Next-gen VPA AI implementations, looking at the competitive landscape of existing challenges and causes

From my recent attendance to the AI & SDV European Summit in Stuttgart, something came out clearly: AI has become one of the core enabler of the software-defined vehicle, underpinning everything from ADAS and automated driving to predictive maintenance and smart manufacturing.

Machine learning, deep learning and computer vision turn high‑volume sensor and usage data into real‑time decisions, allowing OEMs to optimize energy management, tailor driving dynamics, and continuously improve features over the air. In parallel, “smart cockpit” concepts integrate AI-driven perception, voice, and personalization to create a coherent, adaptive human–machine interface that can evolve over the vehicle’s lifetime.

Within this architecture, in‑car VPA systems that use AI are rapidly shifting from “nice to have” voice control to the primary orchestration layer for the cabin experience. Surveys ([1], [2]) show that a significant share of buyers now factor in-car voice assistants into purchase decisions, and that frequent usage deepens the OEM–driver relationship while generating behavioral data that can feed continuous improvement and new service models.

In-car Virtual Personal Assistant (VPA) is becoming the single most important user entry point in the car. Shaping cabin UX, the OEM's monetization model and the perceived intelligence of the vehicle. That makes the VPA the strategic high ground every OEM is now racing to claim.

Competitive landscape

This is reflected by the investment and deployment of varied VPA systems by different OEMs:

As systems become more mature, they start relying on using LLMs to support user requests (VPA 3.0), augmenting existing use cases that a local machine learning system could not handle.

However, having a single LLM system to handle all new requests poses several issues:

1. Required hardware: Comparable model to GPT-4 require anywhere between 70B and 700B parameters, which puts a very strong requirement to embedded hardware, which most legacy architectures ca simply not supply

2. Connectivity: Although a simple solution would be to handle such model on the cloud connection becomes an issue, with users expecting the VPA system to be usable in all conditions, not only where connectivity is available (a similar issue found with embedded maps and navigation)

3. Latency: Research from different OEMs and suppliers align that responses must be between 500ms and 1.5s to be seen as natural ([3], [4])

At this stage is where multi-agent comes into place (VPA 4.0), companies offering state of the art system such as BMW and Cerence agree that intelligent workload partitioning across edge and cloud, and a swarm of smaller specialized models (1B-7B parameters) will solve the three challenges above.

Challenges with implementation

However, UX testing from latest implementations highlight that there are challenges being faced by this state of the art systems:

Even latest implementations are struggling in some cases with the reliability and the AI capability of their systems. Here are some examples of issues:

• Slow handover: usually OEMs make requests go through the embedded system by default (for easiness and reduce token consumption), however when the embedded system cannot resolve the request it hands over to the off-board system, starting the process again and creating double the wait for the consumer (even when the system explicitly highlights the handover, the waiting time feels too long until the action)

• Contradictions between agents: Although OEMs prevent separate critical functions and driver data to be accessed and used by the cloud-based LLM, this means that sometimes a request intended to be solved by the embedded system is wrongly handled by the off-board LLM and as such system does not have the required data, the request is not addressed.

• No integral memory: Although independently the systems have memory of previous requests to support multi-step conversations, sometimes the system does not hand-over correctly those follow up requests to the correct agent, hence answers stop.

Even if the individual agents have been tested and are capable of perform certain use cases, the end user is still sees poor reliability and an apparent poor AI system, when in reality it boil down to  a poor orchestration of the different agents or VPA systems.

Why is poor orchestration an issue?

By discussing with VPA system providers, system integrators and OEMs, we found the following points being commonly mentioned:

1. No strict requirements for VPA system development: Compared to other automotive subsystems, specification documents from OEMs are still very vague or non-existant. VPA SW developers require better defined requirements and use cases to develop better systems.

2. Focus on defined use cases: Both VPA providers and OEMs still focus too much on clearly defined use cases, e.g. navigation from A to B, HVAC adjustments. However, by offering LLM capability, they have adjusted the user expectations to any use case (general knowledge, advice, contextual questions).

3. No clear systems integration testing: VPA providers assume that OEMs will develop the orchestration rules, whereas OEMs sometimes believe VPA providers will have those rules already embedded in their development (e.g. when a request is handled by one of the systems, what data types are out of bounds for off-board systems)

What can be done to mitigate these issues?

1. Define requirements in a structured and detailed manner: Documentation should define not only systems constraints but also type of data available, necessary use cases, orchestration rules and available control vectors (what subsystems and components can the system actuate upon).

2. More stress testing less ODD testing: The goal of a validation engineer should not be to simply test a static list of use cases but to try to break the system to understand how multi-agent systems interact on limit cases. Once the system is broken understand how and what and address it.

3. Treat orchestration as the new critical activity: The decisive layer of automotive AI VPA is the in-vehicle orchestration stack, including the on-device runtime, the swappable LLMs it hosts, app functions and agent-to-tool interfaces.

“The next wave of VPA implementations will be led by brands that orchestrate and explain clearly to their developers what is required from their systems. We at SBD have developed testing methodologies and stress testing for VPA systems to support is. If you are interested in knowing more about this or what other use cases are being augmented by AI and the existing and future challenges of such implementation, contact me. I will be happy to have a conversation on this. Edward Páez - SBD Automotive Senior Consultant

Sources:

[1] SBD internal research, 2026

[2] SoundHound AI Independent Survey, 2024, Link

[3] Sage Journals, The Importance of Timing - An Expert Evaluation on Latencies for In‑Vehicle Voice Assistants”, 2024, Link

[4] Google Duplex: An AI System for Accomplishing Real-World Tasks Over the Phone, 2018 Link