Is on-device AI becoming the next priority for OEMs?
AI is quickly becoming part of the automotive technology conversation. But as OEMs explore where AI can create value inside the vehicle, another question is becoming increasingly important: where should that intelligence actually sit?
Today, much of the attention around automotive AI focuses on the models themselves, particularly generative AI and increasingly capable voice assistants. However, running every AI workload in the cloud is not necessarily the most practical approach for a vehicle.
There is a growing case for moving more AI inference into the vehicle itself.
The question is not whether the cloud disappears. It will remain essential. The more interesting question is how far OEMs will go in building vehicles capable of running AI locally, and whether on-device compute becomes something they actively design their future architectures around.
Training and inference are two very different problems
It is useful to first separate AI training from AI inference. Training is the compute-intensive process of developing and improving a model using large datasets. This is likely to remain predominantly in the cloud or data centre.
Inference is different. Once the model has been trained, inference is the process of using it to produce an output from a new input. This requires less compute and creates a much more realistic opportunity to bring AI directly into the vehicle.
The challenge is making the model small and efficient enough to run within the available hardware.
One approach is to use smaller, task-specific models rather than trying to bring a very large general-purpose model into the vehicle. Another is model distillation, where a smaller model is developed to reproduce much of the useful performance of a larger model while requiring fewer computing resources.
As these techniques improve, the range of AI workloads that can realistically run locally should continue to grow.
Why would an OEM want more AI inside the vehicle?
Cost is one of the strongest motivations.
A cloud-based automotive AI service can create costs at several stages. The OEM may pay to transmit vehicle data, process and store it in the cloud, and then pay again for AI inference.
That may be manageable for occasional requests. Scale the same model across millions of vehicles, with increasingly frequent AI interactions, and the economics start to look different.
This creates an incentive to ask a relatively simple question: does every task really need the cloud?
A hybrid approach could allow relatively simple or frequent tasks to run locally, while more demanding requests are passed to a more capable cloud model. The vehicle could also process and clean data locally before deciding what actually needs to be uploaded.
But cost is only one part of the argument.
Privacy and security provide another reason to process more data inside the vehicle. Cars increasingly generate potentially sensitive information, and keeping more of the initial processing local can reduce the amount of raw data that needs to leave the vehicle.
Then there is latency.
For applications that need predictable, real-time responses, relying on a connection to the cloud is difficult. Network availability and round-trip time are not always deterministic enough. Running the workload locally removes that dependency and opens the possibility of using AI across more vehicle functions.
The hardware strategy is also starting to diverge
There are broadly two routes OEMs can take.
One is to use automotive compute platforms from companies such as NVIDIA or Qualcomm and optimise the software and models around the available CPU, GPU and NPU resources.
The other is much more vertically integrated: design silicon specifically around the workloads the OEM intends to run. This is part of the motivation behind custom-silicon investment from companies such as Tesla, Rivian and some Chinese OEMs. The objective is not simply to own a chip, but to optimise the hardware around the software stack and AI workloads expected to run on it.
More broadly, this level of hardware-software integration can potentially improve performance and power efficiency, but developing custom silicon also requires significantly greater investment and in-house capability.
This means custom silicon is unlikely to be the answer for every OEM. For many, working closely with established semiconductor suppliers may remain the more realistic route.
Putting an AI processor in the car is only the beginning
Perhaps the bigger question is what happens to the rest of the vehicle architecture.
Imagine an OEM creates an AI engine that can support several non-safety-critical applications. The cockpit may want to use it for voice interaction. Another service may want to analyse vehicle condition for prognostics. Energy management could potentially use similar capabilities for optimisation.
Suddenly the challenge is no longer simply having enough TOPS.
How do those different services access the AI engine? What APIs are needed? If several workloads request compute at the same time, which gets priority? How much data needs to move across the vehicle network? And which workloads should remain isolated?
Safety-critical functions such as ADAS are also unlikely to simply share the same environment as every other AI application.
BMW’s Neue Klasse provides one example of how the wider architecture is changing alongside compute. Its four central ‘Superbrains’ consolidate functions including automated driving, infotainment and driving dynamics, while a shared software layer connects functions across the architecture. BMW also says data can be pre-processed within the vehicle before being exchanged with the cloud.
This means greater use of on-device AI has implications for the network, middleware, APIs, workload prioritisation and ultimately the wider E/E architecture.
In other words, on-device AI is not just a silicon decision. It is an architectural one.
Will some OEMs move faster than others?
There are reasons to expect different approaches.
Some Chinese OEMs are already investing in custom silicon and have shown a willingness to experiment quickly with deeper AI integration. Tesla and Rivian are also pursuing greater control over their compute stack.
Traditional OEMs can move in the same direction, but the business case will vary significantly by vehicle segment and architecture.
And cost remains the biggest uncertainty.
A premium vehicle has more room to absorb the cost of a powerful central compute platform capable of supporting local AI workloads. Whether the same approach makes sense in an entry-level vehicle is much less clear.
Centralised architectures may eventually help spread these capabilities further down the range, particularly if the same compute platform is already supporting multiple vehicle functions. But how quickly this happens will depend heavily on the cost of the hardware and the value OEMs can extract from it.
So, how much are OEMs really prioritising on-device compute?
Interest is clearly growing, but OEMs are unlikely to move every AI workload into the vehicle.
What looks more likely is a hybrid model.
Large-scale training and the most demanding AI processing can remain in the cloud. More frequent, latency-sensitive, privacy-sensitive or cost-sensitive inference can increasingly move into the vehicle.
Mercedes-Benz offers a recent example of this direction. Through its partnership with Liquid AI, it plans to bring elements of speech, language understanding and reasoning directly into the vehicle, complementing existing cloud-based large language models. The aim is to support faster, more private interactions without requiring continuous communication with the cloud.
The more important question for OEMs may therefore become less about cloud versus vehicle, and more about deciding which workload belongs where.
That decision could ultimately become an important part of how future vehicle architectures are designed.
“The OEMs that take on-device compute most seriously may not necessarily be those quoting the biggest AI performance numbers. A better indication will be how closely they align the models they want to run, the hardware underneath them, and the E/E architecture that allows those capabilities to be used across the vehicle." Fatemeh Faraji, SBD Automotive, Senior Product Owner |  |
How SBD can help
SBD Automotive can help benchmark your position against the wider industry and identify where action is needed most. To explore how these trends impact your strategy, architecture and supplier roadmap, get in touch with SBD Automotive for a deeper discussion. Email info@sbdautomotive.com |
