OpenAI explores Nvidia chip alternatives as AI inference becomes key focus

OpenAI has been exploring alternatives to some of Nvidia’s artificial intelligence chips since last year, reflecting concerns over how existing hardware performs on certain inference-heavy workloads, according to a report by Reuters.

The shift is showing a growing focus on inference, the stage at which trained AI models generate responses to user queries, as a critical battleground in the next phase of artificial intelligence. While Nvidia continues to dominate chips used to train large-scale AI models, inference performance is becoming increasingly important as AI tools such as ChatGPT scale across consumer and enterprise use cases.

OpenAI’s reassessment comes even as it remains heavily reliant on Nvidia hardware and continues discussions around a potential investment by the chipmaker. Nvidia had earlier signalled plans to invest up to $100 billion in OpenAI, a deal that was expected to conclude quickly but has since faced delays.

During this period, OpenAI has broadened its hardware strategy by engaging with other chipmakers, including AMD, as it looks to diversify its computing stack. People familiar with the matter told Reuters that changes in OpenAI’s product roadmap, particularly its growing emphasis on inference speed, have complicated negotiations with Nvidia.

Also read: Elon Musk merges SpaceX and xAI as company plans space-based AI data centres

Nvidia has pushed back against suggestions of strain in the relationship. Chief Executive Officer Jensen Huang recently dismissed reports of tension, reiterating that the company remains committed to a significant investment in OpenAI. Nvidia has also said its chips continue to deliver leading performance and cost efficiency for inference at scale.

OpenAI, meanwhile, has publicly maintained that Nvidia powers the majority of its inference infrastructure. Chief Executive Sam Altman has described Nvidia’s products as the best AI chips available and said OpenAI expects to remain a major customer for the long term.

However, people cited in the Reuters report said OpenAI has been dissatisfied with the speed at which Nvidia’s hardware handles certain tasks, particularly software development tools and scenarios where AI systems interact directly with other software. OpenAI is said to be evaluating specialised hardware that could eventually support a portion of its inference computing needs.

As part of that effort, OpenAI has discussed working with startups such as Cerebras and Groq, which design chips optimised for inference through large amounts of on-chip memory. Such architectures can reduce delays associated with external memory access, improving response times for chatbots and AI-powered coding tools.

Inference workloads typically place heavier demands on memory access than on raw computation, exposing limitations in general-purpose GPUs that rely on external memory. This has made memory-centric chip designs increasingly attractive as AI applications move from training to real-time deployment.

The performance challenges have been particularly visible in Codex, OpenAI’s AI-powered coding product, where speed is a key requirement for professional users. Altman has previously said that customers place a premium on fast response times for coding tasks.

Also read: Jio Studios acquires 50.1% stake in Sikhya Entertainment for Rs 150 crore

Rival AI models such as Anthropic’s Claude and Google’s Gemini benefit from deployments that rely more heavily on custom-built chips, including Google’s tensor processing units, which are designed specifically for inference and reasoning.

Nvidia has responded by expanding its technology portfolio, including licensing deals and talent acquisitions aimed at strengthening its position in inference-focused hardware. As AI shifts towards real-time reasoning and large-scale deployment, competition in inference chips is shaping up to be a critical test of Nvidia’s long-held dominance.