Nvidia Expands AI Software Development Kit to Accelerate Language Models

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Nvidia aims to dominate the inference side of generative AI

Nvidia, the leading provider of GPUs for training language models, is enhancing its AI-focused software development kit (SDK) to boost the efficiency of large language models (LLMs) and associated tools. The company has integrated support for its TensorRT-LLM SDK on Windows and models such as Stable Diffusion, allowing LLMs to function at a quicker speed. By enhancing the inference process, Nvidia is looking to make a greater impact on the progress and utilization of generative AI.

TensorRT-LLM: Accelerating the LLM Experience

TensorRT-LLM, a component of Nvidia’s SDK, enables LLMs to run more efficiently on Nvidia’s H100 GPUs. This technology is compatible with popular LLMs like Meta’s Llama 2 and AI models such as Stability AI’s Stable Diffusion. By leveraging TensorRT-LLM, users can expect significant performance improvements, especially in the use of sophisticated LLM applications like writing and coding assistants.

Expanding Access and Reducing Reliance on Expensive GPUs

Nvidia plans to make TensorRT-LLM available to the public, allowing anyone to integrate and utilize the SDK for their projects. This move demonstrates Nvidia’s commitment to not only providing powerful GPUs for training and running LLMs, but also offering the necessary software to optimize their performance. The goal is to prevent users from seeking alternative cost-efficient solutions for generative AI.

Competition and the Future of Generative AI

Nvidia currently enjoys a near monopoly in the market for GPUs that train LLMs, resulting in skyrocketing demand and high prices. However, competitors like Microsoft and AMD have announced plans to develop their own chips, aiming to reduce reliance on Nvidia. Additionally, companies such as SambaNova are already offering services that facilitate the running of AI models. While Nvidia remains the hardware leader in generative AI, the company is positioning itself for a future where users are not solely dependent on purchasing large quantities of its GPUs.