The Race to Open-Source Long-Context LLMs Heats Up as Gradient and Crusoe Extend Llama-3's "Context Window" to 1 Million Tokens

In a recent collaboration, AI startup Gradient and cloud compute platform Crusoe extended the “context window” of Llama-3 models to 1 million tokens. The context window determines the number of input and output tokens a large language model (LLM) can process.

VentureBeat Transform 2024 will feature leaders from OpenAI, Chevron, Nvidia, Kaiser Permanente, and Capital One. This three-day event will provide attendees with the opportunity to gain essential insights about GenAI and expand their network.

In a recent collaboration, Gradient and Crusoe extended the context window of Llama-3 models to 1 million tokens. The context window determines the number of input and output tokens a large language model (LLM) can process.

Big tech companies and frontier AI labs are competing to extend the context windows of their LLMs. In a few months, models have gone from supporting a few thousand tokens to more than a million in less than a year. However, LLMs with very long context windows are mostly limited to private models such as Anthropic Claude (200k tokens), OpenAI GPT-4 (128k tokens), and Google Gemini (1 million tokens).

The race to create open-source models with long context windows could reshuffle the LLM market and unlock applications that are not possible with private models.

Gradient works with enterprise customers who want to integrate LLMs into their workflows. Even before Llama-3 came out, the company was facing context pain points in projects they were working on for their customers.