Introducing Bittensor Language Model — A State of the Art 3B Parameter Model For Mobile and Edge Devices

The Opentensor Foundation and Cerebras are pleased to announce Bittensor Language Model (BTLM), a new state-of-the-art 3 billion parameter language model that achieves breakthrough accuracy across a dozen AI benchmarks. What truly sets BTLM apart is its unparalleled versatility. Unlike large models like GPT-3, BTLM seamlessly integrates into mobile and edge devices with as little as 3GB of RAM memory required. This game-changing feature will democratize AI access on a global scale, reaching billions of devices worldwide. Under the hood, BTLM’s exceptional performance is the result of intensive training on the newly created Condor Galaxy 1 (CG-1) supercomputer, generously supported by G42 cloud.

Large Models Don’t Fit on Small Devices

Large language models such as GPT-3 typically have over 100B parameters requiring over 100GB of memory and multiple high-end GPUs in order to perform inference. The release of LLaMA from Meta gave the world high performance models in as little as 7B parameters, making it possible to run LLMs on high end PCs. However even 7B parameter models quantized to 8-bit precision do not fit in many popular devices (i.e., the latest iPhone 13). A 3B parameter model would be compatible with almost all mobile devices, but today’s 3B models substantially underperform their 7B counterparts. Until now.

This May, we began a partnership with Cerebras, a company focused on accelerating AI through innovative hardware and software solutions. Our objective was to train a 3B model that would achieve two key objectives: 1) achieve state-of-the-art accuracy in its class and 2) enable seamless inference with exceptionally long sequence lengths. Today marks a significant milestone as we proudly unveil BTLM, the latest state-of-the-art 3-billion parameter language model, boasting an impressive context window of 65,000 tokens.

By distributing small, efficient models directly onto edge devices, we can harness their latent processing power and reduce the reliance on centralized cloud infrastructure. With Bittensor, edge devices are encouraged to contribute their computational resources and participate in AI tasks through Yuma Consensus¹. By incentivizing edge devices to become active participants in AI inference and training, we lay the foundation for a truly decentralized AI landscape, where collective intelligence and collaborative learning bring about unparalleled advancements in AI capabilities while creating a more democratized and inclusive AI environment for all.

A New Standard for 3B Model Performance

BTLM sets a new standard in 3B parameter model accuracy, outperforming existing models by an average of 12%. This is all the more significant considering BTLM was trained on 627B tokens — significantly lower than RedPajama-INCITE-3B at 800B and OpenLLaMA-3B at 1 trillion tokens.

When looking at individual benchmarks, BTLM scores highest in every category with the exception of TruthfulQA. In RACE-middle it is tied with OpenLLaMA 3B v2.

Not only does BTLM-3B outperform all 3B models, it performs in-line with many 7B models, surpassing the accuracy of RedPajama-INCITE-Base-7B, OpenLLaMA 7B, and Stable-LM-7B. Compared to these models, not only is BTLM-3B on average 7.7% more accurate, it is 2.6x more efficient in memory footprint and inference cost. Given that the most popular model on HuggingFace today is 7B, we believe compacting 7B performance to 3B is an important milestone in enabling AI access on mobile and edge devices.

Bittensor Network Update

Following the implementation of multiple completion scoring mechanisms, the Bittensor network is becoming increasingly unique. Over the last two weeks, 99.97% of unique pairs were generated in relation to all completions. The below chart describes the unique completions per validator per hour over since July 12.

Unique completions per validator per hour in since July 12, 2023

The chart below highlights the full history of diverse completions on the network since the Bittensor 1.1 Validators update.

Unique completions per validator per hour since June 2nd, 2023.

These changes further reinforce Bittensor as the most diverse decentralized neural network in the world. With the network beginning to mature, an app layer is beginning to emerge on top of Bittensor including Chat with Hal, Tao Studio, and BitAPAI Chat.

Bittensor’s ability to absorb any AI technology into its infrastructure enables developers to create applications to suit any modality, including Image, Text, and Audio.

The release of BTLM serves as the next step in democratizing access to AI and positions Bittensor as a leading research institution on the cutting edge of decentralized AI.

If you are interested in learning more about Bittensor, building with us, or leveraging the network for your own use cases, please reach out to us on our Discord. All are welcome — we are excited to build together.

Conclusion

Thanks to a combination of extensive data cleaning and deduplication, variable sequence length training, and muP hyperparameter tuning, BTLM sets a new standard in 3B parameter models and achieves accuracy comparable to many 7B models. Moreover, thanks to its compact size, it can fit in popular mobile devices such as the MacBook Air M1 and iPhone 13.

We graciously thank G42 and IIAI for making the Condor Galaxy 1 supercomputer available for training BTLM. BTLM is available today on HuggingFace. BTLM will be available as a miner on the Bittensor network on July 27, 2023.

(1) Yuma Consensus is a dual proof-of-work and proof-of-stake mechanism which rewards validators with dividends for producing evaluations of miners which are in agreement with the subjective evaluations produced by other validators weighted by stake