Small But Mighty: How NovelAI Trained Clio, an Ultra-Performant 3B-Parameter NLP Model, on CoreWeave [Webinar]

NovelAI’s Clio: In Summary

• Lauren Lundy and Navarre Pratt from CoreWeave chat with Eren Dogan, CEO of NovelAI, about how the AI-assisted storytelling and text-to-image synthesis company came to be where it is today.
• NovelAI, part of Anlatan, was one of the first companies to get access to live instances of NVIDIA H100 Tensor Core GPUs and used them to train, in just five days, Clio, its first model trained from scratch.
• Originally intended to be a proof-of-concept model, Clio surpassed performance expectations and laid the groundwork for NovelAI’s next big model release.

‍

We recently sat down with Eren Dogan, CEO of NovelAI, for a half-hour Q&A session to learn more about their latest model Clio, how they built it, and what’s coming next.

NovelAI is a monthly subscription service for AI-assisted authorship, or simply a GPT-powered sandbox for your imagination. CoreWeave has a longstanding partnership with NovelAI since first helping them serve inference for its beta launch in 2021. 

This spring, Anlatan, developers of NovelAI became one of the first companies to deploy the latest NVIDIA H100 Tensor Core GPUs on CoreWeave, which began offering the new instances to select customers in February. 

The team of developers used these supercomputers to train its latest model, Clio, NovelAI’s first model trained from scratch using its own custom datasets. The model is small in comparison to other NLP and LLM models with 3 billion parameters and 1.5 trillion tokens, but it holds a context size of 8192 tokens, which is 4x as much as Novel AI’s previous models. Originally intended to be a proof-of-concept model, Clio has wowed users with its incredible blazing-fast speed and exceptional performance.

Watch the full on-demand webinar to learn more about how NovelAI started, how the team thought through training and fine-tuning for Clio, and more.

‍

‍

Here’s a breakdown of different sections within the video:

• 0:00 Introduction
• 0:48 About Eren and NovelAI
• 7:10         Training Clio
• 10:37 Curating Datasets
• 13:56 Resource Planning & Strategy
• 17:45 NVIDIA H100 GPUs & Software Optimizations
• 22:40 Clio Results
• 26:42 Looking Ahead

‍

Clio is available on all NovelAI subscription tiers and is testable on a free trial. To keep up with the latest from NovelAI and new model releases, go to the Novel AI Blog or join the Novel AI Discord.