Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker

What changed

EleutherAI's primary goal is to train a model that is equivalent in size to GPT-3 and make it available to the public under an open license. Over the last 6 months, GPT-J gained a lot of interest from Researchers, Data Scientists, and even Software Developers, but it remained very challenging to deploy GPT-J into production for real-world use cases and products. To make the model more accessible, EleutherAI also provides float16 weights, and transformers has new options to reduce the memory footprint when loading large language models.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing the Hugging Face LLM Inference Container for Amazon SageMaker
• Introducing the Hugging Face Embedding Container for Amazon SageMaker
• Llama 2 on Amazon SageMaker a Benchmark

• Introducing the Hugging Face LLM Inference Container for Amazon SageMaker similarity 0.55
• Introducing the Hugging Face Embedding Container for Amazon SageMaker similarity 0.48
• Llama 2 on Amazon SageMaker a Benchmark similarity 0.42