Introducing the Hugging Face LLM Inference Container for Amazon SageMaker

What changed

The example covers: - Setup development environment - Retrieve the new Hugging Face LLM DLC - Deploy Open Assistant 12B to Amazon SageMaker - Run inference and chat with our model - Create Gradio Chatbot backed by Amazon SageMaker You can find the code for the example also in the notebooks repository. Hugging Face LLM DLC is a new purpose-built Inference Container to easily deploy LLMs in a secure and managed environment. The DLC is powered by Text Generation Inference (TGI), an open-source, purpose-built solution for deploying and serving Large Language Models (LLMs).

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Introducing the Hugging Face Embedding Container for Amazon SageMaker
• Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker
• Llama 2 on Amazon SageMaker a Benchmark

• Introducing the Hugging Face Embedding Container for Amazon SageMaker similarity 0.85
• Deploy GPT-J 6B for inference using Hugging Face Transformers and Amazon SageMaker similarity 0.55
• Llama 2 on Amazon SageMaker a Benchmark similarity 0.49