Learning to play Minecraft with Video PreTraining

What changed

We trained a neural network to play Minecraft by Video PreTraining (VPT) on a massive unlabeled video dataset of human Minecraft play, while using only a small amount of labeled contractor data. With fine-tuning, our model can learn to craft diamond tools, a task that usually takes proficient humans over 20 minutes (24,000 actions). Our model uses the native human interface of keypresses and mouse movements, making it quite general, and represents a step towards general computer-using agents.

Why it matters

A concrete addition to Practical AI: Tools, Models & Frameworks: it changes what's available to builders today rather than being general commentary.

How it compares

Related prior coverage to compare against:

• Video generation models as world simulators
• Fine-tuning GPT-3 to scale video creation
• Fine-tuning GPT-2 from human preferences

• Video generation models as world simulators similarity 0.19
• Fine-tuning GPT-3 to scale video creation similarity 0.17
• Fine-tuning GPT-2 from human preferences similarity 0.15