So far we have built and trained tiny models from scratch. Now let us use a real pre-trained model: GPT-2. With 1.5 billion parameters and trained on 40GB of internet text, GPT-2 can generate coherent paragraphs, answer questions, and even write code – all running on a laptop.
Loading GPT-2 with HuggingFace Transformers
The HuggingFace Transformers library makes loading and running GPT-2 trivially simple. With just a few lines, you can load the model, tokenizer, and generate text. The library automatically downloads the pre-trained weights from their model hub and handles all the architecture details – embedding lookup, 12 transformer layers, causal masking, and the language model head.
Local Inference
GPT-2 Small (124M parameters) runs comfortably on a CPU, generating about 10 tokens per second. GPT-2 XL (1.5B) needs a GPU for interactive use. The model card on HuggingFace provides detailed specs and usage examples. The key insight: the same architecture we built in Lesson 5, just scaled up by a factor of ~100,000x in parameters.
Run run_gpt2.py to interact with GPT-2. The model was released by Radford et al. (2019) as a demonstration that language models can perform many NLP tasks without task-specific training – what they called “unsupervised multitask learning.”
In the next lesson, we will build a chat harness around GPT-2, turning our model into a conversational agent.
Model Variants and Performance
GPT-2 comes in four sizes: Small (124M parameters, 12 layers), Medium (355M, 24 layers), Large (774M, 36 layers), and XL (1.5B, 48 layers). Each step up improves coherence and knowledge but requires significantly more compute. The Medium model is a good sweet spot for experimentation on consumer hardware. All variants use the same tokenizer (50,257 tokens) and embedding dimension scaling from 768 to 1600.
You can download and run other open models the same way: Mistral 7B, LLaMA 2/3, and Falcon all use the same HuggingFace interface. The key difference is scale: GPT-2 Small has 12 attention heads and 768-dimensional embeddings; GPT-3 has 96 heads and 12,288-dimensional embeddings.
Prompting GPT-2
GPT-2 responds well to prompt structure. A few-shot prompt with examples dramatically improves output quality. For example, to make GPT-2 answer questions, prepend a few question-answer pairs before the actual query. The model learned during training that text often follows patterns, and it naturally continues the pattern you establish. Unlike instruction-tuned models (GPT-3.5, GPT-4, LLaMA-2-Chat), base GPT-2 has no special understanding of instruction formats – it simply continues whatever text it sees. This is why prompting technique matters so much for base models.
The HuggingFace GPT-2 documentation includes examples for text generation, feature extraction, and even fine-tuning on custom datasets. You can fine-tune GPT-2 on your own text in under an hour on a free Google Colab GPU, adapting its style to specific domains like legal text, poetry, or technical documentation.
Running open-source models locally gives you full control over data privacy, customization, and costs. No API keys, no rate limits, no data leaking to third parties. This is increasingly important as organizations recognize the value of keeping sensitive data on-premises while still leveraging advanced AI capabilities.
Leave a Reply