2020 GPT-3
175B Parameters
By Andy Oberlin, Founder, AImpact Nexus · Updated November 16, 2026

Listen to this chapter
OpenAI released GPT-3 with 175 billion parameters, demonstrating unprecedented language understanding and generation capabilities.
Why it matters
Showed that scaling up models could lead to emergent capabilities, launching the era of foundation models.
The story
GPT-3 (Generative Pre-trained Transformer 3) was a major milestone in the development of large language models. With 175 billion parameters, it was 10 times larger than its predecessor, GPT-2, and was the largest language model ever created at the time. GPT-3 demonstrated a remarkable ability to perform a wide range of natural language tasks with little or no task-specific training, a capability known as 'few-shot' or 'zero-shot' learning.
What was going on at the time
GPT-3 demonstrated the power of scale in language modeling. It showed that by simply making models larger and training them on more data, they could achieve remarkable capabilities without task-specific training. This was a major shift from the previous paradigm, where models were typically fine-tuned for specific tasks. The model was developed by OpenAI and released in June 2020, with the API made available to the public in July 2020.
How it worked
GPT-3 is a Transformer-based language model that was trained on a massive dataset of text and code. Key specifications: Architecture is a Transformer decoder (autoregressive language model), 175 billion parameters (96 layers, 96 attention heads, 12,288-dimensional embeddings), training data of approximately 45TB of text from Common Crawl, WebText2, Books1, Books2, and Wikipedia, context window of 2,048 tokens, and training cost estimated at $4.6 million in compute costs. The model's large size and the diversity of its training data are the keys to its impressive capabilities. GPT-3 is able to perform a wide range of tasks, from writing poetry to generating computer code, by simply being given a natural language prompt.
Why it mattered
The release of the GPT-3 API in June 2020 led to a Cambrian explosion of new applications and startups built on top of the model. Developers created applications for content generation, code generation, customer service chatbots, educational tools, creative writing assistants, and hundreds of other use cases. GPT-3 also sparked a public debate about the potential risks and benefits of large language models, including concerns about bias, misuse, and job displacement.
How people reacted
GPT-3's impressive capabilities generated widespread media coverage and public interest. The model's ability to write coherent essays, generate code, and perform complex reasoning tasks surprised even many AI researchers. However, the release also raised concerns about the potential for misuse, including generating misinformation, automating spam, and displacing human workers.
What it left behind
GPT-3 is a landmark in the history of AI. It demonstrated the power of scale in language modeling and helped to usher in the era of large language models. The model's impressive capabilities and its widespread adoption have had a lasting impact on the field of AI and have helped to shape the development of subsequent models, including GPT-4 and ChatGPT. GPT-3 also highlighted important limitations including hallucinations (generating plausible-sounding but incorrect information), lack of common sense (failing at tasks requiring common sense reasoning), bias (reflecting biases present in training data), no true understanding (generating text based on patterns, not true understanding), and inconsistency (giving different answers to the same question).
In their words
“Few-shot learning is the new transfer learning.”
Timeline
- June 2020 GPT-3 paper published and API released in limited beta
- July 2020 GPT-3 API made available to wider public
- 2020-2022 Thousands of applications built on GPT-3 API
- 2021 GPT-3.5 released with improvements
- November 2022 ChatGPT built on GPT-3.5, reaching mainstream audience
- Present GPT-3 continues to power numerous applications
Key players
- Sam Altman
- Ilya Sutskever
- Dario Amodei
Did you know?
- Trained on 45TB of text data
- Cost $4-12 million to train
- Could write code, poetry, and essays
Further reading
- Wikipedia: GPT-3
Complete history and capabilities
- Language Models are Few-Shot Learners
The original GPT-3 paper by Brown et al. (2020)