How to Build ChatGPT From Scratch: Understanding LLMs Step by Step

Modern AI tools are everywhere.

Developers use ChatGPT to write code, Claude to review pull requests, Gemini to summarize documents, and open-source models to power everything from customer support bots to autonomous agents.

Yet for many people, Large Language Models still feel mysterious.

Terms like:

• Embeddings
• Attention
• Transformers
• Context Windows
• Chain of Thought
• Reasoning Models

appear constantly in AI discussions, but most explanations immediately jump into advanced mathematics.

The problem is that modern LLMs did not appear overnight.

Every major breakthrough solved a limitation in the previous generation of models.

If we follow that progression step by step, modern AI becomes dramatically easier to understand.

The journey looks something like this:

Bigram Model
      ↓
RNN
      ↓
Attention
      ↓
Transformer
      ↓
LLM
      ↓
Reasoning Models

Each stage introduced a new capability:

• Bigram Models learned token relationships
• RNNs introduced memory
• Attention solved long-context limitations
• Transformers removed recurrence
• LLMs scaled the architecture massively
• Reasoning models learned to generate intermediate thinking steps

Let’s start at the very beginning.

Step 1: Building the Simplest Language Model

Imagine a tiny dataset containing developer-related phrases.

Build React Dashboard
Build Next.js Blog
Deploy React Dashboard
Deploy Next.js Blog

The model’s task is simple:

Predict the next token.

Examples:

P(React | Build)

P(Next.js | Build)

P(Blog | Next.js)

P(Dashboard | React)

This is called a Bigram Model.

A toy vocabulary might look like this:

const vocabulary = [
  'Build',
  'Deploy',
  'React',
  'Next.js',
  'Dashboard',
  'Blog',
  '<END>',
];

Internally, the model learns transition probabilities:

Build
  ↓
React   (60%)

Next.js (40%)

This already qualifies as a language model.

But there’s a huge limitation.

The model only sees one token at a time.

It has no memory.

Step 2: Why Memory Matters

Consider these examples:

Build React Dashboard
Build React CRM
Build Next.js Blog
Build Next.js Store

When the model sees:

Dashboard

it no longer remembers that React appeared earlier.

Every prediction depends only on the current token.

This becomes a disaster for longer sequences.

Imagine processing:

Build a multi-tenant analytics platform
with authentication, reporting,
dashboarding, audit logs,
user permissions, notifications,
and billing support

By the end of the sentence, the beginning has effectively disappeared.

Researchers needed a mechanism for memory.

That led to Recurrent Neural Networks.

Step 3: Enter RNNs

Recurrent Neural Networks introduced the concept of a hidden state.

Instead of processing tokens independently, the model carries information forward.

Example:

Build
   ↓
React
   ↓
Dashboard

A simplified implementation:

let hiddenState = [0, 0];

function updateState(input, state) {
  return state.map(
    (value, index) =>
      Math.tanh(
        value + input[index]
      )
  );
}

Each token updates the hidden state.

The model now remembers previous information.

For the first time, context becomes possible.

This was a massive improvement.

But it introduced a new problem.

Step 4: The Bottleneck Problem

Imagine reading a 500-page book.

Then trying to summarize it while keeping only a single sticky note.

That is essentially what an RNN does.

Everything must fit into one hidden state.

As sequences become longer:

• Information gets compressed
• Details disappear
• Relationships are lost

The longer the sequence, the worse the problem becomes.

Researchers started asking a different question:

What if the model didn’t need to remember everything?

What if it could simply look back whenever necessary?

That idea changed AI forever.

Step 5: Attention Changes Everything

Attention is arguably the most important breakthrough in modern AI.

Consider this phrase:

Build React Analytics Dashboard

While processing:

Dashboard

the model can inspect every previous token.

Example attention scores:

Build      → 0.15

React      → 0.55

Analytics  → 0.30

The model is effectively saying:

While generating “Dashboard”, React is the most relevant piece of information.

This is fundamentally different from memory.

Instead of storing everything, the model retrieves what it needs dynamically.

That single idea became the foundation of modern AI.

Step 6: Query, Key, and Value

Attention works through three concepts:

Query
What am I looking for?

Key
What information do I contain?

Value
What information can I provide?

A simplified example:

const query = [0.8, 0.3];

const key = [0.7, 0.4];

const value = [0.9, 0.1];

The model compares Queries and Keys.

A simple similarity calculation:

function similarity(a, b) {
  return a.reduce(
    (sum, value, index) =>
      sum + value * b[index],
    0
  );
}

const score =
  similarity(query, key);

The higher the score, the more attention is assigned.

This mechanism allows the model to decide what information matters.

Step 7: The Transformer Revolution

In 2017, researchers published a paper called:

Attention Is All You Need

The title was intentionally bold.

The argument was simple:

If attention works so well, why keep RNNs at all?

The answer:

Don’t.

The Transformer architecture removed recurrence entirely.

Architecture overview:

Tokens
   ↓
Embeddings
   ↓
Attention Layers
   ↓
Feed Forward Layers
   ↓
Predicted Token

Unlike RNNs:

• Every token can see every other token
• No memory bottleneck exists
• Massive parallel processing becomes possible
• Training becomes dramatically faster

The Transformer quickly became the foundation of modern AI.

Step 8: What Makes an LLM Different?

At a high level, ChatGPT, Claude, Gemini, and Llama all follow the same pattern:

Tokenizer
     ↓
Embeddings
     ↓
Transformer Layers
     ↓
Softmax
     ↓
Next Token

The architecture isn’t fundamentally different.

The scale is.

Instead of:

49 parameters

modern models contain:

Billions

or

Trillions

of parameters

Yet they still perform the same task:

Predict the next token.

Input:

How do I deploy a React application to

Predictions:

Vercel    72%

Netlify   12%

AWS        8%

Railway    5%

The model selects a token.

Then predicts another.

Then another.

And eventually entire paragraphs emerge.

This raises an important question.

If models only predict tokens, where does reasoning come from?

Step 9: Why Chain of Thought Works

Modern LLMs are trained on enormous amounts of human reasoning.

Training data often follows patterns like:

Problem

Analysis

Intermediate Steps

Answer

Example:

Build a SaaS platform

Need authentication
↓
Need database
↓
Need billing
↓
Need deployment

Recommended stack:

Next.js
PostgreSQL
Better Auth
Stripe

The model repeatedly observes that difficult problems are followed by intermediate reasoning.

Eventually it learns:

Complex tasks often require intermediate steps.

This is why prompts such as:

Think step by step.

work surprisingly well.

The model generates reasoning tokens.

Those tokens become part of the context.

The model then reasons using its own generated text.

The process looks like:

Generate Thought
       ↓
Thought Becomes Context
       ↓
Read Context
       ↓
Generate Better Thought
       ↓
Repeat

This is the foundation of Chain of Thought prompting.

Step 10: Generalization Beats Memorization

Many people believe LLMs simply memorize data.

If that were true, they would fail whenever they encountered something new.

Instead, they learn relationships.

Training examples:

Build React Dashboard

Build React CRM

Build Next.js Blog

Build Next.js Store

Eventually embeddings might look like:

React    → [0.81, 0.20]

Vue      → [0.79, 0.24]

Angular  → [0.77, 0.29]

Meanwhile:

Dashboard → [0.14, 0.88]

Analytics → [0.16, 0.85]

Admin     → [0.19, 0.82]

The actual numbers don’t matter.

The relationships do.

Frameworks cluster together.

Dashboards cluster together.

Databases cluster together.

Languages cluster together.

This allows models to solve new tasks.

For example:

Build React Analytics Platform

might never appear in training data.

Yet the model can generate useful output because it understands patterns rather than memorizing sentences.

Why “LLMs Just Predict The Next Token” Is Misleading

Technically, the statement is correct.

Every output token is selected by predicting the most likely continuation.

But that explanation hides almost everything interesting.

It’s similar to saying:

A CPU just executes instructions.

or:

A browser just renders pixels.

The statement is true.

It just doesn’t explain the behavior that emerges from the system.

Modern LLMs combine:

Massive Datasets
        +
Embeddings
        +
Attention
        +
Transformers
        +
Scale
        +
Reasoning

Together they produce:

• Code
• Articles
• Research
• Planning
• Translation
• Analysis
• Agents

Each individual step predicts a token.

The intelligence emerges from the system as a whole.

The Complete Mental Model

If there’s one diagram worth remembering, it’s this:

Bigram Model
      ↓
Learns token relationships

RNN
      ↓
Adds memory

Attention
      ↓
Looks back dynamically

Transformer
      ↓
Removes recurrence

LLM
      ↓
Scales to billions of parameters

Reasoning Models
      ↓
Generate intermediate thoughts

Every major breakthrough solved a limitation in the previous generation.

That progression eventually produced ChatGPT, Claude, Gemini, Llama, and the models powering today’s AI revolution.

Final Thoughts

The easiest way to understand modern AI is not to start with ChatGPT.

Start with the simplest possible language model.

Then follow the evolution:

Bigram
   ↓
Memory
   ↓
Attention
   ↓
Transformer
   ↓
Scale
   ↓
Reasoning

Viewed this way, modern AI stops looking like magic.

It’s decades of engineering, research, and optimization built around one deceptively simple objective:

Predict the next token.

And from that simple objective, an entirely new computing paradigm emerged.