Learning to Reason: When LLMs Stopped Guessing Words and Entered the Test-Time Compute Era

In September 2024, OpenAI announced o1 with a short description: a model trained with reinforcement learning to reason, one that produces a long internal chain of thought before it answers. Read casually, it sounds like a normal product post. But there is one technical detail buried under the headline worth slowing down for: o1’s performance improves when you let it think longer, and it also improves when you train it longer. Two completely different compute axes, not one.

This is where most write-ups about o1 slip. They put o1’s 74% on AIME next to GPT-4o’s 12%, then conclude “the new model is stronger.” True, and technically meaningless. The real questions are: where did the compute go, who pays for it, and how does that bill reshape your backend.

This post covers three things: the second scaling curve that lives at inference time, the RL loop inside the model, and the least fun part of shipping a reasoning model to production, where latency is measured in minutes instead of milliseconds.

The pre-training wall is not a rumor

Before talking about test-time compute, it helps to be clear about why it showed up. Not because the idea is elegant, but because the old road got narrow.

Classical scaling laws (Kaplan et al., 2020) say loss falls as a power law when you add parameters, data, and compute. Chinchilla (Hoffmann et al., 2022) corrected the ratio: for compute-optimal training you want roughly 20 tokens per parameter. A 70B model needs about 1.4 trillion tokens. A 400B model needs around 8 trillion. That is where arithmetic meets geology. The supply of high-quality text on the internet is finite. Serious estimates land somewhere in the low single-digit trillions of tokens after deduplication and filtering, and you cannot get more by crawling faster.

The consequence: doubling compute the old way requires doubling your data. There is no more data. You can train multiple epochs over the same corpus, but past four epochs returns collapse and you start memorizing. You can generate synthetic data, but synthetic data from an older model carries its own distribution problems.

Put differently: the “bigger model, more data” axis still works, but the return on each dollar spent is getting worse. That is the pressure that made another axis worth trying.

The second curve: paying the model to think longer

The idea behind test-time compute is not new. Self-consistency (Wang et al., 2022) samples multiple answers and takes a majority vote. Process reward models (Lightman et al., “Let’s Verify Step by Step,” 2023) score each reasoning step instead of only the final answer. Snell et al. (2024) showed that on some problems, spending extra compute at inference beats adding parameters.

What o1 did was turn that from a technique you bolt on from outside into a capability baked into the model.

graph TD
    subgraph "Curve 1 - Pre-training, train-time compute"
        P1["Double the compute"] --> P2["Loss falls along a power law"]
        P2 --> P3["Need twice the tokens"]
        P3 --> P4["Data wall - high-quality text runs out"]
        P4 --> P5["Diminishing ROI, but not zero"]
    end

    subgraph "Curve 2 - Test-time compute, inference"
        T1["More thinking tokens"] --> T2["Higher accuracy on hard problems"]
        T2 --> T3["Wall-clock grows linearly"]
        T3 --> T4["Output cost grows"]
        T4 --> T5["Backend has to be redesigned"]
    end

    P5 -.->|"Pressure to shift"| T1

These two curves do not replace each other. They stack. A weak model stays weak even with ten minutes of thinking, because it lacks the underlying knowledge to reason over. But a strong model constrained to a single forward pass can go a long way once you let it emit a few thousand reasoning tokens.

The eval table in the announcement’s appendix shows that slope clearly enough to be uncomfortable:

Look at the AIME row. This is an exam where the older model solved an average of 1.8 out of 15, roughly the 12% OpenAI quotes in prose; the appendix lists 9.3 pass@1 and 13.4 cons@64, the gap comes from sampling setup. o1 solves 11.1 of 15 with a single sample. Push to 64 samples and take consensus, you get 12.5 of 15. Rerank 1,000 samples with a learned scoring function and you get 13.9 of 15, which puts it in the top 500 students nationally and above the USA Mathematical Olympiad cutoff.

Those three numbers correspond to three very different cost profiles, and that is where you need to stay awake.

74% is one model call, costing you extra thinking tokens. 83% is 64 model calls. 93% is 1,000 calls plus a verifier. If you copy that 93% onto a pitch deck and design a system around it, you are designing for a workload three orders of magnitude more expensive than what you assumed. In practice, 64 samples only makes sense when the problem has high value and no interactive deadline: tax routing, contract review, an optimization problem you run as an overnight batch. For a chatbot, one sample is all you can justify.

There is also a small detail worth noticing: the MMLU row. o1-preview scores 92.3, o1 scores 90.8. The number goes down. Reasoning training is not free. It concentrates the model’s capacity on problems that need multi-step derivation, and sometimes pays for it with a point or two on tasks that lean on pure recall. o1 still beats GPT-4o on 54 of 57 MMLU subcategories, but losing to its own preview on a general benchmark is a reminder that this is still a trade-off, not a magic trick.

Inside the loop: RL teaches the model to correct itself

The most interesting part of the announcement is the section where the model notices it is wrong. OpenAI published a substitution-cipher example, and o1-preview’s chain of thought reads like the debug log of someone fumbling toward an answer:

“First, what is going on here?”

“Wait a minute.”

“Hmm.”

“It seems that the ciphertext words are exactly twice as long as the plaintext words. (10 vs 5, 8 vs 4, 4 vs 2, 8 vs 4)”

“Idea: Maybe we need to take every other letter or rebuild the plaintext from the ciphertext accordingly.”

GPT-4o on the same prompt counts the letters, splits the words, then gives up and asks the user for more decoding rules. o1-preview tries a hypothesis, finds it does not fit, backs out, tries another, and eventually discovers the rule of adding adjacent letters in the alphabet.

None of those behaviors (decomposing hard steps, noticing a wrong direction, backtracking, trying a different approach) were programmed. They emerged from optimization pressure.

graph TD
    Q["Input problem"] --> P["Policy LLM after RL"]
    P --> S["Emit the next reasoning step"]
    S --> D{"Does this step hold up?"}
    D -->|"Yes"| N{"Enough to answer yet?"}
    D -->|"Wrong or off track"| B["Backtrack, drop the branch, try another route"]
    B --> S
    N -->|"Not yet"| S
    N -->|"Yes"| A["Final answer"]
    A --> R["Verifier scores the trajectory"]
    R --> U["Policy gradient updates weights"]
    U -.->|"Offline training loop"| P

The core mechanism fits in one sentence: the policy samples a trajectory made of a reasoning chain plus an answer, a verifier scores that trajectory, and policy gradient raises the probability of trajectories that led to correct answers while lowering the ones that did not. After enough rounds, the model does not just learn correct answers. It learns the shape of the process that produces them.

Why supervised fine-tuning cannot do this

Supervised fine-tuning needs pairs of (input, ideal output). For reasoning, the “ideal output” is a 5,000-token chain of correct derivation. Nobody can write millions of those, and even if they could, each one is only one of countless valid paths to the answer. Training on them teaches the model to imitate a particular reasoning style, not to find its own.

RL flips the problem: you do not need to know what a correct process looks like, only whether the result was correct. For math, compare the answer. For code, run the unit tests. For some logic problems, check the constraints. This is why RL works well in domains with automatic verifiers, and why it is hard to apply to open-ended writing where “correct” has no function definition.

The credit assignment problem

If it were that easy, everyone would ship it. The real difficulty is that the reward arrives once, at the end of a 5,000-token chain. What did token 2,317 contribute to the final answer being right? Nobody knows. This is long-horizon sparse reward, the hardest problem in RL.

There are three families of mitigation, and o1 plausibly uses all of them:

Process reward models score intermediate steps. Expensive, because you need per-step labels, but the signal is far denser.

Relative value across samples in the same batch: if 8 rollouts of one problem produce 3 correct answers, the advantage of each token inside the correct rollouts goes up. This is the basis of algorithms in the GRPO family, which skip a separate critic network and save a meaningful amount of VRAM.

Shaping: reward intermediate behaviors that correlate with success, such as re-checking the result at the end of a chain. This is the dangerous one, because it invites reward hacking. The model learns to write “let me double check” a lot without actually checking anything.

Why the chain of thought is hidden

OpenAI chose not to expose the raw chain of thought, only a model-generated summary. The announcement gives three reasons: user experience, competitive advantage, and the option to monitor the chain of thought in the future.

The third reason is the technically interesting one. If CoT is used to monitor the model (detecting signs of manipulating the user, of dodging policy), then the CoT has to reflect what the model actually thinks. To stay honest it cannot have policy compliance trained onto it, because that teaches the model to hide its real reasoning. Combine the two requirements: the CoT has to stay free-form, and it cannot be shown to users.

For backend engineers, the practical takeaway is blunt: do not build features that parse the chain of thought. It can change, get truncated, or disappear in a later model. If you need the reasoning, ask the model to put it in the final answer. Do not read its diary.

Consequence one: reasoning is serial, and latency does not scale sideways

This is the part every write-up about o1 should spend more words on.

Decoding is autoregressive. Token n+1 depends on token n. You cannot split 6,000 reasoning tokens across 8 GPUs and get an answer in one-eighth the time. No parallel scan rescues a single request.

So the wall-clock of one request is roughly:

latency ≈ prefill_time + (reasoning_tokens + answer_tokens) / decode_speed

At 100 tokens per second and 6,000 reasoning tokens, the physical floor is 60 seconds. Not “slow.” Not improvable. The more the model thinks, the longer it takes, along a straight line.

There are exactly three ways to move that number, and all three cost something.

Fewer reasoning tokens. This is the most effective knob, and it is why reasoning_effort showed up as a parameter later. But fewer tokens means less accuracy, which is the exact thing you were paying for.

Faster decode. Needs better hardware or a smaller batch. A smaller batch means lower total throughput, which means a higher cost per token. This is a direct latency-versus-dollars trade, and there is no free lunch hiding in it.

Redesign the UX. This is what most real systems end up doing.

The key point: batching improves throughput, not latency. If 100 requests arrive at once, a batch of 100 finishes in roughly the same total time as a batch of 1, but each individual request still waits the full 60 seconds. That is why high GPU utilization and low p99 latency are mutually exclusive goals when serving reasoning models.

And this is where Little’s Law hits you:

required concurrency = arrival_rate × latency

10 requests per second at 300 seconds of latency needs 3,000 in-flight requests. Three thousand open connections, three thousand slots in the provider connection pool, three thousand live state records. If your plan was “just hold the HTTP request open and wait,” that number is the reason it will not work.

One more layer: provider rate limits are usually expressed in tokens per minute, not requests. 10 rps at 8,000 reasoning tokens each is 4.8 million TPM. That exceeds the quota on nearly every standard tier, and it is why reasoning workloads need quota increases negotiated before launch, not after.

Consequence two: reasoning tokens are output tokens, and you pay for tokens you never see

Second shock. Reasoning tokens are billed as output. Invisible to the user, present in the invoice, and present in usage.completion_tokens_details.reasoning_tokens.

The cost formula:

cost ≈ prompt_tokens × p_in + (reasoning_tokens + answer_tokens) × p_out

A typical request: 2,000 prompt tokens, 8,000 reasoning tokens, 600 answer tokens. 10,600 total, 81% of it reasoning. If p_out runs three to five times p_in, which is normal across major providers, the reasoning portion is essentially the entire bill.

Scale it: a million requests a month at 8,000 reasoning tokens each is 8 billion output tokens. Multiply by your unit price. The result is usually big enough to kill a feature before it finds users.

Two operational consequences follow.

First, with o1 the chain of thought is not carried across API calls. Every turn reasons from scratch. That is very different from a standard chat model, where you resend history and the model emits a few dozen new tokens. With a reasoning model, multi-turn cost grows much faster than linearly. If your product is a chat interface, seriously consider enabling reasoning only on the first turn, or only on questions a classifier flags as hard.

Second, there are no logprobs. Reasoning models do not return token probabilities and do not let you set temperature. Which means you cannot route on confidence. A router has to work from task type, or from an independent verifier that runs after the fact. “Trust the model when it feels sure” is not available to you.

When not to use a reasoning model

Plenty of tasks look like they need reasoning but only need lookup and reformatting. Summarizing email. Classifying tickets. Extracting fields from a PDF. Route those through a model that thinks for 60 seconds and you are twenty times slower and thirty times more expensive at identical quality.

A workable rule: reasoning models earn their cost when a problem has many interdependent steps and a checkable answer. Math, code, logical argument, constrained planning, debugging. Outside that, the cheaper faster model wins.

The most common production architecture is a cascade: a small model handles everything, and escalates to the reasoning model only on specific signals, such as a classifier marking the task hard, a unit test failing, or a verifier flagging inconsistent output. Cascades cut 70-90% of cost versus sending everything to the big model, and they are the reason published benchmarks always look better than the actual invoice.

Backend architecture for an API that takes five minutes to answer

Now the practical part. You have an endpoint calling a reasoning model. p50 is 45 seconds, p99 is 4 minutes. Where does a normal request-response architecture break?

It breaks at the load balancer first. Default idle timeouts are 60 seconds on many LBs and 100 seconds on some CDNs. Your request is severed before the model finishes. Then the client retries, and since nothing was persisted yet, each retry is a fresh model call. Three retries is three times the money for one question. That is a retry storm, and it is the fastest route from a latency incident to a billing incident.

It breaks at deploy time: every rolling deploy kills all waiting connections. At five minutes per job, you just destroyed a large amount of already-paid-for work.

It breaks at autoscaling: you scale on in-flight request count, but with this workload each instance holds very few concurrent requests because each one occupies a slot for five minutes. The metric you want is queue depth, not RPS.

The fix is to decouple the job lifecycle from the HTTP request lifecycle entirely. The familiar async job pattern, except this time it is mandatory rather than an optimization.

graph TD
    C["Client - web, mobile, internal service"]

    subgraph "API Layer - stateless"
        API["API Gateway"]
        H["POST /v1/reasoning-jobs"]
        S["GET /v1/jobs/:id/stream - SSE"]
    end

    subgraph "State Layer"
        DB[("`Postgres - jobs, outbox,
        idempotency key`")]
        Q[["`Redis Streams
        job queue`"]]
    end

    subgraph "Worker Tier"
        W1["`Worker pool - Go or Python
        deadline budget, heartbeat lease`"]
    end

    subgraph "Model Boundary"
        LLM["`Reasoning model API
        30 seconds to 10 minutes`"]
    end

    C -->|"1 - create job"| H
    H -->|"2 - write DB, return 202 with job_id"| DB
    H -->|"3 - enqueue"| Q
    Q -->|"4 - consume"| W1
    W1 -->|"5 - call model, hold lease"| LLM
    LLM -->|"6 - return result"| W1
    W1 -->|"7 - persist result"| DB
    DB -->|"8 - push status"| S
    S -->|"9 - SSE"| C
    W1 -->|"HMAC-signed webhook"| C

Four design decisions in that diagram, and the reason for each.

202 instead of 200. The POST writes one row and returns immediately. The client has a job_id within 50ms. No timeout at the load balancer can kill it.

Mandatory idempotency key. Clients retry. Without a key, a retry creates a second job and a second model call, doubling the bill. The key belongs in a unique index at the database layer, not in check-then-insert logic in the application, because check-then-insert loses under race.

Heartbeat lease instead of a fixed timeout. A job running for 8 minutes must not have its lease reaped and handed to another worker. The worker renews periodically. The reaper timeout has to be at least three times the heartbeat interval, or a single GC pause will let a job get stolen.

Signed webhooks, with polling as fallback. Not every client can receive a webhook, and webhooks get lost. Results should live in the database for at least 24 hours so late pollers still get them.

Go: idempotent enqueue and a leased worker

package reasoning

import (
	"context"
	"crypto/sha256"
	"encoding/hex"
	"errors"
	"time"
)

var ErrDuplicate = errors.New("duplicate job")

// Enqueue creates the job idempotently. request_hash carries a unique index
// in Postgres, so a client retry gets the existing job back instead of
// burning another model call.
func (s *Service) Enqueue(ctx context.Context, tenantID, prompt string) (*Job, bool, error) {
	sum := sha256.Sum256([]byte(tenantID + "\x00" + prompt))
	hash := hex.EncodeToString(sum[:])

	job := &Job{
		ID:          newULID(),
		TenantID:    tenantID,
		RequestHash: hash,
		Prompt:      prompt,
		Status:      StatusQueued,
		Deadline:    time.Now().Add(12 * time.Minute),
	}

	if err := s.store.InsertJob(ctx, job); err != nil {
		if isUniqueViolation(err) {
			existing, getErr := s.store.GetJobByHash(ctx, tenantID, hash)
			return existing, true, getErr
		}
		return nil, false, err
	}

	// Outbox: the enqueue happens in the same transaction as the insert
	// at the store layer, so a crash cannot lose the job.
	if err := s.queue.Publish(ctx, job.ID); err != nil {
		return job, false, err
	}
	return job, false, nil
}

const (
	modelTimeout   = 10 * time.Minute
	leaseTTL       = 45 * time.Second
	heartbeatEvery = 15 * time.Second
	maxInflight    = 64 // upper bound on concurrent provider connections
)

func (w *Worker) process(parent context.Context, id string) {
	// The job deadline must be shorter than the HTTP client timeout, and the
	// HTTP client timeout must be shorter than the accumulated lease TTL.
	// Three nested deadline layers. Do not set them equal.
	ctx, cancel := context.WithTimeout(parent, modelTimeout)
	defer cancel()

	job, err := w.store.Claim(ctx, id, leaseTTL)
	if err != nil {
		return // another worker claimed it; not an error worth logging
	}

	stop := w.heartbeat(ctx, job.ID, heartbeatEvery)
	defer stop()

	res, err := w.llm.Reason(ctx, job.Prompt, w.model)
	if err != nil {
		if isRetryable(err) && job.Attempts < 3 {
			_ = w.store.Release(ctx, job.ID)
			_ = w.queue.RetryLater(ctx, job.ID, backoff(job.Attempts))
			return
		}
		_ = w.store.Fail(ctx, job.ID, err.Error())
		_ = w.notify.Failed(ctx, job)
		return
	}

	_ = w.store.Complete(ctx, job.ID, res, res.Usage.ReasoningTokens)
	_ = w.notify.Success(ctx, job) // best effort, retried separately
}

func (w *Worker) heartbeat(ctx context.Context, id string, every time.Duration) func() {
	t := time.NewTicker(every)
	done := make(chan struct{})
	go func() {
		for {
			select {
			case <-done:
				t.Stop()
				return
			case <-t.C:
				// Renew the lease. A failure here is only logged, never fatal:
				// one network blip does not mean the job is dead.
				_ = w.store.ExtendLease(ctx, id, leaseTTL)
			}
		}
	}()
	return func() { close(done) }
}

Python: job creation endpoint and SSE stream

import asyncio
import hashlib
import json

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
from pydantic import BaseModel

app = FastAPI()


class JobRequest(BaseModel):
    prompt: str


@app.post("/v1/reasoning-jobs", status_code=202)
async def create_job(body: JobRequest, request: Request):
    tenant = request.headers["x-tenant-id"]

    # Idempotency-Key comes from the client. Fall back to a payload hash so
    # clients without the header still cannot double-charge themselves on retry.
    key = request.headers.get("idempotency-key") or hashlib.sha256(
        (tenant + "\x00" + body.prompt).encode()
    ).hexdigest()

    existing = await jobs.get_by_key(tenant, key)
    if existing is not None:
        return {"job_id": existing.id, "status": existing.status, "replayed": True}

    job = await jobs.create(tenant=tenant, key=key, prompt=body.prompt)
    await queue.put(job.id)
    return {"job_id": job.id, "status": "queued"}


@app.get("/v1/jobs/{job_id}/stream")
async def stream(job_id: str, request: Request):
    async def gen():
        last_seen = None
        while True:
            if await request.is_disconnected():
                return

            job = await jobs.get(job_id)
            if job is None:
                yield 'event: error\ndata: {"error":"not_found"}\n\n'
                return

            # Metadata only. Never stream the chain of thought: it is hidden
            # by design and can change shape between model versions.
            marker = (job.status, job.elapsed_ms // 2000)
            if marker != last_seen:
                last_seen = marker
                payload = {"status": job.status, "elapsed_ms": job.elapsed_ms}
                yield f"event: status\ndata: {json.dumps(payload)}\n\n"

            if job.status in ("succeeded", "failed"):
                yield f'event: done\ndata: {{"status":"{job.status}"}}\n\n'
                return

            await asyncio.sleep(2)

    return StreamingResponse(
        gen(),
        media_type="text/event-stream",
        headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
    )

import asyncio

async def worker(sem: asyncio.Semaphore) -> None:
    while True:
        job_id = await queue.get()          # BRPOP or XREADGROUP
        async with sem:                      # cap concurrent provider calls
            job = await jobs.claim(job_id, lease_s=45)
            if job is None:
                continue                     # another worker took it

            hb = asyncio.create_task(heartbeat(job_id, every_s=15))
            try:
                # Timeout must sit above the model p99 and below the
                # accumulated lease. Three layers, never equal.
                result = await asyncio.wait_for(
                    call_reasoning_model(job.prompt), timeout=600
                )
                await jobs.complete(job_id, result)
                await notify_webhook(job, result)
            except asyncio.TimeoutError:
                await jobs.fail(job_id, "model_timeout")
            except Exception as exc:
                await jobs.fail(job_id, repr(exc))
            finally:
                hb.cancel()

Two small details in that code get skipped a lot. X-Accel-Buffering: no is mandatory behind nginx; without it SSE gets buffered and the user sees nothing until the stream closes. And request.is_disconnected() has to be checked inside the loop, otherwise every closed tab leaks a task until the job finishes.

The timeout table, and why the layers must nest

The rule: each layer must be at least twice the layer inside it. If the HTTP timeout and the job timeout are equal, you will never get a meaningful error message. The outermost one always wins, and it always wins with something generic.

Things that will hurt in production

Cost amplification through retries. A 4-minute job fails at second 239. If your retry policy is “try three times,” you just paid for 12 minutes of compute and hold nothing. Reasoning workloads need budgeted retries: count total reasoning tokens spent per job, stop when you cross the ceiling, and stop counting attempts.

A circuit breaker for money, not just errors. A conventional breaker trips on error rate. With a reasoning model you need one that trips when spend per minute crosses a threshold, even if every request succeeds. A client bug that loops requests happens most weeks, and it produces no errors at all.

Observability has to count reasoning tokens. The metric that matters is not latency, it is the distribution of reasoning_tokens by tenant and task type. A small prompt change can triple the amount of thinking. Without that metric you find out at the end of the month.

Do not log prompts and outputs unguarded. Reasoning workloads tend to carry sensitive data, because people only hand the hard problems to them. Storage cost comes along for the ride: one job can produce 10KB of log.

Do not assume the model always has an answer. o1 reasons well and is still wrong, and wrong with confidence. In domains with real consequences, an external verifier is a requirement, not an optimization. For code, unit tests. For numbers, constraint checks. For prose, a human.

What is actually worth remembering

Test-time compute does not replace pre-training. It is a second axis with its own limits: thinking longer helps a model exploit knowledge it already has, but it does not create new knowledge. o1 still does not know today’s gold price, and ten more minutes of thinking will not help.

What really changed is how we allocate compute. Compute used to be concentrated in one place: a datacenter, over months, producing a fixed artifact. Now a slice of it moves to every request, at the edge of the system, over tens of seconds, paid per use. Economically that is capex turning into opex. Technically it is a shift from “train a better model” to “design a system that tolerates a dependency with minute-scale latency and a cost that varies with how hard the question is.”

For backend engineers, the hard part is not calling the API. It is accepting that some work in your system cannot complete inside one HTTP request, and designing for that fact instead of fighting it.