From smooth curve to step function

The sensitivity of the sigmoid to its input has a clean closed form:

The maximum is achieved at , where . So the Lipschitz constant of the predictor scales linearly with . The “active region”, the band of where the sigmoid isn’t saturated near or , has width .

This isn’t a thought experiment: it’s what actually happens to the sigmoid as you keep training. Below is the same logistic-regression run from earlier, with the predicted probability surface plotted at four checkpoints in training time. As iterations accumulate and grows logarithmically, the smooth -curve sharpens into a near-vertical step:

The curve at is gently sloped: it answers borderline points with a hedge. The curve at has collapsed to a step, and any input that strays even slightly across the boundary is met with near-total confidence. The model didn’t get more right; it got more emphatic.

	slope at	active region width	for
			saturates to

The same that produces a probability shift at produces full saturation at . The sigmoid has stopped behaving like a smooth probability and started behaving like an indicator function, a step at the decision boundary.

Confidently wrong: a worked case

Take a noisy positive example whose features land at , just past the boundary on the wrong side. The true label is , but a small measurement error has placed it where is slightly negative.

• With : , so . The model hedges: it says roughly , which is well-calibrated for a borderline point.
• With : , so . The model is catastrophically wrong: it says “definitely class ” with confidence on a point whose true label is class .

This is the cost of unbounded weights. High does not make correct answers more correct; it makes wrong answers more confidently wrong. The well-calibrated “hedge” disappears, replaced by a response so steep that any deviation from the training distribution flips the prediction.

For reference: , the edge of float64 precision. Trained classifiers routinely produce logits at this scale, which is exactly the high- regime where the sigmoid has collapsed to a step. Working in raw logits rather than materializing these saturated probabilities is also the numerically stable choice, which I make the case for in Why Your Final Layer Shouldn’t Have Softmax.

Set Your SLOs

“Detect anomalies,” “reduce fraud,” and “improve recommendations” aren’t goals. Without measurable targets, every team member optimizes for a different thing, and you can’t tell whether you’re constrained by latency, precision, coverage, or something else entirely.

The fix is Service Level Objectives (specific, measurable thresholds tied to business outcomes):

These five dimensions force hard conversations. A model with 99% accuracy but 30-minute detection latency fails the TTD target. A model with perfect precision but 500 daily alerts fails the decision budget. The SLOs define the feasible region, and crucially, reveal what’s blocking you from reaching it.

Here’s how our network anomaly detection system instantiated these:

• TTD: p95 ≤ 5 minutes from event to alert
• Alert Budget: ≤ 10 analyst-actionable alerts/day
• Incident-Weighted Recall: ≥ 90%

The same template applies to other domains. A fraud detection team might set TTD ≤ 200ms with ≤ 50 manual reviews/day. A recommendation system might target TTD ≤ 100ms with CTR-weighted precision ≥ X%.

When defining SLOs, involve the people who use the system’s outputs, not just the team that builds it. Security analysts, operations teams, business stakeholders. When they disagree (and they will: security wants recall, ops wants fewer alerts), the SLOs make the tradeoff explicit rather than hiding it inside model thresholds.

SLOs don’t just measure success; they reveal what’s blocking it. If you can’t meet your TTD target, the bottleneck is somewhere in your latency path. If you can’t meet your alert budget, the bottleneck is in precision or triage capacity. Now let’s find exactly where.

Find the Bottleneck

With SLOs defined, you can systematically measure where the system breaks down. Build a constraint ledger, a table measuring capacity, utilization, latency, queue depth, and top failure mode at each pipeline stage:

The diagnostic pattern is simple: high utilization + growing queue = bottleneck. Feature-Tier2 jumps out: 95% utilization with a queue of 1.2K while other stages sit at 10–65%. During peak periods, the system is forced to either sample traffic (missing attacks), queue records (violating TTD), or drop features (hurting accuracy). The model never sees complete feature representations because feature extraction can’t keep pace.

Build this table for your own system. The constraint is almost always obvious once you measure.

Capacity conversion: 10 Gbps network traffic ≈ 100K flows/sec. 1M daily e-commerce orders ≈ 12/sec average, 50/sec peak. 10K IoT sensors at 1Hz ≈ 10K records/sec.

Validate Before You Invest

Before building anything, run a 24-hour experiment: temporarily throw 3x resources at your suspected bottleneck. If system-level metrics improve dramatically, you’ve found the right constraint. If not, look elsewhere. This experiment costs a day; building the wrong solution costs months.

We provisioned 3x compute for Feature-Tier2, enabling 90K records/sec. The results were dramatic: detection time dropped, false positives decreased (the model makes better decisions with complete feature sets), and we nearly met our SLOs. No other improvement (not model accuracy, not infrastructure, not threshold tuning) would have achieved this.

Key takeaway: The constraint is the only thing worth optimizing right now. Everything else is rearranging deck chairs.

Five Whys: With Evidence

The Five Whys technique is simple: ask “why” repeatedly until you reach a root cause, but validate each answer with evidence before proceeding to the next. Unvalidated whys lead to plausible-sounding but wrong root causes.

Here’s how this played out for our Feature-Tier2 bottleneck:

1. Why is Feature-Tier2 at 95% utilization? → It computes 47 features per record. (Validated: profiling shows 89% of computation in 12% of features)
2. Why so many features? → Designed for offline research with unlimited compute. (Validated: 31 features contribute <0.1% to decisions)
3. Why no production constraints in the design? → Development was disconnected from deployment. (Validated: git history shows features added without removal)
4. Why disconnected? → ML team and platform team operate in silos. (Validated: team interviews confirm no shared requirements)
5. Why silos? → No ownership of end-to-end system performance.

Notice where we ended up: the root cause isn’t technical; it’s organizational.

If your Five Whys keep ending at technical causes, go one more level. The technical problem often has an organizational parent: siloed teams, misaligned incentives, no end-to-end ownership. These patterns aren’t unique to our system. Fraud detection, recommendations, and forecasting all exhibit the same failure modes.

Key takeaway: Root causes are usually organizational, not algorithmic. If you fix the technical symptom without fixing the organizational cause, the symptom will return.

Why Teams Get Stuck

Barnard identifies two failure modes that keep teams trapped in these oscillations:

• Getting stuck / procrastinating: Exaggerated fears (fear of losing what the current approach does well, or fear of the effort and risk required to change). (“If we remove features, we’ll miss attacks.”)
• Overreacting / jumping to conclusions: Exaggerated frustration with the current approach’s downsides, or exaggerated expectations of a new solution. (“Let’s just throw out all the expensive features and rely on the model.”)

Most ML teams alternate between these two modes without recognizing the pattern.

Map Your Conflict

The breakthrough comes from asking: what assumptions make this conflict seem unresolvable? To find them, map the conflict explicitly:

We need [rich feature analysis] to achieve [accurate detection].
We need [efficient processing] to achieve [real-time response].
These conflict because we assume:
  1. All records need the same analysis depth
  2. Features must be computed synchronously
  3. One model handles all decisions

Challenge each:
  - Is assumption 1 always true? No, routine DNS queries
    don't need the same scrutiny as connections to unknown IPs.
  - Is assumption 2 always true? No, historical comparisons
    could be asynchronous.
  - Is assumption 3 always true? No, different attack types
    could use specialized models.

This template works for any ML conflict. A fraud detection team might write: “We need comprehensive transaction analysis AND sub-200ms decisions. Hidden assumption: every transaction needs the same analysis depth.” A recommendation team: “We need deep personalization AND instant page load. Hidden assumption: personalization must happen at request time.”

Key takeaway: The tradeoff that’s blocking you is almost never fundamental. It persists because of hidden assumptions. Find the assumption. Challenge it. The conflict evaporates.

The Thinking Process

After mapping your conflict and challenging assumptions (Step 3), work through each challenged assumption systematically:

1. Sketch the system without the assumption. If you challenged “all records need the same analysis depth,” draw the pipeline where they don’t. What would variable-depth processing look like? What decides the depth?

2. Look for the four reusable patterns. Most ML system innovations are combinations of these:

   • Cascade filtering: cheap check first, expensive check only when needed. Applicable whenever most inputs are routine. (Fraud: score transactions with simple rules before running the full model. Recs: serve cached recommendations before running personalization.)
   • Async enrichment: decide now, enrich later. Useful whenever decision speed and decision quality have different time horizons. (Generate an alert with basic info immediately; add forensic context over the next 30 seconds.)
   • Confidence-based routing: let the model decide how much compute each input deserves. Turns a fixed-cost pipeline into an adaptive one. (High-confidence benign traffic exits at Tier-1; uncertain traffic escalates.)
   • Feature caching: never compute the same thing twice across pipeline stages. Obvious but rarely implemented. (Features from early triage stages are reused in deep analysis; we achieved 84% cache hit rates.)

3. Check for async opportunities: what’s being computed before the decision that could move to after?

4. Check for caching opportunities: what’s being computed repeatedly across stages, records, or time windows?

A Worked Example: Progressive Analysis

In our case, the most load-bearing assumption was: “All records need the same analysis depth.” Once you challenge it, the architecture follows from the patterns above, cascade filtering with confidence-based routing between tiers:

Each stage outputs a prediction and a confidence score. High-confidence benign traffic exits immediately. Low confidence escalates. When stages experience backlog, confidence thresholds adjust dynamically: low-risk records defer to async processing during congestion, ensuring high-risk traffic always gets full analysis. And alerts are generated immediately with basic info, then progressively enriched over 30 seconds with connection context, historical patterns, and full forensics.

Fix the Organization Too

Remember: Step 2 told us the root cause was organizational (siloed teams, no end-to-end ownership, features added without production constraints). The progressive architecture only sticks if the organizational structure changes with it. We restructured so ML and platform teams share SLOs, and adding features now requires cross-team cost-benefit approval. Without this, the feature explosion that caused the original bottleneck would have returned within a year.

Key takeaway: The innovation doesn’t have to be novel to the field. It has to be novel to your system. Progressive analysis is a known pattern; applying it to our specific bottleneck was the breakthrough. But the technical fix and the organizational fix are a package deal.

Identify the Riskiest Assumption

Ask: what’s the single assumption that, if wrong, kills the entire approach? For our progressive architecture, it was: “Can Tier-1 triage accurately identify benign traffic without missing attacks?” If lightweight features can’t reliably separate benign from suspicious, the whole cascade fails.

Design the smallest test that answers this question. We trained a logistic regression on 5 cheap features and tested on realistic data with known attacks:

tier1_features = [
    'src_reputation_score',  # Pre-computed reputation
    'dst_port',              # Destination port number
    'protocol',              # TCP/UDP/ICMP
    'packet_rate',           # Packets per second
    'byte_rate'              # Bytes per second
]

tier1_model = LogisticRegression(C=1.0)
tier1_model.fit(X_train[tier1_features], y_train_benign)

Results

What the Iterations Taught Us

The MVE revealed things we couldn’t have predicted from design alone:

1. Confidence calibration: Initial triage was too conservative, with 32% of traffic passed to Tier-2 unnecessarily. The model lacked confidence on legitimate-but-unusual ports. Retraining with expanded examples achieved a 71% triage rate without missing attacks.

2. Dynamic resource allocation: Fixed compute allocation caused bottlenecks when traffic patterns shifted. We implemented stages borrowing compute from idle stages, smoothing throughput across load profiles.

3. Feature pruning: 15 Tier-3 features never influenced decisions in production. Removing them increased throughput 30% without affecting detection. Track a feature value score (importance / computational_cost) and prune ruthlessly.

Production Rollout Checklist

• Shadow mode: Run progressive pipeline parallel to existing system. Compare decisions, measure divergence. Success: no P1 incidents missed for one week.
• Canary (10%): Route 10% of traffic through progressive pipeline. A/B test alert quality with analysts. Success: SLOs maintained, analyst preference ≥ baseline.
• Gradual expansion: 10% → 25% → 50% → 75%, holding each level for 48 hours. Automated rollback on any SLO violation.
• Full production: 100% with old system as instant fallback. Document runbooks, train operations team. Success: one week at 100% with all SLOs met.

Go/No-Go Criteria

After the MVE, the decision is straightforward: does the riskiest assumption hold? If yes (the cascade correctly separates benign from suspicious), proceed to shadow mode. If the assumption fails, you haven’t wasted months; you’ve spent two weeks learning that you need a different innovation. Go back to Step 4 and challenge a different assumption.

Key takeaway: A two-week MVE teaches more than two years of production experience. Test your riskiest assumption first.

Why Tail Latency, Not Average Latency?

During training we optimize throughput: push millions of examples through, extract the most learning per unit of compute. We can afford to wait a day for a better model.

Serving inverts this priority. Now we optimize latency, and specifically not the average. The number that matters is the tail: p95, p99, sometimes p99.9.

It is tempting to read p99 as “1% of users have a bad time,” which sounds survivable. It is not, and the reason is structural (Dean and Barroso, 2013).

Suppose a single user request fans out to backend services (a feature service, a lookup, an embedding store, a model server) and each one independently has a 1% chance of being slow. The request is fast only if every service is fast:

Fan-out	1	10	20	50	100	200
P(user request is slow)	1.0%	9.6%	18.2%	39.5%	63.4%	86.6%

ML Metrics Are Proxies

ML metrics are not business value. They are correlated with it, sometimes, over some range. A recommender’s NDCG is not revenue. A fraud model’s precision is not dollars saved.

I learned this properly on a content recommendation system. We spent three months lifting the ranking model’s AUC from 0.78 to 0.82, which is a real improvement by any offline standard. We deployed it and engagement did not move. The reason took another week to find: nearly all of the gain was in how we ranked items 50 through 100, and users almost never scrolled past item 20. We had spent a quarter optimizing a region of the ranking that almost no one ever looked at.

The problem is NOT AUC, it is that an aggregate metric averages over regions of the input space that carry unequal business weight, and the model will happily buy its gains in the cheap ones. Before you optimize a metric, establish three things with whoever owns the P&L:

1. What user behavior actually drives value?
2. How does the model’s output change that behavior?
3. What is the sensitivity: how much business metric per unit of ML metric?

Question 3 is the one that usually gets skipped, and it is the only one that tells you whether the project is worth doing. Sometimes the answer is that ML is a small term in a large equation. In a loan approval system, the model estimates creditworthiness, but the approval decision is dominated by regulatory floors, capital availability, and portfolio strategy. A better model may move nothing at all, and it is much cheaper to discover that in a spreadsheet than after two quarters of engineering.

The Four Pillars

Production systems need reliability, scalability, maintainability, and adaptability. Most of this is standard software engineering, so I will only dwell on the part that is ML-specific.

Reliability in ML means more than uptime, because ML systems fail silently. A web server that breaks returns a 500. A model whose feature pipeline breaks returns a confident, plausible, wrong number, and it keeps doing so for months. The defense is a degradation ladder that is decided in advance:

primary model  →  simpler/cached model  →  business rules  →  cached prediction  →  safe default

Every level of the system must be implemented and tested, and the lowest/basic level must be safe rather than merely available. A fraud system whose fallback is “approve everything” is not degrading gracefully; it is failing open, and that is expensive.

Scalability has three axes that interact: model complexity, request volume, and number of models. The third is the one that surprises teams. An architecture tuned for one large model often collapses under three hundred small ones, because per-model overhead (loading, memory, monitoring, versioning) was never on anyone’s radar.

Maintainability is versioning, and in ML that means versioning code, data, models, configs, and feature definitions, because any one of them drifting silently produces a different system. Six months from now, nobody remembers why that threshold was 0.37.

Adaptability is the ability to change quickly, which requires modularity, monitoring that tells you when to change, and a safe rollout path (testing in production).

Hierarchical Classification: What It Buys and What It Doesn’t

For classification with many classes, decomposing into a hierarchy (coarse categories first, specialized models at the leaves) is often the right move.

First, different features at different levels. Top-level routing can use broad, cheap features. Leaf models can use narrow features that only make sense within their subtree (“screen size” is meaningful for phones and meaningless for groceries).

Second, the option to abstain. If the top-level classifier is not confident, you can stop and return the coarse label. In production, a confident wrong answer at the leaf (“iPhone 13”) is usually worse than a vague right one (“Electronics”). Flat classifiers do not give you this choice for free.

Third, maintainability. You can retrain one subtree without touching the rest, and errors localize to a level.

Multi-Label: Split or Joint?

If the labels are independent, separate binary classifiers give you flexibility: update, debug, and tune each one alone. If the labels are correlated, a joint model captures the correlation and does better.

Labels are almost never independent. Assume joint unless you have checked.

Multiple Objectives: Decouple Them

When a system must serve several objectives (relevance, engagement, revenue), the tempting move is one model with a blended loss. A single model with three objectives baked into its weights forces you to retrain the model every time the business changes its mind about the trade-off, which it will do quarterly.

Train one model per objective. Combine at serving time with weights:

def score(item, user_ctx, business_ctx, models, rules):
    """One model per objective; trade-off specified in config NOT in weights."""
    scores = {name: m.predict(item) for name, m in models.items()}

    # Changing the business trade-off requires new deployment NOT retraining
    weights = get_weights(user_ctx, business_ctx)
    blended = sum(weights[name] * s for name, s in scores.items())

    return apply_business_rules(blended, item, rules)

This buys four things: you can retune priorities without retraining, update one model without touching the others, debug a single objective in isolation, and use a different architecture per objective. The cost is that you now serve three models instead of one (scalability, axis three), and it is worth paying: the flexibility buys you more than the extra compute costs.

First-, Second-, and Third-Party Data

Data gets more valuable and more chaotic the closer it sits to you. The three tiers differ in how they break.

First-party data (collected directly from your users) is the most valuable, because it is about your users doing your thing. It is also the filthiest data you will ever handle. Users leave fields blank, type asdf@asdf.com, and behave in ways no schema anticipated.

On one system I found that 15% of users were born on January 1st. Our date picker defaulted to January 1st and most people did not bother changing it. Therefore, January 1st now has two meanings (NULL and people who were actually born on January 1st) and we can extract more features out of “date of birth”

The rule for first-party data is defensive processing. Validate everything and assume every field is adversarial, not because users are malicious but because interfaces leak their defaults into your data.

Second-party data (from partners) breaks on semantics rather than syntax. One partner’s “active user” is a 30-day login; another’s is a 90-day purchase. Nothing in the data will tell you this. Build validation that checks distributions, not just types, and alert when a partner’s field shifts.

Third-party data gives you coverage at the price of staleness, unknown bias, and legal exposure. For example, a purchased demographic set may be two years old. Treat it as a dependency that could be withdrawn at any time.

Data Passing Modes: Database, Service, Streaming

Three transports, in increasing order of power and operational cost.

Database-backed is the simplest: features live in a table, you query them at request time. Fine for prototypes and relaxed latency budgets. It scales badly, and worse, it couples your ML availability to your database availability. A slow query does not delay one prediction. It fills your connection pool and takes down the service.

Service-based (REST/gRPC) is what most production systems use, and it is the right default. Independent scaling, independent deploys, language freedom. The failure mode is dependency sprawl: prediction calls feature service calls user service calls database, and each hop is a chance to be slow. Reread the fan-out arithmetic before you add another hop.

Streaming transport (Kafka and friends) decouples data production from consumption, which is what you need when features must be computed continuously rather than on demand. The payoff is that expensive features get precomputed and are simply there when the request arrives.

It is also a standing operational commitment. Out-of-order events, late arrivals, exactly-once semantics, and cluster operations are now your problem, permanently. Price that in honestly before choosing it, because the cost does not show up in the prototype and never goes away afterward.

Batch and Streaming Features

Batch features (also called static) are computed periodically over large datasets: lifetime value, 90-day purchase histogram, demographic aggregates. Because nothing is waiting on them, they can be arbitrarily expensive. They are stable, cheap to serve, and stale.

Streaming features (dynamic) capture the present: items viewed in the last ten minutes, current session depth, current location. They must fit inside the request budget, so they are simple by necessity.

Most real systems need both, and the moment you have both, you have created a new source of Mismatch. The batch pipeline computes avg_order_value in Spark. The streaming pipeline computes it in Flink. They disagree about how to handle nulls, or about whether the window is inclusive, and now your model is trained on one definition and served another. Nobody notices, because both numbers look reasonable.

This is the single most common silent failure in production ML, and the section on training/serving skew covers what to do about it.

Sampling Strategies

The default instinct is to grab the most recent data because it is convenient. This is nonprobability sampling and it is the source of some production headaches.

The rest of the toolkit, briefly, because the choice is usually obvious once you name the constraint:

Method	Use when	Watch out for
Random	You have plenty of every class	Misses rare events entirely at low prevalence
Stratified	Important segments must be represented	Too many strata dimensions gives you meaningless micro-strata
Weighted	Some examples matter more (recency, customer value)	Wrong weights inject bias you will not detect offline
Reservoir	Sampling from an unbounded stream in one pass	Uniform over the stream, which may not be what you want
Importance	Training distribution target distribution	High variance when the two distributions barely overlap

Feedback Signals: Volume, Delay, Noise

Labels from user behavior come in tiers, and the tiers trade off against each other in a way you cannot escape:

An implicit signal is abundant but unreliable: a user viewed an item because it was at the top of the page, not because they wanted it. An explicit signal is honest and arrives too late to steer anything.

The resolution is to use all three at different timescales rather than picking one. Implicit signals drive rapid adaptation and exploration. Engagement signals drive short-cycle updates. Explicit signals drive periodic retraining and, critically, calibration, because they are the only ones you actually trust.

graph LR
    U["User activity stream"] --> A["Implicit<br/>views, hovers, scroll"]
    U --> C["Engagement<br/>clicks, dwell, shares"]
    U --> E["Explicit<br/>ratings, purchases"]

    A -->|"streaming"| M1["Online learner<br/>fast, noisy"]
    C -->|"hourly mini-batch"| M2["Nearline trainer"]
    E -->|"daily/weekly batch"| M3["Batch trainer<br/>ground truth + calibration"]

    M1 --> S["Model registry"]
    M2 --> S
    M3 --> S
    S --> P["Production model"]

When You Do Not Have Labels

Labels are expensive. Four alternative in rough order of how often they work.

Transfer learning is the first thing to try, and it usually wins. Start from a model pretrained on a related task and fine-tune. The amount of task-specific data you need is often shockingly small: hundreds of examples, not hundreds of thousands, because the pretrained model already has the representation and you are only teaching it the decision.

Weak supervision replaces hand-labeling with labeling functions: heuristics, patterns, external knowledge, each individually noisy. A generative model learns how much to trust each one and resolves their disagreements (Ratner et al., 2017). I used this for content moderation, where the labeling functions were things like profanity lists, all-caps ratios, “reported by 3+ distinct users,” and known-bad domains. No single function was better than mediocre. Combined, they were good enough to train on.

Semi-supervision bootstraps from a small labeled set: train, predict on unlabeled data, keep the confident predictions, retrain. It works, but it has a dangerous failure mode. The model is being trained on its own beliefs, so its errors are self-reinforcing. Confidence goes up while accuracy goes down, and the metric you would use to catch this is the one that has been corrupted. Always hold out a genuinely labeled validation set and stop when it degrades.

Active learning spends your labeling budget where it buys the most: uncertain examples, boundary cases, cluster representatives. The chicken-and-egg problem is that selecting informative examples requires a model, and the model requires labels, so start with a small random seed set and iterate.

Why Accuracy Fails Under Imbalance

Everyone knows the first-order version: a model that always predicts “not fraud” scores 99.9% accuracy and catches nothing. The usual response, switch to precision and recall, does not go deep enough, because even a genuinely good model produces a mostly-useless alert queue, and no choice of metric changes that.

Take a fraud model with 99% sensitivity and 99% specificity. Excellent by any standard. At prevalence , its precision is:

At 1% prevalence, a 99%/99% model is a coin flip. At 0.1%, nine out of every ten alerts you send a human are wrong.

Why 0.5 Is the Wrong Threshold

Every classifier ships with a decision threshold. Most ship with 0.5, and 0.5 is almost always wrong.

Suppose a missed fraud costs and a false alarm costs (an analyst’s time, an annoyed customer). Given a calibrated probability that a transaction is fraudulent, flag it when the expected cost of flagging is lower than the expected cost of not:

Solve for :

That is the whole derivation (Elkan, 2001). The optimal threshold does not depend on your model at all. It depends only on the relative cost of the two errors.

Read a few cases off it:

$10	$1,000	0.99%
$10	$100	9.1%
$1	$1	50.0%
$100	$10	90.9%

How much does this actually cost? I simulated two million transactions from a calibrated fraud model (, , prevalence 0.11%) and swept the threshold:

At the system costs $0.119 per transaction. At the default 0.5 it costs $0.757. Shipping the default is a 6.3x cost penalty. On a million transactions a day, that is roughly $640,000 of avoidable cost every day.

Two things stand out in that figure, and the second matters more:

1. The empirical minimum lands on , as the derivation says it must.
2. The basin is flat. Anywhere from roughly 0.002 to 0.05 is nearly optimal. You do not need to compute precisely. You need to not be at 0.5.

Data-Level Fixes and Their Limits

With the threshold question settled, most of the resampling literature reads differently. If you can fix the decision rule for free, why rebalance the data at all?

Sometimes you should, because at severe imbalance the minority class has too few examples for the model to learn a boundary at all, which is a different problem from placing the boundary. But be clear about which problem you have.

• Undersampling the majority class is fast and throws away real information, including the hard negatives that define the boundary.
• Oversampling the minority preserves everything and encourages overfitting: duplicated examples add confidence, not information. Worse, if you duplicate before splitting, the same example lands in train and validation, and your validation score is now fiction (duplicate leakage).
• SMOTE (Chawla et al., 2002) interpolates between minority examples. It works for continuous features and produces nonsense for categorical or discrete ones. The midpoint of two user profiles is not a user.
• Tomek links remove majority examples sitting right on the boundary. Cleaner boundaries, at the price of deleting exactly the ambiguous cases production will throw at you.

What has worked best for me is two-phase training: learn the patterns on balanced data, then fine-tune on the natural distribution at a reduced learning rate to fix calibration.

# Phase 1: balanced data, so the model actually sees the minority class
model.fit(balance(X_train, y_train))

# Phase 2: natural distribution at 0.1x LR, so the outputs mean something.
# Without this, the model's probabilities reflect the resampled prior,
# not the real one, and t* is computed against a lie.
model.set_learning_rate(0.1 * lr)
model.fit(X_train, y_train)

A model trained only on rebalanced data has learned the rebalanced prior, so its probabilities are wrong by exactly the resampling ratio, which breaks the threshold rule.

Algorithm-Level Fixes: Cost-Sensitive and Focal Loss

Often cleaner than touching the data.

Cost-sensitive learning puts the cost matrix directly in the loss, which is the honest version of what the threshold rule does after the fact.

Class-balanced loss weights examples by inverse class frequency. Simple, effective, and the weights are a hyperparameter you now have to tune.

Focal loss (Lin et al., 2017) down-weights examples the model already gets right, so its attention flows to the hard ones. Since majority examples are usually the easy ones, this rebalances implicitly, without a frequency table. For extreme imbalance it often works when explicit reweighting does not.

Missing Values

Before anything else: find out what “missing” means in your system. It is frequently not NULL. It is -999, or "", or "N/A", or 1900-01-01, or 0 in a column where zero is also a legitimate value. Until you have audited this, every statistic you compute is wrong.

Then ask why it is missing, because the mechanism determines the fix:

• MNAR (Missing Not At Random): missingness depends on the unobserved value itself. High earners decline to state income. The missingness is the signal.
• MAR (Missing At Random): missingness depends on other observed variables. Mobile users skip long forms. Predictable from what you do have.
• MCAR (Missing Completely At Random): genuinely random. A dropped packet. Rare in practice, and if you think your data is MCAR, look harder.

Always add a missing indicator. It follows directly from the MNAR definition.

Let be the binary missingness mask for a feature. Under MNAR, depends on the unobserved value, which means:

The two populations have different label distributions given everything else you observed. So carries information that is not recoverable from , and therefore not recoverable from any imputed value, because imputation is a function of . Imputation destroys . The indicator preserves it.

This also tells you exactly when the rule does not apply: under MCAR, the two distributions are equal, is independent of , and the indicator is pure noise. That is the only case where you can skip it, and it is the case you almost never have.

From this, several practical guidelines follow:

• Do not drop columns for being 30% missing. That column is informative for the 70%, and its missingness is informative for the rest.
• Be careful dropping rows. Under MAR or MNAR you are deleting a segment, and it is usually a segment you care about.
• Median/mode imputation plus an indicator is a strong baseline and beats most of what people replace it with.
• Predictive imputation is a model inside your model. Fit it on training data only to avoid leakage(preprocessing leakage).

Scaling

The most know scaling technique is StandardScaler, which has two problems:

The first is that outliers destroy your scaling parameters before they ever touch your model. One value of in a column that otherwise ranges over drags the mean, inflates the standard deviation, and compresses every real value into a sliver near zero. Clip first (1st and 99th percentile), then compute statistics. In that order.

The second is that training-time statistics go stale. You standardized using a mean from six months ago, but user behavior moves continuously in productionand.

Match the transform to the distribution:

Distribution	Transform
Normal	Standardization (z-score)
Uniform	Min-max to
Heavy-tailed	Robust scaling: median and IQR
Power-law	Log first, then scale
Unknown	Quantile / rank transform

Encoding

One-hot is fine until the cardinality is 10,000 and your feature matrix explodes. Target encoding (replace each category with its mean target) is clever and leaks: the target of the row you are encoding is inside the encoding. If you do use it, compute the encoding out-of-fold.

The hashing trick is what tends to survive contact with production. Hash the category into a fixed number of buckets. Bounded dimensionality, no vocabulary to maintain, and (this is the part that matters) unseen categories at serving time just work, because hashing is defined for inputs you have never observed. One-hot and target encoding both have to be told what to do with a new category.

A collision costs a little accuracy, which is a far better failure than an unhandled new category crashing the request path at serving time.

Crossing

Crossing features (latitude × longitude, category × region) lets linear models learn interactions. The combinatorics are brutal: two 100-value features cross to 10,000 combinations, most of which appear a handful of times or never.

Cross selectively, from domain knowledge, and monitor the cardinality of the result. If a cross produces a long tail of singleton values, hash it or group it.

Training/Serving Skew

You compute avg_session_length in a Pandas notebook for training. Someone reimplements it in Scala for the serving path. They handle a null differently, or round differently, or use a half-open interval where you used a closed one. The model now sees a feature at serving time that is subtly not the one it was trained on. Nothing throws an error and both numbers look reasonable, so performance degrades quietly, and every investigation starts at the model, which is not where the problem is.

There is exactly one robust fix: the training path and the serving path must execute the same code, not two separate implementations that are meant to agree.

Temporal Leakage

Temporal leakage is using information from after the prediction time to make the prediction. Stated that way it sounds impossible to do by accident, and yet it is one of the easiest mistakes to make.

I debugged a churn model that scored 99% accuracy in validation. In production it fell to 60%.

The issue was a feature called account_status, which was computed by a nightly job after churn was recorded. The model had not learned to predict churn. It had learned to recognize customers who had already churned. Offline, that field makes the model look 99% accurate. Online, at prediction time, it always reads “active,” so the model has learned nothing it can use.

Any aggregate can hide this. days_since_last_purchase computed relative to today instead of relative to the prediction date. lifetime_value that includes the transaction you are trying to predict. The question to ask about every single feature is: at the moment I need to make this prediction in production, does this value exist yet?

Always split by time, never at random, for any problem with a temporal component. Your validation set must come from strictly after your training set, because that is the shape of the production task.

Preprocessing, Duplicate, and Group Leakage

Preprocessing leakage. You computed the scaler’s mean over the full dataset, then split. The test set’s statistics are now baked into the training transform. Same for imputation values, feature selection, and text vocabularies. The rule: fit every transform on training data only, then apply the frozen transform to test.

Duplicate leakage. The same record (or a near-duplicate) appears in both train and test. Deduplicate before splitting, and check for fuzzy duplicates, not just exact ones.

Group leakage. Correlated records straddle the split. Ten photos of the same person, split across train and test. Multiple transactions from one user on one day. The model learns to identify the group rather than the pattern. Therefore, split by group, not by row.

Detecting Leakage: Adversarial Validation

The best general-purpose leakage detector is a classifier that tries to tell your training set apart from your test set.

The logic is: if train and test are drawn from the same distribution, no classifier can separate them better than chance. Any model, given infinite capacity and infinite data, achieves AUC 0.5. So an AUC materially above 0.5 is evidence that the two sets differ, and the features the classifier leans on are the ones that differ.

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import cross_val_score
import numpy as np, pandas as pd

def adversarial_validation(X_train, X_test, threshold=0.7):
    """AUC ~0.5 means train and test are indistinguishable, which is what we want.
    Materially higher means they differ, and the top features say how."""
    X = pd.concat([X_train, X_test], ignore_index=True)
    is_test = np.r_[np.zeros(len(X_train)), np.ones(len(X_test))]

    clf = RandomForestClassifier(n_estimators=200, random_state=0)
    auc = cross_val_score(clf, X, is_test, cv=5, scoring="roc_auc").mean()

    clf.fit(X, is_test)
    suspects = pd.Series(clf.feature_importances_, index=X.columns).nlargest(5)
    return auc, (suspects if auc > threshold else None)

The 0.7 is a practical tolerance for finite-sample noise. Treat anything above ~0.6 as worth a look.

Two cheaper checks are worth running first.

• Feature importance: if one feature dominates every other, be suspicious. Genuine predictive features are rarely that dominant, and a feature that is “too good” usually contains the answer.
• Ablation: drop one feature, retrain, and see if performance collapses. Genuine predictive signal is usually spread across many features, whereas leakage tends to concentrate in one.

Baselines

Start with the dumbest thing that could work, and make each subsequent model beat the last:

1. Random / mean. Establishes the floor. If you cannot beat this, something is broken.
2. Majority class / historical average. Captures base rates. “Predict yesterday’s value” is a shockingly strong baseline for time series and beats a lot of deep learning.
3. Domain heuristics. Flag transactions over $X. Flag users idle for days. These encode real knowledge and are often most of the achievable value.
4. Simple ML. Logistic regression, gradient-boosted trees.

Then ask whether the complex model’s margin over step 4 is worth its cost, where “cost” includes serving latency, debugging difficulty, retraining time, and the engineer-months it will consume for as long as it lives.

Simple models win in production more often than not. They are faster, more interpretable, degrade more gracefully, and fail in ways you can reason about.

Calibration

The threshold derivation already made the case: the cost-optimal threshold is only meaningful on calibrated probabilities. If your model says 0.70 and the event happens 40% of the time, then every downstream decision keyed to that number is wrong, and the model’s ranking being excellent does not save you.

Check with a reliability diagram: bucket predictions by predicted probability, and plot predicted against observed frequency. A calibrated model sits on the diagonal.

Fix with Platt scaling (fit a logistic regression on the model’s outputs) or isotonic regression (fit a monotonic step function; more flexible, needs more data). Fit either on a held-out set, never on training data.

Modern networks tend to be overconfident, and the effect got worse as models got bigger and more accurate (Guo et al., 2017). Accuracy and calibration are separate properties. You have to check both.

Slice-Based Evaluation

An aggregate metric is an average over a population, and averages hide the segments that matter.

I deployed a model that performed well overall and terribly for new users. We had trained and tuned on users with rich histories, because that was where most of the data was, and the model learned to lean hard on history. New users had none. The aggregate metric was fine, because new users were a small fraction of sessions. They were also the entire growth channel, and the model we shipped was pushing them away.

Evaluate on slices, always, and choose them by business importance rather than by data volume: new versus returning, geography, device, language, high-value versus low-value customers, and the rare-but-critical cases. A model that is 2% better on average and 15% worse on new users is a worse model.

Behavioral and Robustness Testing

Behavioral tests encode domain knowledge as assertions about the model’s behavior, independent of any metric (Ribeiro et al., 2020):

• Monotonicity: raising income should not lower the loan approval probability. If it does, the model is broken no matter what the AUC says.
• Invariance: changing a customer’s name should not change their credit decision.
• Directional expectation: adding a known-fraudulent signal should raise the fraud score.

These catch spurious correlations that validation metrics won’t catch.

Robustness tests perturb inputs slightly and check that predictions do not swing wildly. Production is full of noise: sensor error, truncation, retries, typos. A model that is unstable under small perturbations will be unstable in production.

Batch, Online, and the Hybrid

Batch prediction precomputes everything on a schedule and serves from a lookup table. No latency problem, arbitrary model complexity, cheap serving. Three costs: storage grows with users x items, most of what you compute is never used, and every prediction is stale by up to one cycle. Staleness is worse than it sounds, because intent moves fast. A user shopping for a laptop this morning may be shopping for a monitor now, and a nightly batch cannot reflect that.

Online prediction computes on demand. Always fresh, nothing wasted, and now you are inside a latency budget.

The counterintuitive part is that in an online system, feature computation usually dominates latency, not model inference. The model is a few matrix multiplies; the features need a fan-out to several services and a cache miss, which is the fan-out problem again. Optimizing the model while features are the bottleneck is a common and costly mistake, and the subject of a separate post on finding the constraint that governs a system.

So: hybrid. Precompute the expensive, slow-moving features in batch (user embeddings, 90-day aggregates). Compute the cheap, fast-moving ones online (current session, current context). Join at request time. This is what most mature systems converge on, and keeping the batch and online paths consistent is exactly what feature stores are for.

Compression

Models are wildly over-parameterized, which is good news when you need them smaller. (If you want the mechanics underneath any of this, I built a small PyTorch from scratch and the memory and kernel trade-offs are all there.)

• Quantization: 32-bit floats to 8-bit ints. ~4x smaller, usually negligible accuracy loss. Try this first; it is nearly free.
• Pruning: remove low-magnitude weights. A large fraction of parameters can often go with little damage, and pruning sometimes improves generalization by acting as a regularizer.
• Distillation (Hinton et al., 2015): train a small student on the large teacher’s soft outputs. The soft targets carry more information than hard labels (they encode the teacher’s uncertainty across classes), which is why the student can outperform the same architecture trained from scratch. Compression ratios vary a lot by task.

The unifying observation is that model capacity and useful capacity are different quantities, and compression removes the difference.

Cloud and Edge

Cloud gives you flexibility, elastic scale, and instant updates, and charges you network round-trip time (often tens of milliseconds before your model does any work at all), bandwidth, and a privacy surface.

Edge eliminates the round trip and the privacy problem, and constrains you to a device with a fraction of the memory, no GPU, and a limited power budget.

The pattern converging in practice is partitioning: a small model on device for the immediate response, a large model in the cloud for the accurate one, with the UI designed to accommodate both.

ML Systems Fail Silently

Ordinary software failures are transparent. Servers crash, requests 500, alerts fire, etc. You can handle these with ordinary SRE practice: redundancy, health checks, circuit breakers, gradual rollout. If you do not have these, ML-specific failures are not your biggest problem.

Some ML-specific failures:

• Pipeline corruption. A join key changes, a timestamp format shifts, an upstream system starts emitting a placeholder -> Features become garbage.
• Training/serving skew. Two implementations of one feature, quietly disagreeing.
• Silent degradation. Distribution drifts, accuracy decays, and with no ground-truth labels arriving you have no way to see it.
• Feedback loops. The model’s outputs become its own future training data. A recommender only gets feedback on what it recommended, so it learns that what it recommends is what users want. The system converges on a self-confirming worldview, and the metric that would reveal this has been corrupted by the same loop.

Every one of these leaves the system running and responsive while its outputs are wrong.

Four Layers of Monitoring

You need all four, and the value is in the connections between them.

A drop in the business metric traces to a shift in the prediction distribution, which traces to a feature whose missing-rate jumped, which traces to an upstream schema change that shipped on Tuesday. With only the business layer, you can tell something is wrong but not what. With only the system layer, every dashboard is green while the model quietly fails.

graph TB
    S["System / pipeline<br/>schema change ships (root cause)"]
    D["Data layer<br/>feature missing-rate jumps"]
    M["Model layer<br/>prediction distribution shifts"]
    B["Business layer<br/>conversion drops (what you see first)"]
    S --> D --> M --> B

Causation runs bottom to top: a low-level change propagates up into a business symptom. Detection runs the other way: you notice the business number first and trace back down. A single layer gives you one end of that chain; you need all four to connect them.

Why There Are Exactly Three Kinds of Drift

Mismatch 3 was “ moved.” The question is moved how? We factorize the data distribution, which exists whether or not your model estimates it, so none of this depends on a probabilistic model.

The three types of drift are a consequence of the fact that the joint distribution factorizes two ways:

Drift is the question of which factor moved. The two factorizations enumerate the possibilities exhaustively (Moreno-Torres et al., 2012):

Type	What moves	What stays	Consequence	Fix
Covariate shift			The learned function is still correct, you are just evaluating it in new places	Reweight training data; retrain if the new inputs fall outside the old support
Label shift			Ranking is still correct; the posterior is off by exactly the prior ratio	Recalibrate. The model is fine
Concept drift		(nothing you can lean on)	The relationship you learned is now false	Retrain. Nothing else works

That third row is the dangerous one, and the table explains why: no amount of reweighting or recalibration can repair a function that is estimating the wrong thing.

Detecting Drift

The obvious approach is a statistical test (Kolmogorov-Smirnov, chi-square) on each feature’s distribution. This is a trap, and the reason is subtle.

The power of any statistical test goes to 1 as for any nonzero effect. At 10 million requests a day, every feature will differ from its training distribution at , every single day, because no two empirical distributions are ever exactly equal and you have enough samples to prove it.

The fix is to threshold on effect size, not significance. Population Stability Index is the standard choice:

import numpy as np

def psi(expected, actual, bins=10):
    """Population Stability Index. Effect size, not a p-value, so it does not
    get more alarming just because you have more traffic.
    < 0.1 stable | 0.1-0.2 investigate | > 0.2 act"""
    cuts = np.quantile(expected, np.linspace(0, 1, bins + 1))
    cuts[0], cuts[-1] = -np.inf, np.inf

    e = np.histogram(expected, cuts)[0] / len(expected)
    a = np.histogram(actual, cuts)[0] / len(actual)
    e, a = np.clip(e, 1e-6, None), np.clip(a, 1e-6, None)  # avoid log(0)

    return float(np.sum((a - e) * np.log(a / e)))

The other approach is adversarial validation again, applied to reference-versus-current data instead of train-versus-test. It is more robust than per-feature tests because it catches shifts in the joint distribution that no marginal test would see, and it tells you which features moved.

Performance monitoring is the ground truth, and it requires labels, which are delayed or absent precisely when you most need them.

How Often to Retrain

Stateless retraining rebuilds from scratch on a recent window. Reproducible, no catastrophic forgetting, clean lineage. Expensive and slow to adapt.

Stateful training updates the existing model incrementally. Cheap and fast-adapting. Risks catastrophic forgetting, is permanently corruptible by bad data, and makes “what did this model actually learn from?” hard to answer.

Most mature systems do both: incremental updates for responsiveness, periodic full retrains for stability and lineage.

Trigger retraining on a combination of signals, because each one alone has a gap:

Testing in Production

Each stage catches a different class of problem.

1. Holdout. Necessary, wildly insufficient. Tells you the model learned patterns. Says nothing about production.
2. Backtesting. Evaluate on the most recent data available. If a model trained on last year fails on last month, it will fail tomorrow.
3. Shadow deployment. Run the new model alongside the old on real traffic, serving none of its predictions. You get real latency, real errors, real feature distributions, real infrastructure behavior, and zero user risk. This is the highest-value stage in the list and the most frequently skipped. Run it for at least a week.
4. Canary. Route 1% of traffic, then 5%, 25%, 50%, with automatic rollback on technical and business metrics. A problem at 1% is an incident report. The same problem at 100% is an outage.
5. A/B test. The only stage that measures business impact. Fix the sample size, success criteria, and duration before you start, and then do not peek, because peeking at a running experiment and stopping when it looks good is how you ship noise (Kohavi et al., 2020).
6. Interleaving. For ranking, mix results from both models into one list for the same user. Removes between-user variance and detects smaller effects with far less traffic.

The True Cost of Ownership

Training costs labeling (usually the biggest single line item), the hundreds of failed experiments behind each success, and the engineer-months.

Serving costs feature computation and storage, network, caching, monitoring, and the on-call rotation, of which model inference is often a minority.

Humans cost the most: development, incident response, debugging, stakeholder communication, and the knowledge transfer when someone leaves.

In conclusion: a simple model on more expensive hardware is frequently cheaper than a complex model that requires a dedicated team. Optimize for total cost, and count the humans.

Key Takeaways

1. Tail latency is a property of your architecture, not your model. At a fan-out of 100, a 1% per-service slow rate makes 63% of user requests slow. You cannot optimize your way out of that one service at a time.

2. The default 0.5 threshold asserts that your two errors cost the same. They almost never do. , the cost basin around it is flat, and 0.5 is usually outside it. In the threshold simulation, shipping the default cost 6.3x more per transaction.

3. Calibration is a precondition, not a polish step. Every cost-optimal decision rule assumes the probabilities mean what they say, and modern networks are overconfident by default.

4. Under low prevalence, even an excellent model produces a mostly-false alert queue. That is arithmetic, not a modelling failure, and you must design the human workflow around it.

5. Drift is a question of which factor of moved. Covariate and label shift are bookkeeping problems that reweighting and recalibration solve. Concept drift means your function is wrong, and only retraining helps.

6. Training/serving skew is the most common silent failure in production ML. The only robust fix is that both paths execute the same code.

7. Data quality beats model sophistication, every time. I have never seen a complex model rescue bad data. I have often seen a simple model succeed on good data.

8. The model is 10% of the system (Sculley et al., 2015). The rest is pipelines, features, serving, monitoring, and people.

A Roadmap for a New System

1. Write down the cost matrix. What does a false positive cost? A false negative? If nobody can answer, you do not yet have a problem specification, and every threshold you pick afterward is a guess.
2. Build the baselines and measure the business metric they produce. This is your real bar, and sometimes it is already high enough that you can stop.
3. Establish the latency budget from the architecture, using the fan-out arithmetic, before you choose a model class.
4. Build the feature pipeline so that training and serving share one code path. This is the cheapest it will ever be to do. Retrofitting it costs ten times more.
5. Split by time, and by group. Then run adversarial validation to confirm you have not leaked.
6. Train, calibrate, and set the threshold from step 1. In that order.
7. Evaluate on slices, not just aggregates, with the slices chosen by business importance.
8. Shadow deploy for a week. Then canary. Then A/B.
9. Instrument all four monitoring layers before you take traffic, not after your first incident.
10. Measure your data half-life and set the retraining cadence from the curve.

The gap between a notebook and a production system is real, and almost none of it is modelling. It is the work of keeping a fitted function useful against a world that keeps changing, and every step above is part of that work.

Setup

We start with imports and two small helpers. listify normalizes arguments, and the annealer decorator lets us compose the warmup-then-cosine learning rate schedule further down.

import inspect
import math
import os
import time
from dataclasses import dataclass
from functools import partial, wraps
from typing import Callable, Iterable

import tiktoken
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as opt
from torch.distributed import destroy_process_group, init_process_group
from torch.nn.parallel import DistributedDataParallel as DDP

def listify(obj):
    if obj is None:
        return []
    elif isinstance(obj, str):
        return [obj]
    elif isinstance(obj, list):
        return obj
    elif isinstance(obj, Iterable):
        return list(obj)
    else:
        return [obj]

def annealer(func: Callable):
    @wraps(func)
    def annealer_wrapper(*args, **kwargs):
        return partial(func, *args, **kwargs)

    return annealer_wrapper


@annealer
def lin_sched(start, end, pos):
    """Linear scheduler."""
    return start + (end - start) * pos


@annealer
def cos_sched(start, end, pos):
    """Cosine scheduler."""
    return start + (1 + math.cos(math.pi * (1 - pos))) * (end - start) / 2


def combine_scheds(pcts, scheds):
    """
    Combine multiple schedulers, each run for a given percentage of the
    training process.
    """
    assert len(pcts) == len(scheds), "Each scheduler should have its `pct`."
    assert sum(pcts) == 1.0, "Sum of the `pcts` should be equal to 1."
    pcts = torch.tensor([0] + listify(pcts))
    assert (pcts >= 0).all(), "All percentages should be non-negative."
    pcts = torch.cumsum(pcts, 0)

    def _inner(pos):
        idx = (pos >= pcts).nonzero().max()
        actual_pos = (pos - pcts[idx]) / (pcts[idx + 1] - pcts[idx])
        return scheds[idx](actual_pos)

    return _inner

Configuration

One dataclass holds every architectural and training hyperparameter. The values are GPT-2 124M: 12 layers, 12 heads, 768-dimensional embeddings, and a 1024-token context.

@dataclass
class GPTConfig:
    block_sz: int   = 1024  # Sequence length
    # Originally 50000 BPE merges + 256 byte tokens + 1 for <|endoftext|> token
    # which will delimit different documents. This token's index is 50256
    # However, we found that using 50257 as the vocab size is not a multiple of
    # 64 and we could improve efficiency and performance (through better
    # occupancy) if we round up to the closest multiple of 64, which is 50304.
    vocab_sz: int   = 50257
    n_layer:  int   = 12    # Number of layers
    n_embd:   int   = 768   # Embedding dimension
    n_head:   int   = 12    # Number of attention heads
    lr:       float = 3e-4  # Good for big models
    batch_sz: int   = 4
    dropout:  float = 0.0
    bias:     bool  = True

The Feed-Forward Network

The MLP runs on each token independently, with no interaction between positions. Because it projects out to 4 x n_embd and back, it holds roughly two-thirds of the model’s parameters.

class MLP(nn.Module):
    def __init__(self, config: GPTConfig):
        # Point-wise feed-forward network that applies non-linearity
        # on every token separately. THERE IS NO INTERACTION BETWEEN TOKENS.
        # This is where almost all the capacity and non-linearities of the 
        # model come from especially when we project it to 4 x n_embd
        super().__init__()
        self.c_fc = nn.Linear(config.n_embd, config.n_embd * 4)
        # Found to be better than ReLU in terms of gradient saturation
        self.gelu = nn.GELU(approximate="tanh")
        self.c_proj = nn.Linear(config.n_embd * 4, config.n_embd)
        self.dropout = nn.Dropout(config.dropout)
        self.c_proj.NANOGPT_SCALE_INIT = 1

    def forward(self, x):
        return self.dropout(self.c_proj(self.gelu(self.c_fc(x))))

Causal Self-Attention

Attention is the one operation that lets tokens exchange information. All three projections live in a single fused c_attn matrix that we split after the matmul, and F.scaled_dot_product_attention supplies both Flash Attention and the causal mask.

class CausalSelfAttention(nn.Module):
    def __init__(self, config: GPTConfig):
        super().__init__()
        self.n_head = config.n_head
        self.n_embd = config.n_embd
        # For optimization purposes, QKV matrics will be in one big matrix
        # that will be split after we do matmul with X
        self.c_attn = nn.Linear(config.n_embd, config.n_embd * 3, bias=config.bias)
        # Main job of this projection is for the interaction between heads
        # because they worked in isolation -> cross-head communication
        # that would lead to suppressing or amplify the combination
        self.c_proj = nn.Linear(config.n_embd, config.n_embd, bias=config.bias)
        self.attn_dropout = nn.Dropout(config.dropout)
        self.resid_dropout = nn.Dropout(config.dropout)
        self.dropout = config.dropout
        self.c_proj.NANOGPT_SCALE_INIT = 1
        # NOTE: Mask is not needed when we use Pytorch's Flash attention
        # self.register_buffer(
        #     "mask",
        #     torch.tril(torch.ones(config.block_sz, config.block_sz))
        # )

    def forward(self, x):
        B, T, _ = x.shape
        qkv = self.c_attn(x)
        # q/k/v is B x T x n_embd each
        q, k, v = torch.split(qkv, self.n_embd, dim=-1)
        # Reshape q/k/v to B x n_head x T x (n_embd / n_head)
        # So each head would be learning different kind of relationships
        q = q.view(B, T, self.n_head, self.n_embd // self.n_head).transpose(1, 2)
        k = k.view(B, T, self.n_head, self.n_embd // self.n_head).transpose(1, 2)
        v = v.view(B, T, self.n_head, self.n_embd // self.n_head).transpose(1, 2)
        # attn is B x T x T
        # attn = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(k.shape[-1]))
        # Mask out future tokens
        # attn = attn.masked_fill(self.mask[:T, :T] == 0, float("-inf"))
        # attn = self.attn_dropout(F.softmax(attn, dim=-1))
        # y is B x T x n_embd
        # y = attn @ v
        # Uses Flash attention that never materialize attention matrices for
        # each head and is aware of the memory hierarchy and tries to reduce
        # read/writes with more FLOPs -> Speed up since we're memory bound
        y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
        y = y.transpose(1, 2).contiguous().view(B, T, self.n_embd)
        return self.resid_dropout(self.c_proj(y))

The Transformer Block

A block wraps attention and the MLP in residual connections. The normalization sits before each sublayer instead of after, which departs from Attention Is All You Need. That ordering keeps the residual path an unobstructed identity from input to output, so gradients reach the early layers without crossing a normalization at every step. Deep stacks are much easier to train this way.

class Block(nn.Module):
    def __init__(self, config: GPTConfig):
        super().__init__()
        self.ln_1 = nn.LayerNorm(config.n_embd)
        self.ln_2 = nn.LayerNorm(config.n_embd)
        self.mlp = MLP(config)
        self.attn = CausalSelfAttention(config)

    def forward(self, x):
        # Use Pre-layer normalization which deviates from the
        # transformer original paper that uses post-layer normalization.
        # This should help stabilize training
        x = x + self.attn(self.ln_1(x))
        x = x + self.mlp(self.ln_2(x))
        return x

The Full Model

This class assembles the embeddings, the stack of blocks, and the language modeling head. It also carries weight tying between the token embedding and the output head, which accounts for 38.6M of the model’s 124M parameters, the residual-aware initialization scaling derived below, and weight decay applied only to matrices. One thing to notice in forward: it hands raw logits to F.cross_entropy and never applies a softmax of its own, for the reasons in Why Your Final Layer Shouldn’t Have Softmax.

class GPT2(nn.Module):
    def __init__(self, config: GPTConfig):
        super().__init__()
        self.config = config
        self.transformer = nn.ModuleDict(
            dict(
                wte=nn.Embedding(config.vocab_sz, config.n_embd),
                # Attention operation is a permutation equivariant, this means that
                # if we permute the input then the corresponding output will be
                # permuted in exactly the same way. In other words, attention mechanism
                # is not aware of the relative ordering of the tokens. Therefore, we
                # need some way to encode the positions of the tokens in each sequence.
                # This is where positional encoding comes into play.
                # Here we use a simple positional encoding that is a simple
                # embedding of the position of the token in the sequence.
                wpe=nn.Embedding(config.block_sz, config.n_embd),
                h=nn.ModuleList(
                    [Block(config) for _ in range(config.n_layer)]
                ),
                # Final layer norm after all transformer layers
                ln_f=nn.LayerNorm(config.n_embd),
            )
        )
        # Sometimes called classifier
        self.lm_head = nn.Linear(config.n_embd, config.vocab_sz, bias=False)

        # Weight sharing between the token embedding layer and
        # last linear layer (LM head classifier). The rationale is
        # that tokens that are semantically similar to each other in
        # the embedding space should have similar probabilities in the
        # softmax of the LM head layer.
        # This is also one of the biggest matrices in the model: 50257 * 768 =
        # 38.6M parameters, ~31% of the model's 124M total. Without tying, the
        # same model would carry that matrix twice and weigh in at ~163M.
        self.transformer.wte.weight = self.lm_head.weight
        self.apply(self._init_weights)

    def _init_weights(self, module):
        # The following initialization comes from gpt2 src code
        # NOTE: Because token embedding and classifier weights are shared,
        # our initialization logic will initialize the weight matrix twice
        # but shouldn't be an issue since they're being initialized with the
        # same std and mean
        if isinstance(module, nn.Linear):
            std = 0.02
            # We're changing std because residual path affect std
            # by increasing it on every layer so we need to adjust
            # it so we still have the same std = 0.02
            if hasattr(module, "NANOGPT_SCALE_INIT"):
                # `2` here because every layer has two blocks:
                #   - Attention block
                #   - MLP block
                # `N` is the number of layers in the model (n_layer)
                # Since they are independent, variance of the sum of the two
                # blocks is the sum of the variances
                std *= (2 * self.config.n_layer) ** -0.5
            nn.init.normal_(module.weight, std=std)
            if module.bias is not None:
                nn.init.zeros_(module.bias)
        elif isinstance(module, nn.Embedding):
            # We're initializing the token and positional embeddings
            # with the same std but the paper initialized the positional
            # embedding with std = 0.01
            nn.init.normal_(module.weight, std=0.02)

    def forward(self, x, targets=None):
        T = x.shape[-1]
        assert (
            T <= self.config.block_sz
        ), f"Sequence length must be <= {self.config.block_sz}, got {T}"
        pos_emb = self.transformer.wpe(
            torch.arange(0, T, dtype=torch.long, device=x.device)
        )
        tok_emb = self.transformer.wte(x)
        x = pos_emb + tok_emb
        for block in self.transformer.h:
            x = block(x)
        x = self.transformer.ln_f(x)
        # logits is B x T x vocab_sz
        logits = self.lm_head(x)
        loss = None
        if targets is not None:
            # F.cross_entropy expects the 2nd dimension to be probabilities
            loss = F.cross_entropy(
                logits.view(-1, self.config.vocab_sz), targets.view(-1)
            )
        return logits, loss

    def configure_optimizer(self, weight_decay, lr, device):
        params_dict = {
            pn: p for pn, p in self.named_parameters() if p.requires_grad
        }
        # We're not applying weight decay to bias and layer norm parameters
        # And any 1D parameters. Therefore, we are ONLY applying weight decay
        # to the weight matrices in Embedding and Linear layers
        decay_params = [p for p in params_dict.values() if p.ndim >= 2]
        nondecay_params = [p for p in params_dict.values() if p.ndim < 2]
        params_groups = [
            {"params": decay_params, "weight_decay": weight_decay},
            {"params": nondecay_params, "weight_decay": 0.0},
        ]
        # Fused AdamW is available for PyTorch 2.0+
        fused_available = "fused" in inspect.signature(opt.AdamW).parameters
        use_fused = fused_available and "cuda" in device
        return opt.AdamW(
            params_groups, lr=lr, betas=(0.9, 0.95), eps=1e-8, fused=use_fused
        )

    @torch.no_grad
    def generate(self, idxs: torch.Tensor, max_tokens: int = 5):
        for i in range(max_tokens):
            # x would be B x T x vocab_sz (At most we would have
            # block_sz tokens since we're using fixed block_sz for the
            # positional embedding
            idxs = idxs[:, -self.config.block_sz :]
            logits, _ = self(idxs)
            # Get probs for last token to predict next token
            # This would be B x vocab_sz
            logits = logits[:, -1, :]
            # Apply softmax to get probabilities
            probs = F.softmax(logits, dim=-1)
            # Pick top 50 prob -> we would never pick tokens with
            # very small probs (right tails) -> B x 50
            # probs/idxs are sorted in descending order
            topk_probs, topk_idxs = torch.topk(probs, 50, dim=-1)
            # Sample 1 token from the top 50 tokens -> idx is B x 1
            idx = torch.multinomial(topk_probs, 1)
            # Get the vocab idx as `multinomial` returns only indices that
            # corresponds to the given array
            idx = torch.gather(topk_idxs, -1, idx)
            # TODO: We should check for end_of_text token and break out of
            # the loop (stop generation) even if we have not reached max_tokens
            idxs = torch.cat([idxs, idx], dim=1)
        return idxs

Why Depth Changes the Initialization

The line std *= (2 * self.config.n_layer) ** -0.5 in _init_weights looks like a tuned constant, but it falls out of how the residual stream accumulates.

A block writes into the stream twice, in sequence, exactly as Block.forward does:

with and . These are not independent draws, since is computed from , which already contains . What lets their variances add is the output projection: each sublayer ends in a c_proj initialized zero-mean and independent of its input, so and . A write is uncorrelated with the stream it lands on even though it was computed from it. Pre-normalization then fixes each write’s size: attn and MLP see unit variance however wide the stream has grown, so at every depth. Applying this at each step and unrolling over blocks, from embeddings at :

So the standard deviation grows like , set by depth alone. With and , the stream leaves the stack at , five times the width it entered with. Choosing instead makes the accumulated term , so and total growth is at any depth. The 2 counts the writes per block and the undoes the square root. It is also why NANOGPT_SCALE_INIT is set on c_proj in MLP and CausalSelfAttention and nowhere else: those are the two projections that write into the stream.

Data Loading

The loader tokenizes the corpus once with tiktoken’s GPT-2 BPE encoding (which I build from scratch in Byte Pair Encoding, and place among the alternatives in Tokenization Strategies), then hands out contiguous windows. Sharding is where this gets subtle. Each process must take a disjoint slice, and all processes must wrap around together; otherwise the ranks drift out of step and train on overlapping data with nothing to signal it.

class DataLoaderLight:
    def __init__(
        self,
        file_path: str,
        batch_sz: int,
        block_sz: int,
        process_rank: int = 0,
        number_processes: int = 1,
    ) -> None:
        self.batch_sz = batch_sz
        self.block_sz = block_sz
        self.process_rank = process_rank
        self.number_processes = number_processes
        with open(file_path, "r") as f:
            text = f.read()
        encoder = tiktoken.get_encoding("gpt2")
        tokens = torch.tensor(encoder.encode(text), dtype=torch.long)
        # We can truncate the tokens list/tensor based on a multiple of
        # `batch_sz x block_sz x number_processes`. This is useful
        # for multi-node training and mimics the behavior of DataLoader's
        # `drop_last` parameter. Think of it as for every iteration we sample
        # batch_sz x block_sz x number_processes input tokens where each
        # process would get batch_sz x block_sz tokens -> Each iteration would
        # lead to all processes running the same load and no process is idling
        self.tokens_per_process = self.batch_sz * self.block_sz
        self.tokens_per_iteration = (
            self.tokens_per_process * self.number_processes
        )
        # Every global iteration predicts `tokens_per_iteration` labels, so
        # it needs one additional token to label the final input token.
        self.num_batches = (
            len(tokens) - 1
        ) // self.tokens_per_iteration
        if self.num_batches < 1:
            raise ValueError(
                "dataset must contain at least "
                f"{self.tokens_per_iteration + 1} tokens"
            )
        self.tokens = tokens[:self.num_batches * self.tokens_per_iteration + 1]
        print(f"Loaded {len(self.tokens)} tokens")
        print(f"1 epoch = {len(self)} batches")
        self.current_batch = 0

    def __len__(self):
        return self.num_batches

    def next_batch(self):
        global_start = self.current_batch * self.tokens_per_iteration
        process_start = (
            global_start + self.process_rank * self.tokens_per_process
        )
        process_end = process_start + self.tokens_per_process + 1
        buf = self.tokens[process_start:process_end]
        x = buf[:-1].view(self.batch_sz, self.block_sz)
        y = buf[1:].view(self.batch_sz, self.block_sz)
        # Each process will process batch_sz x block_sz tokens in each
        # iteration -> with number_processes processes, total tokens processed
        # in each iteration is batch_sz x block_sz x number_processes. In the
        # case of one process, total tokens would be batch_sz x block_sz
        # Similar to DataLoader's `drop_last` parameter, we drop the last
        # batch if it's not a multiple of batch_sz x block_sz x number_processes.
        # Tracking the global batch directly makes every process reset together
        # after consuming the same number of complete global batches.
        self.current_batch = (self.current_batch + 1) % self.num_batches
        return x, y

Distributed Setup

torchrun sets RANK, LOCAL_RANK, and WORLD_SIZE in the environment. This cell reads them to initialize the process group and pin each process to its own device.

###########
## Distributed Data Parallel
###########
# Distributed Data Parallel let us run the same model (replica) on different GPUs,
# where each GPU would work on a different slice of data. After we do the backward
# pass, we average the gradients across all processes (GPUs) and synchronize all
# parameters across all devices. We use allReduce op to do this and communicate the
# updates with all processes.
# Each process would go through the same code from top to bottom not aware there
# are other processes running the same thing on other devices
#
# torchrun command sets the following environment variables:
# RANK: Id of the process in the process group. It is an int 0 - WORLD_SIZE
# LOCAL_RANK: In the case of multi-nodes, LOCAL_RANK is the id of
#             the process in the same node. Example: If we have a node
#             with 4 GPUs, the first process will have LOCAL_RANK = 0
#             but RANK of this process may not be 0 if we are running
#             on multiple nodes.
#             This is useful when we have multiple nodes and we want to
#             run the processes on different GPUs in the same node.
#             In this case, we can set the LOCAL_RANK to the GPU id in the
#             node.
# WORLD_SIZE: Total number of processes
ddp = int(os.getenv("RANK", -1)) != -1  # Check if it is a ddp run
if ddp:
    # DDP requires CUDA so we need to set the device for each process
    # so only one process can run per device
    assert torch.cuda.is_available(), "DDP requires CUDA"
    init_process_group(backend="nccl")
    ddp_rank = int(os.getenv("RANK"))
    ddp_local_rank = int(os.getenv("LOCAL_RANK"))
    ddp_world_size = int(os.getenv("WORLD_SIZE"))
    device = f"cuda:{ddp_local_rank}"
    torch.cuda.set_device(device)
    # Master process will do more things such as checkpointing and logging
    # while other processes would assist in the computations.
    # It always has RANK = 0
    master_process = ddp_rank == 0
else:
    ddp_rank = 0
    ddp_local_rank = 0
    ddp_world_size = 1
    master_process = True
    if torch.cuda.is_available():
        device = "cuda"
    elif torch.mps.is_available():
        device = "mps"
        torch.mps.manual_seed(1337)
    else:
        device = "cpu"
print(device)

torch.manual_seed(1337)
if torch.cuda.is_available():
    torch.cuda.manual_seed(1337)

Model and Optimizer

Here we instantiate the model with the padded vocabulary, then build the fused AdamW optimizer and the learning rate schedule it will follow.

##########
## Initialize model and optimizer
##########
# GPU kernels tile in powers of two, so dimensions that divide evenly by
# 64 or 128 make far better use of:
# • Tensor Cores
# • Memory coalescing
# • Shared memory bank alignment
# • Warp scheduling
# Here we round vocab_sz from 50257 up to 50304, the next multiple of 64.
# 50304 is not itself a power of two (it is 64 x 786); what matters is that
# the dimension divides evenly, so the matmul tiles without a ragged
# remainder. We pay for a few hundred embedding rows that can never be
# emitted and get faster kernels everywhere the vocab dimension appears.
model = GPT2(GPTConfig(vocab_sz=50304)).to(device)
# Speed up model by building static graph that analyzes all ops
# and optimizes them such as fusing some of them to avoid unnecessary
# trips to memory. Uncomment to enable: it costs a slow first step while
# the graph is traced and compiled, which is why it is off by default here.
# model = torch.compile(model)
if ddp:
    model = DDP(model, device_ids=[ddp_local_rank])
raw_model = model.module if ddp else model
max_lr = 3e-4
min_lr = max_lr * 0.1
warmup_steps = 10
max_steps = 50
sched = combine_scheds(
    [warmup_steps / max_steps, 1 - (warmup_steps / max_steps)],
    [lin_sched(min_lr, max_lr), cos_sched(max_lr, min_lr)],
)
optimizer = raw_model.configure_optimizer(
    weight_decay=0.1, lr=max_lr, device=device
)

The Training Loop

The loop is where the optimizations come together: gradient accumulation to reach a 0.5M-token batch, bfloat16 autocast, TF32 matmuls, gradient clipping, and the learning rate schedule. Two details in this loop are easy to get wrong in ways that produce no error and no obviously bad number.

##########
## Run training loop
#########
# NOTE: In order to run 0.5M (from GPT3 paper) tokens per fwd/bwd iteration,
# we need to use gradient accumulation because we can't fit it in almost
# any commodity GPU -> We only do backward after we loop through ~0.5M tokens.
total_batch_sz = 2**19  # closest number to 0.5M
assert total_batch_sz % (GPTConfig.batch_sz * GPTConfig.block_sz * ddp_world_size) == 0, "total batch size must be divisible by micro batch_sz x block_sz x ddp_world_size"
grad_accum_steps = total_batch_sz // (
    GPTConfig.batch_sz * GPTConfig.block_sz * ddp_world_size
)
if master_process:
    print(f"Total desired batch size: {total_batch_sz}")
    print(f"Calculated gradient accumulation steps: {grad_accum_steps}")

train_dl = DataLoaderLight(
    "tinyshakespeare.txt",
    batch_sz=GPTConfig.batch_sz,
    block_sz=GPTConfig.block_sz,
    process_rank=ddp_rank,
    number_processes=ddp_world_size
)
# Pytorch will use TensorFloat32 if available, else use FP32
# But the weights will still be stored using FP32 with less precision
# (10 bits for mantissa instead of 23). It is just the
# operations would be executed as TF32 if available
torch.set_float32_matmul_precision("high")

# `autocast` wants a device TYPE ("cuda"), not a full device string. DDP
# assigns each process "cuda:{local_rank}", so comparing against "cuda"
# directly would silently skip mixed precision on every multi-GPU run.
device_type = "cuda" if device.startswith("cuda") else device

for step in range(max_steps):
    start = time.time()
    optimizer.zero_grad()
    loss_accum = 0.0
    for macro_step in range(grad_accum_steps):
        # Draw a fresh micro-batch on every accumulation step. Fetching it
        # outside this loop would add up `grad_accum_steps` copies of the
        # same gradient and leave the effective batch at batch_sz x block_sz
        x, y = train_dl.next_batch()
        x = x.to(device)
        y = y.to(device)
        if device_type == "cuda":
            # Tensors that will be greatly affected by less precision such
            # as loss, layernorm would still be in FP32 while others such
            # as attention weights would be in BF16
            with torch.autocast(device_type=device_type, dtype=torch.bfloat16):
                logits, loss = model(x, y)
        else:
            logits, loss = model(x, y)
        # Just accumulating gradients yield to summation of objective but
        # we want mean so we weight each loss by 1/grad_accum_steps
        loss /= grad_accum_steps
        loss_accum += loss.detach()
        # To avoid syncing the gradients between the processes after every
        # macro step, we disable it and only allows the sync up of
        # gradients after we finish all gradient accumulation in each
        # process
        if ddp:
            model.require_backward_grad_sync = (
                macro_step == grad_accum_steps - 1
            )
        loss.backward()
    # Each process would have its own loss_accum tensor, so to get the
    # average loss_accum across all processes, we want to compute the
    # average of all loss_accum in all processes
    if ddp:
        dist.all_reduce(loss_accum, op=dist.ReduceOp.AVG)
    # Clips gradient to global norm. It is very useful to avoid having a
    # very high loss for some (bad) batch(es) which would lead to very high
    # gradients and huge updates
    # NOTE: In the beginning of training it is normal to have high norms
    # as the model is initialized randomly
    norm = nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # TODO: Use ParamScheduler from `cmn_ai`
    lr = sched(step / max_steps)
    for pg in optimizer.param_groups:
        pg["lr"] = lr
    optimizer.step()
    # CUDA launches kernels asynchronously, so without this barrier we would
    # be timing how long it takes to QUEUE the work, not to finish it
    if device_type == "cuda":
        torch.cuda.synchronize()
    end = time.time()
    elapsed_time = end - start
    token_per_sec = (
        GPTConfig.batch_sz
        * GPTConfig.block_sz
        * grad_accum_steps
        * ddp_world_size
    ) / (elapsed_time)
    print(
        f"step {step}, loss: {loss_accum.item():.4f}, lr {lr:.4e}, norm: {norm:.2f}, time: {elapsed_time:.2f}s, tok/sec: {token_per_sec:.2f}"
    )

if ddp:
    # Kills all processes
    destroy_process_group()

combine_scheds runs a linear warmup over the first 20% of steps, then hands off to cosine decay for the remaining 80%.

Warmup exists because AdamW divides each update by a running estimate of the gradient’s second moment. For the first few steps that denominator is built from almost no samples, so a full-size learning rate applies a large update scaled by a quantity nobody has measured yet. Starting small buys time for the estimate to stabilize. The cosine decay at the other end shrinks the step size as the model approaches a minimum, so the last updates are small enough to settle into it.

Sampling from the Model

With training finished, we can sample from the model. Generation reuses the same forward pass, run one token at a time, with each sampled token appended to the context and fed back in.

# Sample from the trained model. `generate` crops the context to block_sz,
# takes the logits at the final position, and samples from the top 50 tokens
# so we never draw from the long tail of near-zero probabilities.
enc = tiktoken.get_encoding("gpt2")
prompt = torch.tensor(enc.encode("Hello, I'm a language model,"), dtype=torch.long)
# Generate 4 independent continuations from the same prompt
prompt = prompt.unsqueeze(0).repeat(4, 1).to(device)

raw_model.eval()
out = raw_model.generate(prompt, max_tokens=30)
for row in out:
    print(enc.decode(row.tolist()))
    print("-" * 40)

Seeing It in Action

Before formalizing the algorithm, let’s watch it work on a real example. Consider a tiny training corpus containing the words "low lower lowest":

BPE discovered that l and o always appear together, then that lo and w always appear together, building up low as a token, effectively learning the word stem. Then it found lowe as a shared prefix of “lower” and “lowest.” Without any linguistic rules, purely from frequency, BPE learned morphological structure.

Notice what happened in merge 2: the algorithm merged lo with w, where lo itself was created in merge 1. BPE builds tokens hierarchically: later merges compose earlier ones. We can visualize this as a tree:

graph BT
    l["l (byte)"] --> lo["lo (merge 1)"]
    o["o (byte)"] --> lo
    lo --> low["low (merge 2)"]
    w["w (byte)"] --> low
    low --> lowe["lowe (merge 3)"]
    e["e (byte)"] --> lowe

Each level in the tree depends on the levels below it. This is why merge order matters during encoding: you can’t build low until lo exists.

The Algorithm

With the intuition in place, here’s the formal procedure:

1. Initialize: Start with a base vocabulary of all 256 byte values (0–255). Every string can be represented as bytes, so this guarantees full coverage: no <UNK> tokens, ever.

2. Count pairs: Scan the corpus and count every adjacent pair of tokens.

3. Merge the most frequent pair: Create a new token for it and replace all occurrences in the corpus.

4. Repeat steps 2–3 until you’ve done vocab_size - 256 merges.

The output is a merge table: an ordered list of pair → token mappings. This table is the tokenizer.

Training vs. Encoding: A Subtle Difference

There’s an important asymmetry between how BPE learns merges (training) and how it applies them to new text (encoding).

During training, we always merge the globally most frequent pair; that’s how we decide which merges to learn. But during encoding, we apply merges in the order they were learned, not by their frequency in the new text.

Why Bytes, Not Characters?

Starting from bytes (0–255) rather than Unicode code points is a practical decision. Unicode has over 150,000 code points, an impractically large base vocabulary. By working at the byte level, we start with just 256 symbols and can represent any string in any language or script, since UTF-8 encodes every Unicode code point as one to four of these bytes. BPE merges then learn to compose bytes into characters, characters into subwords, and subwords into common words, all driven by frequency in the training data.

Vocabulary Size: A Key Hyperparameter

How many merges should we do? This is the vocabulary size, and it’s a meaningful trade-off:

Larger vocabularies mean each token carries more information, so sequences are shorter and the model sees more context per forward pass. But each token also needs an embedding vector, so the embedding table grows linearly, and so does the output head: the final layer scores every candidate token with a dot product, so a wider vocabulary makes the last layer and its softmax more expensive at every step. And tokens that appear rarely in training will have poorly learned embeddings: they’ve simply not been seen enough times.

There is a subtler ceiling than cost. The Transformer does a fixed amount of work per token, so compressing more text into each token eventually packs in more than a single forward pass can unpack. A token that stands for a whole common phrase leaves the model no intermediate steps in which to reason about its parts. The sweet spot is therefore not as much compression as the hardware allows: it is the point where tokens are dense enough to give long-range context but still coarse enough that one forward pass can digest each one.

Most modern models settle in the 32k–100k range. GPT-2 uses ~50k tokens. LLaMA uses 32k. GPT-4 reportedly uses ~100k. The right size depends on the training data, the target languages, and the compute budget.

2.1 A module is just a namespace object

A module is not a special language construct; it is an ordinary object of type types.ModuleType whose attributes live in its __dict__. Accessing json.loads is a dictionary lookup in json.__dict__; setting an attribute on a module writes to that same dict:

import json
json.custom = 123                 # identical to json.__dict__["custom"] = 123
type(json), json.__name__, "loads" in json.__dict__, json.__dict__["custom"]

(module, 'json', True, 123)

module.x = 10 and module.__dict__["x"] = 10 are the same operation. This is why a module can play so many roles at once: a namespace, an execution environment for statements, and a container of globals. Every module carries a handful of dunder attributes that the import system sets on it:

2.2 A package is a module with a __path__

A package is simply a module that has a __path__ attribute. __path__ is a list of directories to search when importing that package’s submodules; it is to a package what sys.path is to the interpreter. A plain module has no __path__; a package does, and that is what lets import foo.bar know where to find bar.

All packages are modules, but not all modules are packages; only packages have __path__.

For __package__: a package’s __package__ equals its own __name__; a top-level module’s is the empty string; a submodule’s is its parent package’s name.

2.3 Regular vs. namespace packages

A regular package is a directory containing an __init__.py. Importing it runs that __init__.py, and its namespace becomes the package. Crucially, when the finder locates a regular package it stops searching: the first __init__.py of that name on sys.path wins.

A namespace package is a directory without __init__.py (PEP 420). It behaves completely differently during search: the finder does not stop at the first match. It scans all of sys.path, collecting every directory of that name, and, if it never finds a regular package with that name, fuses the collected directories into a single package whose __path__ spans every one of them.

The same package name can pull submodules from completely different directories. Let’s prove it. We create two separate directories, each holding a pck/ folder with no __init__.py, and put a different submodule in each:

# Two directories, each with a `pck/` folder that has NO __init__.py -> namespace pkg
write_module("locA/pck/mod.py",  "X = 100")
write_module("locB/pck/test.py", "X = 200")
sys.path[:0] = [f"{WORKSPACE}/locA", f"{WORKSPACE}/locB"]

from pck.mod import X as from_A
from pck.test import X as from_B
from_A, from_B, list(sys.modules["pck"].__path__)

(100,
 200,
 ['/var/folders/hn/9dlqvjvx427_3m7wj0vxmpx80000gn/T/import-demo-v_vgrtrz/locA/pck',
  '/var/folders/hn/9dlqvjvx427_3m7wj0vxmpx80000gn/T/import-demo-v_vgrtrz/locB/pck'])

The single name pck now resolves to a package whose __path__ lists both temp directories, and pck.mod and pck.test were imported from different places on disk. This is exactly how large namespaced ecosystems (like the google.* cloud libraries) let independently-installed distributions contribute modules under one shared name.

2.4 What __init__.py is for

Beyond marking a directory as a regular package, __init__.py runs the first time the package is imported, which makes it the natural place to shape the package’s public surface. Common uses:

• Re-export the useful names from submodules so callers write from mypkg import Thing instead of from mypkg.internal.things import Thing.
• Assemble __all__ from submodules to control from mypkg import *, e.g. __all__ = submod_a.__all__ + submod_b.__all__ (import the submodules first so their __all__ exists).
• Initialize package-level state, or occasionally monkeypatch another module.

Keep __init__.py cheap: because it runs on first import, expensive work there is paid by every program that imports the package, whether or not it needs that work.

2.5 Dotted imports bind submodules to their parents

Importing foo.bar.baz imports foo, then foo.bar, then foo.bar.baz, in that order, caching each of the three separately in sys.modules. A subtle but important side effect: when a submodule is loaded, it is bound as an attribute on its parent package. After import os.path, the parent module os gains a path attribute pointing at the submodule:

import os.path
"os.path" in sys.modules, hasattr(os, "path"), os.path is sys.modules["os.path"]

(True, True, True)

This is why import os.path lets you write os.path.join(...) even though you only named os: the submodule attached itself to its parent during import.

4.1 The ModuleSpec: a module’s blueprint

Since the Python 3.4 import reboot (PEP 451), the central object of the import system is the ModuleSpec: the metadata bundle that says what a module is and how to load it. You can obtain one without importing anything via importlib.util.find_spec:

importlib.util.find_spec("sys")        # a built-in: origin is 'built-in', no file

ModuleSpec(name='sys', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in')

spec = importlib.util.find_spec("pandas")   # a source package on disk
spec.name, spec.origin, spec.submodule_search_locations

('pandas',
 '/Users/imad/dev/dl-venv/lib/python3.14/site-packages/pandas/__init__.py',
 ['/Users/imad/dev/dl-venv/lib/python3.14/site-packages/pandas'])

The contrast tells the story: sys is a built-in with origin='built-in' and no file; pandas is a source package whose origin is its __init__.py and whose submodule_search_locations is the __path__ it will receive. The spec’s important fields:

4.2 The import algorithm, in pseudocode

Stripped to essentials, importing a not-yet-cached module looks like this:

import sys

def import_module(name):
    if name in sys.modules:              # 1. cache hit -> done
        return sys.modules[name]

    spec = find_spec(name)               # 2. search: finders -> spec (or ImportError)
    module = importlib.util.module_from_spec(spec)   # 3. create the empty module
    sys.modules[name] = module           # 4. cache BEFORE executing (circular-safe)
    spec.loader.exec_module(module)      # 5. execute the body into module.__dict__
    return sys.modules[name]             # 6. return cached (may differ if code swapped it)

Two details are worth internalizing. The module is cached before it is executed, so a circular import finds a partially-initialized module rather than looping forever. And the cached object is what’s returned: a module that reassigns sys.modules[__name__] to something else during execution can hand back a different object than the one that started loading.

4.3 Finders and loaders: create vs. execute are decoupled

The pipeline splits cleanly in two. A finder answers “can I locate this name?” and returns a spec. A loader does the work, and even that is split: create_module(spec) builds the empty module object, and exec_module(module) runs its code. importlib.util.module_from_spec gives you the created-but-not-executed module; nothing runs until you call exec_module:

write_module("heavy.py", """
    print(">>> heavy.py body is executing")
    VALUE = 42
""")
spec = importlib.util.find_spec("heavy")
heavy = importlib.util.module_from_spec(spec)   # created, but NOT executed
hasattr(heavy, "VALUE")                          # -> False: the body hasn't run

False

spec.loader.exec_module(heavy)                   # now the body runs
heavy.VALUE

>>> heavy.py body is executing

42

This decoupling is exactly what makes lazy loading (§6.3) possible: you can hold a fully-formed module object and defer the expensive exec_module until someone actually touches it.

4.4 sys.meta_path: the real controller

Every import is dispatched through sys.meta_path, an ordered list of meta path finders. Python asks each one, front to back, find_spec(name, path, target) until one returns a spec:

sys.meta_path

[<_distutils_hack.DistutilsMetaFinder at 0x1096146e0>,
 <_virtualenv._Finder at 0x109614590>,
 _frozen_importlib.BuiltinImporter,
 _frozen_importlib.FrozenImporter,
 _frozen_importlib_external.PathFinder,
 <six._SixMetaPathImporter at 0x10a44e900>]

The three that always ship with CPython are BuiltinImporter (compiled-in modules like sys), FrozenImporter (modules frozen into the binary, including importlib itself), and PathFinder, the only one that consults sys.path. Libraries and tools can prepend or append their own finders, so you may also see entries from setuptools or editable installs. Conceptually the search is just:

def find_spec(name, path=None, target=None):
    for finder in sys.meta_path:
        spec = finder.find_spec(name, path, target)
        if spec is not None:
            return spec
    raise ModuleNotFoundError(name)

PathFinder in turn delegates to sys.path_hooks: each sys.path entry is handed to these hooks to produce a path-specific finder (a directory gets a FileFinder; a .zip gets a zipimporter). The results are memoized in sys.path_importer_cache.

sys.path_hooks

[zipimport.zipimporter,
 <function _frozen_importlib_external.FileFinder.path_hook.<locals>.path_hook_for_FileFinder(path)>]

This layered design (meta_path → PathFinder → path_hooks → per-entry finder → loader) is the seam that makes §6 possible.

6.1 Tracing every import

A finder that prints and then returns None observes imports without handling them; returning None politely defers to the finders behind it:

class Watcher:
    "A meta-path finder that observes imports, then defers by returning None."
    @classmethod
    def find_spec(cls, name, path, target=None):
        print("find_spec ->", name)
        return None                    # None => let the finders behind us handle it

write_module("observed.py", "VALUE = 1")
sys.meta_path.insert(0, Watcher)       # prepend: we see every import first
try:
    import observed                    # a fresh module, so the finder actually fires
finally:
    sys.meta_path.remove(Watcher)

find_spec -> observed

The same idea one level up (wrapping builtins.__import__) intercepts the import statement itself:

_real_import = builtins.__import__

def traced_import(name, *args, **kwargs):
    if name not in sys.modules:        # only announce first-time imports
        print("importing:", name)
    return _real_import(name, *args, **kwargs)

builtins.__import__ = traced_import
try:
    import graphlib
finally:
    builtins.__import__ = _real_import

importing: graphlib

6.2 Auto-installing missing packages

Append a finder to the end of sys.meta_path and it becomes a last resort, consulted only after every real finder has failed. Here is the (in)famous auto-installer: on the first miss for a top-level name, it pip-installs the package and returns its now-findable spec.

import subprocess

class AutoInstall:
    "Last-resort finder: pip-install a missing top-level module on first miss."
    _tried = set()

    @classmethod
    def find_spec(cls, name, path, target=None):
        if path is not None or name in cls._tried:
            return None                # only top-level names, and only try once
        cls._tried.add(name)
        subprocess.check_call([sys.executable, "-m", "pip", "install", name])
        return importlib.util.find_spec(name)

sys.meta_path.append(AutoInstall)      # append => consulted only after real finders fail

6.3 Lazy imports

The create/execute split from §4.3 turns lazy loading into a few lines. The standard library ships importlib.util.LazyLoader, which wraps a loader so the module’s body doesn’t run until its first attribute access:

def lazy_import(name):
    spec = importlib.util.find_spec(name)
    if spec is None:
        raise ModuleNotFoundError(name)
    spec.loader = importlib.util.LazyLoader(spec.loader)
    module = importlib.util.module_from_spec(spec)
    sys.modules[name] = module
    spec.loader.exec_module(module)    # arms lazy loading; the body still hasn't run
    return module

write_module("expensive.py", """
    print(">>> expensive.py body is executing")
    ANSWER = 99
""")

mod = lazy_import("expensive")
print("bound lazily, nothing above printed yet")
mod.ANSWER                             # first attribute access triggers execution

bound lazily, nothing above printed yet
>>> expensive.py body is executing

99

Nothing printed when we “imported” expensive: the body ran only when we touched mod.ANSWER. Internally, LazyLoader uses exactly the trick you’d invent yourself: it returns a module whose __getattr__ triggers the real exec_module on first access, then swaps its own class back to a plain module so subsequent accesses are normal. This is how libraries with heavy optional dependencies keep import cheap.

8.1 reload creates zombies

importlib.reload does not give you a clean slate. It re-executes the module’s source into the same __dict__, so names from the previous version linger, and, worse, any class statement produces a brand-new class object while the old instances keep pointing at the old one. The two classes share a name but are not the same object:

write_module("widget.py", """
    print("widget.py body ran")
    class Widget:
        pass
""")
import widget
old = widget.Widget()

from importlib import reload
reload(widget)                         # re-runs the body into the SAME module dict
new = widget.Widget()

# Same class *name*, but a different class object -> old instances are "zombies":
old.__class__, new.__class__, type(old) is type(new), isinstance(old, widget.Widget)

widget.py body ran
widget.py body ran

(widget.Widget, widget.Widget, False, False)

Both objects report their class as widget.Widget, yet type(old) is type(new) is False and isinstance(old, widget.Widget) is False: old is now an instance of a class that no longer exists under that name. Reload also does not re-import already-loaded submodules (a reload-ed module that does import pandas keeps the old pandas). This is why reload is a REPL convenience, not a hot-swap mechanism: for anything stateful, restart the interpreter.

8.2 Prefer relative imports inside a package

Absolute imports (from mypackage.sub import thing) hard-code the package’s name into every import statement. Rename the package and every one of them breaks. Relative imports (from .sub import thing) refer to location, not name, and survive renames. Relative imports must use the from … import … form: import .sub is a syntax error, and a leading dot counts one level per dot (from ..pkg import x goes up two).

8.3 The cache is mutable

sys.modules is a plain dict, which is powerful and dangerous. Deleting a key forces a genuine reimport next time (a cleaner reset than reload). Aliasing a module under a second name (sys.modules["new"] = sys.modules["old"]) makes both names resolve to the same object. But setting an entry to None poisons that name: Python treats it as “known to be unimportable” and raises ModuleNotFoundError. Reach into the cache deliberately, not casually.

Key Takeaways

1. import is a cached pipeline, not a keyword that reads a file. Find → create → execute → bind, with sys.modules short-circuiting everything after the first import.
2. A package is just a module with __path__. Regular packages (__init__.py) stop the search and win; namespace packages (no __init__.py) merge directories from across sys.path.
3. sys.path is only the PathFinder’s input. The real dispatch layer is sys.meta_path → PathFinder → sys.path_hooks → per-entry finder → loader.
4. The ModuleSpec is the hub. Finders produce specs; loaders consume them; create and execute are decoupled, which is what makes lazy loading possible.
5. Source is compiled once and cached as version-tagged bytecode in __pycache__, validated against the source’s timestamp and size.
6. Everything is pluggable. Prepend a finder to intercept imports, append one to catch failures: that’s all tracing, lazy imports, and auto-install really are.
7. Know the traps. reload creates zombie objects, absolute imports are brittle to renames, and writing to sys.modules can corrupt the cache.

The Unavoidable Memory Cost

Here’s the catch. To compute gradients during the backward pass, each operation needs its inputs from the forward pass. For a mul node computing , the backward step needs both and to distribute credit:

So the framework must keep every intermediate tensor alive until the backward pass consumes it. The consequence:

• Inference: each layer’s activations can be discarded once the next layer is computed → memory is roughly in depth
• Training: all activations must survive until their gradient is computed → memory is in depth

This is why training a transformer consumes so much more memory than running inference on it. At large batch sizes, forward activations alone can dwarf the parameter memory.

Gradient Checkpointing: Buying Memory Back with Compute

The standard solution to activation memory pressure is gradient checkpointing (Chen et al., 2016) (also called activation recomputation): don’t store all activations during the forward pass. Store only at segment boundaries (checkpoints) and recompute intermediate activations on-the-fly during the backward pass when they’re needed.

flowchart LR
    subgraph s1 ["Segment 1"]
        L1["Layer 1"] --> L2["Layer 2"] --> L3["Layer 3"]
    end
    subgraph s2 ["Segment 2"]
        L4["Layer 4"] --> L5["Layer 5"] --> L6["Layer 6"]
    end
    s1 -->|"✓ checkpoint"| s2

Strategy	Activation memory	Compute overhead
No checkpointing	layers	None
checkpoints	layers	~1 extra forward pass
Recompute everything		Up to extra forward passes

The sweet spot for most LLM training is checkpoints: roughly one extra forward pass in exchange for a meaningful memory reduction. This is what torch.utils.checkpoint.checkpoint_sequential implements.

1.1 Content-Addressable Storage

The key design decision that makes Git work is content-addressable storage. Every object Git stores - every file, every directory listing, every commit - is identified by the SHA-1 hash of its contents. The hash is a 40-character hexadecimal string (160 bits), and it serves as both the object’s name and its address in the database.

# Pseudocode for Git's storage model
def store(obj):
    id = sha1(obj)
    objects[id] = compress(obj)
    return id

def load(id):
    return decompress(objects[id])

Git uses the first two characters of the hash as a directory name and the remaining 38 as a filename, all stored under .git/objects/. For example, a hash of ee5941ab3c... is stored at .git/objects/ee/5941ab3c....

This design gives Git three properties for free:

• Deduplication. Two files with identical contents produce the same hash, so Git stores only one copy. Rename a file? The blob stays the same; only the parent tree changes.
• Integrity. Any corruption - even a single flipped bit - changes the hash, so Git detects it immediately.
• Immutability. Objects are addressed by their content. You cannot change an object without changing its address. This append-only property is why committed data is almost always recoverable.

1.2 Blobs: File Contents Without Names

A blob (binary large object) stores the raw contents of a single file - nothing more. No filename, no permissions, no metadata. Just bytes.

type blob = array<byte>

When Git computes the hash of a blob, it prepends a header: the string "blob", a space, the content length in bytes, and a null byte. The SHA-1 of this combined string becomes the blob’s address:

SHA-1("blob 60\0" + file_contents) → ee5941ab3c...

Because blobs store only contents and not names, two files with identical contents - even in different directories, even with different names - map to the same blob. Git stores it exactly once.

# Inspect a blob - git cat-file -p shows the raw content
# First, find a blob hash from the current tree
!git cat-file -p HEAD^{tree} | tail -2

100755 blob 1b4e54a00da0021ed03ce4facf67cea88d230300    til.qmd
040000 tree 02a16aeaeaa4ae9113498e37832f78ece2bb27f4    til

!zlib-flate -uncompress < 1b/4e54a00da0021ed03ce4facf67cea88d230300 | head -2

blob 528---
title: Today Imad Learned

1.3 Trees: Directory Snapshots

A tree object represents a directory. It contains a list of entries, where each entry maps a name (filename or subdirectory name) to either a blob or another tree, along with a file mode (permissions).

// A directory maps names to blobs or subtrees
type tree = list<(mode, type, hash, name)>

Each line in a tree object looks like:

100644 blob a1b2c3d4...  README.md
040000 tree e5f6a7b8...  src/

Trees have several important properties:

• Trees do not store their own name. A tree’s name is assigned by its parent tree. The root tree - the top-level directory of the project - has no name at all, which is why renaming your repository’s local directory has zero effect on Git.
• Empty directories are invisible. A tree must contain at least one entry. Git cannot track an empty directory. The common workaround is placing a .gitkeep file inside it.
• Renaming is cheap. If you rename a subdirectory, only the parent tree changes. The subtree object and everything below it remain untouched - they have the same hashes and the same addresses.
• Trees are themselves hashed. The hash of a tree is computed over its list of entries. Change any entry (add a file, rename something, update a blob hash), and the tree gets a new hash - which propagates up to every parent tree, all the way to the root.

# Inspect the root tree of the latest commit
!git cat-file -p HEAD^{tree} | head -3

100755 blob bb6df347bcb140d911763ede8d70c72981c6b760    .gitignore
100755 blob e69de29bb2d1d6434b8b29ae775ad8c2e48c5391    .nojekyll
100755 blob 14d0bc4d92bee96cd7b778e97222ed8f18b5b8c5    404.md

1.4 Commits: Snapshots with Context

A commit object ties everything together. It points to a root tree (the snapshot), records who made the change, when, and why, and links to its parent commit(s) to form a history.

type commit = struct {
    tree:    hash           // pointer to the root tree (the snapshot)
    parent:  list<hash>     // zero parents (initial), one (normal), or two+ (merge)
    author:  string         // who wrote the change
    committer: string       // who applied the change
    message: string         // why the change was made (commit message)
}

A commit does not store diffs. When you run git diff between two commits, Git compares their root trees on the fly and computes the difference on demand. This is a key design choice: storing snapshots makes branching and merging fast at the cost of slightly more storage (mitigated by deduplication and packfiles).

Because unchanged files keep the same blob hash, and unchanged directories keep the same tree hash, most of a commit’s tree structure is shared with its parent. A commit that changes one file out of a thousand creates exactly one new blob, one new tree for its parent directory, and new trees up to the root - everything else is reused via identical hashes.

# Inspect a commit object - shows tree, parent, author, message
!git cat-file -p HEAD

tree 94ab6b930e1b2be4b0e68d6eb14f81671498f14d
parent 84381f8ce551c786c0a2b1564cf9a3d8e48319a4
author ImadDabbura <imad.dabbura@hotmail.com> 1775674279 -0500
committer ImadDabbura <imad.dabbura@hotmail.com> 1775674279 -0500

feat: Add few more tips/tricks

1.5 The Three States: Modified, Staged, Committed

Git manages files through three distinct states, mediated by three areas:

The lifecycle of a change:

1. Modified. You edit a file in your working directory. Git knows it has changed (by comparing its hash to the index) but has not recorded the change anywhere.
2. Staged. You run git add file. Git computes the SHA-1 of the file, compresses and stores the blob in .git/objects/, and records the hash in the index file. The index is now a draft of what the next commit’s tree will look like.
3. Committed. You run git commit. Git reads the index, builds tree objects for every directory, creates a commit object pointing to the root tree (with parent, author, and message), and updates the current branch to point at the new commit.

Files also fall into two tracking categories:

• Tracked: files that exist in the last commit or in the staging area. They can be modified, staged, or committed.
• Untracked: files Git does not know about. They appear in git status but are not included in commits until explicitly added.

1.6 Putting It All Together: A Worked Example

Let’s trace exactly what happens when you create a file and commit it.

Step 1 - Create a file:

echo "Hello, Git" > greeting.txt

The file exists only in your working directory. Git status shows it as untracked.

Step 2 - Stage the file (git add greeting.txt):

Git does three things:

1. Computes the SHA-1: SHA-1("blob 11\0Hello, Git\n") → ab3f...
2. Compresses the content and stores it at .git/objects/ab/3f...
3. Adds an entry to .git/index: 100644 blob ab3f... greeting.txt

Step 3 - Commit (git commit -m "Add greeting"):

Git does four things:

1. Reads the index and creates a tree object listing greeting.txt → ab3f... → tree hash d8e7...
2. Creates a commit object: tree d8e7..., parent <previous HEAD>, author ..., message "Add greeting" → commit hash f1a2...
3. Stores both the tree and commit as compressed objects in .git/objects/
4. Updates .git/refs/heads/main to contain f1a2...

The result is a chain: branch → commit → tree → blob(s).

graph LR
    B["main<br/><small>refs/heads/main</small>"] --> C["commit f1a2...<br/><small>tree: d8e7...</small><br/><small>parent: 9c3b...</small><br/><small>msg: Add greeting</small>"]
    C --> T["tree d8e7...<br/><small>greeting.txt → ab3f...</small>"]
    T --> BL["blob ab3f...<br/><small>Hello, Git</small>"]
    C --> PC["commit 9c3b...<br/><small>(parent commit)</small>"]

    style B fill:#f9d71c,stroke:#333,color:#000
    style C fill:#7eb8da,stroke:#333,color:#000
    style T fill:#90c695,stroke:#333,color:#000
    style BL fill:#e8a87c,stroke:#333,color:#000
    style PC fill:#7eb8da,stroke:#333,color:#000

2.1 Branches (Heads)

A branch is a file in .git/refs/heads/ that contains the SHA-1 hash of a single commit - the tip of that branch. When you create a branch, Git creates a 41-byte file (40 hex chars + newline). When you make a commit on that branch, Git updates the file to point to the new commit. That is the entire implementation of branching.

# Creating a branch is literally creating a file
$ cat .git/refs/heads/main
f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0

This is why branches in Git are so cheap: they are not copies of your codebase, not snapshots, not deltas - they are 41-byte text files. Creating a thousand branches costs roughly 41 KB of disk space.

2.2 HEAD: The Singleton Pointer

HEAD is a special reference that identifies where you currently are. Unlike branches, there is only ever one HEAD - it is a singleton. It lives at .git/HEAD and typically contains a symbolic reference to a branch:

$ cat .git/HEAD
ref: refs/heads/main

This tells Git: “the current branch is main.” When you commit, Git follows the chain: HEAD → refs/heads/main → update that file with the new commit hash.

When you check out a specific commit (not a branch), HEAD points directly at that commit hash instead of a branch name. This is a detached HEAD state - you are no longer on any branch, and new commits will not be reachable from any branch unless you explicitly create one.

2.3 Tags: Immutable Bookmarks

Tags are references stored in .git/refs/tags/. Like branches, they point to commits - but unlike branches, they are meant to be immutable. A tag marks a fixed point in history (typically a release: v1.0, v2.3.1).

Git supports two kinds of tags:

• Lightweight tags are simple files containing a commit hash - identical in structure to a branch, just never updated.
• Annotated tags are full Git objects stored in .git/objects/. They contain the tagger’s name and email, a timestamp, a message, and optionally a GPG signature. The tag file in refs/tags/ then points to this tag object, which in turn points to the commit. Annotated tags are recommended because they preserve who tagged, when, and why.

# Lightweight: just a pointer
$ git tag v0.1

# Annotated: a full object with metadata
$ git tag -a v1.0 -m "First stable release"

# Tag a historical commit
$ git tag -a v0.5 ca21323

2.4 The Ref Hierarchy

All references live under .git/refs/ in a clean hierarchy:

.git/refs/
├── heads/          # Local branches
│   ├── main
│   └── feature-x
├── tags/           # Tags
│   ├── v1.0
│   └── v2.0
└── remotes/        # Remote-tracking branches
    └── origin/
        ├── main
        └── feature-x

The following table summarizes the three reference types:

With the object model and reference system in place, we can now see how Git uses them for its most common operation: branching and merging.

3.1 Creating and Switching Branches

Creating a branch copies the current commit hash into a new file:

# Create a new branch (does not switch to it)
$ git branch feature-login

# Create and switch in one step (modern syntax)
$ git switch -c feature-login

# Legacy equivalent (still works, but overloaded)
$ git checkout -b feature-login

When you switch branches, Git does two things: (1) updates HEAD to point at the new branch, and (2) rewrites your working directory to match that branch’s latest commit. Files are added, removed, and modified automatically.

3.2 Fast-Forward Merges

The simplest merge scenario: you created a branch, made commits, and nobody else committed to the base branch in the meantime. The base branch’s tip is a direct ancestor of your branch’s tip - the history is linear.

In this case, Git does a fast-forward merge: it simply moves the base branch pointer forward to your branch’s tip. No new commit is created, no objects are created, no merge logic runs. It is literally updating a 41-byte file.

$ git switch main
$ git merge feature-login --ff-only

graph LR
    A["A"] --> B["B"] --> C["C<br/><small>main (before)</small>"] --> D["D"] --> E["E<br/><small>feature-login</small><br/><small>main (after)</small>"]

    style A fill:#7eb8da,stroke:#333,color:#000
    style B fill:#7eb8da,stroke:#333,color:#000
    style C fill:#7eb8da,stroke:#333,color:#000
    style D fill:#90c695,stroke:#333,color:#000
    style E fill:#90c695,stroke:#333,color:#000

The --ff-only flag tells Git: “only merge if a fast-forward is possible; otherwise, abort.” This is a good default for teams that value linear history.

3.3 Three-Way Merges

When both branches have diverged - the base branch has commits that your feature branch does not, and vice versa - Git cannot fast-forward. Instead, it performs a three-way merge using three reference points:

1. The common ancestor - the most recent commit reachable from both branches
2. The tip of the base branch (e.g., main)
3. The tip of the feature branch

Git compares each branch’s tip against the common ancestor to determine what changed on each side. If changes do not overlap (different files, or different regions of the same file), Git combines them automatically and creates a merge commit - a commit with two parents - that ties the histories together.

$ git switch main
$ git merge feature-login

graph LR
    A["A"] --> B["B"]
    B --> C["C"] --> D["D<br/><small>main</small>"]
    B --> E["E"] --> F["F<br/><small>feature</small>"]
    D --> M["M<br/><small>merge commit</small>"]
    F --> M

    style B fill:#f9d71c,stroke:#333,color:#000
    style D fill:#7eb8da,stroke:#333,color:#000
    style F fill:#90c695,stroke:#333,color:#000
    style M fill:#e8a87c,stroke:#333,color:#000

3.4 Merge Conflicts

When both branches modify the same lines of the same file, Git cannot determine which version is correct. It pauses the merge, marks the conflicting regions in the file with conflict markers (<<<<<<<, =======, >>>>>>>), and asks you to resolve them manually.

The resolution workflow:

1. Open the conflicting file(s) and edit the conflict markers to produce the desired result
2. Stage the resolved files: git add <file>
3. Complete the merge: git commit (this creates the merge commit)

Alternatively, abort the merge entirely: git merge --abort restores the state before the merge began.

For complex conflicts, visual merge tools like vimdiff, VS Code’s built-in merge editor, or dedicated tools like meld can help. Configure your preferred tool with git mergetool.

Branching and merging work locally. The next section extends these ideas to collaboration across repositories - remotes, tracking branches, and the fetch/push protocol.

4.1 The Remote Model

A remote is a named URL pointing to another Git repository. When you clone a repo, Git automatically creates a remote called origin pointing to the source URL. You can have multiple remotes - for example, origin for your fork and upstream for the original project.

# List remotes
$ git remote -v

# Add a second remote
$ git remote add upstream https://github.com/original/repo.git

Remote-tracking branches live under .git/refs/remotes/<remote>/ and act as read-only bookmarks of where each branch was on the remote the last time you communicated with it. You never update them directly with git commit - Git manages them automatically during fetch and push.

4.2 Tracking Branches

A tracking branch (or “upstream branch”) is a local branch that is linked to a remote-tracking branch. This link tells git pull where to fetch from and git push where to push to, without specifying the remote and branch name every time.

Tracking is set up automatically in several cases:

# Cloning: 'main' automatically tracks 'origin/main'
$ git clone https://github.com/user/repo.git

# Switching to a remote branch name creates a tracking branch
$ git switch feature-x  # creates local 'feature-x' tracking 'origin/feature-x'

# Explicit tracking setup
$ git branch --set-upstream-to=origin/feature-x

# Push and set upstream in one step
$ git push -u origin feature-x

You can also track branches from different remotes, or use different local and remote names:

# Track a branch from a different remote
$ git switch -c my-local-name upstream/their-branch

4.3 Cloning: What Actually Happens

When you run git clone https://github.com/user/repo.git, Git performs four steps:

1. Creates a directory named repo/ with a .git/ subdirectory
2. Downloads the entire object store - every blob, tree, and commit in the history
3. Creates remote-tracking branches under refs/remotes/origin/ for every branch on the remote
4. Creates a local main branch tracking origin/main and checks it out

The critical implication: cloning downloads all versions of every file ever committed. If someone committed a 500 MB binary three years ago and then deleted it, that blob is still in the history and gets cloned. For repositories with very long histories, git clone --depth N creates a shallow clone with only the last N commits.

4.4 Pushing and Pulling

The three synchronization commands serve distinct purposes:

Fetch is always safe - it only downloads data and updates bookmarks. It never touches your working directory or local branches.

Pull is fetch + merge (or fetch + rebase if configured). Because it merges, it can create merge commits or conflicts.

Push uploads your commits and asks the remote to update its branch pointer. If the remote has commits you do not have (someone else pushed first), the push is rejected - you must pull and integrate their changes first.

# Fetch all branches from origin
$ git fetch origin

# Pull (fetch + merge) the current tracking branch
$ git pull

# Push the current branch to its upstream
$ git push

# Push with a different remote branch name
$ git push origin local-branch:remote-branch

# Delete a remote branch
$ git push origin --delete old-branch

With local and remote operations covered, the next step is learning to navigate the history that these operations produce.

5.1 git log: Viewing History

git log is the primary tool for browsing commit history. Its power comes from filtering and formatting options:

# Visual overview: graph, one line per commit, all branches
$ git log --all --decorate --graph --oneline

# Last 5 commits
$ git log -5

# Custom format
$ git log --pretty=format:'%C(yellow)%h%C(reset) - %an [%C(green)%ar%C(reset)] %s'

# Commits affecting a specific file
$ git log --oneline -- path/to/file.py

# Search commit messages (extended regex, case-insensitive)
$ git log -E -i --grep 'fix.*login'

Two especially powerful search modes:

• git log -S "term" (the “pickaxe”): finds commits that changed the number of occurrences of a literal string. If a function was added or removed, this finds the commit.
• git log -G "regex": like -S but matches a regex pattern against the diff, finding commits where the patch itself matches.

git show displays a single commit’s metadata and diff in one view - the quickest way to understand what a commit did:

# Show the latest commit's diff
$ git show

# Show a specific commit
$ git show a1b2c3d

5.2 git blame: Per-Line Attribution

git blame annotates each line of a file with the commit that last modified it, who did it, and when. It is indispensable for understanding why code looks the way it does.

# Full blame
$ git blame path/to/file.py

# Restrict to a line range
$ git blame -L 50,75 path/to/file.py

# Detect code moved or copied from other files
$ git blame -C path/to/file.py