CHAD: building a 99M-param Hinglish texting bot from scratch

Everyone is building wrappers around someone else's model. I wanted to know what is actually inside the box, so I built one. From scratch. No pretrained weights, no fine-tuning somebody else's checkpoint. A real decoder-only transformer, hand-written in PyTorch, 98,913,024 parameters, pre trained on four billion tokens of text I had to go out and manufacture myself. I distilled ~100k Q/A dataset from gpt/deepseek models for SFT and lastly did a DPO to nerf the dark humour it picked from Reddit.

1. Writing the transformer by hand

I did not import a model. I wrote every piece: rotary position embeddings (RoPE), grouped-query attention (GQA), a SwiGLU feed-forward, RMSNorm. This is the modern Llama and Mistral recipe, shrunk down. The final shape is d_model 768, 12 layers, 12 query heads but only 3 key/value heads, a 1024-token context, and an output head that shares its weights with the embedding. That lands at exactly 98,913,024 parameters.

Writing each piece by hand was the only way I actually understood why it is there:

• RoPE encodes position by rotating the query and key vectors, so attention depends on the distance between two tokens rather than their absolute index. It adds zero parameters and keeps working past the length you trained on.

• GQA is why there are 3 KV heads instead of 12. At generation time the model caches a key and value for every past token, and that cache is what eats memory. Sharing each KV head across 4 query heads shrinks the cache 4x for almost no quality loss.

• SwiGLU is a gated feed-forward, three matrices and a SiLU gate, and it beats a plain ReLU block at the same parameter count. Every time.

• Weight tying makes the input embedding and the output projection literally the same matrix. That one decision saves 24,576,000 parameters, a quarter of the whole model, and couples the two representations in a way that helps something this small.

Before the real thing I built a throwaway version first: 12.6 million parameters, six layers, small enough to overfit a single batch on my laptop as a test. Once that passed and I understood every line, I scaled the exact same code up and threw the small one away. That is the whole trick to writing a model from scratch. Get it provably right when it is tiny, then only change the numbers.

2. A tokenizer for a language that does not spell consistently

Hinglish has no spelling standard. "bhai", "bhaii", "bhaiya", all valid, all in the wild. People code-switch mid-sentence and drop an emoji into the middle of a word. A normal English tokenizer meets this and emits a stream of garbage.

So I trained a custom 32,000-token byte-level BPE tokenizer on the corpus. Byte-level is the load-bearing word. The base alphabet is literally the 256 possible bytes, so every input is already made of known tokens and there is no such thing as an out-of-vocabulary character, ever. No <unk>, no fallback, just bytes merging up into whole Hinglish words. It comes out to 5 special tokens, 256 byte tokens, and 31,739 learned merges.

I also forced digit-splitting, so every number breaks into single digits and the model never memorizes "2024" as one atom. Hold onto that decision. It is quietly the reason a later chapter has a punchline. The whole tokenizer trained in 159 seconds on my Mac and ships as a single 2.2 MB file, and it compresses the corpus at about 4.31 characters per token.

3. Manufacturing four billion tokens of Hinglish

This was the hardest part of the entire project, and none of it is machine learning. There is no big clean romanized-Hinglish pretraining set. The giant web crawls are overwhelmingly English. The serious Indian-language corpora are in Devanagari script, which is the wrong format, because people do not text in Devanagari, they text in roman letters. So the corpus did not exist, and I had to build it.

The one place on earth with billions of tokens of real, casual, romanized Hinglish is Indian Reddit. I pulled it in bulk from the Pushshift archive torrents, no Reddit API, no rate limits. The whole thing hinges on one detail.

The Pushshift dumps come in two layouts of the same data. One is by-month, which is useless, because grabbing a single community means downloading roughly 4 TB and scanning all of it. The other is by-subreddit, one file per community, which is exactly what you want. A torrent file is just a small catalog, so I parsed its 79,955 entries in memory, name-filtered down to 214 Indian subreddits, and handed only those file indices to the downloader. One non-obvious flag was load-bearing: without telling aria2c to skip preallocation, the client tries to reserve the entire multi-terabyte torrent on disk before pulling a single byte.

Then came cleaning, where the real work hides. Drop the bots and AutoModerator. Drop anything not in Latin script, because some "Indian" subreddits are actually Malayalam or Tamil, the wrong language for this corpus. Drop the junk, the copypasta, the symbol spam. Dedup with a fixed-memory Bloom filter, which cut 6.48 million duplicate documents. What survived was about 4.13 billion clean tokens across 128.9 million comments, packed into a compact binary stream, 21 GB on disk. That pile of unhinged Indian Reddit is the entire personality of the model before I ever fine-tuned it.

4. Training on free TPUs, and the five ways it broke

The tokenizer ran on my Mac. The pretraining ran on Kaggle's free TPU v5e-8.

It was not smooth. Getting one clean 4-billion-token pass took five attempts, and each failure taught me something specific about how PyTorch behaves on a TPU instead of a GPU:

• Moving the model to the TPU silently broke my weight tying. The device copy hands each parameter a fresh tensor, which quietly split my shared embedding back into two separate matrices and inflated the model from 98.9M to 123.5M parameters. Fix: re-tie the two after the model is on the device, and assert they are the same object so it can never drift again.

• The job died at exactly 460 seconds with no obvious error. The cause was a missing barrier. On a TPU the compute graph does not close until you tell it to, and without that one line the graph grew without bound. Step time climbed 14s, 39s, 82s, 142s, 221s, host memory filled with the graph, and the kernel got killed. One line fixed it. One night found it.

• Every 2,900 steps or so the whole thing froze for 17 minutes. I was writing a 1.2 GB checkpoint every 250 steps to a cloud-backed disk, and the dirty pages piled up until the system forced a synchronous flush and stalled all eight chips. Fix: checkpoint 10x less often.

• Batch size 32 ran out of memory before the first step. On a GPU, attention quietly dispatches to a memory-frugal kernel. On a TPU it does not, it builds the full attention-score matrix in memory, which at batch 32 wanted 24.9 GB against 15.75 GB of actual chip memory. Fix: batch 8, which fits with room to spare.

• Stopping cleanly at the session wall could deadlock all eight chips, because one chip would decide to stop while the others waited on it at the next sync. Fix: make the stop decision identically across all eight before anyone acts on it.

The clean run landed at a validation loss of about 3.77, which took two Kaggle sessions stitched together because a single 4-billion-token pass is longer than the 8-hour session wall. The base model wrote fluent, idiomatic Hinglish and had no manners at all. No sense of when a turn ends, because I had pretrained with no boundary between comments, and a habit of leaking raw Reddit. Everything after this exists to fix that.

5. You cannot mine a personality that Reddit downvotes

Here is where the project almost went wrong in an instructive way. My first fine-tune idea was the obvious one: sort the real Reddit comment-and-reply pairs, keep the highly upvoted ones, SFT on those which gave me approxx 132k pairs which had 150+ upvotes. It learned the chat format in about a hundred steps and then it turned wholesome, Supportive and talked like an average reddit commentor which is not what I wanted.

That is not a bug, it is a finding. What gets upvoted on Reddit is kind and relatable, not savage. The voice I wanted is a minority voice that the platform actively buries. I confirmed it the expensive way before accepting it: I spent about 8 dollars running Claude Haiku over roughly 109,000 candidate comments as a judge, scoring each for whether it carried any roast signature. Around 1.1% did. And about 91% of the highest-scored comments were replies to posts whose original text was not in the dump at all, so there was no question to pair the answer to. Mining was a dead end, twice over.

The conclusion is the useful part. If the personality you want contradicts what the source material rewards, you cannot filter your way to it. You have to generate it.

6. Stealing a sense of humor

Here is the idea I am proudest of, and the first real brush with the wall. A 99M model cannot be funny. That is just true. It can learn the shape of a joke, the rhythm, the slang, the gif at the end, but it cannot write one, because a real joke needs world knowledge and a read on what the other person finds cutting, and a model this small has neither. So I stopped asking it to be funny.

Instead I used backtranslation. Take a real, upvoted Hinglish comment from reddit. That comment is already a funny, human-written answer. Now hand it to a large teacher model and ask the easy question in reverse: what are one to three prompts this comment would be the perfect reply to? You get back a clean chat example where the hard part, being funny, was done by a human, and the teacher only did the trivial part, inventing a setup. It sidesteps both mining failures at once. You do not need the comment to be a roast, and you do not need the original post to exist.

Two things made this affordable. The teacher calls would normally be a real bill, tens of thousands of them, so I routed the GPT-5.5 calls through XyPro, a small side project of mine that wraps a Codex subscription behind a plain OpenAI-compatible endpoint on localhost, and sent the cheap bulk to DeepSeek V4 Flash. DeepSeek had one genuinely cursed footgun: you must send thinking: disabled exactly, or it leaves reasoning on, dumps the answer into a hidden field, and hands you back empty content. That one cost me a couple of hours.

The rest of the personality did not come from Reddit at all. I wrote a persona spec by hand and generated a default voice across dozens of everyday situations, greetings, sad messages, smalltalk, homework, roast bait. And the persona itself took six full data rewrites to land. The first three chased a loud, cocky bhai that roasted everything, including people who were genuinely upset, which was just mean and bad. From version four on I threw that out for a chill texter: lowercase, dry, roughly half Hindi and half English woven into every line, with one rule, match the other person's energy and only go savage when invited.

The thing I want to stress: across all six rewrites, the model never changed. Same 98.9M weights, same architecture, byte for byte. Every single improvement came from regenerating the data and re-measuring. That is the actual job. The model is the cheap part.

7. A calculator that cannot do math

I gave chad two tools, a calculator and reaction gifs, under one hard rule: no new tokens. Adding a special token means a bigger embedding table, a resized output head, and re-running the whole export. So the tags are just ordinary text the model learns to type, <calc> and <gif>, and the runtime spots them with a regex on the output.

The calculator is a real little handoff. The model writes the expression, then stops dead. Control returns to the runtime, which does the actual arithmetic with a whitelist-safe evaluator that will only ever see digits and + - * / ( ) . %, never arbitrary code. Then the model gets the answer back and writes its real reply around it.

This is where it got interesting. The mechanism is flawless. On a 16-prompt test the model fired the tool, wrote a valid expression, and used the result in character, 16 times out of 16. But the arithmetic it reaches for is only right about 56% of the time. Direct "A plus B" is perfect. Word problems fall apart, because the model grabs the wrong numbers out of the sentence. Ask it to subtract the price of a phone from your savings and it will happily subtract two numbers that appear nowhere in the problem. Same model, same feature, one half learned perfectly and the other half slammed into a wall. Remember which half was which. It is the whole punchline.

8. Grading a roast

You cannot score a roast with perplexity, because there is no correct answer to grade against. So I used LLM-as-a-judge: DeepSeek V4 Flash scoring every reply on a rubric, relevance, persona, coherence, plus a hard flag for whether it roasted someone who did not ask for it. When the persona pivoted from savage to chill, I had to rewrite the rubric too, because the old grader would have cheerfully rewarded the exact cringe I was removing.

On a held-out set of 100 prompts the shipped model scores 1.82 out of 2 on relevance, 1.98 out of 2 on persona, and leaks the raw Reddit voice zero times out of 102, down from a model that used to do it constantly. It stops cleanly every single time. I also put an earlier version in a five-round insult arena against GPT-5.5 and let the judge call each round. GPT-5.5 won every battle, four to zero. That loss was useful. It surfaced exactly how the early model failed, and it is a big part of why the shipped model is the one it is.

9. Shipping it to a browser for nothing

This is the part I am smug about. The model is small enough to send to the user and run on their machine. No inference server, no GPU, no per-token bill. It downloads once, caches, and every token after that is generated on the visitor's own CPU.

The conversion is almost a cheat. My hand-written architecture is structurally identical to a Llama, same RMSNorm, RoPE, GQA, SwiGLU, tied embeddings, so turning my checkpoint into a HuggingFace Llama is a pure renaming of the weight keys, no surgery. I parity-check the logits of both models on random input and refuse to ship unless they agree to within a thousandth. Then it exports to ONNX and runs through transformers.js on WebAssembly, inside a web worker so generation never freezes the page.

One counterintuitive call: I ship the full-precision model, 472 MB, instead of the quantized one that would be a quarter of the size. Quantizing to int8 quietly wrecks the calculator, because squashing the weights breaks the model's ability to copy digits cleanly. On my calc test the int8 model scored 2 out of 10 against the full model's 7. I would rather make you download the big file than ship broken math. On the load screen you actually pick which brain you want, the raw fine-tuned model or a calmer preference-tuned one, framed as "feral mode" and "big brain mode", because that felt more honest than pretending there is one right answer.

It runs about 78 tokens a second on my Mac. There is a small backend that does nothing but log, and the chat works perfectly even when it is completely down. Every conversation a real person rates becomes a candidate training example, so the thing feeds its own next version. Free inference, free data. I like that a lot.

10. The wall

If you take one thing from this, take this. Across six data rewrites and a preference-tuning pass, every single capability sorted cleanly into one of two buckets.

Some things moved every time I improved the data. Voice, relevance, staying on topic, manners, output format, when to fire a tool, not leaking Reddit. Call these data-addressable. They are pattern-level behaviors, and enough good examples install them.

Other things never moved. Ever. Real humor, factual knowledge, doing math out of a word problem, and holding a fact across a conversation. Tell chad your name, then ask for it two turns later while your name is still sitting right there in its context, and it will confidently say a different name. Look at the raw numbers and the correct name is sitting around rank 4,000 out of 32,000, which is to say the model has essentially no idea. Call these parameter-bound. No amount of cleaner data touches them.

The cleanest proof is that calculator from chapter 7. The tool protocol, which is a pattern, is 16 out of 16. The arithmetic, which is reasoning, is 56%. Same model, same prompt, in a single feature, one half of it is data-addressable and the other half is parameter-bound.

Pattern-level coherence is data-addressable. Variable-binding and reasoning are parameter-bound.

That one line is the whole project. It is also why every fix in this post is the same move underneath. Do not ask the small model to reason. Source the reasoning from outside, a real human's joke, a deterministic calculator, and let the 99M model do the one thing it is genuinely great at, which is sounding exactly like a guy from your group chat.

What I actually learned

I set out to demystify the box and I did. A modern language model is RoPE and grouped-query attention and a gated feed-forward and a mountain of data and a lot of plumbing that nobody puts in the paper. None of it is magic. The part that actually is magic, the reasoning and the facts and the wit, is the part you cannot buy with better data. It shows up with scale and only with scale, and now I have felt exactly where that wall sits with my own hands, on hardware that cost me nothing.

chad is tiny and it is dumb and it talks exactly like my friends, and it runs in a browser tab for free. Honestly could not be prouder of the little guy. 🫡