CARBON

Same idea as GPT completing a sentence, but for DNA. We feed the model a DNA sequence as input and the model produces an output sequence. The model streams the bases one 6-base token at a time. The model is better at predicting sequences of a gene's exons because they are the protein-coding parts of a gene and are under strong evolutionary constraint. As such they should be the most predictable stretches of DNA. The introns serve regulatory purposes on the other hand and are harder to predict. We overlay the real exon/intron annotations on top of the output so you can compare what Carbon produces to what's actually there.

The Carbon model assigns every 6-base chunk a log-probability under the surrounding context: how "expected" or "likely" that stretch of DNA is. The plot with the scores along a real gene shows the curve dips and rises. We overlay the exon/intron annotation on top: confidence reliably climbs in protein-coding regions and falls in repetitive or unconstrained intronic stretches, even though the model never saw a single label. The same score, summed up, is what powers the variant-effect call in §3 below.

§2 showed that Carbon's per-base confidence rises and falls in step with gene structure. Now we use the same log-likelihood, but as a measure for individual mutations. For a real ClinVar variant we score a ~4 kb window of human DNA two ways: once with the original base, once with the mutation. Then we check which version looks more like real, functioning human sequence. Carbon was never trained on what "pathogenic" means; it just learned what natural DNA looks like. Variants that disrupt protein-coding or regulatory function show up as less likely sequence under the model's distribution.

The same gene (insulin, p53) exists in humans, mouse and chicken, but the surrounding sequence has accumulated different mutations along each lineage for hundreds of millions of years. For each species we feed Carbon up to ~400 bp and ask it to continue. Each continuation should match that species' real DNA better than another species' would. The model handles closely-related species well (mouse, chicken, even though they're ~300 My from human); the further you go back in evolutionary time, the more the surrounding sequence drifts and the harder this setup becomes.

When Carbon completes a protein coding region in a gene, the resulting bases translate to a protein: a protein that folds. We feed the resulting sequence into ESMFold (similar to AlphaFold) and render the 3D structure inline, alongside the same protein folded from the reference sequence so you can see whether Carbon's continuation produced something similar.

We embed 571,810 genes from 27 species across six kingdoms (vertebrates, invertebrates, plants, fungi, bacteria, viruses) with Carbon, project to 2D with UMAP, color by attributes. Depending on the attribute, different kinds of organizations emerge from the same points: the model's embedding space encodes multiple axes of biology at once, most of which were never labeled.

If we take the same 571,810 sequences from §6 and average each species' embeddings into a single 3072-dim vector, then cluster those 27 centroids with hierarchical clustering, we can find species the model regards as closely related. Carbon was never trained on what the relation between organisms is. Yet the resulting tree groups vertebrates together, separates bacteria from fungi, and pairs sister clades (primates with primates, rodents with rodents, monocots with monocots).

The most direct way to model DNA is one base per token. It works, but for a L-base sequence Transformer attention costs L², and DNA contexts are long. Carbon instead reads in fixed 6-base blocks. Same DNA span, ⅙ the tokens, and because attention is quadratic, up to 36× cheaper at the same coverage. BPE was a tempting middle ground, but its variable-length tokens collide badly with autoregressive next-token prediction: DNA doesn't have stable "words."

Cross-entropy treats every 6-mer token as atomic: predict TATATT when the target was TATATA, get zero credit even though five of six bases matched. That gets brittle late in training. Carbon switches to Factorized Nucleotide Supervision: instead of one 4096-way classification, the model is supervised on six parallel 4-way nucleotide marginals derived from the same logits. Near-miss tokens get partial credit proportional to how many bases they got right.

The 6-mer tokenizer makes Carbon fast, but it's coarse in both directions of inference. When generating, each step advances the sequence by 6 bases at once and temperature acts on a 4,096-way distribution rather than per nucleotide. When scoring an existing sequence, the raw next-token likelihood answers "how likely is this 6-mer in context?", not "how likely is this exact base at this exact position?", which is the version you want for variant-effect prediction. The same marginalization that powers FNS at training time fixes both: softmax over the 6-mer logits, then for each position p sum the probabilities of every 6-mer that shares a given base at p, and you recover six per-position 4-way base distributions. To generate, sample (or argmax) each independently and force the matching 6-mer token. To score, read P(actual base | context) directly off the marginals at every position. Same logits, same math, two endpoints.

A naive read of "more data is better" misses something specific to DNA: most of a eukaryotic genome is repeats, low-complexity, and weakly-constrained background. Train on raw sequence and a lot of your loss is dominated by easy-to-predict noise. Carbon's corpus is an annotation-aware mixture, biased toward gene-centric, transcript, and bacterial sequence, so the model spends more of its gradient updates on biologically meaningful sequence.

Decoder-only, RMSNorm + SwiGLU + RoPE + grouped-query attention, tied I/O embeddings, 8k-token context. Nothing exotic. The architectural surface is intentionally familiar so that any improvement Carbon shows on genomic tasks is attributable to the data, the tokenizer, and the loss, not to a custom block or a hand-crafted attention variant.

Carbon's nominal training context is short by megabase-scale standards (8k tokens, ≈49 kbp). The reach comes from a two-step extension. First, a training-time long-context phase lifts the context to 32k tokens (≈197 kbp) with RoPE θ rescaled from 500k to 5M. Then, at inference, YaRN pushes that further: 2× to 65k tokens for the 3B model, 4× to 131k tokens for the 8B (≈786 kbp, the size of a small bacterial genome). The 8B has more capacity to absorb the YaRN stretch, which is why it extends further than the 3B.

Eight training-free tasks across four capability axes: generative sequence recovery, variant-effect prediction (BRCA2, TraitGym, ClinVar coding / non-coding), sequence-level perturbation (synthetic motif insertion and synonymous codon shuffling), and long-context retrieval (Genome-NIAH at 393 kbp). No fine-tuning, no head training, all four frozen pretrained models scored under the same protocol. Carbon-3B is competitive with Evo2-7B despite less than half the parameters; Carbon-8B is ahead on five of eight.

The throughput story is a two-factor multiplication, not one big trick. First, the architecture is deliberately vanilla: a stock Llama-3-shaped decoder. That means Carbon drops straight into vLLM and inherits the same paged-attention, fused kernels, and CUDA-graph capture that the open-source LLM stack has been optimizing for two years. Custom blocks would forfeit all of that. Second, 6-mer tokenization compresses a given DNA span by 6× at the input, which under quadratic attention is up to a 36× reduction in prefill cost, and the decode loop emits 6 bases per step instead of one. Stacking the two: standard-stack inference speedups, multiplied by tokenizer compression, gets you the order-of-magnitude gap over Evo2 reported in the paper.