What you see above is a fruit fly flying around a living room, trying to follow a smell cloud, landing on a sugar bowl and extending its proboscis to eat. Which way to turn away from an obstacle, whether the smell is getting stronger or fading, which side the wind blows from, or whether the thing it landed on tastes of sugar is controlled by a complete central nervous system of a real fruit fly - 166,700 neurons and 24.9 million synapses, wired exactly as an electron microscope found them - simulated in real time, in pure C#, on a laptop.

Well… sort of. And this “sort of” is what this post is about.

What is this all hype about?

On September 3, 2026, Google Research and HHMI Janelia published the MaleCNS connectome - the complete wiring diagram of the brain and the “ventral nerve cord” (which carries signals between the brain and the body) of one adult male Drosophila - a fruit fly. Within days the internet did what the internet does: the fly brain was playing Doom, Super Mario 64, jumping over cacti, walking in the browser, and sitting on a Mac desktop.

I got hooked. Just like with dotLLM, I wanted to understand what is really going on there - and for me, with AI, the best way to really understand something is to build it. So I built dotFly: a native .NET “inference engine” of fruit fly-connectome, fast enough to sit inside a Godot game loop. No Python, no native lib wrappers, purely CPU-based.

This post covers: what a connectome actually is and how it was made, what the “fly brain model” everyone runs really is (and even more importantly, what it is not), what all those demos are doing, how dotFly does it in C# with some benchmarks, ending with honest limits of the whole thing.

Wait, is this ethical?

Before we go into the details, a question that pops up immediately: if this is simulating a real brain, should we be doing this at all? Is the simulated fly suffering when it bumps into a wall in Mario? People asked exactly that on r/AskScienceDiscussion and on Hacker News, where one commenter asked: “Is the utility of this program worth a 10% chance that we confidently create perhaps the most cruellest machine possible?”. Do we create a “real living” that we torture in a virtual environment?!

No matter how sci-fi those questions sound, I think it is fair to ask, and I don’t want to dismiss it. But the honest answer requires understanding what is being simulated, and the short version is: it is in some sense simulating the real brain - the wiring is measured from a real animal, neuron by neuron, synapse by synapse - and at the same time it is very far from a living brain. It leaves out much of the brain’s chemistry and detailed ways neurons communicate, and ignores what happens inside each neuron. There are no hormones, no internal state, and no learning. Just kind of “one weight for all 25 million synapses” simplification. The demos “play games” through hand-written adapters that would make any nonlinear dynamical system look like it plays. Nothing in this model establishes that the scanned individual, its memories, or anything resembling experience was preserved.

So no, I don’t think we created “the cruellest experiment possible”. Well, yet. But I also think the discomfort is a useful reminder to be precise about what these demos are - which is what the rest of this post tries to be.

From the basics: what is a connectome?

Three terms are enough to understand everything that follows:

• connectome - the reconstructed “wiring” diagram: which neuron connects to which, with how many synaptic contacts, with what neurotransmitter
• neural model - the equations that make that wiring active (the connectome itself doesn’t compute anything, it’s a graph)
• embodiment - an engineered “interface” between the model and a body, a game, or a world

The internet downloaded that wiring diagram (connectome) and added dynamics and interfaces. It did not download a frozen executable copy of a living fly.

How it was made

It is easy to compress the whole story into “Google mapped a fly brain in September”. The reality is a chain of achievements spanning many years of research:

• 2018 - Zheng et al. publish the first complete electron microscopy volume of an adult female fly brain (the FAFB dataset). Just the images - a ~100 TB stack of slices
• 2024 - the FlyWire consortium publishes the full reconstruction of that brain: ~140,000 neurons, 50 M synapses (Dorkenwald et al., Nature)
• 2024 - Shiu et al. (Nature) publish the first computational model of that whole brain and validate it against real experiments. This is the neural model everybody runs now
• 2025-2026 - MaleCNS: one five-day-old adult male (specimen Z0720-07m) connectome, brain + neck + ventral nerve cord dissected together. Fixed with aldehydes, stained with heavy metals, embedded in resin, cut by a hot knife into 66 slabs ~20 µm thick, and imaged with enhanced focused-ion-beam SEM at 8×8×8 nm resolution: a gallium ion beam mills away a thin layer, the SEM images the new face, repeat. Seven customized instruments, about 160 teravoxels, ~13 months of imaging. Then Google’s flood-filling networks segmented the neurons, automated detectors found the synapses, a ResNet50 classified the neurotransmitter from EM neighborhoods - and 29 expert proofreaders spent about 44 person-years fixing and annotating it all. Result: 166,700 neurons, 11,710 cell types, 25.6 M directed edges, 124 M synaptic contacts. Published in Cell on September 3, 2026 (Berg, Beckett, Costa, Schlegel et al.), with the v1.0 data available since June.

What it is NOT

A connectome is a “photo” of a dead, sliced, metal-stained brain. It is not an fMRI or an electrode recording of a living brain. It does not measure:

• membrane voltages or any activity at all
• the functional strength of a synapse (only how many contacts there are)
• hormones impact
• anything that was learned, remembered, or felt at that time (well, the fly was dead…)

It is a wiring diagram - a very, very good one - with a predicted neurotransmitter label per neuron.

What are the “inputs” and “outputs”?

If the graph is just 166,700 neurons with IDs, how do we know where the eyes are? This is where a century of Drosophila genetics and neuroanatomy comes in. Each neuron in the model carries an annotation - a cell type (LC4, T4a, MN9, DNp04…), a class, a side - assigned by proofreaders from its morphology, location and connectivity, matched to the literature. And the literature is huge, for example we already know that:

• LC4 neurons are known (from calcium imaging, optogenetics, behavior) to respond to looming objects (objects getting closer/bigger),
• T4a-d to motion in four directions,
• ORN_DM1 to a specific odor/smell,
• the so-called sugar gustatory receptor neurons (GRNs) to sugar,
• MN9 is the motor neuron that extends the proboscis,
• DNp04 and the Giant Fiber are the escape descending neurons (‘outputs’) that go down to the nerve cord.

So, the sensory neurons are the inputs for example: photoreceptors, smell- and taste-sensing neurons, movement- and touch-sensing neurons for the fly’s “antenas”.

And the so-called descending and motor neurons are the outputs: ~1,300 descending neurons carry the brain’s commands to the nerve cord, motor neurons drive muscles. That’s not an assumption of the model - it is the anatomy.

Let’s follow one signal through, the simplest reflex a fly has - something is coming at me, get out:

graph LR
    A["a dark spot grows<br/>in the left eye"] --> B["photoreceptors<br/>R1–R6"]
    B --> C["lamina, medulla<br/><small>the optic lobe</small>"]
    C --> D["LC4 (left)<br/><small>71 neurons:<br/>'something is looming'</small>"]
    D --> E["DNp04, DNp02, DNp11,<br/>Giant Fiber<br/><small>descending neurons</small>"]
    E --> F["ventral nerve cord<br/>→ wing & leg motor neurons"]
    F --> G["take off,<br/>turn away"]

Now the same thing as an experiment. I take the MaleCNS graph, drive the 71 left LC4 neurons with random spikes at 120 Hz for one second - this is what “an object is expanding in the left eye” looks like to the brain - and record who fires. One dotfly run --stimulate type:LC4@L --hz 120 later:

4,792 neurons became active in total (out of 166,700), and the top of the list is exactly the escape pathway the literature describes - on the same side as the stimulated eye. Nobody told the model that LC4 is wired to DNp04. The “wiring” is simply there, and activating the input lights the output. That’s the whole game: activate the inputs, observe the outputs.

The model everybody runs: Shiu et al. 2024

The graph does nothing by itself. Shiu et al. made it run with the simplest thing that could possibly work: every neuron is a leaky integrate-and-fire (LIF) point neuron. Forget the shape of the neuron, its dendrites and ion channels. What is left is a single number, its “membrane potential”. Incoming spikes push it up or down, it leaks back toward a resting value, and when it crosses a threshold the neuron emits a spike of its own and resets. That’s it:

Note how much it takes to fire one: three spikes from a neuron with 90 synaptic contacts, within a few milliseconds - a single spike from a single input neuron does nothing on its own (it would need about 162 contacts). Firing is convergence.

How hard a spike pushes is the only place where the connectome enters the equations. Every edge from neuron $i$ to neuron $j$ has a weight:

$n_{ij}$ is the number of synaptic contacts between the two neurons - the microscope counted them. $w_{syn}$ is the same value for all 25 million synapses: 0.275 mV per contact. This is the single calibrated parameter of the whole model. Shiu et al. fitted it so that stimulating the sugar-sensing neurons produces a proboscis-extension response of realistic strength, and left it there. Everything else - time constants, threshold, refractory period, synaptic delay - came from previous electrophysiology.

The $\operatorname{sign}(\text{NT}_i)$ needs a word of explanation for the non-biologists like me. A neuron talks to the next one by releasing a chemical, a neurotransmitter, at the synapse - and each neuron uses (mostly) one kind. Some of those chemicals excite the receiving neuron, pushing its potential up toward the threshold; others inhibit it, pushing it down. The connectome does not measure which chemical a neuron uses either - a classifier predicts it from how the synapse looks in the EM image. The model then reduces the whole chemistry to one bit: in the fly, acetylcholine is the main excitatory transmitter, so +1. GABA and glutamate are inhibitory, so -1. Three others - dopamine, octopamine, serotonin - are in reality modulators: they change how other synapses behave, they carry things like reward and arousal, and a LIF model has no way to express that. Shiu et al. simply treat them as excitatory, +1. It is a modelling convention, not physiology: assigning +1 to a dopamine neuron does not implement reward or learning.

Researchers used the model to make predictions - “if we activate these neurons, those should fire” - and tested 164 of them against real flies with optogenetics and calcium imaging. If paper says 91% agreement - it means exactly that - 91% of those 164 predictions in feeding and grooming circuits. Not 91% of entire fly behavior.

The connectome is fixed - we do not train it

This is the crucial point. The weights are fixed, coming from the number of pairs between each neurons. Nobody trains this network to play Mario, and nobody should: the moment you start fitting the 25M weights to a game score, you no longer have “the fly’s brain” - you have some neural network that happens to have started from a fly-shaped graph. The original model has no plasticity rule at all.

So the whole “game” is, again: activate the inputs, observe the outputs. Drive the sugar neurons with Poisson spikes at 150 Hz, and watch which of the other 127,000 neurons start to fire. Silence one neuron and see what changes. That’s also what makes it valuable - it is a programmable, testable model of neural circuitry, where a hypothesis becomes a run.

So what are all those demos doing?

Every “fly brain plays X” demo, including mine, is three things stacked:

graph LR
    W["World / game"] -->|"encoder<br/><small>engineered</small>"| I["Input neurons<br/><small>measured identity</small>"]
    I --> B["Connectome + LIF dynamics<br/><small>measured wiring, one calibrated weight</small>"]
    B --> O["Readout neurons<br/><small>measured identity</small>"]
    O -->|"decoder<br/><small>engineered (fixed or trained)</small>"| W

The wiring and the dynamics are shared. The adapters are where the demos differ, and where much of the apparent intelligence lives: the encoder that turns pixels or some other state into firing rates (fe. of visual neurons - which ones, and how many Hz for what?), and the decoder that turns firing rates (of descending neurons - the brain’s “motor command” cells) into button presses or body motion. Look at a few:

Xenova’s Fruit Fly Simulation - the full 166,700-neuron MaleCNS LIF running in the browser on WebGPU, driving a 3D fly body. You paint stimulation on neurons, or use presets: “walk” stimulates LC9 on both sides, “turn left” stimulates LC9 on the left plus DNa02 on the left, “fly” stimulates LC4 on both sides. LC4 and LC9 are visual projection neurons - each LC type is a detector that reports one kind of visual event to the central brain (LC4: “something is expanding toward me”); DNa02 is a descending neuron that experiments show drives turning. On the output side, the demo reads a handful of descending neurons: DNp09/DNg100/DNg97 for walking speed, DNa02/DNa11/DNg13 left vs right for turning, MDN (the “moonwalker” backing-up neuron) for a jump, DNp01 (the Giant Fiber) for escape flight - each with a hand-chosen gain (8 mm/s, 3.8 rad/s…) documented in its model.json, which I appreciate a lot. One thing to be aware of: the “turn” presets poke the output neuron DNa02 directly, not just the visual input. The author says why - the visual input alone activates DNa02 too weakly - but it means the fly would still turn even if you cut every connection inside the brain. Still, this is the tool that tries to show real wiring activity without heavy adapters around it.

Fly Dino - a different approach altogether. Instead of the whole brain, an 80-neuron subcircuit: 32 visual detector neurons (LC4, LC11, LPLC2… the same “something is coming” family) and 16 descending neurons, picked because in the connectome they are the most strongly wired path from “visual event” to “motor command”, plus the cells in between. Run as a graded rate model (tanh), not Shiu’s spiking LIF. Eight game features (obstacle distance, size, speed… not pixels) are injected into the visual neurons. Then the decoder adapter: the activity of the 16 output neurons goes into a tiny neural network - 243 numbers in total - that picks Run, Jump or Duck. That tiny network was trained by trial and error over 15,680 games; the fly circuit itself never changed. So what’s happening here is that we trained output adapter to interpret some of the wiring activity and translate them into game actions. This is reservoir computing: the brain is used as a fixed, complicated transformation, and only the last “which action” mapping is fitted from training. The nice part is the control experiment: switch the fly circuit off but keep the trained adapter, and the score drops from 2,886 to 41. So the circuit is doing something - it acts as a fixed, nonlinear transformation of the inputs that the small network learned to read.

“Fly Mario”-style demos (Fly64, fly-games) - the full MaleCNS LIF stepped at a coarse time-slices (20 ms), game frames turned into features for the visual detectors (LC4: threat, LPLC2: something approaching, LC10a: something to chase), and on the output side is neuron-to-button mapping: a dozen descending neuron groups - DNa02 left/right for steering, DNp09 for walking forward, and so on - each mapped to a button, with a threshold. In Fly64 that mapping is hand-made (“if left DNa02 fires more than right, press Left”); in fly-games it is fitted by a small linear regression from recorded example play. Reservoir computing again. It predicts the right button 90% of the time on recorded episodes it never saw - and the fly still dies early in live play, because live situations drift away from the recordings. Mario walks into a wall and stays there.

None of this is a criticism. Those are all honest experiments. The point is - the adapters matter enormously, and a moving body proves nothing about what the wiring contributed unless you also run the controls - switch off the connections inside the brain and/or switch off the input.

This leads us to dotFly

dotFly is, first of all, a library: a .NET inference engine for fly-connectome spiking models that Godot, console apps, tests and benchmarks link in-process. It builds memory-mapped checkpoints from the MaleCNS and FlyWire releases, runs the Shiu et al. model on them - exactly (the float64 backend matches Brian2 spike-for-spike) and fast (the float32 SIMD backend runs the whole MaleCNS in real time on a laptop) - and exposes stimuli in, spikes and rates out, recording and replay, and the causal switches. The room demo is a demo: the thing I built to see whether a thin embodiment, with the fly’s own senses still needs a lot of “adapter”.

The room demo

A Godot 4 (.NET) 3D scene: a sunlit living room, a fly, sugar on a table, and “smell clouds” drifting downwind from few places. The MaleCNS connectome runs on dotFly as the only thing between her senses and her actions. Here is what every panel means:

Senses → inputs (encoders, fixed) - two cameras, one per compound eye, render 32×32 luminance images. Per eye, the dark fraction and its growth become an LC4 (looming) rate. Horizontal and vertical image shift become T4a-d (optic flow) rates. Odor concentration at the two antenas becomes an ORN_DM1 rate, wind side a JO-C/E (Johnston’s organ) rate, and sugar contact when landed drives the fly’s taste-sensing neurons in its feet and mouth. 7,524 input neurons in total.

The connectome (measured) - all 166,700 neurons, drawn at their real positions and lit by their real spikes. Nothing here is drawn from a script.

Readouts (measured circuits) - every circuit read is one that I found in the checkpoint with the CLI (dotfly inspect --upstream-of, dotfly run --stimulate), not assumed - 38 neurons in 13 groups:

• LC4 → DNp04 L/R, Giant Fiber - escape
• T4a → HS, T4b → H2, T4c → VS - optic flow
• ORN_DM1 → DM1 projection neurons - odor amount
• JO-C/E → AMMC012 - wind direction
• GRN → MN9 - feeding

The decoder (fixed, hand-written) - printed on screen next to the readouts it consumes, and every number in it is a choice:

Interesting finding is that smell direction is simply not derivable from any reading (both DM1 candidate neurons respond alike whichever antenna is stimulated), so I was forced to put some “logic” into adapter that the fly searches by flying forward while the smell gets stronger, then turning around when it fades, with the wind signal coming from the brain.

dotFly under the hood: C# performance

dotFly is the sibling of dotLLM in conventions and spirit: a memory-mapped model file, unmanaged state, SIMD kernels, zero allocations on the hot path. Same layered layout, each project a NuGet package:

src/DotFly.Core      .dfb checkpoint reader (mmap), graph tables, NeuronModel, RNG, IBackend
src/DotFly.Data      MaleCNS / FlyWire release readers
src/DotFly.Cpu       SIMD LIF kernel, compute thread pool
src/DotFly.Engine    Simulation, input/output ports, real-time driver, recorder, adapters
src/DotFly.Adapters.*  ONNX Runtime / ML.NET readout adapters
src/DotFly.Cli       dotfly build | inspect | run | bench | explore

And the API demos program against:

// Neurons are addressed by their exact 64-bit IDs from the connectome release (FlyWire root IDs
// or MaleCNS body IDs)
ulong[] sugarGrnIds = [720575940624963786, 720575940630233916, /* 21 sugar GRNs from Shiu et al. */];
const ulong Mn9 = 720575940660219265;   // MN9, the proboscis motor neuron (FlyWire v630 root ID)

using Brain brain = Brain.Open(".data/shiu2024/flywire-v630.dfb");
using Simulation sim = brain.CreateSimulation(new SimulationOptions { Seed = 1 });

NeuronSet sugar = brain.ByBodyIds("sugar_GRN_R", sugarGrnIds);
sim.Input(sugar, InputKind.PoissonToV).Fill(150f);                   // Poisson drive, 150 Hz each
OutputPort mn9 = sim.Output(brain.ByBodyId(Mn9), OutputKind.Rate(50.Ms()));

sim.Run(1.Seconds());                                                 // neural time
Console.WriteLine($"MN9 {mn9.Snapshot.Mean():F1} Hz at {sim.Clock.RealTimeFactor:F1}× real time");

This is the Shiu et al. sugar experiment in ten lines: drive the 21 sugar-sensing gustatory neurons, read the proboscis motor neuron. The full sample (samples/DotFly.Sample.SugarExperiment) runs the paper’s 30 trials plus its silencing experiments and compares with the published results:

flywire-v630: 127,400 neurons, 14,687,178 edges. Sugar GRNs: 21, MN9 index 127193.
30 trials × 1 s in 3.0 s wall (10.1 simulated s per wall s).

Active neurons (mean over trials): 433. Top 10:
  720575940621502051    152.3 Hz  (sugar GRN)
  720575940632425919    150.7 Hz  (sugar GRN)
  ...
MN9 (720575940660219265): 83.3 Hz mean over 30 trials.

Silencing experiments (outgoing synapses zeroed; the neuron still spikes):
  silence 720575940621502051: MN9 83.3 Hz (intact 83.3 Hz)
  silence 720575940632425919: MN9 78.7 Hz (intact 83.3 Hz)
  silence 720575940633143833: MN9 79.6 Hz (intact 83.3 Hz)

Comparison with published results/example/sugarR.parquet (Brian2 2.5.1, 30 trials):
  active neurons: 448 vs 433 (Jaccard 0.949); rate correlation 0.9946; spike-count ratio 0.803
  MN9: 93.3 Hz vs 83.3 Hz

Thirty one-second trials of a 127,400-neuron brain in three seconds, and the answer the paper got: sugar in, proboscis motor neuron out at ~80 Hz. For populations you normally query by annotation instead - brain.Query(type: "LC4", side: Side.Left), brain.Query(cls: "Kenyon_Cell") - and raw IDs are for reproducing a paper or reading a single named cell. For a game loop, new RealtimeDriver(sim).Start() runs the network on its own threads, the game thread writes InputPorts and reads OutputPort.Snapshot frames lock-free.

Some of the design decisions that made the difference:

• Checkpoint as a memory-mapped file (.dfb, the GGUF of dotFly) - the graph in Compressed Sparse Row (CSR) form - rowPtr[N+1], col[E], w[E] (int16 contact counts, signed) - plus annotations, positions and a header recording the data version, filters and sign policy. Building it from the 1 GB Arrow release takes ~10 s, opening it takes milliseconds
• Structure of Arrays state in unmanaged, 64-byte-aligned memory - v[N], g[N] in float32, a refractory countdown, a ring of 19 spike lists. ~2 MB for the whole MaleCNS - it fits in L2
• The LIF kernel is Vector256<float> - spike indices compacted with TrailingZeroCount. Kernels are generic over TFloat : IBinaryFloatingPointIeee754<TFloat> so double (the reference that matches Brian2 to 1e-11 mV) and float (the fast path) share one implementation
• [MethodImpl(MethodImplOptions.AggressiveOptimization)] on every per-step method - the single biggest surprise. Without it the tiered JIT left the kernel and the batch loop at tier-0/OSR for most of a short run: 21 µs/step instead of 6.

Benchmarks

End to end, dotfly bench on my 8-core laptop (Ryzen 7 5800HS, AVX2):

Python, JavaScript, C#

I couldn’t resist checking how does that compare with what the others run? Three references are easy to pin down. The Python one is Brian2 - the general-purpose spiking simulator the published model runs on. The JavaScript and WebGPU ones are both Xenova’s browser demo: its CPU core, BrainCPU in src/brain.js, is a hundred lines of clean, plain JS implementing the same model - exact LIF integration, dt = 0.1 ms, while BrainGPU runs the same thing as a WGSL compute shader . I ran the JavaScript core in Node 24, and the Xenova’s browser demo in headless Chrome, on the same MaleCNS graph (its own data files, 25.6 M edges) with the same stimulus as dotfly bench, back-to-back on the same laptop. Spike counts agree to within the RNG (534-539 k for the T4a scenario in every implementation), so it is the same work:

Thread for thread, C# is about 11× faster than the JS reference on the same work - not because JavaScript is slow at this (V8 does a fine job on typed arrays), but because of what a JIT cannot give you for free: eight neurons per instruction in Vector256, a state layout that fits in L2, no bounds checks on the scatter, and a delivery loop that touches only its own cache-resident slice of g.

What this table does not include: the demos stepping at dt = 20 ms (Fly64, fly-games). With 200× fewer steps they are cheap by construction, at the cost of no longer being the Shiu model.

Limitations - and where this goes next

Now the honest part. Two big limits of the model itself, which perfectly shows the trade offs between using connectome directcly versus filling the gaps in input/output decoders.

The visual boundary is in the wrong place - today C# computes what the “optic lobe” should compute: the retina encoder turns image shift and expansion directly into T4 and LC4 signals - it skips the first four boxes of the looming diagram shown earlier. A fly’s eye is made of about 800 tiny visual units and a whole optic lobe (a large share of all the neurons in the brain!) whose job is precisely to compute motion and looming from photoreceptor signals. The next milestone is to feed light straight into the photoreceptors and see whether the wiring in between computes motion and looming on its own - if it does, the encoder shrinks to “brightness per eye” and the connectome does the rest; if it doesn’t, the fallback is a hybrid with a Flyvis-style trained optic lobe.

The fly’s own navigation circuit is silent - this is the big one. In the demo, the decision of where to go - surge upwind, tumble, cast, land - is made by my decoder. A real fly makes it in its brain, and that circuit is in the wiring: the neurons that keep track of the fly’s heading, combine it with wind and odor, and drive the steering neuron DNa02 left or right. With wind and odor inputs on, none of it fires. We get a fly that smells and feels the wind, but does not steer. The reason is that this circuit is recurrent - it feeds back on itself. Its activity stays alive only because active neurons keep exciting their neighbours while inhibiting the rest. Current Shiu model is not able to represent this and we need to support more complex “recurrent” neural model.

Closing

Two things are true at once, again. The platform: .NET can run a 166,700-neuron spiking network with 25 M synapses in real time on a laptop, bit-identical across thread counts, matching the scientific reference - Vector256, NativeMemory, memory-mapped files and a hand-rolled thread pool are all you need (🔥). And the science: the connectome is a spectacular measurement, the LIF model is a spectacular starting point, and everything that makes a fly “play a game” today is an adapter that should be shown on screen and switched off in a control run.

dotFly is GPLv3 - the data (MaleCNS, FlyWire) is CC BY 4.0 and downloaded separately. If you are a .NET developer who wants to poke a real brain from C#, or a neuroscientist who wants an inner loop three orders of magnitude faster than the reference, give it a try. Stimulate something. Silence something. Tell me what fires.

• GitHub: github.com/kkokosa/dotFly
• Docs: kkokosa.github.io/dotFly - guides, the room demo, the measured-vs-engineered inventory, findings, CLI and API reference
• References: Berg et al. 2026 (MaleCNS), Shiu et al. 2024, Dorkenwald et al. 2024 (FlyWire), Lappalainen et al. 2024 (Flyvis)

A word on runtimelab first. ZeroGC lives on a feature branch of dotnet/runtimelab, the .NET team’s playground “for experimentation and exploring new ideas that may or may not make it into the main dotnet/runtime repo”. Everything there - including this project - should be treated as unsupported by Microsoft. Nothing in this post implies any kind of production-grade support, and it is not a direction of .NET development. Even if I am part of .NET team, it’s a personal experiment. Treat it as one.

With that said: if you ever want to write a new garbage collector, you have to start from something. And the smallest possible something is a GC that only allocates and never reclaims memory. Meet ZeroGC.

The simplest GC that can exist

ZeroGC is a standalone CoreCLR GC, loaded via DOTNET_GCName, that implements the full IGCHeap / IGCHandleManager ABI - but whose allocator is just a bump pointer over a large reserved arena. It never collects, never compacts, never promotes, never frees. Every object you allocate stays resident for the entire life of the process. It’s the .NET cousin of Java’s Epsilon GC - a “no-op” collector.

Because it does nothing, it’s the perfect baseline: it introduces (almost) zero overhead. There are no pauses to measure, no background threads, no write-barrier bookkeeping to pay for.

The “almost” is a fun sidenote. Even a do-nothing GC has to be fast at doing nothing. For example, ZeroGC configures the ephemeral range as empty (ephemeral_low == ephemeral_high) so the JIT-emitted write barriers become near-free, and it hands out per-thread arena slices with a single interlocked add so allocation stays lock-free under many threads. Skipping work correctly still takes engineering.

Of course, there’s this “little” caveat:

Memory is never reclaimed. Long-running or allocation-heavy processes will grow endlessly until the process is killed or the machine runs out of memory. This is a feature of this GC, not a bug.

Whether that’s a dealbreaker or a non-issue depends entirely on your app - and that’s precisely what makes it interesting to measure.

A return of Zero/Upsilon GC

This isn’t a new idea for me. Years ago I built UpsilonGC, exploring the loosened GC ↔ Execution Engine coupling that .NET Core 2.0 introduced - a “Zero GC” that only allocates, and an “Upsilon GC” that actually reclaims. That work produced a couple of blog posts and a talk:

• Zero Garbage Collector for .NET Core
• Zero Garbage Collector for .NET Core 2.1 and ASP.NET Core 2.1
• Make your custom .NET GC - “whys” and “hows” (DotNext Moscow 2018)

All of that is now comfortably outdated - the GC-EE interface has moved on - so ZeroGC is the modern rewrite against today’s CoreCLR.

Why do this at all?

Three reasons:

1. A code baseline - if you want to write a serious collector, you need somewhere to start, and “empty project” is a miserable place to start from. ZeroGC is a working codebase to grow from: the full IGCHeap / IGCHandleManager surface is already implemented and already loads into a real runtime, so the very first thing you do is modify a GC that works rather than debug one that doesn’t yet. It doubles as living documentation of the GC-EE interface - every method you have to implement, with the minimum viable body already written.
2. A performance baseline - this is what a workload looks like with GC overhead at (nearly) zero: no pauses, no background threads, no bookkeeping. That gives you a number to compare a real collector against. But read that number carefully. “GC overhead removed” is not the same as “runtime made as fast as possible”, and it would be wrong to conclude that any app with a GC plugged in will be always slower. A real collector does work that earns its keep:
   • compaction and moving objects around opens possibility to get better cache and TLB behaviour
   • reusing memory keeps the hot working set small enough to actually stay in cache
   • generational collection means fresh allocations land in memory you just touched
3. For the curious (and slightly crazy) - you can drop it into your own app and watch what happens. It’s the cleanest way to answer “how much of my app’s profile is actually GC-bound?”

What running it actually looks like

The repo ships a benchmark report comparing ZeroGC against Workstation and Server GC across eleven workloads, using dotnet-counters to capture GC pause time, memory and throughput second by second over ten-minute runs. The harness and its sample apps live in src/ZeroGC/.

Two charts tell you almost everything about a workload: how long the collector stopped you, and how much memory you were holding while it did. Here are three workloads that land in three completely different places.

The GC was never in your way

An ASP.NET Core minimal API, driven by eight concurrent workers for ten minutes.

Look at the top panel: no pause, under any collector, ever reaches 5 ms - the worst of the entire ten-minute run is 4.6 ms. Add every single pause together and Workstation GC stopped this app for 0.14 seconds in total, Server GC for 0.57. Out of six hundred. There is no big latency problem here, so there is not much for ZeroGC to win back - unless you really want sub-millisecond (zero!) pauses. Throughput is identical to three significant figures (507.3 → 507.5 ops/s).

Now look at the bottom panel, and notice that the two real collectors are the flat line along the bottom - they hold steady around 85 MB for the whole run, because that is what a collector does. ZeroGC’s straight line is every request the API ever served, still resident. It ends the run at 2.7 GB and rising, and it would keep rising until the process died. That is 31× the memory in exchange for a latency win of exactly zero.

Real collections, but nobody notices

GCPerfSim’s cache / large-object-heavy workload - a growing pool of long-lived objects, many of them on the LOH.

This one really does collect, and the two collectors disagree about how. Workstation GC produces the dense blue sawtooth - a collection every second or so, none of them costing more than 4.7 ms all run. Server GC is the opposite shape: long quiet stretches punctuated by gen2 spikes, the worst reaching 21 ms.

But check the axis - the whole panel is 25 ms tall. Totalled over ten minutes, Workstation GC spent 0.71 s collecting and Server GC 0.18 s. Collections everywhere, latency nowhere. Note also that “fewer pauses” and “less time paused” are not the same property: Server GC pauses four times less in total, yet owns every one of the tall spikes.

The bottom panel is where the two real collectors earn their keep. Both hold the working set near 1 GB, roughly flat, while ZeroGC climbs steadily to 5.5 GB - about 6× more. The genuinely-live data is much the same in all three runs; the entire difference is garbage that ZeroGC is contractually unable to release.

The one where it actually pays off

A console app with a growing in-memory cache under a real request workload - the case that naturally escalates into expensive gen2 collections.

Now we see something. Server GC’s gen2 collections arrive on a visible frequency, and they get worse as the cache grows: about 270 ms early in the run, 493 ms by the end. Workstation GC trades those for constant small pauses plus its own 117 ms spike near the end. Totalled up, both real collectors stop the app for roughly 8.2-8.6 seconds across the run. ZeroGC is the flat green line along the bottom: not “fast”, just absent.

And the bottom panel is why this is the one workload where the bargain is worth taking. Because the app retains nearly everything it allocates, all three lines climb together. Never freeing costs about 16% more memory (6.6 GB → 7.7 GB) - and buys away every pause in the panel above. Compare that against 31× for the API that had nothing to gain in the first place.

Try it - for fun

If your app is time-bounded rather than long-running - a CLI tool, a batch job, a benchmark, a short-lived worker, an inference run that fits in RAM - ZeroGC might genuinely finish faster and quieter. You don’t have to build anything: prebuilt, signed binaries are published as NuGet packages, built by Microsoft’s official Arcade/Azure Pipelines infrastructure and pushed to the public dotnet-experimental feed.

1. Get the binary. There is one package per targeted runtime major version - Microsoft.DotNet.RuntimeLab.ZeroGC.Net10 for net10.0 apps (built against v10.0.0 GA), Microsoft.DotNet.RuntimeLab.ZeroGC.Net11 for net11.0 (tracking the previews), x64 Windows and Linux. Add the feed to a nuget.config next to your project:

<configuration>
  <packageSources>
    <add key="dotnet-experimental"
         value="https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-experimental/nuget/v3/index.json" />
  </packageSources>
</configuration>

dotnet add package Microsoft.DotNet.RuntimeLab.ZeroGC.Net10 --version 1.0.0-zerogc.26381.4

(Use whatever prerelease version is currently published - --version and --prerelease can’t be combined in one command.)

2. Put it next to your app. The package carries ZeroGC.dll / libZeroGC.so as a RID-specific native asset, so a plain dotnet build won’t flatten it next to your executable. Either copy the one file out of the NuGet cache into your build output by hand (fine for a quick experiment, easy to automate with a post-build <Copy> target)…

cp ~/.nuget/packages/microsoft.dotnet.runtimelab.zerogc.net10/1.0.0-zerogc.26381.4/runtimes/linux-x64/native/libZeroGC.so \
   bin/Release/net10.0/libZeroGC.so

…or do a RID-specific publish, which picks it up automatically:

dotnet publish -c Release -r win-x64 --self-contained false

3. Turn it on. Same knob as any other standalone GC - just the file name:

DOTNET_GCName=libZeroGC.so dotnet YourApp.dll   # Linux

$env:DOTNET_GCName = "ZeroGC.dll"; dotnet YourApp.dll   # Windows

Unset it to go back to the normal GC. If you’d rather bake it into the app than set an environment variable, the equivalent runtimeconfig.json property (configProperties."System.GC.Name") and AppContext switch work too.

4. Check it’s actually live. ZeroGC reports itself through GC.GetConfigurationValues(), but the cheapest sanity check is the behaviour itself: GC.CollectionCount(0/1/2) should stay pinned at 0 forever, and the working set should climb monotonically.

Which package should I take? Match your app’s target framework - .Net10 for net10.0, .Net11 for net11.0. That is the sanity default and what the docs recommend, though it is probably not load-bearing; what a standalone GC is actually tied to is covered below.

Run it, watch the memory graph climb, and see whether your workload is one of the “GC was invisible anyway” cases or one of the “pauses gone, memory paid” cases. Then share what you find - I’d love to see the graphs.

Building from sources

If you’re on a restricted network, need a runtime version that isn’t published, or want to hack on ZeroGC itself, building from source takes a couple of minutes. Both scripts want a local dotnet/runtime checkout, used read-only as a header and reference dependency - nothing under it is built or modified - and both default that path to my machine, so pass your own:

.\build.ps1 -RuntimeRepo C:\src\runtime          # needs the VC++ build tools

./build-linux.sh --runtime-repo ~/src/runtime    # needs clang++ with C++17

Checking out the dotnet/runtime tag matching your app’s target runtime is the tidy thing to do; any reasonably recent tag will build a working binary, for reasons covered under versioning below. Full details: docs/zerogc/using-prebuilt-binaries.md and src/ZeroGC/README.md.

Versioning: what a standalone GC is tied to

A standalone GC is loaded across a versioned ABI, described by two numbers in gcinterface.h:

#define GC_INTERFACE_MAJOR_VERSION 5
#define GC_INTERFACE_MINOR_VERSION 8

Those numbers cover more than the C++ methods. They version the whole GC/EE contract: which methods exist on IGCHeap and IGCToCLR and what they mean, and also the behaviours and data contracts that never appear as a method signature at all - including the object-model layout the GC reads directly.

There is a second number for the other direction, EE_INTERFACE_MAJOR_VERSION (currently 4), describing what the runtime side offers. The two are exchanged at load time, which is what lets each side adapt to the other.

The loader in gcheaputilities.cpp applies a single rule:

if (g_gc_version_info.MajorVersion < GC_INTERFACE_MAJOR_VERSION)
{
    LOG((LF_GC, LL_FATALERROR, "Loaded GC has incompatible major version number ..."));
    return E_FAIL;
}

The comparison is one-directional. A GC reporting an older major version is rejected; one reporting the same or a newer version is accepted. That is what permits a GC built against a newer runtime to load into an older one. A lower minor version is not rejected at all - it logs at LL_INFO100 and continues.

The major number moves rarely: it has been version 5 since January 2023 (dotnet/runtime#81188, “Fix GC interfaces versioning”), spanning .NET 8, 9, 10 and 11-preview. Keeping a newer GC loadable on older runtimes is an explicit goal on the runtime side - dotnet/runtime#88457, “Ensure GCHeap related debugging still works even when we use newer CLRGC to target older runtimes”, is maintenance for exactly that scenario. When the major version does change, the outcome is a refusal to load with E_FAIL and a logged reason.

The object layout is part of that contract

A standalone GC does not only call the interface. It also compiles against the gcenv.*.h shim headers, which mirror the VM’s private MethodTable and ObjHeader structures field for field. A GC reads object headers directly - it has to, and it cannot avoid the dependency in any case: Object, MethodTable and gc_alloc_context appear throughout the IGCHeap and IGCHandleManager signatures every standalone GC must implement.

That layout is versioned by the same numbers. The MethodTable data a collector needs to walk the object graph is a binary contract kept that way for performance, and changing it incompatibly requires a major version bump even though no C++ signature changes. dotnet/runtime#91821 (“Converge Representations between NativeAOT and CoreCLR”, September 2023) is the worked example: it moved MTFlag_Collectible from 0x10000000 to 0x00200000 and bumped EE_INTERFACE_MAJOR_VERSION from 1 to 2 in the same change, touching no interface method.

The compatibility code handles that signal, and compiles only for standalone GCs:

bool Collectible()
{
#ifdef BUILD_AS_STANDALONE
    if (g_oldMethodTableFlags)
    {
        // This flag is used for .NET 8 or below
        const int Old_MTFlag_Collectible = 0x10000000;
        return (m_flags & Old_MTFlag_Collectible) != 0;
    }
#endif
    return (m_flags & MTFlag_Collectible) != 0;
}

A GC opts into it during the version handshake. The runtime passes its own version in, the GC records it and writes its own version back - ZeroGC’s GC_VersionInfo is the whole mechanism in eight lines:

ZEROGC_EXPORT void GC_VersionInfo(VersionInfo* info)
{
    // On entry, `info` carries the interface version the runtime supports;
    // remember it so we know which optional IGCToCLR members are safe to call.
    g_runtimeSupportedVersion = *info;
    g_oldMethodTableFlags = g_runtimeSupportedVersion.MajorVersion < 2;

    info->MajorVersion = GC_INTERFACE_MAJOR_VERSION;
    info->MinorVersion = GC_INTERFACE_MINOR_VERSION;
    info->BuildVersion = 0;
    info->Name = "ZeroGC";
}

What all this means in practice

• The net10.0 build would run on .NET 11-preview: the GC interface major version has not moved
• Older runtimes (.NET 9, .NET 8) are plausible for the same reason, and where the object model did change, the version handshake is what lets a single binary cope.
• If the major version is bumped, the failure is a refusal at startup with a logged reason, not corruption

For anyone writing their own collector, the practical shape of this is: the version numbers cover more than the vtable, so read them as the contract for the object model too, and use the handshake to branch when you must. Depending on MethodTable is not something to design around - every standalone GC does it, including the one in the box.

If you’ve been building .NET applications and wanted to run LLMs locally, your options have been… limited. You could wrap llama.cpp through LLamaSharp, deal with ONNX Runtime, or orchestrate calls to external Python services. None of these are fully satisfying if you want to run LLM inference purely in .NET.

So I built my own.

Over the past two months I’ve been building dotLLM - a ground-up, high-performance LLM inference engine written natively in C#/.NET 10. Not a wrapper. Not bindings. A full implementation: GGUF model loading, tokenization, attention, sampling, SIMD-optimized CPU inference, CUDA GPU acceleration, an OpenAI-compatible API server, and a built-in chat UI. This week I’ve published the first preview release (v0.1.0-preview.2).

This post covers three things: what dotLLM is, why I built it, and how AI-assisted development - Claude Code, Codex, and Gemini working together in a structured workflow - made it possible for a single developer to ship this in about two months.

What is dotLLM?

The short version

dotLLM is a native C#/.NET 10 LLM inference engine. It runs transformer-based models - Llama, Mistral, Phi, Qwen (and more in future) - from GGUF files, with SIMD-optimized CPU, CUDA GPU backend, or hybrid/offloading mode. It exposes an OpenAI-compatible API, ships as a CLI tool and as NuGet packages you can embed in your own applications.

The key word is native. All orchestration, model loading, tokenization, sampling, scheduling, and CPU compute is implemented in pure C#. The only native code is a thin CUDA library for GPU kernels, loaded via PTX through the CUDA Driver API (P/Invoked).

Architecture

dotLLM is organized as a layered architecture where each layer depends only on the layers below:

graph TD
    subgraph "User-Facing"
        CLI["DotLLM.Cli<br/><small>CLI tool</small>"]
        ChatUI["Built-in Chat UI<br/><small>Browser-based</small>"]
        Server["DotLLM.Server<br/><small>OpenAI-compatible API</small>"]
    end

    subgraph "Engine"
        Eng["DotLLM.Engine<br/><small>KV-cache, scheduler, samplers,<br/>constraints, speculative decoding</small>"]
    end

    subgraph "Model Layer"
        Models["DotLLM.Models<br/><small>GGUF loader, Llama, Mistral,<br/>Phi, Qwen, DeepSeek</small>"]
        Tokenizers["DotLLM.Tokenizers<br/><small>BPE, SentencePiece,<br/>Jinja2 chat templates</small>"]
    end

    subgraph "Compute"
        CPU["DotLLM.Cpu<br/><small>SIMD / AVX2 / AVX-512</small>"]
        CUDA["DotLLM.Cuda<br/><small>PTX kernels, cuBLAS</small>"]
    end

    Core["DotLLM.Core<br/><small>ITensor, IBackend, IModel,<br/>ISamplerStep, IDecodingConstraint</small>"]

    CLI --> Server
    ChatUI --> Server
    Server --> Eng
    CLI --> Eng
    Eng --> Models
    Eng --> Tokenizers
    Models --> CPU
    Models --> CUDA
    CPU --> Core
    CUDA --> Core
    Models --> Core
    Tokenizers --> Core
    Eng --> Core

Each project ships as a separate NuGet package. DotLLM.Core defines all abstractions (ITensor, IBackend, IModel, ISamplerStep, etc.) while concrete implementations live in their respective projects. You pull in only what you need.

Key features

Performance:

• Zero-alloc inference - all tensor data uses NativeMemory.AlignedAlloc (64-byte aligned). No managed heap allocations on the hot path (well, “the best” effort so far). (Almost) no allocs, no GC triggered.
• SIMD vectorization for CPU backend - TensorPrimitives for standard operations, hand-tuned System.Runtime.Intrinsics for quantized matmul, RMSNorm, RoPE, softmax operations. AVX2 and AVX-512 with scalar fallbacks.
• CUDA GPU backend - PTX kernels loaded via the CUDA Driver API with cuBLAS HGEMM for prefill, custom quantized GEMV for decode, and FP16 activation pipeline. Supports full GPU inference, hybrid CPU/GPU layer offloading, and KV-cache quantization.
• Memory-mapped model loading - GGUF files loaded via MemoryMappedFile. OS demand-paging means multi-GB models load in milliseconds.
• Quantized inference - FP16, Q8_0, Q4_K_M and other GGUF formats with fused scale-int dot-product kernels operating directly on quantized blocks.

Serving:

• OpenAI-compatible API - /v1/chat/completions, /v1/completions, tool calling, structured output, streaming SSE via ASP.NET.
• Speculative decoding - draft-verify-accept loop with KV-cache rollback for higher throughput.
• Structured output - FSM/PDA-based constrained decoding guaranteeing valid JSON, JSON Schema, regex, and GBNF grammar.

Extensibility:

• Pluggable backends - IBackend interface with separate packages (CPU, CUDA, future ROCm).
• (planned) LoRA adapters - runtime loading, no weight merging, concurrent multi-adapter serving.
• (planned) Diagnostic hooks - zero-cost IInferenceHook for activation capture, logit lens, SAE integration.
• (planned) OpenTelemetry - System.Diagnostics.Metrics + Activity for throughput, latency, and per-request tracing.

Here’s a minimal streaming generation example:

using var gguf = GgufFile.Open(modelPath);
var config = GgufModelConfigExtractor.Extract(gguf.Metadata);
using var model = TransformerModel.LoadFromGguf(gguf, config);
var tokenizer = GgufBpeTokenizerFactory.Load(gguf.Metadata);

var generator = new TextGenerator(model, tokenizer);
var options = new InferenceOptions
{
    SamplerSteps =
    [
        new TemperatureSampler(0.8f),
        new TopKSampler(40),
        new TopPSampler(0.95f)
    ],
    StopConditions = [new EosStopCondition(tokenizer.EosTokenId)],
    MaxTokens = 128
};

await foreach (var token in generator.GenerateStreamingTokensAsync(prompt, options))
    Console.Write(token.Text);

And here’s what the CLI looks like for a quick generation run with SmolLM-135M:

-- dotllm | Llama 30L/576H | Q8_0 | 16 threads | greedy ──────────────────
The capital of France is Paris. Paris is a city of romance and culture,

╭──────────────────────────────────────────────────────────────────────────╮
│  Generation Complete                                      163.27 tok/s  │
│                                                                         │
│  Prefill            12.3 ms       6 tokens       487.80 tok/s           │
│  Decode             91.8 ms      15 tokens       163.40 tok/s           │
│  Sampling            0.1 ms      15 tokens                              │
│  Total             104.2 ms      21 tokens       201.54 tok/s           │
│  Load              456.7 ms                                             │
│                                                                         │
│  Weights         136.73 MiB      (memory-mapped)                        │
│  KV Cache        158.20 MiB      (192 slots)                            │
╰──────────────────────────────────────────────────────────────────────────╯

There is also build-in serve command to host simple chat UX for testing and research:

Performance reality check

Let’s be honest about where dotLLM stands today. While performance has been a key design consideration from day one (zero-alloc hot paths, SIMD kernels, memory-mapped loading), the primary focus so far has been closing the feature set — getting the full inference pipeline, constrained decoding, tool calling, speculative decoding, and the API server working correctly. Dedicated performance polishing is coming. Here are CPU benchmarks against llama.cpp (AMD Ryzen 9 5950X, 16 threads, same models and quantizations):

Decode throughput (tok/s) - the metric that matters most for interactive chat:

On decode, dotLLM reaches 66-88% of llama.cpp throughput. Decode is largely memory-bandwidth-bound, and C# with SIMD intrinsics can get reasonably close to saturating the memory bus.

Prefill is a different story - dotLLM is roughly 2-5x slower than llama.cpp across the board. Prefill is compute-bound, and llama.cpp has years of hand-tuned GEMM kernels. We hit a specific wall here (outer-product tiled matmul vs. RyuJIT register pressure) which I’ll describe in the lessons learned section below.

The CUDA backend is functional but still early - it currently underperforms CPU on small models due to launch overhead, and the kernel tuning work is ongoing.

This is a preview release. If you need maximum throughput today, llama.cpp is faster. The gap will narrow.

Why build this?

If out-of-the-box performance is not easily achieved, why build this at all? A few reasons, starting with the most important:

Understanding by building. I’ve been writing about LLM internals on this blog - temperature and sampling, logprobs, learning a lot on my own. At some point, it makes you want to implement the whole pipeline. Building dotLLM was the logical next step to accelerate learning: implementing the entire inference pipeline from GGUF parsing through attention to token generation, seeing every allocation and every SIMD instruction up close. And it will generate A LOT of other blog posts…😇

Seeing how far AI-assisted development can go. This was also, explicitly, an experiment. Not vibe coding - not “prompt and pray” loop. Structured, documented, reviewed AI-assisted development where a human makes the architectural decisions and AI handles implementation within well-defined boundaries. I wanted to find out what a solo developer can realistically build in one-two months with this approach. The answer surprised me.

Creating a platform for research and experimentation. Building from the ground up with this goal in mind means the architecture is open to experimentation - adding new features, deeper diagnostics, and interpretability tools into the inference pipeline. All outside the gold-standard HuggingFace monopoly. In .NET 😍

Proving the platform. There’s a persistent assumption that systems-level performance work requires C, C++, or Rust. Twenty years of .NET performance work has taught me that’s not always true. C# with NativeMemory, System.Runtime.Intrinsics, MemoryMappedFile, and Span<T> gives you genuine control over memory and compute. dotLLM is a proof point.

The .NET ecosystem gap is real. If you’re building .NET applications and want local LLM inference, your choices are wrappers (LLamaSharp wrapping llama.cpp), limited runtimes (ONNX Runtime with restricted model support), or orchestration layers (Semantic Kernel, which is about chaining calls, not running inference). Enterprise .NET shops that want to run models in production without Python or C++ dependencies have no native option. dotLLM fills that gap, but it will take a LOT of time to treat it as a serious replacement.

dotLLM is not meant to replace llama.cpp or vLLM in production - at least not yet. It’s built for .NET developers who want native inference without leaving their ecosystem, and for researchers and experimenters who want to explore LLM internals from C#. And everyone who wants to understand how to build a LLM inference engine from scratch.

How AI built an AI engine

Let me be upfront: nearly every commit in dotLLM’s git history has a Co-Authored-By: Claude Opus 4.6 line. This project would not exist in its current form without AI assistance. But the story of how that assistance was structured is more interesting than the fact that it was used.

The development methodology

dotLLM was built over ~60 implementation steps organized into 7 phases, documented in a detailed ROADMAP.md. Each phase was a discrete unit of work with clear scope and acceptance criteria:

• Phase 1 - End-to-end single-token generation (GGUF loader, dequantization, CPU ops, tokenizer, attention, KV-cache, sampling)
• Phase 2 - Practical local inference (Q4_K_M, chat templates, streaming, multi-threading, additional architectures)
• Phase 3 - CPU performance (tiled attention, SIMD tuning, NUMA awareness, operator fusion)
• Phase 4 - GPU acceleration (CUDA backend, hybrid CPU/GPU, KV-cache quantization)
• Phase 5 - Constrained decoding and API (JSON/schema/regex/grammar, tool calling, server, chat UI, prompt caching)
• Phase 6 - Improved serving (warm-up, Native AOT, paged KV-cache, speculative decoding)
• Phase 7 - Diagnostics and interpretability (logprobs, hooks, logit lens - in progress)

Every step started as a GitHub issue. Every issue lived on a branch named issue/{number}-{short-description}. Every PR closed its issue and updated the roadmap. This was relentlessly boring discipline, and it was the single most important factor in the project’s success.

After the initial release, just before making the repository public, I also ran a series of “Waves” - systematic quality passes across the entire codebase:

• Wave 1 (P0): Security and crash fixes - path traversal, CUDA shared memory guards, hybrid GPU edge cases
• Wave 2 (P1): Quick correctness and consistency fixes
• Wave 3: Presentation cleanup - remove dead code, label stubs, fix samples
• Wave 4: Server hardening - request validation, LINQ removal from hot paths
• Wave 5 was skipped - it was earmarked for batch-serving improvements that depend on Phase 9 (continuous batching), which isn’t implemented yet
• Wave 6: CUDA kernel rewrite - tiled softmax, vectorization, grid-stride loops
• Wave 7: CPU performance - TopK sampler optimization, AVX2 gap filling, schema cache tuning

Those “waves” are a bunch of findings that come from in-depth reviews from other models, grouped into GitHub issues.

ROADMAP.md and CLAUDE.md - the highest-ROI investments

If there’s one takeaway from this experiment, it’s this: the time you spend writing structured documentation for AI is not overhead - it IS the development methodology.

ROADMAP.md was the backbone. Each of the ~60 steps had a feature name, a description, key files to modify, and dependencies on other steps. This gave both me and the AI a shared understanding of what to build next, in what order, and why. Without it, AI would be coding in circles - solving the wrong problem, building features in the wrong order, missing dependencies.

The roadmap also forced me to think through the architecture upfront. When you have to discuss things like “Step 31: CUDA backend - PTX kernels loaded via CUDA Driver API, no native shared library, cuBLAS HGEMM for prefill, custom quantized GEMV for decode” before writing any code, you’ve already made the hard decisions. And learnt a lot.

CLAUDE.md was the project’s “constitution” - 180+ lines defining how AI should work in this codebase. Here are some actual rules from it:

**Native .NET first** - All orchestration, model loading, tokenization, sampling, scheduling, CPU compute in pure C#.

**Unmanaged memory for tensors** - `NativeMemory.AlignedAlloc` (64-byte). Zero GC allocations on inference hot path.

**Hybrid GPU architecture** - Thin native C/CUDA lib via `[LibraryImport]`. GPU memory as opaque `IntPtr` - tensor data never crosses P/Invoke boundary.

And specific coding rules:

**NEVER** allocate managed arrays for tensor data. Use `NativeMemory.AlignedAlloc` (64-byte for AVX-512, 32-byte for AVX2).

SIMD: Foundation is `System.Numerics.Tensors.TensorPrimitives` for standard ops. Hot inner loops: `System.Runtime.Intrinsics` - prefer cross-platform `Vector128<T>`/`Vector256<T>`, use platform-specific only when measurably faster. **Always provide scalar fallback.

Beyond CLAUDE.md, there were 22 detailed design documents in /docs/ - one for each major subsystem like ARCHITECTURE, QUANTIZATION, ATTENTION, CUDA, and more. The rule was simple: AI reads the relevant spec before touching a module.

This documentation-first approach had a compound effect. Every implementation step could reference the roadmap for scope, the design docs for architecture, and CLAUDE.md for coding conventions. The AI wasn’t guessing - it was implementing within well-defined constraints.

Claude Code as implementation partner

Claude Code with Opus 4.6 (1M context) was the primary implementation tool from the project start. The workflow was built around six custom Claude Code skills that automated the development lifecycle:

• /plan-step - looks for a given roadmap step in ROADMAP.md plus relevant docs from /docs/, enters plan mode, and produces a step-by-step implementation plan for my approval before any code is written.
• /create-pr - commits remaining changes, pushes the branch, and creates a PR with a detailed description following project conventions.
• /apply-pr-comments - reads review comments from Codex, Gemini, or human reviewers, analyzes them, enters plan mode so I can approve the fixes before any code changes.
• /finish-pr-comments - after fixes are applied and tested, commits the changes, pushes to the PR branch, and replies to each reviewer comment with what was fixed and the commit hash.
• /merge-pr - squash-merges the PR into main, deletes the remote branch, and switches to an updated local main.
• /plan-issue - similar to plan-step but for less frequent case when we start from an issue, not a roadmap’s step.

There was also a GitHub Actions workflow that responds to @claude mentions in PRs and issues, enabling asynchronous interaction.

The typical flow for a feature looked like this: I’d run /plan-step (or /plan-issue), review and adjust the plan, then let Claude implement it step by step while I reviewed each change. The key insight is that planning was always a separate, human-approved step before implementation began. Most of the work and brainstorming happened there.

Codex and Gemini as PR reviewers

Every PR was also reviewed by Codex and Gemini triggered manually via mentions of @codex and @gemini in PR comments.

Gemini was powered by a custom Python bot (.github/scripts/gemini_bot.py) with retry logic and configurable thinking budgets. A separate GEMINI.md file defined its review persona.

The review findings were not cosmetic. They caught genuinely critical bugs that could have shipped. Just as examples:

• KV-cache quantization (PR #75): Codex caught a ring-buffer indexing bug (window reads used linear indexing instead of ring indices, producing garbage after wrap-around) or a pinned buffer scope issue (pointers from fixed blocks used after the scope exited), and a shared-state race condition (per-layer eviction progress stored in a shared counter)
• JSON Schema constrained decoding (PR #79): Found a cache key collision (string substates not included in the hash, collapsing distinct parser states) and a unicode escape flag preservation bug (\u parsing wiped the key-string flag)
• Wave 6 CUDA kernel rewrite (PR #114): Gemini identified thread underutilization in GEMV kernels and uncoalesced memory reads - architectural issues that Codex’s code-level analysis didn’t surface

Once Codex and Gemini leave their comments, the remaining skills close the loop. /apply-pr-comments reads all review comments on the current PR, analyzes them, and enters plan mode — so I can approve which fixes to make before any code changes. This prevents blindly applying every suggestion without human judgment on what’s worth addressing versus deferring.

After the fixes are implemented and tested, straightforward /finish-pr-comments commits the changes, pushes to the PR branch, and has the Claude Code bot (dotllm-claude-code-bot) reply to each reviewer comment with what was fixed and the corresponding commit hash.

This creates a fully traceable chain: Codex/Gemini finds a bug -> Claude fixes it -> the reply references the exact commit. The PR thread becomes a complete audit trail.

Finally, /merge-pr squash-merges the PR into main, deletes the remote branch, and checks out an updated local main. The entire cycle - from roadmap step to merged PR - typically took one Claude Code session.

Lessons learned

What worked brilliantly:

• The implementation is correct. This is the most obvious finding to overlook. But here it is - this setup really worked! Step by step, we were progressing with the implementation, making it more and more sophisticated. And the moment when for the first time I saw an answer from the real model - it felt like magic! Reaching 80-90% of llama.cpp CPU decode performance is a thing, too.
• ROADMAP.md was the highest-ROI time investment. Planning structured documentation saved many hours of correcting misdirected AI implementation. The documentation is the development methodology - it’s not overhead, it’s the thing that makes AI-assisted development work at all. Without the roadmap, the AI has no sense of direction.
• Two-role separation: implementation vs. review. Claude wrote the code. Codex and Gemini reviewed it independently. The AI that writes the code should not be the only AI that reviews it. Different models have genuinely different blind spots - Codex and Gemini almost never flagged the same issues.
• No Ralph Wiggum loop or other YOLO mode. I really wanted to drive the work, not fire & forget it to burn tokens. It worked out perfectly for me. The single task took around several minutes, up to 30-40, so you could return to them from time to time, during breaks whilst waiting for something else. Great background work.

Where AI struggled:

• Fundamental architectural decisions. Which attention strategy to use, how to structure the GPU interop layer, whether to use PagedAttention kernels or staging-buffer gather - these required human judgment informed by comparing to llama.cpp, vLLM, and the research literature. AI can implement an architecture, but it’s less reliable at choosing between architectures.
• The prefill throughput gap. This is the biggest remaining performance wall - but to be fair, it’s not really an AI failure per se. It might be AI struggling to workaround platform’s out-of-the-box behaviours. The 2-5x prefill gap you saw in the benchmarks comes down to one unfinished roadmap step: Step 26, the outer-product tiled matmul. The current batched GEMM (Step 11) uses an inner-product formulation - it tiles over M but still computes one output element at a time. It delivered a 2.2x prefill speedup when implemented. But the “outer-product” approach would share one activation load across all weight-row dot products: a 4x3 AVX2 tile with 12 YMM accumulators. Problem: due to AI that tile needs ~23 YMM registers, but AVX2 only has 16. RyuJIT spills to stack, and the spills eat the reuse benefit the tile was designed to get. AI was actually very good at experimenting here — it iterated through multiple tile sizes, inspected the JIT disassembly for spills, and methodically narrowed down the root cause. The options now seems to be: AVX-512 (32 ZMM registers, fits comfortably), a native C microkernel via [LibraryImport] (what llamafile/tinyBLAS does), or a smaller tile with partial reuse. I’ve left this for a much more manual, still AI-assisted, research phase - it needs careful benchmarking and disassembly analysis rather than the “plan step, implement, make PR” cadence that worked for the rest of the project.
• Keeping it within the guiderails. Small but very annoying. No matter how many times, and in how many places, I ask him to avoid compound tool calls (esp. cd D:\github\ai\dotLLM && git ...) it eventually ignores it, breaking the permissions. It breaks the flow, as instead of an autonomous work it got stuck waiting for permission for tools that otherwise should be accepted (like allowed GitHub read-only commands etc). Now I think I need pre-tool hook that checks that💡
• Claude Code got really stuck 2-3 times. There were a few times when implementing a step took a lot of struggle. Implementing, retrying, failing, in a loop that took literally hours (!). But even more surprising is the fact that he eventually did it. With some of my help and brainstorming, but it was mostly his async work trying to figure it out.

What’s next

dotLLM is at v0.1.0-preview.2 - explicitly a preview. The foundations are solid, but there’s a long road ahead:

• Phase 7 (in progress): Diagnostic hooks, logit lens, Sparse Autoencoders (SAE) integration, LoRA adapters
• Phase 8 (planned): MLA attention (DeepSeek), SmolLM3, Gemma 4, Mixture of Experts
• Phase 9 (planned): Production serving - continuous batching, prefix sharing, advanced scheduling. This will be fun🔥

After those Phases I will consider feature set (mostly) done and start to focus on performance more and more.

BTW, the project is GPLv3 and contributions are welcome! There is still a lot to do. The codebase has 22 design docs, a detailed roadmap, and a CLAUDE.md that makes it easy for both human and AI contributors to get oriented quickly.

• GitHub: github.com/kkokosa/dotLLM
• Website: dotllm.dev
• NuGet packages: DotLLM.Engine, DotLLM.Cpu, DotLLM.Cuda, DotLLM.Server, and more
• Discussions: GitHub Discussions

Closing

Two things are true at once. First: .NET can do native, systems-level AI work. Zero-GC inference, SIMD-vectorized kernels, memory-mapped model loading, paged KV-cache, speculative decoding - all in C#. The platform is more capable than many give it credit for.

Second: a solo developer can build something of this scope in two months with AI assistance - but only with relentless structure. The roadmap, the design docs, the CLAUDE.md constitution, the dual-review workflow - take any of these away and the productivity collapses. AI amplifies discipline; it doesn’t replace it.

If you’re a .NET developer curious about LLM inference, or a researcher who wants to explore model internals from C#, give dotLLM a try. File issues. Break things. Tell me what’s missing.

But looking at raw numbers and graphs is boring. Let’s see how we can use this data to visualize the model’s “confidence” and at least try to spot places when it is “unsure”.

Getting Logprobs from API

If you are using OpenAI API we need to set two parameters: logprobs to true and top_logprobs to some integer $k$ (e.g. 2 or 5) to see top-$k$ alternatives. Information will be provided for all tokens from the response, as they were seen at the time of generation of each token.

Here is what a raw sample HTTP request could look for a question “Who is the President of Poland?”:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Who is the president of Poland?"
      }
    ],
    "logprobs": true,
    "top_logprobs": 2
  }'

And here is a relevant snippet of the JSON response we get back. Notice the logprobs field inside choices:

{
  "id": "chatcmpl-123...",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-4o-2024-08-06",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Andrzej Duda",
        "refusal": null,
        "annotations": []        
      },
      "logprobs": {
        "content": [
          {
            "token": "And",
            "logprob": -0.000035,
            "top_logprobs": [ 
              { "token": "And", "logprob": -0.000035 },
              { "token": "The","logprob": -11.4 }
            ]
          },
          {
            "token": "rzej",
            "logprob": -0.00021,
            "top_logprobs": [
              { "token": "rzej", "logprob": -0.00021 },
              { "token": "rew",  "logprob": -9.1 }
            ]
          },
          {
            "token": " D", "logprob": -0.004,
            "top_logprobs": [
              { "token": " D", "logprob": -0.004 },
              { "token": " Du", "logprob": -6.2 }
            ]
          },
          {
            "token": "uda",
            "logprob": -0.0012,
            "top_logprobs": [
              { "token": "uda", "logprob": -0.0012 },
              { "token": "uda", "logprob": -7.5 }
            ]
          }
        ]
      },
      "finish_reason": "stop"
    }
  ],
  ...
}

As the API returns logprobs (logarithmic probabilities), which are literally the natural logarithm of the probability $ p $. To get the actual probability percentage, we just need to calculate the exponential:

For example, a logprob of -0.004 corresponds to $ e^{-0.004} \approx 0.996 $, or 99.6%, and -0.002 is as small as 0.2%.

Let’s visualize the probabilities for the “Andrzej Duda” tokens. In this case, the model is nearly 100% certain, and the second-best alternatives have almost zero probability.

{
  "data": [
    {
      "x": ["And", "rzej", " D", "uda"],
      "y": [99.99, 99.98, 99.60, 99.88],
      "type": "bar",
      "marker": { "color": "#2ca02c" },
      "name": "Top Choice (%)"
    },
    {
      "x": ["And", "rzej", " D", "uda"],
      "y": [0.001, 0.01, 0.20, 0.05],
      "type": "bar",
      "marker": { "color": "#d62728" },
      "name": "Second Choice (%)"
    }
  ],
  "layout": {
    "title": "Top vs Second Token Probability",
    "barmode": "group",
    "yaxis": { "title": "Probability (%)", "range": [0, 110] },
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

The model is very confident about the response “Andrzej Duda” tokens!

I specifically chose this example to illustrate an important point: the model’s certainty about a given fact does not mean that it is a correct answer or not a hallucination. After the recent elections in 2025, Karol Nawrocki is the President of Poland. However, based on the data available to the model (and no RAG for up to date info), the model is certain that Andrzej Duda is still the President.

Visualizing “Token Clues”

Nevertheless, it may be interesting to see how confident model is about what he thinks is true. We could visualize some “token clues” to have a grasp of his internal thinking and spot some interesting tokens.

I’ve implemented a simple Logprobs app in C# to play with this. Here are the three main cues I found useful:

1. Low Confidence Clue (“uncertainty”)

If the probability of the chosen token is low (e.g., $< 10\%$ ), the model was very lucky - apparently due to the temperature it selected less probable token. Nevertheless, the effect is low confidence about what the model has selected. Our first condition can be:

2. Ambiguity (close alternatives)

Sometimes the model may be fairly confident, but there is another option(s) just as good. I would call it *ambigue** it the model can choose from two or more token with similar probabilities. Out simplified condition can be:

3. Not Top Choice (sampling effect)

If you use temperature > 0, the model might sample a token that wasn’t the most probable one. This is a clear signal that the output is purely a result of sampling randomness (“creativity”). The condition would be:

Putting it all together

By highlighting text with these cues, we can get an “X-Ray” view of the generation.

• Green: confidence > 90%
• Lime: confidence > 70%
• Yellowe: confidence > 50%
• Orange: condifence > 30%
• Red: Low confidence (<30%)
• Annotations: 🔀 as Ambiguous (point 2.), 🎯 as Not Top (point 3.)

Imagine asking the model: “Who is the president of Poland?”

Here, even a factual answer starts with some ambiguity because the model has multiple ways to phrase its knowledge cutoff disclaimer. Also, we can see the sentence starts as “As of October”… but with 75% it could be “As of my”….

The tool visualizes top-$k$ tokens, so beginning of the sentence in details looks like this:

Of course, we can expect some ambiguity when asking some questions, like “Tabs or spaces? Single word answer.”:

Clearly it prefers spaces…😇

Or we can observe the famous non-randomness of random numbers:

Or some interesting “biases”:

Clearly, for more complex queries the answer will be having much more clues like this, but still interesting to observe:

Note: Here for example we can observe that with 33% chance he was going to use Markdown bold ** when listing cat names.

A Note on Hugging Face

If you are running open-weights models locally (or in the cloud) using the Hugging Face transformers library, you can access the same information.

When calling generate(), simply set return_dict_in_generate=True and output_scores=True. The returned object will contain scores, which are the raw logits. You can then apply softmax to convert them into probabilities.

outputs = model.generate(
    inputs, 
    return_dict_in_generate=True, 
    output_scores=True
)
# outputs.scores contains logits for each generated token

Conclusion

Logprobs give us a peek under the hood. They turn a “black box” text generator into a slightly more transparent probabilistic machine. It won’t help you to fight with hallucinations (as we seen with Andrzej Duda example), but at least can give some clues about what the model thinks!

LLMs are smart “next token predictors”. Token by token, they generate responses, based on the probabilities of what token may come next, out of all tokens in the model’s vocabulary.

For example, gpt-4 has a vocabulary of 100278 tokens, each with its own ID (position) and value. We can see vocabulary as a very high-dimensional vector:

Tokens have very different values. There are obvious ones like single letters, digits or special characters (as we see !, #, A, or B above), whole words (like cat or window) or parts (like ing, urity). Tokens are also case sensitive, and many of them have a version with a leading space. Thus, we can have four different tokens like window, Window, _window and _Window (I’ve used _ to denote whitespace for clarity).

There are also some surprises hiding there. For example, the longest token in gpt-4 represents 128 empty space ( ) characters. And there are many similar, like with 96, 88 or 80 * stars (yes, coming from training on source code and other formatted texts). And the longest “word” is .translatesAutoresizingMaskIntoConstraints. You can use a tool like https://tiktokenizer.vercel.app to play with tokens online.

Logits

So, when at some point in time our LLM generated A cat climbed the bookshelf and knocked over a vase. She looked, it calculated the “odds” of every next possible token from its vocabulary. Some are obvious nonsense in such a context. We don’t expect tokens like {, ******* or .translatesAutoresizingMaskIntoConstraints to be selected here. But some are much more “probable”, like _guilty or _down (note that most probably we need whitespace after the previous word).

The output layer of LLM is producing a score for each token in the model’s vocabulary - so called logits (“log-odds,” the logarithm of the odds of each token). Let’s denote them as $l_i$ (a logit for a token $i$). They are raw, not normalized numerical predictions to represent model’s “confidence” about each token being the next in the sequence. And they are just numbers ranging from $-\infty$ to $+\infty$.

At the last stage, LLM converts them to a probability distribution - where all values sum to 1.0 and have values between 0.0 and 1.0 - with the help of Softmax function, which looks like this:

Note: Typically logits are also normalized by substracing their maximum value, to make calculations more stable.

The idea behind Softmax function is simple - it uses the expotential function $e^x$ to produce non-negative value for each logit, from which we can then calculate its ratio against the sum of all other expotentiated logits.

Temperature

But here comes the simple trick called temperature. What if we would scale each logit before expotentiation, by some value $T$:

Such scaling gives a nice consequence, visible on the below graph:

• for $T > 1$ the curve is flattened, making differences between each exponentiated logit (relative probability) smaller. The bigger $T$, the smaller the differences between resulting probabilities
• for $T < 1$ the curve becomes steeper, making bigger differences. The smaller $T$, the bigger the differences between the most probable logits (right side of the graph) vs the least probable ones (left side of the graph)

{
  "data": [
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.049, 0.082, 0.135, 0.223, 0.368, 0.606, 1, 1.648, 2.718, 4.481, 7.389, 12.182, 20.085],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^x (T=1.0)"
    },
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.002, 0.007, 0.018, 0.050, 0.135, 0.368, 1, 2.718, 7.389, 20.086, 54.598, 148.413, 403.429],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^(x/0.5) (T=0.5)"
    },
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.223, 0.287, 0.368, 0.472, 0.607, 0.779, 1, 1.284, 1.649, 2.117, 2.718, 3.490, 4.482],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^(x/2.0) (T=2.0)"
    }
  ],
  "layout": {
    "title": "Effect of Temperature on Exponential Growth",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "title": "Logit value (x)"
    },
    "yaxis": {
      "title": "Exponential value (e^(x/T))",
      "range": [0, 20]
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

Let’s see how these temperatures affect the final probability distribution for a specific set of logit values like $2$, $1$, $-0.5$ and $-3$:

• $T=1.0$ - the standard Softmax distribution. We can treat it as our baseline
• $T=0.5$ (Low Temperature) - the probabilities become “sharper”. The highest logit (of value $2$) becomes significantly more probable (from ~$68\% \rightarrow 88\%$), while the lower ones are “suppressed”
• $T=2.0$ (High Temperature) - the probabilities become “flatter”. The gap between the most and least probable tokens becomes smaller, making the distribution more uniform

{
  "data": [
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.8756, 0.1185, 0.0059, 0.0000],
      "name": "T=0.5 (Sharper)",
      "type": "bar"
    },
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.6865, 0.2525, 0.0564, 0.0046],
      "name": "T=1.0 (Standard)",
      "type": "bar"
    },
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.5062, 0.3071, 0.1451, 0.0415],
      "name": "T=2.0 (Flatter)",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Probability Distribution vs Temperature",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "barmode": "group",
    "xaxis": {
      "title": "Logits"
    },
    "yaxis": {
      "title": "Probability",
      "range": [0, 1]
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

What about $T=0.0$? Mathematically, we can’t divide by zero. So, in practice, LLMs treat this as a limit - as the temperature approaches zero, the distribution becomes infinitely sharp. At $0.0$ the model simply picks the token with the highest logit and gives it 100% probability, turning the “soft” max into a “hard” max (also called $argmax$ function).

Logprobs

What we’ve seen so far is how the model calculates probabilities internally. But some models/providers allow us to observe the output of such probability calculations in the form of so-called logprobs (Log Probabilities, let’s denote them as $lp$). It’s possible via API call options like IncludeLogProbabilities or Logbprobs (we will play with them more in the upcoming blog post - Update: it’s already published).

Logprobs are natural logarithms of the probability ($ln(p)$). They are already normalized by the model’s Softmax and temperature internally - in other words, the Softmax and temperature have already happened inside the model when we receive them from the API. They take values from $-\infty$ to $0$ (always negative or zero), as they represent probabilities from the range $[0.0, 1.0]$. To calculate probabilities out of it we just need to call $e^x$ on them (inverse of natural logaritm). For example:

Let’s return to our A cat climbed the bookshelf and knocked over a vase. She looked example. When we ask for logprobs we will get them for all tokens in the sequence (including the last one, not yet shown here). We can treat logprobes as yet another high-dimensional vector of logprobs values for each token in the model’s vocabulary.

Here’s the real example of logprobes returned for gpt-4 model and the token after looked (13-th index in sequence):

By mapping positions/ids of tokens around the highest values, we will see that they correspond to the following tokens:

Not all tokens were presented in the above $\mathbf{L}$ but we see an example for tokens $519$ (ant, very low value), $520$ (_at, pretty high value) and $521$ (_ase, again with very low value.)

And here’s the graph presenting values for those top 8 the highest valued logprobes:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        -0.61,
        -1.11,
        -2.98,
        -3.86,
        -4.11,
        -4.48,
        -5.23,
        -5.61
      ],
      "name": "Logit",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Raw Logits",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

By just $p_i = e^{{lp}_i}$ we get corresponding probabilities:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.5536,
        0.3358,
        0.0517,
        0.0215,
        0.0167,
        0.0115,
        0.0055,
        0.0037
      ],
      "name": "Probability",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Probabilities (T=1.0)",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "automargin": true
    },
    "yaxis": {
      "automargin": true
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

Note: Logprobes returned by API are truncated to top $n$ values, so calculated probabilities won’t sum to 1.0

With the help of logprobes we can nicely visualize how temperature influences the resulting probabilities for our sentence:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.3596,
        0.28,
        0.1099,
        0.0708,
        0.0625,
        0.0519,
        0.0357,
        0.0295
      ],
      "name": "T=2.0",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.5536,
        0.3358,
        0.0517,
        0.0215,
        0.0167,
        0.0115,
        0.0055,
        0.0037
      ],
      "name": "T=1.0",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.7248,
        0.2667,
        0.0063,
        0.0011,
        0.0007,
        0.0003,
        0.0001,
        0.0
      ],
      "name": "T=0.5",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.9933,
        0.0067,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
      ],
      "name": "T=0.1",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Temperature Comparison (T=1.0 vs T=0.5 vs T=0.1)",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "automargin": true
    },
    "yaxis": {
      "automargin": true
    },
    "barmode": "group"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

We clearly see, in the real-world example, that:

• by setting high temperature like $T = 2.0$ we make many tokens more probable. It could be interpreted as the model being “more creative” (or less constrained)
• by setting lower temperatures like $T = 0.1$ we definitely “boost” most probable tokens at the expense of the remaining tokens. The model becomes more predictable (“less creative”, if you wish)

That’s it for now! In the next blog post we will dive into visualizing logprobs in a nicer way. We also have yet to cover sampling methods like “top-k” and “top-p”, but that’s also a topic for another day.

Since many months I’ve been working with the Model Context Protocol (MCP) and I found myself frustrated with the available tools to easily access and test them. It is not not even for building your own servers, but just to connect to already made and look around what’s possible. I couldn’t find anything that quite suited my needs - something leightweight, simple to use and CLI-based. And I’m not a big fan of the standard MCP Inspector.

So, I decided to build my own: https://github.com/kkokosa/repl-mcp - a simple, interactive CLI tool for testing MCP servers.

The idea was to have a lightweight, REPL-style (Read-Eval-Print Loop) environment where I could quickly poke at tools, prompts, and resources without any overhead.

Key Features:

• Interactive REPL - command history and tab autocompletion
• Inspect Everything - list and inspect available tools, prompts, and resources
• Call Tools - execute tools both with JSON or interactive parameter input
• Transport Support - works with both Stdio and HTTP transports
• Syntax Highlighting - clean, readable terminal output

Examples

Simple local server via stdio - filesystem

You can try it out with the server-filesystem MCP server. In this case we choose --command (or -c) option to run the server on our own (remember to npm install it beforehand):

repl-mcp -c "npx -y @modelcontextprotocol/server-filesystem /tmp"

Remote server with API token - GitHub

If we want to access remote MCP server, we can use --url/-u option, but bear in mind authorization - they (hopefuly!) require some kind of them.

Some of them may accept API token, like in case of official GitHub MCP Server, which we provide as Bearer token (and use HTTP transport):

repl-mcp -u https://api.githubcopilot.com/mcp/ -t http -H Authorization:"Bearer github_pat_..."

Remote servier with OAuth - Atlassian

Many MCP servers rely on OAuth 2.x flow to authorize the user, which is more tricky. In such case we need to support the full OAuth flow, which requires exposing a public endpoint to the Internet (point 8 in the link), to log in to the given service and let the callback return to us. That could be responsibility of my repl-mcp but there’s already a handy mcp-remote proxy MCP server.

Let’s see in action while trying to use Atlassian Remote MCP Server. It is hosted at https://mcp.atlassian.com/v1/mcp and requires OAuth 2.1 authorization flow. “Supported clients” like ChatGPT, Claude or Google Gemini handle authorization flow out of the box. But as we can read in the docs, their MCP server: “also supports any local MCP-compatible client that can run on localhost and connect to the server via the mcp-remote proxy”.

This is exactly what we will do! Here are the steps:

1. run mcp-remote proxy:

   npx mcp-remote https://mcp.atlassian.com/v1/mcp
   

2. see what port it is using, like Using existing client port: 3736
3. run ngrok or similar tool to expose the port:

   ngrok.exe http --domain some-words-here.ngrok-free.app http://localhost:3736
   

4. also, we need to configure Atlassian to accept/trust our callback domain - in Administration panel, we need to add our [[ngrok]]-based domain to Apps -> AI settings -> Rovo MCP server like https://some-words-here.ngrok-free.app/**
5. go to the link printed in the mcp-remote, something like:

   ...
   Please authorize this client by visiting:
   https://mcp.atlassian.com/v1/authorize?response_type=code&client_id=Y...
   ...
   

6. After logging in/authorizing, the callback will be send (via ngrok) to the mcp-proxy and received token will be stored. Close mcp-remote. We won’t need to repeat it every time.
7. Run the proxy under repl-mcp and enjoy your Atlassian access!

   repl-mcp -c "npx mcp-remote https://mcp.atlassian.com/v1/mcp"