If you’ve been building .NET applications and wanted to run LLMs locally, your options have been… limited. You could wrap llama.cpp through LLamaSharp, deal with ONNX Runtime, or orchestrate calls to external Python services. None of these are fully satisfying if you want to run LLM inference purely in .NET.

So I built my own.

Over the past two months I’ve been building dotLLM - a ground-up, high-performance LLM inference engine written natively in C#/.NET 10. Not a wrapper. Not bindings. A full implementation: GGUF model loading, tokenization, attention, sampling, SIMD-optimized CPU inference, CUDA GPU acceleration, an OpenAI-compatible API server, and a built-in chat UI. This week I’ve published the first preview release (v0.1.0-preview.2).

This post covers three things: what dotLLM is, why I built it, and how AI-assisted development - Claude Code, Codex, and Gemini working together in a structured workflow - made it possible for a single developer to ship this in about two months.

What is dotLLM?

The short version

dotLLM is a native C#/.NET 10 LLM inference engine. It runs transformer-based models - Llama, Mistral, Phi, Qwen (and more in future) - from GGUF files, with SIMD-optimized CPU, CUDA GPU backend, or hybrid/offloading mode. It exposes an OpenAI-compatible API, ships as a CLI tool and as NuGet packages you can embed in your own applications.

The key word is native. All orchestration, model loading, tokenization, sampling, scheduling, and CPU compute is implemented in pure C#. The only native code is a thin CUDA library for GPU kernels, loaded via PTX through the CUDA Driver API (P/Invoked).

Architecture

dotLLM is organized as a layered architecture where each layer depends only on the layers below:

graph TD
    subgraph "User-Facing"
        CLI["DotLLM.Cli<br/><small>CLI tool</small>"]
        ChatUI["Built-in Chat UI<br/><small>Browser-based</small>"]
        Server["DotLLM.Server<br/><small>OpenAI-compatible API</small>"]
    end

    subgraph "Engine"
        Eng["DotLLM.Engine<br/><small>KV-cache, scheduler, samplers,<br/>constraints, speculative decoding</small>"]
    end

    subgraph "Model Layer"
        Models["DotLLM.Models<br/><small>GGUF loader, Llama, Mistral,<br/>Phi, Qwen, DeepSeek</small>"]
        Tokenizers["DotLLM.Tokenizers<br/><small>BPE, SentencePiece,<br/>Jinja2 chat templates</small>"]
    end

    subgraph "Compute"
        CPU["DotLLM.Cpu<br/><small>SIMD / AVX2 / AVX-512</small>"]
        CUDA["DotLLM.Cuda<br/><small>PTX kernels, cuBLAS</small>"]
    end

    Core["DotLLM.Core<br/><small>ITensor, IBackend, IModel,<br/>ISamplerStep, IDecodingConstraint</small>"]

    CLI --> Server
    ChatUI --> Server
    Server --> Eng
    CLI --> Eng
    Eng --> Models
    Eng --> Tokenizers
    Models --> CPU
    Models --> CUDA
    CPU --> Core
    CUDA --> Core
    Models --> Core
    Tokenizers --> Core
    Eng --> Core

Each project ships as a separate NuGet package. DotLLM.Core defines all abstractions (ITensor, IBackend, IModel, ISamplerStep, etc.) while concrete implementations live in their respective projects. You pull in only what you need.

Key features

Performance:

• Zero-alloc inference - all tensor data uses NativeMemory.AlignedAlloc (64-byte aligned). No managed heap allocations on the hot path (well, “the best” effort so far). (Almost) no allocs, no GC triggered.
• SIMD vectorization for CPU backend - TensorPrimitives for standard operations, hand-tuned System.Runtime.Intrinsics for quantized matmul, RMSNorm, RoPE, softmax operations. AVX2 and AVX-512 with scalar fallbacks.
• CUDA GPU backend - PTX kernels loaded via the CUDA Driver API with cuBLAS HGEMM for prefill, custom quantized GEMV for decode, and FP16 activation pipeline. Supports full GPU inference, hybrid CPU/GPU layer offloading, and KV-cache quantization.
• Memory-mapped model loading - GGUF files loaded via MemoryMappedFile. OS demand-paging means multi-GB models load in milliseconds.
• Quantized inference - FP16, Q8_0, Q4_K_M and other GGUF formats with fused scale-int dot-product kernels operating directly on quantized blocks.

Serving:

• OpenAI-compatible API - /v1/chat/completions, /v1/completions, tool calling, structured output, streaming SSE via ASP.NET.
• Speculative decoding - draft-verify-accept loop with KV-cache rollback for higher throughput.
• Structured output - FSM/PDA-based constrained decoding guaranteeing valid JSON, JSON Schema, regex, and GBNF grammar.

Extensibility:

• Pluggable backends - IBackend interface with separate packages (CPU, CUDA, future ROCm).
• (planned) LoRA adapters - runtime loading, no weight merging, concurrent multi-adapter serving.
• (planned) Diagnostic hooks - zero-cost IInferenceHook for activation capture, logit lens, SAE integration.
• (planned) OpenTelemetry - System.Diagnostics.Metrics + Activity for throughput, latency, and per-request tracing.

Here’s a minimal streaming generation example:

using var gguf = GgufFile.Open(modelPath);
var config = GgufModelConfigExtractor.Extract(gguf.Metadata);
using var model = TransformerModel.LoadFromGguf(gguf, config);
var tokenizer = GgufBpeTokenizerFactory.Load(gguf.Metadata);

var generator = new TextGenerator(model, tokenizer);
var options = new InferenceOptions
{
    SamplerSteps =
    [
        new TemperatureSampler(0.8f),
        new TopKSampler(40),
        new TopPSampler(0.95f)
    ],
    StopConditions = [new EosStopCondition(tokenizer.EosTokenId)],
    MaxTokens = 128
};

await foreach (var token in generator.GenerateStreamingTokensAsync(prompt, options))
    Console.Write(token.Text);

And here’s what the CLI looks like for a quick generation run with SmolLM-135M:

-- dotllm | Llama 30L/576H | Q8_0 | 16 threads | greedy ──────────────────
The capital of France is Paris. Paris is a city of romance and culture,

╭──────────────────────────────────────────────────────────────────────────╮
│  Generation Complete                                      163.27 tok/s  │
│                                                                         │
│  Prefill            12.3 ms       6 tokens       487.80 tok/s           │
│  Decode             91.8 ms      15 tokens       163.40 tok/s           │
│  Sampling            0.1 ms      15 tokens                              │
│  Total             104.2 ms      21 tokens       201.54 tok/s           │
│  Load              456.7 ms                                             │
│                                                                         │
│  Weights         136.73 MiB      (memory-mapped)                        │
│  KV Cache        158.20 MiB      (192 slots)                            │
╰──────────────────────────────────────────────────────────────────────────╯

There is also build-in serve command to host simple chat UX for testing and research:

Performance reality check

Let’s be honest about where dotLLM stands today. While performance has been a key design consideration from day one (zero-alloc hot paths, SIMD kernels, memory-mapped loading), the primary focus so far has been closing the feature set — getting the full inference pipeline, constrained decoding, tool calling, speculative decoding, and the API server working correctly. Dedicated performance polishing is coming. Here are CPU benchmarks against llama.cpp (AMD Ryzen 9 5950X, 16 threads, same models and quantizations):

Decode throughput (tok/s) - the metric that matters most for interactive chat:

On decode, dotLLM reaches 66-88% of llama.cpp throughput. Decode is largely memory-bandwidth-bound, and C# with SIMD intrinsics can get reasonably close to saturating the memory bus.

Prefill is a different story - dotLLM is roughly 2-5x slower than llama.cpp across the board. Prefill is compute-bound, and llama.cpp has years of hand-tuned GEMM kernels. We hit a specific wall here (outer-product tiled matmul vs. RyuJIT register pressure) which I’ll describe in the lessons learned section below.

The CUDA backend is functional but still early - it currently underperforms CPU on small models due to launch overhead, and the kernel tuning work is ongoing.

This is a preview release. If you need maximum throughput today, llama.cpp is faster. The gap will narrow.

Why build this?

If out-of-the-box performance is not easily achieved, why build this at all? A few reasons, starting with the most important:

Understanding by building. I’ve been writing about LLM internals on this blog - temperature and sampling, logprobs, learning a lot on my own. At some point, it makes you want to implement the whole pipeline. Building dotLLM was the logical next step to accelerate learning: implementing the entire inference pipeline from GGUF parsing through attention to token generation, seeing every allocation and every SIMD instruction up close. And it will generate A LOT of other blog posts…😇

Seeing how far AI-assisted development can go. This was also, explicitly, an experiment. Not vibe coding - not “prompt and pray” loop. Structured, documented, reviewed AI-assisted development where a human makes the architectural decisions and AI handles implementation within well-defined boundaries. I wanted to find out what a solo developer can realistically build in one-two months with this approach. The answer surprised me.

Creating a platform for research and experimentation. Building from the ground up with this goal in mind means the architecture is open to experimentation - adding new features, deeper diagnostics, and interpretability tools into the inference pipeline. All outside the gold-standard HuggingFace monopoly. In .NET 😍

Proving the platform. There’s a persistent assumption that systems-level performance work requires C, C++, or Rust. Twenty years of .NET performance work has taught me that’s not always true. C# with NativeMemory, System.Runtime.Intrinsics, MemoryMappedFile, and Span<T> gives you genuine control over memory and compute. dotLLM is a proof point.

The .NET ecosystem gap is real. If you’re building .NET applications and want local LLM inference, your choices are wrappers (LLamaSharp wrapping llama.cpp), limited runtimes (ONNX Runtime with restricted model support), or orchestration layers (Semantic Kernel, which is about chaining calls, not running inference). Enterprise .NET shops that want to run models in production without Python or C++ dependencies have no native option. dotLLM fills that gap, but it will take a LOT of time to treat it as a serious replacement.

dotLLM is not meant to replace llama.cpp or vLLM in production - at least not yet. It’s built for .NET developers who want native inference without leaving their ecosystem, and for researchers and experimenters who want to explore LLM internals from C#. And everyone who wants to understand how to build a LLM inference engine from scratch.

How AI built an AI engine

Let me be upfront: nearly every commit in dotLLM’s git history has a Co-Authored-By: Claude Opus 4.6 line. This project would not exist in its current form without AI assistance. But the story of how that assistance was structured is more interesting than the fact that it was used.

The development methodology

dotLLM was built over ~60 implementation steps organized into 7 phases, documented in a detailed ROADMAP.md. Each phase was a discrete unit of work with clear scope and acceptance criteria:

• Phase 1 - End-to-end single-token generation (GGUF loader, dequantization, CPU ops, tokenizer, attention, KV-cache, sampling)
• Phase 2 - Practical local inference (Q4_K_M, chat templates, streaming, multi-threading, additional architectures)
• Phase 3 - CPU performance (tiled attention, SIMD tuning, NUMA awareness, operator fusion)
• Phase 4 - GPU acceleration (CUDA backend, hybrid CPU/GPU, KV-cache quantization)
• Phase 5 - Constrained decoding and API (JSON/schema/regex/grammar, tool calling, server, chat UI, prompt caching)
• Phase 6 - Improved serving (warm-up, Native AOT, paged KV-cache, speculative decoding)
• Phase 7 - Diagnostics and interpretability (logprobs, hooks, logit lens - in progress)

Every step started as a GitHub issue. Every issue lived on a branch named issue/{number}-{short-description}. Every PR closed its issue and updated the roadmap. This was relentlessly boring discipline, and it was the single most important factor in the project’s success.

After the initial release, just before making the repository public, I also ran a series of “Waves” - systematic quality passes across the entire codebase:

• Wave 1 (P0): Security and crash fixes - path traversal, CUDA shared memory guards, hybrid GPU edge cases
• Wave 2 (P1): Quick correctness and consistency fixes
• Wave 3: Presentation cleanup - remove dead code, label stubs, fix samples
• Wave 4: Server hardening - request validation, LINQ removal from hot paths
• Wave 5 was skipped - it was earmarked for batch-serving improvements that depend on Phase 9 (continuous batching), which isn’t implemented yet
• Wave 6: CUDA kernel rewrite - tiled softmax, vectorization, grid-stride loops
• Wave 7: CPU performance - TopK sampler optimization, AVX2 gap filling, schema cache tuning

Those “waves” are a bunch of findings that come from in-depth reviews from other models, grouped into GitHub issues.

ROADMAP.md and CLAUDE.md - the highest-ROI investments

If there’s one takeaway from this experiment, it’s this: the time you spend writing structured documentation for AI is not overhead - it IS the development methodology.

ROADMAP.md was the backbone. Each of the ~60 steps had a feature name, a description, key files to modify, and dependencies on other steps. This gave both me and the AI a shared understanding of what to build next, in what order, and why. Without it, AI would be coding in circles - solving the wrong problem, building features in the wrong order, missing dependencies.

The roadmap also forced me to think through the architecture upfront. When you have to discuss things like “Step 31: CUDA backend - PTX kernels loaded via CUDA Driver API, no native shared library, cuBLAS HGEMM for prefill, custom quantized GEMV for decode” before writing any code, you’ve already made the hard decisions. And learnt a lot.

CLAUDE.md was the project’s “constitution” - 180+ lines defining how AI should work in this codebase. Here are some actual rules from it:

**Native .NET first** - All orchestration, model loading, tokenization, sampling, scheduling, CPU compute in pure C#.

**Unmanaged memory for tensors** - `NativeMemory.AlignedAlloc` (64-byte). Zero GC allocations on inference hot path.

**Hybrid GPU architecture** - Thin native C/CUDA lib via `[LibraryImport]`. GPU memory as opaque `IntPtr` - tensor data never crosses P/Invoke boundary.

And specific coding rules:

**NEVER** allocate managed arrays for tensor data. Use `NativeMemory.AlignedAlloc` (64-byte for AVX-512, 32-byte for AVX2).

SIMD: Foundation is `System.Numerics.Tensors.TensorPrimitives` for standard ops. Hot inner loops: `System.Runtime.Intrinsics` - prefer cross-platform `Vector128<T>`/`Vector256<T>`, use platform-specific only when measurably faster. **Always provide scalar fallback.

Beyond CLAUDE.md, there were 22 detailed design documents in /docs/ - one for each major subsystem like ARCHITECTURE, QUANTIZATION, ATTENTION, CUDA, and more. The rule was simple: AI reads the relevant spec before touching a module.

This documentation-first approach had a compound effect. Every implementation step could reference the roadmap for scope, the design docs for architecture, and CLAUDE.md for coding conventions. The AI wasn’t guessing - it was implementing within well-defined constraints.

Claude Code as implementation partner

Claude Code with Opus 4.6 (1M context) was the primary implementation tool from the project start. The workflow was built around six custom Claude Code skills that automated the development lifecycle:

• /plan-step - looks for a given roadmap step in ROADMAP.md plus relevant docs from /docs/, enters plan mode, and produces a step-by-step implementation plan for my approval before any code is written.
• /create-pr - commits remaining changes, pushes the branch, and creates a PR with a detailed description following project conventions.
• /apply-pr-comments - reads review comments from Codex, Gemini, or human reviewers, analyzes them, enters plan mode so I can approve the fixes before any code changes.
• /finish-pr-comments - after fixes are applied and tested, commits the changes, pushes to the PR branch, and replies to each reviewer comment with what was fixed and the commit hash.
• /merge-pr - squash-merges the PR into main, deletes the remote branch, and switches to an updated local main.
• /plan-issue - similar to plan-step but for less frequent case when we start from an issue, not a roadmap’s step.

There was also a GitHub Actions workflow that responds to @claude mentions in PRs and issues, enabling asynchronous interaction.

The typical flow for a feature looked like this: I’d run /plan-step (or /plan-issue), review and adjust the plan, then let Claude implement it step by step while I reviewed each change. The key insight is that planning was always a separate, human-approved step before implementation began. Most of the work and brainstorming happened there.

Codex and Gemini as PR reviewers

Every PR was also reviewed by Codex and Gemini triggered manually via mentions of @codex and @gemini in PR comments.

Gemini was powered by a custom Python bot (.github/scripts/gemini_bot.py) with retry logic and configurable thinking budgets. A separate GEMINI.md file defined its review persona.

The review findings were not cosmetic. They caught genuinely critical bugs that could have shipped. Just as examples:

• KV-cache quantization (PR #75): Codex caught a ring-buffer indexing bug (window reads used linear indexing instead of ring indices, producing garbage after wrap-around) or a pinned buffer scope issue (pointers from fixed blocks used after the scope exited), and a shared-state race condition (per-layer eviction progress stored in a shared counter)
• JSON Schema constrained decoding (PR #79): Found a cache key collision (string substates not included in the hash, collapsing distinct parser states) and a unicode escape flag preservation bug (\u parsing wiped the key-string flag)
• Wave 6 CUDA kernel rewrite (PR #114): Gemini identified thread underutilization in GEMV kernels and uncoalesced memory reads - architectural issues that Codex’s code-level analysis didn’t surface

Once Codex and Gemini leave their comments, the remaining skills close the loop. /apply-pr-comments reads all review comments on the current PR, analyzes them, and enters plan mode — so I can approve which fixes to make before any code changes. This prevents blindly applying every suggestion without human judgment on what’s worth addressing versus deferring.

After the fixes are implemented and tested, straightforward /finish-pr-comments commits the changes, pushes to the PR branch, and has the Claude Code bot (dotllm-claude-code-bot) reply to each reviewer comment with what was fixed and the corresponding commit hash.

This creates a fully traceable chain: Codex/Gemini finds a bug -> Claude fixes it -> the reply references the exact commit. The PR thread becomes a complete audit trail.

Finally, /merge-pr squash-merges the PR into main, deletes the remote branch, and checks out an updated local main. The entire cycle - from roadmap step to merged PR - typically took one Claude Code session.

Lessons learned

What worked brilliantly:

• The implementation is correct. This is the most obvious finding to overlook. But here it is - this setup really worked! Step by step, we were progressing with the implementation, making it more and more sophisticated. And the moment when for the first time I saw an answer from the real model - it felt like magic! Reaching 80-90% of llama.cpp CPU decode performance is a thing, too.
• ROADMAP.md was the highest-ROI time investment. Planning structured documentation saved many hours of correcting misdirected AI implementation. The documentation is the development methodology - it’s not overhead, it’s the thing that makes AI-assisted development work at all. Without the roadmap, the AI has no sense of direction.
• Two-role separation: implementation vs. review. Claude wrote the code. Codex and Gemini reviewed it independently. The AI that writes the code should not be the only AI that reviews it. Different models have genuinely different blind spots - Codex and Gemini almost never flagged the same issues.
• No Ralph Wiggum loop or other YOLO mode. I really wanted to drive the work, not fire & forget it to burn tokens. It worked out perfectly for me. The single task took around several minutes, up to 30-40, so you could return to them from time to time, during breaks whilst waiting for something else. Great background work.

Where AI struggled:

• Fundamental architectural decisions. Which attention strategy to use, how to structure the GPU interop layer, whether to use PagedAttention kernels or staging-buffer gather - these required human judgment informed by comparing to llama.cpp, vLLM, and the research literature. AI can implement an architecture, but it’s less reliable at choosing between architectures.
• The prefill throughput gap. This is the biggest remaining performance wall - but to be fair, it’s not really an AI failure per se. It might be AI struggling to workaround platform’s out-of-the-box behaviours. The 2-5x prefill gap you saw in the benchmarks comes down to one unfinished roadmap step: Step 26, the outer-product tiled matmul. The current batched GEMM (Step 11) uses an inner-product formulation - it tiles over M but still computes one output element at a time. It delivered a 2.2x prefill speedup when implemented. But the “outer-product” approach would share one activation load across all weight-row dot products: a 4x3 AVX2 tile with 12 YMM accumulators. Problem: due to AI that tile needs ~23 YMM registers, but AVX2 only has 16. RyuJIT spills to stack, and the spills eat the reuse benefit the tile was designed to get. AI was actually very good at experimenting here — it iterated through multiple tile sizes, inspected the JIT disassembly for spills, and methodically narrowed down the root cause. The options now seems to be: AVX-512 (32 ZMM registers, fits comfortably), a native C microkernel via [LibraryImport] (what llamafile/tinyBLAS does), or a smaller tile with partial reuse. I’ve left this for a much more manual, still AI-assisted, research phase - it needs careful benchmarking and disassembly analysis rather than the “plan step, implement, make PR” cadence that worked for the rest of the project.
• Keeping it within the guiderails. Small but very annoying. No matter how many times, and in how many places, I ask him to avoid compound tool calls (esp. cd D:\github\ai\dotLLM && git ...) it eventually ignores it, breaking the permissions. It breaks the flow, as instead of an autonomous work it got stuck waiting for permission for tools that otherwise should be accepted (like allowed GitHub read-only commands etc). Now I think I need pre-tool hook that checks that💡
• Claude Code got really stuck 2-3 times. There were a few times when implementing a step took a lot of struggle. Implementing, retrying, failing, in a loop that took literally hours (!). But even more surprising is the fact that he eventually did it. With some of my help and brainstorming, but it was mostly his async work trying to figure it out.

What’s next

dotLLM is at v0.1.0-preview.2 - explicitly a preview. The foundations are solid, but there’s a long road ahead:

• Phase 7 (in progress): Diagnostic hooks, logit lens, Sparse Autoencoders (SAE) integration, LoRA adapters
• Phase 8 (planned): MLA attention (DeepSeek), SmolLM3, Gemma 4, Mixture of Experts
• Phase 9 (planned): Production serving - continuous batching, prefix sharing, advanced scheduling. This will be fun🔥

After those Phases I will consider feature set (mostly) done and start to focus on performance more and more.

BTW, the project is GPLv3 and contributions are welcome! There is still a lot to do. The codebase has 22 design docs, a detailed roadmap, and a CLAUDE.md that makes it easy for both human and AI contributors to get oriented quickly.

• GitHub: github.com/kkokosa/dotLLM
• Website: dotllm.dev
• NuGet packages: DotLLM.Engine, DotLLM.Cpu, DotLLM.Cuda, DotLLM.Server, and more
• Discussions: GitHub Discussions

Closing

Two things are true at once. First: .NET can do native, systems-level AI work. Zero-GC inference, SIMD-vectorized kernels, memory-mapped model loading, paged KV-cache, speculative decoding - all in C#. The platform is more capable than many give it credit for.

Second: a solo developer can build something of this scope in two months with AI assistance - but only with relentless structure. The roadmap, the design docs, the CLAUDE.md constitution, the dual-review workflow - take any of these away and the productivity collapses. AI amplifies discipline; it doesn’t replace it.

If you’re a .NET developer curious about LLM inference, or a researcher who wants to explore model internals from C#, give dotLLM a try. File issues. Break things. Tell me what’s missing.

But looking at raw numbers and graphs is boring. Let’s see how we can use this data to visualize the model’s “confidence” and at least try to spot places when it is “unsure”.

Getting Logprobs from API

If you are using OpenAI API we need to set two parameters: logprobs to true and top_logprobs to some integer $k$ (e.g. 2 or 5) to see top-$k$ alternatives. Information will be provided for all tokens from the response, as they were seen at the time of generation of each token.

Here is what a raw sample HTTP request could look for a question “Who is the President of Poland?”:

curl https://api.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant."
      },
      {
        "role": "user",
        "content": "Who is the president of Poland?"
      }
    ],
    "logprobs": true,
    "top_logprobs": 2
  }'

And here is a relevant snippet of the JSON response we get back. Notice the logprobs field inside choices:

{
  "id": "chatcmpl-123...",
  "object": "chat.completion",
  "created": 1677652288,
  "model": "gpt-4o-2024-08-06",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Andrzej Duda",
        "refusal": null,
        "annotations": []        
      },
      "logprobs": {
        "content": [
          {
            "token": "And",
            "logprob": -0.000035,
            "top_logprobs": [ 
              { "token": "And", "logprob": -0.000035 },
              { "token": "The","logprob": -11.4 }
            ]
          },
          {
            "token": "rzej",
            "logprob": -0.00021,
            "top_logprobs": [
              { "token": "rzej", "logprob": -0.00021 },
              { "token": "rew",  "logprob": -9.1 }
            ]
          },
          {
            "token": " D", "logprob": -0.004,
            "top_logprobs": [
              { "token": " D", "logprob": -0.004 },
              { "token": " Du", "logprob": -6.2 }
            ]
          },
          {
            "token": "uda",
            "logprob": -0.0012,
            "top_logprobs": [
              { "token": "uda", "logprob": -0.0012 },
              { "token": "uda", "logprob": -7.5 }
            ]
          }
        ]
      },
      "finish_reason": "stop"
    }
  ],
  ...
}

As the API returns logprobs (logarithmic probabilities), which are literally the natural logarithm of the probability $ p $. To get the actual probability percentage, we just need to calculate the exponential:

For example, a logprob of -0.004 corresponds to $ e^{-0.004} \approx 0.996 $, or 99.6%, and -0.002 is as small as 0.2%.

Let’s visualize the probabilities for the “Andrzej Duda” tokens. In this case, the model is nearly 100% certain, and the second-best alternatives have almost zero probability.

{
  "data": [
    {
      "x": ["And", "rzej", " D", "uda"],
      "y": [99.99, 99.98, 99.60, 99.88],
      "type": "bar",
      "marker": { "color": "#2ca02c" },
      "name": "Top Choice (%)"
    },
    {
      "x": ["And", "rzej", " D", "uda"],
      "y": [0.001, 0.01, 0.20, 0.05],
      "type": "bar",
      "marker": { "color": "#d62728" },
      "name": "Second Choice (%)"
    }
  ],
  "layout": {
    "title": "Top vs Second Token Probability",
    "barmode": "group",
    "yaxis": { "title": "Probability (%)", "range": [0, 110] },
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

The model is very confident about the response “Andrzej Duda” tokens!

I specifically chose this example to illustrate an important point: the model’s certainty about a given fact does not mean that it is a correct answer or not a hallucination. After the recent elections in 2025, Karol Nawrocki is the President of Poland. However, based on the data available to the model (and no RAG for up to date info), the model is certain that Andrzej Duda is still the President.

Visualizing “Token Clues”

Nevertheless, it may be interesting to see how confident model is about what he thinks is true. We could visualize some “token clues” to have a grasp of his internal thinking and spot some interesting tokens.

I’ve implemented a simple Logprobs app in C# to play with this. Here are the three main cues I found useful:

1. Low Confidence Clue (“uncertainty”)

If the probability of the chosen token is low (e.g., $< 10\%$ ), the model was very lucky - apparently due to the temperature it selected less probable token. Nevertheless, the effect is low confidence about what the model has selected. Our first condition can be:

2. Ambiguity (close alternatives)

Sometimes the model may be fairly confident, but there is another option(s) just as good. I would call it *ambigue** it the model can choose from two or more token with similar probabilities. Out simplified condition can be:

3. Not Top Choice (sampling effect)

If you use temperature > 0, the model might sample a token that wasn’t the most probable one. This is a clear signal that the output is purely a result of sampling randomness (“creativity”). The condition would be:

Putting it all together

By highlighting text with these cues, we can get an “X-Ray” view of the generation.

• Green: confidence > 90%
• Lime: confidence > 70%
• Yellowe: confidence > 50%
• Orange: condifence > 30%
• Red: Low confidence (<30%)
• Annotations: 🔀 as Ambiguous (point 2.), 🎯 as Not Top (point 3.)

Imagine asking the model: “Who is the president of Poland?”

Here, even a factual answer starts with some ambiguity because the model has multiple ways to phrase its knowledge cutoff disclaimer. Also, we can see the sentence starts as “As of October”… but with 75% it could be “As of my”….

The tool visualizes top-$k$ tokens, so beginning of the sentence in details looks like this:

Of course, we can expect some ambiguity when asking some questions, like “Tabs or spaces? Single word answer.”:

Clearly it prefers spaces…😇

Or we can observe the famous non-randomness of random numbers:

Or some interesting “biases”:

Clearly, for more complex queries the answer will be having much more clues like this, but still interesting to observe:

Note: Here for example we can observe that with 33% chance he was going to use Markdown bold ** when listing cat names.

A Note on Hugging Face

If you are running open-weights models locally (or in the cloud) using the Hugging Face transformers library, you can access the same information.

When calling generate(), simply set return_dict_in_generate=True and output_scores=True. The returned object will contain scores, which are the raw logits. You can then apply softmax to convert them into probabilities.

outputs = model.generate(
    inputs, 
    return_dict_in_generate=True, 
    output_scores=True
)
# outputs.scores contains logits for each generated token

Conclusion

Logprobs give us a peek under the hood. They turn a “black box” text generator into a slightly more transparent probabilistic machine. It won’t help you to fight with hallucinations (as we seen with Andrzej Duda example), but at least can give some clues about what the model thinks!

LLMs are smart “next token predictors”. Token by token, they generate responses, based on the probabilities of what token may come next, out of all tokens in the model’s vocabulary.

For example, gpt-4 has a vocabulary of 100278 tokens, each with its own ID (position) and value. We can see vocabulary as a very high-dimensional vector:

Tokens have very different values. There are obvious ones like single letters, digits or special characters (as we see !, #, A, or B above), whole words (like cat or window) or parts (like ing, urity). Tokens are also case sensitive, and many of them have a version with a leading space. Thus, we can have four different tokens like window, Window, _window and _Window (I’ve used _ to denote whitespace for clarity).

There are also some surprises hiding there. For example, the longest token in gpt-4 represents 128 empty space ( ) characters. And there are many similar, like with 96, 88 or 80 * stars (yes, coming from training on source code and other formatted texts). And the longest “word” is .translatesAutoresizingMaskIntoConstraints. You can use a tool like https://tiktokenizer.vercel.app to play with tokens online.

Logits

So, when at some point in time our LLM generated A cat climbed the bookshelf and knocked over a vase. She looked, it calculated the “odds” of every next possible token from its vocabulary. Some are obvious nonsense in such a context. We don’t expect tokens like {, ******* or .translatesAutoresizingMaskIntoConstraints to be selected here. But some are much more “probable”, like _guilty or _down (note that most probably we need whitespace after the previous word).

The output layer of LLM is producing a score for each token in the model’s vocabulary - so called logits (“log-odds,” the logarithm of the odds of each token). Let’s denote them as $l_i$ (a logit for a token $i$). They are raw, not normalized numerical predictions to represent model’s “confidence” about each token being the next in the sequence. And they are just numbers ranging from $-\infty$ to $+\infty$.

At the last stage, LLM converts them to a probability distribution - where all values sum to 1.0 and have values between 0.0 and 1.0 - with the help of Softmax function, which looks like this:

Note: Typically logits are also normalized by substracing their maximum value, to make calculations more stable.

The idea behind Softmax function is simple - it uses the expotential function $e^x$ to produce non-negative value for each logit, from which we can then calculate its ratio against the sum of all other expotentiated logits.

Temperature

But here comes the simple trick called temperature. What if we would scale each logit before expotentiation, by some value $T$:

Such scaling gives a nice consequence, visible on the below graph:

• for $T > 1$ the curve is flattened, making differences between each exponentiated logit (relative probability) smaller. The bigger $T$, the smaller the differences between resulting probabilities
• for $T < 1$ the curve becomes steeper, making bigger differences. The smaller $T$, the bigger the differences between the most probable logits (right side of the graph) vs the least probable ones (left side of the graph)

{
  "data": [
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.049, 0.082, 0.135, 0.223, 0.368, 0.606, 1, 1.648, 2.718, 4.481, 7.389, 12.182, 20.085],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^x (T=1.0)"
    },
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.002, 0.007, 0.018, 0.050, 0.135, 0.368, 1, 2.718, 7.389, 20.086, 54.598, 148.413, 403.429],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^(x/0.5) (T=0.5)"
    },
    {
      "x": [-3, -2.5, -2, -1.5, -1, -0.5, 0, 0.5, 1, 1.5, 2, 2.5, 3],
      "y": [0.223, 0.287, 0.368, 0.472, 0.607, 0.779, 1, 1.284, 1.649, 2.117, 2.718, 3.490, 4.482],
      "type": "scatter",
      "mode": "lines+markers",
      "name": "e^(x/2.0) (T=2.0)"
    }
  ],
  "layout": {
    "title": "Effect of Temperature on Exponential Growth",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "title": "Logit value (x)"
    },
    "yaxis": {
      "title": "Exponential value (e^(x/T))",
      "range": [0, 20]
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

Let’s see how these temperatures affect the final probability distribution for a specific set of logit values like $2$, $1$, $-0.5$ and $-3$:

• $T=1.0$ - the standard Softmax distribution. We can treat it as our baseline
• $T=0.5$ (Low Temperature) - the probabilities become “sharper”. The highest logit (of value $2$) becomes significantly more probable (from ~$68\% \rightarrow 88\%$), while the lower ones are “suppressed”
• $T=2.0$ (High Temperature) - the probabilities become “flatter”. The gap between the most and least probable tokens becomes smaller, making the distribution more uniform

{
  "data": [
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.8756, 0.1185, 0.0059, 0.0000],
      "name": "T=0.5 (Sharper)",
      "type": "bar"
    },
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.6865, 0.2525, 0.0564, 0.0046],
      "name": "T=1.0 (Standard)",
      "type": "bar"
    },
    {
      "x": ["Token A (2.0)", "Token B (1.0)", "Token C (-0.5)", "Token D (-3.0)"],
      "y": [0.5062, 0.3071, 0.1451, 0.0415],
      "name": "T=2.0 (Flatter)",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Probability Distribution vs Temperature",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "barmode": "group",
    "xaxis": {
      "title": "Logits"
    },
    "yaxis": {
      "title": "Probability",
      "range": [0, 1]
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

What about $T=0.0$? Mathematically, we can’t divide by zero. So, in practice, LLMs treat this as a limit - as the temperature approaches zero, the distribution becomes infinitely sharp. At $0.0$ the model simply picks the token with the highest logit and gives it 100% probability, turning the “soft” max into a “hard” max (also called $argmax$ function).

Logprobs

What we’ve seen so far is how the model calculates probabilities internally. But some models/providers allow us to observe the output of such probability calculations in the form of so-called logprobs (Log Probabilities, let’s denote them as $lp$). It’s possible via API call options like IncludeLogProbabilities or Logbprobs (we will play with them more in the upcoming blog post - Update: it’s already published).

Logprobs are natural logarithms of the probability ($ln(p)$). They are already normalized by the model’s Softmax and temperature internally - in other words, the Softmax and temperature have already happened inside the model when we receive them from the API. They take values from $-\infty$ to $0$ (always negative or zero), as they represent probabilities from the range $[0.0, 1.0]$. To calculate probabilities out of it we just need to call $e^x$ on them (inverse of natural logaritm). For example:

Let’s return to our A cat climbed the bookshelf and knocked over a vase. She looked example. When we ask for logprobs we will get them for all tokens in the sequence (including the last one, not yet shown here). We can treat logprobes as yet another high-dimensional vector of logprobs values for each token in the model’s vocabulary.

Here’s the real example of logprobes returned for gpt-4 model and the token after looked (13-th index in sequence):

By mapping positions/ids of tokens around the highest values, we will see that they correspond to the following tokens:

Not all tokens were presented in the above $\mathbf{L}$ but we see an example for tokens $519$ (ant, very low value), $520$ (_at, pretty high value) and $521$ (_ase, again with very low value.)

And here’s the graph presenting values for those top 8 the highest valued logprobes:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        -0.61,
        -1.11,
        -2.98,
        -3.86,
        -4.11,
        -4.48,
        -5.23,
        -5.61
      ],
      "name": "Logit",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Raw Logits",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

By just $p_i = e^{{lp}_i}$ we get corresponding probabilities:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.5536,
        0.3358,
        0.0517,
        0.0215,
        0.0167,
        0.0115,
        0.0055,
        0.0037
      ],
      "name": "Probability",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Probabilities (T=1.0)",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "automargin": true
    },
    "yaxis": {
      "automargin": true
    }
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

Note: Logprobes returned by API are truncated to top $n$ values, so calculated probabilities won’t sum to 1.0

With the help of logprobes we can nicely visualize how temperature influences the resulting probabilities for our sentence:

{
  "data": [
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.3596,
        0.28,
        0.1099,
        0.0708,
        0.0625,
        0.0519,
        0.0357,
        0.0295
      ],
      "name": "T=2.0",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.5536,
        0.3358,
        0.0517,
        0.0215,
        0.0167,
        0.0115,
        0.0055,
        0.0037
      ],
      "name": "T=1.0",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.7248,
        0.2667,
        0.0063,
        0.0011,
        0.0007,
        0.0003,
        0.0001,
        0.0
      ],
      "name": "T=0.5",
      "type": "bar"
    },
    {
      "x": [
        "innocent",
        "at",
        "proud",
        "surprised",
        "down",
        "around",
        "guilty",
        "back"
      ],
      "y": [
        0.9933,
        0.0067,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0,
        0.0
      ],
      "name": "T=0.1",
      "type": "bar"
    }
  ],
  "layout": {
    "title": "Temperature Comparison (T=1.0 vs T=0.5 vs T=0.1)",
    "paper_bgcolor": "rgba(0,0,0,0)",
    "plot_bgcolor": "rgba(0,0,0,0)",
    "xaxis": {
      "automargin": true
    },
    "yaxis": {
      "automargin": true
    },
    "barmode": "group"
  },
  "config": {
    "displayModeBar": false,
    "responsive": true
  }
}

We clearly see, in the real-world example, that:

• by setting high temperature like $T = 2.0$ we make many tokens more probable. It could be interpreted as the model being “more creative” (or less constrained)
• by setting lower temperatures like $T = 0.1$ we definitely “boost” most probable tokens at the expense of the remaining tokens. The model becomes more predictable (“less creative”, if you wish)

That’s it for now! In the next blog post we will dive into visualizing logprobs in a nicer way. We also have yet to cover sampling methods like “top-k” and “top-p”, but that’s also a topic for another day.

Since many months I’ve been working with the Model Context Protocol (MCP) and I found myself frustrated with the available tools to easily access and test them. It is not not even for building your own servers, but just to connect to already made and look around what’s possible. I couldn’t find anything that quite suited my needs - something leightweight, simple to use and CLI-based. And I’m not a big fan of the standard MCP Inspector.

So, I decided to build my own: https://github.com/kkokosa/repl-mcp - a simple, interactive CLI tool for testing MCP servers.

The idea was to have a lightweight, REPL-style (Read-Eval-Print Loop) environment where I could quickly poke at tools, prompts, and resources without any overhead.

Key Features:

• Interactive REPL - command history and tab autocompletion
• Inspect Everything - list and inspect available tools, prompts, and resources
• Call Tools - execute tools both with JSON or interactive parameter input
• Transport Support - works with both Stdio and HTTP transports
• Syntax Highlighting - clean, readable terminal output

Examples

Simple local server via stdio - filesystem

You can try it out with the server-filesystem MCP server. In this case we choose --command (or -c) option to run the server on our own (remember to npm install it beforehand):

repl-mcp -c "npx -y @modelcontextprotocol/server-filesystem /tmp"

Remote server with API token - GitHub

If we want to access remote MCP server, we can use --url/-u option, but bear in mind authorization - they (hopefuly!) require some kind of them.

Some of them may accept API token, like in case of official GitHub MCP Server, which we provide as Bearer token (and use HTTP transport):

repl-mcp -u https://api.githubcopilot.com/mcp/ -t http -H Authorization:"Bearer github_pat_..."

Remote servier with OAuth - Atlassian

Many MCP servers rely on OAuth 2.x flow to authorize the user, which is more tricky. In such case we need to support the full OAuth flow, which requires exposing a public endpoint to the Internet (point 8 in the link), to log in to the given service and let the callback return to us. That could be responsibility of my repl-mcp but there’s already a handy mcp-remote proxy MCP server.

Let’s see in action while trying to use Atlassian Remote MCP Server. It is hosted at https://mcp.atlassian.com/v1/mcp and requires OAuth 2.1 authorization flow. “Supported clients” like ChatGPT, Claude or Google Gemini handle authorization flow out of the box. But as we can read in the docs, their MCP server: “also supports any local MCP-compatible client that can run on localhost and connect to the server via the mcp-remote proxy”.

This is exactly what we will do! Here are the steps:

1. run mcp-remote proxy:

   npx mcp-remote https://mcp.atlassian.com/v1/mcp
   

2. see what port it is using, like Using existing client port: 3736
3. run ngrok or similar tool to expose the port:

   ngrok.exe http --domain some-words-here.ngrok-free.app http://localhost:3736
   

4. also, we need to configure Atlassian to accept/trust our callback domain - in Administration panel, we need to add our [[ngrok]]-based domain to Apps -> AI settings -> Rovo MCP server like https://some-words-here.ngrok-free.app/**
5. go to the link printed in the mcp-remote, something like:

   ...
   Please authorize this client by visiting:
   https://mcp.atlassian.com/v1/authorize?response_type=code&client_id=Y...
   ...
   

6. After logging in/authorizing, the callback will be send (via ngrok) to the mcp-proxy and received token will be stored. Close mcp-remote. We won’t need to repeat it every time.
7. Run the proxy under repl-mcp and enjoy your Atlassian access!

   repl-mcp -c "npx mcp-remote https://mcp.atlassian.com/v1/mcp"