BenchmarkDotNet: measuring C# performance without fooling yourself

You wrap a Stopwatch around two versions of a method, run them once each, and the new one is faster. Ship it. Except the number you just trusted was assembled from tiered JIT warmup, a garbage collection that happened to land on the slower run, and a compiler that may have deleted the work entirely because you never used the result. A single timed call in a managed runtime is not a measurement. It is one sample from a noisy process, reported with no idea how noisy.

BenchmarkDotNet exists because getting this right by hand is genuinely hard, and getting it wrong produces confident, wrong conclusions. The tool is not a stopwatch with nicer output. It is a harness that controls the specific things that make naive .NET timing lie.

One run versus a measured distribution

A number without an error bar is a guess

lower is faster

Stopwatch, single call

n = 1

JIT still warming

No warmup, no repeats, no spread. You cannot tell this bar from noise, a GC pause, or real signal.

BenchmarkDotNet

warmup + many iters

mean

± error

Warmup burns off the JIT and cache effects. The measured band reports a mean with its confidence interval, so the spread is part of the answer.

A benchmark result without an error bar is not a smaller truth than a full one. It is a different thing: a guess that dressed up as a number.

Why the Stopwatch lies, specifically

Each of these is a real mechanism, not a vague "warm up first" superstition:

• Tiered JIT compilation. The runtime first compiles a method to unoptimized code so startup is fast, then recompiles hot methods with full optimization later. Your first calls measure the throwaway tier. Time them and you have benchmarked code the runtime was planning to discard.
• Dead-code elimination. If you compute a value and never use it, the JIT is free to notice and remove the computation. Your "expensive" method can be optimized down to nothing, and you will clock the speed of doing nothing.
• Garbage collection. A GC pause can land on any run. With one run, an unlucky collection triples your number; with one run, a lucky absence of one flatters it. You cannot tell which happened.
• Debug builds. A Debug build disables optimizations and adds instrumentation. Times from it describe a binary you will never deploy.
• A sample size of one. No spread, no confidence interval, no way to know whether a 5% difference between two methods is real or is the difference between two coin flips.

What BenchmarkDotNet does about each

The harness runs each benchmark in a separate process built in Release with optimizations on, so one benchmark cannot pollute another and Debug artifacts stay out. It runs a pilot stage to pick an iteration count, a warmup stage to let tiered compilation and caches settle, and then many measured iterations. From those it reports statistics, not a single number: mean, error, standard deviation, median, and flagged outliers. Add [MemoryDiagnoser] and it also reports bytes allocated and GC collection counts per operation. [Params] runs the whole thing across several input sizes. [Benchmark(Baseline = true)] marks one method as the reference so the others get a Ratio column.

A realistic comparison: building a delimited string three ways, across a small and a large input.

using BenchmarkDotNet.Attributes;

[MemoryDiagnoser]
public class JoinBenchmarks
{
    private int[] _values = null!;

    [Params(100, 10_000)]
    public int N { get; set; }

    [GlobalSetup]
    public void Setup() => _values = Enumerable.Range(0, N).ToArray();

    [Benchmark(Baseline = true)]
    public string Concatenation()
    {
        var result = "";
        foreach (var value in _values)
            result += value + ",";
        return result;
    }

    [Benchmark]
    public string Builder()
    {
        var builder = new System.Text.StringBuilder();
        foreach (var value in _values)
            builder.Append(value).Append(',');
        return builder.ToString();
    }

    [Benchmark]
    public string Join() => string.Join(',', _values);
}

Every method returns its string. That return is what keeps the JIT from eliminating the work. The BenchmarkDotNet getting-started guide covers the project wiring and the BenchmarkRunner.Run entry point.

Read the table like a skeptic

Here is an example run. Your absolute numbers will differ by machine and runtime; the shape is what matters, and the allocation column is the part you can reason about rather than trust.

| Method        | N     | Mean       | Error    | StdDev   | Ratio | Allocated |
|-------------- |------ |----------- |--------- |--------- |------ |---------- |
| Concatenation | 100   |    1.83 us | 0.021 us | 0.019 us | 1.00  |   28.6 KB |
| Builder       | 100   |    0.92 us | 0.008 us | 0.007 us | 0.50  |    2.4 KB |
| Join          | 100   |    0.37 us | 0.004 us | 0.004 us | 0.20  |     704 B |
| Concatenation | 10000 | 9,840.0 us | 96.20 us | 90.00 us | 1.00  |  466.4 MB |
| Builder       | 10000 |    88.6 us |  0.79 us |  0.74 us | 0.009 |  236.4 KB |
| Join          | 10000 |    41.2 us |  0.38 us |  0.36 us | 0.004 |   95.6 KB |

Mean is the average time per operation. Error is the half-width of the confidence interval around that mean, and StdDev is the run-to-run spread. Read them together: Builder at N=100 is 0.92 us with an error of 0.008 us, so it is genuinely faster than Concatenation at 1.83 us. The gap dwarfs the error, so it is signal. If two means sit within each other's error bars, you have not measured a difference, and reporting one is the exact mistake the tool is built to prevent. Ratio normalizes to the baseline: Join at N=10,000 runs at 0.004 of the concatenation time.

The Allocated column is where analytical reasoning beats any timing. A .NET string of n UTF-16 characters costs roughly 2n bytes plus a small object header. The concatenation loop rebuilds the whole accumulated string on every iteration, so it allocates the triangular sum of every intermediate string, which grows with the square of the element count. That is why a hundredfold larger input did not produce a hundredfold larger allocation but a jump from tens of kilobytes to hundreds of megabytes: N-squared, plus the growing width of each number. You do not have to trust the timing to know the concatenation approach is quadratic in garbage; you can derive it. This is the same shape the string concatenation deep dive measures, and the allocation cost is exactly what the allocation tax charges you at the GC.

The mistakes that quietly invalidate a benchmark

• Returning nothing. If a benchmark computes a result and discards it, the JIT may delete the computation. Return the value, or feed it to a Consumer:

private readonly Consumer _consumer = new();

[Benchmark]
public void ParseAll()
{
    foreach (var line in _lines)
        _consumer.Consume(int.Parse(line));
}

• Inputs too small to measure anything but overhead. At N=1 you are timing the harness and the method call, not the algorithm. Use [Params] with sizes that reach the regime you care about, which is why the table above measures 100 and 10,000, not just one.
• Running under the debugger or in Debug. Both disable the optimizations you are trying to measure. BenchmarkDotNet warns you, and you should treat the warning as a failed run.
• Benchmarking the wrong layer. A microbenchmark that shaves 20 microseconds off string formatting is irrelevant if the request spends 40 milliseconds in the database. You optimized a rounding error.

Micro versus macro: profile first, benchmark second

BenchmarkDotNet answers one controlled question precisely: given this input, how does method A compare to method B. It does not tell you whether that method matters to your application. That is a profiler's job. The order that avoids wasted effort is: profile the real workload to find where the time and allocations actually go, form a hypothesis about one suspect, then write a benchmark to compare the current version against a candidate fix under controlled conditions. Benchmarking before profiling is how you end up with a beautifully optimized method nobody's request ever calls, which is the premature optimization trap wearing a lab coat. A micro-measurement is only as useful as the profiling that pointed you at it.

Where the reps happen

Reading about tiered JIT and dead-code elimination is not the same as feeling a benchmark lie to you. The instinct you want is to distrust a single fast run, to check whether a difference clears its error bar, and to reason about allocations from first principles before you believe a column. That comes from measuring many times, against feedback that does not let a warm-JIT fluke pass as a win.

Katabench grades your C# on server-measured time against hidden large inputs and on measured bytes allocated, so the feedback is the same kind BenchmarkDotNet produces: not "it worked," but "here is what it cost, at scale, with the noise controlled." The performance guide explains the time and allocation gates, and how grading works covers why the numbers come from the grading servers rather than your laptop, which is the whole reason they are comparable. Learn to measure honestly on small problems, and you stop being fooled on the large ones.