Horus

Tokens per GPU is a software problem.

Horus is an AI performance engineer for LLM inference. It traces SGLang and vLLM deployments end to end, finds where performance is lost, and uses agents to fix it, delivering benchmark-verified code you keep.

Scope a pilot See how it works

Inference optimization

Same work. Shorter critical path.

One decode iteration, zoomed in on a single block to show why it finishes sooner.

Concept · illustrative timing
Scope One decode iteration
  1. Inputs
  2. Repeated decoder blocks Zoom into one representative block
  3. LM head
  4. Select token

Representative block

Same workload · same horizontal scale

Small kernels Compute Communication Wait

N / S / Q = normalize / scale / quantize. A / B = chunks. D = dispatch; R = return. G = gate. P = post.

  1. 1Kernel fusion

    Fewer boundaries

    Norm Scale Quant One fused kernel

    Merge compatible work; reduce launches and intermediate memory traffic.

  2. 2Faster kernels

    Shorter device execution

    Same logical operation Faster implementation

    The same operation, checked against a reference, in less time on the device.

  3. 3Compute + comm overlap

    Less exposed communication

    Compute Compute A Comm Dispatch B

    Send B while computing A; return A while computing B. Dependencies are preserved.

Hold model, hardware, batch, context and speculation policy constant when comparing measured traces.

Illustration, not measured data. Lanes show logical work, and communication may execute inside a fused GPU kernel. No measured latency, speedup or claim about default SGLang behavior is represented.

The gap

Why the gap stays open

Profilers show what ran, not what to change

A timeline of kernels and API calls still leaves an engineer to connect it to the scheduler, KV cache and batching, and to work out which fix is worth making.

Every speed change looks like an accuracy risk

A configuration change rewrites almost every answer, even when quality is unchanged. Without a way to prove quality held, teams stay on the safe default.

Benchmarks lie by accident

The engine sizes its KV cache from free GPU memory at start-up, and that moves between runs. Without controls at the same cache size, luck reads as progress.

Engine tuning needs deep, specific knowledge

Defaults can be wrong in ways only the source code reveals, and the best settings change with every model and every hardware configuration.

The floorplan

Seven places Horus looks for lost throughput

GPU kernels and communication

Attention, matrix-multiply and MoE kernels, plus GPU-to-GPU collectives. Agents write and tune replacements at the exact shapes your workload uses.

Speculative decoding

Draft depth, draft precision and acceptance, priced against the cost of each decode step.

KV cache and memory

How much cache the engine gets, and what owns the rest of each GPU’s memory.

Engine scheduling and batching

Running batch, waiting queue and the prefill work that interrupts decode.

Parallelism strategy

How one model’s work is split across the GPUs in a node.

Host CPU, Python GIL and garbage collection

CPU work and pauses that leave GPUs waiting.

Compilation and graph capture

Overhead from compiling the model and capturing its GPU graphs.

  1. Step 1

    Trace

    One profiling run of your real workload, on the engine you already use, from engine flags down to individual GPU kernels.

  2. Step 2

    Diagnose

    Find where performance is lost and explain why, whether that is a flag, the parallel layout, memory, communication or a kernel.

  3. Step 3

    Fix

    Specialised agents each take one part of the system and propose changes: configuration, engine patches and rewritten kernels.

  4. Step 4

    Verify

    Every proposal runs on a clean server beside control runs. Only changes that are faster and pass accuracy checks are kept.

  5. Step 5

    Deliver

    You keep the code and the configurations. Horus can run fully on-premises, so your data never has to leave.

Who it’s for

Anyone whose returns run through a GPU bill

GPU clouds and data-center operators
Revenue per rack depends on tokens served, not chips installed.
Inference API providers
You compete on price per million tokens.
AI-native companies
Gross margin is mostly the compute bill.
The funds that back all three
You are underwriting those margins today.

Five questions for any team running inference

  1. How many tokens per GPU per second do you serve on your main model, steady state and over a whole run?
  2. Is that measured against the vendor’s recipe, the engine’s published recipe, or your own tuned baseline?
  3. When you change a setting, how do you prove the answers did not get worse?
  4. Does that throughput still meet your per-user speed and time-to-first-token targets?
  5. Are your benchmark comparisons run beside controls, at the same cache size, more than once?

Horus is built to put a measured number on every one of them.

The first engagement

Bring one model. We’ll find the bottleneck.

A scoped pilot covers one model, one supported deployment and one representative workload, with the baseline, quality checks and experiment budget agreed up front.

What your team gets

  • An evidence bundle
  • Ranked opportunities
  • Tested changes
  • The measured result, including what didn’t work

Talk to us

horus.io.gpu@gmail.com