Security Researcher. HTTP Smuggling, Web Security, CTF.

7 minute read

We have a declarative query language for almost every artifact a defender looks at, except the one this blog spends most of its time on. YARA matches on files. Sigma matches on log events. Suricata and Snort match on packets and reassembled streams. Zeek turns traffic into typed events you can write scripts against. Wireshark’s display filters let you ask precise questions of a capture after the fact. Every one of these is mature, widely deployed, and useful.

None of them can express the thing I actually want to ask, which is whether two HTTP parsers disagree about where a request ends.

That is the whole class. Request smuggling is a front-end and a back-end disagreeing about message boundaries. Web cache deception is a cache and an origin disagreeing about where the meaningful path ends. Both are parser differentials, and the primitive you want to detect is not a byte pattern. It is a disagreement. I keep running into the fact that the tools I would reach for cannot represent a disagreement, because each of them runs exactly one parser.

What one parser can and cannot say

Take the canonical CL.TE case. A request carries both Content-Length and Transfer-Encoding: chunked, the front-end honours one, the back-end honours the other, and the tail of the body becomes the head of the next request in the pipeline.

A Suricata rule can match “a request contains both a Content-Length header and a Transfer-Encoding header.” That is a real signal and people write that rule. But it is a proxy for the actual condition, and it is a bad proxy in both directions. It fires on requests that are perfectly safe because both ends agree to strip Transfer-Encoding per RFC 9112 Section 6.1. It misses the requests that matter, where the ambiguity is not two headers present but one header made invisible to one parser: a bare \n before Transfer-Encoding, a space in Content-Length : 5, an obsolete line fold, a smuggled second Content-Length inside a value. James Kettle’s original 2019 work and everything since is a catalogue of ways to make one parser see a header that the other one does not. The presence of the header string is not the bug. The bug is that two implementations read the same bytes and produce different messages.

To detect that, you would have to run both implementations and compare. No single-parser matcher can do it, no matter how clever the regular expression, because the property is not a property of the bytes. It is a property of the relationship between two parse results over the same bytes.

The atoms of a language that could

If I were designing the language, the primitive type would not be a byte string. It would be a parse result. Concretely, the atoms are:

  1. A raw octet sequence. The bytes on the wire, unmodified, including the framing characters that every existing tool normalises away before you get to see them. \r, \n, and their absence are data here, not formatting.
  2. A named parser applied to those bytes, producing a structured message: a method, a target, an ordered list of header fields with their exact original spelling, and a body with an explicit length and an explicit “how the length was determined” tag.
  3. A comparison across two parse results of the same octet sequence.

With those three, the smuggling condition writes itself. In pseudo-syntax:

rule cl_te_desync {
    input: raw_request R

    parse P_front = parser("nginx-like", R)
    parse P_back  = parser("gunicorn-like", R)

    condition:
        P_front.body_length_source == "content-length"
        and P_back.body_length_source == "transfer-encoding"
        and P_front.body_end_offset != P_back.body_end_offset
}

The condition is the literal definition of the vulnerability: the two ends decided the body ends in two different places. Not “both headers are present.” The bytes that trigger this are exactly the bytes that are dangerous, and the bytes that are safe do not trigger it, because a safe request produces one agreed boundary and the offsets match. Every obfuscation trick that hides a header from one parser is caught for free, because the language never looks at the header spelling. It looks at what each parser did with it.

The same shape covers the cache case. Two parsers, one is the cache’s path-to-cache-key function and one is the origin’s route matcher, and the condition is that they split the path at different offsets while the response carries a Set-Cookie. Same structure, different pair of parsers, different oracle.

Why nobody has built it

The design is not the hard part. The reason this does not exist is the same reason the bugs exist, which I find genuinely funny in a way that does not help me.

The value of the language comes entirely from the parsers being real. A rule is only true if parser("nginx-like") actually parses the way a given nginx version parses, down to the handling of a bare \n and an over-long chunk extension and a duplicated header. A model of nginx is worthless here; you need nginx’s behaviour, at the version the target runs, including its bugs, because its bugs are the whole point. So a serious implementation is a harness that embeds a collection of real HTTP parsers, or faithful reimplementations tracked against them, and keeps them current as they change. That is a heavy, permanent maintenance burden, and it is heaviest exactly where the language is most useful, which is the long tail of proxies and origins nobody has modelled.

There is a second reason, quieter. The existing tools live at fixed points in the stack where you get one view of the traffic. An inline IDS sees the stream as its own TCP reassembler framed it, which is already one parser’s opinion, and it cannot retroactively see the bytes as the back-end will frame them because the back-end is downstream and has not run yet. A log-based tool like Sigma sees the request only after some server already parsed it into fields, so the disagreement has been collapsed before the event was written. The place where you can see the raw octets and apply two parsers of your choice is a research harness with the full byte stream in hand, the kind of setup built around Kettle’s desync research and the manual work that surrounds it, not a deployed detection sensor. The language wants a vantage point that production monitoring does not have.

What it is actually good for

I do not think the answer is a new inline sensor. The vantage-point problem makes that a poor fit, and I would rather be honest about that than pretend a design essay is a product.

Where a language like this would earn its cost is offline, over corpora, in two jobs the current tooling does badly. The first is regression testing your own stack: take the raw requests from a smuggling test suite, run your real front-end and back-end parsers over each, and assert that their boundaries agree. That is a CI check for “did a proxy upgrade just reintroduce a desync,” and it is expressible in this language and awkward in any existing one. The second is triage of research corpora. When you have thousands of captured request variants from fuzzing a proxy pair, the question “which of these produce a boundary disagreement” is the entire result, and today it is answered with one-off scripts that rebuild the same two-parser comparison every time.

Both of those are the parser-differential idea pointed at defence instead of offence, which is a direction I think the smuggling research has been slow to take. We have spent seven years getting very good at finding the bytes that make two parsers disagree. We have spent almost none of that effort on a shared vocabulary for asserting that two parsers agree, which is the thing a defender actually wants to be able to state and check.

I have not built this, and this post is a design sketch rather than a release. The two hard parts, embedding faithful parsers and finding a vantage point with the raw octets, are real and I do not want to wave them away. But the atoms are right, I am fairly sure of that: the primitive type has to be a parse result, and the operator that matters is comparison across two of them over the same bytes. Any tool that keeps treating an HTTP request as a byte string to pattern-match will keep being unable to say the one thing about it that a smuggling researcher most wants to say.