RFC9113 and RFC9114 both require requests with connection-specific
headers to be treated as malformed, with the exception of "te: trailers".
Reject requests containing them.
RFC9110 is clear that the only CTRL character allowed in header values
is HTAB. Conform to the standard, as Varnish, H2O, and (I suspect)
Hyper do. This also makes the whitespace-stripping code simpler, as any
character that is less than 0x21 is either whitespace or rejected.
HTTP headers must be an RFC9110 token, so only a subset of characters
are permitted. RFC9113 and RFC9114 require rejecting invalid header
characters in HTTP/2 and HTTP/3 respectively, so reject them in HTTP/1.0
and HTTP/1.1 for consistency. This also requires removing the ignore
hack for (presumably ancient) versions of IIS.
Per RFC9110, HTTP field values never contain leading or trailing
whitespace. Strip all such whitespace from HTTP and HTTP field values.
The HTTP/1.x parser already stripped spaces but didn't strip tabs, so
change the parser to strip tabs as well. In HTTP/2+, the stripping is
done during validation. This requires modifying the value.
There are three ways to modify the value:
1. Modify the data in-place with memmove().
2. Move the data pointer to point to after the leading whitespace.
3. Allocate a new buffer and replace the data pointer.
Both HPACK and QPACK decompression make a copy of the data, but some
code might assume that the data pointer of a field value can safely be
passed to ngx_pfree(). Therefore, the first option is chosen. Existing
code ensures that header values are NUL-terminated, so the stripping
code NUL-pads header values to ensure that the stripped strings have at
least as many terminating NUL bytes as they did before being stripped.
The stripping code has been tested in a standalone program to make sure
that it works correctly, and it correctly strips leading and trailing
whitespace from a variety of strings. This code has also been tested
with real HTTP/3 requests from Cloudflare's h3i tool.
Fixes: #598
The header validation required by HTTP/2 and HTTP/3 is identical, so use
a common function for both. This will make it easier to add additional
validation in the future. Move the function to ngx_http_parse.c so that
it can share code with the HTTP/1.x parser in the future.
No functional change intended.
Neither r->port_start nor r->port_end were ever used.
The r->port_end is set by the parser, though it was never used by
the following code (and was never usable, since not copied by the
ngx_http_alloc_large_header_buffer() without r->port_start set).
Multi headers are now using linked lists instead of arrays. Notably,
the following fields were changed: r->headers_in.cookies (renamed
to r->headers_in.cookie), r->headers_in.x_forwarded_for,
r->headers_out.cache_control, r->headers_out.link, u->headers_in.cache_control
u->headers_in.cookies (renamed to u->headers_in.set_cookie).
The r->headers_in.cookies and u->headers_in.cookies fields were renamed
to r->headers_in.cookie and u->headers_in.set_cookie to match header names.
The ngx_http_parse_multi_header_lines() and ngx_http_parse_set_cookie_lines()
functions were changed accordingly.
With this change, multi headers are now essentially equivalent to normal
headers, and following changes will further make them equivalent.
In 71edd9192f24 logging of invalid headers which were rejected with the
NGX_HTTP_PARSE_INVALID_HEADER error was restricted to just the "client
sent invalid header line" message, without any attempts to log the header
itself.
This patch returns logging of the header up to the invalid character and
the character itself. The r->header_end pointer is now properly set
in all cases to make logging possible.
The same logging is also introduced when parsing headers from upstream
servers.
Control characters (0x00-0x1f, 0x7f), space, and colon were never allowed in
header names. The only somewhat valid use is header continuation which nginx
never supported and which is explicitly obsolete by RFC 7230.
Previously, such headers were considered invalid and were ignored by default
(as per ignore_invalid_headers directive). With this change, such headers
are unconditionally rejected.
It is expected to make nginx more resilient to various attacks, in particular,
with ignore_invalid_headers switched off (which is inherently unsecure, though
nevertheless sometimes used in the wild).
Control characters (0x00-0x1f, 0x7f) were never allowed in URIs, and must
be percent-encoded by clients. Further, these are not believed to appear
in practice. On the other hand, passing such characters might make various
attacks possible or easier, despite the fact that currently allowed control
characters are not significant for HTTP request parsing.
From now on, requests with spaces in URIs are immediately rejected rather
than allowed. Spaces were allowed in 31e9677b15a1 (0.8.41) to handle bad
clients. It is believed that now this behaviour causes more harm than
good.
No valid CONNECT requests are expected to appear within nginx, since it
is not a forward proxy. Further, request line parsing will reject
proper CONNECT requests anyway, since we don't allow authority-form of
request-target. On the other hand, RFC 7230 specifies separate message
length rules for CONNECT which we don't support, so make sure to always
reject CONNECTs to avoid potential abuse.
When the request line contains request-target in the absolute-URI form,
it can contain path-empty instead of a single slash (see RFC 7230, RFC 3986).
Previously, the ngx_http_parse_request_line() function only accepted empty
path when there was no query string.
With this change, non-empty query is also correctly handled. That is,
request line "GET http://example.com?foo HTTP/1.1" is accepted and results
in $uri "/" and $args "foo".
Note that $request_uri remains "?foo", similarly to how spaces in URIs
are handled. Providing "/?foo", similarly to how "/" is provided for
"GET http://example.com HTTP/1.1", requires allocation.
As defined in HTTP/1.1, body chunks have the following ABNF:
chunk = chunk-size [ chunk-ext ] CRLF chunk-data CRLF
where chunk-data is a sequence of chunk-size octets.
With this change, chunk-data that doesn't end up with CRLF at chunk-size
offset will be treated as invalid, such as in the example provided below:
4
SEE-THIS-AND-
4
THAT
0
It is used at least by SOAP (M-POST method, defined by RFC 2774) and
by WebDAV versioning (VERSION-CONTROL and BASELINE-CONTROL methods,
defined by RFC 3253).
Both minor and major versions are now limited to 999 maximum. In case of
r->http_minor, this limit is already implied by the code. Major version,
r->http_major, in theory can be up to 65535 with current code, but such
values are very unlikely to become real (and, additionally, such values
are not allowed by RFC 7230), so the same test was used for r->http_major.
Minimal data length we expect for further calls was calculated incorrectly
if parsing stopped right after parsing chunk size. This might in theory
affect clients and/or backends using LF instead of CRLF.
Patch by Dmitry Popov.
Windows treats "/directory./" identical to "/directory/". Do the same
when working on Windows. Note that the behaviour is different from one
with last path component (where multiple spaces and dots are ignored by
Windows).
Additional parsing logic added to correctly handle RFC 3986 compliant IPv6 and
IPvFuture characters enclosed in square brackets.
The host validation was completely rewritten. The behavior for non IP literals
was changed in a more proper and safer way:
- Host part is now delimited either by the first colon or by the end of string
if there's no colon. Previously the last colon was used as delimiter which
allowed substitution of a port number in the $host variable.
(e.g. Host: 127.0.0.1:9000:80)
- Fixed stripping of the ending dot in the Host header when the host was also
followed by a port number.
(e.g. Host: nginx.com.:80)
- Fixed upper case characters detection. Previously it was broken which led to
wasting memory and CPU.
