Commit Graph

5295 Commits

Author SHA1 Message Date
Maxim Dounin
05029e775f SSL: restore handlers after blocking.
It is possible that after SSL_read() will return SSL_ERROR_WANT_WRITE,
further calls will return SSL_ERROR_WANT_READ without reading any
application data.  We have to call ngx_handle_write_event() and
switch back to normal write handling much like we do if there are some
application data, or the write there will be reported again and again.

Similarly, we have to switch back to normal read handling if there
is saved read handler and SSL_write() returns SSL_ERROR_WANT_WRITE.
2018-09-10 18:57:39 +03:00
Maxim Dounin
c2f90de0c5 SSL: corrected SSL_ERROR_WANT_WRITE / SSL_ERROR_WANT_READ logging.
While SSL_read() most likely to return SSL_ERROR_WANT_WRITE (and SSL_write()
accordingly SSL_ERROR_WANT_READ) during an SSL renegotiation, it is
not necessary mean that a renegotiation was started.  In particular,
it can never happen during a renegotiation or can happen multiple times
during a renegotiation.

Because of the above, misleading "peer started SSL renegotiation" info
messages were replaced with "SSL_read: want write" and "SSL_write: want read"
debug ones.

Additionally, "SSL write handler" and "SSL read handler" are now logged
by the SSL write and read handlers, to make it easier to understand that
temporary SSL handlers are called instead of normal handlers.
2018-09-10 18:57:19 +03:00
Maxim Dounin
278be041dd Lingering close changed to handle NGX_AGAIN.
The "do { c->recv() } while (c->read->ready)" form used in the
ngx_http_lingering_close_handler() is not really correct, as for
example with SSL c->read->ready may be still set when returning NGX_AGAIN
due to SSL_ERROR_WANT_WRITE.  Therefore the above might be an infinite loop.

This doesn't really matter in lingering close, as we shutdown write side
of the socket anyway and also disable renegotiation (and even without shutdown
and with renegotiation it requires using very large certificate chain and
tuning socket buffers to trigger SSL_ERROR_WANT_WRITE).  But for the sake of
correctness added an NGX_AGAIN check.
2018-09-10 18:57:13 +03:00
Maxim Dounin
07dbfd00ff gRPC: disabled keepalive when sending control frames was blocked.
If sending request body was not completed (u->request_body_sent is not set),
the upstream keepalive module won't save such a connection.  However, it
is theoretically possible (though highly unlikely) that sending of some
control frames can be blocked after the request body was sent.  The
ctx->output_blocked flag introduced to disable keepalive in such cases.
2018-09-03 19:34:02 +03:00
Maxim Dounin
71e152debe gRPC: improved keepalive handling.
The code is now able to parse additional control frames after
the response is received, and can send control frames as well.
This fixes keepalive problems as observed with grpc-c, which can
send window update and ping frames after the response, see
http://mailman.nginx.org/pipermail/nginx/2018-August/056620.html.
2018-09-03 19:34:01 +03:00
Maxim Dounin
b71792b110 Uwsgi: added a check on maximum uwsgi request size.
Requested by Chris Caputo.
2018-09-03 19:17:06 +03:00
Maxim Dounin
07c3e79b64 Uwsgi: style. 2018-09-03 19:17:01 +03:00
Roman Arutyunyan
09cb553c7f Version bump. 2018-08-30 14:42:15 +03:00
Roman Arutyunyan
d9908c6c9a Stream: avoid potential infinite loop at preread phase.
Previously the preread phase code ignored NGX_AGAIN value returned from
c->recv() and relied only on c->read->ready.  But this flag is not reliable and
should only be checked for optimization purposes.  For example, when using
SSL, c->read->ready may be set when no input is available.  This can lead to
calling preread handler infinitely in a loop.
2018-08-29 15:56:42 +03:00
Vladimir Homutov
62821f1c99 Upstream: fixed request chain traversal (ticket #1618).
The problem does not manifest itself currently, because in case of
non-buffered reading, chain link created by u->create_request method
consists of a single element.

Found by PVS-Studio.
2018-08-24 12:19:37 +03:00
Maxim Dounin
d817ceae72 Upstream keepalive: keepalive_requests directive.
The directive configures maximum number of requests allowed on
a connection kept in the cache.  Once a connection reaches the number
of requests configured, it is no longer saved to the cache.
The default is 100.

Much like keepalive_requests for client connections, this is mostly
a safeguard to make sure connections are closed periodically and the
memory allocated from the connection pool is freed.
2018-08-10 21:54:46 +03:00
Maxim Dounin
7de808990b Upstream keepalive: keepalive_timeout directive.
The directive configures maximum time a connection can be kept in the
cache.  By configuring a time which is smaller than the corresponding
timeout on the backend side one can avoid the race between closing
a connection by the backend and nginx trying to use the same connection
to send a request at the same time.
2018-08-10 21:54:23 +03:00
Maxim Dounin
f3d1a925b5 Upstream keepalive: comment added. 2018-08-10 21:54:17 +03:00
Maxim Dounin
1b1b632eed SSL: fixed build with LibreSSL 2.8.0 (ticket #1605).
LibreSSL 2.8.0 "added const annotations to many existing APIs from OpenSSL,
making interoperability easier for downstream applications".  This includes
the const change in the SSL_CTX_sess_set_get_cb() callback function (see
9dd43f4ef67e), which breaks compilation.

To fix this, added a condition on how we redefine OPENSSL_VERSION_NUMBER
when working with LibreSSL (see 382fc7069e3a).  With LibreSSL 2.8.0,
we now set OPENSSL_VERSION_NUMBER to 0x1010000fL (OpenSSL 1.1.0), so the
appropriate conditions in the code will use "const" as it happens with
OpenSSL 1.1.0 and later versions.
2018-08-10 20:49:06 +03:00
Maxim Dounin
0d224602e9 HTTP/2: workaround for clients which fail on table size updates.
There are clients which cannot handle HPACK's dynamic table size updates
as added in 12cadc4669a7 (1.13.6).  Notably, old versions of OkHttp library
are known to fail on it (ticket #1397).

This change makes it possible to work with such clients by only sending
dynamic table size updates in response to SETTINGS_HEADER_TABLE_SIZE.  As
a downside, clients which do not use SETTINGS_HEADER_TABLE_SIZE will
continue to maintain default 4k table.
2018-08-09 20:12:17 +03:00
Maxim Dounin
dd8e1e178c Skipping spaces in configuration files (ticket #1557).
Previously, a chunk of spaces larger than NGX_CONF_BUFFER (4096 bytes)
resulted in the "too long parameter" error during parsing such a
configuration.  This was because the code only set start and start_line
on non-whitespace characters, and hence adjacent whitespace characters
were preserved when reading additional data from the configuration file.
Fix is to always move start and start_line if the last character was
a space.
2018-08-09 12:15:42 +03:00
Maxim Dounin
3b1589173f SSL: support for TLSv1.3 early data with BoringSSL.
Early data AKA 0-RTT mode is enabled as long as "ssl_early_data on" is
specified in the configuration (default is off).

The $ssl_early_data variable evaluates to "1" if the SSL handshake
isn't yet completed, and can be used to set the Early-Data header as
per draft-ietf-httpbis-replay-04.
2018-08-07 02:16:07 +03:00
Maxim Dounin
9f30fda1c2 SSL: enabled TLSv1.3 with BoringSSL.
BoringSSL currently requires SSL_CTX_set_max_proto_version(TLS1_3_VERSION)
to be able to enable TLS 1.3.  This is because by default max protocol
version is set to TLS 1.2, and the SSL_OP_NO_* options are merely used
as a blacklist within the version range specified using the
SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version()
functions.

With this change, we now call SSL_CTX_set_max_proto_version() with an
explicit maximum version set.  This enables TLS 1.3 with BoringSSL.
As a side effect, this change also limits maximum protocol version to
the newest protocol we know about, TLS 1.3.  This seems to be a good
change, as enabling unknown protocols might have unexpected results.

Additionally, we now explicitly call SSL_CTX_set_min_proto_version()
with 0.  This is expected to help with Debian system-wide default
of MinProtocol set to TLSv1.2, see
http://mailman.nginx.org/pipermail/nginx-ru/2017-October/060411.html.

Note that there is no SSL_CTX_set_min_proto_version macro in BoringSSL,
so we call SSL_CTX_set_min_proto_version() and SSL_CTX_set_max_proto_version()
as long as the TLS1_3_VERSION macro is defined.
2018-08-07 02:15:28 +03:00
Sergey Kandaurov
a6baf5e508 Dav: removed dead store after 8e7a5de61664.
Found by Clang Static Analyzer.
2018-08-02 13:19:48 +03:00
Maxim Dounin
04a6d9d3c9 Dav: changed COPY of a file to preserve access mask.
The behaviour is now in line with COPY of a directory with contents,
which preserves access masks on individual files, as well as the "cp"
command.

Requested by Roman Arutyunyan.
2018-08-01 02:12:21 +03:00
Maxim Dounin
9e0d1236f1 Dav: changed ngx_copy_file() to preserve access and mtime.
This fixes wrong permissions and file time after cross-device MOVE
in the DAV module (ticket #1577).  Broken in 8101d9101ed8 (0.8.9) when
cross-device copying was introduced in ngx_ext_rename_file().

With this change, ngx_copy_file() always calls ngx_set_file_time(),
either with the time provided, or with the time from the original file.
This is considered acceptable given that copying the file is costly anyway,
and optimizing cases when we do not need to preserve time will require
interface changes.
2018-08-01 02:12:11 +03:00
Maxim Dounin
6bff9c9967 Dav: fixed ngx_copy_file() to truncate destination file.
Previously, ngx_open_file(NGX_FILE_CREATE_OR_OPEN) was used, resulting
in destination file being partially rewritten if exists.  Notably,
this affected WebDAV COPY command (ticket #1576).
2018-08-01 02:11:58 +03:00
Sergey Kandaurov
a1e2c5566c Version bump. 2018-07-24 18:46:18 +03:00
Maxim Dounin
5a19c034f3 Fixed NGX_TID_T_FMT format specification for uint64_t.
Previously, "%uA" was used, which corresponds to ngx_atomic_uint_t.
Size of ngx_atomic_uint_t can be easily different from uint64_t,
leading to undefined results.
2018-07-22 04:03:40 +03:00
Sergey Kandaurov
b93931ae82 Stream ssl_preread: added SSLv2 Client Hello support.
In particular, it was not possible to obtain SSLv2 protocol version.
2018-07-18 18:51:25 +03:00
Ruslan Ermilov
372b624627 Fixed invalid access to location defined as an empty string. 2018-07-17 15:30:43 +03:00
Sergey Kandaurov
d5a27006e0 SSL: save sessions for upstream peers using a callback function.
In TLSv1.3, NewSessionTicket messages arrive after the handshake and
can come at any time.  Therefore we use a callback to save the session
when we know about it.  This approach works for < TLSv1.3 as well.
The callback function is set once per location on merge phase.

Since SSL_get_session() in BoringSSL returns an unresumable session for
TLSv1.3, peer save_session() methods have been updated as well to use a
session supplied within the callback.  To preserve API, the session is
cached in c->ssl->session.  It is preferably accessed in save_session()
methods by ngx_ssl_get_session() and ngx_ssl_get0_session() wrappers.
2018-07-17 12:53:23 +03:00
Maxim Dounin
e1bebd05cb SSL: use of the SSL_OP_NO_RENEGOTIATION option (ticket #1376).
The SSL_OP_NO_RENEGOTIATION option is available in OpenSSL 1.1.0h+ and can
save some CPU cycles on renegotiation attempts.
2018-07-16 17:47:48 +03:00
Maxim Dounin
1456129902 SSL: fixed SSL_clear_options() usage with OpenSSL 1.1.0+.
In OpenSSL 1.1.0 the SSL_CTRL_CLEAR_OPTIONS macro was removed, so
conditional compilation test on it results in SSL_clear_options()
and SSL_CTX_clear_options() not being used.  Notably, this caused
"ssl_prefer_server_ciphers off" to not work in SNI-based virtual
servers if server preference was switched on in the default server.

It looks like the only possible fix is to test OPENSSL_VERSION_NUMBER
explicitly.
2018-07-16 17:47:20 +03:00
Maxim Dounin
b1734fd800 SSL: logging levels of "unsupported protocol", "version too low".
Starting with OpenSSL 1.1.0, SSL_R_UNSUPPORTED_PROTOCOL instead of
SSL_R_UNKNOWN_PROTOCOL is reported when a protocol is disabled via
an SSL_OP_NO_* option.

Additionally, SSL_R_VERSION_TOO_LOW is reported when using MinProtocol
or when seclevel checks (as set by @SECLEVEL=n in the cipher string)
rejects a protocol, and this is what happens with SSLv3 and @SECLEVEL=1,
which is the default.

There is also the SSL_R_VERSION_TOO_HIGH error code, but it looks like
it is not possible to trigger it.
2018-07-16 17:47:18 +03:00
Maxim Dounin
85b44b46fb Events: added configuration check on the number of connections.
There should be at least one worker connection for each listening socket,
plus an additional connection for channel between worker and master,
or starting worker processes will fail.
2018-07-12 19:50:07 +03:00
Maxim Dounin
751bdd3bb2 Events: moved sockets cloning to ngx_event_init_conf().
Previously, listenings sockets were not cloned if the worker_processes
directive was specified after "listen ... reuseport".

This also simplifies upcoming configuration check on the number
of worker connections, as it needs to know the number of listening
sockets before cloning.
2018-07-12 19:50:02 +03:00
Roman Arutyunyan
a8e38e2a9c Stream ssl_preread: $ssl_preread_protocol variable.
The variable keeps the latest SSL protocol version supported by the client.
The variable has the same format as $ssl_protocol.

The version is read from the client_version field of ClientHello.  If the
supported_versions extension is present in the ClientHello, then the version
is set to TLSv1.3.
2018-07-11 17:56:51 +03:00
Ruslan Ermilov
527cac2979 Allow resetting connections closed by "return 444" (ticket #905).
If reset_timedout_connection is on, TCP connections closed by
"return 444" will be reset instead of a normal close.
2018-07-12 12:50:20 +03:00
Maxim Dounin
f62d460d5b Resolver: retry sending queries on errors (ticket #1511).
Errors when sending UDP datagrams can happen, e.g., when local IP address
changes (see fa0e093b64d7), or an unavailable DNS server on the LAN can cause
send() to fail with EHOSTDOWN on BSD systems.  If this happens during
initial query, retry sending immediately, to a different DNS server when
possible.  If this is not enough, allow normal resend to happen by ignoring
the return code of the second ngx_resolver_send_query() call, much like we
do in ngx_resolver_resend().
2018-07-05 22:21:14 +03:00
Maxim Dounin
f206a112c6 SSL: logging level of "https proxy request" errors.
The "http request" and "https proxy request" errors cannot happen
with HTTP due to pre-handshake checks in ngx_http_ssl_handshake(),
but can happen when SSL is used in stream and mail modules.
2018-07-05 20:45:29 +03:00
Maxim Dounin
c62f9c914f Version bump. 2018-07-05 20:44:58 +03:00
Maxim Dounin
a7186c8f1c Upstream: fixed tcp_nopush with gRPC.
With gRPC it is possible that a request sending is blocked due to flow
control.  Moreover, further sending might be only allowed once the
backend sees all the data we've already sent.  With such a backend
it is required to clear the TCP_NOPUSH socket option to make sure all
the data we've sent are actually delivered to the backend.

As such, we now clear TCP_NOPUSH in ngx_http_upstream_send_request()
also on NGX_AGAIN if c->write->ready is set.  This fixes a test (which
waits for all the 64k bytes as per initial window before allowing more
bytes) with sendfile enabled when the body was written to a file
in a different context.
2018-07-02 19:03:04 +03:00
Maxim Dounin
1331a99f5c Upstream: fixed unexpected tcp_nopush usage on peer connections.
Now tcp_nopush on peer connections is disabled if it is disabled on
the client connection, similar to how we handle c->sendfile.  Previously,
tcp_nopush was always used on upstream connections, regardless of
the "tcp_nopush" directive.
2018-07-02 19:02:31 +03:00
Maxim Dounin
d842b4e5e0 gRPC: clearing buffers in ngx_http_grpc_get_buf().
We copy input buffers to our buffers, so various flags might be
unexpectedly set in buffers returned by ngx_chain_get_free_buf().

In particular, the b->in_file flag might be set when the body was
written to a file in a different context.  With sendfile enabled this
in turn might result in protocol corruption if such a buffer was reused
for a control frame.

Make sure to clear buffers and set only fields we really need to be set.
2018-07-02 19:02:08 +03:00
Ruslan Ermilov
468e37734c Added FreeBSD support for "listen ... reuseport". 2018-07-02 13:54:33 +03:00
Vladimir Homutov
0c4ccbea23 Upstream: ngx_http_upstream_random module.
The module implements random load-balancing algorithm with optional second
choice.  In the latter case, the best of two servers is chosen, accounting
number of connections and server weight.

Example:

upstream u {
    random [two [least_conn]];

    server 127.0.0.1:8080;
    server 127.0.0.1:8081;
    server 127.0.0.1:8082;
    server 127.0.0.1:8083;
}
2018-06-15 11:46:14 +03:00
Ruslan Ermilov
2eab9efbe4 Upstream: improved peer selection concurrency for hash and ip_hash. 2018-06-14 07:03:50 +03:00
Ruslan Ermilov
63e8a1d926 Resolver: require name servers. 2018-06-13 22:37:49 +03:00
Ruslan Ermilov
4542898917 Resolver: allocate resolver from configuration pool.
Before 4a8c9139e579, ngx_resolver_create() didn't use configuration
pool, and allocations were done using malloc().

In 016352c19049, when resolver gained support of several servers,
new allocations were done from the pool.
2018-06-13 22:37:42 +03:00
Maxim Dounin
9b6bc8a578 Upstream: disable body cleanup with preserve_output (ticket #1565).
With u->conf->preserve_output set the request body file might be used
after the response header is sent, so avoid cleaning it.  (Normally
this is not a problem as u->conf->preserve_output is only set with
r->request_body_no_buffering, but the request body might be already
written to a file in a different context.)
2018-06-13 15:28:11 +03:00
Ruslan Ermilov
fb3a9e28b2 HTTP/2: use scheme from original request for pushes (closes #1549).
Instead of the connection scheme, use scheme from the original request.
This fixes pushes when SSL is terminated by a proxy server in front of
nginx.
2018-06-07 20:04:22 +03:00
Ruslan Ermilov
f11a9cbdd0 Added r->schema.
For HTTP/1, it keeps scheme from the absolute form of URI.
For HTTP/2, the :scheme request pseudo-header field value.
2018-06-07 20:01:41 +03:00
Ruslan Ermilov
70b6e7a299 Removed extraneous check while processing request line. 2018-06-07 19:53:43 +03:00
Ruslan Ermilov
94a2ce426f HTTP/2: validate client request scheme.
The scheme is validated as per RFC 3986, Section 3.1.
2018-06-07 11:47:10 +03:00
Ruslan Ermilov
c804eb7748 Allowed digits, '+', '-', and '.' in scheme names as per RFC 3986. 2018-05-24 12:06:35 +03:00
Sergey Kandaurov
1ef7b1ef61 SSL: removed extra prototype. 2018-06-06 13:31:05 +03:00
Maxim Dounin
8dc0f75d0b Added missing space after ngx_close_socket_n. 2018-06-05 17:41:34 +03:00
Maxim Dounin
82b0fe21d6 Version bump. 2018-06-05 17:13:17 +03:00
Roman Arutyunyan
96b6f215b8 Stream: udp streams.
Previously, only one client packet could be processed in a udp stream session
even though multiple response packets were supported.  Now multiple packets
coming from the same client address and port are delivered to the same stream
session.

If it's required to maintain a single stream of data, nginx should be
configured in a way that all packets from a client are delivered to the same
worker.  On Linux and DragonFly BSD the "reuseport" parameter should be
specified for this.  Other systems do not currently provide appropriate
mechanisms.  For these systems a single stream of udp packets is only
guaranteed in single-worker configurations.

The proxy_response directive now specifies how many packets are expected in
response to a single client packet.
2018-06-04 19:50:00 +03:00
Roman Arutyunyan
1028d71695 Events: moved ngx_recvmsg() to new file src/event/ngx_event_udp.c. 2018-06-01 16:55:49 +03:00
Roman Arutyunyan
20f8bfab34 Events: get remote addresses before creating udp connection.
Previously, ngx_event_recvmsg() got remote socket addresses after creating
the connection object.  In preparation to handling multiple UDP packets in a
single session, this code was moved up.
2018-06-01 13:12:57 +03:00
Roman Arutyunyan
26a57486f0 Events: fixed handling zero-length client address.
On Linux recvmsg() syscall may return a zero-length client address when
receiving a datagram from an unbound unix datagram socket.  It is usually
assumed that socket address has at least the sa_family member.  Zero-length
socket address caused buffer over-read in functions which receive socket
address, for example ngx_sock_ntop().  Typically the over-read resulted in
unexpected socket family followed by session close.  Now a fake socket address
is allocated instead of a zero-length client address.
2018-06-01 16:53:02 +03:00
Sergey Kandaurov
dac90a4bff Leave chain in ngx_chain_add_copy() in consistent state on errors. 2018-06-04 18:47:54 +03:00
Maxim Dounin
7b9bc5d4d1 Limit req: improved handling of negative times.
Negative times can appear since workers only update time on an event
loop iteration start.  If a worker was blocked for a long time during
an event loop iteration, it is possible that another worker already
updated the time stored in the node.  As such, time since last update
of the node (ms) will be negative.

Previous code used ngx_abs(ms) in the calculations.  That is, negative
times were effectively treated as positive ones.  As a result, it was
not possible to maintain high request rates, where the same node can be
updated multiple times from during an event loop iteration.
In particular, this affected setups with many SSL handshakes, see
http://mailman.nginx.org/pipermail/nginx/2018-May/056291.html.

Fix is to only update the last update time stored in the node if the
new time is larger than previously stored one.  If a future time is
stored in the node, we preserve this time as is.

To prevent breaking things on platforms without monotonic time available
if system time is updated backwards, a safety limit of 60 seconds is
used.  If the time stored in the node is more than 60 seconds in the future,
we assume that the time was changed backwards and update lr->last
to the current time.
2018-05-30 15:40:34 +03:00
Ruslan Ermilov
e4d173cc88 Core: fixed comment about ngx_current_msec after 81fae70d6cb8.
The value is no longer guaranteed to be based on milliseconds
elapsed since Epoch.
2018-05-29 16:15:19 +03:00
Ruslan Ermilov
4271e48911 Win32: fixed comment in ngx_gettimeofday() calculations. 2018-05-29 11:47:32 +03:00
Maxim Dounin
1ef5c9ce6f Removed glibc crypt_r() bug workaround (ticket #1469).
The bug in question was fixed in glibc 2.3.2 and is no longer expected
to manifest itself on real servers.  On the other hand, the workaround
causes compilation problems on various systems.  Previously, we've
already fixed the code to compile with musl libc (fd6fd02f6a4d), and
now it is broken on Fedora 28 where glibc's crypt library was replaced
by libxcrypt.  So the workaround was removed.
2018-05-23 16:38:16 +03:00
Vladimir Homutov
5568a6598d Syslog: install cleanup handler only once.
If a socket was re-opened due to an error (02c2352d5b01 and fa0e093b64d7),
additional cleanup handler was installed each time.
2018-05-14 22:50:57 +03:00
Ruslan Ermilov
4ac8036e78 Resolver: close UDP socket on error or incomplete send. 2018-05-23 10:41:38 +03:00
Ruslan Ermilov
4fca5bf1e1 Resolver: style. 2018-05-23 10:41:29 +03:00
Maxim Dounin
4f9d83d6d7 Core: silenced getsockopt(TCP_FASTOPEN) messages on FreeBSD.
FreeBSD returns EINVAL when getsockopt(TCP_FASTOPEN) is called on a unix
domain socket, resulting in "getsockopt(TCP_FASTOPEN) ... failed" messages
during binary upgrade when unix domain listen sockets are present in
the configuration.  Added EINVAL to the list of ignored error codes.
2018-05-21 23:11:27 +03:00
Vladimir Homutov
644d26e841 Syslog: re-open syslog udp socket on send error (ticket #1477).
Previously, only unix domain sockets were reopened to tolerate cases when
local syslog server was restarted.  It makes sense to treat other cases
(for example, local IP address changes) similarly.
2018-05-08 19:35:56 +03:00
Sergey Kandaurov
68b50f71e1 Silenced -Wcast-function-type warnings (closes #1546).
Cast to intermediate "void *" to lose compiler knowledge about the original
type and pass the warning.  This is not a real fix but rather a workaround.

Found by gcc8.
2018-05-07 09:54:37 +00:00
Ruslan Ermilov
658a84f425 SSL: deprecated the "ssl" directive. 2018-04-25 14:57:24 +03:00
Maxim Dounin
76be1ea9de SSL: detect "listen ... ssl" without certificates (ticket #178).
In mail and stream modules, no certificate provided is a fatal condition,
much like with the "ssl" and "starttls" directives.

In http, "listen ... ssl" can be used in a non-default server without
certificates as long as there is a certificate in the default one, so
missing certificate is only fatal for default servers.
2018-04-24 15:29:01 +03:00
Maxim Dounin
5d3a854ebd Mail: fixed error message about missing ssl_certificate_key.
In 51e1f047d15d, the "ssl" directive name was incorrectly hardcoded
in the error message shown when there are some SSL keys defined, but
not for all certificates.  Right approach is to use the "mode" variable,
which can be either "ssl" or "starttls".
2018-04-24 15:28:58 +03:00
Maxim Dounin
8b4c7725d2 Cache: fixed cache valid slot to reject incorrect statuses.
Previously, result of ngx_atoi() was assigned to an ngx_uint_t variable,
and errors reported by ngx_atoi() became positive, so the following check
in "status < 100" failed to catch them.  This resulted in the configurations
like "proxy_cache_valid 2xx 30s" being accepted as correct, while they
in fact do nothing.  Changing type to ngx_int_t fixes this, and such
configurations are now properly rejected.
2018-04-18 16:11:41 +03:00
Maxim Dounin
59d806b56f Version bump. 2018-04-18 16:09:08 +03:00
Maxim Dounin
f9e43a3154 Upstream: fixed u->conf->preserve_output (ticket #1519).
Previously, ngx_http_upstream_process_header() might be called after
we've finished reading response headers and switched to a different read
event handler, leading to errors with gRPC proxying.  Additionally,
the u->conf->read_timeout timer might be re-armed during reading response
headers (while this is expected to be a single timeout on reading
the whole response header).
2018-04-05 16:56:12 +03:00
Maxim Dounin
923c9d5f3b Version bump. 2018-04-05 16:53:27 +03:00
Maxim Dounin
d90a75b7cd Upstream: fixed ngx_http_upstream_test_next() conditions.
Previously, ngx_http_upstream_test_next() used an outdated condition on
whether it will be possible to switch to a different server or not.  It
did not take into account restrictions on non-idempotent requests, requests
with non-buffered request body, and the next upstream timeout.

For such requests, switching to the next upstream server was rejected
later in ngx_http_upstream_next(), resulting in nginx own error page
being returned instead of the original upstream response.
2018-04-03 02:43:18 +03:00
Vladimir Homutov
cadc8ca306 Core: fixed build, broken by 63e91f263a49.
Both Solaris and Windows define "s_addr" as a macro.
2018-04-02 20:38:43 +03:00
Ruslan Ermilov
55f08fc9f7 Core: revised the PROXY protocol v2 code.
- use normal prefixes for types and macros
- removed some macros and types
- revised debug messages
- removed useless check of ngx_sock_ntop() returning 0
- removed special processing of AF_UNSPEC
2018-04-02 18:40:04 +03:00
Vladimir Homutov
ecd6e243b6 Core: style. 2018-03-27 18:39:38 +03:00
Vladimir Homutov
9207cc84b2 Core: added processing of version 2 of the PROXY protocol.
The protocol used on inbound connection is auto-detected and corresponding
parser is used to extract passed addresses.  TLV parameters are ignored.

The maximum supported size of PROXY protocol header is 107 bytes
(similar to version 1).
2018-03-22 15:55:28 +03:00
Roman Arutyunyan
f39d5e8b33 Stream: set action before each recv/send while proxying.
Now it's clear from log error message if the error occurred on client or
upstream side.
2018-03-22 18:43:49 +03:00
Maxim Dounin
0ac57648eb gRPC: fixed possible sign extension of error and setting_value.
All cases are harmless and should not happen on valid values, though can
result in bad values being shown incorrectly in logs.

Found by Coverity (CID 1430311, 1430312, 1430313).
2018-03-22 19:26:25 +03:00
Ruslan Ermilov
8e8734ec82 Improved code readability (closes #1512). 2018-03-22 18:13:33 +03:00
Maxim Dounin
25a26bbce4 Version bump. 2018-03-22 15:55:52 +03:00
Sergey Kandaurov
e232421266 gRPC: fixed missing state save in frame header parsing.
Previously, frame state wasn't saved if HEADERS frame payload
that begins with header fragment was not received at once.
2018-03-20 15:58:11 +03:00
Ruslan Ermilov
74ea120f7d HTTP/2: improved frame info debugging. 2018-03-19 21:32:15 +03:00
Sergey Kandaurov
f3ad346952 gRPC: fixed parsing response headers split on CONTINUATION frames. 2018-03-19 16:42:56 +03:00
Ruslan Ermilov
c09aa142d6 Fixed checking ngx_tcp_push() and ngx_tcp_nopush() return values.
No functional changes.
2018-03-19 16:28:23 +03:00
Ruslan Ermilov
510986b806 Upstream: fixed comments after 13f8dec720b5.
The fields "uri", "location", and "url" from ngx_http_upstream_conf_t
moved to ngx_http_proxy_loc_conf_t and ngx_http_proxy_vars_t, reflect
this change in create_loc_conf comments.
2018-03-19 16:22:09 +03:00
Maxim Dounin
6559a42013 gRPC: special handling of "trailer only" responses.
The gRPC protocol makes a distinction between HEADERS frame with
the END_STREAM flag set, and a HEADERS frame followed by an empty
DATA frame with the END_STREAM flag.  The latter is not permitted,
and results in errors not being propagated through nginx.  Instead,
gRPC clients complain that "server closed the stream without sending
trailers" (seen in grpc-go) or "13: Received RST_STREAM with error
code 2" (seen in grpc-c).

To fix this, nginx now returns HEADERS with the END_STREAM flag if
the response length is known to be 0, and we are not expecting
any trailer headers to be added.  And the response length is
explicitly set to 0 in the gRPC proxy if we see initial HEADERS frame
with the END_STREAM flag set.
2018-03-17 23:04:26 +03:00
Maxim Dounin
6a0d9e5b2d gRPC: special handling of the TE request header.
According to the gRPC protocol specification, the "TE" header is used
to detect incompatible proxies, and at least grpc-c server rejects
requests without "TE: trailers".

To preserve the logic, we have to pass "TE: trailers" to the backend if
and only if the original request contains "trailers" in the "TE" header.
Note that no other TE values are allowed in HTTP/2, so we have to remove
anything else.
2018-03-17 23:04:25 +03:00
Maxim Dounin
56ad960e7a The gRPC proxy module.
The module allows passing requests to upstream gRPC servers.
The module is built by default as long as HTTP/2 support is compiled in.
Example configuration:

    grpc_pass 127.0.0.1:9000;

Alternatively, the "grpc://" scheme can be used:

    grpc_pass grpc://127.0.0.1:9000;

Keepalive support is available via the upstream keepalive module.  Note
that keepalive connections won't currently work with grpc-go as it fails
to handle SETTINGS_HEADER_TABLE_SIZE.

To use with SSL:

    grpc_pass grpcs://127.0.0.1:9000;

SSL connections use ALPN "h2" when available.  At least grpc-go works fine
without ALPN, so if ALPN is not available we just establish a connection
without it.

Tested with grpc-c++ and grpc-go.
2018-03-17 23:04:24 +03:00
Maxim Dounin
c67124190c Upstream: u->conf->preserve_output flag.
The flag can be used to continue sending request body even after we've
got a response from the backend.  In particular, this is needed for gRPC
proxying of bidirectional streaming RPCs, and also to send control frames
in other forms of RPCs.
2018-03-17 23:04:23 +03:00
Maxim Dounin
7be60194c1 Upstream: u->request_body_blocked flag.
The flag indicates whether last ngx_output_chain() returned NGX_AGAIN
or not.  If the flag is set, we arm the u->conf->send_timeout timer.
The flag complements c->write->ready test, and allows to stop sending
the request body in an output filter due to protocol-specific flow
control.
2018-03-17 23:04:22 +03:00
Maxim Dounin
7ef115a7e8 Upstream: trailers support, u->conf->pass_trailers flag.
Basic trailer headers support allows one to access response trailers
via the $upstream_trailer_* variables.

Additionally, the u->conf->pass_trailers flag was introduced.  When the
flag is set, trailer headers from the upstream response are passed to
the client.  Like normal headers, trailer headers will be hidden
if present in u->conf->hide_headers_hash.
2018-03-17 23:04:21 +03:00
Maxim Dounin
c554dd1434 HTTP/2: externalized various constants and interfaces. 2018-03-17 23:04:20 +03:00
Roman Arutyunyan
b84b67bc0f Style. 2018-03-12 18:38:53 +03:00
Roman Arutyunyan
1a5604bedd Stream ssl_preread: $ssl_preread_alpn_protocols variable.
The variable keeps a comma-separated list of protocol names from ALPN TLS
extension defined by RFC 7301.
2018-03-12 16:03:08 +03:00
Ruslan Ermilov
84d1e7de0c Improved code readablity.
No functional changes.
2018-03-07 18:28:12 +03:00
Maxim Dounin
83dceda868 HTTP/2: unknown frames now logged at info level. 2018-03-05 21:35:13 +03:00
Maxim Dounin
89ad448f57 Style. 2018-03-05 21:35:08 +03:00
Vladimir Homutov
7647372565 Access log: support for disabling escaping (ticket #1450).
Based on patches by Johannes Baiter <johannes.baiter@bsb-muenchen.de>
and Calin Don.
2018-03-01 11:42:55 +03:00
Maxim Dounin
c7e8a6f212 Core: ngx_current_msec now uses monotonic time if available.
When clock_gettime(CLOCK_MONOTONIC) (or faster variants, _FAST on FreeBSD,
and _COARSE on Linux) is available, we now use it for ngx_current_msec.
This should improve handling of timers if system time changes (ticket #189).
2018-03-01 20:25:50 +03:00
Roman Arutyunyan
6e30556127 Postpone filter: prevented uninitialized r->out.
The r->out chain link could be left uninitialized in case of error.
A segfault could happen if the subrequest handler accessed it.
The issue was introduced in commit 20f139e9ffa8.
2018-03-01 18:38:39 +03:00
Roman Arutyunyan
7c5c15a25d Generic subrequests in memory.
Previously, only the upstream response body could be accessed with the
NGX_HTTP_SUBREQUEST_IN_MEMORY feature.  Now any response body from a subrequest
can be saved in a memory buffer.  It is available as a single buffer in r->out
and the buffer size is configured by the subrequest_output_buffer_size
directive.

Upstream, proxy and fastcgi code used to handle the old-style feature is
removed.
2018-02-28 16:56:58 +03:00
Vladimir Homutov
2d9db482aa Modules compatibility: additional upstream metrics. 2018-02-22 17:25:43 +03:00
Vadim Filimonov
f607032e0c Auth basic: prevent null character in error log (ticket #1494). 2018-02-26 16:52:20 +02:00
Roman Arutyunyan
dd7dba520c Generate error for unsupported IPv6 transparent proxy.
On some platforms (for example, Linux with glibc 2.12-2.25) IPv4 transparent
proxying is available, but IPv6 transparent proxying is not.  The entire feature
is enabled in this case and NGX_HAVE_TRANSPARENT_PROXY macro is set to 1.
Previously, an attempt to enable transparency for an IPv6 socket was silently
ignored in this case and was usually followed by a bind(2) EADDRNOTAVAIL error
(ticket #1487).  Now the error is generated for unavailable IPv6 transparent
proxy.
2018-02-22 13:16:21 +03:00
Ruslan Ermilov
bcda92e843 HTTP/2: style.
Unified the style of validity checks in ngx_http_v2_validate_header().
2018-02-22 12:42:29 +03:00
Sergey Kandaurov
b3b4a98a5c Geo: fixed indentation. 2018-02-21 17:26:00 +03:00
Ruslan Ermilov
7a45918e0b Geo: optimized configuration parser.
If the geo block parser has failed, doing more things is pointless.
2018-02-21 15:50:43 +03:00
Ruslan Ermilov
f98a8c4db6 Geo: fixed memory allocation error handling (closes #1482).
If during configuration parsing of the geo directive the memory
allocation has failed, pool used to parse configuration inside
the block, and sometimes the temporary pool were not destroyed.
2018-02-21 15:50:42 +03:00
Ruslan Ermilov
864d93965b Version bump. 2018-02-21 15:50:35 +03:00
Maxim Dounin
9e2cd6282f HTTP/2: precalculate hash for "Cookie".
There is no need to calculate hashes of static strings at runtime.  The
ngx_hash() macro can be used to do it during compilation instead, similarly
to how it is done in ngx_http_proxy_module.c for "Server" and "Date" headers.
2018-02-15 19:06:22 +03:00
Ruslan Ermilov
89661c0e7d HTTP/2: fixed ngx_http_v2_push_stream() allocation error handling.
In particular, if a stream object allocation failed, and a client sent
the PRIORITY frame for this stream, ngx_http_v2_set_dependency() could
dereference a null pointer while trying to re-parent a dependency node.
2018-02-15 17:51:37 +03:00
Ruslan Ermilov
2437532e7f HTTP/2: push additional request headers (closes #1478).
The Accept-Encoding, Accept-Language, and User-Agent header fields
are now copied from the original request to pushed requests.
2018-02-15 17:51:32 +03:00
Ruslan Ermilov
8a84dd4f32 Expose more headers with NGX_HTTP_HEADERS. 2018-02-15 17:51:26 +03:00
Vladimir Homutov
9d00f9e449 Core: added a stub for additional zone configuration. 2018-02-15 16:08:05 +03:00
Sergey Kandaurov
5e28302dfc HTTP/2: style. 2018-02-15 02:34:16 +03:00
Ruslan Ermilov
c32d9d28fd HTTP/2: fixed null pointer dereference with server push.
r->headers_in.host can be NULL in ngx_http_v2_push_resource().

This happens when a request is terminated with 400 before the :authority
or Host header is parsed, and either pushing is enabled on the server{}
level or error_page 400 redirects to a location with pushes configured.

Found by Coverity (CID 1429156).
2018-02-09 23:20:08 +03:00
Ruslan Ermilov
09eb20c8a7 HTTP/2: fixed build with -Werror=unused-but-set-variable. 2018-02-08 12:11:30 +03:00
Ruslan Ermilov
6e52265b42 HTTP/2: server push.
Resources to be pushed are configured with the "http2_push" directive.

Also, preload links from the Link response headers, as described in
https://www.w3.org/TR/preload/#server-push-http-2, can be pushed, if
enabled with the "http2_push_preload" directive.

Only relative URIs with absolute paths can be pushed.

The number of concurrent pushes is normally limited by a client, but
cannot exceed a hard limit set by the "http2_max_concurrent_pushes"
directive.
2018-02-08 09:55:03 +03:00
Ruslan Ermilov
ac3c8ff364 HTTP/2: changed prototypes of request pseudo-headers parsers.
No functional changes.
2018-02-08 09:54:49 +03:00
Ruslan Ermilov
8590d9d615 Basic support of the Link response header. 2018-02-08 09:54:18 +03:00
Roman Arutyunyan
d31d547dba Dav: added error logging.
Previously, when request body was not available or was previously read in
memory rather than a file, client received HTTP 500 error, but no explanation
was logged in error log.  This could happen, for example, if request body was
read or discarded prior to error_page redirect, or if mirroring was enabled
along with dav.
2018-02-07 16:44:29 +03:00
Sergey Kandaurov
bde18907ac HTTP/2: removed unused field from ngx_http_v2_stream_t. 2018-02-06 20:02:59 +03:00
Ruslan Ermilov
c8f46e070f Upstream: removed X-Powered-By from the list of special headers.
After 1e720b0be7ec, it's neither specially processed nor copied
when redirecting with X-Accel-Redirect.
2018-01-30 22:23:58 +03:00
Sergey Kandaurov
57dde2ab37 SSL: using default server context in session remove (closes #1464).
This fixes segfault in configurations with multiple virtual servers sharing
the same port, where a non-default virtual server block misses certificate.
2018-01-30 17:46:31 +03:00
Ruslan Ermilov
2213695368 HTTP/2: finalize request as bad if parsing of pseudo-headers fails.
This is in line when the required pseudo-headers are missing, and
avoids spurious zero statuses in access.log.
2018-01-30 14:44:31 +03:00
Ruslan Ermilov
0b8b91f45f HTTP/2: more style, comments, and debugging. 2018-01-29 16:06:33 +03:00
Ruslan Ermilov
34cf5d5e6a HTTP/2: handle duplicate INITIAL_WINDOW_SIZE settings. 2018-01-29 15:54:36 +03:00
Ruslan Ermilov
63a4dab7b0 Fixed --test-build-eventport on macOS 10.12 and later.
In macOS 10.12, CLOCK_REALTIME and clockid_t were added, but not timer_t.
2018-01-16 13:52:03 +03:00
Maxim Dounin
3377c00119 Upstream: fixed "header already sent" alerts on backend errors.
Following ad3f342f14ba046c (1.9.13), it is possible that a request where
header was already sent will be finalized with NGX_HTTP_BAD_GATEWAY,
triggering an attempt to return additional error response and the
"header already sent" alert as a result.

In particular, it is trivial to reproduce the problem with a HEAD request
and caching enabled.  With caching enabled nginx will change HEAD to GET
and will set u->pipe->downstream_error to suppress sending the response
body to the client.  When a backend-related error occurs (for example,
proxy_read_timeout expires), ngx_http_finalize_upstream_request() will
be called with NGX_HTTP_BAD_GATEWAY.  After ad3f342f14ba046c this will
result in ngx_http_finalize_request(NGX_HTTP_BAD_GATEWAY).

Fix is to move u->pipe->downstream_error handling to a later point,
where all special response codes are changed to NGX_ERROR.

Reported by Jan Prachar,
http://mailman.nginx.org/pipermail/nginx-devel/2018-January/010737.html.
2018-01-11 21:43:49 +03:00
Maxim Dounin
742f413e91 Version bump. 2017-12-30 00:15:07 +03:00
Roman Arutyunyan
0ad556fe59 Allowed configuration token to start with a variable.
Specifically, it is now allowed to start with a variable expression with braces:
${name}.  The opening curly bracket in such a token was previously considered
the start of a new block.  Variables located anywhere else in a token worked
fine: foo${name}.
2017-12-21 13:29:40 +03:00
Roman Arutyunyan
ce45ded2a8 Fixed capabilities version.
Previously, capset(2) was called with the 64-bit capabilities version
_LINUX_CAPABILITY_VERSION_3.  With this version Linux kernel expected two
copies of struct __user_cap_data_struct, while only one was submitted.  As a
result, random stack memory was accessed and random capabilities were requested
by the worker.  This sometimes caused capset() errors.  Now the 32-bit version
_LINUX_CAPABILITY_VERSION_1 is used instead.  This is OK since CAP_NET_RAW is
a 32-bit capability (CAP_NET_RAW = 13).
2017-12-19 19:00:27 +03:00
Roman Arutyunyan
0e92c213f5 Improved the capabilities feature detection.
Previously included file sys/capability.h mentioned in capset(2) man page,
belongs to the libcap-dev package, which may not be installed on some Linux
systems when compiling nginx.  This prevented the capabilities feature from
being detected and compiled on that systems.

Now linux/capability.h system header is included instead.  Since capset()
declaration is located in sys/capability.h, now capset() syscall is defined
explicitly in code using the SYS_capset constant, similarly to other
Linux-specific features in nginx.
2017-12-18 21:09:39 +03:00
Roman Arutyunyan
752f66bf7d Retain CAP_NET_RAW capability for transparent proxying.
The capability is retained automatically in unprivileged worker processes after
changing UID if transparent proxying is enabled at least once in nginx
configuration.

The feature is only available in Linux.
2017-12-13 20:40:53 +03:00
Debayan Ghosh
d2d737e70b Use sysconf to determine cacheline size at runtime.
Determine cacheline size at runtime if supported
using sysconf(_SC_LEVEL1_DCACHE_LINESIZE). In case not supported,
fallback to compile time defaults.
2017-12-11 16:28:11 +00:00
Roman Arutyunyan
6754a40466 Proxy: escape explicit space in URI in default cache key.
If the flag space_in_uri is set, the URI in HTTP upstream request is escaped to
convert space to %20.  However this flag is not checked while creating the
default cache key.  This leads to different cache keys for requests
'/foo bar' and '/foo%20bar', while the upstream requests are identical.

Additionally, the change fixes background cache updates when the client URI
contains unescaped space.  Default cache key in a subrequest is always based on
escaped URI, while the main request may not escape it.  As a result, background
cache update subrequest may update a different cache entry.
2017-11-20 20:50:35 +03:00
Roman Arutyunyan
891908cd0c Inherit valid_unparsed_uri in cloned subrequests (ticket #1430).
Inheriting this flag will make the cloned subrequest behave consistently with
the parent.  Specifically, the upstream HTTP request and cache key created by
the proxy module may depend directly on unparsed_uri if valid_unparsed_uri flag
is set.  Previously, the flag was zero for cloned requests, which could make
background update proxy a request different than its parent and cache the result
with a different key.  For example, if client URI contained the escaped slash
character %2F, it was used as is by the proxy module in the main request, but
was unescaped in the subrequests.

Similar problems exist in the slice module.
2017-11-20 21:11:19 +03:00
Roman Arutyunyan
de11c2ad09 Proxy: simplified conditions of using unparsed uri.
Previously, the unparsed uri was explicitly allowed to be used only by the main
request.  However the valid_unparsed_uri flag is nonzero only in the main
request, which makes the main request check pointless.
2017-11-20 13:47:17 +03:00
Patryk Lesiewicz
7b3f187bc6 Upstream: flush low-level buffers on write retry.
If the data to write is bigger than what the socket can send, and the
reminder is smaller than NGX_SSL_BUFSIZE, then SSL_write() fails with
SSL_ERROR_WANT_WRITE. The reminder of payload however is successfully
copied to the low-level buffer and all the output chain buffers are
flushed. This means that retry logic doesn't work because
ngx_http_upstream_process_non_buffered_request() checks only if there's
anything in the output chain buffers and ignores the fact that something
may be buffered in low-level parts of the stack.

Signed-off-by: Patryk Lesiewicz <patryk@google.com>
2017-12-01 15:59:14 -08:00
Roman Arutyunyan
e13268714f Upstream keepalive: clean read delayed flag in stored connections.
If a connection with the read delayed flag set was stored in the keepalive
cache, and after picking it from the cache a read timer was set on that
connection, this timer was considered a delay timer rather than a socket read
event timer as expected.  The latter timeout is usually much longer than the
former, which caused a significant delay in request processing.

The issue manifested itself with proxy_limit_rate and upstream keepalive
enabled and exists since 973ee2276300 (1.7.7) when proxy_limit_rate was
introduced.
2017-11-28 14:00:00 +03:00
Maxim Dounin
53e5a746bf Autoindex: discard request body (ticket #1439). 2017-12-04 17:30:02 +03:00
Sergey Kandaurov
e3baa90da9 Removed unused FreeBSD-specific definitions in ngx_posix_config.h. 2017-11-28 13:09:54 +03:00
Ruslan Ermilov
afad219175 Fixed "changing binary" when reaper is not init.
On some systems, it's possible that reaper of orphaned processes is
set to something other than "init" process.  On such systems, the
changing binary procedure did not work.

The fix is to check if PPID has changed, instead of assuming it's
always 1 for orphaned processes.
2017-11-28 12:00:24 +03:00
Maxim Dounin
3656f2eb63 Version bump. 2017-11-23 16:32:58 +03:00
Maxim Dounin
b32cb6b610 Fixed worker_shutdown_timeout in various cases.
The ngx_http_upstream_process_upgraded() did not handle c->close request,
and upgraded connections do not use the write filter.  As a result,
worker_shutdown_timeout did not affect upgraded connections (ticket #1419).
Fix is to handle c->close in the ngx_http_request_handler() function, thus
covering most of the possible cases in http handling.

Additionally, mail proxying did not handle neither c->close nor c->error,
and thus worker_shutdown_timeout did not work for mail connections.  Fix is
to add c->close handling to ngx_mail_proxy_handler().

Also, added explicit handling of c->close to stream proxy,
ngx_stream_proxy_process_connection().  This improves worker_shutdown_timeout
handling in stream, it will no longer wait for some data being transferred
in a connection before closing it, and will also provide appropriate
logging at the "info" level.
2017-11-20 16:31:07 +03:00