Experimental QUIC support for nginx ----------------------------------- 1. Introduction 2. Building from sources 3. Configuration 4. Directives 5. Clients 6. Troubleshooting 7. Contributing 8. Links 1. Introduction This is an experimental QUIC [1] / HTTP/3 [2] support for nginx. The code is developed in a separate "quic" branch available at https://hg.nginx.org/nginx-quic. Currently it is based on nginx mainline 1.23.x. We merge new nginx releases into this branch regularly. The project code base is under the same BSD license as nginx. The code is currently at a beta level of quality, however there are several production deployments with it. NGINX Development Team is working on improving HTTP/3 support to integrate it into the main NGINX codebase. Thus, expect further updates of this code, including features, changes in behaviour, bug fixes, and refactoring. NGINX Development team will be grateful for any feedback and code submissions. Please contact NGINX Development Team via nginx-devel mailing list [3]. What works now: IETF QUIC version 1 is supported. Internet drafts are no longer supported. nginx should be able to respond to HTTP/3 requests over QUIC and it should be possible to upload and download big files without errors. + The handshake completes successfully + One endpoint can update keys and its peer responds correctly + 0-RTT data is being received and acted on + Connection is established using TLS Resume Ticket + A handshake that includes a Retry packet completes successfully + Stream data is being exchanged and ACK'ed + An H3 transaction succeeded + One or both endpoints insert entries into dynamic table and subsequently reference them from header blocks + Version Negotiation packet is sent to client with unknown version + Lost packets are detected and retransmitted properly + Clients may migrate to new address 2. Building from sources The build is configured using the configure command. Refer to http://nginx.org/en/docs/configure.html for details. When configuring nginx, it's possible to enable QUIC and HTTP/3 using the following new configuration options: --with-http_v3_module - enable QUIC and HTTP/3 --with-stream_quic_module - enable QUIC in Stream A library that provides QUIC support is recommended to build nginx, there are several of those available on the market: + BoringSSL [4] + LibreSSL [5] + QuicTLS [6] Alternatively, nginx can be configured with OpenSSL compatibility layer, which emulates BoringSSL QUIC API for OpenSSL. This mode is enabled by default if native QUIC support is not detected. 0-RTT is not supported in OpenSSL compatibility mode. Clone the NGINX QUIC repository $ hg clone -b quic https://hg.nginx.org/nginx-quic $ cd nginx-quic Use the following command to configure nginx with BoringSSL [4] $ ./auto/configure --with-debug --with-http_v3_module \ --with-cc-opt="-I../boringssl/include" \ --with-ld-opt="-L../boringssl/build/ssl \ -L../boringssl/build/crypto" $ make Alternatively, nginx can be configured with QuicTLS [6] $ ./auto/configure --with-debug --with-http_v3_module \ --with-cc-opt="-I../quictls/build/include" \ --with-ld-opt="-L../quictls/build/lib" Alternatively, nginx can be configured with a modern version of LibreSSL [7] $ ./auto/configure --with-debug --with-http_v3_module \ --with-cc-opt="-I../libressl/build/include" \ --with-ld-opt="-L../libressl/build/lib" 3. Configuration The HTTP "listen" directive got a new option "quic" which enables QUIC as client transport protocol instead of TCP. The Stream "listen" directive got a new option "quic" which enables QUIC as client transport protocol instead of TCP or plain UDP. Along with "quic", it's also possible to specify "reuseport" option [8] to make it work properly with multiple workers. To enable address validation: quic_retry on; To enable 0-RTT: ssl_early_data on; Make sure that TLS 1.3 is configured which is required for QUIC: ssl_protocols TLSv1.3; To enable GSO (Generic Segmentation Offloading): quic_gso on; To limit maximum UDP payload size on receive path: quic_mtu ; To set host key for various tokens: quic_host_key ; By default, GSO Linux-specific optimization [10] is disabled. Enable it in case a corresponding network interface is configured to support GSO. A number of directives were added that configure HTTP/3: http3 http3_hq http3_stream_buffer_size http3_max_concurrent_pushes http3_max_concurrent_streams http3_push http3_push_preload In http, an additional variable is available: $http3. The value of $http3 is "h3" for HTTP/3 connections, "hq" for hq connections, or an empty string otherwise. In stream, an additional variable is available: $quic. The value of $quic is "quic" if QUIC connection is used, or an empty string otherwise. Example configuration: http { log_format quic '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$http3"'; access_log logs/access.log quic; server { # for better compatibility it's recommended # to use the same port for quic and https listen 8443 quic reuseport; listen 8443 ssl; ssl_certificate certs/example.com.crt; ssl_certificate_key certs/example.com.key; ssl_protocols TLSv1.3; location / { # required for browsers to direct them into quic port add_header Alt-Svc 'h3=":8443"; ma=86400'; } } } 4. Directives Syntax: quic_bpf on | off; Default: quic_bpf off; Context: main Enables routing of QUIC packets using eBPF. When enabled, this allows to support QUIC connection migration. The directive is only supported on Linux 5.7+. Syntax: quic_retry on | off; Default: quic_retry off; Context: http | stream, server Enables the QUIC Address Validation feature. This includes: - sending a new token in a Retry packet or a NEW_TOKEN frame - validating a token received in the Initial packet Syntax: quic_gso on | off; Default: quic_gso off; Context: http | stream, server Enables sending in optimized batch mode using segmentation offloading. Optimized sending is only supported on Linux featuring UDP_SEGMENT. Syntax: quic_mtu size; Default: quic_mtu 65527; Context: http | stream, server Sets the QUIC max_udp_payload_size transport parameter value. This is the maximum UDP payload that we are willing to receive. Syntax: quic_host_key file; Default: - Context: http | stream, server Specifies a file with the secret key used to encrypt stateless reset and address validation tokens. By default, a randomly generated key is used. Syntax: quic_active_connection_id_limit number; Default: quic_active_connection_id_limit 2; Context: http | stream, server Sets the QUIC active_connection_id_limit transport parameter value. This is the maximum number of connection IDs we are willing to store. Syntax: quic_timeout time; Default: quic_timeout 60s; Context: stream, server Defines a timeout used to negotiate the QUIC idle timeout. In the http module, it is taken from the keepalive_timeout directive. Syntax: quic_stream_buffer_size size; Default: quic_stream_buffer_size 64k; Context: stream, server Syntax: http3_stream_buffer_size size; Default: http3_stream_buffer_size 64k; Context: http, server Sets buffer size for reading and writing of the QUIC STREAM payload. The buffer size is used to calculate initial flow control limits in the following QUIC transport parameters: - initial_max_data - initial_max_stream_data_bidi_local - initial_max_stream_data_bidi_remote - initial_max_stream_data_uni Syntax: http3_max_concurrent_pushes number; Default: http3_max_concurrent_pushes 10; Context: http, server Limits the maximum number of concurrent push requests in a connection. Syntax: http3_max_concurrent_streams number; Default: http3_max_concurrent_streams 128; Context: http, server Sets the maximum number of concurrent HTTP/3 streams in a connection. Syntax: http3_push uri | off; Default: http3_push off; Context: http, server, location Pre-emptively sends (pushes) a request to the specified uri along with the response to the original request. Only relative URIs with absolute path will be processed, for example: http3_push /static/css/main.css; The uri value can contain variables. Several http3_push directives can be specified on the same configuration level. The off parameter cancels the effect of the http3_push directives inherited from the previous configuration level. Syntax: http3_push_preload on | off; Default: http3_push_preload off; Context: http, server, location Enables automatic conversion of preload links specified in the “Link” response header fields into push requests. Syntax: http3 on | off; Default: http3 on; Context: http, server Enables HTTP/3 protocol negotiation. Syntax: http3_hq on | off; Default: http3_hq off; Context: http, server Enables HTTP/0.9 protocol negotiation used in QUIC interoperability tests. 5. Clients * Browsers Known to work: Firefox 90+ and Chrome 92+ (QUIC version 1) Beware of strange issues: sometimes browser may decide to ignore QUIC Cache clearing/restart might help. Always check access.log and error.log to make sure the browser is using HTTP/3 and not TCP https. * Console clients Known to work: ngtcp2, firefox's neqo and chromium's console clients: $ examples/client 127.0.0.1 8443 https://example.com:8443/index.html $ ./neqo-client https://127.0.0.1:8443/ $ chromium-build/out/my_build/quic_client http://example.com:8443 In case everyhing is right, the access log should show something like: 127.0.0.1 - - [24/Apr/2020:11:27:29 +0300] "GET / HTTP/3" 200 805 "-" "nghttp3/ngtcp2 client" "quic" 6. Troubleshooting Here are some tips that may help to identify problems: + Ensure nginx is built with proper SSL library that supports QUIC + Ensure nginx is using the proper SSL library in runtime (`nginx -V` shows what it's using) + Ensure a client is actually sending requests over QUIC (see "Clients" section about browsers and cache) We recommend to start with simple console client like ngtcp2 to ensure the server is configured properly before trying with real browsers that may be very picky with certificates, for example. + Build nginx with debug support [9] and check the debug log. It should contain all details about connection and why it failed. All related messages contain "quic " prefix and can be easily filtered out. + For a deeper investigation, please enable additional debugging in src/event/quic/ngx_event_quic_connection.h: #define NGX_QUIC_DEBUG_PACKETS #define NGX_QUIC_DEBUG_FRAMES #define NGX_QUIC_DEBUG_ALLOC #define NGX_QUIC_DEBUG_CRYPTO 7. Contributing Please refer to http://nginx.org/en/docs/contributing_changes.html 8. Links [1] https://datatracker.ietf.org/doc/html/rfc9000 [2] https://datatracker.ietf.org/doc/html/rfc9114 [3] https://mailman.nginx.org/mailman/listinfo/nginx-devel [4] https://boringssl.googlesource.com/boringssl/ [5] https://www.libressl.org/ [6] https://github.com/quictls/openssl [7] https://github.com/libressl-portable/portable/releases/tag/v3.6.0 [8] https://nginx.org/en/docs/http/ngx_http_core_module.html#listen [9] https://nginx.org/en/docs/debugging_log.html [10] http://vger.kernel.org/lpc_net2018_talks/willemdebruijn-lpc2018-udpgso-paper-DRAFT-1.pdf