Commit Graph

745 Commits

Author SHA1 Message Date
Ruslan Ermilov
31389e4921 Fixed undefined behavior when left shifting signed integer. 2016-08-04 23:42:00 +03:00
Ruslan Ermilov
e62a16e8f6 Win32: added per-thread random seeding.
The change in b91bcba29351 was not enough to fix random() seeding.
On Windows, the srand() seeds the PRNG only in the current thread,
and worse, is not inherited from the calling thread.  Due to this,
worker threads were not properly seeded.

Reported by Marc Bevand.
2016-08-04 01:15:41 +03:00
Otto Kekäläinen
cb963abf03 Fixed spelling. 2016-06-08 08:27:41 +03:00
Maxim Dounin
a3d382397a Core: skip special buffers on writing (ticket #981).
A special last buffer with cl->buf->pos set to NULL can be present in
a chain when writing request body if chunked encoding was used.  This
resulted in a NULL pointer dereference if it happened to be the only
buffer left after a do...while loop iteration in ngx_write_chain_to_file().

The problem originally appeared in nginx 1.3.9 with chunked encoding
support.  Additionally, rev. 3832b608dc8d (nginx 1.9.13) changed the
minimum number of buffers to trigger this from IOV_MAX (typically 1024)
to NGX_IOVS_PREALLOCATE (typically 64).

Fix is to skip such buffers in ngx_chain_to_iovec(), much like it is
done in other places.
2016-05-31 05:13:30 +03:00
Valentin Bartenev
12f4367189 Improved EPOLLRDHUP handling.
When it's known that the kernel supports EPOLLRDHUP, there is no need in
additional recv() call to get EOF or error when the flag is absent in the
event generated by the kernel.  A special runtime test is done at startup
to detect if EPOLLRDHUP is actually supported by the kernel because
epoll_ctl() silently ignores unknown flags.

With this knowledge it's now possible to drop the "ready" flag for partial
read.  Previously, the "ready" flag was kept until the recv() returned EOF
or error.  In particular, this change allows the lingering close heuristics
(which relies on the "ready" flag state) to actually work on Linux, and not
wait for more data in most cases.

The "available" flag is now used in the read event with the semantics similar
to the corresponding counter in kqueue.
2016-05-13 17:19:23 +03:00
Ruslan Ermilov
37a3a2b2e8 Removed redundant "u" format specifier.
It is implied for "x" and "X".
2016-04-08 15:03:38 +03:00
Ruslan Ermilov
4d1f67fd91 Simplified ngx_unix_recv() and ngx_readv_chain().
This makes ngx_unix_recv() and ngx_udp_unix_recv() differ minimally.
2016-04-08 16:49:35 +03:00
Valentin Bartenev
dbabc66b68 Merged implementations of ngx_unix_recv().
There's no real need in two separate implementations,
with and without kqueue support.
2016-04-08 16:41:45 +03:00
Valentin Bartenev
900ef17c47 Fixed small inconsistency in handling EOF among receive functions.
Now all functions always drop the ready flag in this case.
2016-04-08 16:39:49 +03:00
Valentin Bartenev
8e7d8757c6 Merged implementations of ngx_udp_unix_recv().
There's no real need in two separate implementations,
with and without kqueue support.
2016-04-08 16:38:42 +03:00
Josh Soref
73d27510c0 Fixed spelling. 2016-04-07 11:50:13 +03:00
Ruslan Ermilov
5ad6e01b46 Fixed ngx_os_signal_process() prototype. 2016-03-31 22:00:33 +03:00
Sergey Kandaurov
60a6551e1f Fixed ngx_pid_t formatting in ngx_sprintf() and logging. 2016-03-31 02:34:04 +03:00
Sergey Kandaurov
00ef9ff5f0 Fixed logging. 2016-03-31 02:33:57 +03:00
Ruslan Ermilov
7ad57da598 Style. 2016-03-30 11:52:16 +03:00
Maxim Dounin
d259032741 Win32: replaced NGX_EXDEV with more appropriate error code.
Correct error code for NGX_EXDEV on Windows is ERROR_NOT_SAME_DEVICE,
"The system cannot move the file to a different disk drive".

Previously used ERROR_WRONG_DISK is about wrong diskette in the drive and
is not appropriate.

There is no real difference though, as MoveFile() is able to copy files
between disk drives, and will fail with ERROR_ACCESS_DENIED when asked
to copy directories.  The ERROR_NOT_SAME_DEVICE error is only used
by MoveFileEx() when called without the MOVEFILE_COPY_ALLOWED flag.
2016-03-29 09:52:15 +03:00
Maxim Dounin
6871615528 Win32: additional error code NGX_EEXIST_FILE (ticket #910).
On Windows there are two possible error codes which correspond to
the EEXIST error code: ERROR_FILE_EXISTS used by CreateFile(CREATE_NEW),
and ERROR_ALREADY_EXISTS used by CreateDirectory().

MoveFile() seems to use both: ERROR_ALREADY_EXISTS when moving within
one filesystem, and ERROR_FILE_EXISTS when copying a file to a different
drive.
2016-03-29 09:51:46 +03:00
Ruslan Ermilov
f5fff1eda0 Fixed --test-build-*.
Fixes various aspects of --test-build-devpoll, --test-build-eventport, and
--test-build-epoll.

In particular, if --test-build-devpoll was used on Linux, then "devpoll"
event method would be preferred over "epoll".  Also, wrong definitions of
event macros were chosen.
2016-03-28 19:29:18 +03:00
Maxim Dounin
10c8c8d6a4 Threads: offloading of temp files writing to thread pools.
The ngx_thread_write_chain_to_file() function introduced, which
uses ngx_file_t thread_handler, thread_ctx and thread_task fields.
The task context structure (ngx_thread_file_ctx_t) is the same for
both reading and writing, and can be safely shared as long as
operations are serialized.

The task->handler field is now always set (and not only when task is
allocated), as the same task can be used with different handlers.

The thread_write flag is introduced in the ngx_temp_file_t structure
to explicitly enable use of ngx_thread_write_chain_to_file() in
ngx_write_chain_to_temp_file() when supported by caller.

In collaboration with Valentin Bartenev.
2016-03-18 06:44:03 +03:00
Maxim Dounin
ac78b26324 Threads: task pointer stored in ngx_file_t.
This simplifies the interface of the ngx_thread_read() function.

Additionally, most of the thread operations now explicitly set
file->thread_task, file->thread_handler and file->thread_ctx,
to facilitate use of thread operations in other places.

(Potential problems remain with sendfile in threads though - it uses
file->thread_handler as set in ngx_output_chain(), and it should not
be overwritten to an incompatible one.)

In collaboration with Valentin Bartenev.
2016-03-18 06:43:52 +03:00
Maxim Dounin
931ce7f02a Fixed timeouts with threaded sendfile() and subrequests.
If a write event happens after sendfile() but before we've got the
sendfile results in the main thread, this write event will be ignored.
And if no more events will happen, the connection will hang.

Removing the events works in the simple cases, but not always, as
in some cases events are added back by an unrelated code.  E.g.,
the upstream module adds write event in the ngx_http_upstream_init()
to track client aborts.

Fix is to use wev->complete instead.  It is now set to 0 before
a sendfile() task is posted, and it is set to 1 once a write event
happens.  If on completion of the sendfile() task wev->complete is 1,
we know that an event happened while we were executing sendfile(), and
the socket is still ready for writing even if sendfile() did not sent
all the data or returned EAGAIN.
2016-03-18 05:04:45 +03:00
Maxim Dounin
07a7b57762 Truncation detection in sendfilev() on Solaris.
While sendfilev() is documented to return -1 with EINVAL set
if the file was truncated, at least Solaris 11 silently returns 0,
and this results in CPU hog.  Added a test to complain appropriately
if 0 is returned.
2016-03-15 18:26:59 +03:00
Maxim Dounin
681e735a33 Truncation detection in sendfile() on Linux.
This addresses connection hangs as observed in ticket #504, and
CPU hogs with "aio threads; sendfile on" as reported in the mailing list,
see http://mailman.nginx.org/pipermail/nginx-ru/2016-March/057638.html.

The alert is identical to one used on FreeBSD.
2016-03-15 18:26:17 +03:00
Roman Arutyunyan
2ce791f2cd Stream: UDP proxy. 2016-01-20 19:52:12 +03:00
Valentin Bartenev
b3c1abdcd8 Fixed handling of EAGAIN with sendfile in threads. 2016-03-14 22:42:35 +03:00
Maxim Dounin
a6fe512df6 Fixed sendfile in threads (or with aio preload) and subrequests.
If sendfile in threads is used, it is possible that multiple
subrequests will trigger multiple ngx_linux_sendfile_thread() calls,
as operations are only serialized in output chain based on r->aio,
that is, on subrequest level.

This resulted in "task #N already active" alerts, in particular, when
running proxy_store.t with "aio threads; sendfile on;".

Fix is to tolerate duplicate calls, with an additional safety check
that the file is the same as previously used.

The same problem also affects "aio on; sendfile on;" on FreeBSD
(previously known as "aio sendfile;"), where aio->preload_handler()
could be called multiple times due to similar reasons, resulting in
"second aio post" alerts.  Fix is the same as well.

It is also believed that similar problems can arise if a filter
calls the next body filter multiple times for some reason.  These are
mostly theoretical though.
2016-03-03 21:14:12 +03:00
Valentin Bartenev
e243a2d6c2 Introduced the ngx_chain_to_iovec() function.
It's similar to ngx_output_chain_to_iovec() and uses only preallocated memory.
2016-03-03 18:41:05 +03:00
Vladimir Homutov
faa96e82d2 Core: added support for more than 64 CPUs in worker_cpu_affinity. 2016-02-18 13:58:49 +03:00
Maxim Dounin
86cc342a26 Dynamic modules: dlopen() support. 2016-02-04 18:30:21 +03:00
Maxim Dounin
0f67d6355c Dynamic modules: changed ngx_modules to cycle->modules. 2016-02-04 18:30:21 +03:00
Piotr Sikora
9752706bd4 Core: fix typo in error message.
Signed-off-by: Piotr Sikora <piotrsikora@google.com>
2015-12-02 19:17:19 -08:00
Valentin Bartenev
d862ae8e68 Used the pwritev() syscall for writing files where possible.
It is more effective, because it doesn't require a separate lseek().
2015-11-17 19:01:41 +03:00
Valentin Bartenev
906c4462f8 Moved file writev() handling code to a separate function.
No functional changes.
2015-11-17 19:01:41 +03:00
Valentin Bartenev
a9e62b625b Handled EINTR from write() and pwrite() syscalls.
This is in addition to 6fce16b1fc10.
2015-11-17 19:01:41 +03:00
Valentin Bartenev
9ef4b63328 Adjusted file->sys_offset after the write() syscall.
This fixes suboptimal behavior caused by surplus lseek() for sequential writes
on systems without pwrite().  A consecutive read after write might result in an
error on systems without pread() and pwrite().

Fortunately, at the moment there are no widely used systems without these
syscalls.
2015-11-17 19:01:41 +03:00
Valentin Bartenev
a38402e306 Fixed building --with-debug, broken by 6fce16b1fc10. 2015-09-02 19:45:40 +03:00
Valentin Bartenev
7dd3f9ee22 Writing to some file systems can be interrupted.
At least such behavior was observed with CephFS, see:
http://mailman.nginx.org/pipermail/nginx/2015-July/048188.html.
2015-09-02 19:26:40 +03:00
Maxim Dounin
84b0ad63fb Win32: MSVC 2015 compatibility.
Resolved warnings about declarations that hide previous local declarations.
Warnings about WSASocketA() being deprecated resolved by explicit use of
WSASocketW() instead of WSASocket().  When compiling without IPv6 support,
WinSock deprecated warnings are disabled to allow use of gethostbyname().
2015-08-17 18:09:17 +03:00
Valentin Bartenev
50ff8b3c3a Core: idle connections now closed only once on exiting.
Iterating through all connections takes a lot of CPU time, especially
with large number of worker connections configured.  As a result
nginx processes used to consume CPU time during graceful shutdown.
To mitigate this we now only do a full scan for idle connections when
shutdown signal is received.

Transitions of connections to idle ones are now expected to be
avoided if the ngx_exiting flag is set.  The upstream keepalive module
was modified to follow this.
2015-08-11 16:28:55 +03:00
Vladimir Homutov
42c32b733b Core: added support for writing to stdout. 2015-06-16 15:47:40 +03:00
Maxim Dounin
226647e397 Configure: GNU Hurd properly recognized.
With this change it's no longer needed to pass -D_GNU_SOURCE manually,
and -D_FILE_OFFSET_BITS=64 is set to use 64-bit off_t.

Note that nginx currently fails to work properly with master process
enabled on GNU Hurd, as fcntl(F_SETOWN) returns EOPNOTSUPP for sockets
as of GNU Hurd 0.6.  Additionally, our strerror() preloading doesn't
work well with GNU Hurd, as it uses large numbers for most errors.
2015-05-25 17:58:13 +03:00
Maxim Dounin
68b3d28c55 Introduced worker number, ngx_worker. 2015-05-20 15:51:21 +03:00
Maxim Dounin
2ec8cfcd7d Win32: shared memory base addresses and remapping.
Two mechanisms are implemented to make it possible to store pointers
in shared memory on Windows, in particular on Windows Vista and later
versions with ASLR:

- The ngx_shm_remap() function added to allow remapping of a shared memory
  zone to the address originally used for it in the master process.  While
  important, it doesn't solve the problem by itself as in many cases it's
  not possible to use the address because of conflicts with other
  allocations.

- We now create mappings at the same address in all processes by starting
  mappings at predefined addresses normally unused by newborn processes.

These two mechanisms combined allow to use shared memory on Windows
almost without problems, including reloads.

Based on the patch by Sergey Brester:
http://mailman.nginx.org/pipermail/nginx-devel/2015-April/006836.html
2015-04-27 18:25:42 +03:00
Ruslan Ermilov
3180f84b60 Removed the "worker_rlimit_sigpending" directive.
It was only needed by the just removed rtsig module.
2015-04-23 14:26:11 +03:00
Ruslan Ermilov
33b8e5bc06 Removed the obsolete rtsig module. 2015-04-23 14:17:40 +03:00
Ruslan Ermilov
c1882d9f3f Removed the obsolete aio module. 2015-04-22 18:57:32 +03:00
Ruslan Ermilov
9b411460f9 Replaced the remaining NGX_OLD_THREADS check with NGX_WIN32. 2015-03-26 14:20:51 +03:00
Ruslan Ermilov
64c5df4b62 Removed "worker_threads" and "thread_stack_size" directives. 2015-03-26 14:15:17 +03:00
Ruslan Ermilov
324e389e17 Removed unused thread-local-storage code. 2015-03-26 14:15:15 +03:00
Ruslan Ermilov
2b0d2f0583 Removed ngx_threaded and related code. 2015-03-26 14:15:13 +03:00