Commit Graph

119 Commits

Author SHA1 Message Date
Maxim Dounin
ce05841eef Cache: check the whole cache key in addition to hashes.
This prevents a potential attack that discloses cached data if an attacker
will be able to craft a hash collision between some cache key the attacker
is allowed to access and another cache key with protected data.

See http://mailman.nginx.org/pipermail/nginx-devel/2015-September/007288.html.

Thanks to Gena Makhomed and Sergey Brester.
2015-09-11 17:03:56 +03:00
Valentin Bartenev
355a793b89 Cache: added support for reading of the header in thread pools. 2015-04-01 03:49:17 +03:00
Valentin Bartenev
a7ad493aa6 Added support for offloading read() in thread pools. 2015-03-14 17:37:25 +03:00
Roman Arutyunyan
f0b5f83378 Cache: do not inherit last_modified and etag from stale response.
When replacing a stale cache entry, its last_modified and etag could be
inherited from the old entry if the response code is not 200 or 206.  Moreover,
etag could be inherited with any response code if it's missing in the new
response.  As a result, the cache entry is left with invalid last_modified or
etag which could lead to broken revalidation.

For example, when a file is deleted from backend, its last_modified is copied to
the new 404 cache entry and is used later for revalidation.  Once the old file
appears again with its original timestamp, revalidation succeeds and the cached
404 response is sent to client instead of the file.

The problem appeared with etags in 44b9ab7752e3 (1.7.3) and affected
last_modified in 1573fc7875fa (1.7.9).
2015-03-02 19:47:13 +03:00
Ruslan Ermilov
a4ce770a89 Cache: reduced diffs to the plus version of nginx.
No functional changes.
2015-02-17 16:27:52 +03:00
Roman Arutyunyan
f8fecbc94d Cache: added temp_path to file cache.
If use_temp_path is set to off, a subdirectory "temp" is created in the cache
directory.  It's used instead of proxy_temp_path and friends for caching
upstream response.
2015-02-02 19:38:35 +03:00
Valentin Bartenev
843fb552ec Upstream: use_temp_path parameter of proxy_cache_path and friends.
When set to "off", temporary files for cacheable responses will be stored
inside cache directory.
2014-12-26 16:22:59 +03:00
Valentin Bartenev
1858857c25 Cache: update variant while setting header.
Some parts of code related to handling variants of a resource moved into
a separate function that is called earlier.  This allows to use cache file
name as a prefix for temporary file in the following patch.
2014-12-26 16:22:56 +03:00
Valentin Bartenev
7817df480e Upstream: added variables support to proxy_cache and friends. 2014-12-22 12:59:09 +03:00
Piotr Sikora
1ea64b3f8f Cache: send conditional requests only for cached 200/206 responses.
RFC7232 says:

   The 304 (Not Modified) status code indicates that a conditional GET
   or HEAD request has been received and would have resulted in a 200
   (OK) response if it were not for the fact that the condition
   evaluated to false.

which means that there is no reason to send requests with "If-None-Match"
and/or "If-Modified-Since" headers for responses cached with other status
codes.

Also, sending conditional requests for responses cached with other status
codes could result in a strange behavior, e.g. upstream server returning
304 Not Modified for cached 404 Not Found responses, etc.

Signed-off-by: Piotr Sikora <piotr@cloudflare.com>
2014-11-26 18:35:37 -08:00
Maxim Dounin
6d52912a2a Cache: proper wakeup of subrequests.
In case of a cache lock timeout and in the aio handler we now call
r->write_event_handler() instead of a connection write handler,
to make sure to run appropriate subrequest.  Previous code failed to run
inactive subrequests and hence resulted in suboptimal behaviour, see
report by Yichun Zhang:

http://mailman.nginx.org/pipermail/nginx-devel/2013-October/004435.html

(Infinite hang claimed in the report seems impossible without 3rd party
modules, as subrequests will be eventually woken up by the postpone filter.)
2014-12-02 05:54:56 +03:00
Maxim Dounin
10345663c8 Upstream: improved subrequest logging.
To ensure proper logging make sure to set current_request in all event
handlers, including resolve, ssl handshake, cache lock wait timer and
aio read handlers.  A macro ngx_http_set_log_request() introduced to
simplify this.
2014-12-02 05:54:54 +03:00
Roman Arutyunyan
2628cc11d4 Cache: proxy_cache_lock_age and friends.
Once this age is reached, the cache lock is discarded and another
request can acquire the lock.  Requests which failed to acquire
the lock are not allowed to cache the response.
2014-11-18 20:41:12 +03:00
Maxim Dounin
e2bc3c11b2 Cache: removed dead store in ngx_http_file_cache_vary_header().
Found by Clang Static Analyzer.
2014-11-05 18:53:26 +03:00
Maxim Dounin
5f625b7df0 Cache: normalization of some Vary headers.
Spaces in Accept-Charset, Accept-Encoding, and Accept-Language headers
are now ignored.  As per syntax of these headers spaces can only appear
in places where they are optional.
2014-10-27 21:14:12 +03:00
Maxim Dounin
6255935e0f Cache: multiple variants of a resource now can be stored.
If a variant stored can't be used to respond to a request, the variant
hash is used as a secondary key.

Additionally, if we previously switched to a secondary key, while storing
a response to cache we check if the variant hash still apply.  If not, we
switch back to the original key, to handle cases when Vary changes.
2014-10-27 21:14:10 +03:00
Maxim Dounin
83f00adc9b Cache: c->reading flag introduced.
It replaces c->buf in checks in ngx_http_file_cache_open(), making it possible
to reopen the file without clearing c->buf.  No functional changes.
2014-10-27 21:14:07 +03:00
Maxim Dounin
1332e76b20 Cache: hash of Vary headers now stored in cache.
To cache responses with Vary, we now calculate hash of headers listed
in Vary, and return the response from cache only if new request headers
match.

As of now, only one variant of the same resource can be stored in cache.
2014-10-27 21:13:58 +03:00
Roman Arutyunyan
c0b3b9d6ca Upstream: suppressed the file cache slab allocator error messages.
The messages "ngx_slab_alloc() failed: no memory in cache keys zone"
from the file cache slab allocator are suppressed since the allocation
is likely to succeed after the forced expiration of cache nodes.
The second allocation failure is reported.
2014-09-05 18:14:59 +04:00
Maxim Dounin
b812961677 Cache: ETag now saved into cache header. 2014-06-26 02:28:23 +04:00
Maxim Dounin
c2e49a4196 Cache: version in cache files.
This allows to change the structure of cache files without spamming logs
with false alerts.
2014-06-26 02:27:21 +04:00
Ruslan Ermilov
05d717b35d Core: added ngx_slab_calloc() and ngx_slab_calloc_locked().
These functions return zeroed memory, analogous to ngx_pcalloc().
2014-06-04 15:09:19 +04:00
Maxim Dounin
17dad56e4e Cache: added ngx_quit check to ngx_http_file_cache_expire().
While managing big caches it is possible that expiring old cache items
in ngx_http_file_cache_expire() will take a while.  Added a check for
ngx_quit / ngx_terminate to make sure cache manager can be terminated
while in ngx_http_file_cache_expire().
2014-04-30 19:16:35 +04:00
Maxim Dounin
1ac2693a33 Upstream: cache revalidation with conditional requests.
The following new directives are introduced: proxy_cache_revalidate,
fastcgi_cache_revalidate, scgi_cache_revalidate, uwsgi_cache_revalidate.
Default is off.  When set to on, they enable cache revalidation using
conditional requests with If-Modified-Since for expired cache items.

As of now, no attempts are made to merge headers given in a 304 response
during cache revalidation with headers previously stored in a cache item.
Headers in a 304 response are only used to calculate new validity time
of a cache item.
2013-11-18 20:48:22 +04:00
Maxim Dounin
74b7a91013 Win32: Borland C compatibility fixes.
Several false positive warnings silenced, notably W8012 "Comparing
signed and unsigned" (due to u_short values promoted to int), and
W8072 "Suspicious pointer arithmetic" (due to large type values added
to pointers).

With this patch, it's now again possible to compile nginx using bcc32,
with options we normally compile on win32 minus ipv6 and ssl.
2013-09-04 20:48:23 +04:00
Maxim Dounin
d2d8b82b87 Cache: lock timeouts are now logged at info level. 2013-08-23 22:18:54 +04:00
Maxim Dounin
e3cab76758 Backed out f1a91825730a and 7094bd12c1ff.
While ngx_get_full_name() might have a bit more descriptive arguments,
the ngx_conf_full_name() is generally easier to use when parsing
configuration and limits exposure of cycle->prefix / cycle->conf_prefix
details.
2013-08-20 21:11:19 +04:00
Valentin Bartenev
d29d21bade Replaced ngx_conf_full_name() with ngx_get_full_name().
The ngx_get_full_name() function takes more readable arguments list.
2013-08-06 19:58:40 +04:00
Maxim Dounin
40c4e7b73c Valgrind: supressed complaints about uninitialized bytes.
Valgrind complains if we pass uninitialized memory to a syscall:

==36492== Syscall param sendmsg(msg.msg_iov[0]) points to uninitialised byte(s)
==36492==    at 0x6B5E6A: sendmsg (in /usr/lib/system/libsystem_kernel.dylib)
==36492==    by 0x10004288E: ngx_signal_worker_processes (ngx_process_cycle.c:527)
==36492==    by 0x1000417A7: ngx_master_process_cycle (ngx_process_cycle.c:203)
==36492==    by 0x100001F10: main (nginx.c:410)
==36492==  Address 0x7fff5fbff71c is on thread 1's stack

Even initialization of all members of the structure passed isn't enough, as
there is padding which still remains uninitialized and results in Valgrind
complaint.  Note there is no real problem here as data from uninitialized
memory isn't used.
2013-06-05 19:44:22 +04:00
Valentin Bartenev
59db08a6fc Removed zero termination of shm zone names.
It was added in r2717 and no longer needed since r2721,
where the termination was added to ngx_shm_alloc() and
ngx_init_zone_pool().  So then it only corrupts error
messages about ivalid zones.
2013-02-19 17:48:45 +00:00
Maxim Dounin
7ca6c1ff78 Fix of rbtree lookup on hash collisions.
Previous code incorrectly assumed that nodes with identical keys are linked
together.  This might not be true after tree rebalance.

Patch by Lanshun Zhou.
2012-02-27 22:15:39 +00:00
Maxim Dounin
9f38b20db5 Time parsing cleanup.
Nuke NGX_PARSE_LARGE_TIME, it's not used since 0.6.30.  The only error
ngx_parse_time() can currently return is NGX_ERROR, check it explicitly
and make sure to cast it to appropriate type (either time_t or ngx_msec_t)
to avoid signedness warnings on platforms with unsigned time_t (notably QNX).
2012-02-13 15:41:11 +00:00
Maxim Konovalov
f8d59e33f3 Copyright updated. 2012-01-18 15:07:43 +00:00
Maxim Dounin
947bdfa7fe Fixed build without debug. 2011-12-26 11:33:11 +00:00
Maxim Dounin
822fe46934 Cache locks initial implementation.
New directives: proxy_cache_lock on/off, proxy_cache_lock_timeout.  With
proxy_cache_lock set to on, only one request will be allowed to go to
upstream for a particular cache item.  Others will wait for a response
to appear in cache (or cache lock released) up to proxy_cache_lock_timeout.

Waiting requests will recheck if they have cached response ready (or are
allowed to run) every 500ms.

Note: we intentionally don't intercept NGX_DECLINED possibly returned by
ngx_http_file_cache_read().  This needs more work (possibly safe, but needs
further investigation).  Anyway, it's exceptional situation.

Note: probably there should be a way to disable caching of responses
if there is already one request fetching resource to cache (without waiting
at all).  Two possible ways include another cache lock option ("no_cache")
or using proxy_no_cache with some supplied variable.

Note: probably there should be a way to lock updating requests as well.  For
now "proxy_cache_use_stale updating" is available.
2011-12-26 11:15:23 +00:00
Maxim Dounin
614a7ef18d Cache: only complain on long locked entries.
There have been multiple reports of cases where a real locked entry was
removed, resulting in a segmentation fault later in a worker which locked
the entry.  It looks like default inactive timeout isn't enough in real
life.

For now just ignore such locked entries, and move them to the top of the
inactive queue to allow processing of other entries.
2011-12-12 10:49:14 +00:00
Maxim Dounin
64a9f70092 Cache: handling of cache files with long headers.
There are two possible situations which can lead to this: response was
cached with bigger proxy_buffer_size value (and nginx was restared since
then, i.e. shared memory zone content was lost), or due to the race in
the cache update code (see [1]) we've end up with fcn->body_start from
a different response stored in shared memory zone.

[1] http://mailman.nginx.org/pipermail/nginx-devel/2011-September/001287.html
2011-12-12 10:47:48 +00:00
Maxim Dounin
b839e6febb Cache: obsolete code removed.
The ngx_http_cache() and ngx_http_no_cache_set_slot() functions were replaced
by ngx_http_test_predicates() and ngx_http_set_predicate_slot() in 0.8.46 and
no longer used since then.
2011-12-12 10:46:43 +00:00
Maxim Dounin
84c58a30e6 Cache: fix for sending of empty responses.
Revert wrong fix for empty responses introduced in 0.8.31 and apply new one,
rewritten to match things done by static module as close as possible.
2011-09-27 11:15:35 +00:00
Igor Sysoev
8b2bf08165 Cache size accounting fix: actual cache size on disk was less than
needed by sum of sizes of files loaded by worker processes themselves
while cache loader was running.

The bug has been introduced in r3900.
2011-08-25 17:29:34 +00:00
Igor Sysoev
0d18687b03 The change in adaptive loader behaviour introduced in r3975:
now cache loader processes either as many files as specified by loader_files
or works no more than time specified by loader_threshold during each iteration.

loader_threshold was previously used to decrease loader_files or
to increase loader_timeout and this might eventually result in
downgrading loader_files to 1 and increasing loader_timeout to large values
causing loading cache for forever.
2011-08-22 10:16:49 +00:00
Igor Sysoev
922f106582 loader_files, loader_sleep, and loader_threshold 2011-07-29 15:09:02 +00:00
Igor Sysoev
d5a25a4b97 set correct configuration file values while adding path
patch by Maxim Dounin
2011-07-25 10:22:38 +00:00
Igor Sysoev
1850ce09a1 rename ngx_http_file_cache_manager_sleep() to ngx_http_file_cache_loader_sleep()
and do not use it all in cache manager: this is a vestige of the times when
cache manager loaded cache
2011-07-25 09:45:00 +00:00
Igor Sysoev
d8ce166e2e do not close connection if cache file is too small: replace it with valid one 2011-07-24 20:04:47 +00:00
Igor Sysoev
c2cafadfa5 elimination of reading cache files by cache loader 2011-07-24 19:46:53 +00:00
Igor Sysoev
92161bac79 fix r3968 2011-07-24 19:43:52 +00:00
Igor Sysoev
6cb6fc4a09 update the previous commit:
removing dependencies on file uniq since WIN32_FIND_DATA has no such field
2011-07-24 17:36:26 +00:00
Igor Sysoev
c1250b6a49 The cache loader performs two tasks: inserting cache objects in inactivity
list and evaluating total cache size. Reading just directory is enough for
this purpose. Elimination of reading cache files saves at least one disk I/O
operation per file.

Preparation for elimination of reading cache files by cache loader:
removing dependencies on the reading:
*) cache node valid_sec and valid_msec are used only for caching errors;
*) upstream buffer size can be used instead of cache node body_start.
2011-07-24 16:47:31 +00:00
Igor Sysoev
bb389f6111 fuse two if's in one condition 2011-07-24 16:10:06 +00:00