Commit Graph

793 Commits

Author SHA1 Message Date
Bruce
5428229347
fix file read crash (#6021) 2024-09-14 08:33:35 -07:00
chrislu
6063a889ed 3.73 2024-09-12 13:11:55 -07:00
Eugeniy E. Mikhailov
dab0bb8097
Feature limit caching to prescribed number of bytes per file (#6009)
* feature: we can check if a fileId is already in the cache

We using this to protect cache from adding the same needle to
the cache over and over.

* fuse mount: Do not start dowloader if needle is already in the cache

* added maxFilePartSizeInCache property to ChunkCache

If file very large only first maxFilePartSizeInCache bytes
are going to be put to the cache (subject to the needle size
constrains).

* feature: for large files put in cache no more than prescribed number of bytes

Before this patch only the first needle of a large file was intended for
caching. This patch uses maximum prescribed amount of bytes to be put in
cache. This allows to bypass default 2MB maximum for a file part stored
in the cache.

* added dummy mock methods to satisfy interfaces of ChunkCache
2024-09-11 21:09:20 -07:00
Eugeniy E. Mikhailov
c04edeed68
bug fix in the data received from cache processing (#6002)
The patch addresses #3745.

The cache should return the exact amount of data requested by the buffer.
By construction of the cache it is always all requested data range
or we have error happening.

The old use of minsize miscalculate the requested data size,
if non zero offset is requested.
2024-09-10 13:30:18 -07:00
chrislu
18afdb15b6 Revert "weed mount, weed dav add option to force cache"
This reverts commit 7367b976b0.
2024-09-04 01:38:29 -07:00
chrislu
7367b976b0 weed mount, weed dav add option to force cache 2024-09-04 01:19:14 -07:00
chrislu
e50d85c0f3 3.72 2024-08-18 21:51:53 -07:00
chrislu
57dc39c451 randomizing next file handle id 2024-08-07 10:42:00 -07:00
chrislu
ed7e721efe 3.71 2024-07-21 21:03:27 -07:00
chrislu
4020149690 3.70 2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1 add version to filer 2024-07-21 20:13:51 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
chrislu
33b11f92bc Squashed commit of the following:
commit 465b47b8f72668dfc12eaa72befb232513c9e1db
Merge: f39272b54 9c440d472
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:56:51 2024 -0700

    Merge branch 'master' into merged-persisted-logs

commit f39272b5416bdd4d1714ce3e99aded0fe62e7d89
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:49:26 2024 -0700

    merge meta logs from multiple files
2024-07-11 00:04:24 -07:00
chrislu
114ac39917 add a simple test 2024-07-07 23:44:36 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
chrislu
fbe1316c92 3.69 2024-06-30 23:08:49 -07:00
chrislu
c030cb3ce9 bootstrap filer from one peer 2024-06-28 14:57:20 -07:00
chrislu
a9cd9b0542 3.68 2024-06-09 21:10:46 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
d3032d1e80 3.67 2024-05-07 08:47:30 -07:00
Patrick Schmidt
96c48bd0da
Fix deadlock in lock table locks (#5566) 2024-05-07 05:44:53 -07:00
chrislu
3682eb929f 3.66 2024-05-05 22:23:18 -07:00
chrislu
a5645d50a7 ensure head index is within range 2024-05-05 12:20:08 -07:00
chrislu
8948fb51e3 Merge branch 'master' into mq-subscribe 2024-04-25 23:42:19 -07:00
M@
abf01a0eb7
Fixes unlocked read from logBuffer.LastTsNs that is racey. (#5536) 2024-04-25 15:46:12 -07:00
Konstantin Lebedev
2f3fee9bb9
filer read chunk retry if status code 499 (#5528) 2024-04-24 06:51:05 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
chrislu
0edc5aad9a 3.65 2024-04-14 20:15:46 -07:00
chrislu
e53a972483 Merge branch 'master' into mq-subscribe 2024-04-11 19:48:00 -07:00
Konstantin Lebedev
5189a09de0
[volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
chrislu
cc1c69f312 Merge branch 'master' into mq-subscribe 2024-04-08 11:03:14 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server (#5480) 2024-04-08 07:27:00 -07:00
clonefetch
9e07a87fcb
chore: fix function names in comment (#5478) 2024-04-08 07:19:02 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
chrislu
d935f70e3c shutdown follower when leader stops 2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1 send flush message to follower before shutting down logBuffer 2024-04-01 16:01:26 -07:00
chrislu
03f18e43df clean up logs 2024-03-21 23:30:49 -07:00
chrislu
26e24db721 adjust log level 2024-03-21 23:27:19 -07:00
chrislu
5c0fcae039 clean up logs 2024-03-21 23:24:18 -07:00
chrislu
407a52fc93 adjust log level 2024-03-21 23:23:26 -07:00
chrislu
50023fe1cf adjust log level 2024-03-21 08:07:56 -07:00
chrislu
3c559dc294 fix test2 2024-03-16 23:48:31 -07:00
chrislu
b74e8082ba 3.64 2024-03-16 11:46:40 -07:00
chrislu
054d0b3487 wait for data 2024-03-16 11:14:14 -07:00
chrislu
f8957b84c1 rename to LastTsNs 2024-03-16 10:52:47 -07:00
chrislu
529fa8c4c8 go fmt 2024-03-16 10:51:47 -07:00
chrislu
059df3957d rename to lastFlushDataTime 2024-03-16 10:51:16 -07:00
chrislu
6f75a0af55 ask follower to follow 2024-03-10 14:34:28 -07:00
chrislu
28f8f33d6a include key in LogEntry 2024-03-07 10:53:30 -08:00