1
0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-31 05:27:51 +08:00
Commit Graph

804 Commits

Author SHA1 Message Date
chrislu
c9f3448692 ReadAt may return io.EOF t end of file
related to https://github.com/seaweedfs/seaweedfs/issues/6219
2024-11-21 00:37:38 -08:00
chrislu
7b3c0e937f 3.80 2024-11-18 09:48:33 -08:00
Max Denushev
a5fe6e21bc
feat(filer.backup): add ignore errors option ()
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-11-14 08:40:55 -08:00
Chris Lu
dc784bf217
merge current message queue code changes ()
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
Bruce
0060a2cf9c
Fix 6181/6182 ()
* set larger buf size for LogBuffer

* jump to next day when no more entry found

* Update weed/filer/filer_notify_read.go

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-10-31 08:40:05 -07:00
chrislu
228946369c 3.79 2024-10-28 19:44:39 -07:00
chrislu
ebbb35d533 3.78 2024-10-27 17:44:22 -07:00
chrislu
b28b1a3402 3.77 2024-10-10 01:28:06 -07:00
chrislu
82ed61c655 3.76 2024-10-06 19:37:03 -07:00
chrislu
117c3aebda 3.75 2024-10-03 08:33:35 -07:00
chrislu
e3fa87bcc1 3.74 2024-09-30 11:29:01 -07:00
Bruce
5428229347
fix file read crash () 2024-09-14 08:33:35 -07:00
chrislu
6063a889ed 3.73 2024-09-12 13:11:55 -07:00
Eugeniy E. Mikhailov
dab0bb8097
Feature limit caching to prescribed number of bytes per file ()
* feature: we can check if a fileId is already in the cache

We using this to protect cache from adding the same needle to
the cache over and over.

* fuse mount: Do not start dowloader if needle is already in the cache

* added maxFilePartSizeInCache property to ChunkCache

If file very large only first maxFilePartSizeInCache bytes
are going to be put to the cache (subject to the needle size
constrains).

* feature: for large files put in cache no more than prescribed number of bytes

Before this patch only the first needle of a large file was intended for
caching. This patch uses maximum prescribed amount of bytes to be put in
cache. This allows to bypass default 2MB maximum for a file part stored
in the cache.

* added dummy mock methods to satisfy interfaces of ChunkCache
2024-09-11 21:09:20 -07:00
Eugeniy E. Mikhailov
c04edeed68
bug fix in the data received from cache processing ()
The patch addresses .

The cache should return the exact amount of data requested by the buffer.
By construction of the cache it is always all requested data range
or we have error happening.

The old use of minsize miscalculate the requested data size,
if non zero offset is requested.
2024-09-10 13:30:18 -07:00
chrislu
18afdb15b6 Revert "weed mount, weed dav add option to force cache"
This reverts commit 7367b976b0.
2024-09-04 01:38:29 -07:00
chrislu
7367b976b0 weed mount, weed dav add option to force cache 2024-09-04 01:19:14 -07:00
chrislu
e50d85c0f3 3.72 2024-08-18 21:51:53 -07:00
chrislu
57dc39c451 randomizing next file handle id 2024-08-07 10:42:00 -07:00
chrislu
ed7e721efe 3.71 2024-07-21 21:03:27 -07:00
chrislu
4020149690 3.70 2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1 add version to filer 2024-07-21 20:13:51 -07:00
vadimartynov
86d92a42b4
Added tls for http clients ()
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce () 2024-07-16 09:15:55 -07:00
chrislu
33b11f92bc Squashed commit of the following:
commit 465b47b8f72668dfc12eaa72befb232513c9e1db
Merge: f39272b54 9c440d472
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:56:51 2024 -0700

    Merge branch 'master' into merged-persisted-logs

commit f39272b5416bdd4d1714ce3e99aded0fe62e7d89
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:49:26 2024 -0700

    merge meta logs from multiple files
2024-07-11 00:04:24 -07:00
chrislu
114ac39917 add a simple test 2024-07-07 23:44:36 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const () 2024-07-01 01:00:39 -07:00
chrislu
fbe1316c92 3.69 2024-06-30 23:08:49 -07:00
chrislu
c030cb3ce9 bootstrap filer from one peer 2024-06-28 14:57:20 -07:00
chrislu
a9cd9b0542 3.68 2024-06-09 21:10:46 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
d3032d1e80 3.67 2024-05-07 08:47:30 -07:00
Patrick Schmidt
96c48bd0da
Fix deadlock in lock table locks () 2024-05-07 05:44:53 -07:00
chrislu
3682eb929f 3.66 2024-05-05 22:23:18 -07:00
chrislu
a5645d50a7 ensure head index is within range 2024-05-05 12:20:08 -07:00
chrislu
8948fb51e3 Merge branch 'master' into mq-subscribe 2024-04-25 23:42:19 -07:00
M@
abf01a0eb7
Fixes unlocked read from logBuffer.LastTsNs that is racey. () 2024-04-25 15:46:12 -07:00
Konstantin Lebedev
2f3fee9bb9
filer read chunk retry if status code 499 () 2024-04-24 06:51:05 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
chrislu
0edc5aad9a 3.65 2024-04-14 20:15:46 -07:00
chrislu
e53a972483 Merge branch 'master' into mq-subscribe 2024-04-11 19:48:00 -07:00
Konstantin Lebedev
5189a09de0
[volume] Reduce the number of buffers for uploading one chunk () 2024-04-11 04:47:21 -07:00
chrislu
cc1c69f312 Merge branch 'master' into mq-subscribe 2024-04-08 11:03:14 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server () 2024-04-08 07:27:00 -07:00
clonefetch
9e07a87fcb
chore: fix function names in comment () 2024-04-08 07:19:02 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
chrislu
d935f70e3c shutdown follower when leader stops 2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1 send flush message to follower before shutting down logBuffer 2024-04-01 16:01:26 -07:00
chrislu
03f18e43df clean up logs 2024-03-21 23:30:49 -07:00
chrislu
26e24db721 adjust log level 2024-03-21 23:27:19 -07:00