Commit Graph

1196 Commits

Author SHA1 Message Date
Numblgw
aebf3952b7
filer sync: source path and exclude path support dir suffix (#6268)
filer sync: source path and exclude paht support dir suffix

Co-authored-by: liguowei <liguowei@xinye.com>
2024-11-21 08:25:12 -08:00
zemul
6d3e38f439
Adjust threshold config (#6246)
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2024-11-17 20:51:26 -08:00
Max Denushev
a5fe6e21bc
feat(filer.backup): add ignore errors option (#6235)
* feat(filer.backup): add ignore errors option

* feat(filer.backup): fix 404 error wrap

* feat(filer.backup): fix wrapping function

* feat(filer.backup): fix wrapping errors in genProcessFunction

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

* Update weed/command/filer_backup.go

---------

Co-authored-by: Max Denushev <denushev@tochka.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-11-14 08:40:55 -08:00
chrislu
113c9ce6a8 remove the direct_io flag, as it is not well-supported on macOS 2024-11-07 14:18:59 -08:00
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
6c7824c813 consistent sorted tags elastic,gocdk,rclone,sqlite,tikv,ydb 2024-10-21 22:53:29 -07:00
chrislu
804e765397 remove unused 2024-10-13 23:12:10 -07:00
Guang Jiong Lou
6c986e9d70
improve worm support (#5983)
* improve worm support

Signed-off-by: lou <alex1988@outlook.com>

* worm mode in filer

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* move to fs configure

Signed-off-by: lou <alex1988@outlook.com>

* remove flag

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* support worm hardlink

Signed-off-by: lou <alex1988@outlook.com>

* update after review

Signed-off-by: lou <alex1988@outlook.com>

* typo

Signed-off-by: lou <alex1988@outlook.com>

* sync filer conf

Signed-off-by: lou <alex1988@outlook.com>

---------

Signed-off-by: lou <alex1988@outlook.com>
2024-09-16 21:02:21 -07:00
zouyixiong
881c9a009e
[master] master missing start LoopPushingMetric routine fixed. (#6018) 2024-09-13 20:01:34 -07:00
chrislu
d660d5c7d4 increasing default cache size 2024-09-10 10:30:19 -07:00
chrislu
eb02946c97 support write once read many
fix https://github.com/seaweedfs/seaweedfs/issues/5954
2024-09-04 02:25:07 -07:00
chrislu
18afdb15b6 Revert "weed mount, weed dav add option to force cache"
This reverts commit 7367b976b0.
2024-09-04 01:38:29 -07:00
chrislu
7367b976b0 weed mount, weed dav add option to force cache 2024-09-04 01:19:14 -07:00
chrislu
66ac82bb8f default cacheDirWrite to cacheDir 2024-09-04 00:05:58 -07:00
Shaun Alexander
0d435f9da4
filer.remote.gateway not filer.remote.sync for filer.remote.gateway help string. (#5935) 2024-08-23 20:42:59 -07:00
chrislu
4463296811 add parallel vacuuming 2024-08-21 22:53:54 -07:00
chrislu
2074e7db67 fix invalid toml 2024-08-08 00:03:42 -07:00
qinguoyi
bf2366ba3a
Fix KeyMaterial nil bug (#5849) 2024-08-02 11:06:48 -07:00
qinguoyi
e88da0be1b
Fix KeyMaterial nil bug (#5848) 2024-08-02 09:47:11 -07:00
qinguoyi
606d0cba64
Fix extractToFile gz file nil bug (#5847) 2024-08-02 09:46:37 -07:00
qinguoyi
b74b506e52
add command backup destory volume error log (#5846) 2024-08-02 00:04:57 -07:00
chrislu
d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
zuzuviewer
5ee50d26d8
fix CVE-2016-2183 (#5844)
* fix CVE-2016-2183

* allow tls.min_version and tls.max_version not set
2024-08-01 20:00:22 -07:00
Konstantin Lebedev
e1f9336688
[filer] fix panic filer on webdav.maxMB (#5822)
fix panic filer on webdav.maxMB
2024-07-27 13:40:22 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
19d14b4c71 minor 2024-07-16 09:21:01 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
chrislu
3a82f5ffad ensure metadata follow a specific folder
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
vadimartynov
ec9e7493b3
-metricsIp cmd flag (#5773)
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;

* Fixed cmd flags
2024-07-12 10:56:26 -07:00
vadimartynov
de60f383de
Added tls for http handlers (#5764)
* Added https handler for filer

* Added example for security.toml
2024-07-11 07:53:18 -07:00
chrislu
8943a572d8 log error message 2024-07-03 09:50:47 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
chrislu
ac6fd36c06 using fixed bootstrap from peer filer 2024-06-30 23:16:54 -07:00
Nikita Borzykh
64756c2d79
fix: correct command flag for master.raftBootstrap option in server.go (#5701) 2024-06-22 19:28:37 -07:00
chrislu
a829f36d17 adjust error handling
fix https://github.com/seaweedfs/seaweedfs/issues/5677
2024-06-17 10:59:41 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628)
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
MeenakshiSachdeva
d2516d99cc
Meenakshi fix filer sync (#5672)
* Fix Filer Sync Issue: 5455

* Fix Filer Sync Issue: 5455
2024-06-10 11:16:29 -07:00
MeenakshiSachdeva
1caa012a76
Fix Filer Sync Issue: 5455 (#5663) 2024-06-10 09:19:38 -07:00
sb
a3a00d9499
Feature/mongodb security (#5602) 2024-05-18 12:19:10 -07:00
sb
3fae87632f
add filer configuration options for username/password and tls (#5601) 2024-05-18 09:02:03 -07:00
Konstantin Lebedev
d389c5b27e
fix: recreate index include deleted files (#5579)
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508

* fix: counting the number of files

* fix: log
2024-05-12 11:31:34 -07:00
Nikita Korolev
2dd2bb3e16
fix filer address parsing (#5415) 2024-03-24 11:20:33 -07:00
Nikita Borzykh
1f3742850d
Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage (#5403) 2024-03-24 08:36:21 -07:00
Konstantin Lebedev
d821cb3b18
fix: sync without dir /buckets/some/.uploads/hash_hash (#5402) 2024-03-20 12:54:29 -07:00
Michał Frąckiewicz
205829fa22
Document default method used by compact (#5379) 2024-03-15 12:16:44 -07:00
rustrover
ab70aa92da
remove repetitive words (#5364) 2024-03-09 07:35:47 -08:00
Konstantin Lebedev
170b63d6f8
[filer.backup] add param uploader_part_size for S3sink (#5352)
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
2024-03-07 08:35:51 -08:00
Sean Ross
0833057503
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… (#5351)
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.
2024-02-29 02:16:28 -08:00
jerebear12
85d6d5371b
Disable filer UI in configuration (#5297)
* Add filer.ui.enabled configuration property

* Add filer.expose_directory_metadata to config

* Ammend commit

* Remove ShowUI reference

* Update all routes that allow directory metadata

* Add cmd flag to server.go
2024-02-27 08:38:55 -08:00
chrislu
bb867694d8 skip updating multipart .upload files 2024-01-18 09:24:52 -08:00