mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 00:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,052 Commits 32 Branches 296 Tags 152 MiB
7bd638de47
Commit Graph

7125 Commits

Author SHA1 Message Date
chrislu
54e9877093 Merge branch 'master' into mq-subscribe 2024-01-02 13:25:00 -08:00
chrislu
d3688938d9 avoid nil 
fix https://github.com/seaweedfs/seaweedfs/issues/5153

introduced from #5109
 2024-01-02 12:55:34 -08:00
chrislu
923b1e5dad Merge branch 'master' into mq-subscribe 2024-01-01 20:56:12 -08:00
chrislu
d4e91b6ad9 3.60 2024-01-01 19:11:23 -08:00
chrislu
8b4cd50aac log errors 2024-01-01 19:09:17 -08:00
chrislu
bf9ae74384 add logs 2024-01-01 19:09:05 -08:00
chrislu
32bc8d6a38 adjust wait time 2023-12-31 17:42:44 -08:00
chrislu
458ddbf919 clean up dead code 2023-12-31 13:25:33 -08:00
chrislu
2845230329 passing broker into the assignments 2023-12-28 21:13:49 -08:00
chrislu
093fdc1621 subscriber can get assignments 2023-12-28 20:35:15 -08:00
Varun Upadhyay
77626666c5
Minor cleanup & gitignore update (#5144) 2023-12-28 20:25:43 -08:00
chrislu
9ed26cd7b0 Merge branch 'master' into mq-subscribe 2023-12-28 11:57:30 -08:00
chrislu
c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency (#4529) 
* fix: increase speed cmd fs meta load

* fix: add wg
 2023-12-28 11:54:37 -08:00
Lars Lehtonen
edc0532b1d
weed/server: remove unneeded err from streamReadOneVolume() signature (#5143) 2023-12-28 11:45:13 -08:00
chrislu
bebbc9fe44 create volume grow request if the selected volume is close to full 2023-12-27 11:45:44 -08:00
chrislu
bb9269f3f7 Revert "Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134)" 
This reverts commit 838578b55f.
 2023-12-24 12:31:08 -08:00
chrislu
c6b1dc7058 remove unused code 2023-12-24 11:11:41 -08:00
vivekkoya
838578b55f
Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134) 2023-12-23 15:01:57 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck (#5133) 
* Add modifyTimeAgo to volume.fsck

* Fix AppendAtNs
 2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler (#5132) 2023-12-23 09:18:16 -08:00
chrislu
03c4b2e988 correct lock accounting 
possibly fix https://github.com/seaweedfs/seaweedfs/issues/5082
 2023-12-22 14:16:23 -08:00
chrislu
c4badf7396 adjust options 2023-12-22 11:41:10 -08:00
chrislu
6324062acc fix 2023-12-22 11:37:15 -08:00
chrislu
e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
chrislu
ee1c9bc314 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-22 11:33:02 -08:00
chrislu
034db049a0 comment fix 2023-12-22 11:33:00 -08:00
chrislu
6df44d3c95 option to filer.copy with public volume server url 
fix https://github.com/seaweedfs/seaweedfs/issues/5126
 2023-12-22 11:32:43 -08:00
chrislu
7a0d1e7fc1 minor 2023-12-22 11:29:54 -08:00
Farbod
c278f49bca
Using filer.remote.sync concurrency in filer.remote.gateway (#5123) 
- Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync

Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>
 2023-12-20 16:25:54 -08:00
jerebear12
06343f8976
Set allowed origins in config (#5109) 
* Add a way to use a JWT in an HTTP only cookie

If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.

* Added a way to specify allowed origins header from config

* Removed unecessary log

* Check list of domains from config or command flag

* Handle default wildcard and change name of config value to cors
 2023-12-20 16:21:11 -08:00
Konstantin Lebedev
3c9bcfb864
chore: add dsn for connection to mysql (#5060) 
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
 2023-12-20 16:20:58 -08:00
chrislu
915416a277 fix wrong parameter 
fix https://github.com/seaweedfs/seaweedfs/pull/5102
 2023-12-13 12:02:57 -08:00
Farbod
d5c01e5ceb
Remote gateway Bug fix (#5100) 2023-12-12 07:46:45 -08:00
chrislu
61b05fad95 fix test 2023-12-11 20:57:29 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
Konstantin Lebedev
125ad8fe63 falls back to update only if error contains msg "duplicate entry" 
https://github.com/seaweedfs/seaweedfs/issues/5062
 2023-12-09 10:18:18 -08:00
chrislu
c1a6e624e5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-07 08:13:46 -08:00
chrislu
5fe4c04a1a add missing error in the log 
related to https://github.com/seaweedfs/seaweedfs/issues/5084
 2023-12-07 08:13:42 -08:00
Lars Lehtonen
43a5973a66 weed/storage: fix dropped test errors 2023-12-06 12:05:22 -08:00
jerebear12
d5d9fbb8aa Add a way to use a JWT in an HTTP only cookie 
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
 2023-12-05 08:57:01 -08:00
chrislu
756bcc032d adjust logs 2023-11-27 12:57:29 -08:00
Konstantin Lebedev
dd78397fea logging PickForWrite error 
https://github.com/seaweedfs/seaweedfs/issues/3886
 2023-11-27 12:56:15 -08:00
chrislu
81f11883e3 go fmt 2023-11-26 11:47:20 -08:00
chrislu
7a50762b9e reduce binary size by skipping rclone 2023-11-23 00:48:27 -08:00
zehweh
d818ca9631 fix TestMisplacedChecking() and add test-case 2023-11-21 08:24:34 -08:00
julor
2fa37e0287 Update filer_server_handlers_write_autochunk.go 2023-11-20 11:17:14 -08:00
julor
9e03b916d8 verify content-md5 in header 
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
 2023-11-20 11:17:14 -08:00
SmsS4
6c1ce18541 Fix weed fix 2023-11-20 11:14:19 -08:00
Lars Lehtonen
57801687de weed/command: fix dropped error 2023-11-20 08:05:24 -08:00
Viktor Kuzmin
06e723f629 Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952 2023-11-17 07:38:14 -08:00
Konstantin Lebedev
2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
chrislu
17710e1ecb adjust logs 2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr
6f559fe838 fs.mergeVolumes: change dir to a named parameter 2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be Fix typo (dist => dest) 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3 fs.mergeVolumes: Make a plan based on volumes size 2023-11-13 06:41:24 -08:00
Konstantin Lebedev
e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev
3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev
5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev
7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr
b05d682c48 make help command in shell case insensitive 2023-11-13 06:06:45 -08:00
chrislu
27b34f3793 3.59 2023-11-12 22:12:08 -08:00
jsh
47112917ff ec.decode: mount the collected ec shards 2023-11-10 00:04:42 -08:00
chrislu
85bed4d92e Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-09 08:02:57 -08:00
chrislu
c563386675 rename parameters 2023-11-09 08:02:39 -08:00
Lars Lehtonen
5db82e594a weed/server: fix dropped webdav error 2023-11-07 19:05:21 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go (#4988) 2023-11-07 07:58:19 -08:00
Konstantin Lebedev
d693beea87
Replace an invalid method in a metric with an invalid one (#4987) 2023-11-06 10:51:47 -08:00
chrislu
a667f0ad99 fix error reporting 
related to https://github.com/seaweedfs/seaweedfs/issues/4978
 2023-11-05 07:17:16 -08:00
chrislu
b40ba7c7fe Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-01 16:28:05 -07:00
chrislu
310eec9a9c skip if path changed 
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
 2023-11-01 16:27:59 -07:00
Nico D'Cotta
3834ad5c48
refactor webdav subdirectory, fixes #4967 (#4969) 
* refactor webdav subdirectory, fixes #4967

* fix bug where Name() was not called on delegate wrappedFileInfo
 2023-11-01 13:12:49 -07:00
tired-engineer
4ed06e9ba5
Pass correct Filer's gRPC port to S3 server. (#4972) 
Fixes seaweedfs/seaweedfs#4971
 2023-11-01 10:11:41 -07:00
SmsS4
ba11331cc1
Fix twice increase metric counter (#4961) 2023-10-30 08:05:27 -07:00
SmsS4
f95848ba7d
Add ErrorGetNotFound and ErrorGetInternal to volume server metrics (#4960) 2023-10-30 07:38:03 -07:00
chrislu
d1e83a3b4d 3.58 2023-10-29 23:04:20 -07:00
duanhongyi
bf839651ec
Fix: issues #4950 (#4952) 2023-10-25 22:09:39 -07:00
zehweh
8b39bbbe2f
fix copying .vif file in VolumeCopy (#4943) 
closes #4934
fixes #2633
might fix #3528
 2023-10-24 09:54:24 -07:00
chrislu
2bc05f70e7 log full percentage 2023-10-22 12:59:34 -07:00
chrislu
612fd7206e fix 2023-10-19 08:10:33 -07:00
Konstantin Lebedev
6c82d7ddfc
[s3] do reload s3 static config (#4923) 
* do reload s3 config

* print error on reload s3 config

* print success msg

* Update weed/s3api/s3api_server.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-10-18 23:26:49 -07:00
lakeland1990
3df0e11ad3
remove compare to fix #4668 issue (#4924) 2023-10-18 09:06:56 -07:00
chrislu
cbc24c7b24 disconnect from old subscribers 2023-10-13 16:48:14 -07:00
James Hedley
6acb43bbbb
Add optional flags to enable mTLS with verification of client certificate (#4910) 2023-10-13 09:02:24 -07:00
chrislu
b62066d38f adding DiskType to /dir/status output 2023-10-12 21:39:11 -07:00
chrislu
deb86ff4a6 upgrading grpc 2023-10-12 21:38:34 -07:00
chrislu
de0b969b36 Revert "rename" 
This reverts commit 35b5264ab7.
 2023-10-12 20:28:11 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file (#4881) 
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero (#4909) 
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 08:15:20 -07:00
chrislu
35b5264ab7 rename 2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr
3e8604a4c5
add fs.mergeVolumes command into shell (#4907) 
* add fs.compact command into shell

* rename fs.compact to fs.mergeVolumes
 2023-10-10 12:47:18 -07:00
Konstantin Lebedev
2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment (#4893) 
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-09 09:57:26 -07:00
Yuval Yacoby
3fe00996b2
added healthz endpoint to filer (#4899) 2023-10-08 07:03:18 -07:00
Andrew Garrett
082f67bfe9
Fix filer.backup local sink to propagate file mode changes (#4896) 2023-10-06 05:40:20 -07:00
chrislu
0fd7222d65 default to skip if less than 4 nodes 2023-10-05 11:13:48 -07:00
chrislu
7b3c63a8af handle wrapped errors 2023-10-05 08:15:11 -07:00
chrislu
4e493d596c minor 2023-10-05 08:13:53 -07:00
Lars Lehtonen
530bdedf76
weed/operation: fix dropped error (#4888) 2023-10-03 10:55:20 -07:00
chrislu
6ebe26a765 Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)"" 
This reverts commit 567d788928.
 2023-10-03 08:28:52 -07:00
chrislu
7540d43ee9 Revert "Revert "fix compilation"" 
This reverts commit f9abfd0b03.
 2023-10-03 08:28:02 -07:00
chrislu
249c0e06ef Revert "fix compilation" 
This reverts commit 451ec6504d.
 2023-10-03 08:27:50 -07:00
chrislu
451ec6504d fix compilation 2023-10-03 08:15:18 -07:00
chrislu
f9abfd0b03 Revert "fix compilation" 
This reverts commit 0483ba3889.
 2023-10-03 07:58:11 -07:00
chrislu
0483ba3889 fix compilation 2023-10-02 11:55:50 -07:00
chrislu
567d788928 Revert "Revert "Add disk type to prometheus metrics" (#4777)" 
This reverts commit 9215ba24be.
 2023-10-02 11:49:54 -07:00
chrislu
b5070d6280 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-10-02 10:52:40 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
chrislu
bf13f3ced7 add subscriber coordinator 2023-10-02 01:02:27 -07:00
chrislu
734178093e refactor TopicPartition struct 2023-10-02 01:01:45 -07:00
chrislu
2a578b9033 refactor 2023-10-01 11:59:19 -07:00
chrislu
321fda552c comment 2023-10-01 11:36:27 -07:00
chrislu
b02fdeabff RetryForever => RetryUntil 2023-10-01 11:33:56 -07:00
chrislu
ebd6f96d35 adjust mq.proto 2023-09-30 21:47:57 -07:00
chrislu
02cb799481 add a test framework for grpc 2023-09-30 21:36:58 -07:00
chrislu
c4e6545197 fix compilation 2023-09-30 21:26:09 -07:00
chrislu
358cba43ef update proto generated files 2023-09-30 13:19:25 -07:00
chrislu
597ab1e4bf add design for mq balancer 2023-09-30 13:19:05 -07:00
chrislu
d7c211fe8c adjust mq proto 2023-09-30 13:18:49 -07:00
chrislu
40888b5612 align 64bit atomic operation 
fix https://github.com/seaweedfs/seaweedfs/issues/4878
 2023-09-28 07:58:43 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync (#4875) 
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-27 23:12:10 -07:00
Konstantin Lebedev
44906f1f3b
fix: avoid error file name too long when writing a file (#4876) 2023-09-27 05:40:51 -07:00
chrislu
9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica (#4873) 
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4 describe a topic 2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance (#4864) 
* fix(ec): ignore 0 byte data files

refers to parts of #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

* fix(ec): ignore volumes not from the current collection during balance

fixes #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

---------

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
 2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"" 
This reverts commit 8cb42c39
 2023-09-25 09:35:16 -07:00
chrislu
3d07895518 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-25 09:16:35 -07:00
chrislu
a6e35e4f68 fix compilation 2023-09-25 09:16:33 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) 
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-25 08:34:12 -07:00
chrislu
c9177c92e5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-25 08:20:16 -07:00
chrislu
81e57bd45e fix test 2023-09-25 08:20:03 -07:00
Lars Lehtonen
28a3a31b27
weed/replication/sub: fix dropped error (#4865) 2023-09-25 07:33:18 -07:00
chrislu
23f334dd13 comment 2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71 ensure latest stats are reported 2023-09-24 23:34:31 -07:00
chrislu
dff2ce5d2f read existing broker stats 2023-09-24 23:10:33 -07:00
chrislu
1492bf7552 fix listing topics 2023-09-24 23:05:41 -07:00
chrislu
3cf9b8d621 create topic and report topic 2023-09-24 22:00:43 -07:00
chrislu
b3f94feede do create topic 2023-09-24 22:00:43 -07:00
chrislu
d74348048a implement create topic 2023-09-24 22:00:43 -07:00
chrislu
0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu
0f8168c0c9 3.57 2023-09-24 21:29:31 -07:00
chrislu
1df4223195 minor 2023-09-24 09:54:57 -07:00
chrislu
10ac25f857 avoid deleting the entry too early 2023-09-23 12:43:00 -07:00
chrislu
c43238b30a fix waiting condition 2023-09-23 12:28:02 -07:00
chrislu
322af1fc04 fix compilation 2023-09-23 12:13:32 -07:00
chrislu
0414b68a4f if shared lock, still need to wait for exclusive lock to release 2023-09-23 12:08:23 -07:00
chrislu
9ba4639a27 adjust for better logs 2023-09-23 11:53:44 -07:00
chrislu
186b1f029b read source uses shared lock 2023-09-23 10:33:44 -07:00
chrislu
2df70ce95f adjust logs 2023-09-21 23:51:32 -07:00
chrislu
3e23421608 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-21 20:08:30 -07:00
chrislu
49a470bcdf reduce logs 2023-09-21 20:08:20 -07:00
Konstantin Lebedev
750a0ba1b2
[iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) 
Replace action read/write to readAcp/writeAcp for handlers with acl query
 https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 14:07:04 -07:00
chrislu
8b2c39f2c0 reduce binary size 2023-09-21 13:22:08 -07:00
chrislu
b388957340 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-21 11:08:30 -07:00
chrislu
31fc165715 mount switch to ordered lock requests 2023-09-21 11:08:26 -07:00
chrislu
7669852241 adjust logs 2023-09-21 11:04:12 -07:00
chrislu
e3b1bacf3f add lock table to ensure ordered locks 2023-09-21 10:24:34 -07:00
Konstantin Lebedev
a46f873edd
[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) 
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-21 08:20:05 -07:00
Konstantin Lebedev
d8b424d123
[s3] optimization iam lookup for reducing algorithm complexity (#4857) 
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 08:19:11 -07:00
chrislu
411bdda08d fix compilation 2023-09-19 16:22:41 -07:00
chrislu
d531b7cbbf merge some previous missing changes 2023-09-19 14:10:54 -07:00
chrislu
b18112ae10 fix compilation bugs during merge 2023-09-19 14:08:17 -07:00
chrislu
6c6655de9d wip 2023-09-19 14:03:42 -07:00
chrislu
cf28108ea4 fix grpd dial option 2023-09-19 14:02:51 -07:00
chrislu
27af11f1e8 Revert "Revert "Merge branch 'master' into sub"" 
This reverts commit 0bb97709d4.
 2023-09-18 18:47:34 -07:00
chrislu
0bb97709d4 Revert "Merge branch 'master' into sub" 
This reverts commit 4d414f54a2, reversing
changes made to 4827425146.
 2023-09-18 16:13:20 -07:00
chrislu
8cb42c39ad Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs" 
This reverts commit 2e5aa06026, reversing
changes made to 4d414f54a2.
 2023-09-18 16:12:50 -07:00
chrislu
2e5aa06026 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-18 16:06:41 -07:00
chrislu
4d414f54a2 Merge branch 'master' into sub 2023-09-18 16:04:59 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850) 
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* API changes

* go mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2023-09-18 14:43:05 -07:00
chrislu
b64fc44e8d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-18 09:01:17 -07:00
chrislu
340aea9b34 add broker leader 2023-09-18 09:01:14 -07:00
chrislu
0d390ec882 fix nil 2023-09-18 08:52:35 -07:00
Lars Lehtonen
d83107efe5 fix: weed/storage: dropped error 2023-09-18 08:18:57 -07:00
zemul
91cbaef02b fix: mount proxyByFiler write error: wrong jwt 2023-09-18 07:44:50 -07:00
chrislu
4e17d399a6 refactor 2023-09-17 00:04:20 -07:00
chrislu
7a49d3541a avoid nil 2023-09-17 00:04:13 -07:00
chrislu
a65088d739 broker report stats to balancer 2023-09-16 23:00:51 -07:00
chrislu
a1cb49d078 adjust log 2023-09-16 23:00:28 -07:00
chrislu
b590a68e75 revert to non-streaming mode to assign file id 
fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685
 2023-09-16 16:13:08 -07:00
chrislu
4650c4c65f Revert "turn on streaming assign file id" 
This reverts commit 733db2bc88.
 2023-09-16 16:09:20 -07:00
chrislu
fa62926347 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-16 15:06:22 -07:00
chrislu
89a1fd1751 Squashed commit of the following: 
commit 4827425146
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Sep 16 15:05:38 2023 -0700

    balancer works

commit 3b50139f68
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:22:32 2023 -0700

    comments

commit 7f685ce7ba
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 15 22:20:05 2023 -0700

    adjust APIs

commit 436d99443b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 14 23:49:05 2023 -0700

    receive broker stats

commit b771fefa37
Merge: 0a851ec00 890881037
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 13 00:03:47 2023 -0700

    Merge branch 'master' into sub

commit 0a851ec00b
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Sep 10 22:01:25 2023 -0700

    Create balancer.go

commit 39941edc0b
Author: chrislu <chris.lu@gmail.com>
Date:   Thu Sep 7 23:55:19 2023 -0700

    add publisher shutdown

commit 875f562779
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:16:41 2023 -0700

    server side send response at least once per second

commit 984b6c54cf
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 23:15:29 2023 -0700

    ack interval 128

commit 2492a45499
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Sep 6 22:39:46 2023 -0700

    ack interval

commit ba67e6ca29
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:50 2023 -0700

    api for sub

commit 9e4f985698
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Sep 4 21:43:30 2023 -0700

    publish, benchmark

commit cb470d44df
Author: chrislu <chris.lu@gmail.com>
Date:   Fri Sep 1 00:36:51 2023 -0700

    can pub and sub

commit 1eb2da46d5
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:02:12 2023 -0700

    connect and publish

commit 504ae8383a
Author: chrislu <chris.lu@gmail.com>
Date:   Mon Aug 28 09:01:25 2023 -0700

    protoc version

commit dbcba75271
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:59:04 2023 -0700

    rename to lookup

commit c9caf33119
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 18:33:46 2023 -0700

    move functions

commit 4d6c18d86f
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 17:50:59 2023 -0700

    pub sub initial tests

commit 4eb8e8624d
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:14:39 2023 -0700

    rename

commit 1990456670
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 27 13:13:14 2023 -0700

    sub

commit 905911853d
Author: chrislu <chris.lu@gmail.com>
Date:   Sat Aug 26 13:39:21 2023 -0700

    adjust proto
 2023-09-16 15:06:16 -07:00
chrislu
4827425146 balancer works 2023-09-16 15:05:38 -07:00
chrislu
3b50139f68 comments 2023-09-15 22:22:32 -07:00
chrislu
7f685ce7ba adjust APIs 2023-09-15 22:20:05 -07:00
chrislu
436d99443b receive broker stats 2023-09-14 23:49:05 -07:00
wang wusong
9e5b1eef9b avoid passe lock by value 2023-09-14 06:51:12 -07:00
chrislu
b771fefa37 Merge branch 'master' into sub 2023-09-13 00:03:47 -07:00
annluda
8908810376 Fix s3 api object list with params max-keys and prefix 2023-09-11 21:35:15 -07:00
chrislu
5407b86766 3.56 2023-09-10 22:05:41 -07:00
chrislu
07c2d75ac7 ensure single threaded access 2023-09-10 22:03:44 -07:00
chrislu
0a851ec00b Create balancer.go 2023-09-10 22:01:25 -07:00
chrislu
39941edc0b add publisher shutdown 2023-09-07 23:55:19 -07:00
chrislu
875f562779 server side send response at least once per second 2023-09-06 23:16:41 -07:00
chrislu
984b6c54cf ack interval 128 2023-09-06 23:15:29 -07:00
chrislu
2492a45499 ack interval 2023-09-06 22:39:46 -07:00
Lars Lehtonen
f24c7e803f weed/storage/erasure_coding: Close() after error handling 2023-09-06 07:35:54 -07:00
Lars Lehtonen
d762b12e11 weed/storage/erasure_coding: fix dropped test error 2023-09-06 07:35:54 -07:00
Lars Lehtonen
a879e1bbb5 weed/storage/erasure_coding: remove unused err from encodeDatFile() signature 2023-09-06 07:35:54 -07:00
Patrick Schmidt
98dcec0ee2 Clean up old signature hash pools 2023-09-05 10:33:27 -07:00
Patrick Schmidt
cdd817edf9 Improve S3 request signing performance 
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
 2023-09-05 10:33:27 -07:00
chrislu
ba67e6ca29 api for sub 2023-09-04 21:43:50 -07:00
chrislu
9e4f985698 publish, benchmark 2023-09-04 21:43:30 -07:00
Lars Lehtonen
e6385e5e8b
weed/util: fix dropped errors (#4803) 2023-09-01 13:10:45 -07:00
chrislu
cb470d44df can pub and sub 2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5 connect and publish 2023-08-28 09:02:12 -07:00
chrislu
504ae8383a protoc version 2023-08-28 09:01:25 -07:00
chrislu
dbcba75271 rename to lookup 2023-08-27 18:59:04 -07:00
chrislu
c9caf33119 move functions 2023-08-27 18:33:46 -07:00
chrislu
4d6c18d86f pub sub initial tests 2023-08-27 17:50:59 -07:00
chrislu
4eb8e8624d rename 2023-08-27 13:14:39 -07:00
chrislu
1990456670 sub 2023-08-27 13:13:14 -07:00
chrislu
905911853d adjust proto 2023-08-26 13:39:21 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer (#4767) 2023-08-24 07:08:56 -07:00
chrislu
733db2bc88 turn on streaming assign file id 2023-08-23 00:34:21 -07:00
chrislu
99f037b958 streaming assign file ids 2023-08-23 00:31:33 -07:00
chrislu
94b7e2a37c add stream assign server side implementation 2023-08-22 09:59:04 -07:00
chrislu
ccedad5196 refactor files 2023-08-22 09:54:06 -07:00
chrislu
010c5e91e3 add stream assign proto 2023-08-22 09:53:54 -07:00
chrislu
a1149f060b upgrade jwt 2023-08-22 08:26:31 -07:00
chrislu
c49caff5a1 change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims 2023-08-21 23:42:39 -07:00
chrislu
c45665eb97 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-08-21 09:47:36 -07:00
SmsS4
f61490966f
Add time to first byte metric for s3 (#4768) 
* Add time to first byte metric for s3

* Change second to millisecond
 2023-08-21 00:42:39 -07:00
chrislu
01d70c21f3 Squashed commit of the following: 
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:52:19 2023 -0700

    fix compilation

commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 22:46:44 2023 -0700

    pub

commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 13:27:39 2023 -0700

    fix generic type

commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 12:47:14 2023 -0700

    Merge branch 'master' into pubsub

commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21
Author: chrislu <chris.lu@gmail.com>
Date:   Sun Aug 20 11:10:58 2023 -0700

    wip

commit f8b00980bc2f3879bb43decffd9a08d842f196f2
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:35 2023 -0700

    add design document

commit 08d2bebe42a26ebc39f1542f54d99e73620727dd
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:14:06 2023 -0700

    minor

commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da
Author: chrislu <chris.lu@gmail.com>
Date:   Tue Jul 25 09:13:49 2023 -0700

    rename
 2023-08-20 22:53:05 -07:00
chrislu
16e3f2d528 fix log buffer test 2023-08-20 12:46:15 -07:00
chrislu
3e9c32a3f0 refactoring to use atomic bool 
fix compilation
 2023-08-20 12:13:06 -07:00
SmsS4
1e72579ac7
Fix s3.clean.uploads (#4766) 2023-08-19 06:30:18 -07:00
chrislu
6792db21dc ErrInvalidPart when completed parts has extra parts 
fix https://github.com/seaweedfs/seaweedfs/discussions/4746
 2023-08-17 00:51:42 -07:00
chrislu
f5c4b642bb re-ordering 2023-08-16 23:54:23 -07:00
chrislu
3852307e94 renaming 2023-08-16 23:47:43 -07:00
chrislu
6c7fa567d4 add separate cache directory for write buffers 2023-08-16 23:39:21 -07:00
SmsS4
ea1b259c12
Update weed fix description (#4745) 2023-08-13 03:50:53 -07:00
Nikita Mochalov
4b1ba7f5b2
Save disk space metrics immediately (#4740) 2023-08-10 06:52:32 -07:00
Nikita Mochalov
e6a49dc533
Fix resource leaks (#4737) 
* Fix division by zero

* Fix file handle leak

* Fix file handle leak

* Fix file handle leak

* Fix goroutine leak
 2023-08-09 15:30:36 -07:00
chrislu
3365468d0d added an error message 2023-08-08 20:35:21 -07:00
wusong
3ec977cd6c
minor (#4723) 2023-08-01 02:34:59 -07:00
chrislu
c09100f928 fix building mongo index 2023-07-24 11:09:51 -07:00
chrislu
7eafa3420b 3.55 2023-07-23 23:11:37 -07:00
SmoothDenis
51bcc219ea
s3api should return 500 code from filer (#4699) 2023-07-22 07:22:38 -07:00
wusong
33b9192536
Fix large file chunks deleted by mistake (#4678) 2023-07-17 07:52:14 -07:00
chrislu
358b3a4894 3.54 2023-07-16 22:47:57 -07:00
chrislu
6f588b5b18 fix refactoring mistake 
fix https://github.com/seaweedfs/seaweedfs/issues/4639
 2023-07-16 11:22:48 -07:00
mervynzhang
df400e6c71
Concurrency works better (#4663) 
Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>
 2023-07-12 23:04:54 -07:00
wusong
0cb9ddd8ec
Fix data loss: add lock for metacache (#4664) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-07-11 22:23:32 -07:00
Henco Appel
89542db2d0
Fix S3 API multipart upload on Windows (#4666) 
The file paths were being having '/' changed to '\' by filepath.Dir()
resulting in a file being created with '\' separators, but when trying
to read the files, the same wasn't happening.

Co-authored-by: Hendrik Appel <happel@europe.altair.com>
 2023-07-11 09:05:14 -07:00
Henco Appel
680c2e76f1
Fix the segfault from not setting s3 https port (#4667) 2023-07-11 08:03:20 -07:00
Konstantin Lebedev
2c076ebd96
avoid logging nil error (#4659) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-07-10 23:38:52 -07:00
LHHDZ
3730772e05
refresh max volume count after adjust (#4662) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-07-10 23:37:33 -07:00
chrislu
c278bac263 avoid nil needle map 
fix https://github.com/seaweedfs/seaweedfs/issues/4640
 2023-07-07 22:16:58 -07:00
chrislu
0606b59a3e mount: create mount root on filer 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/127
 2023-07-07 09:30:32 -07:00
chrislu
14546f0ad8 minor 2023-07-06 09:12:28 -07:00
chrislu
ab30f489c3 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-07-06 00:33:02 -07:00
chrislu
31b2751aff clone volume locations in case they are changed 
fix https://github.com/seaweedfs/seaweedfs/issues/4642
 2023-07-06 00:32:58 -07:00
wusong
61553beba2
Fix DataBackend nil pointer (#4641) 2023-07-04 23:22:10 -07:00
chrislu
a315490f7d proxy to master uses http address 
fix https://github.com/seaweedfs/seaweedfs/issues/4607
 2023-07-04 11:45:21 -07:00
chrislu
2aa59ab37c fix copying level db files during commi 
fix https://github.com/seaweedfs/seaweedfs/issues/4635
 2023-07-04 11:28:12 -07:00
wusong
c44a6552d2
Fix buff use (#4628) 2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr
f0d3c874b6
Unix Socket listener for S3 server (#4621) 
Add localSocket option to s3 server
 2023-06-26 16:22:45 -07:00
chrislu
2c4c2f0994 3.53 2023-06-25 22:51:14 -07:00
chrislu
f193942adc lock on append metadata operation 2023-06-25 22:07:23 -07:00
chrislu
61c42f9991 adjust lock APIs 2023-06-25 20:30:20 -07:00
chrislu
aafb874ad2 lock until cancelled 2023-06-25 19:31:25 -07:00
chrislu
c4f44d56f7 add distributed lock client 2023-06-25 17:38:34 -07:00
chrislu
f8aa5ea844 adjust filer.proto 2023-06-25 17:38:19 -07:00
chrislu
13a8593ab9 adjust errors 2023-06-25 17:37:54 -07:00
chrislu
06471dac9d init lock ring 2023-06-25 15:28:16 -07:00
chrislu
5149b3d07b filer can proxy to peer filer holding the lock 2023-06-25 15:28:01 -07:00
chrislu
868f7875d7 refactor 2023-06-25 14:30:58 -07:00
chrislu
26eff062c8 refactor 2023-06-25 14:17:58 -07:00
chrislu
464a71a373 add distributed lock manager 2023-06-25 14:14:40 -07:00
chrislu
ee4f7cd636 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-06-25 00:58:24 -07:00
chrislu
3fd659df2a add distributed lock manager 2023-06-25 00:58:21 -07:00
Damiano Albani
0d9ba8c612
Support templating name of files stored via Rclone backend (#4606) 2023-06-24 07:08:24 -07:00
chrislu
a82c44972b refactor 2023-06-20 00:12:59 -07:00
chrislu
14a19170e2 refactor 2023-06-19 23:51:04 -07:00
chrislu
84ced2abf5 add LockRing 2023-06-19 23:48:33 -07:00
chrislu
fe1716ccdf remove deprecated tests 2023-06-19 18:43:51 -07:00
chrislu
1ce725728d minor 2023-06-19 18:43:41 -07:00
chrislu
11549706ef refactor 2023-06-19 18:21:07 -07:00
chrislu
8ec1bc2c99 remove unused cluster node leader 2023-06-19 18:19:13 -07:00
Bai Jie
3b88ab42aa
remove duplicate fileCount query (#4588) 2023-06-18 00:14:14 -07:00
Bai Jie
44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil (#4587) 2023-06-18 00:13:40 -07:00
chrislu
94fbf02ea2 generate jwt token when looking up needle id 
fix https://github.com/seaweedfs/seaweedfs/issues/4577
 2023-06-17 00:11:23 -07:00
柏杰
0b0fb9b9e4
avoid data race read volume.IsEmpty (#4574) 
* avoid data race read volume.IsEmpty

-   avoid phantom read isEmpty for onlyEmpty
-   use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope

* add Destroy(onlyEmpty: true) test

* add Destroy(onlyEmpty: false) test

* remove unused `IsEmpty()`

* change literal `8` to `SuperBlockSize`
 2023-06-14 14:39:58 -07:00
Konstantin Lebedev
1e22d5caf2
fix get file stats for IsEmpty (#4576) 2023-06-14 01:43:30 -07:00
Konstantin Lebedev
4dd890d4a2
optional https port for s3 (#4482) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-13 23:58:49 -07:00
chrislu
3fbf4f6189 Revert "Initial patch for redis version 7 support (#4572)" 
This reverts commit a25bca0692.
 2023-06-12 22:27:38 -07:00
Konstantin Lebedev
4527ead295
fix from comment delete volume is empty (#4573) 
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561

* fix tests

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 22:22:46 -07:00
Aaron Gipson
a25bca0692
Initial patch for redis version 7 support (#4572) 2023-06-12 13:30:03 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty (#4561) 
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559

* fix IsEmpty

* fix test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-12 10:42:44 -07:00
柏杰
68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560) 2023-06-10 23:15:27 -07:00
Konstantin Lebedev
5ee04d20fa
Healthz check for deadlocks (#4558) 2023-06-09 09:42:48 -07:00
wusong
d6af3a8eb6
fix 404 not found (#4536) 
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-06 09:14:49 -07:00
Konstantin Lebedev
5caff2c0f0
fix: deletes newly created volumes (#4551) 
https://github.com/seaweedfs/seaweedfs/issues/4531

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-06 09:08:27 -07:00
wusong
9bdbf9c880
revert #4491 (#4550) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-06 00:17:51 -07:00
chrislu
0bffb3770f support TTL when creating folders 
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
 2023-06-05 23:29:25 -07:00
wusong
26f15d0079
Fix no more writable volumes by delay judgment (#4548) 
* fix nomore writables volumes while disk free space is sufficient by time delay

* reset

---------

Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-05 10:17:21 -07:00
chrislu
fb4b61036c 3.52 2023-06-04 22:34:26 -07:00
Konstantin Lebedev
a0931be0c0
S3 TLS credentials Refreshing (#4506) 
* S3 TLS credentials Refreshing

* fix: logging

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-06-04 14:27:56 -07:00
wusong
5aec6da8a3
fix deadlock for filer upload (#4527) 
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-06-03 13:38:27 -07:00
chrislu
e23f3d6eca set df.File to nil after it is closed 
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
 2023-06-02 17:16:45 -07:00
chrislu
ca7cc61319 reduce the window size between unregistering a volume server and creating volumes on that server 
fix https://github.com/seaweedfs/seaweedfs/issues/4467
 2023-05-30 08:20:00 -07:00
chrislu
68c3cf767c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-29 16:45:28 -07:00
chrislu
d1d1757520 adjust error message 2023-05-29 16:45:25 -07:00
wusong
e7f4418855
add return for error handle (#4502) 2023-05-24 07:45:06 -07:00
chrislu
4310e1fac4 3.51 2023-05-21 23:43:39 -07:00
wusong
8fffe3e822
fix no more writables volumes while disk free space is sufficient (#4491) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-05-21 22:18:50 -07:00
mervynzhang
1ebb549f77
support swift (#4480) 2023-05-19 06:39:25 -07:00
Mesar Hameed
a90b777ff4
Feat: etcd filer store keys should have customizable prefix (#4484) 
An etcd cluster is not necessarily only dedicated to seaweedfs.
This security enhancement adds a customizable key_prefix option to the etcd filer store.
This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the
key_prefix, instead of all keys on the etcd cluster.
 2023-05-18 23:08:56 -07:00
SmsS4
17e91d2917
Use filerGroup for s3 buckets collection prefix (#4465) 
* Use filerGroup for s3 buckets collection prefix

* Fix templates

* Remove flags

* Remove s3CollectionPrefix
 2023-05-16 09:39:43 -07:00
chrislu
b7f011f777 fix test 2023-05-15 17:46:56 -07:00
Mesar Hameed
65484e80ef
Feat: support username/password authentication for etcd filer store s… (#4477) 
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262

Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
 2023-05-15 17:45:14 -07:00
chrislu
9204ee2d2d 3.50 2023-05-11 08:03:11 -07:00
chrislu
d999f1f0e2 update raft version 
fix #4460
 2023-05-09 22:54:23 -07:00
chrislu
25b237b83c reset wait time on a successful retry 2023-05-07 23:39:33 -07:00
chrislu
59f55c1a66 3.49 2023-05-07 14:00:19 -07:00
chrislu
b2a3860290 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-05-07 13:33:47 -07:00
chrislu
4511edc871 update raft 2023-05-07 13:33:44 -07:00
themarkchen
7592d013fe
fix shell volume.balance bug (#4447) 2023-05-04 07:35:37 -07:00
chrislu
42766bccc8 3.48 2023-04-30 22:20:18 -07:00
zemul
e9fda774f4
[Filer] post add param:saveInside (#4434) 
* fix:mount deadlock

* feat: filer http upload to metadata

* feat: /etc save inside

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-29 08:31:05 -07:00
wusong
4867aa03ab
Fix: http rename move dir to subdir (#4432) 
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2023-04-26 20:54:36 -07:00
Konstantin Lebedev
d75a7b7f62
allow deleting only older empty dir without recursion (#4430) 2023-04-25 08:31:14 -07:00
zemul
f352616b7d
Volume range read use sync.pool (#4422) 2023-04-24 07:00:48 -07:00
chrislu
18686b7375 3.47 2023-04-23 22:45:41 -07:00
zemul
68f1cc34a5
[mount]Fix copy file range (#4407) 
* fix:mount deadlock

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-23 22:34:38 -07:00
chrislu
e1b92bc701 reset local file content if remote file is updated 
fix https://github.com/seaweedfs/seaweedfs/issues/4289
 2023-04-22 22:32:21 -07:00
chrislu
ff7b6d779e avoid overwriting variables 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-22 11:49:59 -07:00
wusong
2e240704ab
Writables inconsistency (#4417) 
fix: inconsistent read and write permissions between master and volume server

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-21 00:14:41 -07:00
wusong
19245dde50
mount: add retry for read only case (#4416) 
* mount: add retry for read only case

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* add new util retry function for mount

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

* change error list param

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>

---------

Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-21 00:09:31 -07:00
SmsS4
4131874fa8
use env for flags wtih dot in name (#4415) 2023-04-20 09:33:27 -07:00
Damiano Albani
9f55c7c90e
Rclone storage backend (#4402) 
* Add Rclone storage backend

* Support templating the name of files stored via Rclone

* Enable Rclone accounting

* Remove redundant type conversion

* Provide progress information for Rclone download/upload operations

* Log error when Rclone can't instantiate filesystem

* Remove filename templating functionality for Rclone storage

To (maybe) be later reintroduced as a generic functionality for all
storage backends.

* Remove S3 specific check

* Move Rclone config initialisation to init() method
 2023-04-18 14:12:12 -07:00
wusong
a408b46d95
compilation fail (#4414) 
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
 2023-04-18 00:36:55 -07:00
chrislu
0a22eea55d collect ec shard from multiple locations 
fix https://github.com/seaweedfs/seaweedfs/issues/4365
 2023-04-17 22:56:21 -07:00
chrislu
ae9388723f adjust error message 2023-04-17 22:01:39 -07:00
chrislu
8ecdf958ab 3.46 2023-04-16 21:57:43 -07:00
Chengyu Liu
0f653d2153
update fuse client log which should output the fh of FileHandle. (#4404) 2023-04-15 22:15:16 -07:00
zemul
0122e022ea
Mount concurrent read (#4400) 
* fix:mount deadlock

* feat: concurrent read

* fix

* Remove useless code

* fix

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-04-13 22:32:45 -07:00
Konstantin Lebedev
5614ad0000
fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399) 
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter

* fix list with delimiter and start token

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 16:53:49 -07:00
Konstantin Lebedev
88b2bbadd1
try fix s3test test_bucket_listv2_delimiter_prefix (#4396) 2023-04-12 08:47:20 -07:00
Konstantin Lebedev
095cdb0c7f
fix test prefix is a directory (#4393) 
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-12 00:08:13 -07:00
Konstantin Lebedev
44ad07276c
s3 fix get list of dir object key with slash suffix (#4391) 
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086

* list only entry dir eq prefix

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-04-11 23:52:35 -07:00
Konstantin Lebedev
130bc3e668
s3 fix get fake dir object key (#4390) 2023-04-11 07:36:22 -07:00
Matt
da1d3b5a33
adjusted regex to be from 1 to 255 for the value (#4377) 
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation

* adjusted regex to be from 1 to 255 for the value

* added better error message

* fixed regex
 2023-04-10 12:24:38 -07:00
Matt
2d213d87e1
File Path Configuration TTL Validation (#4376) 
* compatibility patch for csi driver

* added namespace to all component parameters

* added namespace to all component parameters

* dereference in range

* added namespace to values.yml defaults

* added namespace to s3 component

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* added helm chart to github pages

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* push on all tags

* changed helm directory structure

* update charts location

* fixed dereference

* updated permissions

* updated permissions

* match current action schema

* added helm chart liniting CI

* modified chart list changed

* fixed nodejs warning

* standardized a few defaults

* added chart dirs

* lowerd period seconds for volume startup to test chart lint changes

* lowerd period seconds for volume startup to test chart lint changes

* test

* changed back

* adjustment

* debug ls statment

* removed change detection

* remvoed change detection

* always lint the charts

* added missing )

* fixed typo

* added spaces in front of all comments

* pdated values.yaml

* pdated values.yaml

* don't validate maintainers for now

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* update helm_ci.yml

* - adds helm chart lint on changes
- adds test helm chart install on k8s

* updated helm chart readme.md

* added artifact hub

* added artifact hub

* added ttl validation
 2023-04-04 22:07:17 -07:00
chrislu
efef6e94bf use UPSERT for postgres style databases 2023-04-04 11:52:36 -07:00
chrislu
422bfaed69 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-04-02 22:53:02 -07:00
chrislu
b9ab65dba5 3.45 2023-04-02 22:52:58 -07:00
wusong
2b82d0f871
style: clerical error (#4361) 2023-03-31 07:36:21 -07:00
LHHDZ
8e80ce9476
fix key corrupt when fs.configure copy path trie (#4353) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-03-29 11:29:44 -07:00
chrislu
3cc7b90414 print wait in progress 2023-03-26 12:22:48 -07:00
chrislu
f1f14e28bc adjust name 2023-03-26 12:17:23 -07:00
chrislu
d54798c3ae Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-03-25 22:03:00 -07:00
chrislu
6bfb91b539 update entry Attributes.Mtime = time.Now().Unix() 
fix https://github.com/seaweedfs/seaweedfs/issues/4337
 2023-03-25 22:02:57 -07:00
renweijun
bbd461b2c6
weed mount default EnableACL，Support chmod chown (#4335) 2023-03-25 04:47:53 -07:00
chrislu
5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Konstantin Lebedev
de4545c28b
add validate config for raft (#4332) 2023-03-21 06:36:13 -07:00
chrislu
adb90bd252 avoid lower casing the command 
fix https://github.com/seaweedfs/seaweedfs/pull/4321
 2023-03-19 21:20:46 -07:00
chrislu
5b43c4bb98 3.44 2023-03-19 20:27:55 -07:00
chrislu
50dc2fe96b cleaning variables 2023-03-19 18:48:40 -07:00
Konstantin Lebedev
29d1312625
Fs.verify.concurrency (#4293) 
* init

* refactor

* refactor print msg

* param just concurrency

* fix race

* next fix race

* reset variables before each run

* clean

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-03-19 18:38:57 -07:00
chrislu
784daea1fa fix volume not found if marked as read only 
fix https://github.com/seaweedfs/seaweedfs/issues/4088
 2023-03-19 18:30:13 -07:00
Stewart Miles
264be0d2d4
Retry until a leader is selected. (#4318) 
Fixes regression introduced in
https://github.com/seaweedfs/seaweedfs/pull/4313

Related to #4307
 2023-03-16 20:50:38 -07:00
LHHDZ
d695119073
The filer does not set defaultReplication to the defaultReplication o… (#4315) 
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-03-15 22:10:39 -07:00
Stewart Miles
57ab1f8516
Use exponential backoff to query leader. (#4313) 
`topology.Leader()` was using a backoff that typically
resulted in at least a 5s delay when initially starting
a master and raft server. This changes the backoff
algorithm to use exponential backoff starting with 100ms
and waiting up to 20s for leader selection.

Related to #4307
 2023-03-15 17:49:46 -07:00
Stewart Miles
dd71f54c6b
Fix -raftHashicorp and -raftBootstrap flag propagation. (#4309) 
`weed server` was not correctly propagating
`-master.raftHashicorp` and `-master.raftBootstrap` flags when
starting the master server.

Related to #4307
 2023-03-15 13:03:20 -07:00
Patrick Schmidt
71b33faef0
Skip parent directory creation in mount (#4310) 
A POSIX filesystem does not implicitly create parent directories when
they do not exist. Directories must be explicitly created and
permissions be set.

This also fixes a bug where asynchronous operations would create a
file in the filer before the parent directory was created. If the
file was a symlink or another special type of file the directory
would inherit that type and become unusable in the mounted FS.
 2023-03-15 12:37:25 -07:00
chrislu
f5854d13df fix test 2023-03-14 22:05:16 -07:00
chrislu
0454bb2d88 fix test 
when size is zero, the "LastModified" is not written
 2023-03-14 22:02:49 -07:00
chrislu
2268d38263 fix naming 
fix https://github.com/seaweedfs/seaweedfs/issues/4305
 2023-03-14 08:38:02 -07:00
chrislu
98bc93cf31 fix tests 2023-03-13 10:00:27 -07:00
Konstantin Lebedev
e17429223e
shell script unclean variables (#4298) 2023-03-13 07:16:31 -07:00
Thomas Anderson
bd70683871
fix(weed/topology/node.go): typo in error message (#4292) 2023-03-08 23:58:03 -08:00
Rohit Chormale
59706c89fb
#4270 set http status code to 409 if dir already exists (#4287) 2023-03-08 07:45:08 -08:00
chrislu
7111c08289 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-03-08 01:48:30 -08:00
chrislu
c589e9837c fix tests 2023-03-08 01:48:28 -08:00
Kevin Liu
244385bf0d
Fix binding metrics to ipv6 (#4286) 
* Fix binding metrics to ipv6

* Update weed/stats/metrics.go

---------

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-03-08 01:04:02 -08:00
zemul
bd6f2716d3
fix_read_tomb_needle_meta (#4263) 
Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:26:55 -08:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs (#4264) 
* fix

* fix mtime s > ns

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-03-02 08:24:36 -08:00
chrislu
8241c9a829 adjust ui pagination size 
fix https://github.com/seaweedfs/seaweedfs/issues/4258
 2023-02-28 08:31:26 -08:00
lfhy
1976ca9160
add -disk to filer command (#4247) 
* add -disk to filer command

* add diskType to filer.grpc

* use filer.disk when filerWebDavOptions.disk is empty

* add filer.disk to weed server command.

---------

Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
 2023-02-25 09:48:59 -08:00
chrislu
214b7cd286 volume.fix.replication: adjust the retry checking times 2023-02-22 10:47:52 -08:00
LHHDZ
db5515eada
Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists (#4240) 
Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2023-02-21 23:08:52 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency (#4239) 2023-02-21 19:47:33 -08:00
chrislu
3227e4175e 3.43 2023-02-20 16:56:05 -08:00
chrislu
9bb0a9e306 clean comments 2023-02-17 01:31:44 -08:00
chrislu
545d5d0cc3 fix for io.ReaderAt used in filer.remote.sync 
fix https://github.com/seaweedfs/seaweedfs/issues/4194
 2023-02-17 01:30:55 -08:00
wusong
79af6b41d4
[mount] Fix read file-data in entry.content (#4207) 2023-02-15 07:18:56 -08:00
chrislu
5869945f16 avoid infinite loop 
fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904
 2023-02-10 13:05:04 -08:00
chrislu
e037c71ec3 adjust text 2023-02-10 13:04:29 -08:00
chrislu
67b8c2853a add line return 2023-02-10 12:53:43 -08:00
chrislu
302155a6fa fix compilation 2023-02-10 12:32:59 -08:00
chrislu
5520b81551 add some error messages for args 
related to https://github.com/seaweedfs/seaweedfs/issues/4115
 2023-02-10 12:30:53 -08:00
chrislu
dc4ed2cd9b do not move cloud tier volumes 
fix https://github.com/seaweedfs/seaweedfs/issues/4195
 2023-02-09 22:17:54 -08:00
chrislu
31bb91583f fix bug when vid not found 
fix https://github.com/seaweedfs/seaweedfs/issues/4193
 2023-02-09 17:30:44 -08:00
chrislu
79d68a81fd text change 2023-02-07 15:00:41 -08:00
chrislu
999a126a3f typo 2023-02-07 14:59:47 -08:00
chrislu
1c2364936a detect invalid bucket name 
fix https://github.com/seaweedfs/seaweedfs/issues/4143
 2023-02-07 14:57:29 -08:00
chrislu
5de93fe442 refactoring 2023-02-07 14:45:20 -08:00
Eric Yang
5083429704
ADHOC: s3 bucket name verification (#4189) 
* ADHOC: s3 bucket name verification

* add test
 2023-02-07 14:37:29 -08:00
Neo
d5f77706a8
volume:fix return if Replicate write error (#4188) 2023-02-06 22:34:56 -08:00
chrislu
8821d6b161 3.42 2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region (#4177) 
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
 2023-02-05 21:24:21 -08:00
chrislu
84e9934bf9 fix filer.remote.sync on a S3 cloud mount 
fix https://github.com/seaweedfs/seaweedfs/issues/4175
 2023-02-01 20:44:00 -08:00
chrislu
0932437a1c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-02-01 02:38:09 -08:00
chrislu
c47f75eb5b fix null content type caused exception in hdfs client 
fix https://github.com/seaweedfs/seaweedfs/issues/4170
 2023-02-01 02:38:05 -08:00
wusong
fe573f26a8
memchunk save content twice (#4174) 2023-01-31 19:36:59 -08:00
Neo
b9b613a78e
filter system log dir does not make subscribe event (#4172) 2023-01-31 18:56:11 -08:00
wusong
665917fdf1
[mount]: delete redundant code (#4173) 2023-01-31 18:54:46 -08:00
a
0d085ec175
make directory lister trigger if mime type is httpd/unix-directory (#4169) 2023-01-31 06:54:40 -08:00
chrislu
4d71af87f3 3.41 2023-01-29 22:56:01 -08:00
chrislu
ef70415c70 skipping manifest chunk volume id change 2023-01-29 22:55:06 -08:00
chrislu
6e8dbf8868 add fs.meta.changeVolumeId 2023-01-29 22:51:58 -08:00
wusong
de081c0d64
[mount] fix metacache update (#4161) 2023-01-29 07:55:27 -08:00
Guo Lei
b63d1e9c6b
fix index (#4158) 2023-01-28 06:55:27 -08:00
chrislu
81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6 use one readerCache for the whole file 2023-01-16 22:43:02 -08:00
chrislu
4614e85efa adjust help message 2023-01-15 21:28:36 -08:00
chrislu
c24678153c use memory chunks only for sequential writes 2023-01-15 21:10:08 -08:00
chrislu
2885ba0e50 3.40 2023-01-15 20:12:05 -08:00
chrislu
12f8251d32 avoid index out of slice range 2023-01-14 13:31:22 -08:00
chrislu
313fd17ae4 allow random writes for mem chunk 2023-01-13 20:32:18 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload (#4130) 
* merge chunks during upload

* fix test
 2023-01-12 23:07:36 -08:00
chrislu
2452f3b2f7 sort chunks by offset 2023-01-12 20:29:07 -08:00
chrislu
361d97941b mysql: Missed escaping column name 
fix https://github.com/seaweedfs/seaweedfs/issues/4129
 2023-01-11 08:46:56 -08:00
chrislu
efcffe4f77 no need to set chunks during flush 2023-01-10 10:01:52 -08:00
chrislu
85f1b70207 reduce group size from 256MB to 64MB 2023-01-10 01:35:44 -08:00
chrislu
86657ed3a8 avoid repeatedly adding manifest chunks 2023-01-10 01:35:10 -08:00
chrislu
28fe578944 avoid possible nil entry 2023-01-10 01:07:34 -08:00
chrislu
47fa5ef979 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-01-10 00:52:50 -08:00
chrislu
9e0d526163 fix compilation 2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14 cluster.ps add filer meta sync progress 2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e chunk group remove manifestChunks and reset sections in SetChunks() 2023-01-10 00:49:31 -08:00
chrislu
e650c8397d ensure section.chunks is always garbage free 2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1 refactor 2023-01-10 00:46:46 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. (#4119) 2023-01-09 06:47:08 -08:00
chrislu
51d4a4b28d 3.39 2023-01-08 18:20:22 -08:00
chrislu
0b52f08226 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-01-08 18:03:25 -08:00
chrislu
469c959e6a add missing options for webdav 2023-01-08 18:03:22 -08:00
monchickey
3e2c9ea73d
Add image cropping. (#4117) 2023-01-06 09:28:07 -08:00
chrislu
296fdc296c mount: faster add chunks 2023-01-06 01:03:29 -08:00
chrislu
2abf817580 fix for stream reader 
fix https://github.com/seaweedfs/seaweedfs/issues/4112
 2023-01-05 11:19:21 -08:00
chrislu
e367444586 add notes for full version 2023-01-04 09:52:25 -08:00
chrislu
5423790b2c add back previous chunk upload selection algo 2023-01-03 22:50:39 -08:00
chrislu
7bdae5172e batch delete EC needles 
fix https://github.com/seaweedfs/seaweedfs/issues/4107
 2023-01-03 22:05:26 -08:00
zemul
6b4c033431
add mount log (#4101) 
* filer.backup use replication.source.filer

* add mount log

* Revert "filer.backup use replication.source.filer"

This reverts commit 07bf6f956c.

* fix

Co-authored-by: zemul <zhouzemiao@ihuman.com>
 2023-01-03 00:00:45 -08:00
chrislu
48f2edc065 3.38 2023-01-02 23:23:02 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089) 
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
 2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax (#4096) 2023-01-01 05:06:57 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query (#4095) 2023-01-01 05:06:41 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding (#4092) 2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum (#4087) 
* stop vacuum

* suspend/resume vacuum

* remove unused code

* rename

* rename param
 2022-12-28 01:36:44 -08:00
Guo Lei
5f993a5f20
Fix ide warning (#4082) 
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
 2022-12-26 00:03:47 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file (#4060) 
* fs.verify needles status of file

* fix parse args

* refactor logging

* set modifyTimeAgoAtSec
 2022-12-21 15:54:17 -08:00
chrislu
77b7c9f7cc lock and unlock chunks for read 2022-12-21 14:15:35 -08:00
chrislu
d1797deccb remove duplicated logic of removing from writableChunks 2022-12-21 14:15:11 -08:00
chrislu
6ede19e825 add a simple file replication progress bar 2022-12-20 19:47:21 -08:00
chrislu
f7beba8515 unlock before submitting the uploading jobs 2022-12-20 12:52:30 -08:00
chrislu
28d479e5c0 mount: adjust locking for upload pipeline 2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305 filer sink retries reading file chunks, skipping missing chunks 
if the file chunk is not available during replication time, the file is skipped
 2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c metadata skip reading un-available logs 
this means the volumes for metadata histories have been lost.
 2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993 more detailed logs 2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed (#4057) 2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key (#4047) 
Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-12-11 22:49:57 -08:00
chrislu
438146249f 3.37 2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9 rotate log files 2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c volume server logs add url info 
debug https://github.com/seaweedfs/seaweedfs/issues/3964
 2022-12-11 13:15:38 -08:00
chrislu
38c63c429b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb add chunk etag when downloading from remote storage 
fix https://github.com/seaweedfs/seaweedfs/issues/3987
 2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting (#4042) 
Signed-off-by: wusong <wangwusong@virtaitech.com>

Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
 2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming (#4039) 2022-12-07 07:47:40 -08:00
chrislu
659f28b855 simple refactoring 2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8 duplicated logic 2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" (#4035) 
* Revert "refactor: moved to locked entry"

This reverts commit 94bc9afd9d.

* only add LockedEntry, no changes to entryLock

* fix compilation
 2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb 3.36 2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d refactor: moved to locked entry 2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory (#4027) 2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer (#4025) 2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn (#4024) 2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories (#4016) 2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip (#4012) 2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata (#4010) 2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles (#4005) 2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo (#3993) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles (#3991) 
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.

Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
 2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks() 
for later locking on reading chunks
 2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57 3.35 2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading (#3958) 
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f filer store: skip disabled location specific filer store initialization 
fix https://github.com/seaweedfs/seaweedfs/issues/3971
 2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f file name escape uri 
fix https://github.com/seaweedfs/seaweedfs/issues/3969
 2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part (#3968) 2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid (#3963) 2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers (#3965) 2022-11-10 07:09:53 -08:00
chrislu
47db75a695 3.34 2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType (#3949) 
* fix filer.remote.sync to azure with ContentType

* fix pass X-Amz-Meta to X-Ms-Meta
 2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration (#3941) 2022-11-04 00:34:40 -07:00
chrislu
4193dafce1 azure metadata: skip metadata prefixed with "X-" 
fix https://github.com/seaweedfs/seaweedfs/issues/3875
 2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete (#3939) 2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934) 
collect all file ids from the file without cut off time for finding missing data from volumes
 2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933) 
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted (#3932) 2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server (#3926) 
check needles status from volume server
 2022-10-31 11:33:04 -07:00
chrislu
0f360862bf 3.33 2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies (#3924) 
avoid vacuum if not enough replica copies
 2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923) 
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
 2022-10-30 20:32:46 -07:00
chrislu
8b9957d461 add back "/" prefix if it is missing in object 
fix https://github.com/seaweedfs/seaweedfs/issues/3737
 2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-28 12:53:23 -07:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922) 
needles older than the cutoff time are not missing yet

https://github.com/seaweedfs/seaweedfs/issues/3919
 2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048 master: delete partially created replicated volumes during volume growth 
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
 2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles (#3909) 2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns (#3906) 
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic (#3907) 2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit (#3905) 2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase (#3894) 
* fix: set user metadata key to lowercase

* feat: simplify a bit
 2022-10-24 17:29:52 -07:00
chrislu
091f999e8c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-24 16:50:41 -07:00
chrislu
377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo (#3893) 
* types packages is imported more than onece

* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
 2022-10-24 09:25:48 -07:00
chrislu
d5364218b2 adjust help message 2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3 fix merge 2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place (#3880) 
https://github.com/seaweedfs/seaweedfs/issues/3852

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred (#3883) 
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40 skip reading needle if size is invalid 
related to https://github.com/seaweedfs/seaweedfs/issues/3882
 2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d remove volume server tcp client 2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a fix size variable 2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50 volume server: remote tier volumes only soft delete in local index 
fix https://github.com/seaweedfs/seaweedfs/issues/3889
 2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite (#3881) 2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0 3.32 2022-10-16 20:43:37 -07:00
chrislu
de286fe662 shell: volume.move handles volume moved to cloud tier 
fix https://github.com/seaweedfs/seaweedfs/issues/3803
 2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15 shell: volume.tier.upload clean up replicated copies 
fix https://github.com/seaweedfs/seaweedfs/issues/3804
 2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping (#3859) 2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861) 
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
 2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict (#3858) 
fix conflict

https://github.com/seaweedfs/seaweedfs/pull/3841
 2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9 Revert "mount: remove mount directory from generating cache directory id" 
This reverts commit 109dc7fdf6.
 2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume (#3851) 
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
 2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9 ensure monotonic n.AppendAtNs 
fix https://github.com/seaweedfs/seaweedfs/issues/3852
 2022-10-13 23:15:00 -07:00
chrislu
e05637c42c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35 remove unused ReadNeedleBlobRequest.needle_id 
fix https://github.com/seaweedfs/seaweedfs/issues/3853
 2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp (#3855) 
* add acl helper functionalities

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* add tests

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* remove 0 when create map

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* delete when empty at `AssembleEntryWithAcp`

`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp`  shouldn't skip empty value

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73 filer: redis store reduce from 2 redis operations to 1 for updates. 2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece (#3838) 2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb default hasSlowRead to true 2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true (#3710) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit

* volume server:set the default value of "hasSlowRead" to true
 2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities (#3831) 2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata (#3833) 2022-10-12 07:14:49 -07:00
chrislu
e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
chrislu
cea73ac008 serialize sqlite operations 
fix https://github.com/seaweedfs/seaweedfs/issues/3827
 2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71 fix error handling 2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-11 21:58:19 -07:00
chrislu
0d817bc347 fix invalid memory address or nil pointer dereference on filer.sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3826
 2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824) 
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency (#3813) 2022-10-10 06:44:29 -07:00
chrislu
0711870f43 3.31 2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6 mount: remove mount directory from generating cache directory id 
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
 2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9 tests add parsing ec shard info 2022-10-09 22:47:55 -07:00
chrislu
0623bf582e include ec shard for capacityByFreeVolumeCount 2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15 move volume: find target volume server by exiting/max ratio 2022-10-09 22:30:59 -07:00
chrislu
049f040c3c refactor 2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7 parsing ec volumes 2022-10-09 19:21:23 -07:00
chrislu
1e27d36576 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-10-09 18:58:14 -07:00
chrislu
f9383aa726 refactor to change capacity data type 2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie (#3748) 
* fix: compact_map get error

* fix: CompactSection delete lock and move test to compact_map

Co-authored-by: shibinbin <shibinbin@megvii.com>
 2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options (#3807) 
fuse: Ignore "-o" command
 2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not (#3799) 
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452

* fix metric MasterAdminLock
 2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. (#3802) 2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics (#3798) 2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell (#3796) 2022-10-06 07:10:41 -07:00
chrislu
ea271600ec fix parameters 2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c filer.sync: limit concurrency when fetching file chunks 
fix https://github.com/seaweedfs/seaweedfs/issues/3787
 2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels (#3785) 2022-10-04 01:20:01 -07:00
chrislu
29632d5a34 3.30 2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis (#3765) 2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7 better fix for option format 2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb mount: fix extra option format 2022-09-30 08:44:41 -07:00
chrislu
8e81619d02 mount: accept all extra mount options 
fix https://github.com/seaweedfs/seaweedfs/issues/3767
 2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId (#3754) 2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account (#3753) 
associate `Account` and `Identity` by accountId
 2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1 skip debug message 2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server (#3708) 
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer (#3750) 
Adjusted filer response headers
 2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" (#3747) 
* fs.meta.load load any dirs with prefix "important"

* replace dirPattern to dirPrefix

* help dirPrefix
 2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda skip truncation on error 
fix https://github.com/seaweedfs/seaweedfs/issues/3746
 2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e 3.29 2022-09-25 19:20:45 -07:00
chrislu
640c53cda8 add future list 2022-09-25 17:43:42 -07:00
chrislu
cc570a4477 assert results are sorted 2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1 ordered execution async wait 2022-09-25 13:45:55 -07:00
chrislu
973b8ff067 rename 2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b rename file 2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2 s3 to watch specific directories 2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9 grpc: watch metadata changes by directory 2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f filer replication: compare content changes directly 
Fix https://github.com/seaweedfs/seaweedfs/issues/3714

The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.

For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
 2022-09-20 08:35:10 -07:00
chrislu
b2a171632b mount: fix 0 file size in linux 2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf adjust logs 2022-09-18 18:14:24 -07:00
chrislu
901e8157de 3.28 2022-09-18 17:46:04 -07:00
chrislu
9d230521a4 report write error 2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b volume server: rename readBufferSize to readBufferSizeMB 2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-16 23:54:39 -07:00
chrislu
06748f2771 remove unused code 2022-09-16 23:54:37 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store (#3707) 2022-09-16 09:05:38 -07:00
Ryan Russell
824f7ad9e1
refactor(shell): readability improvements (#3704) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:49 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements (#3703) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35 mark "hasSlowRead" as experimental 2022-09-15 23:33:46 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements (#3696) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… (#3695) 
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:48 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep (#3694) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet (#3693) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var (#3692) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:09:06 -07:00
Ryan Russell
d65bdeef08
docs(meta_cache): readability fixes (#3691) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:04:57 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… (#3690) 
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:04:29 -07:00
Ryan Russell
a635973847
docs(topology): readability improvements (#3688) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 02:03:10 -07:00
Ryan Russell
0fc242b084
docs: panicing -> panicking (#3687) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:38:04 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes (#3686) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:33:04 -07:00
Ryan Russell
150ce9f3f6
docs(wdclient): readability fix (#3685) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 01:29:26 -07:00
ImFantuan
b64411bda8
fix 0 size (#3683) 2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default (#3682) 
* simplify a bit

* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
 2022-09-15 00:27:15 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu
c8645fd232 master: implement grpc VolumeMarkWritable 
fix https://github.com/seaweedfs/seaweedfs/issues/3657
 2022-09-14 23:05:30 -07:00
Ryan Russell
c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr (#3679) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 13:10:27 -07:00
Ryan Russell
12914af4d8
Character readability (#3678) 
* refactor(pb): `quote_charactoer` -> `quote_character`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 13:09:53 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime (#3676) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported (#3677) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:16:39 -07:00
Ryan Russell
bd2dc6d641
refactor(shell): Decending -> Descending (#3675) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:06:48 -07:00
Ryan Russell
dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… (#3674) 
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:02:33 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive (#3673) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 12:00:16 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements (#3672) 
* refactor(net_timeout): `listner` -> `listener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(filer): `localPublicListner` -> `localPublicListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(command): `masterLocalListner` -> `masterLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(net_timeout): `ipListner` -> `ipListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:59:55 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction (#3671) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:45 -07:00
Ryan Russell
c4363c4b5e
refactor(command_fs_rm): entiries -> entries (#3670) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:27 -07:00
Ryan Russell
a36d7b77be
refactor(cipher_test): plantext -> plaintext (#3669) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 11:24:09 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates (#3668) 
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `SafeRenewInteval` -> `SafeRenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `InitLockInteval` -> `InitLockInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor: `RenewInteval` -> `RenewInterval`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-14 10:29:55 -07:00
Ryan Russell
e22335ba78
refactor(queue_unbounded): inbountLen -> inboundLen (#3666) 2022-09-14 10:12:45 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
Ryan Russell
d734fff322
docs: replicte -> replicate (#3664) 2022-09-14 10:01:18 -07:00
Ryan Russell
dfaa602661
refactor(notification_kafka): parition -> partition (#3663) 2022-09-14 09:15:21 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662) 2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original (#3661) 2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer (#3654) 2022-09-14 06:33:00 -07:00
Ryan Russell
c9f573eed8
docs: indicates readability comment fix (#3659) 2022-09-14 06:31:07 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename (#3658) 2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix (#3655) 2022-09-14 04:53:04 -07:00
chrislu
58d18b68d8 rename variable 2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas (#3591) 
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578

* put with in the util.Retry

* just purge on any errors
 2022-09-13 22:33:35 -07:00
chrislu
0baf3d38c2 adjust logs 2022-09-13 10:33:28 -07:00
chrislu
ce55922884 fixing 2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero (#3630) 
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629

* fix GrpcPort

* skip url :0

* skip empty DataCenter or Rack

* skip empty heartbeat Ip

* dell msg add DataCenter

* comment todo

* fix
 2022-09-11 22:31:53 -07:00
chrislu
b834027c5a refactor 2022-09-11 21:53:15 -07:00
chrislu
d8ca7d34fe 3.27 2022-09-11 19:47:53 -07:00
chrislu
22064c3425 mount: ensure ordered file handle lock and unlock 2022-09-11 19:44:34 -07:00
chrislu
b9112747b5 volume server: synchronously report volume readonly status to master 
fix https://github.com/seaweedfs/seaweedfs/issues/3628
 2022-09-11 19:29:10 -07:00
chrislu
6690236754 rename variables 2022-09-11 19:18:42 -07:00
chrislu
4af8c427eb refactor 2022-09-10 16:56:03 -07:00
chrislu
4957d8eec6 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 15:36:02 -07:00
chrislu
b1501f61dd Revert "a new test case" 
This reverts commit eb7cf3de81.
 2022-09-10 15:35:42 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
chrislu
2c6b68b40e simplify a bit 2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a
[sync] override amz storage class, None to delete (#3639) 
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
 2022-09-10 14:15:42 -07:00
chrislu
3cb914f7e1 avoid dead lock 2022-09-10 11:26:19 -07:00
chrislu
ebef579ebb Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-10 11:22:18 -07:00
chrislu
576c113c59 replace PR https://github.com/seaweedfs/seaweedfs/pull/3621 
replace https://github.com/seaweedfs/seaweedfs/pull/3621
 2022-09-10 11:22:16 -07:00
Brian
4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640) 2022-09-10 08:05:28 -07:00
chrislu
205ecb5d03 skip directories under ".uploads" directory 
related to https://github.com/seaweedfs/seaweedfs/issues/3636

skipping all directories under ".uploads" directory.
 2022-09-09 22:57:38 -07:00
chrislu
fc4208d128 volume.balance: default to balance ALL_COLLECTIONS 2022-09-09 16:06:05 -07:00
gfx
48db56ddad
arangodb s3 bucket name compatibility (#3588) 
* Update arangodb_store.go

* update readme, properly escape queries, add name patching

* use underscore

* use underscore

* better comment

* fix readme

Co-authored-by: a <a@a.a>
 2022-09-09 09:43:42 -07:00
VictorSFF
10d545060f
mount: fix new inode allocation (#3633) 2022-09-09 06:32:19 -07:00
famosss
41af5a6ee5
simplify a bit (#3631) 2022-09-08 21:01:21 -07:00
chrislu
64e75a286e simplify a bit 2022-09-08 19:07:57 -07:00
chrislu
d14d029c73 reduce v.dataFileAccessLock lock scope 2022-09-08 19:05:31 -07:00
chrislu
406a80da4b adjust variable names 2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
chrislu
eb7cf3de81 a new test case 2022-09-08 14:04:42 -07:00
chrislu
4260804613 volume.balance: avoid moving out volume with max=1 2022-09-08 14:04:21 -07:00
chrislu
d653c5f811 unused 2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43 rename 2022-09-08 12:06:03 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625) 2022-09-07 23:23:33 -07:00
chrislu
7de112943b Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195 return err for processRangeRequest() 2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown (#3618) 2022-09-07 18:57:13 -07:00
Konstantin Lebedev
695e3a909c
avoid data race on currentDiskUsage.maxVolumeCount (#3592) 
https://github.com/seaweedfs/seaweedfs/issues/3590
 2022-09-07 09:51:15 -07:00
Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount (#3594) 
https://github.com/seaweedfs/seaweedfs/issues/3593
 2022-09-07 09:50:17 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error (#3616) 2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown (#3613) 
* fix: volume Deadlock when exception is thrown

* adjust log level
 2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
chrislu
c07ab9c060 3.26 2022-09-06 08:26:20 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket (#3609) 
https://github.com/seaweedfs/seaweedfs/issues/3607
 2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level (#3610) 2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap (#3606) 2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic (#3596) 2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log (#3598) 2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e 3.25 2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level (#3589) 2022-09-04 22:37:32 -07:00
chrislu
f4abefc109 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 22:23:31 -07:00
chrislu
39340f7e42 cloud tier: s3 consume all read response body 
fix https://github.com/seaweedfs/seaweedfs/issues/3584
 2022-09-04 22:23:28 -07:00
chrislu
7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket (#3587) 
* Update helpers.go

* Update arangodb_store_bucket.go
 2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context (#3586) 
Update arangodb_store.go
 2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d minor optimization 2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca filer.backup and filer.sync: include headers during backup and sync 
fix https://github.com/seaweedfs/seaweedfs/issues/3532
 2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level (#3585) 2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move (#3583) 2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync (#3579) 
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
 2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed (#3575) 2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c filer: rocksdb store report not found correctly 
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
 2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e Revert "avoid data race on grace.hooks (#3572)" 
This reverts commit c37d6fc01a.
 2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks (#3572) 
https://github.com/seaweedfs/seaweedfs/issues/3564
 2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab align memory 2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566) 
https://github.com/seaweedfs/seaweedfs/issues/3565
 2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1 add more logs 2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541) 
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0

Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update code

* more code fix

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount (#3562) 
* Add an e2e workflow to test FUSE mount

* Fix deadlocks during concurrent r/w
 2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c upload_content: upload close response.Body 2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced refactor 2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c just a bit safer 2022-08-31 00:10:17 -07:00
chrislu
9c944377dc minor 2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6 s3: close response body with copying object 2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae print process before do the work 2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
chrislu
de5263364b master client: use GetMaster() for latest master address 2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a more detailed error 2022-08-30 09:43:17 -07:00
chrislu
d81db3c703 s3: fix configuring IAM for the same user 
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]

s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
 2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0 rename variable 2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster (#3538) 
https://github.com/seaweedfs/seaweedfs/issues/3510
 2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00