chrislu
bf9ae74384
add logs
2024-01-01 19:09:05 -08:00
chrislu
32bc8d6a38
adjust wait time
2023-12-31 17:42:44 -08:00
chrislu
458ddbf919
clean up dead code
2023-12-31 13:25:33 -08:00
chrislu
2845230329
passing broker into the assignments
2023-12-28 21:13:49 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
Varun Upadhyay
77626666c5
Minor cleanup & gitignore update ( #5144 )
2023-12-28 20:25:43 -08:00
chrislu
9ed26cd7b0
Merge branch 'master' into mq-subscribe
2023-12-28 11:57:30 -08:00
chrislu
c950a40aad
subscriber keep connected to the balancer
2023-12-28 11:56:37 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency ( #4529 )
...
* fix: increase speed cmd fs meta load
* fix: add wg
2023-12-28 11:54:37 -08:00
Lars Lehtonen
edc0532b1d
weed/server: remove unneeded err from streamReadOneVolume() signature ( #5143 )
2023-12-28 11:45:13 -08:00
chrislu
bebbc9fe44
create volume grow request if the selected volume is close to full
2023-12-27 11:45:44 -08:00
chrislu
bb9269f3f7
Revert "Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. ( #5134 )"
...
This reverts commit 838578b55f
2023-12-24 12:31:08 -08:00
chrislu
c6b1dc7058
remove unused code
2023-12-24 11:11:41 -08:00
vivekkoya
838578b55f
Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. ( #5134 )
2023-12-23 15:01:57 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck ( #5133 )
...
* Add modifyTimeAgo to volume.fsck
* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler ( #5132 )
2023-12-23 09:18:16 -08:00
chrislu
03c4b2e988
correct lock accounting
...
possibly fix https://github.com/seaweedfs/seaweedfs/issues/5082
2023-12-22 14:16:23 -08:00
chrislu
c4badf7396
adjust options
2023-12-22 11:41:10 -08:00
chrislu
6324062acc
fix
2023-12-22 11:37:15 -08:00
chrislu
e0727071c8
go fmt
2023-12-22 11:33:50 -08:00
chrislu
ee1c9bc314
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-12-22 11:33:02 -08:00
chrislu
034db049a0
comment fix
2023-12-22 11:33:00 -08:00
chrislu
6df44d3c95
option to filer.copy with public volume server url
...
fix https://github.com/seaweedfs/seaweedfs/issues/5126
2023-12-22 11:32:43 -08:00
chrislu
7a0d1e7fc1
minor
2023-12-22 11:29:54 -08:00
Farbod
c278f49bca
Using filer.remote.sync concurrency in filer.remote.gateway ( #5123 )
...
- Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync
Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>
2023-12-20 16:25:54 -08:00
jerebear12
06343f8976
Set allowed origins in config ( #5109 )
...
* Add a way to use a JWT in an HTTP only cookie
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
* Added a way to specify allowed origins header from config
* Removed unecessary log
* Check list of domains from config or command flag
* Handle default wildcard and change name of config value to cors
2023-12-20 16:21:11 -08:00
Konstantin Lebedev
3c9bcfb864
chore: add dsn for connection to mysql ( #5060 )
...
* chore: add dsn for connection to mysql
* add comment
* new comment
* fix: validate dsn and adapt password
2023-12-20 16:20:58 -08:00
chrislu
915416a277
fix wrong parameter
...
fix https://github.com/seaweedfs/seaweedfs/pull/5102
2023-12-13 12:02:57 -08:00
Farbod
d5c01e5ceb
Remote gateway Bug fix ( #5100 )
2023-12-12 07:46:45 -08:00
chrislu
61b05fad95
fix test
2023-12-11 20:57:29 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
Konstantin Lebedev
125ad8fe63
falls back to update only if error contains msg "duplicate entry"
...
https://github.com/seaweedfs/seaweedfs/issues/5062
2023-12-09 10:18:18 -08:00
chrislu
c1a6e624e5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-12-07 08:13:46 -08:00
chrislu
5fe4c04a1a
add missing error in the log
...
related to https://github.com/seaweedfs/seaweedfs/issues/5084
2023-12-07 08:13:42 -08:00
Lars Lehtonen
43a5973a66
weed/storage: fix dropped test errors
2023-12-06 12:05:22 -08:00
jerebear12
d5d9fbb8aa
Add a way to use a JWT in an HTTP only cookie
...
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
2023-12-05 08:57:01 -08:00
chrislu
756bcc032d
adjust logs
2023-11-27 12:57:29 -08:00
Konstantin Lebedev
dd78397fea
logging PickForWrite error
...
https://github.com/seaweedfs/seaweedfs/issues/3886
2023-11-27 12:56:15 -08:00
chrislu
81f11883e3
go fmt
2023-11-26 11:47:20 -08:00
chrislu
7a50762b9e
reduce binary size by skipping rclone
2023-11-23 00:48:27 -08:00
zehweh
d818ca9631
fix TestMisplacedChecking() and add test-case
2023-11-21 08:24:34 -08:00
julor
2fa37e0287
Update filer_server_handlers_write_autochunk.go
2023-11-20 11:17:14 -08:00
julor
9e03b916d8
verify content-md5 in header
...
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
2023-11-20 11:17:14 -08:00
SmsS4
6c1ce18541
Fix weed fix
2023-11-20 11:14:19 -08:00
Lars Lehtonen
57801687de
weed/command: fix dropped error
2023-11-20 08:05:24 -08:00
Viktor Kuzmin
06e723f629
Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952
2023-11-17 07:38:14 -08:00
Konstantin Lebedev
2b229e98ce
fix: add doDeleteFile option for filer backup
2023-11-17 07:37:28 -08:00
chrislu
17710e1ecb
adjust logs
2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr
6f559fe838
fs.mergeVolumes: change dir to a named parameter
2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be
Fix typo (dist => dest)
2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3
fs.mergeVolumes: Make a plan based on volumes size
2023-11-13 06:41:24 -08:00
Konstantin Lebedev
e55fa68848
fix: excludeFileName is empty
2023-11-13 06:23:46 -08:00
Konstantin Lebedev
3c5295a1a6
filer backup add option for exclude file names that match the regexp to sync on filer
2023-11-13 06:23:46 -08:00
Konstantin Lebedev
5e41ab1370
s3 api add not implemented response for PutBucketVersioning
2023-11-13 06:22:23 -08:00
Konstantin Lebedev
7f6152f130
s3 api add default response for GetBucketVersioning
2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr
b05d682c48
make help command in shell case insensitive
2023-11-13 06:06:45 -08:00
chrislu
27b34f3793
3.59
2023-11-12 22:12:08 -08:00
jsh
47112917ff
ec.decode: mount the collected ec shards
2023-11-10 00:04:42 -08:00
chrislu
85bed4d92e
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-11-09 08:02:57 -08:00
chrislu
c563386675
rename parameters
2023-11-09 08:02:39 -08:00
Lars Lehtonen
5db82e594a
weed/server: fix dropped webdav error
2023-11-07 19:05:21 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go ( #4988 )
2023-11-07 07:58:19 -08:00
Konstantin Lebedev
d693beea87
Replace an invalid method in a metric with an invalid one ( #4987 )
2023-11-06 10:51:47 -08:00
chrislu
a667f0ad99
fix error reporting
...
related to https://github.com/seaweedfs/seaweedfs/issues/4978
2023-11-05 07:17:16 -08:00
chrislu
b40ba7c7fe
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-11-01 16:28:05 -07:00
chrislu
310eec9a9c
skip if path changed
...
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
2023-11-01 16:27:59 -07:00
Nico D'Cotta
3834ad5c48
refactor webdav subdirectory, fixes #4967 ( #4969 )
...
* refactor webdav subdirectory, fixes #4967
* fix bug where Name() was not called on delegate wrappedFileInfo
2023-11-01 13:12:49 -07:00
tired-engineer
4ed06e9ba5
Pass correct Filer's gRPC port to S3 server. ( #4972 )
...
Fixes seaweedfs/seaweedfs#4971
2023-11-01 10:11:41 -07:00
SmsS4
ba11331cc1
Fix twice increase metric counter ( #4961 )
2023-10-30 08:05:27 -07:00
SmsS4
f95848ba7d
Add ErrorGetNotFound and ErrorGetInternal to volume server metrics ( #4960 )
2023-10-30 07:38:03 -07:00
chrislu
d1e83a3b4d
3.58
2023-10-29 23:04:20 -07:00
duanhongyi
bf839651ec
Fix: issues #4950 ( #4952 )
2023-10-25 22:09:39 -07:00
zehweh
8b39bbbe2f
fix copying .vif file in VolumeCopy ( #4943 )
...
closes #4934
fixes #2633
might fix #3528
2023-10-24 09:54:24 -07:00
chrislu
2bc05f70e7
log full percentage
2023-10-22 12:59:34 -07:00
chrislu
612fd7206e
fix
2023-10-19 08:10:33 -07:00
Konstantin Lebedev
6c82d7ddfc
[s3] do reload s3 static config ( #4923 )
...
* do reload s3 config
* print error on reload s3 config
* print success msg
* Update weed/s3api/s3api_server.go
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-10-18 23:26:49 -07:00
lakeland1990
3df0e11ad3
remove compare to fix #4668 issue ( #4924 )
2023-10-18 09:06:56 -07:00
chrislu
cbc24c7b24
disconnect from old subscribers
2023-10-13 16:48:14 -07:00
James Hedley
6acb43bbbb
Add optional flags to enable mTLS with verification of client certificate ( #4910 )
2023-10-13 09:02:24 -07:00
chrislu
b62066d38f
adding DiskType to /dir/status output
2023-10-12 21:39:11 -07:00
chrislu
deb86ff4a6
upgrading grpc
2023-10-12 21:38:34 -07:00
chrislu
de0b969b36
Revert "rename"
...
This reverts commit 35b5264ab7
2023-10-12 20:28:11 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file ( #4881 )
...
* fix: disallow file name too long when writing a file
* bool LongerName to MaxFilenameLength
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero ( #4909 )
...
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 08:15:20 -07:00
chrislu
35b5264ab7
rename
2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr
3e8604a4c5
add fs.mergeVolumes command into shell ( #4907 )
...
* add fs.compact command into shell
* rename fs.compact to fs.mergeVolumes
2023-10-10 12:47:18 -07:00
Konstantin Lebedev
2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment ( #4893 )
...
* fix: skipping checking active volumes with the same number of files at the moment
https://github.com/seaweedfs/seaweedfs/issues/4140
* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140
* add TestShouldSkipVolume
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-09 09:57:26 -07:00
Yuval Yacoby
3fe00996b2
added healthz endpoint to filer ( #4899 )
2023-10-08 07:03:18 -07:00
Andrew Garrett
082f67bfe9
Fix filer.backup local sink to propagate file mode changes ( #4896 )
2023-10-06 05:40:20 -07:00
chrislu
0fd7222d65
default to skip if less than 4 nodes
2023-10-05 11:13:48 -07:00
chrislu
7b3c63a8af
handle wrapped errors
2023-10-05 08:15:11 -07:00
chrislu
4e493d596c
minor
2023-10-05 08:13:53 -07:00
Lars Lehtonen
530bdedf76
weed/operation: fix dropped error ( #4888 )
2023-10-03 10:55:20 -07:00
chrislu
6ebe26a765
Revert "Revert "Revert "Add disk type to prometheus metrics" ( #4777 )""
...
This reverts commit 567d788928
2023-10-03 08:28:52 -07:00
chrislu
7540d43ee9
Revert "Revert "fix compilation""
...
This reverts commit f9abfd0b03
2023-10-03 08:28:02 -07:00
chrislu
249c0e06ef
Revert "fix compilation"
...
This reverts commit 451ec6504d
2023-10-03 08:27:50 -07:00
chrislu
451ec6504d
fix compilation
2023-10-03 08:15:18 -07:00
chrislu
f9abfd0b03
Revert "fix compilation"
...
This reverts commit 0483ba3889
2023-10-03 07:58:11 -07:00
chrislu
0483ba3889
fix compilation
2023-10-02 11:55:50 -07:00
chrislu
567d788928
Revert "Revert "Add disk type to prometheus metrics" ( #4777 )"
...
This reverts commit 9215ba24be
2023-10-02 11:49:54 -07:00
chrislu
b5070d6280
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-10-02 10:52:40 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg ( #4880 )
2023-10-02 08:29:09 -07:00
chrislu
bf13f3ced7
add subscriber coordinator
2023-10-02 01:02:27 -07:00
chrislu
734178093e
refactor TopicPartition struct
2023-10-02 01:01:45 -07:00
chrislu
2a578b9033
refactor
2023-10-01 11:59:19 -07:00
chrislu
321fda552c
comment
2023-10-01 11:36:27 -07:00
chrislu
b02fdeabff
RetryForever => RetryUntil
2023-10-01 11:33:56 -07:00
chrislu
ebd6f96d35
adjust mq.proto
2023-09-30 21:47:57 -07:00
chrislu
02cb799481
add a test framework for grpc
2023-09-30 21:36:58 -07:00
chrislu
c4e6545197
fix compilation
2023-09-30 21:26:09 -07:00
chrislu
358cba43ef
update proto generated files
2023-09-30 13:19:25 -07:00
chrislu
597ab1e4bf
add design for mq balancer
2023-09-30 13:19:05 -07:00
chrislu
d7c211fe8c
adjust mq proto
2023-09-30 13:18:49 -07:00
chrislu
40888b5612
align 64bit atomic operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/4878
2023-09-28 07:58:43 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync ( #4875 )
...
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647
* Update weed/shell/command_volume_fix_replication.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix: revert this existing do option to positive
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-27 23:12:10 -07:00
Konstantin Lebedev
44906f1f3b
fix: avoid error file name too long when writing a file ( #4876 )
2023-09-27 05:40:51 -07:00
chrislu
9d589b48e6
rename function
2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica ( #4873 )
...
https://github.com/seaweedfs/seaweedfs/issues/4647
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4
describe a topic
2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance ( #4864 )
...
* fix(ec): ignore 0 byte data files
refers to parts of #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
* fix(ec): ignore volumes not from the current collection during balance
fixes #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
---------
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b
Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs ""
...
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu
3d07895518
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 09:16:35 -07:00
chrislu
a6e35e4f68
fix compilation
2023-09-25 09:16:33 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration ( #4859 )
...
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface
https://github.com/seaweedfs/seaweedfs/issues/4519
* fix: test bucket acl default and
adjust the variable names
* fix: s3 api config test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
chrislu
c9177c92e5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 08:20:16 -07:00
chrislu
81e57bd45e
fix test
2023-09-25 08:20:03 -07:00
Lars Lehtonen
28a3a31b27
weed/replication/sub: fix dropped error ( #4865 )
2023-09-25 07:33:18 -07:00
chrislu
23f334dd13
comment
2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71
ensure latest stats are reported
2023-09-24 23:34:31 -07:00
chrislu
dff2ce5d2f
read existing broker stats
2023-09-24 23:10:33 -07:00
chrislu
1492bf7552
fix listing topics
2023-09-24 23:05:41 -07:00
chrislu
3cf9b8d621
create topic and report topic
2023-09-24 22:00:43 -07:00
chrislu
b3f94feede
do create topic
2023-09-24 22:00:43 -07:00
chrislu
d74348048a
implement create topic
2023-09-24 22:00:43 -07:00
chrislu
0361c321b4
add CreateTopic API
2023-09-24 22:00:43 -07:00
chrislu
0f8168c0c9
3.57
2023-09-24 21:29:31 -07:00
chrislu
1df4223195
minor
2023-09-24 09:54:57 -07:00
chrislu
10ac25f857
avoid deleting the entry too early
2023-09-23 12:43:00 -07:00
chrislu
c43238b30a
fix waiting condition
2023-09-23 12:28:02 -07:00
chrislu
322af1fc04
fix compilation
2023-09-23 12:13:32 -07:00
chrislu
0414b68a4f
if shared lock, still need to wait for exclusive lock to release
2023-09-23 12:08:23 -07:00
chrislu
9ba4639a27
adjust for better logs
2023-09-23 11:53:44 -07:00
chrislu
186b1f029b
read source uses shared lock
2023-09-23 10:33:44 -07:00
chrislu
2df70ce95f
adjust logs
2023-09-21 23:51:32 -07:00
chrislu
3e23421608
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-21 20:08:30 -07:00
chrislu
49a470bcdf
reduce logs
2023-09-21 20:08:20 -07:00
Konstantin Lebedev
750a0ba1b2
[iam] Replace action read/write to readAcp/writeAcp for handlers with acl ( #4858 )
...
Replace action read/write to readAcp/writeAcp for handlers with acl query
https://github.com/seaweedfs/seaweedfs/issues/4519
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-21 14:07:04 -07:00
chrislu
8b2c39f2c0
reduce binary size
2023-09-21 13:22:08 -07:00
chrislu
b388957340
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-21 11:08:30 -07:00
chrislu
31fc165715
mount switch to ordered lock requests
2023-09-21 11:08:26 -07:00
chrislu
7669852241
adjust logs
2023-09-21 11:04:12 -07:00
chrislu
e3b1bacf3f
add lock table to ensure ordered locks
2023-09-21 10:24:34 -07:00
Konstantin Lebedev
a46f873edd
[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. ( #4856 )
...
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private
https://github.com/seaweedfs/seaweedfs/issues/4519
* Update weed/s3api/s3api_bucket_handlers.go
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-21 08:20:05 -07:00
Konstantin Lebedev
d8b424d123
[s3] optimization iam lookup for reducing algorithm complexity ( #4857 )
...
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-21 08:19:11 -07:00
chrislu
411bdda08d
fix compilation
2023-09-19 16:22:41 -07:00
chrislu
d531b7cbbf
merge some previous missing changes
2023-09-19 14:10:54 -07:00
chrislu
b18112ae10
fix compilation bugs during merge
2023-09-19 14:08:17 -07:00
chrislu
6c6655de9d
wip
2023-09-19 14:03:42 -07:00
chrislu
cf28108ea4
fix grpd dial option
2023-09-19 14:02:51 -07:00
chrislu
27af11f1e8
Revert "Revert "Merge branch 'master' into sub""
...
This reverts commit 0bb97709d4
2023-09-18 18:47:34 -07:00
chrislu
0bb97709d4
Revert "Merge branch 'master' into sub"
...
This reverts commit 4d414f54a24827425146
2023-09-18 16:13:20 -07:00
chrislu
8cb42c39ad
Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs "
...
This reverts commit 2e5aa060264d414f54a2
2023-09-18 16:12:50 -07:00
chrislu
2e5aa06026
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-18 16:06:41 -07:00
chrislu
4d414f54a2
Merge branch 'master' into sub
2023-09-18 16:04:59 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 ( #4850 )
...
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone ) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases )
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md )
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0 )
---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* API changes
* go mod
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2023-09-18 14:43:05 -07:00
chrislu
b64fc44e8d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-18 09:01:17 -07:00
chrislu
340aea9b34
add broker leader
2023-09-18 09:01:14 -07:00
chrislu
0d390ec882
fix nil
2023-09-18 08:52:35 -07:00
Lars Lehtonen
d83107efe5
fix: weed/storage: dropped error
2023-09-18 08:18:57 -07:00
zemul
91cbaef02b
fix: mount proxyByFiler write error: wrong jwt
2023-09-18 07:44:50 -07:00
chrislu
4e17d399a6
refactor
2023-09-17 00:04:20 -07:00
chrislu
7a49d3541a
avoid nil
2023-09-17 00:04:13 -07:00
chrislu
a65088d739
broker report stats to balancer
2023-09-16 23:00:51 -07:00
chrislu
a1cb49d078
adjust log
2023-09-16 23:00:28 -07:00
chrislu
b590a68e75
revert to non-streaming mode to assign file id
...
fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685
2023-09-16 16:13:08 -07:00
chrislu
4650c4c65f
Revert "turn on streaming assign file id"
...
This reverts commit 733db2bc88
2023-09-16 16:09:20 -07:00
chrislu
fa62926347
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-16 15:06:22 -07:00
chrislu
89a1fd1751
Squashed commit of the following:
...
commit 48274251463b50139f687f685ce7ba436d99443bb771fefa370a851ec008908810370a851ec00b39941edc0b875f562779984b6c54cf2492a45499ba67e6ca299e4f985698cb470d44df1eb2da46d5504ae8383adbcba75271c9caf331194d6c18d86f4eb8e8624d1990456670905911853d
2023-09-16 15:06:16 -07:00
chrislu
4827425146
balancer works
2023-09-16 15:05:38 -07:00
chrislu
3b50139f68
comments
2023-09-15 22:22:32 -07:00
chrislu
7f685ce7ba
adjust APIs
2023-09-15 22:20:05 -07:00
chrislu
436d99443b
receive broker stats
2023-09-14 23:49:05 -07:00
wang wusong
9e5b1eef9b
avoid passe lock by value
2023-09-14 06:51:12 -07:00
chrislu
b771fefa37
Merge branch 'master' into sub
2023-09-13 00:03:47 -07:00
annluda
8908810376
Fix s3 api object list with params max-keys and prefix
2023-09-11 21:35:15 -07:00
chrislu
5407b86766
3.56
2023-09-10 22:05:41 -07:00
chrislu
07c2d75ac7
ensure single threaded access
2023-09-10 22:03:44 -07:00
chrislu
0a851ec00b
Create balancer.go
2023-09-10 22:01:25 -07:00
chrislu
39941edc0b
add publisher shutdown
2023-09-07 23:55:19 -07:00
chrislu
875f562779
server side send response at least once per second
2023-09-06 23:16:41 -07:00
chrislu
984b6c54cf
ack interval 128
2023-09-06 23:15:29 -07:00
chrislu
2492a45499
ack interval
2023-09-06 22:39:46 -07:00
Lars Lehtonen
f24c7e803f
weed/storage/erasure_coding: Close() after error handling
2023-09-06 07:35:54 -07:00
Lars Lehtonen
d762b12e11
weed/storage/erasure_coding: fix dropped test error
2023-09-06 07:35:54 -07:00
Lars Lehtonen
a879e1bbb5
weed/storage/erasure_coding: remove unused err from encodeDatFile() signature
2023-09-06 07:35:54 -07:00
Patrick Schmidt
98dcec0ee2
Clean up old signature hash pools
2023-09-05 10:33:27 -07:00
Patrick Schmidt
cdd817edf9
Improve S3 request signing performance
...
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
2023-09-05 10:33:27 -07:00
chrislu
ba67e6ca29
api for sub
2023-09-04 21:43:50 -07:00
chrislu
9e4f985698
publish, benchmark
2023-09-04 21:43:30 -07:00
Lars Lehtonen
e6385e5e8b
weed/util: fix dropped errors ( #4803 )
2023-09-01 13:10:45 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5
connect and publish
2023-08-28 09:02:12 -07:00
chrislu
504ae8383a
protoc version
2023-08-28 09:01:25 -07:00
chrislu
dbcba75271
rename to lookup
2023-08-27 18:59:04 -07:00
chrislu
c9caf33119
move functions
2023-08-27 18:33:46 -07:00
chrislu
4d6c18d86f
pub sub initial tests
2023-08-27 17:50:59 -07:00
chrislu
4eb8e8624d
rename
2023-08-27 13:14:39 -07:00
chrislu
1990456670
sub
2023-08-27 13:13:14 -07:00
chrislu
905911853d
adjust proto
2023-08-26 13:39:21 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer ( #4767 )
2023-08-24 07:08:56 -07:00
chrislu
733db2bc88
turn on streaming assign file id
2023-08-23 00:34:21 -07:00
chrislu
99f037b958
streaming assign file ids
2023-08-23 00:31:33 -07:00
chrislu
94b7e2a37c
add stream assign server side implementation
2023-08-22 09:59:04 -07:00
chrislu
ccedad5196
refactor files
2023-08-22 09:54:06 -07:00
chrislu
010c5e91e3
add stream assign proto
2023-08-22 09:53:54 -07:00
chrislu
a1149f060b
upgrade jwt
2023-08-22 08:26:31 -07:00
chrislu
c49caff5a1
change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims
2023-08-21 23:42:39 -07:00
chrislu
c45665eb97
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-08-21 09:47:36 -07:00
SmsS4
f61490966f
Add time to first byte metric for s3 ( #4768 )
...
* Add time to first byte metric for s3
* Change second to millisecond
2023-08-21 00:42:39 -07:00
chrislu
01d70c21f3
Squashed commit of the following:
...
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
2023-08-20 22:53:05 -07:00
chrislu
16e3f2d528
fix log buffer test
2023-08-20 12:46:15 -07:00
chrislu
3e9c32a3f0
refactoring to use atomic bool
...
fix compilation
2023-08-20 12:13:06 -07:00
SmsS4
1e72579ac7
Fix s3.clean.uploads ( #4766 )
2023-08-19 06:30:18 -07:00
chrislu
6792db21dc
ErrInvalidPart when completed parts has extra parts
...
fix https://github.com/seaweedfs/seaweedfs/discussions/4746
2023-08-17 00:51:42 -07:00
chrislu
f5c4b642bb
re-ordering
2023-08-16 23:54:23 -07:00
chrislu
3852307e94
renaming
2023-08-16 23:47:43 -07:00
chrislu
6c7fa567d4
add separate cache directory for write buffers
2023-08-16 23:39:21 -07:00
SmsS4
ea1b259c12
Update weed fix description ( #4745 )
2023-08-13 03:50:53 -07:00
Nikita Mochalov
4b1ba7f5b2
Save disk space metrics immediately ( #4740 )
2023-08-10 06:52:32 -07:00
Nikita Mochalov
e6a49dc533
Fix resource leaks ( #4737 )
...
* Fix division by zero
* Fix file handle leak
* Fix file handle leak
* Fix file handle leak
* Fix goroutine leak
2023-08-09 15:30:36 -07:00
chrislu
3365468d0d
added an error message
2023-08-08 20:35:21 -07:00
wusong
3ec977cd6c
minor ( #4723 )
2023-08-01 02:34:59 -07:00
chrislu
c09100f928
fix building mongo index
2023-07-24 11:09:51 -07:00
chrislu
7eafa3420b
3.55
2023-07-23 23:11:37 -07:00
SmoothDenis
51bcc219ea
s3api should return 500 code from filer ( #4699 )
2023-07-22 07:22:38 -07:00
wusong
33b9192536
Fix large file chunks deleted by mistake ( #4678 )
2023-07-17 07:52:14 -07:00
chrislu
358b3a4894
3.54
2023-07-16 22:47:57 -07:00
chrislu
6f588b5b18
fix refactoring mistake
...
fix https://github.com/seaweedfs/seaweedfs/issues/4639
2023-07-16 11:22:48 -07:00
mervynzhang
df400e6c71
Concurrency works better ( #4663 )
...
Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>
2023-07-12 23:04:54 -07:00
wusong
0cb9ddd8ec
Fix data loss: add lock for metacache ( #4664 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-07-11 22:23:32 -07:00
Henco Appel
89542db2d0
Fix S3 API multipart upload on Windows ( #4666 )
...
The file paths were being having '/' changed to '\' by filepath.Dir()
resulting in a file being created with '\' separators, but when trying
to read the files, the same wasn't happening.
Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2023-07-11 09:05:14 -07:00
Henco Appel
680c2e76f1
Fix the segfault from not setting s3 https port ( #4667 )
2023-07-11 08:03:20 -07:00
Konstantin Lebedev
2c076ebd96
avoid logging nil error ( #4659 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-07-10 23:38:52 -07:00
LHHDZ
3730772e05
refresh max volume count after adjust ( #4662 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-07-10 23:37:33 -07:00
chrislu
c278bac263
avoid nil needle map
...
fix https://github.com/seaweedfs/seaweedfs/issues/4640
2023-07-07 22:16:58 -07:00
chrislu
0606b59a3e
mount: create mount root on filer
...
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/127
2023-07-07 09:30:32 -07:00
chrislu
14546f0ad8
minor
2023-07-06 09:12:28 -07:00
chrislu
ab30f489c3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-07-06 00:33:02 -07:00
chrislu
31b2751aff
clone volume locations in case they are changed
...
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-07-06 00:32:58 -07:00
wusong
61553beba2
Fix DataBackend nil pointer ( #4641 )
2023-07-04 23:22:10 -07:00
chrislu
a315490f7d
proxy to master uses http address
...
fix https://github.com/seaweedfs/seaweedfs/issues/4607
2023-07-04 11:45:21 -07:00
chrislu
2aa59ab37c
fix copying level db files during commi
...
fix https://github.com/seaweedfs/seaweedfs/issues/4635
2023-07-04 11:28:12 -07:00
wusong
c44a6552d2
Fix buff use ( #4628 )
2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr
f0d3c874b6
Unix Socket listener for S3 server ( #4621 )
...
Add localSocket option to s3 server
2023-06-26 16:22:45 -07:00
chrislu
2c4c2f0994
3.53
2023-06-25 22:51:14 -07:00
chrislu
f193942adc
lock on append metadata operation
2023-06-25 22:07:23 -07:00
chrislu
61c42f9991
adjust lock APIs
2023-06-25 20:30:20 -07:00
chrislu
aafb874ad2
lock until cancelled
2023-06-25 19:31:25 -07:00
chrislu
c4f44d56f7
add distributed lock client
2023-06-25 17:38:34 -07:00
chrislu
f8aa5ea844
adjust filer.proto
2023-06-25 17:38:19 -07:00
chrislu
13a8593ab9
adjust errors
2023-06-25 17:37:54 -07:00
chrislu
06471dac9d
init lock ring
2023-06-25 15:28:16 -07:00
chrislu
5149b3d07b
filer can proxy to peer filer holding the lock
2023-06-25 15:28:01 -07:00
chrislu
868f7875d7
refactor
2023-06-25 14:30:58 -07:00
chrislu
26eff062c8
refactor
2023-06-25 14:17:58 -07:00
chrislu
464a71a373
add distributed lock manager
2023-06-25 14:14:40 -07:00
chrislu
ee4f7cd636
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-06-25 00:58:24 -07:00
chrislu
3fd659df2a
add distributed lock manager
2023-06-25 00:58:21 -07:00
Damiano Albani
0d9ba8c612
Support templating name of files stored via Rclone backend ( #4606 )
2023-06-24 07:08:24 -07:00
chrislu
a82c44972b
refactor
2023-06-20 00:12:59 -07:00
chrislu
14a19170e2
refactor
2023-06-19 23:51:04 -07:00
chrislu
84ced2abf5
add LockRing
2023-06-19 23:48:33 -07:00
chrislu
fe1716ccdf
remove deprecated tests
2023-06-19 18:43:51 -07:00
chrislu
1ce725728d
minor
2023-06-19 18:43:41 -07:00
chrislu
11549706ef
refactor
2023-06-19 18:21:07 -07:00
chrislu
8ec1bc2c99
remove unused cluster node leader
2023-06-19 18:19:13 -07:00
Bai Jie
3b88ab42aa
remove duplicate fileCount query ( #4588 )
2023-06-18 00:14:14 -07:00
Bai Jie
44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil ( #4587 )
2023-06-18 00:13:40 -07:00
chrislu
94fbf02ea2
generate jwt token when looking up needle id
...
fix https://github.com/seaweedfs/seaweedfs/issues/4577
2023-06-17 00:11:23 -07:00
柏杰
0b0fb9b9e4
avoid data race read volume.IsEmpty ( #4574 )
...
* avoid data race read volume.IsEmpty
- avoid phantom read isEmpty for onlyEmpty
- use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope
* add Destroy(onlyEmpty: true) test
* add Destroy(onlyEmpty: false) test
* remove unused `IsEmpty()`
* change literal `8` to `SuperBlockSize`
2023-06-14 14:39:58 -07:00
Konstantin Lebedev
1e22d5caf2
fix get file stats for IsEmpty ( #4576 )
2023-06-14 01:43:30 -07:00
Konstantin Lebedev
4dd890d4a2
optional https port for s3 ( #4482 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-13 23:58:49 -07:00
chrislu
3fbf4f6189
Revert "Initial patch for redis version 7 support ( #4572 )"
...
This reverts commit a25bca0692
2023-06-12 22:27:38 -07:00
Konstantin Lebedev
4527ead295
fix from comment delete volume is empty ( #4573 )
...
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561
* fix tests
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 22:22:46 -07:00
Aaron Gipson
a25bca0692
Initial patch for redis version 7 support ( #4572 )
2023-06-12 13:30:03 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty ( #4561 )
...
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
柏杰
68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value ( #4560 )
2023-06-10 23:15:27 -07:00
Konstantin Lebedev
5ee04d20fa
Healthz check for deadlocks ( #4558 )
2023-06-09 09:42:48 -07:00
wusong
d6af3a8eb6
fix 404 not found ( #4536 )
...
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 09:14:49 -07:00
Konstantin Lebedev
5caff2c0f0
fix: deletes newly created volumes ( #4551 )
...
https://github.com/seaweedfs/seaweedfs/issues/4531
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-06 09:08:27 -07:00
wusong
9bdbf9c880
revert #4491 ( #4550 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 00:17:51 -07:00
chrislu
0bffb3770f
support TTL when creating folders
...
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
2023-06-05 23:29:25 -07:00
wusong
26f15d0079
Fix no more writable volumes by delay judgment ( #4548 )
...
* fix nomore writables volumes while disk free space is sufficient by time delay
* reset
---------
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-05 10:17:21 -07:00
chrislu
fb4b61036c
3.52
2023-06-04 22:34:26 -07:00
Konstantin Lebedev
a0931be0c0
S3 TLS credentials Refreshing ( #4506 )
...
* S3 TLS credentials Refreshing
* fix: logging
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-04 14:27:56 -07:00
wusong
5aec6da8a3
fix deadlock for filer upload ( #4527 )
...
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-03 13:38:27 -07:00
chrislu
e23f3d6eca
set df.File to nil after it is closed
...
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
2023-06-02 17:16:45 -07:00
chrislu
ca7cc61319
reduce the window size between unregistering a volume server and creating volumes on that server
...
fix https://github.com/seaweedfs/seaweedfs/issues/4467
2023-05-30 08:20:00 -07:00
chrislu
68c3cf767c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-05-29 16:45:28 -07:00
chrislu
d1d1757520
adjust error message
2023-05-29 16:45:25 -07:00
