mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-19 21:47:50 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,938 Commits 32 Branches 296 Tags 152 MiB
fe3e7db9d1
Commit Graph

494 Commits

Author SHA1 Message Date
famosss
25e012d30b
fix: set user metadata key to lowercase (#3894) 
* fix: set user metadata key to lowercase

* feat: simplify a bit
 2022-10-24 17:29:52 -07:00
Guo Lei
97edb40275
Fix errinfo (#3893) 
* types packages is imported more than onece

* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
 2022-10-24 09:25:48 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp (#3855) 
* add acl helper functionalities

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* add tests

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* remove 0 when create map

Signed-off-by: changlin.shi <changlin.shi@ly.com>

* delete when empty at `AssembleEntryWithAcp`

`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp`  shouldn't skip empty value

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-13 22:50:44 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities (#3831) 2022-10-12 20:15:16 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… (#3824) 
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call

Signed-off-by: changlin.shi <changlin.shi@ly.com>

Signed-off-by: changlin.shi <changlin.shi@ly.com>
 2022-10-11 20:14:14 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency (#3813) 2022-10-10 06:44:29 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels (#3785) 2022-10-04 01:20:01 -07:00
LHHDZ
e9584d9661
add ownership rest apis (#3765) 2022-10-01 19:18:00 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer (#3759) 2022-09-29 12:29:01 -07:00
LHHDZ
aacdcc4cad
s3: add account (#3753) 
associate `Account` and `Identity` by accountId
 2022-09-28 12:45:18 -07:00
chrislu
31922b2bf2 s3 to watch specific directories 2022-09-20 09:30:05 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements (#3696) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… (#3695) 
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-15 03:12:48 -07:00
ImFantuan
b64411bda8
fix 0 size (#3683) 2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default (#3682) 
* simplify a bit

* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
 2022-09-15 00:27:15 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability (#3665) 2022-09-14 10:11:31 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket (#3571) 
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
 2022-09-01 10:33:23 -07:00
chrislu
a54f30b6c6 s3: close response body with copying object 2022-08-31 00:09:23 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
chrislu
9fce75607d s3: report http.StatusOK if the directory is explicitly created 
fix https://github.com/seaweedfs/seaweedfs/issues/3457
 2022-08-23 01:16:46 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests (#3496) 2022-08-22 23:39:34 -07:00
famosss
bdba3da2e4
Handle preflight cors requests (#3481) 2022-08-22 08:21:38 -07:00
chrislu
42c6e52513 s3: fix regression on HEAD directory operation 2022-08-18 02:13:58 -07:00
chrislu
4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
Andrey Triumfov
31faa6d43d
Remove duplicate slashes in object path to prevent 500 errors (#3442) 2022-08-15 08:19:28 -07:00
Chris Lu
e88392b50f
Fix s3 pagination (#3436) 
* Revert previous changes

* s3: use cursor to track tree traversal

fix https://github.com/seaweedfs/seaweedfs/issues/3166

* special cases for empty prefix and empty directory

* use constants

* address empty folder

* undo local changes

* fix IsTruncated

* adjust counting directories

* fix cases when prefix is a directory

* s3: handle directory object

works for

aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
 2022-08-15 00:30:19 -07:00
chrislu
7457c746f0 s3: fix aws s3api head-object 2022-08-14 23:52:35 -07:00
chrislu
67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
chrislu
90db4b5a44 minor 2022-08-03 00:10:47 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag 
File upload should succeed should return `ETag` instead of `Etag`
 2022-07-14 15:35:41 -07:00
LHHDZ
c08f6d0ff7 File upload should succeed should return ETag instead of Etag 
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
 2022-07-14 19:45:01 +08:00
guosj
ab1b9697e6 supplement check duplicate accesskey 2022-07-13 17:28:20 +08:00
chrislu
69ef6459c3 S3 ListObjectsV2 fix pagination under bucket root 
fix https://github.com/chrislusf/seaweedfs/issues/3166
 2022-07-06 02:28:34 -07:00
Konstantin Lebedev
3d5f1cbc5e fix pagination 2022-07-01 14:17:49 +05:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder 
avoid error bucket NotEmpty if multipart uploads folder exist
 2022-06-29 16:21:16 +05:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy 
validate tags on copy object and add regex for validating tags
 2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535 remove println 2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2 fix return 2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c fix return 2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913 fix parseTagging function and add test for validate tags 2022-06-28 14:48:55 +03:00
Lapshin Vitaliy
1a5981d583 fix test 2022-06-27 18:46:21 +03:00
Lapshin Vitaliy
d7c3493d15 Merge branch 'master' into bufix/validate-tags-on-copy 2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a adjust log message 2022-06-26 23:12:16 -07:00
石昌林
15bfc60a88 Remove the collection query param of s3api and let the collection be determined by the filer 2022-06-27 13:47:53 +08:00
石昌林
be5c901bea Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems 2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48 validate tags on copy object and add regex for validating tags 2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode 
AbortMultipartUploadHandler should return 204 instead of 200
 2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0 AbortMultipartUploadHandler should return 204 instead of 200 
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
 2022-06-20 13:26:22 +08:00
石昌林
9e036df356 remove go.uber.org/atomic 2022-06-20 12:35:29 +08:00
石昌林
3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林
37df209195 add some unit tests and some code optimizes 2022-06-17 17:11:18 +08:00
zzq09494
62f74f5e3c add bucket label to s3 prometheus metrics 2022-06-16 13:21:25 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
shichanglin5
f01dd27752 fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup 2022-06-13 13:23:26 +08:00
Konstantin Lebedev
49f058da85 fix s3 tests: 
ranged_request_invalid_range
ranged_request_empty_object
 2022-06-08 13:25:13 +05:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f fix s3 tests 
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
 2022-06-07 14:43:10 +05:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d fix show empty dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-06-01 20:10:52 +05:00
chrislu
27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu
596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers 
Filer bootstrap from peers
 2022-05-30 21:38:31 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud 
Create folders in s3 via nextcloud
 2022-05-30 13:10:10 -07:00
Konstantin Lebedev
af562e1a20 rm log info 2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c list self dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961 show fake dirs 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46 s3: keep auth enabled in case identities are set to empty 
fix https://github.com/chrislusf/seaweedfs/issues/3084
 2022-05-29 19:00:03 -07:00
Konstantin Lebedev
54356211c2 list empty folder 2022-05-27 13:17:18 +05:00
chrislu
866fed1eab avoid grpc name conflict 
fix https://github.com/chrislusf/seaweedfs/issues/3055
 2022-05-15 21:31:29 -07:00
chrislu
4076d74813 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-15 00:43:39 -07:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
shichanglin5
688d55488c test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go 2022-05-14 10:40:29 +08:00
shichanglin5
1166dead00 fix the problem of metadata and tagging loss when files are copied 
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
 2022-05-13 19:46:20 +08:00
chrislu
f7366a9668 skip unix socket mode for windows 
fix https://github.com/chrislusf/seaweedfs/issues/3013
 2022-05-04 10:14:34 -07:00
chrislu
f17cd0d5cd return false if not found 
fix https://github.com/chrislusf/seaweedfs/issues/3011
 2022-05-03 07:18:34 -07:00
chrislu
00c1dfec4f go fmt 2022-05-01 23:16:29 -07:00
Konstantin Lebedev
306cf70c4a avoid empty listMultipartUploads response 2022-04-27 19:27:44 +05:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
guol-fnst
44d810d163 rename functions and remove uncessary check 2022-04-12 13:40:53 +08:00
guol-fnst
180aa88a92 check uploadid using object name hash string 2022-04-12 11:04:38 +08:00
guol-fnst
2232cfb5b7 Check object name and uploadID when processing multipart uploading 2022-04-11 19:53:44 +08:00
root
f15a737a0a Add "Location:" in response when creating bucket 
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html"
 2022-04-06 13:56:27 +08:00
root
b7e34016d4 fix error response in case of wrong expire time format 2022-04-02 11:01:33 +08:00
Konstantin Lebedev
691ed50957 use constant for ErrBucketNotEmpty 2022-03-31 08:44:48 +05:00
Konstantin Lebedev
b6a2b43efe allowDeleteBucketNotEmpty 2022-03-30 22:46:13 +05:00
Konstantin Lebedev
a8b15c676c s3 test bucket delete nonempty 2022-03-30 21:06:26 +05:00
Chris Lu
ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit 
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
 2022-03-30 01:27:13 -07:00
Konstantin Lebedev
6317f6a8a9 fix Description 2022-03-30 13:04:15 +05:00
Konstantin Lebedev
993283bb1d revert sort.Search 2022-03-29 19:53:12 +05:00