Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode
...
AbortMultipartUploadHandler should return 204 instead of 200
2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0
AbortMultipartUploadHandler should return 204 instead of 200
...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
2022-06-20 13:26:22 +08:00
石昌林
9e036df356
remove go.uber.org/atomic
2022-06-20 12:35:29 +08:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
zzq09494
62f74f5e3c
add bucket label to s3 prometheus metrics
2022-06-16 13:21:25 +08:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
shichanglin5
f01dd27752
fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup
2022-06-13 13:23:26 +08:00
Konstantin Lebedev
49f058da85
fix s3 tests:
...
ranged_request_invalid_range
ranged_request_empty_object
2022-06-08 13:25:13 +05:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f
fix s3 tests
...
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d
fix show empty dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-06-01 20:10:52 +05:00
chrislu
27732ecfa4
move s3 related constants from package http to s3_constants
2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-30 22:47:29 -07:00
chrislu
596c3860ca
use final destination to resolve fs configuration
...
related to https://github.com/chrislusf/seaweedfs/issues/3075
2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers
...
Filer bootstrap from peers
2022-05-30 21:38:31 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud
...
Create folders in s3 via nextcloud
2022-05-30 13:10:10 -07:00
Konstantin Lebedev
af562e1a20
rm log info
2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c
list self dir
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961
show fake dirs
...
https://github.com/chrislusf/seaweedfs/issues/3086
2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46
s3: keep auth enabled in case identities are set to empty
...
fix https://github.com/chrislusf/seaweedfs/issues/3084
2022-05-29 19:00:03 -07:00
Konstantin Lebedev
54356211c2
list empty folder
2022-05-27 13:17:18 +05:00
chrislu
866fed1eab
avoid grpc name conflict
...
fix https://github.com/chrislusf/seaweedfs/issues/3055
2022-05-15 21:31:29 -07:00
chrislu
4076d74813
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-05-15 00:43:39 -07:00
chrislu
29198720f2
s3: add grpc server to accept configuration changes
2022-05-15 00:43:37 -07:00
shichanglin5
688d55488c
test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go
2022-05-14 10:40:29 +08:00
shichanglin5
1166dead00
fix the problem of metadata and tagging loss when files are copied
...
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
2022-05-13 19:46:20 +08:00
chrislu
f7366a9668
skip unix socket mode for windows
...
fix https://github.com/chrislusf/seaweedfs/issues/3013
2022-05-04 10:14:34 -07:00
chrislu
f17cd0d5cd
return false if not found
...
fix https://github.com/chrislusf/seaweedfs/issues/3011
2022-05-03 07:18:34 -07:00
chrislu
00c1dfec4f
go fmt
2022-05-01 23:16:29 -07:00
Konstantin Lebedev
306cf70c4a
avoid empty listMultipartUploads response
2022-04-27 19:27:44 +05:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc
2022-04-18 10:39:29 +08:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
guol-fnst
44d810d163
rename functions and remove uncessary check
2022-04-12 13:40:53 +08:00
guol-fnst
180aa88a92
check uploadid using object name hash string
2022-04-12 11:04:38 +08:00
guol-fnst
2232cfb5b7
Check object name and uploadID when processing multipart uploading
2022-04-11 19:53:44 +08:00
root
f15a737a0a
Add "Location:" in response when creating bucket
...
according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html "
2022-04-06 13:56:27 +08:00
root
b7e34016d4
fix error response in case of wrong expire time format
2022-04-02 11:01:33 +08:00
Konstantin Lebedev
691ed50957
use constant for ErrBucketNotEmpty
2022-03-31 08:44:48 +05:00
Konstantin Lebedev
b6a2b43efe
allowDeleteBucketNotEmpty
2022-03-30 22:46:13 +05:00
Konstantin Lebedev
a8b15c676c
s3 test bucket delete nonempty
2022-03-30 21:06:26 +05:00
Chris Lu
ce8ef60c59
Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit
...
fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit
2022-03-30 01:27:13 -07:00
Konstantin Lebedev
6317f6a8a9
fix Description
2022-03-30 13:04:15 +05:00
Konstantin Lebedev
993283bb1d
revert sort.Search
2022-03-29 19:53:12 +05:00
Konstantin Lebedev
5b90a39954
fix s3 tests:
...
multipart_upload_incorrect_etag
multipart_resend_first_finishes_last
2022-03-29 19:21:09 +05:00
Konstantin Lebedev
9231971c88
fix test_s3.test_multi_object_delete_key_limit and
...
test_multi_objectv2_delete_key_limit
2022-03-29 16:55:09 +05:00
chrislu
e48764be75
s3: multipart upload verifies uploaded parts
2022-03-23 01:05:14 -07:00
chrislu
18ae8943ef
better clean up upload directory
2022-03-23 01:03:51 -07:00
chrislu
da3d330616
s3 and filer transport using unix domain socket instead of tcp
2022-03-07 02:00:14 -08:00
chrislu
0cb17b45b1
refactoring
2022-03-07 01:59:01 -08:00
chrislu
a96d4254e9
filer, s3, volume server: a bit memory optimization
2022-03-02 20:15:28 -08:00
Chris Lu
e1d3dd5e18
Merge pull request #2712 from guo-sj/correct_comments
2022-02-26 09:16:12 -08:00
guosj
82cad5e330
correct comments
2022-02-26 22:44:26 +08:00
Chris Lu
9873bae115
Merge pull request #2706 from guo-sj/fix_log_info
2022-02-24 22:48:19 -08:00
guosj
121b31f750
fix incorrect log information
2022-02-25 13:41:20 +08:00
chrislu
61811dc2f1
comments
2022-02-18 22:14:40 -08:00
chrislu
9014d00fd0
Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist"
...
This reverts commit 6cf2e7d493
.
2022-02-18 20:54:54 -08:00
chrislu
81f86c381d
s3: avoid nil response
...
fix https://github.com/chrislusf/seaweedfs/issues/2636
2022-02-07 11:37:59 -08:00
Chris Lu
247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors
...
add s3api error for copy in file, not directory
2022-02-04 22:34:34 -08:00
zerospiel
f3364fec99
weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting
...
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).
Added generic comment rule of traversing methods
2022-02-04 15:14:48 +03:00
Lapshinn Vitaly
6bdc274d4d
add s3api error for copy in file, not directory
2022-02-04 03:28:37 +03:00
chrislu
6cf2e7d493
s3: listObjectParts return ErrNoSuchUpload if does not exist
...
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
"Initiator": null,
"Owner": null,
"StorageClass": "STANDARD"
}
If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
2022-02-03 12:34:16 -08:00
zerospiel
b54a65ba5a
weed/s3api: added new bucket handlers for more compatibility with AWS S3
...
Protocol
Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.
Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment
Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
2022-02-03 17:17:05 +03:00
chrislu
77362700e1
S3: fail fast when "X-Amz-Copy-Source" is a folder
...
fix #2593
2022-01-18 12:04:40 -08:00
chrislu
c87b8f4c30
S3: fail fast when "X-Amz-Copy-Source" is a folder
...
fix https://github.com/chrislusf/seaweedfs/issues/2593
2022-01-17 23:09:37 -08:00
Konstantin Lebedev
edb753ab4d
https://github.com/chrislusf/seaweedfs/issues/2583
2022-01-12 16:04:59 +05:00
chrislu
e76105e2ab
fix auth permission checking
2022-01-03 21:05:20 -08:00
chrislu
a7887166cf
wildcard prefix to restrict access to directories in s3 bucket
...
https://github.com/chrislusf/seaweedfs/discussions/2551
2022-01-03 15:39:36 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d
BUGFIX: ensure Authorization header is only added once
2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
chrislu
5788bf2270
s3: increase timeout limit
...
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu
7210558c7b
s3: pass through s3 presigned headers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
Konstantin Lebedev
969f513265
disable audit log for s3 statusHandler
2021-12-13 16:08:19 +05:00
Konstantin Lebedev
34779e8f38
force enable asynchronous I/O sending events to Fluentd
2021-12-13 13:39:39 +05:00
kmlebedev
4f98553ba9
audit log SignatureVersion
2021-12-10 19:40:32 +05:00
Konstantin Lebedev
98251fe16a
non blocking audit log
2021-12-09 19:47:16 +05:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log
2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81
audit log config
2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20
audit log
2021-12-07 12:15:48 +05:00
Konstantin Lebedev
3ac48cd540
audit log
2021-12-07 12:13:21 +05:00
chrislu
5c3b783310
s3: copy object API needs to escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/2482
2021-12-04 23:24:53 -08:00
Konstantin Lebedev
40abae7caa
Avoid forbiddening for HeadBucketHandler if owners are specified
...
https://github.com/chrislusf/seaweedfs/issues/2434
2021-11-17 14:42:24 +05:00
Konstantin Lebedev
42f4aa824b
force overwrite s3-is-admin
...
https://github.com/chrislusf/seaweedfs/issues/2433
2021-11-11 11:34:01 +05:00
Bl1tz23
c683409e92
s3: add RWMutex to iam, use RLock for concurrent reading
2021-11-09 18:11:06 +03:00
Bl1tz23
5018b22f36
s3: fix potencial iam identities data race
2021-11-09 12:19:50 +03:00
Chris Lu
59d1435d65
s3: avoid possible attacks by version
2021-11-08 17:47:56 -08:00
Chris Lu
244841f080
adjust logs
2021-11-07 13:27:57 -08:00
Chris Lu
6e999f1176
s3: fix ListObject if more than 10000 objects
2021-11-07 12:39:36 -08:00
Chris Lu
55973e8572
log unknown access key
2021-11-07 12:37:46 -08:00
Chris Lu
35c37562bc
check auth only when enabled
2021-11-03 12:11:36 -07:00
Chris Lu
e1ab8b01d0
s3: adjust permission for HEAD bucket operation
...
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
2021-11-02 16:28:55 -07:00
Chris Lu
b25661c6df
s3: skip permission checking for creating bucket if the bucket already exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2417
Rclone was trying to create the bucket even though the bucket already exists.
2021-11-02 13:13:36 -07:00
Chris Lu
9cf756b188
S3: support CORS
...
related to https://github.com/chrislusf/seaweedfs/issues/1271
2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5
refactoring
2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8
refactoring
2021-10-31 18:02:08 -07:00
Konstantin Lebedev
dc3ae5e8a6
DeleteBucketLifecycleConfiguration return 204
2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af
fix DeleteBucketLifecycleConfiguration
2021-10-28 18:30:33 +05:00
Chris Lu
54b6e0f3fd
adjust logs
2021-10-27 23:46:07 -07:00
Konstantin Lebedev
235329a92a
fix Tagging add xmlns
2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9
fix Tagging test
2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5
fix PutObjectTaggingHandler Unmarshal
2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2
fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389
2021-10-20 16:01:06 +05:00
Chris Lu
309f46e3af
S3: fix upload limit if the size is more than 8GB
2021-10-18 14:14:18 -07:00
Chris Lu
a6bb509cb7
adjust mime detection
2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf
fix https://github.com/chrislusf/seaweedfs/issues/2387
2021-10-18 16:27:57 +05:00
Chris Lu
004e56c1a6
save updated lastTsNs
2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f
s3: continuously listen to file updates
...
fix https://github.com/chrislusf/seaweedfs/issues/2342
2021-10-17 01:04:19 -07:00
Konstantin Lebedev
8de7915950
s3 mime detect
2021-10-14 15:03:11 +05:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy
...
Bucket policy
2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1
GetBucketLifecycleConfigurationHandler
2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e
S3 MultipartUpload pass contentType to meta
2021-10-12 17:14:54 +05:00
Konstantin Lebedev
be4b3ed509
AclHandlers
2021-10-11 15:03:56 +05:00
Chris Lu
84fdda85e7
go fmt
2021-09-19 12:06:15 -07:00
Chris Lu
5abdc0be77
s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests
2021-09-19 03:24:47 -07:00
Chris Lu
59dd271734
more glog
2021-09-19 00:29:51 -07:00
Chris Lu
4cbba2b1c6
add more glog to s3
2021-09-19 00:28:22 -07:00
Chris Lu
71175461ef
add glog for s3 handlers
2021-09-19 00:18:59 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
f62df7f695
fix wrong srcPath checking
2021-09-05 23:37:40 -07:00
Chris Lu
491f7636f8
s3: CopyObject return http Status 400 Bad Request for non-existing source
...
fix https://github.com/chrislusf/seaweedfs/issues/2306
2021-09-03 17:38:39 -07:00
Chris Lu
5a0f92423e
use grpc and jwt
2021-08-12 21:40:33 -07:00
Bl1tz23
1c94b3d013
merge master, resolve conflicts
2021-08-10 13:45:24 +03:00
Bl1tz23
e6e57db530
Add liveness\readiness probe for s3 api handler on /status path
2021-08-10 13:42:46 +03:00
Chris Lu
6b743dbbf9
refactor client subscribe metadata
2021-08-04 16:25:46 -07:00
Chris Lu
7359193e97
go fmt
2021-07-21 14:38:12 -07:00
Chris Lu
f0042f62dd
readable logs
2021-07-15 12:17:48 -07:00
Chris Lu
ecce300964
s3 config read via grpc
2021-07-09 02:48:03 -07:00
Chris Lu
a024254ad7
logging
2021-07-03 14:51:01 -07:00
Chris Lu
d39b2689a5
S3 authorization: StreamingSigned enforces access control
...
fix https://github.com/chrislusf/seaweedfs/issues/2180
2021-07-03 14:50:53 -07:00
lyg
1a4db87e19
fix s3 metadata error with multipart upload
2021-07-02 11:00:42 +08:00
Chris Lu
c6d4c16079
S3: add metadata with multipart upload
...
fix https://github.com/chrislusf/seaweedfs/issues/2173
2021-07-01 19:12:11 -07:00
Chris Lu
b71c3cfba4
avoid possible empty object
2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e
adjust the error output
...
fix https://github.com/chrislusf/seaweedfs/issues/2123
2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209
refactor
2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef
refactoring
2021-06-10 16:54:36 -07:00
Chris Lu
2b60e2abb1
only disallow streaming signed when no auth enabled
...
fix https://github.com/chrislusf/seaweedfs/issues/2101
2021-05-31 01:03:04 -07:00
Konstantin Lebedev
b612d5aebd
s3 test get w/ If-Match: bogus ETag
2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b
- object write cache control
...
- object write expires
2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks
2021-05-24 12:28:19 +05:00
Chris Lu
431684798b
s3: add errors if requests are signed by no authentication is setup
...
fix https://github.com/chrislusf/seaweedfs/issues/2075
2021-05-21 14:08:47 -07:00
Konstantin Lebedev
03d1199d5f
Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader"
...
This reverts commit ac71117e
2021-05-20 11:45:21 +05:00
Chris Lu
26a55bbb5c
Adjust error message when bucket name conflicts with existing collections
...
fix https://github.com/chrislusf/seaweedfs/issues/2069
2021-05-12 22:30:39 -07:00
Chris Lu
ac71117ee6
revert PR #1903 avoid http error: superfluous response.WriteHeader
2021-05-05 15:11:39 -07:00
Chris Lu
e87e6ef33c
s3: return 404 if bucket does not exist
...
fix https://github.com/chrislusf/seaweedfs/issues/2039
2021-05-02 21:30:37 -07:00
Konstantin Lebedev
a48785c7df
auth use bucket wild cards
2021-04-27 21:45:40 +05:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers
...
IAM handlers
2021-04-12 12:07:45 -07:00
Merlin Gaillard
4d4acc715e
s3api: handle 304 response code from filer
2021-04-09 12:13:19 +02:00
Konstantin Lebedev
ba175f81b5
add auth aws signV4
2021-04-08 17:40:47 +05:00
Chris Lu
b1a86cf808
s3: copy object to itself
...
fix https://github.com/chrislusf/seaweedfs/issues/1922
2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920
add line
2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35
dir + slash key does not exist
...
https://github.com/chrislusf/seaweedfs/issues/1917
2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e
Merge branch 'upstreamMaster' into check_chunkviews_mr
...
# Conflicts:
# weed/filer/filechunk_manifest.go
# weed/filer/stream.go
# weed/replication/repl_util/replication_util.go
# weed/util/fasthttp_util.go
2021-03-16 15:29:49 +05:00
wuh-fnst
050a7f9599
s3: count correction for the number of files in the dir
2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b
Status PartialContent for Content-Range response
2021-03-15 23:30:22 +05:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder
...
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-11 11:01:44 -08:00
Chris Lu
79280e1513
escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-11 09:49:40 -08:00
wuheng
828fbf3fb7
s3: "isLast" returns true when the file does not exist
2021-03-11 15:20:50 +08:00
Chris Lu
ca546f47e1
s3: escape object key if containing special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/1884
2021-03-10 13:19:28 -08:00
Konstantin Lebedev
bb3af2d70b
no directoriesWithDeletion
2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34
s3 delete-objects pass OK for fail to delete non-empty folder
2021-03-10 19:41:35 +05:00
Chris Lu
387c6f4218
Revert "delete the folder object in multi_object_delete"
...
This reverts commit 2e89c8c9ae
.
2021-03-09 10:07:27 -08:00
Chris Lu
2e89c8c9ae
delete the folder object in multi_object_delete
...
fix https://github.com/chrislusf/seaweedfs/issues/1857
2021-03-04 14:03:40 -08:00
Chris Lu
5511722420
s3: list permission is needed to list my buckets
...
fix https://github.com/chrislusf/seaweedfs/issues/1837
2021-03-01 12:41:55 -08:00
bingoohuang
352ac2f271
Merge remote-tracking branch 'origin/master'
2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34
use backticks instead of double quotes to avoid escaped additionally in regex
2021-02-18 14:05:28 +08:00
Chris Lu
8b169aa47c
s3: Critical - S3 Delete Multiple Objects, delete the bucket
...
fix https://github.com/chrislusf/seaweedfs/issues/1806
2021-02-15 13:38:29 -08:00
Chris Lu
487e435679
adjust http max idle connections per host
...
related to https://github.com/chrislusf/seaweedfs/issues/1802
2021-02-12 03:47:15 -08:00
Chris Lu
a0e84c4fbc
go fmt
2021-02-10 23:41:05 -08:00
Chris Lu
5353e38469
S3: should return 204 on DELETE to nonexistent file
...
fix https://github.com/chrislusf/seaweedfs/issues/1776
2021-02-03 00:35:44 -08:00
Chris Lu
609daaf387
s3: DeleteMultipleObjectsHandler clean up leftover empty folders
...
fix https://github.com/chrislusf/seaweedfs/issues/1772
2021-02-01 10:49:17 -08:00
Chris Lu
990fa69bfe
add back AdjustedUrl() related code
2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2
fmt
2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac
delete from the deepest directory first when checking empty folders
2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925
s3: batch purge empty folders
2021-01-28 13:20:06 -08:00
Chris Lu
00707ec00f
mount: outsideContainerClusterMode proxy through filer
...
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8
Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
...
This reverts commit 096e088d7b
.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b
mount: when outside cluster network, use filer as proxy to access volume servers
2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89
s3: fix for listing objects if more than 1000 in the folder
...
many existing tools have max key set to 1000
2021-01-24 00:27:52 -08:00
Chris Lu
389426bbb7
s3: listing may repeat on the edge
2021-01-17 22:59:31 -08:00
Chris Lu
8a7302ce37
s3: KeyCount is zero
...
fix https://github.com/chrislusf/seaweedfs/issues/1741
2021-01-11 03:50:45 -08:00
Chris Lu
e327385644
go fmt
2021-01-06 04:22:00 -08:00
Chris Lu
6d2c979999
s3: fix regression
...
fix https://github.com/chrislusf/seaweedfs/issues/1707
2020-12-27 21:09:45 -08:00
Chris Lu
561a2ff0bc
s3: add option for "alllowEmptyFolder"
2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7
s3: restore V2 auth type
...
The added headers were included in calculating the signatures, failing the comparison
2020-12-26 22:01:16 -08:00
Chris Lu
090f85be4b
s3: support config action Admin:bucket
2020-12-25 00:38:56 -08:00
Chris Lu
8e48a235e2
s3: avoid looping if the directory is empty
...
fix https://github.com/chrislusf/seaweedfs/issues/1701
2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae
added more logs
2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986
add debug messages
2020-12-24 01:42:15 -08:00
Chris Lu
3be3635799
just log error when checking empty dir failed
2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589
s3: break loop if error
...
fix #1701
2020-12-23 14:34:59 -08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
5c465293e9
correctly determine whether a folder is empty
...
avoid edge cases that deleting the folder if previous 32 directories are all empty
early terminate if one file is found
2020-12-12 16:19:29 -08:00