mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-05 03:29:03 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,067 Commits 32 Branches 296 Tags 149 MiB
bf5a5abfb1
Commit Graph

392 Commits

Author SHA1 Message Date
Chris Lu
9cf756b188 S3: support CORS 
related to https://github.com/chrislusf/seaweedfs/issues/1271
 2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Konstantin Lebedev
dc3ae5e8a6 DeleteBucketLifecycleConfiguration return 204 2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af fix DeleteBucketLifecycleConfiguration 2021-10-28 18:30:33 +05:00
Chris Lu
54b6e0f3fd adjust logs 2021-10-27 23:46:07 -07:00
Konstantin Lebedev
235329a92a fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9 fix Tagging test 2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5 fix PutObjectTaggingHandler Unmarshal 2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu
309f46e3af S3: fix upload limit if the size is more than 8GB 2021-10-18 14:14:18 -07:00
Chris Lu
a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Chris Lu
004e56c1a6 save updated lastTsNs 2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f s3: continuously listen to file updates 
fix https://github.com/chrislusf/seaweedfs/issues/2342
 2021-10-17 01:04:19 -07:00
Konstantin Lebedev
8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy 
Bucket policy
 2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e S3 MultipartUpload pass contentType to meta 2021-10-12 17:14:54 +05:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
84fdda85e7 go fmt 2021-09-19 12:06:15 -07:00
Chris Lu
5abdc0be77 s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests 2021-09-19 03:24:47 -07:00
Chris Lu
59dd271734 more glog 2021-09-19 00:29:51 -07:00
Chris Lu
4cbba2b1c6 add more glog to s3 2021-09-19 00:28:22 -07:00
Chris Lu
71175461ef add glog for s3 handlers 2021-09-19 00:18:59 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
f62df7f695 fix wrong srcPath checking 2021-09-05 23:37:40 -07:00
Chris Lu
491f7636f8 s3: CopyObject return http Status 400 Bad Request for non-existing source 
fix https://github.com/chrislusf/seaweedfs/issues/2306
 2021-09-03 17:38:39 -07:00
Chris Lu
5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Bl1tz23
1c94b3d013 merge master, resolve conflicts 2021-08-10 13:45:24 +03:00
Bl1tz23
e6e57db530 Add liveness\readiness probe for s3 api handler on /status path 2021-08-10 13:42:46 +03:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
f0042f62dd readable logs 2021-07-15 12:17:48 -07:00
Chris Lu
ecce300964 s3 config read via grpc 2021-07-09 02:48:03 -07:00
Chris Lu
a024254ad7 logging 2021-07-03 14:51:01 -07:00
Chris Lu
d39b2689a5 S3 authorization: StreamingSigned enforces access control 
fix https://github.com/chrislusf/seaweedfs/issues/2180
 2021-07-03 14:50:53 -07:00
lyg
1a4db87e19 fix s3 metadata error with multipart upload 2021-07-02 11:00:42 +08:00
Chris Lu
c6d4c16079 S3: add metadata with multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/2173
 2021-07-01 19:12:11 -07:00
Chris Lu
b71c3cfba4 avoid possible empty object 2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e adjust the error output 
fix https://github.com/chrislusf/seaweedfs/issues/2123
 2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209 refactor 2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef refactoring 2021-06-10 16:54:36 -07:00
Chris Lu
2b60e2abb1 only disallow streaming signed when no auth enabled 
fix https://github.com/chrislusf/seaweedfs/issues/2101
 2021-05-31 01:03:04 -07:00
Konstantin Lebedev
b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Chris Lu
431684798b s3: add errors if requests are signed by no authentication is setup 
fix https://github.com/chrislusf/seaweedfs/issues/2075
 2021-05-21 14:08:47 -07:00
Konstantin Lebedev
03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu
26a55bbb5c Adjust error message when bucket name conflicts with existing collections 
fix https://github.com/chrislusf/seaweedfs/issues/2069
 2021-05-12 22:30:39 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
e87e6ef33c s3: return 404 if bucket does not exist 
fix https://github.com/chrislusf/seaweedfs/issues/2039
 2021-05-02 21:30:37 -07:00
Konstantin Lebedev
a48785c7df auth use bucket wild cards 2021-04-27 21:45:40 +05:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Konstantin Lebedev
27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu
781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu
7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
ruitao.liu
a9990a1dc6 adjust check bucket if exist or has access. 2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a enable admin to access all buckets 2020-11-12 13:57:54 -08:00
Chris Lu
7094492428 fix ErrNotFound 2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019 rename function 2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master 
check permission for bucket delete/head.
 2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b skip if entry.Extended map is nil. 2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2 return NoSuchBucket instead of InternalError delete non-existed bucket. 2020-11-12 16:44:16 +08:00
ruitao.liu
e06676f007 check permission for bucket delete/head. 2020-11-12 16:15:59 +08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master 
add owner attr for bucket.
 2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0 check if bucket already exists. 2020-11-12 14:11:03 +08:00
ruitao.liu
dbba8cb57e fix if nil map in entry. 2020-11-12 11:50:19 +08:00
Chris Lu
86cdb2a3e5 adjust logs 2020-11-11 02:01:24 -08:00
ruitao.liu
5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Konstantin Lebedev
6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Konstantin Lebedev
e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
ruitao.liu
22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
ruitao.liu
b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Chris Lu
6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
ruitao.liu
8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Konstantin Lebedev
05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00