mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-24 01:17:56 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,069 Commits 37 Branches 296 Tags 152 MiB
1c2b10303a
Commit Graph

544 Commits

Author SHA1 Message Date
Chris Lu
e1ab8b01d0 s3: adjust permission for HEAD bucket operation 
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
 2021-11-02 16:28:55 -07:00
Chris Lu
b25661c6df s3: skip permission checking for creating bucket if the bucket already exists 
fix https://github.com/chrislusf/seaweedfs/issues/2417

Rclone was trying to create the bucket even though the bucket already exists.
 2021-11-02 13:13:36 -07:00
Chris Lu
9cf756b188 S3: support CORS 
related to https://github.com/chrislusf/seaweedfs/issues/1271
 2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5 refactoring 2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8 refactoring 2021-10-31 18:02:08 -07:00
Konstantin Lebedev
dc3ae5e8a6 DeleteBucketLifecycleConfiguration return 204 2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af fix DeleteBucketLifecycleConfiguration 2021-10-28 18:30:33 +05:00
Chris Lu
54b6e0f3fd adjust logs 2021-10-27 23:46:07 -07:00
Konstantin Lebedev
235329a92a fix Tagging add xmlns 2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9 fix Tagging test 2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5 fix PutObjectTaggingHandler Unmarshal 2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu
309f46e3af S3: fix upload limit if the size is more than 8GB 2021-10-18 14:14:18 -07:00
Chris Lu
a6bb509cb7 adjust mime detection 2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf fix https://github.com/chrislusf/seaweedfs/issues/2387 2021-10-18 16:27:57 +05:00
Chris Lu
004e56c1a6 save updated lastTsNs 2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f s3: continuously listen to file updates 
fix https://github.com/chrislusf/seaweedfs/issues/2342
 2021-10-17 01:04:19 -07:00
Konstantin Lebedev
8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy 
Bucket policy
 2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e S3 MultipartUpload pass contentType to meta 2021-10-12 17:14:54 +05:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
84fdda85e7 go fmt 2021-09-19 12:06:15 -07:00
Chris Lu
5abdc0be77 s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests 2021-09-19 03:24:47 -07:00
Chris Lu
59dd271734 more glog 2021-09-19 00:29:51 -07:00
Chris Lu
4cbba2b1c6 add more glog to s3 2021-09-19 00:28:22 -07:00
Chris Lu
71175461ef add glog for s3 handlers 2021-09-19 00:18:59 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
f62df7f695 fix wrong srcPath checking 2021-09-05 23:37:40 -07:00
Chris Lu
491f7636f8 s3: CopyObject return http Status 400 Bad Request for non-existing source 
fix https://github.com/chrislusf/seaweedfs/issues/2306
 2021-09-03 17:38:39 -07:00
Chris Lu
5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Bl1tz23
1c94b3d013 merge master, resolve conflicts 2021-08-10 13:45:24 +03:00
Bl1tz23
e6e57db530 Add liveness\readiness probe for s3 api handler on /status path 2021-08-10 13:42:46 +03:00
Chris Lu
6b743dbbf9 refactor client subscribe metadata 2021-08-04 16:25:46 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
f0042f62dd readable logs 2021-07-15 12:17:48 -07:00
Chris Lu
ecce300964 s3 config read via grpc 2021-07-09 02:48:03 -07:00
Chris Lu
a024254ad7 logging 2021-07-03 14:51:01 -07:00
Chris Lu
d39b2689a5 S3 authorization: StreamingSigned enforces access control 
fix https://github.com/chrislusf/seaweedfs/issues/2180
 2021-07-03 14:50:53 -07:00
lyg
1a4db87e19 fix s3 metadata error with multipart upload 2021-07-02 11:00:42 +08:00
Chris Lu
c6d4c16079 S3: add metadata with multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/2173
 2021-07-01 19:12:11 -07:00
Chris Lu
b71c3cfba4 avoid possible empty object 2021-06-10 22:17:53 -07:00
Chris Lu
310e31424e adjust the error output 
fix https://github.com/chrislusf/seaweedfs/issues/2123
 2021-06-10 21:55:13 -07:00
Chris Lu
8b382a8209 refactor 2021-06-10 21:50:21 -07:00
Chris Lu
33b87244ef refactoring 2021-06-10 16:54:36 -07:00
Chris Lu
2b60e2abb1 only disallow streaming signed when no auth enabled 
fix https://github.com/chrislusf/seaweedfs/issues/2101
 2021-05-31 01:03:04 -07:00
Konstantin Lebedev
b612d5aebd s3 test get w/ If-Match: bogus ETag 2021-05-24 16:59:44 +05:00
Konstantin Lebedev
69c768870b - object write cache control 
- object write expires
 2021-05-24 15:43:55 +05:00
Konstantin Lebedev
84dce32a57
Merge branch 'master' into head_check_all_chunks 2021-05-24 12:28:19 +05:00
Chris Lu
431684798b s3: add errors if requests are signed by no authentication is setup 
fix https://github.com/chrislusf/seaweedfs/issues/2075
 2021-05-21 14:08:47 -07:00
Konstantin Lebedev
03d1199d5f Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" 
This reverts commit ac71117e
 2021-05-20 11:45:21 +05:00
Chris Lu
26a55bbb5c Adjust error message when bucket name conflicts with existing collections 
fix https://github.com/chrislusf/seaweedfs/issues/2069
 2021-05-12 22:30:39 -07:00
Chris Lu
ac71117ee6 revert PR #1903 avoid http error: superfluous response.WriteHeader 2021-05-05 15:11:39 -07:00
Chris Lu
e87e6ef33c s3: return 404 if bucket does not exist 
fix https://github.com/chrislusf/seaweedfs/issues/2039
 2021-05-02 21:30:37 -07:00
Konstantin Lebedev
a48785c7df auth use bucket wild cards 2021-04-27 21:45:40 +05:00
Chris Lu
f5de42fae3
Merge pull request #1975 from kmlebedev/iam_handlers 
IAM handlers
 2021-04-12 12:07:45 -07:00
Merlin Gaillard
4d4acc715e s3api: handle 304 response code from filer 2021-04-09 12:13:19 +02:00
Konstantin Lebedev
ba175f81b5 add auth aws signV4 2021-04-08 17:40:47 +05:00
Chris Lu
b1a86cf808 s3: copy object to itself 
fix https://github.com/chrislusf/seaweedfs/issues/1922
 2021-03-19 01:31:49 -07:00
Konstantin Lebedev
bf94149920 add line 2021-03-17 23:41:34 +05:00
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Konstantin Lebedev
27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu
781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu
7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
ruitao.liu
a9990a1dc6 adjust check bucket if exist or has access. 2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a enable admin to access all buckets 2020-11-12 13:57:54 -08:00
Chris Lu
7094492428 fix ErrNotFound 2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019 rename function 2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master 
check permission for bucket delete/head.
 2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b skip if entry.Extended map is nil. 2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2 return NoSuchBucket instead of InternalError delete non-existed bucket. 2020-11-12 16:44:16 +08:00
ruitao.liu
e06676f007 check permission for bucket delete/head. 2020-11-12 16:15:59 +08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master 
add owner attr for bucket.
 2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0 check if bucket already exists. 2020-11-12 14:11:03 +08:00
ruitao.liu
dbba8cb57e fix if nil map in entry. 2020-11-12 11:50:19 +08:00
Chris Lu
86cdb2a3e5 adjust logs 2020-11-11 02:01:24 -08:00
ruitao.liu
5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Konstantin Lebedev
6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Konstantin Lebedev
e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
ruitao.liu
22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
ruitao.liu
b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Chris Lu
6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
ruitao.liu
8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Konstantin Lebedev
05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd
1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
limd
a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu
9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu
b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu
9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu
d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu
dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu
5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu
2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu
3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Chris Lu
81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu
c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu
828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu
2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
limd
b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu
a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd
92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu
54cd582653 refactor 2020-07-26 12:58:58 -07:00
Chris Lu
c50ec1599c auth for presigned put object part 2020-07-26 12:09:50 -07:00
Chris Lu
ea0180de4e auth for presigned put operation 2020-07-26 12:06:45 -07:00
popstk
a09fcfcf20 fix s3api auth bug 2020-07-25 17:14:49 +08:00
popstk
feb52400be fix s3api copy object handler SerializationError 2020-07-25 15:52:31 +08:00
Chris Lu
bfabb4159b refactoring 2020-07-11 09:11:15 -07:00
Chris Lu
a4e3cffe0a S3: DeleteObject API also support deleting a folder recursively 2020-06-22 10:01:00 -07:00
Rinat Shigapov
320d70d305 return last modified in UTC 2020-06-19 10:26:40 +03:00
Chris Lu
00f96d86a1 fix typo 2020-06-11 10:53:25 -07:00
Rinat Shigapov
283b749ff1 keep 204 response 2020-06-11 19:00:34 +03:00
Rinat Shigapov
fafc41a27f return xml encoded NotFound status code for s3 delete 2020-06-11 17:53:15 +03:00
Rinat Shigapov
2d2c5dfa39 proxy status code of filer delete response 2020-06-11 15:00:47 +03:00
Chris Lu
a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu
d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu
fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
bc38b72a20 s3: implemented DeleteMultipleObjects 2020-02-25 14:38:36 -08:00
Chris Lu
35dde56711 refactoring 2020-02-25 12:58:45 -08:00
Chris Lu
986d63cd5e s3: increase list objects or parts limit from 1000 to 10000 
10x of Amazon S3 limits
 2020-02-25 11:45:40 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
4ed6b584e2 s3: access control limited by bucket 2020-02-22 21:34:18 -08:00
Chris Lu
48b7ad5fa8 s3: deny anonymous type 2020-02-22 14:01:04 -08:00
Chris Lu
dbabdd418e add unhandled request auth type 
fix

2020-02-18 11:43:57.396699 I | http: panic serving 172.28.0.43:50658: runtime error: invalid memory address or nil pointer dereference
goroutine 595 [running]:
net/http.(*conn).serve.func1(0xc0001fe3c0)
    /usr/lib/go/src/net/http/server.go:1767 +0x13b
panic(0x55c4e35f3820, 0x55c4e48b3c40)
    /usr/lib/go/src/runtime/panic.go:679 +0x1b6
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).authRequest(0xc0004b84e0, 0xc000115900, 0xc0000bb650, 0x1, 0x1, 0x55c4e399d740)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:143 +0x11c
github.com/chrislusf/seaweedfs/weed/s3api.(*IdentityAccessManagement).Auth.func1(0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /go/src/github.com/chrislusf/seaweedfs/weed/s3api/auth_credentials.go:111 +0x5e
net/http.HandlerFunc.ServeHTTP(0xc0004b87e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115900)
    /usr/lib/go/src/net/http/server.go:2007 +0x46
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0004ba000, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /root/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go:212 +0xe4
net/http.serverHandler.ServeHTTP(0xc00011e0e0, 0x55c4e3994c40, 0xc0007808c0, 0xc000115700)
    /usr/lib/go/src/net/http/server.go:2802 +0xa6
net/http.(*conn).serve(0xc0001fe3c0, 0x55c4e399d680, 0xc000894180)
    /usr/lib/go/src/net/http/server.go:1890 +0x877
created by net/http.(*Server).Serve
    /usr/lib/go/src/net/http/server.go:2927 +0x390
 2020-02-18 09:16:04 -08:00
Chris Lu
0a57db7026
Merge pull request #1200 from chrislusf/add_s3_acl 
Add s3 ACL
 2020-02-17 12:53:30 -08:00
Chris Lu
4cdde5f569 configuration stores the identity list 2020-02-17 12:31:59 -08:00
Chris Lu
96c1ae8471 refactoring the close http response 2020-02-14 09:46:36 -08:00
Chris Lu
cf5064d702 properly close http response 2020-02-14 09:09:15 -08:00
Chris Lu
b4abe3c081 unused 2020-02-09 18:02:17 -08:00
Chris Lu
f3ce3166ad add streaming v4 2020-02-09 17:42:17 -08:00
Chris Lu
b90ad6f452 add v2 support 2020-02-09 16:02:05 -08:00
Chris Lu
9ed364f053 support acl 2020-02-09 14:30:02 -08:00
Chris Lu
b2743afaee remove println 2020-01-31 00:21:18 -08:00
Chris Lu
a80ecbfe84 s3: add s3 copy 
fix https://github.com/chrislusf/seaweedfs/issues/1190
 2020-01-31 00:11:12 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
ba73c053c3 change uuid package 
fix https://github.com/chrislusf/seaweedfs/issues/1126
 2019-11-18 19:24:37 -08:00
Lei Liu
2cb348a2c2 s3 api: fix listbucket common_prefixes issue 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-09-29 21:15:40 +08:00
Chris Lu
972e881d48 add more logs 
add more logs
 2019-09-16 09:48:30 -07:00
Chris Lu
20ae559437 add logs 
add logs for https://github.com/chrislusf/seaweedfs/issues/1064
 2019-09-16 08:19:10 -07:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
1e2127e261 s3: adding Location for multipart upload 
fix https://github.com/chrislusf/seaweedfs/issues/1002
 2019-07-08 12:37:45 -07:00
Chris Lu
62843991f2 s3: adjust object key 2019-07-08 12:37:20 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
c789b496d8 use cached grpc client 2019-04-05 20:31:58 -07:00
Chris Lu
20dcb44077 fix tests 2019-04-01 23:59:31 -07:00
Chris Lu
78b9db34d5 weed s3: generate time format acceptable by aws cli s3 
fix https://github.com/chrislusf/seaweedfs/issues/911
 2019-04-01 16:13:29 -07:00
Chris Lu
c5a3ff1c35 weed s3: multipart upload goes to the right bucket 
fix https://github.com/chrislusf/seaweedfs/issues/908
 2019-03-31 11:10:19 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
cb07d15254 add namespace for s3 2019-02-27 00:21:37 -08:00
Chris Lu
70ac2f6ea8 add namespace for ListAllMyBucketsResult 2019-02-26 23:23:04 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
3339325334 fix s3 ListAllMyBucketsResult to work with s3cmd 2019-01-02 11:36:29 -08:00
Chris Lu
43db7ac123 fix spelling 2019-01-02 02:38:32 -08:00
Chris Lu
39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu
a4ceb051a7 use MD5 for ETag to be consistent with Amazon S3 2018-11-08 21:41:02 -08:00
Chris Lu
fdc8a267de remove unused function 2018-09-22 22:11:13 -07:00
Chris Lu
d9b32db8d7 adjust uploads to a hidden folder 2018-09-19 22:03:16 -07:00
Chris Lu
f5471bcebf add bucket name in the redirection 2018-09-19 22:01:41 -07:00
Chris Lu
bea4f6ca14 go fmt 2018-09-12 01:00:57 -07:00
Chris Lu
f628953c33 cleanup tmp upload directories 2018-09-12 01:00:51 -07:00
Chris Lu
f6d8525d1d working S3 multipart uploads 2018-09-12 00:46:12 -07:00
Chris Lu
439e88b5a6 fix format error 2018-09-11 13:01:51 -07:00
Chris Lu
296acc4a0a go fmt 2018-09-09 16:26:11 -07:00
Chris Lu
164091c269 add s3 multipart upload 2018-09-09 16:25:43 -07:00
Chris Lu
dd37f0540f add extended attributes 2018-09-07 13:12:52 -07:00
Chris Lu
d614716ff2 fix compilation error 2018-09-04 00:43:46 -07:00
Chris Lu
ce6a2140a2 preparing to support S3 multipart uploads 2018-09-04 00:42:44 -07:00
Chris Lu
7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu
2ed6d8cca6 refactoring filer rm 2018-09-03 13:16:26 -07:00
Chris Lu
03f852c799 refactoring 2018-09-03 13:03:16 -07:00
Chris Lu
0b0ece9649 add place holders for multiplarts upload 2018-09-03 11:38:10 -07:00
Chris Lu
88f1d32cc4 add s3ChunkedReader 
fix https://github.com/chrislusf/seaweedfs/issues/718
 2018-09-02 14:20:47 -07:00
Chris Lu
77fadab560 go fmt 2018-07-24 01:38:08 -07:00
Chris Lu
6039b25bc9 s3API working with listing objects 2018-07-23 01:55:26 -07:00
Chris Lu
3c7ffa2a2c go fmt 2018-07-23 01:16:17 -07:00
Chris Lu
ebad3a44ab s3api do not proxy directory requests 2018-07-23 01:15:59 -07:00
Chris Lu
c133e72f62 print out unsupported methods 2018-07-22 19:55:53 -07:00
Chris Lu
19c537b687 releasing 0.91 2018-07-22 02:12:02 -07:00
Chris Lu
20853972b2 s3 API add ListObjectsV2Handler 2018-07-22 02:04:07 -07:00
Chris Lu
0655151b8c go fmt 2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu
49375d6031 adjust DELETE response to S3 format 2018-07-21 19:12:44 -07:00
Chris Lu
5068b6ae7d fix compilation 2018-07-21 19:06:09 -07:00
Chris Lu
12825938d7 unused method 2018-07-21 18:50:56 -07:00
Chris Lu
8f240276db s3 add HEAD DELETE 2018-07-21 18:49:47 -07:00
Chris Lu
7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu
8480008a9a add s3 upload, and removing mono and multi part upload analyzer 
removing mono and multi part upload analyzer, which were used just to determine the file name
 2018-07-21 10:39:02 -07:00
Chris Lu
dad733086a fix string printing 2018-07-20 00:58:10 -07:00
Chris Lu
568132c0c9 delete bucket also removes collection data 2018-07-20 00:10:01 -07:00
Chris Lu
feea33d5e0 add HeadBucket 2018-07-19 01:43:27 -07:00
Chris Lu
c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu
702fbdf731 refactoring 2018-07-18 02:39:12 -07:00
Chris Lu
834a25f084 add list all my buckets 2018-07-18 02:37:09 -07:00