mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-11 15:59:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,087 Commits 32 Branches 296 Tags 150 MiB
320637dc7a
Commit Graph

333 Commits

Author SHA1 Message Date
Konstantin Lebedev
c5705e7a35 dir + slash key does not exist 
https://github.com/chrislusf/seaweedfs/issues/1917
 2021-03-17 23:40:42 +05:00
Konstantin Lebedev
eb54993a4e Merge branch 'upstreamMaster' into check_chunkviews_mr 
# Conflicts:
#	weed/filer/filechunk_manifest.go
#	weed/filer/stream.go
#	weed/replication/repl_util/replication_util.go
#	weed/util/fasthttp_util.go
 2021-03-16 15:29:49 +05:00
wuh-fnst
050a7f9599 s3: count correction for the number of files in the dir 2021-03-16 09:22:03 +08:00
Konstantin Lebedev
3a3699867b Status PartialContent for Content-Range response 2021-03-15 23:30:22 +05:00
Chris Lu
d084334ffd
Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder 
s3 delete-objects pass OK for fail to delete non-empty folder
 2021-03-11 11:01:44 -08:00
Chris Lu
79280e1513 escape special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-11 09:49:40 -08:00
wuheng
828fbf3fb7 s3: "isLast" returns true when the file does not exist 2021-03-11 15:20:50 +08:00
Chris Lu
ca546f47e1 s3: escape object key if containing special characters 
fix https://github.com/chrislusf/seaweedfs/issues/1884
 2021-03-10 13:19:28 -08:00
Konstantin Lebedev
bb3af2d70b no directoriesWithDeletion 2021-03-10 19:52:41 +05:00
Konstantin Lebedev
15b018da34 s3 delete-objects pass OK for fail to delete non-empty folder 2021-03-10 19:41:35 +05:00
Chris Lu
387c6f4218 Revert "delete the folder object in multi_object_delete" 
This reverts commit 2e89c8c9ae.
 2021-03-09 10:07:27 -08:00
Chris Lu
2e89c8c9ae delete the folder object in multi_object_delete 
fix https://github.com/chrislusf/seaweedfs/issues/1857
 2021-03-04 14:03:40 -08:00
Chris Lu
5511722420 s3: list permission is needed to list my buckets 
fix https://github.com/chrislusf/seaweedfs/issues/1837
 2021-03-01 12:41:55 -08:00
bingoohuang
352ac2f271 Merge remote-tracking branch 'origin/master' 2021-02-18 14:05:51 +08:00
bingoohuang
eab6e31d34 use backticks instead of double quotes to avoid escaped additionally in regex 2021-02-18 14:05:28 +08:00
Chris Lu
8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu
609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu
c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu
e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu
389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu
8a7302ce37 s3: KeyCount is zero 
fix https://github.com/chrislusf/seaweedfs/issues/1741
 2021-01-11 03:50:45 -08:00
Chris Lu
e327385644 go fmt 2021-01-06 04:22:00 -08:00
Chris Lu
6d2c979999 s3: fix regression 
fix https://github.com/chrislusf/seaweedfs/issues/1707
 2020-12-27 21:09:45 -08:00
Chris Lu
561a2ff0bc s3: add option for "alllowEmptyFolder" 2020-12-27 10:50:27 -08:00
Chris Lu
aa020ee7e7 s3: restore V2 auth type 
The added headers were included in calculating the signatures, failing the comparison
 2020-12-26 22:01:16 -08:00
Chris Lu
090f85be4b s3: support config action Admin:bucket 2020-12-25 00:38:56 -08:00
Chris Lu
8e48a235e2 s3: avoid looping if the directory is empty 
fix https://github.com/chrislusf/seaweedfs/issues/1701
 2020-12-24 11:34:52 -08:00
Chris Lu
1620de08ae added more logs 2020-12-24 01:52:06 -08:00
Chris Lu
a09cd28986 add debug messages 2020-12-24 01:42:15 -08:00
Chris Lu
3be3635799 just log error when checking empty dir failed 2020-12-23 14:36:48 -08:00
Chris Lu
0ca9d89589 s3: break loop if error 
fix #1701
 2020-12-23 14:34:59 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
5c465293e9 correctly determine whether a folder is empty 
avoid edge cases that deleting the folder if previous 32 directories are all empty

early terminate if one file is found
 2020-12-12 16:19:29 -08:00
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Konstantin Lebedev
27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu
781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu
7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu
0ea5c087ce go fmt 2020-11-15 16:59:28 -08:00
ruitao.liu
a9990a1dc6 adjust check bucket if exist or has access. 2020-11-13 17:13:20 +08:00
Chris Lu
e6333da65a enable admin to access all buckets 2020-11-12 13:57:54 -08:00
Chris Lu
7094492428 fix ErrNotFound 2020-11-12 13:30:46 -08:00
Chris Lu
c6a0704019 rename function 2020-11-12 13:30:08 -08:00
Chris Lu
ed3b0f81c6
Merge pull request #1613 from taozix/master 
check permission for bucket delete/head.
 2020-11-12 13:25:28 -08:00
ruitao.liu
c4f0fd6e1b skip if entry.Extended map is nil. 2020-11-12 17:59:31 +08:00
ruitao.liu
ab966410d2 return NoSuchBucket instead of InternalError delete non-existed bucket. 2020-11-12 16:44:16 +08:00
ruitao.liu
e06676f007 check permission for bucket delete/head. 2020-11-12 16:15:59 +08:00
Chris Lu
0fbc27b640
Merge pull request #1612 from taozix/master 
add owner attr for bucket.
 2020-11-11 22:19:47 -08:00
ruitao.liu
d7cc0498e0 check if bucket already exists. 2020-11-12 14:11:03 +08:00
ruitao.liu
dbba8cb57e fix if nil map in entry. 2020-11-12 11:50:19 +08:00
Chris Lu
86cdb2a3e5 adjust logs 2020-11-11 02:01:24 -08:00
ruitao.liu
5b636b3242 Add bucket owner attr. 2020-11-11 16:20:59 +08:00
Konstantin Lebedev
6cd6ff6962 avoid blank response for 404 2020-11-04 22:54:47 +05:00
Konstantin Lebedev
e1190b3224 load S3 config from filer 
https://github.com/chrislusf/seaweedfs/issues/1500
 2020-11-03 21:45:56 +05:00
ruitao.liu
22a9ea0512 adjust s3 header file. 2020-10-29 16:05:40 +08:00
ruitao.liu
b917be7955 S3 bucket list, response with uploaded storageclass. 2020-10-29 14:57:19 +08:00
Chris Lu
6da87720eb shorter tag prefix 2020-10-27 15:33:26 -07:00
ruitao.liu
8766ca1b95 rename s3 tag prefix. 2020-10-27 17:33:24 +08:00
ruitao.liu
19026ae55d return x-amz-tag-count header when GET object. 2020-10-27 16:49:31 +08:00
Chris Lu
da36abf033 go fmt 2020-10-24 20:12:04 -07:00
Konstantin Lebedev
05b5f12f2e multiplate DomainNames through comma 2020-10-21 20:48:51 +05:00
Chris Lu
b3aa2fab9a s3: report error to s3 when updating an object but it is already a directory 
fix https://github.com/chrislusf/seaweedfs/issues/1545
 2020-10-20 10:25:16 -07:00
Chris Lu
ace0ea3d28 s3: avoid duplicated bucket 2020-10-15 10:52:20 -07:00
Chris Lu
723ae11db4 refactoring in order to adjust volume server url later 2020-10-11 20:15:10 -07:00
Chris Lu
9e7a2772b1 s3: only admin can list all buckets 2020-10-08 10:12:09 -07:00
Chris Lu
62d3d3aea0 go fmt 2020-10-07 23:30:54 -07:00
Chris Lu
e91b9c85a2 s3: Added support for "List" action in weed s3 -config=... in the config file. 
fix https://github.com/chrislusf/seaweedfs/issues/1511
 2020-10-07 23:22:35 -07:00
Chris Lu
f781cce500 s3: support object tagging 
* GetObjectTagging
* PutObjectTagging
* DeleteObjectTagging
 2020-10-02 22:21:51 -07:00
Konstantin Lebedev
68463e92c1 add status code in S3RequestCounter 2020-10-01 00:59:39 +05:00
limd
402aef8f30 s3: 
1.fix spark reading S3 directory wildcard problem
2.fix the problem of the spark history service writing S3 directory

reference git revsion number: b41b7ea4d0
 2020-09-25 14:37:02 +08:00
Chris Lu
043b063136
Merge pull request #1482 from hilimd/master 
Fix: s3 delete object
 2020-09-24 18:21:34 -07:00
Chris Lu
dbf5327b98 s3: handle response-content-encoding 
fix https://github.com/chrislusf/seaweedfs/issues/1487
 2020-09-24 18:09:52 -07:00
limd
d506080c36 rollback 2020-09-23 14:29:53 +08:00
limd
8f9f29b773 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:33:13 +08:00
limd
1892677b22 fixed the problem of empty directory when S3 deleted the directory 2020-09-23 13:15:06 +08:00
limd
a99f63cb4d 1.add S3 copy directory function 
2.fixed the problem of empty directory when S3 deleted the directory
 2020-09-23 11:26:01 +08:00
Chris Lu
9cdbfc1a49 refactor 2020-09-21 17:34:38 -07:00
Chris Lu
b9c1f3e9de s3: fixes for list multipart upload 2020-09-21 10:51:24 -07:00
Chris Lu
9a3b564508 adjust printout 2020-09-20 16:01:56 -07:00
Chris Lu
62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu
d013d09a9b adjust logging 2020-09-20 15:38:59 -07:00
Chris Lu
dc4d2145ff open up 2020-09-19 20:23:12 -07:00
Chris Lu
29abe980df s3: add support for PostPolicy 
fix https://github.com/chrislusf/seaweedfs/issues/1426
 2020-09-19 20:14:19 -07:00
Chris Lu
5b40a2690a refactoring 2020-09-19 14:09:58 -07:00
Chris Lu
2cbd1cf121 fix compilation 2020-09-18 00:15:54 -07:00
Chris Lu
23e9ede068 s3: collect metrics 2020-09-18 00:09:04 -07:00
Chris Lu
3984c3962f add comment 2020-09-11 15:07:19 -07:00
Chris Lu
baa6bdf4d4 s3: listMultipartUploads fix output format 2020-09-11 15:04:01 -07:00
Chris Lu
3eda8d6dfc s3: ListParts output xml format 
fix https://github.com/chrislusf/seaweedfs/issues/1461
 2020-09-11 14:53:50 -07:00
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
83012f543a s: avoid possible completeMultipartUpload re-tries that can overwrite the uploaded file 2020-08-26 08:50:24 -07:00
Chris Lu
81e5124faf s3: list bucket permission change from admin to read 
fix https://github.com/chrislusf/seaweedfs/issues/1430
 2020-08-24 11:22:45 -07:00
Chris Lu
c45ba5d7d4 fix listObjectsV2 response format 
fix https://github.com/chrislusf/seaweedfs/issues/1426 issue 1
 2020-08-19 12:08:04 -07:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
f86c7d911a remove println 2020-08-09 14:42:25 -07:00
Chris Lu
9ecc1170a3 istObjects和listObjectsV2不能查询子目录 #1418 
fix https://github.com/chrislusf/seaweedfs/issues/1418
 2020-08-09 14:35:53 -07:00
Chris Lu
3f4aff5dde s3: fix delimiter in list response 2020-08-09 09:09:35 -07:00
Chris Lu
828a5ae429 check signature only when auth is enabled 2020-08-08 09:11:40 -07:00
Chris Lu
2b74abf766 S3: configurable access for anonymous user 
fix https://github.com/chrislusf/seaweedfs/issues/1413
 2020-08-06 03:41:34 -07:00
limd
b41b7ea4d0 fix spark read s3 bug (sc.binaryFiles) 2020-08-01 01:08:30 +08:00
Chris Lu
a566bfc6e1 s3: use bucket in the domain 
fix https://github.com/chrislusf/seaweedfs/issues/1405
 2020-07-28 08:47:27 -07:00
limd
92c32f9d46 PutObject: fix create folder bug 2020-07-28 00:58:42 +08:00
Chris Lu
54cd582653 refactor 2020-07-26 12:58:58 -07:00
Chris Lu
c50ec1599c auth for presigned put object part 2020-07-26 12:09:50 -07:00
Chris Lu
ea0180de4e auth for presigned put operation 2020-07-26 12:06:45 -07:00
popstk
a09fcfcf20 fix s3api auth bug 2020-07-25 17:14:49 +08:00
popstk
feb52400be fix s3api copy object handler SerializationError 2020-07-25 15:52:31 +08:00
Chris Lu
bfabb4159b refactoring 2020-07-11 09:11:15 -07:00
Chris Lu
a4e3cffe0a S3: DeleteObject API also support deleting a folder recursively 2020-06-22 10:01:00 -07:00
Rinat Shigapov
320d70d305 return last modified in UTC 2020-06-19 10:26:40 +03:00
Chris Lu
00f96d86a1 fix typo 2020-06-11 10:53:25 -07:00
Rinat Shigapov
283b749ff1 keep 204 response 2020-06-11 19:00:34 +03:00
Rinat Shigapov
fafc41a27f return xml encoded NotFound status code for s3 delete 2020-06-11 17:53:15 +03:00
Rinat Shigapov
2d2c5dfa39 proxy status code of filer delete response 2020-06-11 15:00:47 +03:00
Chris Lu
a25a8d8822 s3: print out time in UTC format 
fix https://github.com/chrislusf/seaweedfs/issues/1297
 2020-04-29 21:35:24 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
Chris Lu
d57559d3f7 chunk rememober encryption or not 2020-04-29 02:41:47 -07:00
Chris Lu
ec2eb8bc48 add If-None-Match and If-Modified-Since 
fix https://github.com/chrislusf/seaweedfs/issues/1269
 2020-04-08 08:12:00 -07:00
Chris Lu
86398d1efe S3 API: set deletion to recursive 
fix https://github.com/chrislusf/seaweedfs/issues/1251
 2020-03-26 11:05:54 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
c07bcd5065 refactoring 2020-03-23 00:30:02 -07:00
Chris Lu
654a69ff52 refactoring 2020-03-23 00:06:24 -07:00
Chris Lu
fbca6b29bd refactoring 2020-03-22 23:52:55 -07:00
Chris Lu
c4bea45099 S3 API: fix DeleteMultipleObjectsHandler 
fix https://github.com/chrislusf/seaweedfs/issues/1241
 2020-03-20 14:17:31 -07:00
Chris Lu
9b3109a5d8 filer: processing all response headers, no pass through to volume server 
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
 2020-03-08 15:42:44 -07:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
ed0acd1722 go fmt 2020-02-26 16:52:57 -08:00