mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-27 04:39:25 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,813 Commits 32 Branches 296 Tags 148 MiB
b5e2be635a
Commit Graph

3014 Commits

Author SHA1 Message Date
Chris Lu
f930c713fc more efficient recursion 2020-12-12 13:25:19 -08:00
Chris Lu
316d1b4e69 refactor APIs 2020-12-12 12:42:53 -08:00
Chris Lu
03637d6f57 s3: move "delete-directory-if-empty" to read time 
move "delete-directory-if-empty" to read time instead of entry deletion time

the listing speed for a s3 bucket folder will slow down if it has many sub folders

related to 0d345ac97d

fix https://github.com/chrislusf/seaweedfs/issues/1647

fix https://github.com/chrislusf/seaweedfs/issues/1670
 2020-12-12 03:38:34 -08:00
Chris Lu
37075a414d adjust logs 2020-12-12 03:33:57 -08:00
Chris Lu
ae655033ac adjust logging 2020-12-11 16:57:53 -08:00
Chris Lu
d986c7196d use append time instead of filer's own modification time 
fix https://github.com/chrislusf/seaweedfs/issues/1669
 2020-12-11 16:55:18 -08:00
Chris Lu
3fedfec1e7 check cross device rename error 2020-12-10 23:50:32 -08:00
Chris Lu
c2f18a10cb minor 2020-12-10 22:23:22 -08:00
Chris Lu
4b0c2a846b add logs 2020-12-10 19:55:28 -08:00
Chris Lu
83078ac6ce filer: change to /etc/seaweedfs folder on filer 
fix https://github.com/chrislusf/seaweedfs/issues/1666
 2020-12-10 11:11:02 -08:00
Chris Lu
26731694f8 s3: use static configuration by default 
So that users can still use the previous configuration files.

If leave it empty, s3 will try to use the version from filer
 2020-12-10 00:59:04 -08:00
Chris Lu
765b3ef1be save /etc/iam/identity.json inside filer store 2020-12-10 00:15:22 -08:00
Chris Lu
97c942b5ee paginate through large directories 2020-12-09 23:23:38 -08:00
Chris Lu
02a160c3fd s3: recursively iterate all sub folders 
fix https://github.com/chrislusf/seaweedfs/issues/1656
 2020-12-09 16:47:34 -08:00
Chris Lu
a9c619c270 added error handling for saving 2020-12-09 01:34:24 -08:00
Chris Lu
4211601eab set file handle reader to nil 2020-12-08 22:48:18 -08:00
Chris Lu
b52ae9cef8 add s3 debug 2020-12-08 22:27:04 -08:00
Chris Lu
8e78187a97 add back last read chunk cache to reader and properly close the reader 2020-12-08 22:26:46 -08:00
Chris Lu
900d22c6ec mount: avoid memory leaking read buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1654

the reader goes together with the file handle, which may stay for a long time.
 2020-12-08 02:38:53 -08:00
Chris Lu
06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path 
related to https://reviews.freebsd.org/D27391
 2020-12-07 16:46:48 -08:00
Chris Lu
fce8803087 break import cycle 2020-12-07 00:29:17 -08:00
Chris Lu
606051cc79 filer: cache "/etc" filder small files to filer store 2020-12-07 00:11:35 -08:00
Chris Lu
ac22f1dd26 filer subscribe: handle rename subscription 2020-12-07 00:10:49 -08:00
Chris Lu
eed87791b7 s3: subscribe to s3.configure changes 2020-12-07 00:10:29 -08:00
Chris Lu
01e2da5782 refactoring 2020-12-06 23:16:20 -08:00
Chris Lu
77286f8bea fix test 2020-12-06 21:56:13 -08:00
Chris Lu
57578a6cd1 change parameter help message 2020-12-06 21:54:55 -08:00
Chris Lu
78f3ab439c refactoring 2020-12-06 21:54:34 -08:00
Chris Lu
9acda432fe fix import cycle 2020-12-06 20:12:52 -08:00
Chris Lu
ae5eb85a06 refactoring 2020-12-06 20:05:06 -08:00
Chris Lu
5931a2f53f filer conf: support json conf only 2020-12-06 19:47:06 -08:00
Chris Lu
4c72482536 shell: fix fs.configure 2020-12-06 19:43:31 -08:00
Chris Lu
46b91228af 2.14 2020-12-06 18:56:58 -08:00
Chris Lu
263eb29e9f filer: add option to cache small files to filer store 2020-12-04 22:39:43 -08:00
Chris Lu
d171d9f988 volume: ensure the volume dat and idx files are always closed 
fix https://github.com/chrislusf/seaweedfs/issues/1646
 2020-12-04 21:50:26 -08:00
Konstantin Lebedev
eadbba5c95 security master use access.ui 2020-12-04 12:57:59 +05:00
Chris Lu
cc839f935d better unit detection 2020-12-03 19:23:59 -08:00
Chris Lu
b219ccfe68 Update volume_ttl_test.go 2020-12-03 12:55:44 -08:00
Chris Lu
a83b8e7b01 ttl fix: 3y was converted into 96m 2020-12-03 10:53:29 -08:00
Chris Lu
5fb60b7135
Merge pull request #1596 from kmlebedev/store_s3cred 
S3 credentials store in filer
 2020-12-03 00:40:20 -08:00
Konstantin Lebedev
14699dfcef use content field of entry 2020-12-02 17:19:05 +05:00
Konstantin Lebedev
a3d4b50a49 use entry content filed 2020-12-02 16:12:13 +05:00
Chris Lu
003b6245e7 fix nil 2020-12-02 00:09:19 -08:00
Chris Lu
f3bb645018 file open error 2020-12-01 23:37:49 -08:00
Chris Lu
dc0bc48257 return file open error 2020-12-01 23:36:49 -08:00
Chris Lu
45dba088cd fix tests 2020-12-01 19:37:21 -08:00
Chris Lu
04062c56c7 webdav: improve webdav upload speed 2020-12-01 15:32:27 -08:00
Konstantin Lebedev
03620776ec Merge branch 'upstream_master' into store_s3cred 2020-12-01 16:03:34 +05:00
Chris Lu
005a6123e9
Merge pull request #1643 from hilimd/master 
fix tls grpc ca path
 2020-11-30 17:16:46 -08:00
Chris Lu
f4abd01adf filer: cache small file to filer store 2020-11-30 04:34:04 -08:00
Chris Lu
a9c6be5fc3 set creation time 2020-11-30 03:11:52 -08:00
Chris Lu
141ce67c09 close http request body 2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf 2.13 2020-11-29 17:00:03 -08:00
Chris Lu
965413c21b shell: add volume.vacuum command 2020-11-28 23:18:02 -08:00
Chris Lu
96c48bc8a8 fix test 2020-11-28 03:03:43 -08:00
Chris Lu
c57a7839ec adjust comments 2020-11-28 03:01:03 -08:00
Chris Lu
8438fb0cc3 fix ec shard spreading error 2020-11-28 02:21:16 -08:00
Chris Lu
536202e4e0 minor 2020-11-28 02:16:06 -08:00
Chris Lu
99c4e50d3d minor 2020-11-28 00:14:11 -08:00
Chris Lu
2c913dde04 volume: detect and drop volumes with disk IO error 
from Jethro in slack:
is it possible to make the assign request a bit smarter? Currently I’m in the state that a disk failed but all assign request are being send to this volume. It would be cool if the master sees this and stopped using this volume.
e=HTTP(http://x:8089/913,045a782b63176edf) not 200 but 500 Internal Server Error
Body={"size":740167,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"ee4381e202212ff3aee647704c036689"}
e=HTTP(http://x:8089/913,045a782c90240077) not 200 but 500 Internal Server Error
Body={"size":792779,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"c43463ccc11eb6eb2fc306f407a6a953"}
e=HTTP(http://x:8089/913,045a782e6b7901ea) not 200 but 500 Internal Server Error
Body={"size":3962392,"error":"failed to write to local disk: write /mnt/v9/913.dat: input/output error","eTag":"04c91198e9b276c81f11dbf189af5d28"}
 2020-11-28 00:09:29 -08:00
Chris Lu
9ac4935f22 read from volume index file directly instead of open a separate file 
fix https://github.com/chrislusf/seaweedfs/issues/1640

read from volume index file directly instead of open a separate file,
to ensure reading latest index entries.
 2020-11-27 16:18:48 -08:00
Chris Lu
85554bea38 filer: readonly handle static resources 
fix https://github.com/chrislusf/seaweedfs/issues/1641
 2020-11-27 13:42:14 -08:00
Chris Lu
6d30b21b10 volume: add "-dir.idx" option for separate index storage 
fix https://github.com/chrislusf/seaweedfs/issues/1265
 2020-11-27 03:17:10 -08:00
Chris Lu
3c229eb677 refactor 2020-11-26 17:21:55 -08:00
Chris Lu
c0fb4a3766 add comments 2020-11-26 15:22:42 -08:00
Chris Lu
6783ae5f8f rename 2020-11-26 15:19:43 -08:00
Chris Lu
983fd90010 marked as WIP 2020-11-26 15:08:52 -08:00
Chris Lu
97f706f35b just refactoring 2020-11-26 15:08:05 -08:00
Chris Lu
b11449f955 pass in dir.idx parameter to DiskLocation 2020-11-26 14:59:03 -08:00
Chris Lu
0d345ac97d s3: remove empty parent folder on delete 
fix https://github.com/chrislusf/seaweedfs/issues/1637
 2020-11-26 12:21:58 -08:00
Chris Lu
cc2bd97ad9 refactor 2020-11-26 11:25:56 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
effa00ed08 refactor 2020-11-26 11:21:58 -08:00
Chris Lu
513bcd6e0d filer: avoid duplicated FindEntry for deletion 2020-11-26 11:14:56 -08:00
Chris Lu
1ae108efca add debug message 2020-11-26 10:41:10 -08:00
Chris Lu
a102157020 do not retry if failed with a non transport related error 2020-11-26 01:29:30 -08:00
Konstantin Lebedev
4e55baf5b1 s3 config changes 2020-11-26 03:50:53 +05:00
Konstantin Lebedev
52c8f2fc9a s3iam test 2020-11-26 02:26:45 +05:00
Konstantin Lebedev
a26f1b2040 new pkg s3iam 2020-11-26 01:30:11 +05:00
Konstantin Lebedev
6206737df2 s3 configure 2020-11-25 21:02:31 +05:00
Chris Lu
eab53ea80d filer leveldb store: a bit more efficient directory listing with prefix 2020-11-22 21:10:41 -08:00
Chris Lu
d3be2ceedd 2.12 2020-11-22 17:17:33 -08:00
Chris Lu
c7ebadc25d avoid possible concurrent access inside ensureCorrectWritables() 2020-11-22 17:15:59 -08:00
hilimd
9aa990f80f
Merge pull request #39 from chrislusf/master 
sync
 2020-11-22 20:29:17 +08:00
limd
e8296104fc fix tls grpc ca path 2020-11-22 20:27:15 +08:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Aleksandr Demshin
176950ea78 fix typo 2020-11-20 19:50:46 +05:00
Chris Lu
7385103057 fix volume placement validating 
fix https://github.com/chrislusf/seaweedfs/issues/1626
 2020-11-20 01:12:25 -08:00
Konstantin Lebedev
27e73de797 Merge branch 'upstream_master' into store_s3cred 
# Conflicts:
#	weed/s3api/filer_util.go
 2020-11-19 18:16:44 +05:00
Chris Lu
290b5e2cd0 directly delete file chunks 
keeping current async deletions for now
 2020-11-17 17:20:21 -08:00
Chris Lu
781585b195 remove unnecessary checking 2020-11-17 17:00:06 -08:00
Chris Lu
8cb8cd4cc5 add locks 2020-11-17 16:59:48 -08:00
Chris Lu
7c9e592c2d s3: avoid nil resp when having error 
fix https://github.com/chrislusf/seaweedfs/issues/1622
 2020-11-17 11:23:17 -08:00
Chris Lu
dc304342b2 fs.configure: configurable volume growth 2020-11-17 01:00:02 -08:00
Chris Lu
b04375a3d1 use alternative assign request 2020-11-17 00:36:21 -08:00
Chris Lu
9add554feb fail fast if user configures collection for a bucket 2020-11-16 19:57:08 -08:00
Chris Lu
6204dc6e85 add comment 2020-11-16 16:57:31 -08:00
Chris Lu
ef3e068c51 deprecating "filer.options.buckets_fsync" in filer.toml 2020-11-16 16:56:49 -08:00
Chris Lu
5f19e81dab filer confi: support hierachical configuration 2020-11-16 16:50:12 -08:00
Chris Lu
ed1ce3f299 adjust help message 2020-11-16 09:54:26 -08:00