mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-30 07:09:01 +08:00
Code Issues Packages Projects Releases Wiki Activity
4,718 Commits 32 Branches 296 Tags 148 MiB
8e78187a97
Commit Graph

421 Commits

Author SHA1 Message Date
Chris Lu
628b27ef3b purge old cache implementation 2020-06-11 01:50:00 -07:00
Chris Lu
4598473251 fix possible EOF problem 2020-06-09 18:04:40 -07:00
Chris Lu
bb3e86fbd5 auto create non-existing cache directory 
fix https://github.com/chrislusf/seaweedfs/issues/1350
 2020-06-08 08:49:57 -07:00
Chris Lu
ac48c899e8 add printouts for synchronizing metadata which can be slow for large folders 2020-05-18 10:31:12 -07:00
Chris Lu
47234760f4 log adjust format 2020-05-02 01:33:49 -07:00
Chris Lu
97a504b0bb fix possible directory deletion error 2020-05-02 01:11:53 -07:00
Chris Lu
871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
Chris Lu
e93588ec78 FUSE mount: atomic local cache updates 2020-04-29 18:20:54 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
wuyuxiang
6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu
2a7957b4ca FUSE: subcribe meta events based on timestamp 2020-04-27 16:52:20 -07:00
Chris Lu
5d0e1d8d74 also writes to local meta cache 
before waiting for subscribed meta events
 2020-04-22 15:40:47 -07:00
Chris Lu
a207285af7 cache metadata on startup 2020-04-21 22:00:34 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
4f02f7121d read from meta cache 
meta cache is not initialized
 2020-04-21 18:50:30 -07:00
Chris Lu
b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu
9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu
2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu
e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu
13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu
eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu
bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu
006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu
6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu
9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu
54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu
e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu
e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu
1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu
45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
36d6595658 remove ctx 2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e remove ctx 2020-02-25 22:37:54 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu
24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu
72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
fca6152c3a fix print format 2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106 mount: fix problems found in issue 1182 
fix https://github.com/chrislusf/seaweedfs/issues/1182

always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
 2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5 mount: able to trim a file 2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466 mount: rename also clear the cache 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161 mount: deletion ignore old vid 2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0 mount: modify file size 2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f adjust logs 2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018 logs 2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33 filer: no entry if not found 2020-01-15 19:08:54 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
2f21beaccd loading xattr value 2019-12-17 21:45:32 -08:00
Chris Lu
c42b023960 mount: invalidate xattr 2019-12-17 20:38:56 -08:00
Chris Lu
9c781da34a mount: fix nil directory problem 2019-12-17 15:08:38 -08:00
Chris Lu
ee6952fddd twaking 2019-12-17 14:03:57 -08:00
Chris Lu
f27a1abf1c mount: fix dir nil 2019-12-17 13:50:32 -08:00
bs106
ce79a31448 Fix missing leading dot in extension 2019-12-17 09:47:12 +00:00
Chris Lu
5961e0d9eb fix logging 2019-12-15 22:24:06 -08:00
Chris Lu
33b4b1868b mount: support extended attributes 2019-12-15 21:07:01 -08:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
33da30d2b1 tweak 2019-12-14 11:04:20 -08:00
Chris Lu
de3eb7c380 update cache if updating the attributes 2019-12-13 11:14:09 -08:00
Chris Lu
357ea4b944 rm will clean the cache first 2019-12-13 11:08:26 -08:00
Chris Lu
deb249736e fix nill 2019-12-13 10:35:23 -08:00
Chris Lu
f59119cd1d mount: rename to dirListCacheLimit to configure cache for long format directory list 2019-12-13 10:22:24 -08:00
Chris Lu
3e0cd122d8 mount: directory listing cache ttl set to 5 minutes 2019-12-13 10:08:27 -08:00
Chris Lu
05c3b795dc filer: configurable directory list cache size 2019-12-13 10:05:43 -08:00
Chris Lu
0fa1269bc7 filer: streaming file listing 2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0 filer: increase directory listing pagination size 2019-12-12 09:11:10 -08:00
Chris Lu
f51a6d8ba5 increase filer pagination limit 2019-12-11 22:21:37 -08:00
Chris Lu
40ff4eea14 fix nil problem 2019-12-10 23:13:14 -08:00
Chris Lu
5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu
a14bd31f5b go fmt 2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9 refactoring 2019-06-22 20:04:56 -07:00
Chris Lu
ff97acae62 more logs 2019-06-21 12:14:40 -07:00
Chris Lu
d0ce5b020b logging 2019-06-21 12:05:00 -07:00
Chris Lu
78994024ff adjust logging 2019-06-21 11:46:12 -07:00
Chris Lu
11be0b5e91 more logging 2019-06-20 23:45:30 -07:00
Chris Lu
4b0f084b5d delete chunks only when file writing is successful 2019-06-20 21:58:35 -07:00
Chris Lu
eebc44cce3 refactoring 2019-06-20 21:57:17 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
630cf1e33e fix 2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c fix 2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c logging 2019-04-24 12:30:36 -07:00
Chris Lu
97406333a5 support atomic renaming for mysql/postgres filer store 2019-03-30 23:08:29 -07:00
Chris Lu
c7e7b6229f detect mime type 2019-03-27 14:25:18 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
d312c55bbe file path supports windows, avoiding back slashes 
fix https://github.com/chrislusf/seaweedfs/issues/868
 2019-03-04 13:00:08 -08:00
Chris Lu
9a4dda3011
Merge pull request #855 from chrislusf/add_jwt 
Add jwt
 2019-02-18 15:05:32 -08:00
Chris Lu
a1c7dc3806 avoid changing attributes for mount directory 2019-02-18 12:14:28 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
712a0e19c8 Merge branch 'master' into add_jwt 2019-02-15 10:01:16 -08:00
Chris Lu
2ec6a679c2 avoid "fchmod failed" during cp 2019-02-15 10:00:27 -08:00
Chris Lu
157c0f7c01 add jwt for writes 2019-02-15 09:59:22 -08:00
Chris Lu
a3b0e39b06 weed mount deletion always goes to weed filer 2019-02-14 07:07:38 -08:00
Chris Lu
e70f740deb set file attributes 
fix https://github.com/chrislusf/seaweedfs/issues/847
 2019-01-24 19:38:04 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu
bd32108a90 disable async file deletion 2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1 incrementally calculate visible intervals 2019-01-05 22:23:44 -08:00
Chris Lu
fe50224ea0 refactoring 2019-01-05 15:21:56 -08:00
Chris Lu
d9f9aca453 refactoring 2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5 increase batch deletion size to 1024 2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832 reduce directory list cache usage 2019-01-03 22:21:39 -08:00
Chris Lu
9539823c35 fix log 2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a async file chunk deletion 2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a get bytes from sync pool 2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a reduce pointer usage 2018-12-31 15:10:14 -08:00
Chris Lu
d6181076bb FUSE remove adjust error 2018-12-30 22:29:27 -08:00
Chris Lu
0388d421d2 caching visible intervals for read 
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
 2018-12-30 00:51:44 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f set attributes include crtime 2018-12-28 22:37:18 -08:00
Chris Lu
79168ab0bb adjust error and log 2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2 reuse buffer by sync pool 2018-12-28 03:27:48 -08:00
Chris Lu
253f190f48 weed mount add symlink support 2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a avoid error log if not found 2018-12-24 17:45:26 -08:00
Chris Lu
d3839fe279 go fmt 2018-12-22 11:11:57 -08:00
Chris Lu
3ac54792e1 paginate when filer deleting and FUSE mount renaming 2018-12-16 23:20:08 -08:00
Chris Lu
42b8f1145a weed mount ls paginate the results 
fix https://github.com/chrislusf/seaweedfs/issues/787
 2018-12-11 07:41:50 -08:00
Chris Lu
39ea8484c1 adjust proto API 2018-12-11 07:17:10 -08:00
Chris Lu
1bfb96f34d optimization for reading whole chunk with gzip encoding 2018-12-07 01:57:55 -08:00
Chris Lu
29f1673d97 refactoring 2018-12-07 01:25:01 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
2e32b44061 add template for fs.FSStatfser 2018-11-22 23:05:22 -08:00
Chris Lu
2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
60421b23f0 reduce weed mount memory usage 2018-11-18 07:49:14 -08:00
Chris Lu
d3b7965c76 caching grpc clients 2018-11-14 23:50:46 -08:00
Chris Lu
d89774cb7f cache grpc client connections to improve performance 2018-11-14 23:31:39 -08:00
Chris Lu
2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu
11b0a9cb76 fix possible error when reusing file handles 2018-11-13 23:53:17 -08:00
Chris Lu
8178fc9973 adjust logging 2018-11-11 00:07:46 -08:00
Chris Lu
cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu
ed8efb5aef much improved "ls -al" performance 2018-11-07 11:35:13 -08:00
Chris Lu
ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu
9db53cf256 use the configurable chunk size 2018-10-14 00:18:52 -07:00
Chris Lu
3ddcd87098 increase weed mount IO perforance 2018-10-14 00:10:17 -07:00
Chris Lu
08266b7256 go fmt 2018-10-11 00:08:13 -07:00