mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 00:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
5,657 Commits 32 Branches 296 Tags 152 MiB
aabea37af1
Commit Graph

504 Commits

Author SHA1 Message Date
Chris Lu
daf0a449f7 properly cancel context for streaming grpc 2020-09-09 12:07:15 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
1d56ea24ef fix 2020-09-03 00:08:37 -07:00
Chris Lu
7e1aad0b54 mount: map uid/gid between local and filer 2020-09-03 00:07:22 -07:00
Chris Lu
eb7929a971 rename filer2 to filer 2020-09-01 00:21:19 -07:00
Chris Lu
408e339c53 also delete the manifest chunk itself 2020-08-31 00:16:03 -07:00
Chris Lu
33d8c6c617 change log to warning 2020-08-30 21:01:44 -07:00
Chris Lu
e62d7f221f a bit more logs 2020-08-30 21:00:24 -07:00
Chris Lu
22fe4ae573 mount: fix for UrBackup 2020-08-30 21:00:09 -07:00
Chris Lu
ae84a9ee2e add logs 2020-08-30 20:59:04 -07:00
Chris Lu
a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu
f2a8574448 filer and mount deletion resolves manifest chunks also 2020-08-30 02:07:14 -07:00
Chris Lu
99ecf63276 go fmt 2020-08-29 22:28:33 -07:00
Chris Lu
063c9ddac5 adjust logs 2020-08-29 11:56:22 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
51346a5930 always keep the manifest list of chunks 2020-08-23 17:15:12 -07:00
Chris Lu
c1d1677a28 keep manifest chunks forever 2020-08-23 16:59:01 -07:00
Chris Lu
98175548c2 adjust deletion ordering 2020-08-23 16:27:12 -07:00
Chris Lu
aee27ccbe1 multiple fixes 
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
 2020-08-23 15:48:02 -07:00
Chris Lu
77393d3d30 add file handle locking when changing file entry 2020-08-23 00:35:50 -07:00
Chris Lu
d60bcbf08a sorting chunks 2020-08-23 00:00:36 -07:00
Chris Lu
5e6b714836 add random test 2020-08-22 16:33:00 -07:00
Chris Lu
f48567c5c6 remove unused function 2020-08-19 22:53:49 -07:00
Chris Lu
839634097f also do flush on release 2020-08-19 01:27:10 -07:00
Chris Lu
a78772d5ea avoid shutdown in the middle of running 2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508 logs 2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419 adjust logs 2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5 delete chunks in the last step 2020-08-18 23:40:53 -07:00
Chris Lu
618b2f6829 release resources only when needed to 2020-08-18 12:53:08 -07:00
Chris Lu
208849702d logs 2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3 printout data size 2020-08-18 09:09:29 -07:00
Chris Lu
be4d42b8e2 rename 2020-08-17 20:15:53 -07:00
Chris Lu
abdaf9958d possibly read more 2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78 rename 2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d prevent nil 2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4 minor optimization 2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000 report error first 2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc fix possible out of range bytes 
avoid buff out of range resp.Data = buff[:totalRead]
 2020-08-16 23:47:34 -07:00
Chris Lu
22e5132b3a adjust log level 2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be reduce memory allocation 2020-08-16 16:24:40 -07:00
Chris Lu
4a77f0820a clean up logs 2020-08-16 01:37:50 -07:00
Chris Lu
aec7f32b02 fix reader_at 2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7 adjust logs 2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a update metadata only if changed 2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2 adjust log level 2020-08-15 18:24:35 -07:00
Chris Lu
a22ee30596 fix nil 2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816 ensure meta data changes are updated 2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20 proper deletion ordering 
delete central file store first, then delete local cache
 2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1 file size support set file length 
use Attr.FileSize and TotalChunkSize to determine file size
 2020-08-15 09:32:47 -07:00
Chris Lu
a7f669044e rename also applies to open file handle 2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f adjust log level 2020-08-14 00:22:21 -07:00
Chris Lu
090612492c fix compilation 2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f add back handles lock 2020-08-12 23:52:13 -07:00
Chris Lu
e74dc4e4bc add back fs node cache for renaming 2020-08-09 21:56:09 -07:00
Chris Lu
9832653e1d FUSE mount: proper error with deleting non empty folder 2020-08-08 21:37:36 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
eed525b717 FUSE mount: remove DirListCacheLimit 
outdated parameter
 2020-07-28 09:24:39 -07:00
Chris Lu
d40de39e75 FUSE: do not change crtime, uid, gid on save 2020-07-24 10:13:45 -07:00
Chris Lu
8dfeba8023 FUSE: skip changing to empty uid and gid during flush 
Skip uid and gid changes if request uid and gid are zero

mongodb lock file, and many interim files, has empty uid and gid

I0724 10:05:51 93643 filehandle.go:173] /db/diagnostic.data/metrics.interim fh 1333342842031408359 flush Flush [ID=0x3 Node=0x50 Uid=0 Gid=0 Pid=178] 0x10 fl=0x0 lk=0x0
 2020-07-24 10:06:43 -07:00
Chris Lu
dd29b8c81d fix compilation 2020-07-24 09:36:05 -07:00
Chris Lu
5788f1f6bf remove fsnode cache 
fix https://github.com/chrislusf/seaweedfs/issues/1400
 2020-07-24 09:25:54 -07:00
Chris Lu
6962ca48d3 fix compilation 2020-07-23 21:31:23 -07:00
Chris Lu
df7eee523b FUSE: implement dir fsync 
fix https://github.com/chrislusf/seaweedfs/issues/1398
 2020-07-23 21:09:40 -07:00
Chris Lu
3bbac0820b FUSE: faster rename implementation 2020-07-23 21:08:42 -07:00
Chris Lu
60d14a9800 mount: fix difference with storage format in local cache 2020-07-20 00:17:50 -07:00
Chris Lu
97d97f3528 go code can read and write chunk manifest 2020-07-19 17:59:43 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
2d1df97933 mount: fatal error if can not process meta data 2020-07-11 06:16:48 -07:00
Chris Lu
e3e00d36c5 reset cache for each new release 2020-07-11 06:16:17 -07:00
Chris Lu
7ad0bce2f5 remove old code 2020-07-10 23:03:22 -07:00
Chris Lu
54d6b3c30f filer: support running multiple mount to different filers avoiding duplicated cache 
fix https://github.com/chrislusf/seaweedfs/issues/1382
 2020-07-07 23:18:44 -07:00
Chris Lu
ce3630e7a4 refactoring 2020-07-03 22:16:59 -07:00
Chris Lu
31e23e9783 filer: support active<=>active filer replication 2020-06-30 22:53:57 -07:00
Chris Lu
42a338d7b0 FUSE mount: clean up file handles during renaming 
resolve dir rename when file is still open. Need to clean the file handles as soon as possible.

These can happen out of order:
file rename, then file release
file release, then file rename
 2020-06-28 13:41:00 -07:00
Chris Lu
95a1860d65 remove empty calls for list cache 2020-06-28 10:25:54 -07:00
Chris Lu
6498d0c86b remove list cache 2020-06-28 10:21:52 -07:00
Chris Lu
9033a7d369 removed async option 2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d reverting and working 2020-06-28 10:14:17 -07:00
Chris Lu
3cec4b3c49 FUSE mount: fix file id written twice? 
fix https://github.com/chrislusf/seaweedfs/issues/1373
 2020-06-25 20:11:49 -07:00
Chris Lu
e8e0d629af ensure correct file mode 2020-06-24 23:26:54 -07:00
Chris Lu
f7a45d448f FUSE mount: lazy loading meta cache 2020-06-19 09:45:42 -07:00
Chris Lu
b74eced7a3 mount: a fix to prevent possible repeated calls 
related to https://github.com/bazil/fuse/issues/130
 2020-06-17 13:55:16 -07:00
Chris Lu
628b27ef3b purge old cache implementation 2020-06-11 01:50:00 -07:00
Chris Lu
4598473251 fix possible EOF problem 2020-06-09 18:04:40 -07:00
Chris Lu
bb3e86fbd5 auto create non-existing cache directory 
fix https://github.com/chrislusf/seaweedfs/issues/1350
 2020-06-08 08:49:57 -07:00
Chris Lu
ac48c899e8 add printouts for synchronizing metadata which can be slow for large folders 2020-05-18 10:31:12 -07:00
Chris Lu
47234760f4 log adjust format 2020-05-02 01:33:49 -07:00
Chris Lu
97a504b0bb fix possible directory deletion error 2020-05-02 01:11:53 -07:00
Chris Lu
871efa4fc1 refactoring 
some previous chunk etag was using md5, which should be wrong.
 2020-04-30 17:20:47 -07:00
Chris Lu
e93588ec78 FUSE mount: atomic local cache updates 2020-04-29 18:20:54 -07:00
Chris Lu
9e72e9e4b8 able to subscribe any topic from any point of time 2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5 refactoring 2020-04-29 13:26:02 -07:00
wuyuxiang
6850d28d6b refacotr: move signal handling and pprof to grace package 2020-04-28 14:10:23 +08:00
Chris Lu
2a7957b4ca FUSE: subcribe meta events based on timestamp 2020-04-27 16:52:20 -07:00
Chris Lu
5d0e1d8d74 also writes to local meta cache 
before waiting for subscribed meta events
 2020-04-22 15:40:47 -07:00
Chris Lu
a207285af7 cache metadata on startup 2020-04-21 22:00:34 -07:00
Chris Lu
e24b25de78 async meta caching: can stream updates now 2020-04-21 21:16:13 -07:00
Chris Lu
4f02f7121d read from meta cache 
meta cache is not initialized
 2020-04-21 18:50:30 -07:00
Chris Lu
b8e4238ad2 add placeholder for cached meta data reading 2020-04-21 18:02:08 -07:00
Chris Lu
9cacaf5eca better detect file mime type 2020-04-14 11:32:31 -07:00
Chris Lu
2b5c4fbbf3 tiered caching 
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
 2020-04-13 22:19:27 -07:00
Chris Lu
7764e0465c refactoring 2020-04-12 21:00:55 -07:00
Chris Lu
211d87cf4c mount: option to disable caching 2020-04-12 00:52:54 -07:00
Chris Lu
df97da25f9 mount: add on disk caching 2020-04-11 21:12:41 -07:00
Chris Lu
d7f3acb2c0 refactor 2020-04-11 12:45:24 -07:00
Chris Lu
e909b55633 refactoring 2020-04-11 12:39:46 -07:00
Chris Lu
13ab1fdaf0 mount: use file default permission 0666 2020-04-10 02:23:23 -07:00
Chris Lu
eb39df2704 avoid dead loop 2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513 fix to avoid nil file an dirty pages 2020-04-09 21:43:05 -07:00
Chris Lu
bb78ab9915 wait for master to be aware of the failed volumes 
fix https://github.com/chrislusf/seaweedfs/issues/1268
 2020-04-08 23:12:37 -07:00
Chris Lu
006f78b70b make a copy of request data 
to avoid concurrency issues
 2020-04-08 22:31:19 -07:00
Chris Lu
6630541399 ensure correct file size when opening existing files 2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375 add locking to fs cache 2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82 simplify file handle management 2020-04-08 12:50:20 -07:00
Chris Lu
9dc0b1df8f refactoring to compile for windows OS 
windows os does not like to work with fuse
 2020-03-29 21:07:55 -07:00
Chris Lu
54768d0761 fix: delete a file and then create a directory with the same name 2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4 return part of the chunk if chunkview is not the full chunk 2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad adjust print out 2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8 FUSE: add configurable in memory chunk cache size 2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3 FUSE: add chunk cache for recently accessed file chunks 2020-03-28 13:43:31 -07:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
d1439c5bd3 fix FUSE read for large files 
FUSE expects ReadAt do not return partial filled buffer with a nil error.
 2020-03-27 04:50:51 -07:00
Chris Lu
e1911760a7 refactoring 2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37 clean up 2020-03-27 00:30:39 -07:00
Chris Lu
e7e8691720 add some tests 2020-03-26 22:21:52 -07:00
Chris Lu
1fef598347 set target Dir object when moving 2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238 tweak 2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753 rename: set new name for renamed files 2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5 refactor 2020-03-26 03:30:02 -07:00
Chris Lu
dd5b582d05 go fmt 2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206 tree structured fs cache 
FsCache for FsNode, wrapping fs.Node
 2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a fix fscache move 2020-03-25 22:19:19 -07:00
Chris Lu
45ee3736aa refactoring 2020-03-23 01:25:38 -07:00
Chris Lu
c0f0fdb3ba refactoring 2020-03-23 00:01:34 -07:00
Chris Lu
0bf148f49d logging 2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593 commented prints 2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64 FUSE mount: rename correctly across folders 2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45 FUSE mount: rename under root 
fix https://github.com/chrislusf/seaweedfs/issues/1242
 2020-03-22 13:09:16 -07:00
Chris Lu
ae2ee379c0 consistent 64bit size 2020-03-22 01:37:46 -07:00
Chris Lu
65d2ea9fb0 FUSE mount: stream read data with buffer 
fix https://github.com/chrislusf/seaweedfs/issues/1244
 2020-03-22 01:00:36 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
afb20de14c breaks dependency loop 2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b fuse mount: avoid lookup nil entry 
fix https://github.com/chrislusf/seaweedfs/issues/1221
 2020-03-07 16:51:46 -08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d refactoring 2020-03-04 00:39:47 -08:00
Chris Lu
0156e2975a mount: add mode to run external to SeaweedFS container cluster 2020-02-26 16:46:01 -08:00
Chris Lu
36d6595658 remove ctx 2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e remove ctx 2020-02-25 22:37:54 -08:00
Chris Lu
97ab8a1976 remove ctx if possible 2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f adjust logging 2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15 move filer assign volume grpc errror to response 2020-02-25 17:15:09 -08:00
Chris Lu
7d10fdf737 fix directory lookup nil 2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491 minor 2020-02-25 00:42:48 -08:00
Chris Lu
6ab7368ef2 filer: dynamically create bucket under /buckets folder 2020-02-24 22:28:45 -08:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
f724057051 reduce logs 2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51 avoid retry delay 2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25 mount: able to write to any part of a file 2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979 logging 2020-01-27 00:54:21 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a mount: able to handle large git clone 2020-01-26 13:01:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152 log 2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca logs 2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f mount: reuse the entry object for the new directory 
avoid mkdir and then query for the same directory
reduces these "context canceled" issues

attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
 2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170 log 2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221 adjust logging 2020-01-25 00:31:53 -08:00
Chris Lu
24f62ebe76 minor changes 2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143 reduce log 2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654 mount: return IO error for mkdir 2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea retry context canceled request 2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7 mount: avoid use uint64 for greater than zero test 2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa mount: update the file size so file Attr() can read 
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
 2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202 mount: better combines connected intervals to write to volume servers 2020-01-22 23:00:04 -08:00
Chris Lu
72e1aff237 add lock on dirty pages 2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a adjust logging 2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5 mount: read data that is just written 
able read on data not flushed
multiple file open shares the same file handle

fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
 2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a mount: reuse previous file node 2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18 filer: option to create file only if it is new, O_EXCL 2020-01-22 11:42:40 -08:00
Chris Lu
fca6152c3a fix print format 2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106 mount: fix problems found in issue 1182 
fix https://github.com/chrislusf/seaweedfs/issues/1182

always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
 2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5 mount: able to trim a file 2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59 refactoring 2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466 mount: rename also clear the cache 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161 mount: deletion ignore old vid 2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0 mount: modify file size 2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c filer: set inode 
avoid "getcwd: No such file or directory"
 2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f adjust logs 2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018 logs 2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33 filer: no entry if not found 2020-01-15 19:08:54 -08:00
Chris Lu
8a2d529759 go fmt 2019-12-23 12:48:53 -08:00
Chris Lu
2f21beaccd loading xattr value 2019-12-17 21:45:32 -08:00
Chris Lu
c42b023960 mount: invalidate xattr 2019-12-17 20:38:56 -08:00
Chris Lu
9c781da34a mount: fix nil directory problem 2019-12-17 15:08:38 -08:00
Chris Lu
ee6952fddd twaking 2019-12-17 14:03:57 -08:00
Chris Lu
f27a1abf1c mount: fix dir nil 2019-12-17 13:50:32 -08:00
bs106
ce79a31448 Fix missing leading dot in extension 2019-12-17 09:47:12 +00:00
Chris Lu
5961e0d9eb fix logging 2019-12-15 22:24:06 -08:00