Chris Lu
5e6b714836
add random test
2020-08-22 16:33:00 -07:00
Chris Lu
f48567c5c6
remove unused function
2020-08-19 22:53:49 -07:00
Chris Lu
839634097f
also do flush on release
2020-08-19 01:27:10 -07:00
Chris Lu
a78772d5ea
avoid shutdown in the middle of running
2020-08-19 00:42:02 -07:00
Chris Lu
ae9bc4a508
logs
2020-08-18 23:42:26 -07:00
Chris Lu
ed4b43b419
adjust logs
2020-08-18 23:42:09 -07:00
Chris Lu
9f1e0aeef5
delete chunks in the last step
2020-08-18 23:40:53 -07:00
Chris Lu
618b2f6829
release resources only when needed to
2020-08-18 12:53:08 -07:00
Chris Lu
208849702d
logs
2020-08-18 12:52:54 -07:00
Chris Lu
1fcd083db3
printout data size
2020-08-18 09:09:29 -07:00
Chris Lu
be4d42b8e2
rename
2020-08-17 20:15:53 -07:00
Chris Lu
abdaf9958d
possibly read more
2020-08-17 16:04:56 -07:00
Chris Lu
9d46c7bc78
rename
2020-08-17 11:12:10 -07:00
Chris Lu
4ccfdaeb4d
prevent nil
2020-08-17 10:07:34 -07:00
Chris Lu
24c8e6bcb4
minor optimization
2020-08-17 10:03:34 -07:00
Chris Lu
f5837b7000
report error first
2020-08-16 23:49:10 -07:00
Chris Lu
2ac27616bc
fix possible out of range bytes
...
avoid buff out of range resp.Data = buff[:totalRead]
2020-08-16 23:47:34 -07:00
Chris Lu
22e5132b3a
adjust log level
2020-08-16 16:25:11 -07:00
Chris Lu
ee0f92a6be
reduce memory allocation
2020-08-16 16:24:40 -07:00
Chris Lu
4a77f0820a
clean up logs
2020-08-16 01:37:50 -07:00
Chris Lu
aec7f32b02
fix reader_at
2020-08-16 00:49:08 -07:00
Chris Lu
5d80fc2ec7
adjust logs
2020-08-15 21:09:31 -07:00
Chris Lu
e0bfd3161a
update metadata only if changed
2020-08-15 19:56:24 -07:00
Chris Lu
003d48da21
adjust logs
2020-08-15 19:55:28 -07:00
Chris Lu
6ee8d952d2
adjust log level
2020-08-15 18:24:35 -07:00
Chris Lu
a22ee30596
fix nil
2020-08-15 17:01:42 -07:00
Chris Lu
0d60e67816
ensure meta data changes are updated
2020-08-15 14:15:07 -07:00
Chris Lu
5b43bddf20
proper deletion ordering
...
delete central file store first, then delete local cache
2020-08-15 09:33:41 -07:00
Chris Lu
c647deace1
file size support set file length
...
use Attr.FileSize and TotalChunkSize to determine file size
2020-08-15 09:32:47 -07:00
Chris Lu
a7f669044e
rename also applies to open file handle
2020-08-14 00:22:49 -07:00
Chris Lu
edfa73782f
adjust log level
2020-08-14 00:22:21 -07:00
Chris Lu
090612492c
fix compilation
2020-08-13 00:07:56 -07:00
Chris Lu
d43129d27f
add back handles lock
2020-08-12 23:52:13 -07:00
Chris Lu
e74dc4e4bc
add back fs node cache for renaming
2020-08-09 21:56:09 -07:00
Chris Lu
9832653e1d
FUSE mount: proper error with deleting non empty folder
2020-08-08 21:37:36 -07:00
Chris Lu
20e2ac1add
filer: store md5 metadata for files uploaded by filer
...
fix https://github.com/chrislusf/seaweedfs/issues/1412
2020-08-06 10:04:17 -07:00
Chris Lu
eed525b717
FUSE mount: remove DirListCacheLimit
...
outdated parameter
2020-07-28 09:24:39 -07:00
Chris Lu
d40de39e75
FUSE: do not change crtime, uid, gid on save
2020-07-24 10:13:45 -07:00
Chris Lu
8dfeba8023
FUSE: skip changing to empty uid and gid during flush
...
Skip uid and gid changes if request uid and gid are zero
mongodb lock file, and many interim files, has empty uid and gid
I0724 10:05:51 93643 filehandle.go:173] /db/diagnostic.data/metrics.interim fh 1333342842031408359 flush Flush [ID=0x3 Node=0x50 Uid=0 Gid=0 Pid=178] 0x10 fl=0x0 lk=0x0
2020-07-24 10:06:43 -07:00
Chris Lu
dd29b8c81d
fix compilation
2020-07-24 09:36:05 -07:00
Chris Lu
5788f1f6bf
remove fsnode cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1400
2020-07-24 09:25:54 -07:00
Chris Lu
6962ca48d3
fix compilation
2020-07-23 21:31:23 -07:00
Chris Lu
df7eee523b
FUSE: implement dir fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/1398
2020-07-23 21:09:40 -07:00
Chris Lu
3bbac0820b
FUSE: faster rename implementation
2020-07-23 21:08:42 -07:00
Chris Lu
60d14a9800
mount: fix difference with storage format in local cache
2020-07-20 00:17:50 -07:00
Chris Lu
97d97f3528
go code can read and write chunk manifest
2020-07-19 17:59:43 -07:00
Chris Lu
ddec7b2bb9
go fmt
2020-07-14 07:34:16 -07:00
Chris Lu
2d1df97933
mount: fatal error if can not process meta data
2020-07-11 06:16:48 -07:00
Chris Lu
e3e00d36c5
reset cache for each new release
2020-07-11 06:16:17 -07:00
Chris Lu
7ad0bce2f5
remove old code
2020-07-10 23:03:22 -07:00
Chris Lu
54d6b3c30f
filer: support running multiple mount to different filers avoiding duplicated cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1382
2020-07-07 23:18:44 -07:00
Chris Lu
ce3630e7a4
refactoring
2020-07-03 22:16:59 -07:00
Chris Lu
31e23e9783
filer: support active<=>active filer replication
2020-06-30 22:53:57 -07:00
Chris Lu
42a338d7b0
FUSE mount: clean up file handles during renaming
...
resolve dir rename when file is still open. Need to clean the file handles as soon as possible.
These can happen out of order:
file rename, then file release
file release, then file rename
2020-06-28 13:41:00 -07:00
Chris Lu
95a1860d65
remove empty calls for list cache
2020-06-28 10:25:54 -07:00
Chris Lu
6498d0c86b
remove list cache
2020-06-28 10:21:52 -07:00
Chris Lu
9033a7d369
removed async option
2020-06-28 10:18:32 -07:00
Chris Lu
1bb8cae65d
reverting and working
2020-06-28 10:14:17 -07:00
Chris Lu
3cec4b3c49
FUSE mount: fix file id written twice?
...
fix https://github.com/chrislusf/seaweedfs/issues/1373
2020-06-25 20:11:49 -07:00
Chris Lu
e8e0d629af
ensure correct file mode
2020-06-24 23:26:54 -07:00
Chris Lu
f7a45d448f
FUSE mount: lazy loading meta cache
2020-06-19 09:45:42 -07:00
Chris Lu
b74eced7a3
mount: a fix to prevent possible repeated calls
...
related to https://github.com/bazil/fuse/issues/130
2020-06-17 13:55:16 -07:00
Chris Lu
628b27ef3b
purge old cache implementation
2020-06-11 01:50:00 -07:00
Chris Lu
4598473251
fix possible EOF problem
2020-06-09 18:04:40 -07:00
Chris Lu
bb3e86fbd5
auto create non-existing cache directory
...
fix https://github.com/chrislusf/seaweedfs/issues/1350
2020-06-08 08:49:57 -07:00
Chris Lu
ac48c899e8
add printouts for synchronizing metadata which can be slow for large folders
2020-05-18 10:31:12 -07:00
Chris Lu
47234760f4
log adjust format
2020-05-02 01:33:49 -07:00
Chris Lu
97a504b0bb
fix possible directory deletion error
2020-05-02 01:11:53 -07:00
Chris Lu
871efa4fc1
refactoring
...
some previous chunk etag was using md5, which should be wrong.
2020-04-30 17:20:47 -07:00
Chris Lu
e93588ec78
FUSE mount: atomic local cache updates
2020-04-29 18:20:54 -07:00
Chris Lu
9e72e9e4b8
able to subscribe any topic from any point of time
2020-04-29 17:40:08 -07:00
Chris Lu
ed3cf811f5
refactoring
2020-04-29 13:26:02 -07:00
wuyuxiang
6850d28d6b
refacotr: move signal handling and pprof to grace package
2020-04-28 14:10:23 +08:00
Chris Lu
2a7957b4ca
FUSE: subcribe meta events based on timestamp
2020-04-27 16:52:20 -07:00
Chris Lu
5d0e1d8d74
also writes to local meta cache
...
before waiting for subscribed meta events
2020-04-22 15:40:47 -07:00
Chris Lu
a207285af7
cache metadata on startup
2020-04-21 22:00:34 -07:00
Chris Lu
e24b25de78
async meta caching: can stream updates now
2020-04-21 21:16:13 -07:00
Chris Lu
4f02f7121d
read from meta cache
...
meta cache is not initialized
2020-04-21 18:50:30 -07:00
Chris Lu
b8e4238ad2
add placeholder for cached meta data reading
2020-04-21 18:02:08 -07:00
Chris Lu
9cacaf5eca
better detect file mime type
2020-04-14 11:32:31 -07:00
Chris Lu
2b5c4fbbf3
tiered caching
...
1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files
2020-04-13 22:19:27 -07:00
Chris Lu
7764e0465c
refactoring
2020-04-12 21:00:55 -07:00
Chris Lu
211d87cf4c
mount: option to disable caching
2020-04-12 00:52:54 -07:00
Chris Lu
df97da25f9
mount: add on disk caching
2020-04-11 21:12:41 -07:00
Chris Lu
d7f3acb2c0
refactor
2020-04-11 12:45:24 -07:00
Chris Lu
e909b55633
refactoring
2020-04-11 12:39:46 -07:00
Chris Lu
13ab1fdaf0
mount: use file default permission 0666
2020-04-10 02:23:23 -07:00
Chris Lu
eb39df2704
avoid dead loop
2020-04-09 21:43:26 -07:00
Chris Lu
56ec8c4513
fix to avoid nil file an dirty pages
2020-04-09 21:43:05 -07:00
Chris Lu
bb78ab9915
wait for master to be aware of the failed volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/1268
2020-04-08 23:12:37 -07:00
Chris Lu
006f78b70b
make a copy of request data
...
to avoid concurrency issues
2020-04-08 22:31:19 -07:00
Chris Lu
6630541399
ensure correct file size when opening existing files
2020-04-08 12:50:59 -07:00
Chris Lu
b524a40375
add locking to fs cache
2020-04-08 12:50:34 -07:00
Chris Lu
bd56172b82
simplify file handle management
2020-04-08 12:50:20 -07:00
Chris Lu
9dc0b1df8f
refactoring to compile for windows OS
...
windows os does not like to work with fuse
2020-03-29 21:07:55 -07:00
Chris Lu
54768d0761
fix: delete a file and then create a directory with the same name
2020-03-29 01:39:48 -07:00
Chris Lu
057722bbf4
return part of the chunk if chunkview is not the full chunk
2020-03-29 00:54:39 -07:00
Chris Lu
af6d9976ad
adjust print out
2020-03-28 14:07:25 -07:00
Chris Lu
a75d50bbb8
FUSE: add configurable in memory chunk cache size
2020-03-28 14:07:16 -07:00
Chris Lu
826bc0b7e3
FUSE: add chunk cache for recently accessed file chunks
2020-03-28 13:43:31 -07:00
Chris Lu
eedd33dda3
refactoring
2020-03-28 13:41:58 -07:00
Chris Lu
d1439c5bd3
fix FUSE read for large files
...
FUSE expects ReadAt do not return partial filled buffer with a nil error.
2020-03-27 04:50:51 -07:00
Chris Lu
e1911760a7
refactoring
2020-03-27 00:30:55 -07:00
Chris Lu
2f5ed29c37
clean up
2020-03-27 00:30:39 -07:00
Chris Lu
e7e8691720
add some tests
2020-03-26 22:21:52 -07:00
Chris Lu
1fef598347
set target Dir object when moving
2020-03-26 10:56:40 -07:00
Chris Lu
9e2e07b238
tweak
2020-03-26 10:56:18 -07:00
Chris Lu
cbce793753
rename: set new name for renamed files
2020-03-26 03:30:23 -07:00
Chris Lu
c763a492f5
refactor
2020-03-26 03:30:02 -07:00
Chris Lu
dd5b582d05
go fmt
2020-03-26 00:09:01 -07:00
Chris Lu
7f0d87b206
tree structured fs cache
...
FsCache for FsNode, wrapping fs.Node
2020-03-26 00:08:14 -07:00
Chris Lu
2e4fadd10a
fix fscache move
2020-03-25 22:19:19 -07:00
Chris Lu
45ee3736aa
refactoring
2020-03-23 01:25:38 -07:00
Chris Lu
c0f0fdb3ba
refactoring
2020-03-23 00:01:34 -07:00
Chris Lu
0bf148f49d
logging
2020-03-22 16:19:00 -07:00
Chris Lu
bda13ed593
commented prints
2020-03-22 13:09:51 -07:00
Chris Lu
74e23e8d64
FUSE mount: rename correctly across folders
2020-03-22 13:09:33 -07:00
Chris Lu
64000f5c45
FUSE mount: rename under root
...
fix https://github.com/chrislusf/seaweedfs/issues/1242
2020-03-22 13:09:16 -07:00
Chris Lu
ae2ee379c0
consistent 64bit size
2020-03-22 01:37:46 -07:00
Chris Lu
65d2ea9fb0
FUSE mount: stream read data with buffer
...
fix https://github.com/chrislusf/seaweedfs/issues/1244
2020-03-22 01:00:36 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
afb20de14c
breaks dependency loop
2020-03-07 17:01:39 -08:00
Chris Lu
8645283a7b
fuse mount: avoid lookup nil entry
...
fix https://github.com/chrislusf/seaweedfs/issues/1221
2020-03-07 16:51:46 -08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
f90c43635d
refactoring
2020-03-04 00:39:47 -08:00
Chris Lu
0156e2975a
mount: add mode to run external to SeaweedFS container cluster
2020-02-26 16:46:01 -08:00
Chris Lu
36d6595658
remove ctx
2020-02-25 22:38:27 -08:00
Chris Lu
fd9612d66e
remove ctx
2020-02-25 22:37:54 -08:00
Chris Lu
97ab8a1976
remove ctx if possible
2020-02-25 22:23:59 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
bd3254b53f
adjust logging
2020-02-25 17:24:08 -08:00
Chris Lu
0841bedb15
move filer assign volume grpc errror to response
2020-02-25 17:15:09 -08:00
Chris Lu
7d10fdf737
fix directory lookup nil
2020-02-25 11:13:06 -08:00
Chris Lu
e86da5a491
minor
2020-02-25 00:42:48 -08:00
Chris Lu
6ab7368ef2
filer: dynamically create bucket under /buckets folder
2020-02-24 22:28:45 -08:00
Chris Lu
d335f04de6
support env variables to overwrite toml file
2020-01-29 09:09:55 -08:00
Chris Lu
f724057051
reduce logs
2020-01-27 21:20:41 -08:00
Chris Lu
c8d543ef51
avoid retry delay
2020-01-27 00:58:26 -08:00
Chris Lu
081bc1ea25
mount: able to write to any part of a file
2020-01-27 00:54:52 -08:00
Chris Lu
2f6bb57979
logging
2020-01-27 00:54:21 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
08e4b56a8a
mount: able to handle large git clone
2020-01-26 13:01:11 -08:00
Chris Lu
c48fc8b4de
grpc send error via response instead of grpc error
2020-01-25 09:17:19 -08:00
Chris Lu
ccf3859152
log
2020-01-25 08:16:43 -08:00
Chris Lu
4433c7bdca
logs
2020-01-25 07:34:09 -08:00
Chris Lu
8cccccce9f
mount: reuse the entry object for the new directory
...
avoid mkdir and then query for the same directory
reduces these "context canceled" issues
attr read directory:"/seaweedfs/other/java/s3copier/src/main/java/com" name:"seaweedfs" : rpc error: code = Canceled desc = context canceled
2020-01-25 01:15:54 -08:00
Chris Lu
90e30c0170
log
2020-01-25 00:32:18 -08:00
Chris Lu
8a42aa8221
adjust logging
2020-01-25 00:31:53 -08:00
Chris Lu
24f62ebe76
minor changes
2020-01-24 10:02:53 -08:00
Chris Lu
5f1109f143
reduce log
2020-01-24 01:41:51 -08:00
Chris Lu
9d93c83654
mount: return IO error for mkdir
2020-01-24 01:41:31 -08:00
Chris Lu
107e8a56ea
retry context canceled request
2020-01-24 01:40:51 -08:00
Chris Lu
2f75264ec7
mount: avoid use uint64 for greater than zero test
2020-01-23 22:12:57 -08:00
Chris Lu
c936a12afa
mount: update the file size so file Attr() can read
...
if file Attr() during file writes, the OS, at least for Mac, may try to resend the data to write, which usually are empty.
2020-01-23 21:59:58 -08:00
Chris Lu
c2e589f202
mount: better combines connected intervals to write to volume servers
2020-01-22 23:00:04 -08:00
Chris Lu
72e1aff237
add lock on dirty pages
2020-01-22 15:38:25 -08:00
Chris Lu
66ded8804a
adjust logging
2020-01-22 15:37:59 -08:00
Chris Lu
6b48d246a5
mount: read data that is just written
...
able read on data not flushed
multiple file open shares the same file handle
fix https://github.com/chrislusf/seaweedfs/issues/1182 on linux
2020-01-22 13:42:03 -08:00
Chris Lu
09f4ceef3a
mount: reuse previous file node
2020-01-22 11:43:43 -08:00
Chris Lu
d4e75a0d18
filer: option to create file only if it is new, O_EXCL
2020-01-22 11:42:40 -08:00
Chris Lu
fca6152c3a
fix print format
2020-01-20 20:24:23 -08:00
Chris Lu
a990ef2106
mount: fix problems found in issue 1182
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
always use the non-duplicated fs.Node
Forget() the fs.Node
Rename will also use the right fs.Node
Avoid using the same file handle for the same file
2020-01-20 20:21:01 -08:00
Chris Lu
630f72f8c5
mount: able to trim a file
2020-01-20 00:00:08 -08:00
Chris Lu
1b0bfbaf59
refactoring
2020-01-19 23:59:46 -08:00
Chris Lu
2f15e93466
mount: rename also clear the cache
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-01-19 12:31:56 -08:00
Chris Lu
04019aa161
mount: deletion ignore old vid
2020-01-19 12:07:26 -08:00
Chris Lu
6a40c688e0
mount: modify file size
2020-01-19 12:07:04 -08:00
Chris Lu
39edcfde4c
filer: set inode
...
avoid "getcwd: No such file or directory"
2020-01-19 12:06:19 -08:00
Chris Lu
e9bb0c179f
adjust logs
2020-01-18 13:25:17 -08:00
Chris Lu
061a61e018
logs
2020-01-15 19:09:00 -08:00
Chris Lu
230a0e4a33
filer: no entry if not found
2020-01-15 19:08:54 -08:00
Chris Lu
8a2d529759
go fmt
2019-12-23 12:48:53 -08:00
Chris Lu
2f21beaccd
loading xattr value
2019-12-17 21:45:32 -08:00
Chris Lu
c42b023960
mount: invalidate xattr
2019-12-17 20:38:56 -08:00
Chris Lu
9c781da34a
mount: fix nil directory problem
2019-12-17 15:08:38 -08:00
Chris Lu
ee6952fddd
twaking
2019-12-17 14:03:57 -08:00
Chris Lu
f27a1abf1c
mount: fix dir nil
2019-12-17 13:50:32 -08:00
bs106
ce79a31448
Fix missing leading dot in extension
2019-12-17 09:47:12 +00:00
Chris Lu
5961e0d9eb
fix logging
2019-12-15 22:24:06 -08:00
Chris Lu
33b4b1868b
mount: support extended attributes
2019-12-15 21:07:01 -08:00
Chris Lu
392678f8f3
upload skipping mimetype if not needed
2019-12-15 20:57:08 -08:00
Chris Lu
33da30d2b1
tweak
2019-12-14 11:04:20 -08:00
Chris Lu
de3eb7c380
update cache if updating the attributes
2019-12-13 11:14:09 -08:00
Chris Lu
357ea4b944
rm will clean the cache first
2019-12-13 11:08:26 -08:00
Chris Lu
deb249736e
fix nill
2019-12-13 10:35:23 -08:00
Chris Lu
f59119cd1d
mount: rename to dirListCacheLimit to configure cache for long format directory list
2019-12-13 10:22:24 -08:00
Chris Lu
3e0cd122d8
mount: directory listing cache ttl set to 5 minutes
2019-12-13 10:08:27 -08:00
Chris Lu
05c3b795dc
filer: configurable directory list cache size
2019-12-13 10:05:43 -08:00
Chris Lu
0fa1269bc7
filer: streaming file listing
2019-12-13 00:22:37 -08:00
Chris Lu
d0b423bbc0
filer: increase directory listing pagination size
2019-12-12 09:11:10 -08:00
Chris Lu
f51a6d8ba5
increase filer pagination limit
2019-12-11 22:21:37 -08:00
Chris Lu
40ff4eea14
fix nil problem
2019-12-10 23:13:14 -08:00
Chris Lu
5956dfd08d
mount: add umask option
...
related to https://github.com/chrislusf/seaweedfs/issues/978
2019-07-24 00:03:05 -07:00
Chris Lu
a14bd31f5b
go fmt
2019-06-22 20:05:25 -07:00
Chris Lu
3fa1f150d9
refactoring
2019-06-22 20:04:56 -07:00
Chris Lu
ff97acae62
more logs
2019-06-21 12:14:40 -07:00
Chris Lu
d0ce5b020b
logging
2019-06-21 12:05:00 -07:00
Chris Lu
78994024ff
adjust logging
2019-06-21 11:46:12 -07:00
Chris Lu
11be0b5e91
more logging
2019-06-20 23:45:30 -07:00
Chris Lu
4b0f084b5d
delete chunks only when file writing is successful
2019-06-20 21:58:35 -07:00
Chris Lu
eebc44cce3
refactoring
2019-06-20 21:57:17 -07:00
Chris Lu
3f9ecee40f
working with reading remote intervals
2019-05-28 21:29:07 -07:00
Chris Lu
1ca1ec906a
add weedfuse
2019-05-10 15:03:31 -07:00
Chris Lu
b30c14b631
webdav: can read now
2019-05-03 00:24:35 -07:00
Chris Lu
630cf1e33e
fix
2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c
fix
2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c
logging
2019-04-24 12:30:36 -07:00
Chris Lu
97406333a5
support atomic renaming for mysql/postgres filer store
2019-03-30 23:08:29 -07:00
Chris Lu
c7e7b6229f
detect mime type
2019-03-27 14:25:18 -07:00
Chris Lu
55bab1b456
add context.Context
2019-03-15 17:20:24 -07:00
Chris Lu
d312c55bbe
file path supports windows, avoiding back slashes
...
fix https://github.com/chrislusf/seaweedfs/issues/868
2019-03-04 13:00:08 -08:00
Chris Lu
9a4dda3011
Merge pull request #855 from chrislusf/add_jwt
...
Add jwt
2019-02-18 15:05:32 -08:00
Chris Lu
a1c7dc3806
avoid changing attributes for mount directory
2019-02-18 12:14:28 -08:00
Chris Lu
77b9af531d
adding grpc mutual tls
2019-02-18 12:11:52 -08:00
Chris Lu
712a0e19c8
Merge branch 'master' into add_jwt
2019-02-15 10:01:16 -08:00
Chris Lu
2ec6a679c2
avoid "fchmod failed" during cp
2019-02-15 10:00:27 -08:00
Chris Lu
157c0f7c01
add jwt for writes
2019-02-15 09:59:22 -08:00
Chris Lu
a3b0e39b06
weed mount deletion always goes to weed filer
2019-02-14 07:07:38 -08:00
Chris Lu
e70f740deb
set file attributes
...
fix https://github.com/chrislusf/seaweedfs/issues/847
2019-01-24 19:38:04 -08:00
bingoohuang
ab6be025d7
go fmt and fix some typo
2019-01-17 09:17:19 +08:00
Chris Lu
bd32108a90
disable async file deletion
2019-01-05 22:51:50 -08:00
Chris Lu
be9a7592a1
incrementally calculate visible intervals
2019-01-05 22:23:44 -08:00
Chris Lu
fe50224ea0
refactoring
2019-01-05 15:21:56 -08:00
Chris Lu
d9f9aca453
refactoring
2019-01-05 15:16:39 -08:00
Chris Lu
251b596bc5
increase batch deletion size to 1024
2019-01-03 22:21:57 -08:00
Chris Lu
aac11fb832
reduce directory list cache usage
2019-01-03 22:21:39 -08:00
Chris Lu
9539823c35
fix log
2019-01-01 03:04:44 -08:00
Chris Lu
525db94d9a
async file chunk deletion
2019-01-01 02:33:57 -08:00
Chris Lu
ffb5d3f93a
get bytes from sync pool
2019-01-01 02:14:40 -08:00
Chris Lu
832ac8129a
reduce pointer usage
2018-12-31 15:10:14 -08:00
Chris Lu
d6181076bb
FUSE remove adjust error
2018-12-30 22:29:27 -08:00
Chris Lu
0388d421d2
caching visible intervals for read
...
speeds up 4x in single thread mode
speeds up 30% in 32 threads mode
2018-12-30 00:51:44 -08:00
Chris Lu
d7bf6c5b91
switching_to_seaweedfs_fuse
2018-12-29 10:31:36 -08:00
Chris Lu
03c7447ad6
set mount point to existing permissions
...
fix https://github.com/chrislusf/seaweedfs/issues/806
2018-12-28 23:36:13 -08:00
Chris Lu
a8af1e3f5f
set attributes include crtime
2018-12-28 22:37:18 -08:00
Chris Lu
79168ab0bb
adjust error and log
2018-12-28 15:10:23 -08:00
Chris Lu
308ac1d0d2
reuse buffer by sync pool
2018-12-28 03:27:48 -08:00
Chris Lu
253f190f48
weed mount add symlink support
2018-12-25 22:45:44 -08:00
Chris Lu
9eb2bee86a
avoid error log if not found
2018-12-24 17:45:26 -08:00
Chris Lu
d3839fe279
go fmt
2018-12-22 11:11:57 -08:00
Chris Lu
3ac54792e1
paginate when filer deleting and FUSE mount renaming
2018-12-16 23:20:08 -08:00
Chris Lu
42b8f1145a
weed mount ls paginate the results
...
fix https://github.com/chrislusf/seaweedfs/issues/787
2018-12-11 07:41:50 -08:00
Chris Lu
39ea8484c1
adjust proto API
2018-12-11 07:17:10 -08:00
Chris Lu
1bfb96f34d
optimization for reading whole chunk with gzip encoding
2018-12-07 01:57:55 -08:00
Chris Lu
29f1673d97
refactoring
2018-12-07 01:25:01 -08:00
Chris Lu
86dd933596
go fmt
2018-11-23 00:26:15 -08:00
Chris Lu
444dfded84
add fs.FSStatfser for SeaweedFS weed mount
2018-11-23 00:24:51 -08:00
Chris Lu
2e32b44061
add template for fs.FSStatfser
2018-11-22 23:05:22 -08:00
Chris Lu
2f2e417b68
detailed export
2018-11-21 15:12:47 -08:00
Chris Lu
4fcfc9410f
cleanup
2018-11-18 11:51:38 -08:00
Chris Lu
60421b23f0
reduce weed mount memory usage
2018-11-18 07:49:14 -08:00
Chris Lu
d3b7965c76
caching grpc clients
2018-11-14 23:50:46 -08:00
Chris Lu
d89774cb7f
cache grpc client connections to improve performance
2018-11-14 23:31:39 -08:00
Chris Lu
2e6a3c7b16
reduce memory usage
2018-11-14 22:48:54 -08:00
Chris Lu
11b0a9cb76
fix possible error when reusing file handles
2018-11-13 23:53:17 -08:00
Chris Lu
8178fc9973
adjust logging
2018-11-11 00:07:46 -08:00
Chris Lu
cbd94b18a5
improve "ls -al" performance for large directory
2018-11-08 07:37:34 -08:00
Chris Lu
ed8efb5aef
much improved "ls -al" performance
2018-11-07 11:35:13 -08:00
Chris Lu
ff4189ab25
go fmt
2018-11-04 11:59:08 -08:00
Chris Lu
9db53cf256
use the configurable chunk size
2018-10-14 00:18:52 -07:00
Chris Lu
3ddcd87098
increase weed mount IO perforance
2018-10-14 00:10:17 -07:00
Chris Lu
08266b7256
go fmt
2018-10-11 00:08:13 -07:00
Chris Lu
f7ff84f17f
Revert "smalller block size for better accuracy"
...
This reverts commit 97ecd20fc9
.
2018-09-27 09:42:24 -07:00
Chris Lu
97ecd20fc9
smalller block size for better accuracy
2018-09-27 09:39:17 -07:00
Chris Lu
f53632a84b
add block size for linux du
...
du already working on Mac
2018-09-27 09:33:36 -07:00
Chris Lu
60c1ada4c1
remove filer grpc API GetEntryAttributes
2018-09-25 09:18:06 -07:00
Chris Lu
578f8a6dec
fix empty file reading on linux
2018-09-23 10:01:00 -07:00
Chris Lu
200d9b9ac2
add file name to log
2018-09-23 09:48:55 -07:00
Chris Lu
7d6b2a4740
add ETag to upload results and chunks
2018-09-22 22:12:21 -07:00
Chris Lu
402ab598b6
go fmt
2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21
delay file creation without chunks from dir.Create to file.Flush
2018-09-22 00:11:46 -07:00
Chris Lu
97575e7185
do not set attributes if the file is still open
2018-09-16 12:37:06 -07:00
Chris Lu
267201ff44
handle non continuous writes
...
fix #728
2018-09-10 02:39:41 -07:00
Chris Lu
109385e955
simplifying logic to avoid handling non continuous writes
...
fix https://github.com/chrislusf/seaweedfs/issues/728
2018-09-10 02:21:57 -07:00
Chris Lu
296acc4a0a
go fmt
2018-09-09 16:26:11 -07:00
Chris Lu
747377142b
fix file permission during writes
2018-09-07 13:57:06 -07:00
Chris Lu
75e749039b
fix data writes error when consecutive calls overlaps with previous writes
2018-09-07 13:11:43 -07:00
Chris Lu
f177cec509
avoid slice out of bounds
...
avoid this problem
2018/09/04 16:27:14 fuse: panic in handler for Write [ID=0x27c0d Node=0x2 Uid=0 Gid=0 Pid=0] 0x1 131072 @10607788032 fl=WriteCache lock=0 ffl=OpenReadOnly: runtime error: slice bounds out of range
goroutine 211141 [running]:
bazil.org/fuse/fs.(*Server).serve.func2(0x10d3e60, 0xc00014be30, 0xc00052fef8, 0xc00052fe77)
/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:857 +0x1ac
panic(0xe2d080, 0x17f62b0)
/home/travis/.gimme/versions/go/src/runtime/panic.go:513 +0x1b9
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0xc0005dc000, 0x20000, 0x1000000, 0x276720000, 0xc0003feaa0, 0x0, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:142 +0x8ec
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).saveExistingPagesToStorage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x0, 0x0, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:107 +0x6c
github.com/chrislusf/seaweedfs/weed/filesys.(*ContinuousDirtyPages).AddPage(0xc0000aca80, 0x10d7ba0, 0xc0003fcc00, 0x278460000, 0xc011966050, 0x20000, 0x20fb0, 0x6fc23ac00, 0x4a817c800, 0x0, ...)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/dirty_page.go:70 +0x8f
github.com/chrislusf/seaweedfs/weed/filesys.(*FileHandle).Write(0xc000548410, 0x10d7ba0, 0xc0003fcc00, 0xc00014be30, 0xc011946af8, 0x47fa01, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/filesys/filehandle.go:141 +0x245
bazil.org/fuse/fs.(*Server).handleRequest(0xc0002cc0c0, 0x10d7ba0, 0xc0003fcc00, 0x10cb020, 0xc000394140, 0xc0000acac0, 0x10d3e60, 0xc00014be30, 0xc00052fef8, 0x10ca6a0, ...)
/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:1265 +0x1599
bazil.org/fuse/fs.(*Server).serve(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:878 +0x410
bazil.org/fuse/fs.(*Server).Serve.func1(0xc0002cc0c0, 0x10d3e60, 0xc00014be30)
/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:425 +0x6e
created by bazil.org/fuse/fs.(*Server).Serve
/home/travis/gopath/src/bazil.org/fuse/fs/serve.go:423 +0x321
2018-09-05 02:17:04 -07:00
Chris Lu
0655151b8c
go fmt
2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42
s3 API add ListObjectsV1
2018-07-22 01:14:36 -07:00
Chris Lu
7e2031b18f
go fmt
2018-07-21 17:39:10 -07:00
Chris Lu
13e5541e17
FUSE can change file or folder attributes
...
FUSE can change file or folder attributes
2018-07-19 02:17:36 -07:00
Chris Lu
7abfab8e77
add feature to mount a specific filer path to local directory
2018-07-14 21:09:21 -07:00
Chris Lu
842dab07b4
weed mount can request to prioritize to write to a data center
2018-07-14 13:36:28 -07:00
Chris Lu
c8234a5af6
ensure using none nil attributes
...
fix https://github.com/chrislusf/seaweedfs/issues/674
2018-07-05 23:16:34 -07:00
Chris Lu
77fc8c5914
keep alive for gRpc calls
2018-07-03 19:07:55 -07:00
Chris Lu
d2a811eef2
fix compilation
2018-06-11 23:29:09 -07:00
Chris Lu
5bd72696ac
weed mount add ttl option
2018-06-11 23:13:33 -07:00
Chris Lu
f1273073fc
switch to fs.NodeRequestLookuper
...
in order to set the entry valid duration
2018-06-07 00:07:37 -07:00
Chris Lu
cec1d97035
recursively move files and directories
2018-06-06 23:39:30 -07:00
Chris Lu
ae23e46313
better error log
2018-06-06 23:06:19 -07:00
Chris Lu
e755540be9
fix caching during directory listing
2018-06-06 22:55:59 -07:00
Chris Lu
daabdfe357
remove nodemap, fix directory listing cache
2018-06-06 22:48:51 -07:00
Chris Lu
5c25d29272
support renaming files
2018-06-06 22:11:01 -07:00
Chris Lu
b3447f4375
adjust logging
2018-06-06 02:21:36 -07:00
Chris Lu
6816661b0f
fixed file handle by file full path
2018-06-06 02:09:57 -07:00
Chris Lu
299312c805
use separate filer grpc port
2018-06-05 23:37:41 -07:00
Chris Lu
a218eaf1f0
fix log
2018-05-30 22:09:24 -07:00
Chris Lu
430eb67489
handle large file copy when write request is larger than buffer
2018-05-30 22:02:21 -07:00
Chris Lu
0301504184
add mime, use simple insert and update filer store API
...
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
2018-05-30 20:24:57 -07:00
Chris Lu
4e3ea49cff
properly working local write buffer
2018-05-29 01:21:21 -07:00
Chris Lu
c4b92e17d0
fix isPerfectAppend
2018-05-28 22:45:52 -07:00
Chris Lu
74332e1a61
minor
2018-05-28 14:32:16 -07:00
Chris Lu
be0e88a606
fix chunk size limit for default 0
2018-05-28 13:44:27 -07:00
Chris Lu
5c4480ec6c
add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug
2018-05-28 13:42:25 -07:00
Chris Lu
8ab7dd9d08
weed mount add options for collection and replication
2018-05-28 13:24:48 -07:00
Chris Lu
d0b238d2db
cache local writes before flushing to volume server
2018-05-28 12:30:17 -07:00
Chris Lu
07e0d13d2d
filer support reading multiple chunks, with range support
2018-05-28 05:39:12 -07:00
Chris Lu
458ada173e
go fmt
2018-05-27 11:52:26 -07:00
Chris Lu
6de84c64c6
adding create time
2018-05-25 23:26:40 -07:00
Chris Lu
ac66c133a5
do not read attributes when file is opened
2018-05-25 01:27:21 -07:00
Chris Lu
6d1bcd4b8c
use existing attributes instead of fetching from filer
2018-05-25 01:22:31 -07:00
Chris Lu
0a223838bd
refactoring
2018-05-25 00:57:25 -07:00
Chris Lu
7b81cf3762
better logs
2018-05-24 23:20:26 -07:00
Chris Lu
9f4f8de9ad
skip printout
2018-05-24 23:20:12 -07:00
Chris Lu
d773e11c7a
file handler directly read from volume servers
...
this mostly works fine now!
next: need to cache files to local disk
2018-05-24 01:22:37 -07:00
Chris Lu
849b6ec28d
seems editing already working
...
Need to handle multiple chunks read.
Need to cache local file changes.
2018-05-23 20:55:24 -07:00
Chris Lu
536559f62d
copy works, edit somehow still fails
2018-05-23 03:08:46 -07:00
Chris Lu
873868cc10
not working now
...
need to add file handler
2018-05-22 04:31:44 -07:00
Chris Lu
7362de9a18
weed mount can work well
...
TODO: somehow filer url is returning empty content
2018-05-22 03:26:38 -07:00
Chris Lu
9dd228747c
filer copy added uid/gid
2018-05-21 01:25:30 -07:00
Chris Lu
f07482382b
able to update file content
...
having some issue when vi reports file changed.
2018-05-21 00:00:28 -07:00
Chris Lu
7ca5052942
create files correctly!
2018-05-19 13:51:44 -07:00
Chris Lu
793dd81ca2
skip permission checking when creating dir or files
2018-05-19 12:40:24 -07:00
Chris Lu
e31c514b00
adding modified time to file chunk
2018-05-16 00:54:44 -07:00
Chris Lu
6bf31467c7
adding empty fsync
...
less error now. But still:
vi on write: E514: write error (file system full?)
cp: Input/output error
2018-05-16 00:54:27 -07:00
Chris Lu
b303a02461
cp file can work
...
1. consolidate to filer_pb.FileChunk
2. dir add file, mkdir
3. file flush, write
updates having issue
2018-05-16 00:08:44 -07:00
Chris Lu
58954bf46f
pass file attributes from filer to mount
2018-05-14 02:02:17 -07:00
Chris Lu
9f345da20f
mv filer proto to filer_pb
2018-05-09 23:18:02 -07:00
Chris Lu
43a69d20bf
change filer API to gRPC
2018-05-08 01:59:43 -07:00
Chris Lu
be58993f47
ensure the same Node is returned
2018-05-05 23:50:34 -07:00
Chris Lu
b857cf9d9b
format
2018-05-05 23:39:29 -07:00
Chris Lu
456738ba64
refactoring fuse
2018-05-05 22:47:16 -07:00