Chris Lu
54b835e1ae
filer: redis cluster add option for password
...
fix https://github.com/chrislusf/seaweedfs/issues/971
2019-05-20 09:00:30 -07:00
Chris Lu
fbbc74abb4
adds VolumeEcGenerateSlices, VolumeEcCopy
2019-05-20 00:53:17 -07:00
Chris Lu
ae499fd5aa
Merge branch 'master' into erasure_coding
2019-05-19 21:38:18 -07:00
Chris Lu
698dea779c
refactoring
2019-05-19 21:37:49 -07:00
Chris Lu
693d6c0f1b
clean up tests
2019-05-19 21:16:01 -07:00
Chris Lu
0094ca3f8a
mark the webdav as unstable
2019-05-19 21:08:26 -07:00
Chris Lu
6386a3174b
able to validate by randomly selected ec files
2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376
add locating data inside the ec files
2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08
generate ec01~ec14, generate ecx file with sorted needle values
2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0
Merge branch 'master' into erasure_coding
2019-05-18 11:16:07 -07:00
Chris Lu
f2c4c888f6
1.34
2019-05-17 22:53:30 -07:00
Chris Lu
8123a488cd
filer: optimize leveldb parameters
2019-05-17 18:14:25 -07:00
Chris Lu
76cf57a0e1
volume: optimzation for leveldb
2019-05-17 17:34:03 -07:00
Chris Lu
0fe286a6ca
filer: optimization for leveldb, add leveldb2
...
large filer db may see high CPU and disk usage due to background compaction
2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732
filer: adjust recommended mysql meta data type to LONGBLOB
...
fix https://github.com/chrislusf/seaweedfs/issues/966
2019-05-17 09:11:08 -07:00
Chris Lu
bf9d490538
fix for tests
2019-05-17 02:28:20 -07:00
Chris Lu
82b0759493
filer: migrating filer store from persisting shorter structured file id instead of a string
2019-05-17 02:03:23 -07:00
Chris Lu
8156958ee9
move function to make travis happy
2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6
testing RS coding
2019-05-15 01:02:00 -07:00
Chris Lu
ab22407763
filer ip only use bind ip
2019-05-14 23:43:40 -07:00
Chris Lu
017d0957c3
avoid switching master node if same ip and host name
...
fix https://github.com/chrislusf/seaweedfs/issues/955
2019-05-11 18:42:46 -07:00
Chris Lu
4dfcd2169e
weedfuse: adapt to fstab command line pattern
2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0
weedfuse: daemonize
2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a
add weedfuse
2019-05-10 15:03:31 -07:00
Chris Lu
69b7dd398d
adjust help message
2019-05-07 14:02:01 -07:00
Chris Lu
d01807ba05
shell: volume.balance add option for all_collections, each_collection
2019-05-06 14:12:19 -07:00
Chris Lu
8ffdd5dcdb
writel throttling moving volumes
2019-05-06 13:58:42 -07:00
Chris Lu
cf58fc0e63
refactor: extract out the write throttler
2019-05-06 13:56:08 -07:00
Chris Lu
4e42e7b5e7
shell: volume.balance add dataCenter filter
2019-05-06 13:30:14 -07:00
Chris Lu
4c392ca762
use ip.bind to listen for grpc calls
...
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
2019-05-06 08:10:18 -07:00
Chris Lu
8656102127
shell: volume.balance runs for each collection, and for ALL at the end
2019-05-05 22:28:14 -07:00
Chris Lu
f08bbb72de
refactoring
2019-05-05 22:21:28 -07:00
Chris Lu
0df6346611
add error checking
2019-05-05 21:58:46 -07:00
Chris Lu
3ff92037cc
shell: add volume.balance command
2019-05-05 21:17:23 -07:00
Chris Lu
3f2a30bd05
shell: volume.list output sorted voume list
2019-05-05 20:23:50 -07:00
Chris Lu
46a675ecc2
align memory to avoid nil with atomic operations
2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81
adjust memory layout to avoid possible nil pointer
2019-05-04 08:47:11 -07:00
Chris Lu
25941e0500
master: add jwt expires_after_seconds
2019-05-04 08:42:25 -07:00
Chris Lu
bd8af92b54
remove unused code
2019-05-04 08:20:22 -07:00
Chris Lu
4b27f892e6
remove 128 bit needle id build option
...
fix https://github.com/chrislusf/seaweedfs/issues/949
2019-05-03 22:39:00 -07:00
Chris Lu
3fbe22b203
better error handling for nil raftServer
...
addressing https://github.com/chrislusf/seaweedfs/issues/717
2019-05-03 22:33:08 -07:00
Chris Lu
5681208f2d
refactor: remove logger
2019-05-03 22:21:06 -07:00
Chris Lu
8c982db25d
1.33
2019-05-03 17:27:50 -07:00
Chris Lu
b335f81a4f
volume: add option to limit compaction speed
2019-05-03 17:22:39 -07:00
Chris Lu
f0f981e7c8
webdav: fix directory renaming
2019-05-03 14:12:51 -07:00
Chris Lu
bfbecd7253
webdav: fix directory creation
2019-05-03 13:13:11 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav
...
add WebDav
2019-05-03 01:25:10 -07:00
Chris Lu
7064b2ea48
webdav: can write now
2019-05-03 00:55:52 -07:00
Chris Lu
b30c14b631
webdav: can read now
2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7
scaffolding, but seems hard to satisfy mac, windows, and different os versions
...
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e
style: go fmt for all
2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b
mount: allow nonempty fuse mount to work with Samba
...
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
2019-04-29 10:42:09 -07:00
Chris Lu
108d0fb08d
adjust log level
2019-04-26 09:32:37 -07:00
Chris Lu
575926cdec
minor
2019-04-26 09:32:37 -07:00
Chris Lu
630cf1e33e
fix
2019-04-24 12:31:52 -07:00
Chris Lu
ea4592161c
fix
2019-04-24 12:31:14 -07:00
Chris Lu
c5a931b22c
logging
2019-04-24 12:30:36 -07:00
Chris Lu
9beea63960
memory alignment
...
fix https://github.com/chrislusf/seaweedfs/issues/939
2019-04-24 09:23:19 -07:00
Chris Lu
9774ede3cf
adjust usage description
2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0
weed server: also optionally start S3 gateway
2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89
filer: listen on specified ip address
2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c
filer: remove customizable grpc port, fixed to filer port + 10000
2019-04-23 23:58:52 -07:00
Chris Lu
067b935763
report unknown commands
2019-04-23 22:29:36 -07:00
Chris Lu
53e2e73de2
fix metrics
...
fix https://github.com/chrislusf/seaweedfs/issues/935
2019-04-22 12:32:10 -07:00
Chris Lu
0c96cfdb2e
minor
2019-04-21 15:50:51 -07:00
Chris Lu
00b6f653fa
shell: add fs.cat
2019-04-21 15:43:43 -07:00
Chris Lu
59d532a8c3
1.31
2019-04-21 14:09:50 -07:00
Chris Lu
5c411f3e5f
minor
2019-04-21 13:33:32 -07:00
Chris Lu
440111a349
volume: support http status 304 for the same file id
2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1
maintain layout correctness when changing volumes
2019-04-21 13:32:36 -07:00
Chris Lu
0302b9496c
volume: also check cookie for possible duplicates
2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41
adjust log or error
2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75
add remove volumes with version info
2019-04-20 23:53:37 -07:00
Chris Lu
64a9a0e104
shell: fix volume.copy
2019-04-20 20:48:07 -07:00
Chris Lu
5f3d0e33a1
shell: add commands volume.copy volume.delete volume.mount volume.unmount
2019-04-20 20:31:35 -07:00
Chris Lu
d06d3c57d4
add notes
2019-04-20 20:12:07 -07:00
Chris Lu
2ffe98443b
refactoring
2019-04-20 12:05:28 -07:00
Chris Lu
6fc1f53018
shell: add command volume.move
2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9
add function ParseFileIdFromString
2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3
volume: atomic copying file, adds version and stopOffset
2019-04-19 12:29:49 -07:00
Chris Lu
730a032137
filer mysql: adjust scaffolding instructions
2019-04-19 09:55:46 -07:00
Chris Lu
0be2d51c96
read volume lastAppendAtNs when loading a volume
2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f
fix needle map entry size
2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0
refactoring
2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a
refactoring
2019-04-18 21:17:43 -07:00
Chris Lu
a2d34d4802
filer.copy: add adjustable compression level
2019-04-18 20:21:28 -07:00
Chris Lu
fa176fe80f
volume tailing chunks large files
2019-04-18 19:22:13 -07:00
Chris Lu
3dce1016cb
add volume tailer
2019-04-18 11:05:02 -07:00
Chris Lu
b142f9f1d5
go fmt
2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37
add VolumeStreamFollow, but not used yet
2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966
refactoring
2019-04-17 22:04:49 -07:00
Chris Lu
3b9446606a
adjust log level
2019-04-16 23:31:44 -07:00
Chris Lu
07091c44cd
lock btree writes
...
fix https://github.com/chrislusf/seaweedfs/issues/930
2019-04-16 19:48:07 -07:00
Chris Lu
4393b99332
add notes
2019-04-16 13:57:45 -07:00
Chris Lu
338e6d60a5
refactor: prepare for snapshotting
2019-04-16 09:55:37 -07:00
Chris Lu
af49aea0c6
weed shell: fs.meta.save adjusts meta data file name
2019-04-16 09:40:27 -07:00
Chris Lu
8ea1ee6dfa
weed shell: add fs.meta.notify, removing filer.export
2019-04-16 01:58:28 -07:00
Chris Lu
014906ec79
weed shell: add command fs.meta.load to restore meta data
2019-04-16 01:37:11 -07:00
Chris Lu
79c2cca9c1
better error message
2019-04-16 01:15:30 -07:00
Chris Lu
d35023c713
weed shell: add command fs.meta.save
2019-04-16 01:06:32 -07:00
Chris Lu
b3b42bc947
replicate need to include new entry path
2019-04-16 00:44:31 -07:00
Chris Lu
967e108b9a
increase max idle conn per hosts
2019-04-15 22:00:50 -07:00
Chris Lu
f567777d64
volume: fix needle checking for deleted needles
2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1
Revert "Revert "volume: close request body if any parsing error""
...
This reverts commit 359b7c1650
.
2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650
Revert "volume: close request body if any parsing error"
...
This reverts commit 5996656141
.
2019-04-15 21:35:12 -07:00
Chris Lu
5996656141
volume: close request body if any parsing error
2019-04-15 21:11:45 -07:00
Chris Lu
6e116b3d67
volume: reset leader if error happens
2019-04-15 09:09:46 -07:00
Chris Lu
e85048bcdc
http exhaust and close response body
2019-04-14 23:28:24 -07:00
Chris Lu
3e8a3a8fec
fix race detector found problems
2019-04-14 23:00:37 -07:00
Chris Lu
2e03f3adef
filer: rename -port.public to -port.readonly
2019-04-11 20:42:55 -07:00
Chris Lu
59a9e15733
1.30
2019-04-11 19:40:36 -07:00
Chris Lu
e11e127d91
close grpc connections during errors
2019-04-11 09:27:08 -07:00
Wine93
361912224d
typo: remove blank
2019-04-11 09:18:53 +00:00
Chris Lu
0e33272f56
small refactoring
2019-04-10 23:39:53 -07:00
Chris Lu
cb2061f86a
Merge pull request #925 from stlpmo-jn/checkUnhealthReplication
...
the implement of repair the error replications of the volume
2019-04-10 23:37:12 -07:00
Chris Lu
c0d6cc0067
weed mount: mount to the same folder name
2019-04-10 23:25:41 -07:00
stlpmo-jn
c3fa50d3b3
remove the health chekcer, because it's the same as command_volume_fix_replication
2019-04-11 13:40:31 +08:00
Chris Lu
2f76681d62
refactor: adjust grpc API name
2019-04-10 21:41:17 -07:00
stlpmo-jn
f2031884f0
fix bug : CI build failed
2019-04-11 09:53:31 +08:00
Chris Lu
4df10e50b3
update glide.yaml
2019-04-10 14:27:28 -07:00
stlpmo-jn
c1a0403da2
repair the error replications of the volume
2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5
weed volume: fix bug, remove boltdb, btree
2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96
weed volume: remove boltdb, btree options, add options for leveldb medium, large
2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a
weed volume: disable block cache on leveldb
...
saving 8MB for each volume, but at the cost of slower lookup
2019-04-09 00:24:32 -07:00
Chris Lu
35aba35c07
Revert "weed volume: disable block cache on leveldb"
...
This reverts commit a55e708d36
.
2019-04-09 00:22:40 -07:00
Chris Lu
a55e708d36
weed volume: disable block cache on leveldb
...
saving 8MB for each volume, but at the cost of slower (offset,size) lookup
2019-04-09 00:18:32 -07:00
Chris Lu
6d4c44a85c
1.29
2019-04-08 22:41:59 -07:00
Chris Lu
88ba08cb9e
add large disk support
2019-04-08 22:32:42 -07:00
Chris Lu
784c5bb73a
add build option support 5-byte offset
2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1
divide offset into higher and lower sections
2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc
refactor Offset into a struct of bytes
2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive
...
replication: add GoCDK PubSub support
2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407
weed filer.copy: use existing file owner and gropu id
2019-04-07 11:31:50 -07:00
Chris Lu
a32abda1a3
adjust compression to optimize for speed
2019-04-07 11:01:28 -07:00
Chris Lu
c731d9be6d
weed filer.copy: add option to profile the execution
2019-04-07 09:13:24 -07:00
Chris Lu
174bf1e8b2
more efficient client side gzip compression
2019-04-06 14:14:28 -07:00
Chris Lu
868913aa04
printout statistics
2019-04-06 11:12:35 -07:00
Chris Lu
de7626bd22
weed shell: volume.list add summary statistics
2019-04-06 09:25:29 -07:00
Chris Lu
94302935b2
weed volume: only store compressed data if compression is effective
2019-04-06 00:10:52 -07:00
Chris Lu
b2d92a2992
weed shell: fs.tree fix directory tree listing
2019-04-05 23:42:36 -07:00
Chris Lu
24c020104a
weed filer.copy: parallelize the copying to increase throughput
...
improvement https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 23:35:30 -07:00
Chris Lu
c789b496d8
use cached grpc client
2019-04-05 20:31:58 -07:00
Chris Lu
5808caa2f5
use cached grpc client
2019-04-05 20:26:52 -07:00
Chris Lu
cd6d35aa52
weed shell: fs.tree improvements
2019-04-05 11:40:54 -07:00
Jonathan Amsterdam
72920efc20
added entry to scaffold
2019-04-05 13:43:38 -04:00
Jonathan Amsterdam
a6a5d80401
fix typo
2019-04-05 08:13:32 -04:00
Chris Lu
300b39b5af
weed filer.copy: use cached filer grpc connection
...
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 01:09:06 -07:00
Chris Lu
921231268b
weed filer.copy: skip files that can not be opened
...
such as socket
2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22
default weed filer.copy and weed upload chunk size to 32MB
2019-04-04 23:14:32 -07:00
Chris Lu
c94823f52c
set default http idle connection per host
...
possible fix https://github.com/chrislusf/seaweedfs/issues/915
2019-04-04 20:28:40 -07:00
Chris Lu
0bd7ced7c2
fix test
2019-04-04 20:11:59 -07:00
Chris Lu
9cc73f4a9a
fix test
2019-04-04 19:34:15 -07:00
Chris Lu
af37b374cb
weed shell: fs.cd change current directory
2019-04-04 19:27:51 -07:00
Chris Lu
766396d249
weed master: atomic volume counting
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
2019-04-04 19:27:00 -07:00
Jonathan Amsterdam
c0c9a8bad5
replication: add GoCDK PubSub support
2019-04-04 17:22:45 -04:00
Chris Lu
715a38da1e
weed shell: add fs.cd, fs.pwd to change to a directory and print current directory
2019-04-03 00:20:00 -07:00
Chris Lu
20dcb44077
fix tests
2019-04-01 23:59:31 -07:00
Chris Lu
beaa2bd71a
weed filer: properly delete cached directory entry
...
fix https://github.com/chrislusf/seaweedfs/issues/910
2019-04-01 17:03:04 -07:00
Chris Lu
78b9db34d5
weed s3: generate time format acceptable by aws cli s3
...
fix https://github.com/chrislusf/seaweedfs/issues/911
2019-04-01 16:13:29 -07:00
Chris Lu
2a52e70a41
weed filer: increase default mysql/postgres dir or name length to 65535
2019-04-01 12:37:54 -07:00
Chris Lu
b4c276263f
weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename
2019-04-01 11:03:04 -07:00
Chris Lu
c5a3ff1c35
weed s3: multipart upload goes to the right bucket
...
fix https://github.com/chrislusf/seaweedfs/issues/908
2019-03-31 11:10:19 -07:00
Chris Lu
189c890715
weed replicate: replicate atomic rename to other systems
2019-03-31 08:10:47 -07:00
Chris Lu
8c823abe1f
1.28
2019-03-30 23:22:41 -07:00
Chris Lu
78ac2bef3c
go fmt
2019-03-30 23:09:16 -07:00
Chris Lu
97406333a5
support atomic renaming for mysql/postgres filer store
2019-03-30 23:08:29 -07:00
Chris Lu
920b4e56aa
refactoring
2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947
weed filer.copy: support multiple masters
2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491
weed filer.export: fix pagination error
2019-03-28 00:46:15 -07:00
Chris Lu
07775998a9
some comments
2019-03-28 00:38:30 -07:00
Chris Lu
53b81fcfcd
weed shell: add fs.tree
2019-03-28 00:05:04 -07:00
Chris Lu
c7e7b6229f
detect mime type
2019-03-27 14:25:18 -07:00
Chris Lu
65757ae2fd
weed filer: set content-type consistent with filer store
2019-03-27 11:41:11 -07:00
Chris Lu
37ce4c5269
weed shell: add fs.ls
2019-03-26 12:43:51 -07:00
Chris Lu
7a14cdc90c
refactoring, go fmt
2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6
remove deprecated code
2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c
weed backup: efficient delta backup
...
fix https://github.com/chrislusf/seaweedfs/issues/399
2019-03-25 23:01:53 -07:00
Chris Lu
c913583e2d
Merge branch 'master' into add_volume_follow
2019-03-25 21:19:34 -07:00
Chris Lu
91b9a68493
udpate glide
2019-03-25 09:39:54 -07:00
Chris Lu
a32797518b
weed master: redirect to leader for default admin UI
...
fix https://github.com/chrislusf/seaweedfs/issues/898
2019-03-25 09:20:05 -07:00
Chris Lu
70815e9124
WIP
2019-03-25 09:16:12 -07:00
Chris Lu
eaa42c3865
1.27
2019-03-23 14:14:40 -07:00
Chris Lu
cd8a3b99bb
textual changes
2019-03-23 12:57:35 -07:00
Chris Lu
bd1c0735e0
weed shell: adjust help text format
2019-03-23 11:54:26 -07:00
Chris Lu
6b70b36105
weed shell: add "volume.fix.replication"
2019-03-23 11:34:09 -07:00
Chris Lu
95e0520182
weed volume: add grpc operation to relicate a volume to local
2019-03-23 11:33:34 -07:00
Chris Lu
a3490b600c
weed filer, weed master: add option to disable http
2019-03-21 16:00:46 -07:00
Chris Lu
531add52c2
weed master: skip proxied file read in http read only mode
2019-03-21 09:49:04 -07:00
Jonathan Amsterdam
e966033098
Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk
2019-03-20 16:02:38 -04:00
Chris Lu
977b30e992
weed filer: http HEAD response header add ETag
...
fix https://github.com/chrislusf/seaweedfs/issues/892
2019-03-20 10:13:53 -07:00
Jonathan Amsterdam
8db82e2b75
notification: add Go CDK pubsub support
...
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.
Link in all current providers.
Update the notification scaffold.
2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd
benchmark: add replication flag
2019-03-20 16:38:11 +08:00
Chris Lu
88ab932f7d
refactoring function parameter
2019-03-19 23:01:23 -07:00
Chris Lu
5ae4b963a4
avoid using global rand
2019-03-19 22:20:14 -07:00
Chris Lu
f3d316a846
weed shell: adding file system disk usage fs.du
...
fix https://github.com/chrislusf/seaweedfs/issues/889
echo "fs.du http://localhost:8888/some/path " | weed shell
2019-03-19 21:58:00 -07:00
Chris Lu
cea5c53bf7
fix length checking
...
fix https://github.com/chrislusf/seaweedfs/issues/890
2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3
weed filer: set grpc port to port + 10000
2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08
add error checking
2019-03-19 05:34:43 -07:00
Chris Lu
eb4a54d9fe
refactoring
2019-03-19 05:19:37 -07:00
Chris Lu
ece9d13312
volume info collect compact revision number
2019-03-18 09:32:21 -07:00