Commit Graph

436 Commits

Author SHA1 Message Date
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
c48fc8b4de grpc send error via response instead of grpc error 2020-01-25 09:17:19 -08:00
Chris Lu
e4b660699f correct comments
fix https://github.com/chrislusf/seaweedfs/issues/1181
2020-01-12 20:31:33 -08:00
Chris Lu
dc9457fb87 fix compilation error 2020-01-07 09:03:47 -08:00
Chris Lu
aa73364a24 weed fix: add back .idx file generation 2020-01-06 16:29:59 -08:00
Chris Lu
3eafec4b29 volume: add option to limit file size 2020-01-03 00:37:26 -08:00
Chris Lu
86d4b18eb5 filer: optional recursive deletion
fix https://github.com/chrislusf/seaweedfs/issues/1176
2019-12-31 11:52:54 -08:00
Chris Lu
efd2f50ede compaction changed to .idx based deletion 2019-12-24 14:55:50 -08:00
Chris Lu
abffe857a1 change btree map to in memory level db 2019-12-24 10:18:56 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
a9b1ee18af default redis reads options
details in https://github.com/chrislusf/seaweedfs/pull/1165
2019-12-21 09:30:51 -08:00
Chris Lu
fcbd04c425 default to false in scaffold.go 2019-12-21 08:31:27 -08:00
Dakota Sullivan
f934a85d58
Add readOnly and routeByLatency options to scaffold.go 2019-12-20 19:27:15 -07:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
f59119cd1d mount: rename to dirListCacheLimit to configure cache for long format directory list 2019-12-13 10:22:24 -08:00
Chris Lu
05c3b795dc filer: configurable directory list cache size 2019-12-13 10:05:43 -08:00
Chris Lu
bbb6ebc3c0 filer: DeleteFolderChildren for deleting large folders 2019-12-13 00:23:05 -08:00
Chris Lu
5847c5afff return meaningful error log 2019-12-10 22:09:29 -08:00
Chris Lu
0da7b894cc pass backend config from master to volume servers 2019-11-29 01:05:09 -08:00
Chris Lu
a9dbd52e06
Merge pull request #1145 from stlpmo-jn/volume_graceful_stop
let volume server graceful stop
2019-11-29 00:11:57 -08:00
stlpmo
ea9d1ebd2f resolve the Duration.Milliseconds() is not exist in golang v11 & v12 2019-11-29 15:17:17 +08:00
Chris Lu
61bc1d6ffd tweaking 2019-11-28 18:47:51 -08:00
Chris Lu
09c6d8fa32 fix test 2019-11-28 18:44:27 -08:00
Chris Lu
f60154f330 master load backend storage config from master.toml 2019-11-28 18:33:18 -08:00
Chris Lu
641b92f53c fix scaffold error 2019-11-28 18:04:25 -08:00
Chris Lu
0b822a6fb5
Merge pull request #1142 from iliul/wip-fuse-mount
fuse: check mount point available before do runmount
2019-11-27 19:22:51 -08:00
Lei Liu
13b6f09fea fuse: adjust linux specific logic
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-28 10:21:31 +08:00
Chris Lu
46294b42e6 filer: support memsql
fix https://github.com/chrislusf/seaweedfs/issues/1141
2019-11-27 12:34:03 -08:00
Lei Liu
1ba3456ac2
go fmt
Signed-off-by: Lei Liu <liul.stone@gmail.com>
2019-11-27 23:08:51 +08:00
Lei Liu
4c87b222f1
fuse: check mount point available before do runmount
1. Use more readable mount point information
2. Fix some typos

eg:

$ df -Th
Filesystem              Type            Size  Used Avail Use% Mounted on
localhost:8888:/        fuse.seaweedfs  206G   512  206G   1% /mnt/weedfs

$ mount | grep weedfs
localhost:8888:/ on /mnt/weedfs type fuse.seaweedfs (rw,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-27 22:56:44 +08:00
Chris Lu
67e5f5b558 typo 2019-11-27 03:09:45 -08:00
Chris Lu
9711a6ffaa WIP 2019-11-18 19:24:37 -08:00
stlpmo
753138a3f7 test passed 2019-11-17 11:40:36 +08:00
Chris Lu
de0c983808 refactoring scaffolding for master.toml 2019-11-14 22:20:38 -08:00
Lei Liu
46755ea1e1 fix master maintenance logic
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-11-12 14:46:10 +08:00
Chris Lu
ee90236a97 Merge branch 'master' into refactoring_dat_backend 2019-11-11 21:30:14 -08:00
stlpmo
62d393d6c9 ut pass 2019-11-11 10:52:21 +08:00
stlpmo
802a0eb3fe move from cmd-line to scaffold 2019-11-11 09:15:17 +08:00
stlpmo
1c8bed3810 delete the var etcdClient and comments
refactor the code
add sequencer cmd-line
delete nerver used codes
2019-11-05 15:45:25 +08:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Lei Liu
c2884cace2 misc updated
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-29 21:28:28 +08:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
eb2172f63f FUSE: add os specific mount options
related to https://github.com/chrislusf/seaweedfs/issues/1094
2019-10-27 09:12:10 -07:00
Chris Lu
69de05f6cb go fmt 2019-10-25 07:45:12 -07:00
Chris Lu
288c45a690 fix iitial filer url 2019-10-25 07:44:37 -07:00
Chris Lu
05fe7a2366
Merge pull request #1092 from binbinshi/master
fix: weed shell filer config is constant
2019-10-25 07:08:59 -07:00
shibinbin
fe1f469978 fix : change default filer.port to 8888 2019-10-25 17:39:29 +08:00
shibinbin
3464f21b5c fix: weed shell add filer runtime attributes 2019-10-25 17:34:03 +08:00
Chris Lu
9573c0f4b3 filer: filer.copy cleanup in case of failed uploads 2019-10-24 10:16:01 -07:00
Chris Lu
5d78871487 filer: parallize weed filer.copy for single file 2019-10-24 07:26:23 -07:00
Chris Lu
23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c262526d8a
Merge pull request #1087 from joeslay/master
Storing files in in-memory collections on windows
2019-10-21 22:30:35 -07:00
zhangsong
65f1095a54 fix benchmark cmd usage info 2019-10-14 20:23:32 +08:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
xushuxun
5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu
6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu
b9f6236f8a filer: fix typo
fix https://github.com/chrislusf/seaweedfs/issues/1031
2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect
improve scalability
2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu
344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d mount: add umask option
related to https://github.com/chrislusf/seaweedfs/issues/978
2019-07-24 00:03:05 -07:00
Chris Lu
61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu
23d36dfbaf 1.40
enable azure replication
2019-07-02 21:27:05 -07:00
Chris Lu
88fed85bba temp skipping azure
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
2019-06-30 23:17:45 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options
fix https://github.com/chrislusf/seaweedfs/issues/968
2019-06-23 01:57:35 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password
fix https://github.com/chrislusf/seaweedfs/issues/971
2019-05-20 09:00:30 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2
large filer db may see high CPU and disk usage due to background compaction
2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB
fix https://github.com/chrislusf/seaweedfs/issues/966
2019-05-17 09:11:08 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
2019-05-06 08:10:18 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer
addressing https://github.com/chrislusf/seaweedfs/issues/717
2019-05-03 22:33:08 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav
add WebDav
2019-05-03 01:25:10 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
2019-04-29 10:42:09 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Wine93
361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu
c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive
replication: add GoCDK PubSub support
2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu
c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu
24c020104a weed filer.copy: parallelize the copying to increase throughput
improvement https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 23:35:30 -07:00
Jonathan Amsterdam
72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Chris Lu
300b39b5af weed filer.copy: use cached filer grpc connection
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
2019-04-05 01:09:06 -07:00
Chris Lu
921231268b weed filer.copy: skip files that can not be opened
such as socket
2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu
715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup
fix https://github.com/chrislusf/seaweedfs/issues/399
2019-03-25 23:01:53 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost
fix https://github.com/chrislusf/seaweedfs/issues/886
2019-03-15 09:57:33 -07:00