mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-24 01:17:56 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,322 Commits 37 Branches 296 Tags 152 MiB
b5b5c67cf6
Commit Graph

1098 Commits

Author SHA1 Message Date
stlpmo
802a0eb3fe move from cmd-line to scaffold 2019-11-11 09:15:17 +08:00
stlpmo
1c8bed3810 delete the var etcdClient and comments 
refactor the code
add sequencer cmd-line
delete nerver used codes
 2019-11-05 15:45:25 +08:00
Chris Lu
5b950c735e Merge branch 'master' into refactoring_dat_backend 2019-10-29 23:18:41 -07:00
Lei Liu
c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu
19b6a16003 changed from os.file to backend.DataStorageBackend 2019-10-29 00:35:16 -07:00
Chris Lu
eb2172f63f FUSE: add os specific mount options 
related to https://github.com/chrislusf/seaweedfs/issues/1094
 2019-10-27 09:12:10 -07:00
Chris Lu
69de05f6cb go fmt 2019-10-25 07:45:12 -07:00
Chris Lu
288c45a690 fix iitial filer url 2019-10-25 07:44:37 -07:00
Chris Lu
05fe7a2366
Merge pull request #1092 from binbinshi/master 
fix: weed shell filer config is constant
 2019-10-25 07:08:59 -07:00
shibinbin
fe1f469978 fix : change default filer.port to 8888 2019-10-25 17:39:29 +08:00
shibinbin
3464f21b5c fix: weed shell add filer runtime attributes 2019-10-25 17:34:03 +08:00
Chris Lu
9573c0f4b3 filer: filer.copy cleanup in case of failed uploads 2019-10-24 10:16:01 -07:00
Chris Lu
5d78871487 filer: parallize weed filer.copy for single file 2019-10-24 07:26:23 -07:00
Chris Lu
23240efa63 filer: add tikv support 2019-10-23 00:31:31 -07:00
Chris Lu
fc412e428b refactor ScanVolumeFileFrom() 2019-10-22 00:50:30 -07:00
Chris Lu
c262526d8a
Merge pull request #1087 from joeslay/master 
Storing files in in-memory collections on windows
 2019-10-21 22:30:35 -07:00
zhangsong
65f1095a54 fix benchmark cmd usage info 2019-10-14 20:23:32 +08:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock
1f01eb78e8 Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct 2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22 Changes to try and pass the URL parameters through - in memory flag not working still 2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
xushuxun
5904d78bd4 weed backup: add ttl and replication parameter 2019-08-16 11:05:22 +08:00
Chris Lu
6f75df8660 server: fix starting default master list 2019-08-10 03:10:03 -07:00
Chris Lu
b9f6236f8a filer: fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/1031
 2019-08-06 00:42:19 -07:00
Chris Lu
c79274f7eb notes on supported meta data store 2019-08-05 16:45:21 -07:00
Feng Ye
fc0498798b Add etcd filestore 2019-08-01 10:16:45 +08:00
Chris Lu
8afd8d35b3 master: followers can also lookup and redirect 
improve scalability
 2019-07-28 03:58:13 -07:00
Chris Lu
2c6cf72e73 refactoring 2019-07-28 01:55:05 -07:00
Chris Lu
344ce90315 remove weedfuse 2019-07-24 00:42:12 -07:00
Chris Lu
5956dfd08d mount: add umask option 
related to https://github.com/chrislusf/seaweedfs/issues/978
 2019-07-24 00:03:05 -07:00
Chris Lu
61b7a650f4 filer.copy default to 32MB chunks 2019-07-10 21:48:27 -07:00
Chris Lu
23d36dfbaf 1.40 
enable azure replication
 2019-07-02 21:27:05 -07:00
Chris Lu
88fed85bba temp skipping azure 
can not release due to https://github.com/Azure/azure-pipeline-go/issues/23
 2019-06-30 23:17:45 -07:00
Chris Lu
15ecf4e3bf filer: default to leveldb2 2019-06-30 00:44:57 -07:00
Chris Lu
c777102da9 go fmt 2019-06-23 15:30:16 -07:00
Chris Lu
d5560f2705 simplify metrics settings 2019-06-23 15:29:49 -07:00
Chris Lu
6f8b335007 fix duplicated flags 2019-06-23 03:11:21 -07:00
Chris Lu
f16375621f big refactoring 2019-06-23 03:08:27 -07:00
Chris Lu
8da5d5b094 filer.copy: use filer settings, avoid unnecessary command line options 
fix https://github.com/chrislusf/seaweedfs/issues/968
 2019-06-23 01:57:35 -07:00
Chris Lu
d8ed73926d volume servers get metrics address and interval from the master 2019-06-17 14:51:47 -07:00
Chris Lu
8b43679ae3 refactoring 2019-06-15 12:21:44 -07:00
Chris Lu
5f6c9825f8 volume server: adds basic metrics 2019-06-14 00:54:56 -07:00
Chris Lu
a11525fe4e filer: adds basic metrics pushing to Prometheus gateway 2019-06-13 02:01:54 -07:00
Chris Lu
e66bddd84c customizable sleep intervals 2019-06-06 00:39:08 -07:00
Chris Lu
50aa769554 jwt for read access control 2019-06-06 00:29:02 -07:00
Chris Lu
784141c5e6 adjust script parameter 2019-06-05 01:48:03 -07:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
17ac1290c0 volume: load ec shards during heartbeats to master 2019-05-21 22:41:20 -07:00
Chris Lu
54b835e1ae filer: redis cluster add option for password 
fix https://github.com/chrislusf/seaweedfs/issues/971
 2019-05-20 09:00:30 -07:00
Chris Lu
0094ca3f8a mark the webdav as unstable 2019-05-19 21:08:26 -07:00
Chris Lu
0fe286a6ca filer: optimization for leveldb, add leveldb2 
large filer db may see high CPU and disk usage due to background compaction
 2019-05-17 17:33:49 -07:00
Chris Lu
7e0c3bb732 filer: adjust recommended mysql meta data type to LONGBLOB 
fix https://github.com/chrislusf/seaweedfs/issues/966
 2019-05-17 09:11:08 -07:00
Chris Lu
ab22407763 filer ip only use bind ip 2019-05-14 23:43:40 -07:00
Chris Lu
4dfcd2169e weedfuse: adapt to fstab command line pattern 2019-05-11 01:17:55 -07:00
Chris Lu
9a4fb14ea0 weedfuse: daemonize 2019-05-10 17:08:15 -07:00
Chris Lu
1ca1ec906a add weedfuse 2019-05-10 15:03:31 -07:00
Chris Lu
d01807ba05 shell: volume.balance add option for all_collections, each_collection 2019-05-06 14:12:19 -07:00
Chris Lu
4c392ca762 use ip.bind to listen for grpc calls 
help to avoid https://github.com/chrislusf/seaweedfs/issues/952
 2019-05-06 08:10:18 -07:00
Chris Lu
25941e0500 master: add jwt expires_after_seconds 2019-05-04 08:42:25 -07:00
Chris Lu
3fbe22b203 better error handling for nil raftServer 
addressing https://github.com/chrislusf/seaweedfs/issues/717
 2019-05-03 22:33:08 -07:00
Chris Lu
b335f81a4f volume: add option to limit compaction speed 2019-05-03 17:22:39 -07:00
Chris Lu
9d3c945b00
Merge pull request #950 from chrislusf/webdav 
add WebDav
 2019-05-03 01:25:10 -07:00
Chris Lu
b30c14b631 webdav: can read now 2019-05-03 00:24:35 -07:00
Chris Lu
47f14775d7 scaffolding, but seems hard to satisfy mac, windows, and different os versions 
https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/
 2019-05-02 14:22:10 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
82e06a780b mount: allow nonempty fuse mount to work with Samba 
possibly fix https://github.com/chrislusf/seaweedfs/issues/936
 2019-04-29 10:42:09 -07:00
Chris Lu
9774ede3cf adjust usage description 2019-04-24 00:25:20 -07:00
Chris Lu
b04d7e3ac0 weed server: also optionally start S3 gateway 2019-04-24 00:18:01 -07:00
Chris Lu
dabc9c9a89 filer: listen on specified ip address 2019-04-24 00:01:39 -07:00
Chris Lu
21fe5bde7c filer: remove customizable grpc port, fixed to filer port + 10000 2019-04-23 23:58:52 -07:00
Chris Lu
3b3651dea3 volume: atomic copying file, adds version and stopOffset 2019-04-19 12:29:49 -07:00
Chris Lu
730a032137 filer mysql: adjust scaffolding instructions 2019-04-19 09:55:46 -07:00
Chris Lu
ac2727853f fix needle map entry size 2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
a2d34d4802 filer.copy: add adjustable compression level 2019-04-18 20:21:28 -07:00
Chris Lu
13ad5c1966 refactoring 2019-04-17 22:04:49 -07:00
Chris Lu
8ea1ee6dfa weed shell: add fs.meta.notify, removing filer.export 2019-04-16 01:58:28 -07:00
Chris Lu
2e03f3adef filer: rename -port.public to -port.readonly 2019-04-11 20:42:55 -07:00
Wine93
361912224d typo: remove blank 2019-04-11 09:18:53 +00:00
Chris Lu
c0d6cc0067 weed mount: mount to the same folder name 2019-04-10 23:25:41 -07:00
Chris Lu
3555628ad5 weed volume: fix bug, remove boltdb, btree 2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96 weed volume: remove boltdb, btree options, add options for leveldb medium, large 2019-04-09 09:42:06 -07:00
Chris Lu
88ba08cb9e add large disk support 2019-04-08 22:32:42 -07:00
Chris Lu
000ee725fc refactor Offset into a struct of bytes 2019-04-08 19:40:56 -07:00
Chris Lu
a41ba79119
Merge pull request #914 from jba/gocdk-receive 
replication: add GoCDK PubSub support
 2019-04-08 10:31:02 -07:00
Chris Lu
d14b614407 weed filer.copy: use existing file owner and gropu id 2019-04-07 11:31:50 -07:00
Chris Lu
c731d9be6d weed filer.copy: add option to profile the execution 2019-04-07 09:13:24 -07:00
Chris Lu
24c020104a weed filer.copy: parallelize the copying to increase throughput 
improvement https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 23:35:30 -07:00
Jonathan Amsterdam
72920efc20 added entry to scaffold 2019-04-05 13:43:38 -04:00
Chris Lu
300b39b5af weed filer.copy: use cached filer grpc connection 
another attemp for https://github.com/chrislusf/seaweedfs/issues/915
 2019-04-05 01:09:06 -07:00
Chris Lu
921231268b weed filer.copy: skip files that can not be opened 
such as socket
 2019-04-05 00:04:00 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
af37b374cb weed shell: fs.cd change current directory 2019-04-04 19:27:51 -07:00
Chris Lu
715a38da1e weed shell: add fs.cd, fs.pwd to change to a directory and print current directory 2019-04-03 00:20:00 -07:00
Chris Lu
2a52e70a41 weed filer: increase default mysql/postgres dir or name length to 65535 2019-04-01 12:37:54 -07:00
Chris Lu
920b4e56aa refactoring 2019-03-29 22:53:35 -07:00
Chris Lu
f60d383947 weed filer.copy: support multiple masters 2019-03-29 22:50:14 -07:00
Chris Lu
487ca51491 weed filer.export: fix pagination error 2019-03-28 00:46:15 -07:00
Chris Lu
7a14cdc90c refactoring, go fmt 2019-03-25 23:18:40 -07:00
Chris Lu
df95ce0b6c weed backup: efficient delta backup 
fix https://github.com/chrislusf/seaweedfs/issues/399
 2019-03-25 23:01:53 -07:00
Chris Lu
70815e9124 WIP 2019-03-25 09:16:12 -07:00
Chris Lu
a3490b600c weed filer, weed master: add option to disable http 2019-03-21 16:00:46 -07:00
Jonathan Amsterdam
e966033098 Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk 2019-03-20 16:02:38 -04:00
Jonathan Amsterdam
8db82e2b75 notification: add Go CDK pubsub support 
Add the gocdk_pub_sub package, which supports the Go Cloud Development
Kit pubsub API.

Link in all current providers.

Update the notification scaffold.
 2019-03-20 07:57:58 -04:00
chenwanli
45a52b17fd benchmark: add replication flag 2019-03-20 16:38:11 +08:00
Chris Lu
da871896c3 weed filer: set grpc port to port + 10000 2019-03-19 05:47:41 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
Chris Lu
55bab1b456 add context.Context 2019-03-15 17:20:24 -07:00
Chris Lu
cece860bfd add context to all filer APIs 2019-03-15 15:55:34 -07:00
Chris Lu
d30600e6d6 fix filer starting under "weed server" when ip is not localhost 
fix https://github.com/chrislusf/seaweedfs/issues/886
 2019-03-15 09:57:33 -07:00
Chris Lu
a4e8aa98a0 weed server: mkdir first, and set the master ip address 
fix https://github.com/chrislusf/seaweedfs/issues/885
 2019-03-15 00:32:42 -07:00
Chris Lu
6fe071175d mount: add option to disable allow others 
fix https://github.com/chrislusf/seaweedfs/issues/877
 2019-03-10 19:45:40 -07:00
Chris Lu
4773497d2c add back "/cluster/status" 
fix https://github.com/chrislusf/seaweedfs/issues/870
 2019-03-06 12:10:45 -08:00
Chris Lu
ab32556a51 weed server: fix error coming from moving raft to grpc 
fix https://github.com/chrislusf/seaweedfs/issues/872
 2019-03-06 10:21:49 -08:00
Chris Lu
2812c14520 master: add option to disable http operations 2019-03-02 05:10:05 -08:00
Chris Lu
7bbe24dd28 volume server directly support https 2019-02-25 00:43:36 -08:00
Chris Lu
0fa1be8c4b adjust formatting and comments 2019-02-19 21:10:10 -08:00
Chris Lu
07af52cb6f raft change from http to grpc 
master grpc port is fixed to http port + 10000
 2019-02-18 22:38:14 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
215cd27b37 add authorizing fileId write access 
need to secure upload/update/delete for benchmark/filer/mount
need to add secure grpc
 2019-02-14 00:08:20 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Chris Lu
501bd72b1c wip: add security.toml file 2019-02-09 21:07:12 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
a6bffc58f9 go fmt 2019-01-18 14:15:10 -08:00
Chris Lu
67e2ea72be master add separate grpc port 
due to https://github.com/soheilhy/cmux/issues/64

fix https://github.com/chrislusf/seaweedfs/issues/820
fix https://github.com/chrislusf/seaweedfs/issues/840
fix https://github.com/chrislusf/seaweedfs/issues/841
 2019-01-18 14:14:47 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb extract VolumeFileScanner for ScanVolumeFile 2019-01-16 17:48:59 +08:00
Chris Lu
6740427638 weed mount: reduce default chunk size 2019-01-05 15:17:03 -08:00
Chris Lu
55f7d8f801 remove extra log 2018-12-29 18:00:19 -08:00
Chris Lu
170ed53da5 weed mount: mac add AutoXattr 2018-12-29 13:55:38 -08:00
Chris Lu
9f3836072a weed mount: no mac finder broswe mode 2018-12-29 13:16:23 -08:00
Chris Lu
d7bf6c5b91 switching_to_seaweedfs_fuse 2018-12-29 10:31:36 -08:00
Chris Lu
03c7447ad6 set mount point to existing permissions 
fix https://github.com/chrislusf/seaweedfs/issues/806
 2018-12-28 23:36:13 -08:00
Chris Lu
2a19511ba8 avoid gzip benchmark data 
related to https://github.com/chrislusf/seaweedfs/issues/804
 2018-12-27 13:10:46 -08:00
Chris Lu
ba923e2864 error if the last arguement does not end with forward slash 2018-12-23 18:20:11 -08:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
93d0ad09af error log format 2018-12-16 17:37:10 -08:00
Chris Lu
c28e8a2397 refactoring 2018-12-06 00:44:41 -08:00
Chris Lu
ffa2827ab1 fail fast if two notification queues or inputs are enabled 2018-12-06 00:37:59 -08:00
Chris Lu
52b24a9902 default "weed server -filer" to same directory as -mdir 2018-12-05 23:24:25 -08:00
Chris Lu
fb43a9d7b9 adjust message 2018-12-05 02:03:03 -08:00
Chris Lu
1cbd53c01c WIP SeaweedFileSystem added mkdirs, getFileStatus, listStatus, delete 2018-11-25 13:43:26 -08:00
Chris Lu
86dd933596 go fmt 2018-11-23 00:26:15 -08:00
Chris Lu
6344469a9b set fuse subtype 2018-11-22 23:04:23 -08:00
Chris Lu
60d3681371 default images.fix.orientation to false 2018-11-21 16:50:42 -08:00
Chris Lu
c9b3ef58de options to control filer.export verbosity 2018-11-21 15:15:40 -08:00
Chris Lu
2f2e417b68 detailed export 2018-11-21 15:12:47 -08:00
Chris Lu
2e6a3c7b16 reduce memory usage 2018-11-14 22:48:54 -08:00
Chris Lu
a62adf62ca a bit faster 2018-11-12 11:19:13 -08:00
Chris Lu
affe708961 a bit faster 2018-11-11 00:43:30 -08:00
Chris Lu
a09eaf8756 adjusting, seems there are some perf benefit 2018-11-11 00:08:22 -08:00
Chris Lu
cbd94b18a5 improve "ls -al" performance for large directory 2018-11-08 07:37:34 -08:00
Chris Lu
85d1d99459 default "weed export to current enabled store" 2018-11-04 12:59:29 -08:00
Chris Lu
6c45787a37 selectively export files 2018-11-04 12:07:33 -08:00
Chris Lu
ff4189ab25 go fmt 2018-11-04 11:59:08 -08:00
Chris Lu
8dfac6a4cf working b2 sink 2018-11-04 11:58:59 -08:00
Chris Lu
a64613172d bootstrap filer replication with weed filer.export -targetStore=notification 2018-11-03 12:43:45 -07:00
Chris Lu
3674ad9f8e go fmt 2018-11-01 01:12:21 -07:00
Chris Lu
db584ff7f8 separate into notification.toml, add gcp pub/sub message queue 2018-11-01 01:11:09 -07:00
Chris Lu
4c97ff3717 support AWS SQS as file change notification message queue 2018-10-31 01:11:19 -07:00
Chris Lu
fe31f389b0 add configuration instructions 2018-10-30 02:29:11 -07:00
Chris Lu
c58b73ad9b adding BackBlaze, Azure 2018-10-23 23:59:40 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
556382ff5f adding basic grpc to volume server 2018-10-11 01:16:33 -07:00
Chris Lu
08266b7256 go fmt 2018-10-11 00:08:13 -07:00
Chris Lu
1d7e01b754 reformat 2018-10-11 00:05:54 -07:00
Chris Lu
da6154b29c refactor volume server to startVolumeServer() 2018-10-11 00:04:31 -07:00
Chris Lu
60d2f1557d refactor func name 2018-10-10 23:19:54 -07:00
Chris Lu
9f4c2f87d3 adding filer replication sink to Azure 2018-10-09 01:35:48 -07:00
Chris Lu
96a23591e7 remove comments 2018-10-07 22:21:00 -07:00
Chris Lu
a5b5887efc add filer replication to aws s3 2018-10-06 17:10:15 -07:00
Chris Lu
84d4a2c8a0 add filer replication to google cloud storage 2018-10-06 13:08:38 -07:00
Chris Lu
14858f6224 add logging 2018-10-06 13:01:38 -07:00
Chris Lu
56c5c7b1b6 add google cloud storage 2018-10-04 01:14:44 -07:00
Chris Lu
e8ef501f02 add s3 replication sink 2018-10-03 23:36:52 -07:00
Chris Lu
98384b62fd adjust description 2018-09-28 01:58:34 -07:00
Chris Lu
12ba8b9799 generate useable filer.toml 2018-09-23 09:26:25 -07:00
Chris Lu
b16d47030e save and use last processed kafka offsets 2018-09-23 01:34:40 -07:00
Chris Lu
9fe24991d5 refactoring 2018-09-23 00:40:36 -07:00
Chris Lu
7d6b2a4740 add ETag to upload results and chunks 2018-09-22 22:12:21 -07:00
Chris Lu
420f068376 add more instructions 2018-09-22 22:12:06 -07:00
Chris Lu
01ceace18e adjust sink options 2018-09-22 00:53:52 -07:00
Chris Lu
402ab598b6 go fmt 2018-09-22 00:12:10 -07:00
Chris Lu
7bb62b9f21 delay file creation without chunks from dir.Create to file.Flush 2018-09-22 00:11:46 -07:00
Chris Lu
db69ce89f0 go fmt 2018-09-21 01:56:43 -07:00
Chris Lu
a6cfaba018 able to sync the changes 2018-09-21 01:54:29 -07:00
Chris Lu
a8cd7df44a let filer choose default replication 2018-09-20 09:00:15 -07:00
Chris Lu
d5149e592e add logging on kafka reading 2018-09-19 22:31:45 -07:00
Chris Lu
25fb6f9a46 fix compilation 2018-09-17 02:23:21 -07:00
Chris Lu
779641e9d4 adjust replicated entry name 2018-09-17 01:37:24 -07:00
Chris Lu
b7a375f5aa set large limit for "weed filer.export" directory listing 2018-09-17 00:42:36 -07:00
Chris Lu
4e1f1a0b92 set large limit for "weed mount" directory listing 2018-09-17 00:41:33 -07:00
Chris Lu
788acdf527 add WIP filer.replicate 2018-09-17 00:27:56 -07:00
Chris Lu
7ad5a90127 go fmt 2018-09-03 16:47:00 -07:00
Chris Lu
6e3f4d1079 add working filer.export command 2018-08-23 00:02:04 -07:00
Chris Lu
98b8f8649d adding filer.export prototype 2018-08-22 23:54:22 -07:00
Chris Lu
ae4e654d2a add example configuration for default redis create-cluster setup 2018-08-22 18:23:19 -07:00
Chris Lu
748870b03f add "weed scaffold" command 2018-08-19 15:36:30 -07:00
Chris Lu
f827ada811 merge notification config with filer.toml 2018-08-19 15:17:55 -07:00
Chris Lu
f036ef8a3c add filer notification 2018-08-13 01:20:49 -07:00
Chris Lu
75d63db60d randomize raft server startup 
also some go fmt
 2018-08-12 14:27:14 -07:00
Chris Lu
eca4b928d2 ensure master server count is odd 2018-08-12 14:25:31 -07:00
Chris Lu
2e427c66ac small refactoring 2018-08-10 23:47:17 -07:00
Chris Lu
44d8e35988 add listener for volumd id location for benchmark tool 2018-07-31 19:12:36 -07:00
Chris Lu
d09606c65b refactoring 2018-07-29 02:25:24 -07:00
Chris Lu
4f317c7e3d working version3 2018-07-24 01:36:04 -07:00
Chris Lu
7f32eb1e25 s3 add https support 2018-07-22 21:28:54 -07:00
Chris Lu
056871242b remove println 2018-07-22 01:17:49 -07:00
Chris Lu
0655151b8c go fmt 2018-07-22 01:15:11 -07:00
Chris Lu
6319d84f42 s3 API add ListObjectsV1 2018-07-22 01:14:36 -07:00
Chris Lu
922c614bde fix compilation problem 2018-07-21 20:42:14 -07:00
Chris Lu
16c66e097b adjust default buckets folder 2018-07-21 20:29:33 -07:00
Chris Lu
7e2031b18f go fmt 2018-07-21 17:39:10 -07:00
Chris Lu
c7228fefa7 add bucket creation and deletion 
1. option for "weed s3 -filer.dir.buckets" to choose a folder for buckets
2. create a bucket
3. delete a bucket, recursively delete all metadata on filer
 2018-07-19 01:21:44 -07:00
Chris Lu
834a25f084 add list all my buckets 2018-07-18 02:37:09 -07:00
Chris Lu
7abfab8e77 add feature to mount a specific filer path to local directory 2018-07-14 21:09:21 -07:00
Chris Lu
3edfe1d28f extend export command to show tombstone + change output format to CSV 
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
 2018-07-14 20:51:17 -07:00
Chris Lu
6d58916cfd fix compilation error 2018-07-14 13:39:57 -07:00
Chris Lu
842dab07b4 weed mount can request to prioritize to write to a data center 2018-07-14 13:36:28 -07:00
Chris Lu
1e1c368ae8 adjust filer help message 2018-07-10 00:22:11 -07:00
Chris Lu
50d351944f filer add data center option 2018-07-09 02:22:48 -07:00
Chris Lu
d4d7ced922 refactoring: add type for needle id, offset 
later the type size can possibly be adjusted
 2018-07-08 02:28:04 -07:00
Chris Lu
922032b9bb add filer sub dir list size limit to "weed server" 2018-07-08 02:11:36 -07:00
Chris Lu
702ff48cda add dir list limit option 2018-07-07 02:18:47 -07:00
Chris Lu
2ad45ca04f refactor 2018-07-07 02:03:25 -07:00
Chris Lu
77fc8c5914 keep alive for gRpc calls 2018-07-03 19:07:55 -07:00
Chris Lu
5bd72696ac weed mount add ttl option 2018-06-11 23:13:33 -07:00
Chris Lu
98110c1697 add collection, replication, ttl to each file 2018-06-10 16:57:32 -07:00
Chris Lu
299312c805 use separate filer grpc port 2018-06-05 23:37:41 -07:00
Chris Lu
43e3f5724c use fixed list of masters in both filer and volume servers 2018-06-01 00:39:39 -07:00
Chris Lu
dc13e10637 fix copy error on 0 size files 2018-05-30 22:28:14 -07:00
Chris Lu
44acf4b756 fix file name 
f.Name() is not the base file name
 2018-05-30 20:48:13 -07:00
Chris Lu
0301504184 add mime, use simple insert and update filer store API 
1. add mime type to file in filer
2. purge old chunks if overwrite during insert
 2018-05-30 20:24:57 -07:00
Chris Lu
26e7cd8c75 assign a different volume on large file copying 2018-05-30 01:05:26 -07:00
Chris Lu
2fe0d479f1 filer.copy supports chunking 2018-05-30 00:54:56 -07:00
Chris Lu
2d7dea1a09 a little refactoring 2018-05-29 23:52:27 -07:00
Chris Lu
8a48baa056 add single chunk file copying to new filer 2018-05-29 23:46:45 -07:00
Chris Lu
4e3ea49cff properly working local write buffer 2018-05-29 01:21:21 -07:00
Chris Lu
5c4480ec6c add mountOptions.chunkSizeLimitMB, remove cmdMount.IsDebug 2018-05-28 13:42:25 -07:00
Chris Lu
8ab7dd9d08 weed mount add options for collection and replication 2018-05-28 13:24:48 -07:00
Chris Lu
458ada173e go fmt 2018-05-27 11:52:26 -07:00
Chris Lu
9e77563c99 add leveldb store 
1. switch to viper for filer store configuration
2. simplify FindEntry() return values, removing “found”
3. add leveldb store
 2018-05-26 03:49:46 -07:00
Chris Lu
9dd228747c filer copy added uid/gid 2018-05-21 01:25:30 -07:00
Chris Lu
c5cf9bd290 properly working filer 2018-05-13 23:56:16 -07:00
Chris Lu
9f345da20f mv filer proto to filer_pb 2018-05-09 23:18:02 -07:00
Chris Lu
942c2cbd7b mv pb to master_pb 2018-05-09 23:11:54 -07:00
Chris Lu
43a69d20bf change filer API to gRPC 2018-05-08 01:59:43 -07:00
Chris Lu
4936d6c342 adjusting for FUSE 2018-05-07 00:37:47 -07:00
Chris Lu
456738ba64 refactoring fuse 2018-05-05 22:47:16 -07:00
Chris Lu
fffb14bc87 better support FUSE Lookup() 2018-05-05 02:01:50 -07:00
Chris Lu
4b9b9547d4 unused option 2018-02-27 13:57:15 -08:00
Chris Lu
61ff330e99 adjust instructions 2018-01-15 20:33:12 -08:00
Chris Lu
ef6224e45d filer default to auto chunking file size to 32MB 2017-09-15 08:24:30 -07:00
Chris Lu
58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu
dbee816205 fix typo 
fix https://github.com/chrislusf/seaweedfs/issues/549
 2017-08-10 10:26:39 -07:00
Chris Lu
e0dfa7a069 ensure 30GB or less 
avoid https://github.com/chrislusf/seaweedfs/issues/535
 2017-07-16 21:40:55 -07:00
Chris Lu
1fbb8723dc add cpu and mem pprof 2017-06-22 01:33:58 -07:00
Chris Lu
5047bdb4a2 skip bytes cache 2017-06-03 11:44:24 -07:00
Chris Lu
478fe0ecf2 filer add readonly public port 2017-05-27 20:14:22 -07:00
Chris Lu
7ecc0f4b11 rename variables 2017-05-27 18:11:18 -07:00
Chris Lu
82c3ccc8dd add btree for volume index 2017-05-26 22:51:25 -07:00
Chris Lu
4ce6586710 use local random generator to avoid global lock 2017-05-19 23:33:36 -07:00
sparklxb
c46e91d229 complement weed mount: add read and delete 2017-03-17 15:02:27 +08:00
sparklxb
3a67b6ddc1 complement weed mount 2017-03-10 14:46:00 +08:00
sparklxb
7f85e15990 complement weed mount 2017-03-10 10:28:19 +08:00
sparklxb
813a4f7456 Fixes #266 2017-03-06 13:04:52 +08:00
Chris Lu
41f686ab9d remove weeds.conf 2017-02-12 21:58:44 -08:00
Chris Lu
d953ed442c auto detect volume server ip address 2017-01-12 13:42:53 -08:00
Chris Lu
1be354bdce default IP to "localhost" 
so that the master can know the ip address.
 2017-01-10 23:57:59 -08:00
Chris Lu
e767c3ea4f disable master connection timeout 
temporarily disable master connection timeout due to heartbeat
connection timeout
 2017-01-10 01:30:00 -08:00
Chris Lu
e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
sparklxb
66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
Chris Lu
53cf1b4900 change the basic unit to MB 2017-01-08 14:57:32 -08:00
Chris Lu
ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
Chris Lu
13e7069eb9 keep track of total data file size 2017-01-06 10:22:20 -08:00
lixianbin
d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu
dcaf1796fe add option to enable caching 2016-11-16 07:09:57 -08:00
Chris Lu
1bc041b46d add a new way to manually compact corrupted volume 
fix https://github.com/chrislusf/seaweedfs/issues/371
 2016-09-22 20:31:20 -07:00
霍晓栋
1c2affea8e Merge remote-tracking branch 'upstream/master' 2016-09-08 10:34:24 +08:00
Chris Lu
c4b7966dbe minor help message change 2016-09-07 18:21:05 -07:00
霍晓栋
b0035747e3 add filer support 2016-08-31 11:32:30 +08:00
Mike Tolman
01d3f69c52 Adding AutoChunk/MaxMB Support to Filer API 
This is related to the following issue I added to chrislusf/seaweedfs:
https://github.com/chrislusf/seaweedfs/issues/342
 2016-08-05 16:01:30 -06:00
Mike Tolman
a89a3c86d0 Revert "Add AutoChunking to the Filer API, so that you can upload really large files through the filer API." 
This reverts commit 09059bfdcc.
 2016-08-05 15:47:46 -06:00
Mike Tolman
ce99bb927d Revert "Adding HTTP verb whitelisting options." 
This reverts commit 34837afc7a.
 2016-08-05 15:45:48 -06:00
Mike Tolman
34837afc7a Adding HTTP verb whitelisting options. 2016-08-05 15:23:43 -06:00
Mike Tolman
09059bfdcc Add AutoChunking to the Filer API, so that you can upload really large files through the filer API. 2016-08-05 15:19:10 -06:00
Chris Lu
185a916f5e adjusting command options 2016-07-21 15:00:07 -07:00
Chris Lu
cdae9fc680 add "weed copy" command to copy files to filer 2016-07-20 23:45:55 -07:00
Chris Lu
d3ae3a8e14 collect proof when stopping 2016-07-05 10:23:14 -07:00
Chris Lu
b38ff39e8b add master profiling option 2016-07-03 15:39:57 -07:00
霍晓栋
1f63094542 refactor Volume Assign function 2016-06-26 10:50:18 +08:00
Chris Lu
5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00