mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 06:57:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
890 Commits 32 Branches 296 Tags 152 MiB
86cd40fba8
Commit Graph

50 Commits

Author SHA1 Message Date
chrislusf
86cd40fba8 Add "weed backup" command. 
This is a pre-cursor for asynchronous replication.
 2015-05-26 00:58:41 -07:00
chrislusf
c7cf4fa46c Fix looking for master nodes. 2015-05-22 08:12:51 -07:00
chrislusf
3188382ea7 More debug information for connecting to master 2015-05-14 21:46:59 -07:00
yanyiwu
b7a18580b8 Sort VolumeInfos by VolumeId in Store.Status(); 
Ordered VolumeInfos is more Human-readable,
especially when there is a lot of volumes.
 2015-04-21 01:21:21 +08:00
Stuart P. Bentley
f0c2a2dcb3 Change all chrislusf/weed-fs links to point to chrislu/seaweedfs 2015-04-16 19:18:06 +00:00
chrislusf
1b6ab2f6af Add boltdb for volume needle map 
boltdb is fairly slow to write, about 6 minutes for recreating index
for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from
disk, and generate the boltdb as large as 134,217,728 bytes in 6
minutes.

To compare, for leveldb, it recreates index in leveldb as large as
27,188,148 bytes in 8 seconds.
For in memory version, it loads the index in

To test the memory consumption, the leveldb or boltdb index are
created. And the server is restarted. Using the benchmark tool to read
lots of files. There are 7 volumes in benchmark collection, each with
about 1553K files.
For leveldb, the memory starts at 142,884KB, and stays at 179,340KB.
For boltdb, the memory starts at 73,756KB, and stays at 144,564KB.
For in-memory, the memory starts at 368,152KB, and stays at 448,032KB.
 2015-03-29 11:04:32 -07:00
chrislusf
020ba6c9a8 add leveldb support for needle map 
This supposedly should reduce memory consumption. However, for tests
with millions of, this shows consuming more memories. Need to see
whether this will work out. If not, later boltdb will be tested.
 2015-03-27 16:34:58 -07:00
chrislusf
78bc7b393f tweaking ui 2015-03-22 12:50:04 -07:00
chrislusf
b07d81fb08 follow golint suggestions 2015-03-10 00:20:34 -07:00
chrislusf
f511b507a5 Add read only public port on volume server 
Add read only public port on volume server
 2015-03-09 01:10:04 -07:00
Chris Lu
f7998f8652 merge conflicts 2015-02-07 15:35:28 -08:00
Chris Lu
7b4a53b2c1 Add optional admin port to volume server, to seperate admin operations from normal file operations. 2015-01-18 17:03:38 -08:00
Lei Xue
029e3a3822 fix some typos 2015-01-13 18:46:56 +08:00
yanyiwu
5b7628cf08 use github.com/golang/protobuf/proto instead of code.google.com/p/goprotobuf/proto 2014-12-26 16:59:53 +08:00
yanyiwu
5a40f539f2 fix bug: upload a file which already existed return a wrong file size. 2014-12-26 15:36:33 +08:00
Chris Lu
179d36ba0e formatting code by: goimports -w=true . 2014-10-26 11:34:55 -07:00
wyy
4126280d55 use github.com/chrislusf instead of github.com/aszxqw 2014-09-25 16:57:22 +08:00
wyy
1cd19447e3 use github.com/aszxqw instead of code.google.com/p 2014-09-25 00:47:09 +08:00
Chris Lu
b9aee2defb add TTL support 
The volume TTL and file TTL are not necessarily the same. as long as
file TTL is smaller than volume TTL, it'll be fine.

volume TTL is used when assigning file id, e.g.
http://.../dir/assign?ttl=3h

file TTL is used when uploading
 2014-09-20 12:38:59 -07:00
Chris Lu
e8e8d11bd3 Refactor out weedfs vacuum. 2014-05-19 19:18:39 -07:00
Chris Lu
8c04c5ed5f remove the println 2014-04-22 23:10:01 -07:00
Chris Lu
1818a2a2da Change to protocol buffer for volume-join-masster message 
Reduced size to about 1/5 of the previous json format message
 2014-04-21 02:11:10 -07:00
Chris Lu
637469e656 log the volume server connected to which master server 2014-04-20 23:28:05 -07:00
Chris Lu
3b5035c468 1. v0.54 
2. go vet found many printing format errors
 2014-04-17 00:16:44 -07:00
Chris Lu
51939efeac 1. volume server now sends master server its max file key, so that 
master server does not need to store the sequence on disk any more
2. fix raft server's failure to init cluster during bootstrapping
 2014-04-16 23:43:27 -07:00
Chris Lu
9653a54766 added typed join result 2014-04-16 17:29:58 -07:00
Chris Lu
56a3d30e75 batch delete on volume servers 2014-04-14 01:00:09 -07:00
Chris Lu
59f6a13609 adding lots of different stats 2014-03-26 13:22:27 -07:00
Chris Lu
37dd41ab91 print out log message 2014-03-19 04:41:41 -07:00
Chris Lu
3fec41b911 remove unnecessary code 2014-03-18 22:32:29 -07:00
Chris Lu
cd10c277b2 can now delete a collection! Is this a dangerous feature? Only enabling 
deleting "benchmark" collections for now.
 2014-03-10 11:43:54 -07:00
Chris Lu
27c74a7e66 Major: 
change replication_type to ReplicaPlacement, hopefully cleaner code
works for 9 possible ReplicaPlacement
xyz
x : number of copies on other data centers
y : number of copies on other racks
z : number of copies on current rack
x y z each can be 0,1,2

Minor:
weed server "-mdir" default to "-dir" if empty
 2014-03-02 22:16:54 -08:00
Chris Lu
edae676913 1. volume server auto detect clustered master nodes 
2. remove operation package dependency on storage
 2014-02-14 17:10:49 -08:00
Chris Lu
0e5c4e432d report when size is closing to the volume limit 
fix error
 2014-02-05 10:36:37 -08:00
Chris Lu
2a8c60f71b be lenient when writing, but report right away when volume size limit is 
exceeded
 2014-02-05 10:22:32 -08:00
Chris Lu
3b68711139 support for collections! 2013-11-12 02:21:22 -08:00
Chris Lu
3f5f8657d2 add a command to force compaction of a volume, removing deleted files 2013-09-28 22:18:52 -07:00
Chris Lu
82b74c7940 issue 43 "go fmt" chagnes from "Ryan S. Brown" <sb@ryansb.com> 
some basic changes to parse upload url
 2013-09-01 23:58:21 -07:00
Chris Lu
ed154053c8 switching to temporarily use glog library 2013-08-08 23:57:22 -07:00
Chris Lu
123b0cc2df fix for issue #35 2013-07-19 20:38:00 -07:00
Chris Lu
ff1c04c486 fix issue 34 2013-07-19 19:37:10 -07:00
Chris Lu
d4105f9b46 add support for multiple folders and multiple max limit: eg 
-dir=folder1,folder2,folder3 -max=7,8,9
 2013-07-13 11:38:01 -07:00
Chris Lu
d1fcf1fd67 add error when size limit is reached 2013-07-03 22:14:16 -07:00
Chris Lu
50269b74ce add dataCenter option when assign file keys 
add dataCenter option when starting volume servers
some work related to freeze a volume. Not tested yet.
 2013-06-19 18:10:38 -07:00
Chris Lu
fb635146a1 refactoring needle mapper interface to separate index file storage logic 
out
 2013-04-18 00:23:14 -07:00
Chris Lu
3222d40871 fix compilation error 2013-04-14 19:37:29 -07:00
Chris Lu
95dc977608 accurate error messages during writing 2013-04-14 19:34:37 -07:00
Chris Lu
a4369b35a7 merge changes from about dealing with read only volumes. 
97482255d5.diff
 2013-04-14 19:30:26 -07:00
Chris Lu
db8e27be6e add lots of error checking by GThomas 2013-02-26 22:54:22 -08:00
Chris Lu
5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00