chrislusf
1b6ab2f6af
Add boltdb for volume needle map
...
boltdb is fairly slow to write, about 6 minutes for recreating index
for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from
disk, and generate the boltdb as large as 134,217,728 bytes in 6
minutes.
To compare, for leveldb, it recreates index in leveldb as large as
27,188,148 bytes in 8 seconds.
For in memory version, it loads the index in
To test the memory consumption, the leveldb or boltdb index are
created. And the server is restarted. Using the benchmark tool to read
lots of files. There are 7 volumes in benchmark collection, each with
about 1553K files.
For leveldb, the memory starts at 142,884KB, and stays at 179,340KB.
For boltdb, the memory starts at 73,756KB, and stays at 144,564KB.
For in-memory, the memory starts at 368,152KB, and stays at 448,032KB.
2015-03-29 11:04:32 -07:00
chrislusf
0d23f49f74
removing unused files
2015-03-27 16:35:42 -07:00
chrislusf
020ba6c9a8
add leveldb support for needle map
...
This supposedly should reduce memory consumption. However, for tests
with millions of, this shows consuming more memories. Need to see
whether this will work out. If not, later boltdb will be tested.
2015-03-27 16:34:58 -07:00
chrislusf
78bc7b393f
tweaking ui
2015-03-22 12:50:04 -07:00
chrislusf
b07d81fb08
follow golint suggestions
2015-03-10 00:20:34 -07:00
chrislusf
f511b507a5
Add read only public port on volume server
...
Add read only public port on volume server
2015-03-09 01:10:04 -07:00
chrislusf
d48d76cb4f
adding special handling to recover data if possible
...
For bug #87 and #93 , add special handling to recover data if possible.
2015-03-09 01:10:04 -07:00
chrislusf
853701cb6b
Avoid wrong way to delete on replication failure
...
Avoid wrong way to delete on replication failure. This deletion has bug
to write. The better fix is not to use the deletion on failure at all.
2015-03-09 01:10:04 -07:00
chrislusf
9d8a6d2562
adjust log printing
2015-03-09 01:10:04 -07:00
chrislusf
4ee766e966
avoid auto unzip tar.gz files
...
fix https://github.com/chrislusf/weed-fs/issues/90
2015-02-25 23:59:44 -08:00
Chris Lu
f7998f8652
merge conflicts
2015-02-07 15:35:28 -08:00
Chris Lu
7b4a53b2c1
Add optional admin port to volume server, to seperate admin operations from normal file operations.
2015-01-18 17:03:38 -08:00
yourchanges
a86042e413
fix the #59 and #56
2015-01-16 17:30:23 +08:00
Chris Lu
41bd5179f3
Resolve Conflicts
2015-01-14 09:56:13 -08:00
Chris Lu
af416189f1
Cleanup error printing.
2015-01-13 17:04:41 -08:00
Lei Xue
029e3a3822
fix some typos
2015-01-13 18:46:56 +08:00
yourchanges
7481b7376d
Fix the first form multi-part item is not a "file type part" problem in "ParseUpload()"
2015-01-10 03:22:16 +08:00
yanyiwu
cacfc85869
add some String() to make codes easier to read and debug
2015-01-08 15:54:50 +08:00
yanyiwu
5b7628cf08
use github.com/golang/protobuf/proto instead of code.google.com/p/goprotobuf/proto
2014-12-26 16:59:53 +08:00
yanyiwu
5a40f539f2
fix bug: upload a file which already existed return a wrong file size.
2014-12-26 15:36:33 +08:00
yanyiwu
089eb8ad39
add String function for needle to print
2014-12-26 13:29:44 +08:00
Chris Lu
179d36ba0e
formatting code by: goimports -w=true .
2014-10-26 11:34:55 -07:00
Chris Lu
e9a8999f63
print error the correct way.
2014-10-21 01:27:06 -07:00
wyy
4126280d55
use github.com/chrislusf instead of github.com/aszxqw
2014-09-25 16:57:22 +08:00
wyy
1cd19447e3
use github.com/aszxqw instead of code.google.com/p
2014-09-25 00:47:09 +08:00
Chris Lu
7920b4685e
Adding unit tests for volume ttl.
2014-09-20 20:51:24 -07:00
Chris Lu
b9aee2defb
add TTL support
...
The volume TTL and file TTL are not necessarily the same. as long as
file TTL is smaller than volume TTL, it'll be fine.
volume TTL is used when assigning file id, e.g.
http://.../dir/assign?ttl=3h
file TTL is used when uploading
2014-09-20 12:38:59 -07:00
Chris Lu
69343c5951
adding ttl field to volume super block
2014-08-31 23:25:54 -07:00
Chris Lu
57a4549d86
wrap etag value with double quotes
2014-08-26 10:15:12 -07:00
Chris Lu
4c58cef24a
a bit refactoring to prepare for volume format change and backward
...
compatibility.
2014-08-25 11:37:00 -07:00
Chris Lu
530927db64
Add Etag support
2014-07-22 00:24:50 -07:00
Chris Lu
5f19af8fce
compress *.htm file
2014-07-08 09:32:55 -07:00
Chris Lu
4b7b439be9
Reduce memory usage for "weed fix"
2014-05-31 17:10:51 -07:00
Chris Lu
faf497feb8
Important Fix: Avoid overwriting the index file!!!
2014-05-29 19:19:03 -07:00
Chris Lu
3e5b4da361
fix compilation error
2014-05-26 17:36:41 -07:00
Chris Lu
625f880b15
Adjust "weed server" parameters.
2014-05-26 17:34:54 -07:00
Chris Lu
fe3f06435e
Refactor out volume vacuum.
2014-05-19 20:54:39 -07:00
Chris Lu
e7aaa24da8
Refactor out volume vacuum.
2014-05-19 19:24:35 -07:00
Chris Lu
e8e8d11bd3
Refactor out weedfs vacuum.
2014-05-19 19:18:39 -07:00
Chris Lu
c1307103b2
fix orientation for all jpg files
2014-05-16 01:10:46 -07:00
Chris Lu
34e03e7cf6
iphone usually has upper cased .JPG extension
...
refactor
2014-05-15 01:56:08 -07:00
Chris Lu
8c04c5ed5f
remove the println
2014-04-22 23:10:01 -07:00
Chris Lu
1818a2a2da
Change to protocol buffer for volume-join-masster message
...
Reduced size to about 1/5 of the previous json format message
2014-04-21 02:11:10 -07:00
Chris Lu
637469e656
log the volume server connected to which master server
2014-04-20 23:28:05 -07:00
Chris Lu
3b5035c468
1. v0.54
...
2. go vet found many printing format errors
2014-04-17 00:16:44 -07:00
Chris Lu
51939efeac
1. volume server now sends master server its max file key, so that
...
master server does not need to store the sequence on disk any more
2. fix raft server's failure to init cluster during bootstrapping
2014-04-16 23:43:27 -07:00
Chris Lu
9653a54766
added typed join result
2014-04-16 17:29:58 -07:00
Chris Lu
56a3d30e75
batch delete on volume servers
2014-04-14 01:00:09 -07:00
Chris Lu
6084e7670a
fix bug when reading back the replica settings!
2014-04-13 03:06:15 -07:00
Chris Lu
59f6a13609
adding lots of different stats
2014-03-26 13:22:27 -07:00
