21 Commits

Author	SHA1	Message	Date
chrislusf	1b6ab2f6af	Add boltdb for volume needle map ... boltdb is fairly slow to write, about 6 minutes for recreating index for 1553934 files. Boltdb loads 1,553,934 x 16 = 24,862,944bytes from disk, and generate the boltdb as large as 134,217,728 bytes in 6 minutes. To compare, for leveldb, it recreates index in leveldb as large as 27,188,148 bytes in 8 seconds. For in memory version, it loads the index in To test the memory consumption, the leveldb or boltdb index are created. And the server is restarted. Using the benchmark tool to read lots of files. There are 7 volumes in benchmark collection, each with about 1553K files. For leveldb, the memory starts at 142,884KB, and stays at 179,340KB. For boltdb, the memory starts at 73,756KB, and stays at 144,564KB. For in-memory, the memory starts at 368,152KB, and stays at 448,032KB.	2015-03-29 11:04:32 -07:00
chrislusf	020ba6c9a8	add leveldb support for needle map ... This supposedly should reduce memory consumption. However, for tests with millions of, this shows consuming more memories. Need to see whether this will work out. If not, later boltdb will be tested.	2015-03-27 16:34:58 -07:00
chrislusf	9d8a6d2562	adjust log printing	2015-03-09 01:10:04 -07:00
Chris Lu	af416189f1	Cleanup error printing.	2015-01-13 17:04:41 -08:00
Chris Lu	179d36ba0e	formatting code by: goimports -w=true .	2014-10-26 11:34:55 -07:00
wyy	4126280d55	use github.com/chrislusf instead of github.com/aszxqw	2014-09-25 16:57:22 +08:00
wyy	1cd19447e3	use github.com/aszxqw instead of code.google.com/p	2014-09-25 00:47:09 +08:00
Chris Lu	faf497feb8	Important Fix: Avoid overwriting the index file!!!	2014-05-29 19:19:03 -07:00
Chris Lu	3b5035c468	1. v0.54 ... 2. go vet found many printing format errors	2014-04-17 00:16:44 -07:00
Chris Lu	51939efeac	1. volume server now sends master server its max file key, so that ... master server does not need to store the sequence on disk any more 2. fix raft server's failure to init cluster during bootstrapping	2014-04-16 23:43:27 -07:00
Chris Lu	3dbebfd1e1	Thread-safe fixes: ... 1. avoid sharing []byte 2. switch to use ReadAt()	2014-03-19 04:41:16 -07:00
Chris Lu	cd10c277b2	can now delete a collection! Is this a dangerous feature? Only enabling ... deleting "benchmark" collections for now.	2014-03-10 11:43:54 -07:00
Chris Lu	82b74c7940	issue 43 "go fmt" chagnes from "Ryan S. Brown" <sb@ryansb.com> ... some basic changes to parse upload url	2013-09-01 23:58:21 -07:00
Chris Lu	ed154053c8	switching to temporarily use glog library	2013-08-08 23:57:22 -07:00
Chris Lu	50269b74ce	add dataCenter option when assign file keys ... add dataCenter option when starting volume servers some work related to freeze a volume. Not tested yet.	2013-06-19 18:10:38 -07:00
Chris Lu	fb635146a1	refactoring needle mapper interface to separate index file storage logic ... out	2013-04-18 00:23:14 -07:00
Chris Lu	db8e27be6e	add lots of error checking by GThomas	2013-02-26 22:54:22 -08:00
Chris Lu	a4cef2fbd4	only cares about error other than EOF	2013-02-11 23:54:21 -08:00
Chris Lu	91492b4947	avoid EOF error on windows. This is a "try to load" function anyway.	2013-02-11 23:38:48 -08:00
Chris Lu	aecea22666	Issue 15: Stress test corrupts volume	2013-02-10 14:00:06 -08:00
Chris Lu	5071f528f6	testing compilation with remove package	2013-02-10 03:49:51 -08:00