Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
...
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
chrislu
59e58c4b23
volume: fix loading old volume format
...
fix https://github.com/chrislusf/seaweedfs/issues/2487
2021-12-05 01:06:01 -08:00
Chris Lu
c46d16b2f3
skip network error when remote volumes start
2021-03-01 01:20:06 -08:00
Chris Lu
6e43e8ce17
add logs for remote file loading
2021-03-01 00:48:59 -08:00
Chris Lu
015d16f43f
add vif file versions in case loading superblock fails
2021-03-01 00:48:30 -08:00
Chris Lu
885ca34748
volume: fail fast if idx files are missing
...
fix https://github.com/chrislusf/seaweedfs/issues/1796
2021-02-11 00:44:40 -08:00
bingoohuang
94ea3bd3a5
renaming NeedleMapType to NeedleMapKind
2021-02-07 09:00:03 +08:00
Chris Lu
4f31c1bb94
go fmt
2020-12-22 02:34:08 -08:00
Chris Lu
d171d9f988
volume: ensure the volume dat and idx files are always closed
...
fix https://github.com/chrislusf/seaweedfs/issues/1646
2020-12-04 21:50:26 -08:00
Chris Lu
6d30b21b10
volume: add "-dir.idx" option for separate index storage
...
fix https://github.com/chrislusf/seaweedfs/issues/1265
2020-11-27 03:17:10 -08:00
Chris Lu
744c5594da
volume: automatically trim out unreachable entries
2020-10-28 01:14:39 -07:00
Chris Lu
faa5c2e89a
refactoring
2020-07-03 16:34:31 -07:00
Chris Lu
c8ca234773
refactoring
2020-04-11 14:27:25 -07:00
Chris Lu
df9d538044
rename function
2020-04-11 14:19:44 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
288baf37fd
saving .vif files correctly
2020-01-08 09:45:26 -08:00
Chris Lu
9134092a8d
endure creating .vif files
2019-12-28 21:36:15 -08:00
Chris Lu
0cc8347a3a
go fmt
2019-12-28 21:13:10 -08:00
Chris Lu
c06f7eb48a
load volume info from .vif file, use superblock as a backup
2019-12-28 12:28:58 -08:00
Chris Lu
f4a74e03d1
refactoring: separating .vif from tier file loading
2019-12-28 11:35:27 -08:00
Chris Lu
2000284435
rename volume tier info to volume info
2019-12-28 11:21:49 -08:00
Chris Lu
efd2f50ede
compaction changed to .idx based deletion
2019-12-24 14:55:50 -08:00
Chris Lu
f42b5bd0f5
rename .sdb to .sdx
2019-12-24 10:19:12 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
ab966d7192
refactoring variable name
2019-12-19 00:44:46 -08:00
Chris Lu
f61de28c69
volume: add deletion capability for previously readonly volumes
2019-12-19 00:42:46 -08:00
Chris Lu
39810a9951
tiered volume is read only
2019-12-18 01:26:22 -08:00
Chris Lu
58f88e530c
volume: use sorted index map for readonly volumes
2019-12-18 01:21:21 -08:00
Chris Lu
ec8de250e2
tiered storage: can copy to s3, read from s3
...
master not aware tiered volume yet, file assigning is not working yet
2019-12-02 15:08:31 -08:00
Chris Lu
85f8649320
refactor memory mapped file into backend storage
2019-11-09 00:10:59 -08:00
Chris Lu
4b5ba4927f
adjust NewDiskFile() paramaters
2019-10-29 22:37:36 -07:00
Chris Lu
19b6a16003
changed from os.file to backend.DataStorageBackend
2019-10-29 00:35:16 -07:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
joeslay
d53aee179b
Merge pull request #10 from chrislusf/master
...
merge seaweed master
2019-10-14 16:03:40 +01:00
Chris Lu
09874f0d16
volume: return error if superblock is not initialized
...
fix https://github.com/chrislusf/seaweedfs/issues/1079
2019-10-09 00:02:50 -07:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
j.laycock
5885ab67b3
rename volume property MemoryMap to MemoryMapMaxSizeMB
2019-09-05 11:56:48 +01:00
j.laycock
1f01eb78e8
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
xushuxun
9af9f36b89
volume: read dat files' last modified time no matter dat files are catWrite or readonly
2019-08-21 11:26:01 +08:00
Chris Lu
115558e5f5
adjust counters
2019-06-17 21:02:50 -07:00
Chris Lu
289fd7eb39
count number of volumes and ec shards
2019-06-16 02:24:15 -07:00
Chris Lu
76cf57a0e1
volume: optimzation for leveldb
2019-05-17 17:34:03 -07:00
Chris Lu
0be2d51c96
read volume lastAppendAtNs when loading a volume
2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f
fix needle map entry size
2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0
refactoring
2019-04-18 21:43:36 -07:00
Chris Lu
8a5ce16e96
weed volume: remove boltdb, btree options, add options for leveldb medium, large
2019-04-09 09:42:06 -07:00
Chris Lu
9b3bf0e46c
fix "weed backup" rerunning
...
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
2018-09-09 02:48:58 -07:00
