Kimbsen
|
38626cb584
|
don't forget the last few bytes
|
2020-06-23 16:53:49 +02:00 |
|
Kimbsen
|
8d1f6cd567
|
added optional md5 verification of uploaded data
|
2020-06-23 16:24:11 +02:00 |
|
Chris Lu
|
de5ca9b258
|
remove fixJpgOrientation
|
2020-06-20 12:50:40 -07:00 |
|
Chris Lu
|
3080c197e3
|
rename UnCompressData to DecompressData
|
2020-06-20 08:16:16 -07:00 |
|
Chris Lu
|
e912fd15e3
|
renaming
|
2020-06-19 22:45:27 -07:00 |
|
Chris Lu
|
16fe132a20
|
detect mime type before replicating to other volume servers
|
2020-06-19 22:11:36 -07:00 |
|
Chris Lu
|
ae1994cbc1
|
erasure coding: fix cases where there are no .ecj files
|
2020-06-18 09:52:35 -07:00 |
|
Chris Lu
|
7ccc31853c
|
hard limit volume file size
|
2020-06-16 00:27:48 -07:00 |
|
Chris Lu
|
bdbe5ddbe9
|
volume server: add a fix for dead lock
https://github.com/chrislusf/seaweedfs/issues/1353
|
2020-06-08 08:12:59 -07:00 |
|
Chris Lu
|
45a9d852d3
|
go fmt
|
2020-06-05 15:27:10 -07:00 |
|
Evgenii Kozlov
|
2410ffbd79
|
Name fix
|
2020-06-05 19:00:38 +03:00 |
|
Evgenii Kozlov
|
0e0db70f55
|
Set volumes ReadOnly if low free disk space
|
2020-06-05 18:18:15 +03:00 |
|
Chris Lu
|
ef2b3a0801
|
leveldb recover error handling
|
2020-05-26 00:03:44 -07:00 |
|
Chris Lu
|
e5a0787653
|
filer: leveldb store add auto recovery, report correct error
fix https://github.com/chrislusf/seaweedfs/issues/1326
|
2020-05-22 10:54:42 -07:00 |
|
Chris Lu
|
7ff2b0121d
|
specify constant data type to uint64 instead of default int
fix https://github.com/chrislusf/seaweedfs/issues/1324
|
2020-05-19 07:55:23 -07:00 |
|
Chris Lu
|
11b33c0760
|
fix compilation problem
|
2020-05-18 10:53:31 -07:00 |
|
zhangsong
|
b161baa3f0
|
fix volume vacuum test unit
|
2020-05-07 19:58:06 +08:00 |
|
zhangsong
|
3703c7e176
|
write request keep compatible with before
|
2020-05-06 21:35:32 +08:00 |
|
zhangsong
|
f9e8702bb4
|
use async write to persistent file to disk - part1
|
2020-05-04 17:39:44 +08:00 |
|
Chris Lu
|
6190fd665d
|
printout error
|
2020-04-27 12:41:31 -07:00 |
|
Chris Lu
|
bafa95045b
|
volume: deletion checks all disk locations
fix https://github.com/chrislusf/seaweedfs/issues/1283
|
2020-04-21 14:49:58 -07:00 |
|
Chris Lu
|
7764e0465c
|
refactoring
|
2020-04-12 21:00:55 -07:00 |
|
Chris Lu
|
e4af63a721
|
volume server: accept fsync=true in write requests
|
2020-04-11 21:39:16 -07:00 |
|
Chris Lu
|
c8ca234773
|
refactoring
|
2020-04-11 14:27:25 -07:00 |
|
Chris Lu
|
df9d538044
|
rename function
|
2020-04-11 14:19:44 -07:00 |
|
Chris Lu
|
9fa065f600
|
typo in logs
|
2020-04-09 23:43:09 -07:00 |
|
Chris Lu
|
e63a79ade8
|
better handle lock in case of exception
|
2020-03-24 18:41:25 -07:00 |
|
Chris Lu
|
3137777d83
|
volume: automatically detect max volume count
|
2020-03-22 16:21:42 -07:00 |
|
Chris Lu
|
cbfe31a9a8
|
idx file sync before compaction
|
2020-03-20 23:38:46 -07:00 |
|
Chris Lu
|
81797a059a
|
volume: sync volume file right before compaction
fix https://github.com/chrislusf/seaweedfs/issues/1237
|
2020-03-19 23:54:52 -07:00 |
|
Chris Lu
|
709f231e23
|
tiered storage: add s3 endpoint for private s3 implementation
fix https://github.com/chrislusf/seaweedfs/issues/1238
|
2020-03-19 21:13:56 -07:00 |
|
Chris Lu
|
b964bbab3d
|
fix compilation
|
2020-03-17 10:01:24 -07:00 |
|
Chris Lu
|
c3cb6fa1d7
|
volume: compaction can cause readonly volumes
address https://github.com/chrislusf/seaweedfs/issues/1233
|
2020-03-17 09:43:57 -07:00 |
|
Chris Lu
|
22400c6633
|
consistent gzip logic
local store gzip same as replicated writes
|
2020-03-15 04:15:40 -07:00 |
|
Chris Lu
|
560df51def
|
refactoring
|
2020-03-15 03:11:26 -07:00 |
|
Chris Lu
|
d439d83772
|
volume: follow compactionBytePerSecond
related to https://github.com/chrislusf/seaweedfs/issues/1108
|
2020-03-11 10:32:17 -07:00 |
|
Chris Lu
|
d3e1ef64b3
|
ensure error checking on windows
|
2020-03-10 13:33:26 -07:00 |
|
Chris Lu
|
88a110e67e
|
release file handle
|
2020-03-10 13:32:22 -07:00 |
|
Chris Lu
|
43e6261579
|
volume: fix memory leak
|
2020-03-10 13:31:23 -07:00 |
|
Chris Lu
|
0871d2cff0
|
volume: fix memory leak during compaction
fix https://github.com/chrislusf/seaweedfs/issues/1222
|
2020-03-09 22:29:02 -07:00 |
|
Chris Lu
|
89eb05b50f
|
filer: support TTL for all filer stores
|
2020-03-09 01:02:01 -07:00 |
|
Chris Lu
|
9b3109a5d8
|
filer: processing all response headers, no pass through to volume server
* filer calculate MD5 etag
* filer handle response headers, instread of pass it to volume servers
|
2020-03-08 15:42:44 -07:00 |
|
Chris Lu
|
1ae83c2938
|
go fmt
|
2020-03-07 06:12:57 -08:00 |
|
Chris Lu
|
ea1169dc80
|
filer cipher: single chunk http POST and PUT and read
|
2020-03-07 06:08:08 -08:00 |
|
Chris Lu
|
7335e62199
|
volume: PUT also conditionally gzip compress
|
2020-03-01 23:37:02 -08:00 |
|
Chris Lu
|
151114ff59
|
volume: fix readonly status reporting
|
2020-03-01 17:57:37 -08:00 |
|
Chris Lu
|
f87f2045b3
|
delete old file first on windows
fix https://github.com/chrislusf/seaweedfs/issues/1210
|
2020-02-27 00:59:35 -08:00 |
|
Chris Lu
|
757c7d67ad
|
avoid nil DataBackend
|
2020-02-27 00:07:25 -08:00 |
|
Chris Lu
|
430b5a49e2
|
unused code
|
2020-02-26 19:35:00 -08:00 |
|
Chris Lu
|
ed0acd1722
|
go fmt
|
2020-02-26 16:52:57 -08:00 |
|
Chris Lu
|
97ab8a1976
|
remove ctx if possible
|
2020-02-25 22:23:59 -08:00 |
|
Chris Lu
|
892e726eb9
|
avoid reusing context object
fix https://github.com/chrislusf/seaweedfs/issues/1182
|
2020-02-25 21:50:12 -08:00 |
|
panyc16
|
7b3764fd9e
|
make loadExistingVolume() return bool
|
2020-02-13 15:19:00 +08:00 |
|
Chris Lu
|
8d94564f41
|
refactor
|
2020-02-04 21:16:34 -08:00 |
|
Chris Lu
|
c1288e9eb4
|
volume: sdx generation uses memdb instead of compactMap
fix https://github.com/chrislusf/seaweedfs/issues/1194
|
2020-02-04 21:12:09 -08:00 |
|
Chris Lu
|
08e4702542
|
sync before closing index file
merge from 0181f87d91
|
2020-02-04 10:38:12 -08:00 |
|
Chris Lu
|
3a35632d58
|
sync before closing file
merge changes from 1747fc2d52
|
2020-02-04 10:37:14 -08:00 |
|
Chris Lu
|
40ae533fa3
|
shell: add volume.configure.replication to change replication for a volume
fix https://github.com/chrislusf/seaweedfs/issues/1192
|
2020-02-02 15:37:23 -08:00 |
|
Chris Lu
|
d335f04de6
|
support env variables to overwrite toml file
|
2020-01-29 09:09:55 -08:00 |
|
Chris Lu
|
72a64a5cf8
|
use the same context object in order to retry
|
2020-01-26 14:42:11 -08:00 |
|
Chris Lu
|
b6e6ca8595
|
adjust logs
|
2020-01-24 20:06:58 -08:00 |
|
Chris Lu
|
c8b2dac6c1
|
volume: avoid sharing volume dat file handle
possibly help on https://github.com/chrislusf/seaweedfs/issues/1184
|
2020-01-21 21:18:01 -08:00 |
|
Chris Lu
|
f8a20ef35e
|
add the old way to compact as a comment
|
2020-01-08 09:45:42 -08:00 |
|
Chris Lu
|
288baf37fd
|
saving .vif files correctly
|
2020-01-08 09:45:26 -08:00 |
|
Chris Lu
|
acf7ca7b93
|
volume: fix compaction
|
2020-01-08 09:45:03 -08:00 |
|
Chris Lu
|
943f4986ef
|
fix possible nil
|
2020-01-08 08:49:18 -08:00 |
|
Chris Lu
|
9995d3bcb5
|
remove println
|
2020-01-07 21:49:28 -08:00 |
|
Chris Lu
|
3eafec4b29
|
volume: add option to limit file size
|
2020-01-03 00:37:26 -08:00 |
|
Chris Lu
|
672868b460
|
always create .vif file
|
2019-12-28 21:52:06 -08:00 |
|
Chris Lu
|
b6f7033478
|
go fmt
|
2019-12-28 21:37:29 -08:00 |
|
Chris Lu
|
9134092a8d
|
endure creating .vif files
|
2019-12-28 21:36:15 -08:00 |
|
Chris Lu
|
0cc8347a3a
|
go fmt
|
2019-12-28 21:13:10 -08:00 |
|
Chris Lu
|
37b64a50b4
|
ec: generate and copy .vif file
|
2019-12-28 12:44:59 -08:00 |
|
Chris Lu
|
c06f7eb48a
|
load volume info from .vif file, use superblock as a backup
|
2019-12-28 12:28:58 -08:00 |
|
Chris Lu
|
f4a74e03d1
|
refactoring: separating .vif from tier file loading
|
2019-12-28 11:35:27 -08:00 |
|
Chris Lu
|
2000284435
|
rename volume tier info to volume info
|
2019-12-28 11:21:49 -08:00 |
|
Chris Lu
|
e8b357fd53
|
rename from .tier to .vif
|
2019-12-28 11:17:39 -08:00 |
|
Chris Lu
|
48d28d3eb2
|
tier: support remote file attributes and remember the file extension
|
2019-12-25 21:37:24 -08:00 |
|
Chris Lu
|
1346437d71
|
Merge pull request #1171 from stlpmo-jn/redundant_type_conversion
remove the redundant type conversion
|
2019-12-25 17:02:05 -08:00 |
|
Chris Lu
|
35393b4a02
|
cleanly destroy remote files
|
2019-12-25 16:17:58 -08:00 |
|
Chris Lu
|
eecad6062a
|
go fmt
|
2019-12-25 09:56:42 -08:00 |
|
Chris Lu
|
d960b3474a
|
tier storage: support downloading the remote dat files
|
2019-12-25 09:53:13 -08:00 |
|
stlpmo
|
f3de4b6c18
|
remove the redundant type conversion
|
2019-12-25 10:13:45 +08:00 |
|
Chris Lu
|
9ff72f616a
|
go fmt
|
2019-12-24 14:56:16 -08:00 |
|
Chris Lu
|
efd2f50ede
|
compaction changed to .idx based deletion
|
2019-12-24 14:55:50 -08:00 |
|
Chris Lu
|
d8b39fe92a
|
testing
|
2019-12-24 11:29:26 -08:00 |
|
Chris Lu
|
f42b5bd0f5
|
rename .sdb to .sdx
|
2019-12-24 10:19:12 -08:00 |
|
Chris Lu
|
abffe857a1
|
change btree map to in memory level db
|
2019-12-24 10:18:56 -08:00 |
|
Chris Lu
|
72a561ab7c
|
refactoring
|
2019-12-24 08:13:01 -08:00 |
|
Chris Lu
|
409a3fe41f
|
Merge pull request #1168 from stlpmo-jn/optimize_DeleteCollectionFromDiskLocation
decouple the volume.Destroy() from the operation of unmountVolume()
|
2019-12-24 07:08:04 -08:00 |
|
stlpmo
|
38e4b79125
|
decouple the volume.Destroy() from the operation of unmountVolume()
|
2019-12-24 17:20:34 +08:00 |
|
stlpmo
|
f300c35266
|
resolve issue : when the host disk full, the volume server will core dump
|
2019-12-24 16:36:15 +08:00 |
|
Chris Lu
|
a18f62bbe7
|
only copy required shards
|
2019-12-23 18:06:13 -08:00 |
|
Chris Lu
|
8a2d529759
|
go fmt
|
2019-12-23 12:48:53 -08:00 |
|
Chris Lu
|
09ca936c78
|
shell: add ec.decode command
|
2019-12-23 12:48:20 -08:00 |
|
Chris Lu
|
ab966d7192
|
refactoring variable name
|
2019-12-19 00:44:46 -08:00 |
|
Chris Lu
|
f61de28c69
|
volume: add deletion capability for previously readonly volumes
|
2019-12-19 00:42:46 -08:00 |
|
Chris Lu
|
deb03ecfa3
|
remove duplicated checking
already checked in store.go
|
2019-12-18 23:59:53 -08:00 |
|
Chris Lu
|
ef3ae3cd41
|
clean up sorted index file
|
2019-12-18 20:59:15 -08:00 |
|
Chris Lu
|
39810a9951
|
tiered volume is read only
|
2019-12-18 01:26:22 -08:00 |
|
Chris Lu
|
58f88e530c
|
volume: use sorted index map for readonly volumes
|
2019-12-18 01:21:21 -08:00 |
|
Chris Lu
|
356bd1b629
|
adjust text
|
2019-12-10 22:09:29 -08:00 |
|
Chris Lu
|
10bd3c6b4b
|
refactoring
|
2019-12-08 19:44:16 -08:00 |
|
Chris Lu
|
2b8e20f122
|
remove unused functions
|
2019-12-08 18:33:25 -08:00 |
|
Chris Lu
|
f38f90b7ea
|
cleanly remove .tier volume file
|
2019-12-06 22:19:47 -08:00 |
|
Chris Lu
|
6827cabfbb
|
volume: dataFileAccessLock change to RW Lock
|
2019-12-06 06:59:57 -08:00 |
|
Chris Lu
|
e426bd541e
|
grow volumes on volume servers with slots freed by cloud storage
|
2019-12-03 21:36:42 -08:00 |
|
Chris Lu
|
7ae8b1cc86
|
show volume info in volume.list in weed shell
|
2019-12-02 23:38:56 -08:00 |
|
Chris Lu
|
1becbce657
|
display remote volumes on volume server ui page
|
2019-12-02 23:23:54 -08:00 |
|
Chris Lu
|
70648d35ad
|
go fmt
|
2019-12-02 20:49:58 -08:00 |
|
Chris Lu
|
6383b45bd0
|
add lock variable
|
2019-12-02 20:49:50 -08:00 |
|
Chris Lu
|
caae543a9f
|
fix test
|
2019-12-02 15:54:24 -08:00 |
|
Chris Lu
|
ec8de250e2
|
tiered storage: can copy to s3, read from s3
master not aware tiered volume yet, file assigning is not working yet
|
2019-12-02 15:08:31 -08:00 |
|
Chris Lu
|
0da7b894cc
|
pass backend config from master to volume servers
|
2019-11-29 01:05:09 -08:00 |
|
Chris Lu
|
61bc1d6ffd
|
tweaking
|
2019-11-28 18:47:51 -08:00 |
|
Chris Lu
|
f60154f330
|
master load backend storage config from master.toml
|
2019-11-28 18:33:18 -08:00 |
|
Chris Lu
|
ee2072dff9
|
volume: avoid inifinite loop reading ec volume info if error happens
|
2019-11-27 23:07:53 -08:00 |
|
Chris Lu
|
0f9ba84274
|
s3
|
2019-11-27 03:09:45 -08:00 |
|
Chris Lu
|
e4c63ff2f8
|
go fmt
|
2019-11-22 14:51:09 -08:00 |
|
Chris Lu
|
9711a6ffaa
|
WIP
|
2019-11-18 19:24:37 -08:00 |
|
chenwanli
|
295f3beed5
|
rm ldb and bdb directory
|
2019-11-19 09:35:06 +08:00 |
|
Chris Lu
|
c6e8225a20
|
adjust memory mapped file size
related to https://github.com/chrislusf/seaweedfs/pull/1087
|
2019-11-12 21:05:48 -08:00 |
|
Lei Liu
|
46755ea1e1
|
fix master maintenance logic
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
|
2019-11-12 14:46:10 +08:00 |
|
Chris Lu
|
85f8649320
|
refactor memory mapped file into backend storage
|
2019-11-09 00:10:59 -08:00 |
|
Chris Lu
|
c5c1d83d91
|
Merge branch 'master' into refactoring_dat_backend
|
2019-11-08 22:54:41 -08:00 |
|
Chris Lu
|
c34ffed43f
|
go fmt
|
2019-11-08 22:47:50 -08:00 |
|
Chris Lu
|
84c503c6a7
|
adjust ec reading log level
|
2019-11-08 22:41:02 -08:00 |
|
zhangsong
|
1dd101f782
|
use read lock to avoid io hang during heartbeat
|
2019-11-08 20:27:55 +08:00 |
|
Chris Lu
|
db30a46050
|
skip memory mapped file when compacting
|
2019-10-29 23:34:38 -07:00 |
|
Chris Lu
|
5b950c735e
|
Merge branch 'master' into refactoring_dat_backend
|
2019-10-29 23:18:41 -07:00 |
|
Chris Lu
|
57e441d67b
|
fix compaction logic
|
2019-10-29 23:18:01 -07:00 |
|
Chris Lu
|
b7156291a8
|
adjust variable names
|
2019-10-29 23:16:43 -07:00 |
|
Chris Lu
|
4b5ba4927f
|
adjust NewDiskFile() paramaters
|
2019-10-29 22:37:36 -07:00 |
|
Chris Lu
|
cd7ce720c7
|
fix test
|
2019-10-29 00:37:15 -07:00 |
|
Chris Lu
|
19b6a16003
|
changed from os.file to backend.DataStorageBackend
|
2019-10-29 00:35:16 -07:00 |
|
Chris Lu
|
7a51a9a582
|
refactor: simplifying to ReadAt() and WriteAt()
|
2019-10-25 00:11:25 -07:00 |
|
Chris Lu
|
46ed2ca902
|
refactoring
|
2019-10-24 23:41:32 -07:00 |
|
Chris Lu
|
f937933b3d
|
refactoring
|
2019-10-23 22:25:53 -07:00 |
|
Chris Lu
|
3c865ee39b
|
fix missing needle header read
|
2019-10-22 00:57:47 -07:00 |
|
Chris Lu
|
fc412e428b
|
refactor ScanVolumeFileFrom()
|
2019-10-22 00:50:30 -07:00 |
|
Chris Lu
|
c9a183eb69
|
refactor memory map related code
|
2019-10-22 00:49:42 -07:00 |
|
Chris Lu
|
fec07c829d
|
go fmt
|
2019-10-21 23:03:48 -07:00 |
|
Chris Lu
|
faec9076a4
|
adjust parameter names
|
2019-10-21 22:57:01 -07:00 |
|
j.laycock
|
2e2fe00dbd
|
Comma, no space?
|
2019-10-18 11:31:25 +01:00 |
|
j.laycock
|
248f3be6e3
|
using a space instead of a comma to hopefully fix the build!
|
2019-10-18 11:23:02 +01:00 |
|
j.laycock
|
2c455841ea
|
Make volumeCreate more consistent between all 3 implementations.
|
2019-10-18 11:01:45 +01:00 |
|
j.laycock
|
b0ddad6889
|
Fix volume_create breaking the build
|
2019-10-18 10:32:07 +01:00 |
|
joeslay
|
d53aee179b
|
Merge pull request #10 from chrislusf/master
merge seaweed master
|
2019-10-14 16:03:40 +01:00 |
|
Chris Lu
|
87e5a02a99
|
fix logic error
|
2019-10-09 00:07:18 -07:00 |
|
Chris Lu
|
09874f0d16
|
volume: return error if superblock is not initialized
fix https://github.com/chrislusf/seaweedfs/issues/1079
|
2019-10-09 00:02:50 -07:00 |
|
j.laycock
|
eb27c2b037
|
Make releaseMemory private and return byte array instead, fix other platform compilation issues, reduce in-memory chunk size.
|
2019-10-01 12:21:44 +01:00 |
|
j.laycock
|
d5f5acb734
|
limit locking physical memory to 80% of max physical memory
|
2019-09-20 12:44:29 +01:00 |
|
j.laycock
|
476140fd6b
|
minor change to setProcessWorkingSetSize function
|
2019-09-19 11:59:00 +01:00 |
|
j.laycock
|
6fc6322c90
|
Change joeslay paths to chrislusf paths
|
2019-09-12 14:18:21 +01:00 |
|
joeslay
|
d8c34b032f
|
Merge pull request #6 from chrislusf/master
merge seaweed master
|
2019-09-12 11:44:09 +01:00 |
|
Chris Lu
|
5e9c65469e
|
volume: skip readonly checking when Destroy()
fix https://github.com/chrislusf/seaweedfs/issues/1063
|
2019-09-11 09:45:09 -07:00 |
|
j.laycock
|
c1a928887a
|
Add some comments/documentation
|
2019-09-11 10:42:37 +01:00 |
|
joeslay
|
9ea9a5e97a
|
Merge pull request #5 from chrislusf/master
merge seaweed master
|
2019-09-11 10:25:27 +01:00 |
|
j.laycock
|
15a4c91c28
|
Remove Finalizer
|
2019-09-10 17:05:54 +01:00 |
|
j.laycock
|
9c9dff7386
|
Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming
|
2019-09-10 16:18:51 +01:00 |
|
Chris Lu
|
02710f9624
|
refactoring: remove unused functions
|
2019-09-09 10:03:41 -07:00 |
|
Chris Lu
|
0f861d23a2
|
avoid nil needle map
fix https://github.com/chrislusf/seaweedfs/issues/1061
|
2019-09-09 09:48:08 -07:00 |
|
j.laycock
|
63acc8c972
|
rename mem_buffer to mBuffer
|
2019-09-05 13:46:29 +01:00 |
|
j.laycock
|
44ae041e80
|
rename more memorymapped variables to memoryMapMaxSizeMB
|
2019-09-05 13:42:23 +01:00 |
|
j.laycock
|
5885ab67b3
|
rename volume property MemoryMap to MemoryMapMaxSizeMB
|
2019-09-05 11:56:48 +01:00 |
|
j.laycock
|
5a6dfb05fc
|
Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size
|
2019-09-05 11:46:12 +01:00 |
|
j.laycock
|
f6146d6ad2
|
use 0 instead of false for max memory flag
|
2019-09-05 11:43:18 +01:00 |
|
j.laycock
|
4257582db5
|
Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks
|
2019-09-05 11:15:01 +01:00 |
|
Tom Maxwell
|
4a878c0006
|
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
|
2019-09-04 15:27:14 +01:00 |
|
j.laycock
|
cc756ddd79
|
Rename End_of_file variable, implement platform indepenent memory_map.go
|
2019-09-04 14:24:17 +01:00 |
|
j.laycock
|
de5a7e60a2
|
remove temp hack
|
2019-09-03 18:20:09 +01:00 |
|
j.laycock
|
cbd0a98fa1
|
Add InMemory to Volume Grow structure
|
2019-09-03 18:19:02 +01:00 |
|
j.laycock
|
1f01eb78e8
|
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
|
2019-09-03 17:00:59 +01:00 |
|
Tom Maxwell
|
d637d86d22
|
Changes to try and pass the URL parameters through - in memory flag not working still
|
2019-09-03 15:41:28 +01:00 |
|
j.laycock
|
9a459d984b
|
Do not vacuum memory mapped files
|
2019-09-03 15:05:43 +01:00 |
|
j.laycock
|
0123c7a898
|
Change to max 2GB for now
|
2019-09-02 17:46:22 +01:00 |
|
joeslay
|
e91139348e
|
Merge pull request #2 from chrislusf/master
merge seaweed master
|
2019-09-02 11:39:40 +01:00 |
|
j.laycock
|
595a1beff0
|
Swap imports to use joeslay
|
2019-09-02 11:28:40 +01:00 |
|
j.laycock
|
1e62a2b233
|
Fix memory not being unmapped
|
2019-08-30 16:55:33 +01:00 |
|
j.laycock
|
840ccdc35d
|
Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly
|
2019-08-30 12:30:00 +01:00 |
|
j.laycock
|
dc50701e7d
|
Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete
|
2019-08-30 12:30:00 +01:00 |
|
j.laycock
|
523f3a12b3
|
use Os.File/uintptr instead of windows.Handle in memory map struct
|
2019-08-30 12:30:00 +01:00 |
|
j.laycock
|
0e5d3b1a70
|
Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files
|
2019-08-30 12:30:00 +01:00 |
|
Chris Lu
|
58168a8c52
|
volume: truncate dat files when creating
|
2019-08-26 13:01:57 -07:00 |
|
joeslay
|
36bde84000
|
Merge pull request #1 from chrislusf/master
merge changes from chrislusf/seaweed master
|
2019-08-23 17:03:37 +01:00 |
|
j.laycock
|
1e89e719b3
|
Add windows memory map functions to storage package
|
2019-08-23 16:19:00 +01:00 |
|
xushuxun
|
9af9f36b89
|
volume: read dat files' last modified time no matter dat files are catWrite or readonly
|
2019-08-21 11:26:01 +08:00 |
|
Chris Lu
|
f9c1cd7370
|
avoid dead lock from dataFileAccessLock
fix https://github.com/chrislusf/seaweedfs/issues/1043
|
2019-08-15 09:24:54 -07:00 |
|
Chris Lu
|
d829df4f59
|
volume: protect against nil needle map
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
|
2019-08-14 01:08:01 -07:00 |
|
Chris Lu
|
e40634e6b4
|
volume: fail the volume deletion if compaction is in progress
fix https://github.com/chrislusf/seaweedfs/issues/1035
|
2019-08-12 00:53:50 -07:00 |
|
Chris Lu
|
f3b99cbfe5
|
fix wrong alignment fix
|
2019-07-21 13:50:42 -07:00 |
|
Chris Lu
|
644c5c27b4
|
refactoring
|
2019-07-21 13:50:24 -07:00 |
|
Chris Lu
|
33b3366043
|
volume: avoid compacting wait group
|
2019-07-21 13:49:59 -07:00 |
|
Chris Lu
|
fd16adbde1
|
stricter checking before writes
|
2019-07-21 11:21:30 -07:00 |
|
Chris Lu
|
5a135fef87
|
purge old code
|
2019-07-21 10:45:36 -07:00 |
|
Chris Lu
|
c54d9221b9
|
write requests also checks cookie if overwrites
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
|
2019-07-17 23:57:34 -07:00 |
|
Chris Lu
|
898d943b25
|
refactoring
|
2019-07-17 23:43:48 -07:00 |
|
Chris Lu
|
c33f423955
|
stop early if compaction fails
fix https://github.com/chrislusf/seaweedfs/issues/1015
|
2019-07-17 23:22:01 -07:00 |
|
Chris Lu
|
a7b1b23c58
|
fix wrong volume count
fix https://github.com/chrislusf/seaweedfs/issues/1013
|
2019-07-17 00:03:15 -07:00 |
|
Chris Lu
|
a3d1296ed9
|
go fmt
|
2019-06-27 12:18:59 -07:00 |
|
Chris Lu
|
ec75b2d761
|
volume: fix bug with 8000GB version if using in memory index
fix https://github.com/chrislusf/seaweedfs/issues/994
|
2019-06-27 12:18:45 -07:00 |
|
Chris Lu
|
6883f9e322
|
mark volume readonly before ec encoding
|
2019-06-26 23:02:22 -07:00 |
|
Chris Lu
|
c2960c7a86
|
defaulting file ttl to volume ttl
fix https://github.com/chrislusf/seaweedfs/issues/992
|
2019-06-26 11:18:23 -07:00 |
|
Chris Lu
|
a14bd31f5b
|
go fmt
|
2019-06-22 20:05:25 -07:00 |
|
Chris Lu
|
f88a8bda7b
|
ec deletion works
|
2019-06-21 01:14:10 -07:00 |
|
Chris Lu
|
e63317fb08
|
ec deletion code complete, not tested yet
|
2019-06-20 00:55:30 -07:00 |
|
Chris Lu
|
4cea8aefd0
|
add grpc VolumeEcBlobDelete
|
2019-06-20 00:17:11 -07:00 |
|
Chris Lu
|
856da7aae2
|
ec volume support deletes
|
2019-06-19 22:57:14 -07:00 |
|
Chris Lu
|
115558e5f5
|
adjust counters
|
2019-06-17 21:02:50 -07:00 |
|
Chris Lu
|
935639b908
|
metrics: disk size for <collection, normal/EC>
|
2019-06-16 21:56:41 -07:00 |
|
Chris Lu
|
0fdb1e705d
|
collect volume disk usage metrics
|
2019-06-16 02:44:20 -07:00 |
|
Chris Lu
|
289fd7eb39
|
count number of volumes and ec shards
|
2019-06-16 02:24:15 -07:00 |
|
Chris Lu
|
57092d69ac
|
go fmt
|
2019-06-10 21:33:32 -07:00 |
|
Chris Lu
|
d344e0a035
|
fix ec related bugs
|
2019-06-05 23:20:26 -07:00 |
|
Chris Lu
|
450f4733ce
|
report empty volume or ec shards
|
2019-06-05 13:32:33 -07:00 |
|
Chris Lu
|
ca8a2bb534
|
go fmt
|
2019-06-04 22:04:10 -07:00 |
|
Chris Lu
|
2215e81be7
|
ui add ec shard statuses
|
2019-06-04 21:52:37 -07:00 |
|
Chris Lu
|
0e52862586
|
avoid the test that depends on ordering in a different folder
|
2019-06-04 02:22:32 -07:00 |
|
Chris Lu
|
2f7710a75d
|
debug sporadic test error on travis
|
2019-06-04 02:09:06 -07:00 |
|
Chris Lu
|
11cffb3168
|
fix ec.rebuild bugs
|
2019-06-03 11:50:54 -07:00 |
|
Chris Lu
|
7e80b2b882
|
fix multiple bugs
|
2019-06-03 02:26:31 -07:00 |
|
Chris Lu
|
133b772fb5
|
destroy ec volume if it is empty
|
2019-06-01 01:51:28 -07:00 |
|
Chris Lu
|
ba18314aab
|
ec shard delete also check ec volumes, in addition to volumes
|
2019-06-01 01:41:22 -07:00 |
|
Chris Lu
|
f919d0235c
|
ec encode volumes quiet for a period of time
|
2019-05-31 23:41:17 -07:00 |
|
Chris Lu
|
de1a35acdd
|
conditionally fresh the shard locations
|
2019-05-31 02:22:12 -07:00 |
|
Chris Lu
|
47f1901843
|
ask for the ec volume version
|
2019-05-31 00:58:51 -07:00 |
|
Chris Lu
|
689930f092
|
forget shards that are broken
|
2019-05-31 00:19:13 -07:00 |
|
Chris Lu
|
aaab2c25f5
|
resolve concurrent modification problem
n and err can be modified concurrently
|
2019-05-30 13:59:07 -07:00 |
|
Chris Lu
|
40ca2f2903
|
add collection.delete
|
2019-05-30 09:47:54 -07:00 |
|
Chris Lu
|
5c6c757619
|
clean up test files
|
2019-05-29 00:34:30 -07:00 |
|
Chris Lu
|
5dd67f9acf
|
reading by recover from other shards
|
2019-05-28 23:48:39 -07:00 |
|
Chris Lu
|
3f9ecee40f
|
working with reading remote intervals
|
2019-05-28 21:29:07 -07:00 |
|
Chris Lu
|
302d9fbc6d
|
adjust error message for file not found
|
2019-05-28 14:27:21 -07:00 |
|
Chris Lu
|
4f76342cbc
|
WIP
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
|
2019-05-28 00:51:01 -07:00 |
|
Chris Lu
|
2858a954b3
|
read ec volume shards locations from master
|
2019-05-28 00:13:13 -07:00 |
|
Chris Lu
|
713596e781
|
caching ec shard locations
|
2019-05-27 22:54:58 -07:00 |
|
Chris Lu
|
3a8c1055a2
|
refactoring ecx to ecVolume
|
2019-05-27 22:00:36 -07:00 |
|
Chris Lu
|
217cde0a3b
|
refactoring
|
2019-05-27 21:40:51 -07:00 |
|
Chris Lu
|
03b9291e5d
|
volum server passes grpc option and master location to store
|
2019-05-27 21:22:23 -07:00 |
|
Chris Lu
|
b4b407e403
|
add grpc ec shard read
|
2019-05-27 11:59:03 -07:00 |
|
Chris Lu
|
a4f3d82c57
|
convert needle id to ec intervals to read from
|
2019-05-27 01:29:46 -07:00 |
|
Chris Lu
|
a463759edf
|
adjust error message
|
2019-05-26 01:14:42 -07:00 |
|
Chris Lu
|
af67d99ca4
|
incrementally update master ec shards state
|
2019-05-26 00:21:17 -07:00 |
|
Chris Lu
|
db94a41f9e
|
mount/unmount ec shards
|
2019-05-25 23:23:19 -07:00 |
|
Chris Lu
|
f0e6574d5e
|
allocate ec shards to volume servers
|
2019-05-25 02:02:44 -07:00 |
|
Chris Lu
|
6f4b09b6a4
|
pb shard info uses ShardBits instead one message for one shard
|
2019-05-24 13:28:44 -07:00 |
|
Chris Lu
|
228850d588
|
shard id starts from zero
|
2019-05-24 11:52:23 -07:00 |
|
Chris Lu
|
8a96445f40
|
register ec shards to each data node
|
2019-05-23 22:51:18 -07:00 |
|
Chris Lu
|
4659d80035
|
prepare to register ec shard info in master
|
2019-05-23 00:42:28 -07:00 |
|
Chris Lu
|
7180520889
|
ec shard info can be queried via VolumeList()
|
2019-05-22 22:44:28 -07:00 |
|
Chris Lu
|
17ac1290c0
|
volume: load ec shards during heartbeats to master
|
2019-05-21 22:41:20 -07:00 |
|
Chris Lu
|
fbbc74abb4
|
adds VolumeEcGenerateSlices, VolumeEcCopy
|
2019-05-20 00:53:17 -07:00 |
|
Chris Lu
|
693d6c0f1b
|
clean up tests
|
2019-05-19 21:16:01 -07:00 |
|
Chris Lu
|
6386a3174b
|
able to validate by randomly selected ec files
|
2019-05-19 14:24:33 -07:00 |
|
Chris Lu
|
7c2c60c376
|
add locating data inside the ec files
|
2019-05-19 03:01:58 -07:00 |
|
Chris Lu
|
87f63b9c08
|
generate ec01~ec14, generate ecx file with sorted needle values
|
2019-05-18 22:46:24 -07:00 |
|
Chris Lu
|
12dc6608f0
|
Merge branch 'master' into erasure_coding
|
2019-05-18 11:16:07 -07:00 |
|
Chris Lu
|
76cf57a0e1
|
volume: optimzation for leveldb
|
2019-05-17 17:34:03 -07:00 |
|
Chris Lu
|
8156958ee9
|
move function to make travis happy
|
2019-05-15 10:02:44 -07:00 |
|
Chris Lu
|
0a36f628c6
|
testing RS coding
|
2019-05-15 01:02:00 -07:00 |
|
Chris Lu
|
cf58fc0e63
|
refactor: extract out the write throttler
|
2019-05-06 13:56:08 -07:00 |
|
Chris Lu
|
46a675ecc2
|
align memory to avoid nil with atomic operations
|
2019-05-04 21:33:05 -07:00 |
|
Chris Lu
|
177f3add81
|
adjust memory layout to avoid possible nil pointer
|
2019-05-04 08:47:11 -07:00 |
|
Chris Lu
|
4b27f892e6
|
remove 128 bit needle id build option
fix https://github.com/chrislusf/seaweedfs/issues/949
|
2019-05-03 22:39:00 -07:00 |
|
Chris Lu
|
b335f81a4f
|
volume: add option to limit compaction speed
|
2019-05-03 17:22:39 -07:00 |
|
Wine93
|
32f93fb09e
|
style: go fmt for all
|
2019-04-30 03:22:19 +00:00 |
|
Chris Lu
|
9beea63960
|
memory alignment
fix https://github.com/chrislusf/seaweedfs/issues/939
|
2019-04-24 09:23:19 -07:00 |
|
Chris Lu
|
53e2e73de2
|
fix metrics
fix https://github.com/chrislusf/seaweedfs/issues/935
|
2019-04-22 12:32:10 -07:00 |
|
Chris Lu
|
440111a349
|
volume: support http status 304 for the same file id
|
2019-04-21 13:33:23 -07:00 |
|
Chris Lu
|
0302b9496c
|
volume: also check cookie for possible duplicates
|
2019-04-21 13:31:45 -07:00 |
|
Chris Lu
|
132921ad41
|
adjust log or error
|
2019-04-21 10:14:17 -07:00 |
|
Chris Lu
|
316bd27f75
|
add remove volumes with version info
|
2019-04-20 23:53:37 -07:00 |
|
Chris Lu
|
6fc1f53018
|
shell: add command volume.move
|
2019-04-20 11:35:23 -07:00 |
|
stlpmo-jn
|
2200ea9cb9
|
add function ParseFileIdFromString
|
2019-04-20 18:39:06 +08:00 |
|
Chris Lu
|
3b3651dea3
|
volume: atomic copying file, adds version and stopOffset
|
2019-04-19 12:29:49 -07:00 |
|
Chris Lu
|
0be2d51c96
|
read volume lastAppendAtNs when loading a volume
|
2019-04-19 01:56:38 -07:00 |
|
Chris Lu
|
ac2727853f
|
fix needle map entry size
|
2019-04-19 00:39:34 -07:00 |
|
Chris Lu
|
e5506152c0
|
refactoring
|
2019-04-18 21:43:36 -07:00 |
|
Chris Lu
|
33c92b819a
|
refactoring
|
2019-04-18 21:17:43 -07:00 |
|
Chris Lu
|
fa176fe80f
|
volume tailing chunks large files
|
2019-04-18 19:22:13 -07:00 |
|
Chris Lu
|
b142f9f1d5
|
go fmt
|
2019-04-18 00:19:18 -07:00 |
|
Chris Lu
|
b09e8dbb37
|
add VolumeStreamFollow, but not used yet
|
2019-04-18 00:18:29 -07:00 |
|
Chris Lu
|
13ad5c1966
|
refactoring
|
2019-04-17 22:04:49 -07:00 |
|
Chris Lu
|
f567777d64
|
volume: fix needle checking for deleted needles
|
2019-04-15 21:58:43 -07:00 |
|
Chris Lu
|
bd76ad0ff1
|
Revert "Revert "volume: close request body if any parsing error""
This reverts commit 359b7c1650 .
|
2019-04-15 21:43:28 -07:00 |
|
Chris Lu
|
359b7c1650
|
Revert "volume: close request body if any parsing error"
This reverts commit 5996656141 .
|
2019-04-15 21:35:12 -07:00 |
|
Chris Lu
|
5996656141
|
volume: close request body if any parsing error
|
2019-04-15 21:11:45 -07:00 |
|
Chris Lu
|
3e8a3a8fec
|
fix race detector found problems
|
2019-04-14 23:00:37 -07:00 |
|
Chris Lu
|
0e33272f56
|
small refactoring
|
2019-04-10 23:39:53 -07:00 |
|
stlpmo-jn
|
c1a0403da2
|
repair the error replications of the volume
|
2019-04-10 19:41:55 +08:00 |
|
Chris Lu
|
3555628ad5
|
weed volume: fix bug, remove boltdb, btree
|
2019-04-09 10:08:59 -07:00 |
|
Chris Lu
|
8a5ce16e96
|
weed volume: remove boltdb, btree options, add options for leveldb medium, large
|
2019-04-09 09:42:06 -07:00 |
|
Chris Lu
|
9924fa3b1a
|
weed volume: disable block cache on leveldb
saving 8MB for each volume, but at the cost of slower lookup
|
2019-04-09 00:24:32 -07:00 |
|
Chris Lu
|
784c5bb73a
|
add build option support 5-byte offset
|
2019-04-08 22:01:29 -07:00 |
|
Chris Lu
|
7a4b234ea1
|
divide offset into higher and lower sections
|
2019-04-08 21:44:06 -07:00 |
|
Chris Lu
|
000ee725fc
|
refactor Offset into a struct of bytes
|
2019-04-08 19:40:56 -07:00 |
|
Chris Lu
|
94302935b2
|
weed volume: only store compressed data if compression is effective
|
2019-04-06 00:10:52 -07:00 |
|
Chris Lu
|
7a14cdc90c
|
refactoring, go fmt
|
2019-03-25 23:18:40 -07:00 |
|
Chris Lu
|
19728fe3f6
|
remove deprecated code
|
2019-03-25 23:12:14 -07:00 |
|
Chris Lu
|
df95ce0b6c
|
weed backup: efficient delta backup
fix https://github.com/chrislusf/seaweedfs/issues/399
|
2019-03-25 23:01:53 -07:00 |
|
Chris Lu
|
70815e9124
|
WIP
|
2019-03-25 09:16:12 -07:00 |
|
Chris Lu
|
95e0520182
|
weed volume: add grpc operation to relicate a volume to local
|
2019-03-23 11:33:34 -07:00 |
|
Chris Lu
|
cea5c53bf7
|
fix length checking
fix https://github.com/chrislusf/seaweedfs/issues/890
|
2019-03-19 20:56:27 -07:00 |
|
Chris Lu
|
da871896c3
|
weed filer: set grpc port to port + 10000
|
2019-03-19 05:47:41 -07:00 |
|
Chris Lu
|
916b809c08
|
add error checking
|
2019-03-19 05:34:43 -07:00 |
|
Chris Lu
|
ece9d13312
|
volume info collect compact revision number
|
2019-03-18 09:32:21 -07:00 |
|
Chris Lu
|
44647a46c0
|
needle scanner read appendAtNs
|
2019-03-17 23:28:43 -07:00 |
|
Chris Lu
|
8b26d15740
|
refactoring: simplify function parameter
|
2019-03-17 22:32:01 -07:00 |
|
Chris Lu
|
2fcc88116e
|
go fmt
|
2019-03-17 20:27:36 -07:00 |
|
Chris Lu
|
aca653c08b
|
weed shell: list volumes
|
2019-03-17 20:27:08 -07:00 |
|
Chris Lu
|
95ef4513c8
|
print out memory size
|
2019-03-13 23:14:40 -07:00 |
|
Chris Lu
|
346541a101
|
print out per entry memory usage
|
2019-03-13 23:07:24 -07:00 |
|
chenwanli
|
fd27ed7755
|
Fix https://github.com/chrislusf/seaweedfs/issues/861
|
2019-02-26 17:12:39 +08:00 |
|
Chris Lu
|
77b9af531d
|
adding grpc mutual tls
|
2019-02-18 12:11:52 -08:00 |
|
Chris Lu
|
09471b46f9
|
fix related tests
|
2019-01-26 00:10:29 -06:00 |
|
Chris Lu
|
5668ed798d
|
memory needle map mark size to be TombstoneFileSize
fix https://github.com/chrislusf/seaweedfs/issues/850
|
2019-01-26 00:05:44 -06:00 |
|
bingoohuang
|
cf0fc797ac
|
use tempFile.Truncate to reset file size to speedup test
|
2019-01-23 16:24:51 +08:00 |
|
bingoohuang
|
ab6be025d7
|
go fmt and fix some typo
|
2019-01-17 09:17:19 +08:00 |
|
bingoohuang
|
6ddfaf33cb
|
extract VolumeFileScanner for ScanVolumeFile
|
2019-01-16 17:48:59 +08:00 |
|
bingoohuang
|
27093bc2e5
|
fix needle Append return offset to avoid uint32 overflow
|
2019-01-16 16:48:03 +08:00 |
|
Chris Lu
|
badd8fdf8f
|
shortcut for empty file
|
2019-01-08 09:03:28 -08:00 |
|
Chris Lu
|
6b5d6bb5a6
|
adjust error names
|
2019-01-05 19:52:38 -08:00 |
|
Chris Lu
|
9383c91eb1
|
wait to read again if the volume is compacting
|
2019-01-05 19:52:17 -08:00 |
|
Chris Lu
|
d5ebfd5b0c
|
fix s3cmd put
1. fix volume put data size
|
2019-01-02 12:57:33 -08:00 |
|
Chris Lu
|
1478d7ea21
|
reduce file seek when writing
|
2018-12-31 15:08:32 -08:00 |
|
Chris Lu
|
0c932d1738
|
add volume delete grpc API
fix https://github.com/chrislusf/seaweedfs/issues/808
|
2018-12-29 00:03:30 -08:00 |
|
Chris Lu
|
c043fd17cb
|
detect more gzippable content
|
2018-12-22 15:05:31 -08:00 |
|
Chris Lu
|
852ee21835
|
avoid .gz auto decompression
|
2018-12-22 13:58:16 -08:00 |
|
Chris Lu
|
be946c9e54
|
filer store original data size instead of data size after gzip
|
2018-12-22 13:11:07 -08:00 |
|
Chris Lu
|
141d302492
|
fix binarySearchCompactSection
|
2018-12-22 11:52:58 -08:00 |
|
Chris Lu
|
5333f2984a
|
streaming updates for large entries or large index file
fix https://github.com/chrislusf/seaweedfs/issues/801
|
2018-12-22 11:10:08 -08:00 |
|
Chris Lu
|
36d13355bb
|
compact map fix bug when iterating
|
2018-12-22 11:05:29 -08:00 |
|
Chris Lu
|
f64dc7465b
|
remove index file
fix https://github.com/chrislusf/seaweedfs/issues/793
|
2018-12-17 20:33:32 -08:00 |
|
Chris Lu
|
42cb9b76c4
|
reduce needle map memory usage by 25%
|
2018-12-15 05:55:56 -08:00 |
|
Chris Lu
|
168f461761
|
fix MaxUint8
|
2018-12-10 20:57:51 -08:00 |
|
Chris Lu
|
8f78a069c6
|
fix possible name overflow when len(name)>255
|
2018-12-10 20:57:01 -08:00 |
|
Chris Lu
|
4a490fe35a
|
go fmt
|
2018-12-09 01:27:25 -08:00 |
|
Chris Lu
|
5741134ca0
|
change overflow from map to slice
|
2018-12-09 01:27:11 -08:00 |
|
Chris Lu
|
d9189c857c
|
remove unused file
|
2018-12-09 00:30:53 -08:00 |
|
Chris Lu
|
df4b9df142
|
add memory usage report
|
2018-12-09 00:15:23 -08:00 |
|
Chris Lu
|
ec5a305624
|
refactor
|
2018-12-09 00:12:37 -08:00 |
|
Chris Lu
|
9d3be33e5c
|
optimize a little bit
|
2018-12-08 21:45:14 -08:00 |
|
Chris Lu
|
86dd933596
|
go fmt
|
2018-11-23 00:26:15 -08:00 |
|
Chris Lu
|
2697d6a4fb
|
fix deleting bucket
|
2018-11-08 22:15:21 -08:00 |
|
Chris Lu
|
a4ceb051a7
|
use MD5 for ETag to be consistent with Amazon S3
|
2018-11-08 21:41:02 -08:00 |
|
Chris Lu
|
f9dbaf6292
|
delete file by name instead of referencing the dataFile pointer
|
2018-11-05 08:53:38 -08:00 |
|
alex8224
|
8d3f9c0b75
|
fix invalid point when delete collection
|
2018-11-05 19:53:55 +08:00 |
|
Chris Lu
|
f050b22d6c
|
close file in ScanVolumeFile()
fix https://github.com/chrislusf/seaweedfs/issues/761
|
2018-11-04 00:28:24 -07:00 |
|
Chris Lu
|
496a7398d1
|
adjust logging
|
2018-10-18 20:47:30 -07:00 |
|
Chris Lu
|
eec951cad2
|
migrate volume sync to gRpc
|
2018-10-15 21:44:41 -07:00 |
|
Chris Lu
|
66a353dcb5
|
remove volume server /admin/volume/delete
|
2018-10-15 01:26:49 -07:00 |
|
Chris Lu
|
fda771c83f
|
migrate volume sync status to grpc API on volume server
|
2018-10-15 01:19:15 -07:00 |
|
Chris Lu
|
b423bb9e2d
|
migrate assign volume to grpc API on volume server
|
2018-10-15 00:40:46 -07:00 |
|
Chris Lu
|
333709657c
|
adjust log level
|
2018-10-15 00:40:12 -07:00 |
|
Chris Lu
|
b1daede91b
|
move volume vacuum to gRpc
|
2018-10-14 23:12:43 -07:00 |
|
Chris Lu
|
a362261972
|
avoid possible racing issue if leveldb or boltdb is used.
remove the to-be-generated files during commit
fix https://github.com/chrislusf/seaweedfs/issues/738
|
2018-09-26 18:45:51 -07:00 |
|
Chris Lu
|
164091c269
|
add s3 multipart upload
|
2018-09-09 16:25:43 -07:00 |
|
Chris Lu
|
9b3bf0e46c
|
fix "weed backup" rerunning
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
|
2018-09-09 02:48:58 -07:00 |
|
Chris Lu
|
bc8d34143d
|
Revert "fix "weed backup" rerunning"
This reverts commit 1937be2bb2 .
|
2018-09-09 02:38:30 -07:00 |
|
Chris Lu
|
1937be2bb2
|
fix "weed backup" rerunning
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
|
2018-09-09 02:31:12 -07:00 |
|
Chris Lu
|
76cbe8bf33
|
instant notification of new volumes added or deleted
|
2018-08-24 01:26:56 -07:00 |
|
Chris Lu
|
ac793a3c5a
|
clean remove all left over files when deleting a collection
|
2018-08-23 23:33:16 -07:00 |
|
Chris Lu
|
d361a1ddf5
|
fix id String() instead
|
2018-08-23 22:44:30 -07:00 |
|
Chris Lu
|
694f93de80
|
parse needle id correctly
fix https://github.com/chrislusf/seaweedfs/issues/713
|
2018-08-23 20:48:43 -07:00 |
|
Chris Lu
|
1f030af06f
|
adjusting startup logging
|
2018-08-22 13:59:04 -07:00 |
|
Chris Lu
|
75d63db60d
|
randomize raft server startup
also some go fmt
|
2018-08-12 14:27:14 -07:00 |
|
Chris Lu
|
495a776671
|
support 128 bit NeedleId
|
2018-07-31 23:25:26 -07:00 |
|
Chris Lu
|
10bb76fe6b
|
go fmt
|
2018-07-27 02:10:42 -07:00 |
|
Chris Lu
|
4d322df95e
|
avoid reparsing the multipart form
|
2018-07-27 02:10:10 -07:00 |
|
Chris Lu
|
57ded4fd80
|
speed up leveldb bolddb loading
|
2018-07-24 22:17:56 -07:00 |
|
Chris Lu
|
ea8ecf35f4
|
set needle AppendAtNs value during writes
|
2018-07-24 02:44:33 -07:00 |
|
Chris Lu
|
77fadab560
|
go fmt
|
2018-07-24 01:38:08 -07:00 |
|
Chris Lu
|
4f317c7e3d
|
working version3
|
2018-07-24 01:36:04 -07:00 |
|
Chris Lu
|
852af28f91
|
needle priting format
|
2018-07-21 17:41:21 -07:00 |
|
Chris Lu
|
7e2031b18f
|
go fmt
|
2018-07-21 17:39:10 -07:00 |
|
Chris Lu
|
c98df05ed0
|
support PUT
|
2018-07-21 15:58:48 -07:00 |
|
Chris Lu
|
feb8eeb830
|
refactor: separate multipart parsing file
|
2018-07-21 12:01:35 -07:00 |
|
Chris Lu
|
3edfe1d28f
|
extend export command to show tombstone + change output format to CSV
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
|
2018-07-14 20:51:17 -07:00 |
|
Chris Lu
|
a51aa4c586
|
avoid extra log messages
fix https://github.com/chrislusf/seaweedfs/issues/689
|
2018-07-13 22:07:32 -07:00 |
|
Chris Lu
|
8a12fd9ec4
|
add error checking when loading dat file
|
2018-07-09 23:31:25 -07:00 |
|
Chris Lu
|
d0982cafa7
|
refactoring
|
2018-07-09 00:22:50 -07:00 |
|
Chris Lu
|
5b1fd374be
|
fix all tests
|
2018-07-08 02:39:04 -07:00 |
|
Chris Lu
|
d4d7ced922
|
refactoring: add type for needle id, offset
later the type size can possibly be adjusted
|
2018-07-08 02:28:04 -07:00 |
|
Chris Lu
|
5bfb72d058
|
faster loading boltdb or leveldb needle map metrics by bloomfilter
avoid btree
|
2018-07-07 00:51:17 -07:00 |
|
Chris Lu
|
aba1fe01b3
|
use constant NeedleIndexSize instead of the number 16
|
2018-07-07 00:50:14 -07:00 |
|
Chris Lu
|
defcd4edc5
|
adjust log level
|
2018-07-07 00:48:58 -07:00 |
|
Chris Lu
|
9bcaa65574
|
avoid divided by 0
fix https://github.com/chrislusf/seaweedfs/issues/650
|
2018-06-25 01:20:15 -07:00 |
|
Chris Lu
|
233a9a9b6c
|
fix log error
|
2018-06-24 16:12:41 -07:00 |
|
Chris Lu
|
49f4216280
|
fix compilation error
|
2018-06-24 15:37:54 -07:00 |
|
Chris Lu
|
7f6500ae07
|
super block reserves a couple of bits for future extension
|
2018-06-24 15:23:03 -07:00 |
|
Chris Lu
|
f339e9c284
|
add support for extra super block info
|
2018-06-24 15:19:57 -07:00 |
|
Chris Lu
|
9ba335a7c4
|
prepare for flexible super block
|
2018-06-24 11:37:08 -07:00 |
|
Chris Lu
|
3a6f9c9bcf
|
increase test size
|
2018-06-23 18:30:13 -07:00 |
|
Chris Lu
|
7f7e4e9885
|
fix error for deleted files during compaction
deletion during commit may cause trouble when make up the difference during commitCompact()
|
2018-06-23 18:24:59 -07:00 |
|
Chris Lu
|
3699996985
|
add compaction test
|
2018-06-23 16:48:19 -07:00 |
|
Chris Lu
|
0467195f07
|
fix file closing on windows
|
2018-06-13 00:36:51 -07:00 |
|
Chris Lu
|
eaf000eff1
|
fix error on go tip
|
2018-06-11 23:39:42 -07:00 |
|
Chris Lu
|
43e3f5724c
|
use fixed list of masters in both filer and volume servers
|
2018-06-01 00:39:39 -07:00 |
|
Chris Lu
|
458ada173e
|
go fmt
|
2018-05-27 11:52:26 -07:00 |
|
Chris Lu
|
1675243f29
|
maybe speed up a little when loading index
|
2018-05-23 02:38:11 -07:00 |
|
Chris Lu
|
69b9d8c3c2
|
fix boltdb variable usage
|
2018-05-22 10:18:09 -07:00 |
|
Chris Lu
|
942c2cbd7b
|
mv pb to master_pb
|
2018-05-09 23:11:54 -07:00 |
|
Chris Lu
|
57c5405657
|
add todo
|
2018-05-09 04:18:31 -07:00 |
|
Chris Lu
|
6db1c8bafb
|
Revert "adjust the needle mapper"
This reverts commit e2c7be3a2e .
|
2018-05-09 03:57:07 -07:00 |
|
Chris Lu
|
e2c7be3a2e
|
adjust the needle mapper
as discussed on
https://github.com/chrislusf/seaweedfs/pull/642#issuecomment-387606960
|
2018-05-09 03:50:14 -07:00 |
|
WenLong LI
|
b9068c1304
|
use volumes variable
fix leveldb mount bug
|
2018-05-03 12:13:53 +08:00 |
|
liwenlong05
|
02a1597642
|
fix leveldb mount bug
|
2018-05-03 11:20:38 +08:00 |
|
mtolman
|
e4b1e5c3c8
|
Fix for issue #635
Fix for the following issue on master: https://github.com/chrislusf/seaweedfs/issues/635
|
2018-04-17 13:26:48 -06:00 |
|
Yang Pengcheng
|
a2112e1be0
|
fix ldb dir deleting
|
2017-08-30 22:55:03 +08:00 |
|
Chris Lu
|
58344980e4
|
preallocate disk space during compaction also, add cleanup for failed compaction
|
2017-08-29 23:59:53 -07:00 |
|
Chris Lu
|
5047bdb4a2
|
skip bytes cache
|
2017-06-03 11:44:24 -07:00 |
|
Chris Lu
|
82c3ccc8dd
|
add btree for volume index
|
2017-05-26 22:51:25 -07:00 |
|
Chris Lu
|
bd97cbc523
|
skip busy reporting back to master
possible fix for https://github.com/chrislusf/seaweedfs/issues/499
|
2017-05-23 21:12:03 -07:00 |
|
Chris Lu
|
043b7a7c65
|
add locking to access l.volumes
|
2017-01-20 10:18:43 -08:00 |
|
brstgt
|
0656838fe5
|
Don't return actual file size from CheckVolumeDataIntegrity, it will be 0 if last needle is a tombstone, so it's not reliable anyway
|
2017-01-20 16:37:45 +01:00 |
|
brstgt
|
4fb5bb09b6
|
Remove obsolete property Volume.dataFileSize
|
2017-01-20 16:31:11 +01:00 |
|
brstgt
|
e074a54a20
|
Delete volumes online without restarting volume server
|
2017-01-20 13:02:37 +01:00 |
|
brstgt
|
492f93416d
|
Mount and unmount volumes online without restarting volume server
|
2017-01-20 12:49:20 +01:00 |
|
brstgt
|
18b3afc97a
|
Log volumeId if dataFileSize != actual data file size - improve log types
|
2017-01-20 12:22:25 +01:00 |
|
brstgt
|
0c898df430
|
Log volumeId if dataFileSize != actual data file size
|
2017-01-20 11:52:23 +01:00 |
|
Chris Lu
|
5b0f5f456e
|
volume server get notified of leader change
|
2017-01-18 09:34:27 -08:00 |
|
Chris Lu
|
59022b6fe0
|
fix byte counter on loading index file
fix https://github.com/chrislusf/seaweedfs/issues/441
|
2017-01-18 08:40:39 -08:00 |
|
Chris Lu
|
3065506b38
|
volume servers always connect to the master leader
|
2017-01-13 10:20:40 -08:00 |
|
Chris Lu
|
e46c3415f7
|
gRpc for master~volume heartbeat
|
2017-01-10 01:01:12 -08:00 |
|
Chris Lu
|
4beaaa0650
|
simplify the gzip optimization logic
fix https://github.com/chrislusf/seaweedfs/issues/436
|
2017-01-09 19:31:58 -08:00 |
|
Chris Lu
|
ed44f12f6d
|
support Fallocate on linux
|
2017-01-08 11:01:46 -08:00 |
|
Chris Lu
|
7b6837cbc2
|
move back the section
|
2017-01-08 10:35:47 -08:00 |
|
sparklxb
|
da9b672d1b
|
support additional header name-value pairs
|
2017-01-08 22:34:42 +08:00 |
|
sparklxb
|
86a7c56275
|
support additional header name-value pairs
|
2017-01-08 09:16:40 +08:00 |
|
Chris Lu
|
13e7069eb9
|
keep track of total data file size
|
2017-01-06 10:22:20 -08:00 |
|
lixianbin
|
d96d0a87cf
|
fix bug: upload big .gz file more than maxMB
|
2017-01-04 11:23:40 +08:00 |
|
Chris Lu
|
dcaf1796fe
|
add option to enable caching
|
2016-11-16 07:09:57 -08:00 |
|
Chris Lu
|
f54f530ada
|
adjust verbose logging
|
2016-11-13 14:07:51 -08:00 |
|
霍晓栋
|
b9f385bd66
|
refactor concurrent loading code logic
|
2016-11-13 13:24:52 +08:00 |
|
霍晓栋
|
096ffa9744
|
concurrent loading volume
|
2016-11-11 11:53:22 +08:00 |
|
Chris Lu
|
36f9633223
|
add locks for location.volumes
fix https://github.com/chrislusf/seaweedfs/issues/392
|
2016-11-06 20:55:22 -08:00 |
|
霍晓栋
|
7d73bbb073
|
comment UT case
|
2016-10-07 16:40:51 +08:00 |
|
霍晓栋
|
7d382ba5fe
|
comment UT case
|
2016-10-07 16:34:22 +08:00 |
|
霍晓栋
|
ce1f7ab662
|
makediff func with UT case
|
2016-10-07 16:22:24 +08:00 |
|
霍晓栋
|
ed848425c7
|
supplemental data between compacting and commit compacting
|
2016-09-29 13:57:23 +08:00 |
|
Chris Lu
|
dffad65f2f
|
fix compilation
|
2016-09-26 22:30:44 -07:00 |
|
Chris Lu
|
7e29218327
|
add a template for makeupDiff
|
2016-09-26 22:26:41 -07:00 |
|
Chris Lu
|
1bc041b46d
|
add a new way to manually compact corrupted volume
fix https://github.com/chrislusf/seaweedfs/issues/371
|
2016-09-22 20:31:20 -07:00 |
|
Chris Lu
|
01cbd5cb58
|
lock
fix https://github.com/chrislusf/seaweedfs/issues/367
|
2016-09-08 09:50:31 -07:00 |
|
霍晓栋
|
b9b3651a98
|
deleted index entry could not point to deleted needle
|
2016-07-25 14:56:58 +08:00 |
|
霍晓栋
|
09bd3d015d
|
deleted index entry could not point to deleted needle
|
2016-07-25 14:54:40 +08:00 |
|
霍晓栋
|
78678f4bcb
|
deleted needle does not need checksum verification
|
2016-07-25 11:40:35 +08:00 |
|
Chris Lu
|
d981eb282f
|
refactor volume_read_write.go out of volume.go
|
2016-07-03 00:10:27 -07:00 |
|
Chris Lu
|
576573711a
|
refactoring
|
2016-07-02 23:58:23 -07:00 |
|
Chris Lu
|
3d8df0f709
|
refactor volume_loading.go out of volume.go
|
2016-07-02 23:56:49 -07:00 |
|