j.laycock
d5f5acb734
limit locking physical memory to 80% of max physical memory
2019-09-20 12:44:29 +01:00
j.laycock
476140fd6b
minor change to setProcessWorkingSetSize function
2019-09-19 11:59:00 +01:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
joeslay
d8c34b032f
Merge pull request #6 from chrislusf/master
...
merge seaweed master
2019-09-12 11:44:09 +01:00
Chris Lu
5e9c65469e
volume: skip readonly checking when Destroy()
...
fix https://github.com/chrislusf/seaweedfs/issues/1063
2019-09-11 09:45:09 -07:00
j.laycock
c1a928887a
Add some comments/documentation
2019-09-11 10:42:37 +01:00
joeslay
9ea9a5e97a
Merge pull request #5 from chrislusf/master
...
merge seaweed master
2019-09-11 10:25:27 +01:00
j.laycock
15a4c91c28
Remove Finalizer
2019-09-10 17:05:54 +01:00
j.laycock
9c9dff7386
Virtual lock memory pages and Set Process Working set size to hint to windows harder not to write pages to disk as much, add finalize function, minor renaming
2019-09-10 16:18:51 +01:00
Chris Lu
02710f9624
refactoring: remove unused functions
2019-09-09 10:03:41 -07:00
Chris Lu
0f861d23a2
avoid nil needle map
...
fix https://github.com/chrislusf/seaweedfs/issues/1061
2019-09-09 09:48:08 -07:00
j.laycock
63acc8c972
rename mem_buffer to mBuffer
2019-09-05 13:46:29 +01:00
j.laycock
44ae041e80
rename more memorymapped variables to memoryMapMaxSizeMB
2019-09-05 13:42:23 +01:00
j.laycock
5885ab67b3
rename volume property MemoryMap to MemoryMapMaxSizeMB
2019-09-05 11:56:48 +01:00
j.laycock
5a6dfb05fc
Only use truncate flag when creating non memory map files, rename and use memory map size for reserving max file size
2019-09-05 11:46:12 +01:00
j.laycock
f6146d6ad2
use 0 instead of false for max memory flag
2019-09-05 11:43:18 +01:00
j.laycock
4257582db5
Allocate in 16MB chunks, make creation of memory maps always aligned to 16MB chunks
2019-09-05 11:15:01 +01:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
j.laycock
cc756ddd79
Rename End_of_file variable, implement platform indepenent memory_map.go
2019-09-04 14:24:17 +01:00
j.laycock
de5a7e60a2
remove temp hack
2019-09-03 18:20:09 +01:00
j.laycock
cbd0a98fa1
Add InMemory to Volume Grow structure
2019-09-03 18:19:02 +01:00
j.laycock
1f01eb78e8
Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct
2019-09-03 17:00:59 +01:00
Tom Maxwell
d637d86d22
Changes to try and pass the URL parameters through - in memory flag not working still
2019-09-03 15:41:28 +01:00
j.laycock
9a459d984b
Do not vacuum memory mapped files
2019-09-03 15:05:43 +01:00
j.laycock
0123c7a898
Change to max 2GB for now
2019-09-02 17:46:22 +01:00
joeslay
e91139348e
Merge pull request #2 from chrislusf/master
...
merge seaweed master
2019-09-02 11:39:40 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
j.laycock
1e62a2b233
Fix memory not being unmapped
2019-08-30 16:55:33 +01:00
j.laycock
840ccdc35d
Refactor to pass memory maps by reference instead of value, fix memory maps not being created properly or written to properly
2019-08-30 12:30:00 +01:00
j.laycock
dc50701e7d
Overload createFile function, expose File in memory_map struct, delete from memory_map map on delete
2019-08-30 12:30:00 +01:00
j.laycock
523f3a12b3
use Os.File/uintptr instead of windows.Handle in memory map struct
2019-08-30 12:30:00 +01:00
j.laycock
0e5d3b1a70
Move memory_map_windows to new folder, intercept most of the read and write calls to volume dat files
2019-08-30 12:30:00 +01:00
Chris Lu
58168a8c52
volume: truncate dat files when creating
2019-08-26 13:01:57 -07:00
joeslay
36bde84000
Merge pull request #1 from chrislusf/master
...
merge changes from chrislusf/seaweed master
2019-08-23 17:03:37 +01:00
j.laycock
1e89e719b3
Add windows memory map functions to storage package
2019-08-23 16:19:00 +01:00
xushuxun
9af9f36b89
volume: read dat files' last modified time no matter dat files are catWrite or readonly
2019-08-21 11:26:01 +08:00
Chris Lu
f9c1cd7370
avoid dead lock from dataFileAccessLock
...
fix https://github.com/chrislusf/seaweedfs/issues/1043
2019-08-15 09:24:54 -07:00
Chris Lu
d829df4f59
volume: protect against nil needle map
...
fix @mastak reported nil problem in https://github.com/chrislusf/seaweedfs/issues/1037
2019-08-14 01:08:01 -07:00
Chris Lu
e40634e6b4
volume: fail the volume deletion if compaction is in progress
...
fix https://github.com/chrislusf/seaweedfs/issues/1035
2019-08-12 00:53:50 -07:00
Chris Lu
f3b99cbfe5
fix wrong alignment fix
2019-07-21 13:50:42 -07:00
Chris Lu
644c5c27b4
refactoring
2019-07-21 13:50:24 -07:00
Chris Lu
33b3366043
volume: avoid compacting wait group
2019-07-21 13:49:59 -07:00
Chris Lu
fd16adbde1
stricter checking before writes
2019-07-21 11:21:30 -07:00
Chris Lu
5a135fef87
purge old code
2019-07-21 10:45:36 -07:00
Chris Lu
c54d9221b9
write requests also checks cookie if overwrites
...
protect against edge cases, avoid https://github.com/chrislusf/seaweedfs/issues/1014
2019-07-17 23:57:34 -07:00
Chris Lu
898d943b25
refactoring
2019-07-17 23:43:48 -07:00
Chris Lu
c33f423955
stop early if compaction fails
...
fix https://github.com/chrislusf/seaweedfs/issues/1015
2019-07-17 23:22:01 -07:00
Chris Lu
a7b1b23c58
fix wrong volume count
...
fix https://github.com/chrislusf/seaweedfs/issues/1013
2019-07-17 00:03:15 -07:00
Chris Lu
a3d1296ed9
go fmt
2019-06-27 12:18:59 -07:00
Chris Lu
ec75b2d761
volume: fix bug with 8000GB version if using in memory index
...
fix https://github.com/chrislusf/seaweedfs/issues/994
2019-06-27 12:18:45 -07:00
Chris Lu
6883f9e322
mark volume readonly before ec encoding
2019-06-26 23:02:22 -07:00
Chris Lu
c2960c7a86
defaulting file ttl to volume ttl
...
fix https://github.com/chrislusf/seaweedfs/issues/992
2019-06-26 11:18:23 -07:00
Chris Lu
a14bd31f5b
go fmt
2019-06-22 20:05:25 -07:00
Chris Lu
f88a8bda7b
ec deletion works
2019-06-21 01:14:10 -07:00
Chris Lu
e63317fb08
ec deletion code complete, not tested yet
2019-06-20 00:55:30 -07:00
Chris Lu
4cea8aefd0
add grpc VolumeEcBlobDelete
2019-06-20 00:17:11 -07:00
Chris Lu
856da7aae2
ec volume support deletes
2019-06-19 22:57:14 -07:00
Chris Lu
115558e5f5
adjust counters
2019-06-17 21:02:50 -07:00
Chris Lu
935639b908
metrics: disk size for <collection, normal/EC>
2019-06-16 21:56:41 -07:00
Chris Lu
0fdb1e705d
collect volume disk usage metrics
2019-06-16 02:44:20 -07:00
Chris Lu
289fd7eb39
count number of volumes and ec shards
2019-06-16 02:24:15 -07:00
Chris Lu
57092d69ac
go fmt
2019-06-10 21:33:32 -07:00
Chris Lu
d344e0a035
fix ec related bugs
2019-06-05 23:20:26 -07:00
Chris Lu
450f4733ce
report empty volume or ec shards
2019-06-05 13:32:33 -07:00
Chris Lu
ca8a2bb534
go fmt
2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7
ui add ec shard statuses
2019-06-04 21:52:37 -07:00
Chris Lu
0e52862586
avoid the test that depends on ordering in a different folder
2019-06-04 02:22:32 -07:00
Chris Lu
2f7710a75d
debug sporadic test error on travis
2019-06-04 02:09:06 -07:00
Chris Lu
11cffb3168
fix ec.rebuild bugs
2019-06-03 11:50:54 -07:00
Chris Lu
7e80b2b882
fix multiple bugs
2019-06-03 02:26:31 -07:00
Chris Lu
133b772fb5
destroy ec volume if it is empty
2019-06-01 01:51:28 -07:00
Chris Lu
ba18314aab
ec shard delete also check ec volumes, in addition to volumes
2019-06-01 01:41:22 -07:00
Chris Lu
f919d0235c
ec encode volumes quiet for a period of time
2019-05-31 23:41:17 -07:00
Chris Lu
de1a35acdd
conditionally fresh the shard locations
2019-05-31 02:22:12 -07:00
Chris Lu
47f1901843
ask for the ec volume version
2019-05-31 00:58:51 -07:00
Chris Lu
689930f092
forget shards that are broken
2019-05-31 00:19:13 -07:00
Chris Lu
aaab2c25f5
resolve concurrent modification problem
...
n and err can be modified concurrently
2019-05-30 13:59:07 -07:00
Chris Lu
40ca2f2903
add collection.delete
2019-05-30 09:47:54 -07:00
Chris Lu
5c6c757619
clean up test files
2019-05-29 00:34:30 -07:00
Chris Lu
5dd67f9acf
reading by recover from other shards
2019-05-28 23:48:39 -07:00
Chris Lu
3f9ecee40f
working with reading remote intervals
2019-05-28 21:29:07 -07:00
Chris Lu
302d9fbc6d
adjust error message for file not found
2019-05-28 14:27:21 -07:00
Chris Lu
4f76342cbc
WIP
...
no errors, but not returning file content
* the interval needs to use actual file zie
* need to read the actual version instead of the current version
2019-05-28 00:51:01 -07:00
Chris Lu
2858a954b3
read ec volume shards locations from master
2019-05-28 00:13:13 -07:00
Chris Lu
713596e781
caching ec shard locations
2019-05-27 22:54:58 -07:00
Chris Lu
3a8c1055a2
refactoring ecx to ecVolume
2019-05-27 22:00:36 -07:00
Chris Lu
217cde0a3b
refactoring
2019-05-27 21:40:51 -07:00
Chris Lu
03b9291e5d
volum server passes grpc option and master location to store
2019-05-27 21:22:23 -07:00
Chris Lu
b4b407e403
add grpc ec shard read
2019-05-27 11:59:03 -07:00
Chris Lu
a4f3d82c57
convert needle id to ec intervals to read from
2019-05-27 01:29:46 -07:00
Chris Lu
a463759edf
adjust error message
2019-05-26 01:14:42 -07:00
Chris Lu
af67d99ca4
incrementally update master ec shards state
2019-05-26 00:21:17 -07:00
Chris Lu
db94a41f9e
mount/unmount ec shards
2019-05-25 23:23:19 -07:00
Chris Lu
f0e6574d5e
allocate ec shards to volume servers
2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4
pb shard info uses ShardBits instead one message for one shard
2019-05-24 13:28:44 -07:00
Chris Lu
228850d588
shard id starts from zero
2019-05-24 11:52:23 -07:00
Chris Lu
8a96445f40
register ec shards to each data node
2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035
prepare to register ec shard info in master
2019-05-23 00:42:28 -07:00
Chris Lu
7180520889
ec shard info can be queried via VolumeList()
2019-05-22 22:44:28 -07:00
Chris Lu
17ac1290c0
volume: load ec shards during heartbeats to master
2019-05-21 22:41:20 -07:00
Chris Lu
fbbc74abb4
adds VolumeEcGenerateSlices, VolumeEcCopy
2019-05-20 00:53:17 -07:00
Chris Lu
693d6c0f1b
clean up tests
2019-05-19 21:16:01 -07:00
Chris Lu
6386a3174b
able to validate by randomly selected ec files
2019-05-19 14:24:33 -07:00
Chris Lu
7c2c60c376
add locating data inside the ec files
2019-05-19 03:01:58 -07:00
Chris Lu
87f63b9c08
generate ec01~ec14, generate ecx file with sorted needle values
2019-05-18 22:46:24 -07:00
Chris Lu
12dc6608f0
Merge branch 'master' into erasure_coding
2019-05-18 11:16:07 -07:00
Chris Lu
76cf57a0e1
volume: optimzation for leveldb
2019-05-17 17:34:03 -07:00
Chris Lu
8156958ee9
move function to make travis happy
2019-05-15 10:02:44 -07:00
Chris Lu
0a36f628c6
testing RS coding
2019-05-15 01:02:00 -07:00
Chris Lu
cf58fc0e63
refactor: extract out the write throttler
2019-05-06 13:56:08 -07:00
Chris Lu
46a675ecc2
align memory to avoid nil with atomic operations
2019-05-04 21:33:05 -07:00
Chris Lu
177f3add81
adjust memory layout to avoid possible nil pointer
2019-05-04 08:47:11 -07:00
Chris Lu
4b27f892e6
remove 128 bit needle id build option
...
fix https://github.com/chrislusf/seaweedfs/issues/949
2019-05-03 22:39:00 -07:00
Chris Lu
b335f81a4f
volume: add option to limit compaction speed
2019-05-03 17:22:39 -07:00
Wine93
32f93fb09e
style: go fmt for all
2019-04-30 03:22:19 +00:00
Chris Lu
9beea63960
memory alignment
...
fix https://github.com/chrislusf/seaweedfs/issues/939
2019-04-24 09:23:19 -07:00
Chris Lu
53e2e73de2
fix metrics
...
fix https://github.com/chrislusf/seaweedfs/issues/935
2019-04-22 12:32:10 -07:00
Chris Lu
440111a349
volume: support http status 304 for the same file id
2019-04-21 13:33:23 -07:00
Chris Lu
0302b9496c
volume: also check cookie for possible duplicates
2019-04-21 13:31:45 -07:00
Chris Lu
132921ad41
adjust log or error
2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75
add remove volumes with version info
2019-04-20 23:53:37 -07:00
Chris Lu
6fc1f53018
shell: add command volume.move
2019-04-20 11:35:23 -07:00
stlpmo-jn
2200ea9cb9
add function ParseFileIdFromString
2019-04-20 18:39:06 +08:00
Chris Lu
3b3651dea3
volume: atomic copying file, adds version and stopOffset
2019-04-19 12:29:49 -07:00
Chris Lu
0be2d51c96
read volume lastAppendAtNs when loading a volume
2019-04-19 01:56:38 -07:00
Chris Lu
ac2727853f
fix needle map entry size
2019-04-19 00:39:34 -07:00
Chris Lu
e5506152c0
refactoring
2019-04-18 21:43:36 -07:00
Chris Lu
33c92b819a
refactoring
2019-04-18 21:17:43 -07:00
Chris Lu
fa176fe80f
volume tailing chunks large files
2019-04-18 19:22:13 -07:00
Chris Lu
b142f9f1d5
go fmt
2019-04-18 00:19:18 -07:00
Chris Lu
b09e8dbb37
add VolumeStreamFollow, but not used yet
2019-04-18 00:18:29 -07:00
Chris Lu
13ad5c1966
refactoring
2019-04-17 22:04:49 -07:00
Chris Lu
f567777d64
volume: fix needle checking for deleted needles
2019-04-15 21:58:43 -07:00
Chris Lu
bd76ad0ff1
Revert "Revert "volume: close request body if any parsing error""
...
This reverts commit 359b7c1650
.
2019-04-15 21:43:28 -07:00
Chris Lu
359b7c1650
Revert "volume: close request body if any parsing error"
...
This reverts commit 5996656141
.
2019-04-15 21:35:12 -07:00
Chris Lu
5996656141
volume: close request body if any parsing error
2019-04-15 21:11:45 -07:00
Chris Lu
3e8a3a8fec
fix race detector found problems
2019-04-14 23:00:37 -07:00
Chris Lu
0e33272f56
small refactoring
2019-04-10 23:39:53 -07:00
stlpmo-jn
c1a0403da2
repair the error replications of the volume
2019-04-10 19:41:55 +08:00
Chris Lu
3555628ad5
weed volume: fix bug, remove boltdb, btree
2019-04-09 10:08:59 -07:00
Chris Lu
8a5ce16e96
weed volume: remove boltdb, btree options, add options for leveldb medium, large
2019-04-09 09:42:06 -07:00
Chris Lu
9924fa3b1a
weed volume: disable block cache on leveldb
...
saving 8MB for each volume, but at the cost of slower lookup
2019-04-09 00:24:32 -07:00
Chris Lu
784c5bb73a
add build option support 5-byte offset
2019-04-08 22:01:29 -07:00
Chris Lu
7a4b234ea1
divide offset into higher and lower sections
2019-04-08 21:44:06 -07:00
Chris Lu
000ee725fc
refactor Offset into a struct of bytes
2019-04-08 19:40:56 -07:00
Chris Lu
94302935b2
weed volume: only store compressed data if compression is effective
2019-04-06 00:10:52 -07:00
Chris Lu
7a14cdc90c
refactoring, go fmt
2019-03-25 23:18:40 -07:00
Chris Lu
19728fe3f6
remove deprecated code
2019-03-25 23:12:14 -07:00
Chris Lu
df95ce0b6c
weed backup: efficient delta backup
...
fix https://github.com/chrislusf/seaweedfs/issues/399
2019-03-25 23:01:53 -07:00
Chris Lu
70815e9124
WIP
2019-03-25 09:16:12 -07:00
Chris Lu
95e0520182
weed volume: add grpc operation to relicate a volume to local
2019-03-23 11:33:34 -07:00
Chris Lu
cea5c53bf7
fix length checking
...
fix https://github.com/chrislusf/seaweedfs/issues/890
2019-03-19 20:56:27 -07:00
Chris Lu
da871896c3
weed filer: set grpc port to port + 10000
2019-03-19 05:47:41 -07:00
Chris Lu
916b809c08
add error checking
2019-03-19 05:34:43 -07:00
Chris Lu
ece9d13312
volume info collect compact revision number
2019-03-18 09:32:21 -07:00
Chris Lu
44647a46c0
needle scanner read appendAtNs
2019-03-17 23:28:43 -07:00
Chris Lu
8b26d15740
refactoring: simplify function parameter
2019-03-17 22:32:01 -07:00
Chris Lu
2fcc88116e
go fmt
2019-03-17 20:27:36 -07:00
Chris Lu
aca653c08b
weed shell: list volumes
2019-03-17 20:27:08 -07:00
Chris Lu
95ef4513c8
print out memory size
2019-03-13 23:14:40 -07:00
Chris Lu
346541a101
print out per entry memory usage
2019-03-13 23:07:24 -07:00
chenwanli
fd27ed7755
Fix https://github.com/chrislusf/seaweedfs/issues/861
2019-02-26 17:12:39 +08:00
Chris Lu
77b9af531d
adding grpc mutual tls
2019-02-18 12:11:52 -08:00
Chris Lu
09471b46f9
fix related tests
2019-01-26 00:10:29 -06:00
Chris Lu
5668ed798d
memory needle map mark size to be TombstoneFileSize
...
fix https://github.com/chrislusf/seaweedfs/issues/850
2019-01-26 00:05:44 -06:00
bingoohuang
cf0fc797ac
use tempFile.Truncate to reset file size to speedup test
2019-01-23 16:24:51 +08:00
bingoohuang
ab6be025d7
go fmt and fix some typo
2019-01-17 09:17:19 +08:00
bingoohuang
6ddfaf33cb
extract VolumeFileScanner for ScanVolumeFile
2019-01-16 17:48:59 +08:00
bingoohuang
27093bc2e5
fix needle Append return offset to avoid uint32 overflow
2019-01-16 16:48:03 +08:00
Chris Lu
badd8fdf8f
shortcut for empty file
2019-01-08 09:03:28 -08:00
Chris Lu
6b5d6bb5a6
adjust error names
2019-01-05 19:52:38 -08:00
Chris Lu
9383c91eb1
wait to read again if the volume is compacting
2019-01-05 19:52:17 -08:00
Chris Lu
d5ebfd5b0c
fix s3cmd put
...
1. fix volume put data size
2019-01-02 12:57:33 -08:00
Chris Lu
1478d7ea21
reduce file seek when writing
2018-12-31 15:08:32 -08:00
Chris Lu
0c932d1738
add volume delete grpc API
...
fix https://github.com/chrislusf/seaweedfs/issues/808
2018-12-29 00:03:30 -08:00
Chris Lu
c043fd17cb
detect more gzippable content
2018-12-22 15:05:31 -08:00
Chris Lu
852ee21835
avoid .gz auto decompression
2018-12-22 13:58:16 -08:00
Chris Lu
be946c9e54
filer store original data size instead of data size after gzip
2018-12-22 13:11:07 -08:00
Chris Lu
141d302492
fix binarySearchCompactSection
2018-12-22 11:52:58 -08:00
Chris Lu
5333f2984a
streaming updates for large entries or large index file
...
fix https://github.com/chrislusf/seaweedfs/issues/801
2018-12-22 11:10:08 -08:00
Chris Lu
36d13355bb
compact map fix bug when iterating
2018-12-22 11:05:29 -08:00
Chris Lu
f64dc7465b
remove index file
...
fix https://github.com/chrislusf/seaweedfs/issues/793
2018-12-17 20:33:32 -08:00
Chris Lu
42cb9b76c4
reduce needle map memory usage by 25%
2018-12-15 05:55:56 -08:00
Chris Lu
168f461761
fix MaxUint8
2018-12-10 20:57:51 -08:00
Chris Lu
8f78a069c6
fix possible name overflow when len(name)>255
2018-12-10 20:57:01 -08:00
Chris Lu
4a490fe35a
go fmt
2018-12-09 01:27:25 -08:00
Chris Lu
5741134ca0
change overflow from map to slice
2018-12-09 01:27:11 -08:00
Chris Lu
d9189c857c
remove unused file
2018-12-09 00:30:53 -08:00
Chris Lu
df4b9df142
add memory usage report
2018-12-09 00:15:23 -08:00
Chris Lu
ec5a305624
refactor
2018-12-09 00:12:37 -08:00
Chris Lu
9d3be33e5c
optimize a little bit
2018-12-08 21:45:14 -08:00
Chris Lu
86dd933596
go fmt
2018-11-23 00:26:15 -08:00
Chris Lu
2697d6a4fb
fix deleting bucket
2018-11-08 22:15:21 -08:00
Chris Lu
a4ceb051a7
use MD5 for ETag to be consistent with Amazon S3
2018-11-08 21:41:02 -08:00
Chris Lu
f9dbaf6292
delete file by name instead of referencing the dataFile pointer
2018-11-05 08:53:38 -08:00
alex8224
8d3f9c0b75
fix invalid point when delete collection
2018-11-05 19:53:55 +08:00
Chris Lu
f050b22d6c
close file in ScanVolumeFile()
...
fix https://github.com/chrislusf/seaweedfs/issues/761
2018-11-04 00:28:24 -07:00
Chris Lu
496a7398d1
adjust logging
2018-10-18 20:47:30 -07:00
Chris Lu
eec951cad2
migrate volume sync to gRpc
2018-10-15 21:44:41 -07:00
Chris Lu
66a353dcb5
remove volume server /admin/volume/delete
2018-10-15 01:26:49 -07:00
Chris Lu
fda771c83f
migrate volume sync status to grpc API on volume server
2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d
migrate assign volume to grpc API on volume server
2018-10-15 00:40:46 -07:00
Chris Lu
333709657c
adjust log level
2018-10-15 00:40:12 -07:00
Chris Lu
b1daede91b
move volume vacuum to gRpc
2018-10-14 23:12:43 -07:00
Chris Lu
a362261972
avoid possible racing issue if leveldb or boltdb is used.
...
remove the to-be-generated files during commit
fix https://github.com/chrislusf/seaweedfs/issues/738
2018-09-26 18:45:51 -07:00
Chris Lu
164091c269
add s3 multipart upload
2018-09-09 16:25:43 -07:00
Chris Lu
9b3bf0e46c
fix "weed backup" rerunning
...
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
2018-09-09 02:48:58 -07:00
Chris Lu
bc8d34143d
Revert "fix "weed backup" rerunning"
...
This reverts commit 1937be2bb2
.
2018-09-09 02:38:30 -07:00
Chris Lu
1937be2bb2
fix "weed backup" rerunning
...
"weed backup" rerunning will already have ReplicaPlacement set, while version is not set.
2018-09-09 02:31:12 -07:00
Chris Lu
76cbe8bf33
instant notification of new volumes added or deleted
2018-08-24 01:26:56 -07:00
Chris Lu
ac793a3c5a
clean remove all left over files when deleting a collection
2018-08-23 23:33:16 -07:00
Chris Lu
d361a1ddf5
fix id String() instead
2018-08-23 22:44:30 -07:00
Chris Lu
694f93de80
parse needle id correctly
...
fix https://github.com/chrislusf/seaweedfs/issues/713
2018-08-23 20:48:43 -07:00
Chris Lu
1f030af06f
adjusting startup logging
2018-08-22 13:59:04 -07:00
Chris Lu
75d63db60d
randomize raft server startup
...
also some go fmt
2018-08-12 14:27:14 -07:00
Chris Lu
495a776671
support 128 bit NeedleId
2018-07-31 23:25:26 -07:00
Chris Lu
10bb76fe6b
go fmt
2018-07-27 02:10:42 -07:00
Chris Lu
4d322df95e
avoid reparsing the multipart form
2018-07-27 02:10:10 -07:00
Chris Lu
57ded4fd80
speed up leveldb bolddb loading
2018-07-24 22:17:56 -07:00
Chris Lu
ea8ecf35f4
set needle AppendAtNs value during writes
2018-07-24 02:44:33 -07:00
Chris Lu
77fadab560
go fmt
2018-07-24 01:38:08 -07:00
Chris Lu
4f317c7e3d
working version3
2018-07-24 01:36:04 -07:00
Chris Lu
852af28f91
needle priting format
2018-07-21 17:41:21 -07:00
Chris Lu
7e2031b18f
go fmt
2018-07-21 17:39:10 -07:00
Chris Lu
c98df05ed0
support PUT
2018-07-21 15:58:48 -07:00
Chris Lu
feb8eeb830
refactor: separate multipart parsing file
2018-07-21 12:01:35 -07:00
Chris Lu
3edfe1d28f
extend export command to show tombstone + change output format to CSV
...
merging https://github.com/chrislusf/seaweedfs/pull/610 and add "-limit" option
2018-07-14 20:51:17 -07:00
Chris Lu
a51aa4c586
avoid extra log messages
...
fix https://github.com/chrislusf/seaweedfs/issues/689
2018-07-13 22:07:32 -07:00
Chris Lu
8a12fd9ec4
add error checking when loading dat file
2018-07-09 23:31:25 -07:00
Chris Lu
d0982cafa7
refactoring
2018-07-09 00:22:50 -07:00
Chris Lu
5b1fd374be
fix all tests
2018-07-08 02:39:04 -07:00
Chris Lu
d4d7ced922
refactoring: add type for needle id, offset
...
later the type size can possibly be adjusted
2018-07-08 02:28:04 -07:00
Chris Lu
5bfb72d058
faster loading boltdb or leveldb needle map metrics by bloomfilter
...
avoid btree
2018-07-07 00:51:17 -07:00
Chris Lu
aba1fe01b3
use constant NeedleIndexSize instead of the number 16
2018-07-07 00:50:14 -07:00
Chris Lu
defcd4edc5
adjust log level
2018-07-07 00:48:58 -07:00
Chris Lu
9bcaa65574
avoid divided by 0
...
fix https://github.com/chrislusf/seaweedfs/issues/650
2018-06-25 01:20:15 -07:00
Chris Lu
233a9a9b6c
fix log error
2018-06-24 16:12:41 -07:00
Chris Lu
49f4216280
fix compilation error
2018-06-24 15:37:54 -07:00
Chris Lu
7f6500ae07
super block reserves a couple of bits for future extension
2018-06-24 15:23:03 -07:00
Chris Lu
f339e9c284
add support for extra super block info
2018-06-24 15:19:57 -07:00
Chris Lu
9ba335a7c4
prepare for flexible super block
2018-06-24 11:37:08 -07:00
Chris Lu
3a6f9c9bcf
increase test size
2018-06-23 18:30:13 -07:00
Chris Lu
7f7e4e9885
fix error for deleted files during compaction
...
deletion during commit may cause trouble when make up the difference during commitCompact()
2018-06-23 18:24:59 -07:00
Chris Lu
3699996985
add compaction test
2018-06-23 16:48:19 -07:00
Chris Lu
0467195f07
fix file closing on windows
2018-06-13 00:36:51 -07:00
Chris Lu
eaf000eff1
fix error on go tip
2018-06-11 23:39:42 -07:00
Chris Lu
43e3f5724c
use fixed list of masters in both filer and volume servers
2018-06-01 00:39:39 -07:00
Chris Lu
458ada173e
go fmt
2018-05-27 11:52:26 -07:00
Chris Lu
1675243f29
maybe speed up a little when loading index
2018-05-23 02:38:11 -07:00
Chris Lu
69b9d8c3c2
fix boltdb variable usage
2018-05-22 10:18:09 -07:00
Chris Lu
942c2cbd7b
mv pb to master_pb
2018-05-09 23:11:54 -07:00
Chris Lu
57c5405657
add todo
2018-05-09 04:18:31 -07:00
Chris Lu
6db1c8bafb
Revert "adjust the needle mapper"
...
This reverts commit e2c7be3a2e
.
2018-05-09 03:57:07 -07:00
Chris Lu
e2c7be3a2e
adjust the needle mapper
...
as discussed on
https://github.com/chrislusf/seaweedfs/pull/642#issuecomment-387606960
2018-05-09 03:50:14 -07:00
WenLong LI
b9068c1304
use volumes variable
...
fix leveldb mount bug
2018-05-03 12:13:53 +08:00
liwenlong05
02a1597642
fix leveldb mount bug
2018-05-03 11:20:38 +08:00
mtolman
e4b1e5c3c8
Fix for issue #635
...
Fix for the following issue on master: https://github.com/chrislusf/seaweedfs/issues/635
2018-04-17 13:26:48 -06:00
Yang Pengcheng
a2112e1be0
fix ldb dir deleting
2017-08-30 22:55:03 +08:00
Chris Lu
58344980e4
preallocate disk space during compaction also, add cleanup for failed compaction
2017-08-29 23:59:53 -07:00
Chris Lu
5047bdb4a2
skip bytes cache
2017-06-03 11:44:24 -07:00
Chris Lu
82c3ccc8dd
add btree for volume index
2017-05-26 22:51:25 -07:00
Chris Lu
bd97cbc523
skip busy reporting back to master
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/499
2017-05-23 21:12:03 -07:00
Chris Lu
043b7a7c65
add locking to access l.volumes
2017-01-20 10:18:43 -08:00
brstgt
0656838fe5
Don't return actual file size from CheckVolumeDataIntegrity, it will be 0 if last needle is a tombstone, so it's not reliable anyway
2017-01-20 16:37:45 +01:00
brstgt
4fb5bb09b6
Remove obsolete property Volume.dataFileSize
2017-01-20 16:31:11 +01:00
brstgt
e074a54a20
Delete volumes online without restarting volume server
2017-01-20 13:02:37 +01:00
brstgt
492f93416d
Mount and unmount volumes online without restarting volume server
2017-01-20 12:49:20 +01:00
brstgt
18b3afc97a
Log volumeId if dataFileSize != actual data file size - improve log types
2017-01-20 12:22:25 +01:00
brstgt
0c898df430
Log volumeId if dataFileSize != actual data file size
2017-01-20 11:52:23 +01:00
Chris Lu
5b0f5f456e
volume server get notified of leader change
2017-01-18 09:34:27 -08:00
Chris Lu
59022b6fe0
fix byte counter on loading index file
...
fix https://github.com/chrislusf/seaweedfs/issues/441
2017-01-18 08:40:39 -08:00
Chris Lu
3065506b38
volume servers always connect to the master leader
2017-01-13 10:20:40 -08:00
Chris Lu
e46c3415f7
gRpc for master~volume heartbeat
2017-01-10 01:01:12 -08:00
Chris Lu
4beaaa0650
simplify the gzip optimization logic
...
fix https://github.com/chrislusf/seaweedfs/issues/436
2017-01-09 19:31:58 -08:00
Chris Lu
ed44f12f6d
support Fallocate on linux
2017-01-08 11:01:46 -08:00
Chris Lu
7b6837cbc2
move back the section
2017-01-08 10:35:47 -08:00
sparklxb
da9b672d1b
support additional header name-value pairs
2017-01-08 22:34:42 +08:00
sparklxb
86a7c56275
support additional header name-value pairs
2017-01-08 09:16:40 +08:00
Chris Lu
13e7069eb9
keep track of total data file size
2017-01-06 10:22:20 -08:00
lixianbin
d96d0a87cf
fix bug: upload big .gz file more than maxMB
2017-01-04 11:23:40 +08:00
Chris Lu
dcaf1796fe
add option to enable caching
2016-11-16 07:09:57 -08:00
Chris Lu
f54f530ada
adjust verbose logging
2016-11-13 14:07:51 -08:00
霍晓栋
b9f385bd66
refactor concurrent loading code logic
2016-11-13 13:24:52 +08:00
霍晓栋
096ffa9744
concurrent loading volume
2016-11-11 11:53:22 +08:00
Chris Lu
36f9633223
add locks for location.volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/392
2016-11-06 20:55:22 -08:00
霍晓栋
7d73bbb073
comment UT case
2016-10-07 16:40:51 +08:00
霍晓栋
7d382ba5fe
comment UT case
2016-10-07 16:34:22 +08:00
霍晓栋
ce1f7ab662
makediff func with UT case
2016-10-07 16:22:24 +08:00
霍晓栋
ed848425c7
supplemental data between compacting and commit compacting
2016-09-29 13:57:23 +08:00
Chris Lu
dffad65f2f
fix compilation
2016-09-26 22:30:44 -07:00
Chris Lu
7e29218327
add a template for makeupDiff
2016-09-26 22:26:41 -07:00
Chris Lu
1bc041b46d
add a new way to manually compact corrupted volume
...
fix https://github.com/chrislusf/seaweedfs/issues/371
2016-09-22 20:31:20 -07:00
Chris Lu
01cbd5cb58
lock
...
fix https://github.com/chrislusf/seaweedfs/issues/367
2016-09-08 09:50:31 -07:00
霍晓栋
b9b3651a98
deleted index entry could not point to deleted needle
2016-07-25 14:56:58 +08:00
霍晓栋
09bd3d015d
deleted index entry could not point to deleted needle
2016-07-25 14:54:40 +08:00
霍晓栋
78678f4bcb
deleted needle does not need checksum verification
2016-07-25 11:40:35 +08:00
Chris Lu
d981eb282f
refactor volume_read_write.go out of volume.go
2016-07-03 00:10:27 -07:00
Chris Lu
576573711a
refactoring
2016-07-02 23:58:23 -07:00
Chris Lu
3d8df0f709
refactor volume_loading.go out of volume.go
2016-07-02 23:56:49 -07:00
Chris Lu
582d5d526c
refactoring
2016-07-02 23:50:58 -07:00
Chris Lu
3713119ab9
adjust verification logic.
...
1. log file name for easier debugging
2. remove unused version in readIndexEntryAtOffset()
3. if indexSize==0, the dataFile usually still has a super block of
size 8. So skipping this case.
2016-07-02 23:45:37 -07:00
霍晓栋
16f6984d02
refactor data integrity checking code v2
2016-07-03 14:11:25 +08:00
霍晓栋
dda13def2a
refactor data integrity checking code
2016-07-03 12:53:03 +08:00
霍晓栋
2e74fb60c8
add data integrity checking
2016-07-02 21:34:30 +08:00
Chris Lu
101e784577
add locking on possible concurrent map access
...
fix https://github.com/chrislusf/seaweedfs/issues/328
2016-06-23 09:10:25 -07:00
霍晓栋
1ef81ac518
filer could detect master nodes healthy status and choose a working one when encountering SPOF
2016-06-07 11:38:40 +08:00
Chris Lu
5ce6bbf076
directory structure change to work with glide
...
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00
Chris Lu
5071f528f6
testing compilation with remove package
2013-02-10 03:49:51 -08:00
Chris Lu
55f2627fcf
testing new directory structure
2013-02-10 03:31:40 -08:00
Chris Lu
ab6fb13ad7
avoid the "src" folder
2013-02-10 03:25:35 -08:00