mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 06:57:51 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,054 Commits 32 Branches 296 Tags 152 MiB
1e13b6879c
Commit Graph

873 Commits

Author SHA1 Message Date
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs (#3856) 
* avoid data race of TraverseBfs

* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]

goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
        /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
 2022-10-16 11:30:41 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch (#3841) 
* avoid load volume file with BytesOffset mismatch

https://github.com/seaweedfs/seaweedfs/issues/2966

* set BytesOffset if has not VolumeInfoFile

* typos fail => failed

* exit if bytesOffset mismatch
 2022-10-14 00:18:09 -07:00
chrislu
9c8678ded9 ensure monotonic n.AppendAtNs 
fix https://github.com/seaweedfs/seaweedfs/issues/3852
 2022-10-13 23:15:00 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files (#3832) 2022-10-13 00:51:20 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece (#3838) 2022-10-12 22:59:07 -07:00
chrislu
e55076c46f cloud tier: add retry when copying data file 
fix https://github.com/seaweedfs/seaweedfs/issues/3828
 2022-10-12 00:38:32 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric (#3830) 
* optimiz updating mapmetric for leveldb

* import loading leveldb

* add comments
 2022-10-11 21:13:25 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac (#3814) 2022-10-10 07:28:02 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie (#3748) 
* fix: compact_map get error

* fix: CompactSection delete lock and move test to compact_map

Co-authored-by: shibinbin <shibinbin@megvii.com>
 2022-10-09 16:56:40 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. (#3802) 2022-10-07 11:27:15 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763) 
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-29 12:48:36 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal (#3755) 
Add new volumes to HUP(reload) signal
 2022-09-28 12:44:13 -07:00
chrislu
3fc261d27c Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-16 23:54:39 -07:00
chrislu
06748f2771 remove unused code 2022-09-16 23:54:37 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements (#3703) 
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
 2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702) 
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
 2022-09-16 00:30:40 -07:00
chrislu
896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
chrislu
0baf3d38c2 adjust logs 2022-09-13 10:33:28 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626) 
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2022-09-10 15:29:17 -07:00
famosss
41af5a6ee5
simplify a bit (#3631) 2022-09-08 21:01:21 -07:00
chrislu
64e75a286e simplify a bit 2022-09-08 19:07:57 -07:00
chrislu
d14d029c73 reduce v.dataFileAccessLock lock scope 2022-09-08 19:05:31 -07:00
chrislu
406a80da4b adjust variable names 2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2 refactor: move ReadNeedleDataInto into volume_read.go 2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc (#3581) 
* ADHOC: add read needle meta grpc

* add test

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-09-06 23:51:27 -07:00
chrislu
39340f7e42 cloud tier: s3 consume all read response body 
fix https://github.com/seaweedfs/seaweedfs/issues/3584
 2022-09-04 22:23:28 -07:00
chrislu
7c6324b114 adjust log level 2022-09-04 22:21:24 -07:00
chrislu
8c5759a06d minor optimization 2022-09-04 18:50:45 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version (#3539) 
* avoid race conditions access to  SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515

* superBlockAccessLock replace to sync.Mutex
 2022-08-30 00:08:00 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu
e0f4366f4c more detailed error 
related to https://github.com/seaweedfs/seaweedfs/issues/3528
 2022-08-26 16:49:48 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount (#3526) 2022-08-26 08:41:42 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact (#3388) 
* optimiz vacuuming volume

* fix bugx

* rename parameters

* fix conflict

* change copyDataBasedOnIndexFile to an instance method

* close needlemap

* optimiz commiting Vacuum volume for  leveldb index

* fix bugs

* fix leveldb loading bugs

* refactor

* fix leveldb loading bug

* add leveldb recovery

* add test case for levelDB

* modify test case to cover all the new branches

* use one tmpNm instead of two instances

* refactor

* refactor

* move setWatermark to the end

* add test for watermark and updating leveldb

* fix error logic

* refactor, add test

* check nil before close needlemapeer
add test case
fix metric bug

* add tests, fix bugs

* adjust log level
remove wrong test case
refactor

* avoid duplicate  updating metric for leveldb index
 2022-08-23 23:53:35 -07:00
Guo Lei
762dc219a6
fix deletecount bug (#3499) 2022-08-23 08:53:24 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same (#3477) 
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc)
 2022-08-21 19:11:18 -07:00
chrislu
c7892bc7c4 volume: file counter should be all files 
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
 2022-08-20 23:35:31 -07:00
chrislu
4d72a1ce6e s3: remove deleted collection during metrics reporting 
fix https://github.com/seaweedfs/seaweedfs/issues/3453
 2022-08-18 02:39:07 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
93261f5199 Revert "Refactor for Sync method (#3426)" 
This reverts commit 670cb759f8.

with the pr
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	18.486s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.025s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.864s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.110s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.414s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.203s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
weed/storage () - (master) >
weed/storage () - (master) >

without the pr
weed/storage () - (master) >
weed/storage () - (master) > go test -count=1 ./...
ok  	github.com/seaweedfs/seaweedfs/weed/storage	1.617s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map	0.026s
?   	github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding	0.906s
?   	github.com/seaweedfs/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle	0.202s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/needle_map	24.533s
ok  	github.com/seaweedfs/seaweedfs/weed/storage/super_block	0.280s
?   	github.com/seaweedfs/seaweedfs/weed/storage/types	[no test files]
?   	github.com/seaweedfs/seaweedfs/weed/storage/volume_info	[no test files]
 2022-08-15 15:35:31 -07:00
Rain Li
670cb759f8
Refactor for Sync method (#3426) 2022-08-10 08:21:57 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code (#3423) 
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
 2022-08-09 01:24:43 -07:00
Konstantin Lebedev
3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
Evgeny Kuzhelev
47c72e6f35 remove all (currently existing) collection volume metrics 2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c delete disk_size metrics when collection deleted 2022-07-27 15:02:41 +05:00
guosj
cc7a4b0a6e correct comment 2022-07-25 11:46:41 +08:00
guol-fnst
ac694f0c8f rename parameter and reuse functions 
rename milestone to  watermark
 2022-07-20 17:00:40 +08:00
guol-fnst
91285bb51d remove uncessary code 2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34 optimiz 2022-07-19 16:58:53 +08:00
guol-fnst
bec9f79659 use snapshot 2022-07-19 11:14:41 +08:00
guol-fnst
300b383cdf use 10 or numCPU workers if env is not found 2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2 optimiz concurrency 
user can customize number of workers via env "GOMAXPROCS"
 2022-07-14 09:57:25 +08:00
guol-fnst
26313060a3 speeding up loading volumes 2022-07-12 16:19:33 +08:00
chrislu
06a8b174b5 also remove Sync() for idx file 2022-06-30 13:50:53 -07:00
chrislu
52580743b9 stuck with file.Sync() 
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s

goroutine 61 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
	/usr/local/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
	/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
	/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
	/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
	_testmain.go:57 +0x1aa

goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf

goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
	/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
	/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
	/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
	/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1486 +0x35f

goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL	github.com/chrislusf/seaweedfs/weed/storage	600.194s
?   	github.com/chrislusf/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/erasure_coding	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle_map	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/super_block	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/types	[no test files]
?   	github.com/chrislusf/seaweedfs/weed/storage/volume_info	[no test files]
FAIL
 2022-06-30 12:32:55 -07:00
garenchan
8aa19577f4 fix 3238: handle errors for GenerateDirUuid method 2022-06-27 22:04:50 +08:00
chrislu
509a9047db test compact map with snowflake sequencer 2022-06-23 21:45:51 -07:00
chrislu
625fd16a2e reduce upfront memory usage for low density volume 2022-06-16 15:39:29 -07:00
chrislu
36d2756978 volume: avoid writing too much data for range requests in large files 
related to https://github.com/chrislusf/seaweedfs/issues/3178
 2022-06-14 11:16:47 -07:00
chrislu
9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" 
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
 2022-06-10 06:38:17 -07:00
shichanglin5
f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal 
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
 2022-06-09 20:41:16 +08:00
chrislu
11f1e7996b fix test 2022-06-05 18:50:41 -07:00
chrislu
056c480eb0 volume: checksum remove the hashing step in Value() 2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu
685643d506 needle read into a writer 2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23 add needle reading in chunks 2022-06-04 18:15:39 -07:00
chrislu
18b29f709c refactoring, split file 2022-06-03 23:31:14 -07:00
chrislu
81d6159290 volume: report error if a volume has nil data backend 
fix https://github.com/chrislusf/seaweedfs/issues/3105
 2022-05-29 16:59:30 -07:00
guol-fnst
b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775 rename UUID file 
fix typo
move locationUUID  into DiskLocation
 2022-05-17 15:41:47 +08:00
guol-fnst
de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
chrislu
3e20336bde fix atomic add int64 
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
 2022-05-11 00:44:35 -07:00
chrislu
76148ce0f7 use storage_class for backend configuration 
fix https://github.com/chrislusf/seaweedfs/issues/3008
 2022-05-02 12:20:01 -07:00
chrislu
1aae7a3f1b volume.tier.upload progress starts negative #2992 
fix https://github.com/chrislusf/seaweedfs/issues/2992
 2022-04-30 18:10:01 -07:00
chrislu
192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
chrislu
70e5a1b632 volume close should wait for committing compaction 2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression 
waite volume being closed during compression idx
 2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0 use two flags: v.isCompacting and v.isCommitCompacting 2022-04-26 23:28:34 -07:00
chrislu
94f824e1ce volume: sync to disk before copying volume files 
address https://github.com/chrislusf/seaweedfs/issues/2976
 2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039 waite volume being closed during compression idx 2022-04-26 13:40:42 +05:00
chrislu
8e9ad7db5a fix bug deleting volume or unmount volume if a volume server has multiple directories 2022-04-19 22:58:57 -07:00
Konstantin Lebedev
1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
Konstantin Lebedev
9438738693 avoid invalid memory address or nil pointer dereference 2022-04-18 12:10:22 +05:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
justin
d51a724101 fix: encode small chunk return error maybe have some bug. 2022-03-28 13:11:24 +08:00
Konstantin Lebedev
f53cff045f checks disk file exist 2022-03-10 18:58:56 +05:00
Chris Lu
97a4b66df7
Merge pull request #2704 from guo-sj/fix_bugs_in_return_value 
fix return value in storage/volume_vacuum.go:444
 2022-02-24 00:49:29 -08:00
guosj
3e7aa1caf5 fix return value in storage/volume_vacuum.go:444 2022-02-24 15:54:36 +08:00
chrislu
c29bc9a367 fix error handling 2022-02-23 15:34:25 -08:00
Chris Lu
da58c748bc
Merge pull request #2698 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:41:30 -08:00
Chris Lu
bd092d8318
Merge pull request #2697 from guo-sj/fix_bugs_in_return_value 2022-02-23 00:30:52 -08:00
guosj
d68c27f82d fix another return value bug 2022-02-23 16:21:25 +08:00
guosj
8f9aa0cddd fix bugs in return value 2022-02-23 16:17:48 +08:00
chrislu
a129bda7d9 sync data first before stopping 2022-02-16 09:11:34 -08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory 
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2022-02-14 10:38:13 +08:00
Konstantin Lebedev
ef541972f8 updated needle with fsync 2022-02-08 00:10:53 +05:00
Konstantin Lebedev
fb97e234c9 skips compact if store is stopping 2022-02-07 20:16:15 +05:00
chrislu
433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
chrislu
76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
chrislu
50ddd8c8e2 remove debug messages 
fix https://github.com/chrislusf/seaweedfs/issues/2514
 2021-12-16 00:58:15 -08:00
chrislu
488afa5002 volume: load volume can optionally be skipped, if ec volume exists 
fix https://github.com/chrislusf/seaweedfs/issues/2489
 2021-12-05 02:28:52 -08:00
chrislu
59e58c4b23 volume: fix loading old volume format 
fix https://github.com/chrislusf/seaweedfs/issues/2487
 2021-12-05 01:06:01 -08:00
Chris Lu
0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu
c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu
d04cdcf40d s3 header add user agent 2021-10-29 12:28:24 -07:00
Chris Lu
3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
f0b928ff5e go fmt 2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc volume: find a non-empty offset when binary searching by timestamp 2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327 volume: find a non-empty offset when binary searching by timestamp 
fix https://github.com/chrislusf/seaweedfs/issues/2364
 2021-10-11 16:11:50 -07:00
Chris Lu
ce1efeb0eb adds more error message 2021-10-09 04:54:14 -07:00
Chris Lu
d688e10ed1 do not try to compress if not sure about the file content 2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8 revert temporary changes that disabled compression 
fix https://github.com/chrislusf/seaweedfs/issues/2362
 2021-10-07 13:24:16 -07:00
Chris Lu
f0d1e7bd05 skip ec volumes when loading normal volumes 2021-10-05 02:31:44 -07:00
Chris Lu
f58ea6a2ee add source name to error message 2021-10-01 02:19:30 -07:00
Chris Lu
2e9372dcf7 volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
Chris Lu
5956a8b05a adjust comment 2021-09-27 01:58:03 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
8c6ff55226 add volume not found error type, to reduce error log 2021-09-11 14:26:41 -07:00
Chris Lu
1b942dfa3c add error message 2021-09-02 12:32:46 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
f811fd0903 fix build 2021-08-30 02:03:08 -07:00
Chris Lu
5adfdd7982 change to a new bloom filter library 2021-08-29 21:26:38 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi 
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
 2021-08-25 17:34:29 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Konstantin Lebedev
865b06b7fe The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library. 
https://github.com/klauspost/crc32
 2021-08-17 13:06:48 +05:00
Chris Lu
78e8ddf910 Only when tailing volume, the zero-ed cookie should skip checking. 
This only happens when checkCookie == false and fsync == false.
 2021-08-13 02:09:35 -07:00
Chris Lu
a8617c1a39 tail volume: fix zero cookie problem from batch deletion 2021-08-13 01:54:35 -07:00
Chris Lu
734c980040 volume: support concurrent download data size limit 2021-08-08 23:25:16 -07:00
Chris Lu
9cc84a910f volume: deletion can skip volume size checking 
fix https://github.com/chrislusf/seaweedfs/issues/2225
 2021-08-01 00:32:51 -07:00
Chris Lu
49c66e88a0 volume: change all writes to fsync during graceful stopping 
fix https://github.com/chrislusf/seaweedfs/issues/2193
 2021-07-13 01:29:57 -07:00
Chris Lu
b624090398 go fmt 2021-07-01 01:21:14 -07:00
Chris Lu
24e11d1e90 look back when adding to sorted values 
look back when adding to sorted values, before adding it to overflow
 2021-06-28 22:46:49 -07:00
Chris Lu
fc8dd58aea volume: large_volume version has bug when using in memory index 
fix https://github.com/chrislusf/seaweedfs/issues/2162
 2021-06-28 15:48:07 -07:00
Chris Lu
56eb522b13 fix stats when a collection is deleted 2021-06-15 21:11:31 -07:00
Chris Lu
6c82326575 use bytes.Buffer to reduce memory allocation and gc 2021-06-06 13:42:36 -07:00
Konstantin Lebedev
99ef280c7c avoid data loss after restarting a container with a volum server 2021-06-02 17:07:19 +05:00
Chris Lu
1aa7e99a89 skip file not found error when deleting 2021-05-15 09:37:39 -07:00
qieqieplus
ac26080bd2 fix concurrent vacuum & delete panic 2021-05-05 17:54:50 +08:00
bingoohuang
cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00