Commit Graph

74 Commits

Author SHA1 Message Date
NyaMisty
579ebbdf60
Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed (#5636) 2024-06-02 14:25:42 -07:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck (#5133)
* Add modifyTimeAgo to volume.fsck

* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler (#5132) 2023-12-23 09:18:16 -08:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs (#4264)
* fix

* fix mtime s > ns

---------

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:24:36 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency (#4239) 2023-02-21 19:47:33 -08:00
chrislu
e037c71ec3 adjust text 2023-02-10 13:04:29 -08:00
chrislu
67b8c2853a add line return 2023-02-10 12:53:43 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934)
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id (#3933)
volume.fsck param volumeId is comma separated the volume id

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server (#3926)
check needles status from volume server
2022-10-31 11:33:04 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix (#3923)
* sync of deletions the fix

* avoid return if only partiallyDeletedNeedles

* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
ea2637734a refactor filer proto chunk variable from mtime to modified_ts_ns 2022-10-28 12:53:19 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns (#3906)
* ADHOC: volume fsck using append at ns

* nit

* nit

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
chrislu
377870f4a9 keep system log data 2022-10-24 16:50:39 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861)
* fix apply purging and add verifyNeedle

* common readSourceNeedleBlob

* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume (#3851)
* refactor

* refactor args verbose and writer

* refactor readFilerFileIdFile

* fix filter by collectMtime

* skip system log collection
2022-10-13 23:30:30 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search (#3763)
* ADHOC: filter deleted files from idx file binary search

* remove unwanted check

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
chrislu
b6d7556dda skip truncation on error
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param (#3626)
* ADHOC: cut off volumn fsck

* more

* fix typo

* add test

* modify name

* fix comment

* fix comments

* nit

* fix typo

* Update weed/shell/command_volume_fsck.go

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-10 15:29:17 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
271b5aed96 shell: volume.fsck add a note for -reallyDeleteFromVolume option 2022-05-15 11:07:04 -07:00
Konstantin Lebedev
d4343ab7da forcePurging desc 2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f change forcePurging to a pointer 2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2 delete missing data from volumes in one replica 2022-04-25 22:59:46 +05:00
Konstantin Lebedev
7f1383a41e findExtraChunksInVolumeServers in consideration of replication 2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0 fix collect filer files 2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7 skip new entities 2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0 fsck replicas 2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02 collecting only bucket volumes 2022-03-31 13:35:58 +05:00
banjiaojuhao
f7f2a597dd minor 2022-03-08 16:22:55 +08:00
Chris Lu
64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete
Added basic deletion capabilities to fsck.
2022-02-14 23:00:01 -08:00
Radtoo
724ce04b1c We picked the second flag's name. 2022-02-13 00:53:35 +01:00
Radtoo
fbe7ed7927 Revert the the flag to "reallyDeleteFromVolume". 2022-02-10 20:01:48 +01:00
chrislu
f18803424a volume.balance: add delay during tight loop
fix https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:53:55 -08:00
Radtoo
fbb14e0ea8 Adding separate toggle to purge absent vols
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.

Also renamed forcePurge to just purge.
2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2 Also delete paths for which a volume is entirely absent. 2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu
ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu
60dc450091 skip fixing read only volumes
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu
94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
Chris Lu
9ccfc1cfee Update command_volume_fsck.go
minor
2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9 Update command_volume_fsck.go
remove unused code
2021-11-28 22:37:11 -08:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
119d5908dd shell: do not need to lock to see volume -h 2021-09-13 22:13:34 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
333cdce485 add verbose message 2021-08-13 02:57:14 -07:00