chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
93261f5199
Revert "Refactor for Sync method ( #3426 )"
...
This reverts commit 670cb759f8
2022-08-15 15:35:31 -07:00
Rain Li
670cb759f8
Refactor for Sync method ( #3426 )
2022-08-10 08:21:57 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code ( #3423 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-09 01:24:43 -07:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
Evgeny Kuzhelev
47c72e6f35
remove all (currently existing) collection volume metrics
2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c
delete disk_size metrics when collection deleted
2022-07-27 15:02:41 +05:00
guosj
cc7a4b0a6e
correct comment
2022-07-25 11:46:41 +08:00
guol-fnst
ac694f0c8f
rename parameter and reuse functions
...
rename milestone to watermark
2022-07-20 17:00:40 +08:00
guol-fnst
91285bb51d
remove uncessary code
2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34
optimiz
2022-07-19 16:58:53 +08:00
guol-fnst
bec9f79659
use snapshot
2022-07-19 11:14:41 +08:00
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2
optimiz concurrency
...
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
guol-fnst
26313060a3
speeding up loading volumes
2022-07-12 16:19:33 +08:00
chrislu
06a8b174b5
also remove Sync() for idx file
2022-06-30 13:50:53 -07:00
chrislu
52580743b9
stuck with file.Sync()
...
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
2022-06-30 12:32:55 -07:00
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
2022-06-27 22:04:50 +08:00
chrislu
509a9047db
test compact map with snowflake sequencer
2022-06-23 21:45:51 -07:00
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
2022-06-16 15:39:29 -07:00
chrislu
36d2756978
volume: avoid writing too much data for range requests in large files
...
related to https://github.com/chrislusf/seaweedfs/issues/3178
2022-06-14 11:16:47 -07:00
chrislu
9f8b72a54d
Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID"
...
This reverts commit 37da68931900d53c34c4
2022-06-10 06:38:17 -07:00
shichanglin5
f5b0c04b14
perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal
...
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
2022-06-09 20:41:16 +08:00
chrislu
11f1e7996b
fix test
2022-06-05 18:50:41 -07:00
chrislu
056c480eb0
volume: checksum remove the hashing step in Value()
2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc
stream read large files
2022-06-05 11:54:04 -07:00
chrislu
685643d506
needle read into a writer
2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23
add needle reading in chunks
2022-06-04 18:15:39 -07:00
chrislu
18b29f709c
refactoring, split file
2022-06-03 23:31:14 -07:00
chrislu
81d6159290
volume: report error if a volume has nil data backend
...
fix https://github.com/chrislusf/seaweedfs/issues/3105
2022-05-29 16:59:30 -07:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775
rename UUID file
...
fix typo
move locationUUID into DiskLocation
2022-05-17 15:41:47 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
chrislu
3e20336bde
fix atomic add int64
...
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
2022-05-11 00:44:35 -07:00
chrislu
76148ce0f7
use storage_class for backend configuration
...
fix https://github.com/chrislusf/seaweedfs/issues/3008
2022-05-02 12:20:01 -07:00
chrislu
1aae7a3f1b
volume.tier.upload progress starts negative #2992
...
fix https://github.com/chrislusf/seaweedfs/issues/2992
2022-04-30 18:10:01 -07:00
chrislu
192983b464
s3 backend support customizing storage class
2022-04-30 17:36:40 -07:00
chrislu
70e5a1b632
volume close should wait for committing compaction
2022-04-26 23:34:05 -07:00
Chris Lu
ffe028f8d0
Merge pull request #2974 from kmlebedev/wait_volume_closed_compression
...
waite volume being closed during compression idx
2022-04-26 23:29:22 -07:00
chrislu
37ab8909b0
use two flags: v.isCompacting and v.isCommitCompacting
2022-04-26 23:28:34 -07:00
chrislu
94f824e1ce
volume: sync to disk before copying volume files
...
address https://github.com/chrislusf/seaweedfs/issues/2976
2022-04-26 13:03:43 -07:00
Konstantin Lebedev
7315d1d039
waite volume being closed during compression idx
2022-04-26 13:40:42 +05:00
chrislu
8e9ad7db5a
fix bug deleting volume or unmount volume if a volume server has multiple directories
2022-04-19 22:58:57 -07:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
Konstantin Lebedev
9438738693
avoid invalid memory address or nil pointer dereference
2022-04-18 12:10:22 +05:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
justin
d51a724101
fix: encode small chunk return error maybe have some bug.
2022-03-28 13:11:24 +08:00
Konstantin Lebedev
f53cff045f
checks disk file exist
2022-03-10 18:58:56 +05:00
