chrislu
6c6655de9d
wip
2023-09-19 14:03:42 -07:00
chrislu
4827425146
balancer works
2023-09-16 15:05:38 -07:00
chrislu
436d99443b
receive broker stats
2023-09-14 23:49:05 -07:00
chrislu
b771fefa37
Merge branch 'master' into sub
2023-09-13 00:03:47 -07:00
chrislu
2492a45499
ack interval
2023-09-06 22:39:46 -07:00
chrislu
ba67e6ca29
api for sub
2023-09-04 21:43:50 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
chrislu
504ae8383a
protoc version
2023-08-28 09:01:25 -07:00
chrislu
dbcba75271
rename to lookup
2023-08-27 18:59:04 -07:00
chrislu
4d6c18d86f
pub sub initial tests
2023-08-27 17:50:59 -07:00
chrislu
1990456670
sub
2023-08-27 13:13:14 -07:00
chrislu
905911853d
adjust proto
2023-08-26 13:39:21 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer ( #4767 )
2023-08-24 07:08:56 -07:00
chrislu
010c5e91e3
add stream assign proto
2023-08-22 09:53:54 -07:00
chrislu
01d70c21f3
Squashed commit of the following:
...
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
2023-08-20 22:53:05 -07:00
chrislu
f8aa5ea844
adjust filer.proto
2023-06-25 17:38:19 -07:00
chrislu
5149b3d07b
filer can proxy to peer filer holding the lock
2023-06-25 15:28:01 -07:00
chrislu
464a71a373
add distributed lock manager
2023-06-25 14:14:40 -07:00
chrislu
3fd659df2a
add distributed lock manager
2023-06-25 00:58:21 -07:00
chrislu
8ec1bc2c99
remove unused cluster node leader
2023-06-19 18:19:13 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty ( #4561 )
...
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
wusong
26f15d0079
Fix no more writable volumes by delay judgment ( #4548 )
...
* fix nomore writables volumes while disk free space is sufficient by time delay
* reset
---------
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-05 10:17:21 -07:00
mervynzhang
1ebb549f77
support swift ( #4480 )
2023-05-19 06:39:25 -07:00
Konstantin Lebedev
d75a7b7f62
allow deleting only older empty dir without recursion ( #4430 )
2023-04-25 08:31:14 -07:00
chrislu
5db9fcccd4
refactoring
2023-03-21 23:01:49 -07:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region ( #4177 )
...
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
chrislu
e1ca6308cb
add chunk etag when downloading from remote storage
...
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2022-11-23 15:59:38 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
...
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
21c0587900
go fmt
2022-09-14 23:06:44 -07:00
chrislu
c8645fd232
master: implement grpc VolumeMarkWritable
...
fix https://github.com/seaweedfs/seaweedfs/issues/3657
2022-09-14 23:05:30 -07:00
Ryan Russell
12914af4d8
Character readability ( #3678 )
...
* refactor(pb): `quote_charactoer` -> `quote_character`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:09:53 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction ( #3671 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:45 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero ( #3630 )
...
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629
* fix GrpcPort
* skip url :0
* skip empty DataCenter or Rack
* skip empty heartbeat Ip
* dell msg add DataCenter
* comment todo
* fix
2022-09-11 22:31:53 -07:00
chrislu
b9112747b5
volume server: synchronously report volume readonly status to master
...
fix https://github.com/seaweedfs/seaweedfs/issues/3628
2022-09-11 19:29:10 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc ( #3581 )
...
* ADHOC: add read needle meta grpc
* add test
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-06 23:51:27 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long ( #3398 )
...
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
2022-08-23 23:18:21 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files ( #3478 )
...
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Konstantin Lebedev
fc65122766
rename to LoadAvg_1M
2022-08-01 21:32:21 +05:00
Konstantin Lebedev
5209ebbeef
remove percent
2022-08-01 20:40:38 +05:00
Konstantin Lebedev
634bc97840
fix conflicts
2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
chrislu
afef014832
scaffold
2022-07-31 13:23:44 -07:00
chrislu
13b9a52f80
fix deprecated functions
2022-07-29 01:34:39 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
chrislu
de3afd4199
fix rebasing
2022-07-28 23:25:46 -07:00
chrislu
06cd491abc
add message batch
2022-07-28 23:24:38 -07:00
chrislu
7576c244c4
add flatbuffer serde for message
2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47
segment serde
2022-07-28 23:24:38 -07:00
chrislu
9f479aab98
allocate brokers to serve segments
2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c
clean up
2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5
remove old code
2022-07-28 23:24:38 -07:00
chrislu
f25e273e32
display data center and rack in cluster.ps
2022-07-28 23:22:52 -07:00
chrislu
68065128b8
add dc and rack
2022-07-28 23:22:51 -07:00
chrislu
21b6b07dd8
renaming
2022-07-28 23:22:06 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
449ecc8890
remove max connection age
...
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
chrislu
530da0cc89
remove unused code
2022-06-22 00:34:25 -07:00
chrislu
41dfe27102
Revert "remove max connection age"
...
This reverts commit b9b684194f
2022-06-20 14:49:39 -07:00
chrislu
1d0c53ea56
remote storage: stop supporting hdfs as a remote storage
2022-06-20 14:15:59 -07:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
chrislu
d12f431d98
collect volume server status
2022-06-12 11:56:23 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f
fix s3 tests
...
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
2022-06-07 14:43:10 +05:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
creeew
02ae102731
fix filer.sync missing source srv uploaded files to target when target down
2022-06-02 01:28:47 +08:00
chrislu
682382648e
collect cluster node start time
2022-05-30 16:23:52 -07:00
chrislu
a2b101a737
subscribe metadata between a range
2022-05-30 15:04:19 -07:00
chrislu
ca16fbf0ef
shell: cluster.ps display filer group
2022-05-29 16:37:14 -07:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
...
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879
revert changes of pb file
...
orignal protoc-gen-go-grpc verion is 1.1.0
2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
chrislu
866fed1eab
avoid grpc name conflict
...
fix https://github.com/chrislusf/seaweedfs/issues/3055
2022-05-15 21:31:29 -07:00
chrislu
29198720f2
s3: add grpc server to accept configuration changes
2022-05-15 00:43:37 -07:00
chrislu
8bc16f1149
rename
2022-05-06 23:38:22 -07:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
chrislu
4bd6bea429
fix test
2022-05-01 09:02:01 -07:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
chrislu
b4be56bb3b
add timing info during ping operation
2022-04-16 12:45:49 -07:00
