mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 17:07:57 +08:00
Code Issues Packages Projects Releases Wiki Activity
9,697 Commits 32 Branches 296 Tags 152 MiB
440ebd7b6d
Commit Graph

485 Commits

Author SHA1 Message Date
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu
4573c99ae3 fix tests 2022-08-18 00:15:46 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev
fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev
5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Konstantin Lebedev
634bc97840 fix conflicts 2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
chrislu
afef014832 scaffold 2022-07-31 13:23:44 -07:00
chrislu
13b9a52f80 fix deprecated functions 2022-07-29 01:34:39 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
chrislu
de3afd4199 fix rebasing 2022-07-28 23:25:46 -07:00
chrislu
06cd491abc add message batch 2022-07-28 23:24:38 -07:00
chrislu
7576c244c4 add flatbuffer serde for message 2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47 segment serde 2022-07-28 23:24:38 -07:00
chrislu
9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c clean up 2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu
f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu
449ecc8890 remove max connection age 
related to https://github.com/chrislusf/seaweedfs/pull/3226
 2022-06-29 02:44:12 -07:00
chrislu
530da0cc89 remove unused code 2022-06-22 00:34:25 -07:00
chrislu
41dfe27102 Revert "remove max connection age" 
This reverts commit b9b684194f.
 2022-06-20 14:49:39 -07:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
石昌林
3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
chrislu
d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f fix s3 tests 
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
 2022-06-07 14:43:10 +05:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol 
avoid loading duplicated volume directory
 2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879 revert changes of pb file 
orignal protoc-gen-go-grpc verion is 1.1.0
 2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst
de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
chrislu
866fed1eab avoid grpc name conflict 
fix https://github.com/chrislusf/seaweedfs/issues/3055
 2022-05-15 21:31:29 -07:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
chrislu
8bc16f1149 rename 2022-05-06 23:38:22 -07:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
chrislu
4bd6bea429 fix test 2022-05-01 09:02:01 -07:00
Konstantin Lebedev
1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
chrislu
b4be56bb3b add timing info during ping operation 2022-04-16 12:45:49 -07:00
Konstantin Lebedev
35bc67f030 fix conflicts 2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
Konstantin Lebedev
85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
Konstantin Lebedev
357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu
bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
chrislu
958f880b70 mount: add grpc method to adjust quota 2022-04-02 15:14:37 -07:00
chrislu
daca2d22a5 use original server address string as map key 2022-04-01 17:34:42 -07:00
chrislu
bbbbbd70a4 master supports grpc ping 2022-04-01 16:50:58 -07:00
chrislu
743ad690b6 filer supports grpc ping 2022-04-01 16:44:58 -07:00
chrislu
800cbc004c volume server adds ping function 2022-04-01 16:37:06 -07:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu
a0bad1c997 remove any go mod changes 
This reverts commit 6c7f7d6887.
 2022-03-21 23:04:00 -07:00
chrislu
6c7f7d6887 Revert "Merge pull request #2782 from SadmiB/upstream" 
This reverts commit a644b7236a, reversing
changes made to 349257f822.
 2022-03-21 23:00:50 -07:00
Chris Lu
0d692a6254
Merge branch 'master' into upstream 2022-03-21 11:02:04 -07:00
SadmiB
d12540c9f2 Add contabo api client 2022-03-21 17:16:49 +01:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
chrislu
91d6785cf3 define metadata action types 2022-02-25 00:54:16 -08:00
chrislu
be3fc77391 mount2: use consistent inode 2022-02-25 00:53:27 -08:00
chrislu
ceaf993a27 mount2: add rdev 2022-02-24 14:51:25 -08:00
chrislu
b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
chrislu
b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu
fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu
6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
banjiaojuhao
08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu
b70cb3e0b2 upgrade protoc to 3.17.3 
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
 2021-12-05 00:42:25 -08:00
Chris Lu
e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu
4b9c42996a refactor grpc API 2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Chris Lu
cca62fdb30 mount: streaming renaming folders 2021-10-17 04:22:42 -07:00
Chris Lu
004e56c1a6 save updated lastTsNs 2021-10-17 02:01:20 -07:00
Chris Lu
0a856241fe avoid int bigger than math.MaxInt32 
fix https://github.com/chrislusf/seaweedfs/issues/2363
 2021-10-07 21:12:57 -07:00
Chris Lu
225b019fe0 stream read multiple volumes in a volume server 2021-09-27 02:51:31 -07:00
Chris Lu
c4d7ee6c5c volume server: read all files in a volume 2021-09-27 01:45:32 -07:00
Chris Lu
52fe86df45 use default 10000 for grpc port 2021-09-20 14:05:59 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu
889a39a24f ipv6 2021-09-07 02:30:25 -07:00
Chris Lu
2b1feb732c remote.cache supports replication 2021-09-06 18:30:44 -07:00
Chris Lu
c218ef20c7 filer.remote.sync: automatically detect the primary remote storage 2021-09-06 15:10:55 -07:00