Konstantin Lebedev
|
3c75479e2b
|
Merge branch 'master' into gentle_vacuum
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
|
2022-08-01 14:45:22 +05:00 |
|
chrislu
|
afef014832
|
scaffold
|
2022-07-31 13:23:44 -07:00 |
|
chrislu
|
13b9a52f80
|
fix deprecated functions
|
2022-07-29 01:34:39 -07:00 |
|
chrislu
|
26dbc6c905
|
move to https://github.com/seaweedfs/seaweedfs
|
2022-07-29 00:17:28 -07:00 |
|
Konstantin Lebedev
|
2f0dda384d
|
vacuum show LA
|
2022-07-29 11:59:33 +05:00 |
|
chrislu
|
de3afd4199
|
fix rebasing
|
2022-07-28 23:25:46 -07:00 |
|
chrislu
|
06cd491abc
|
add message batch
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
7576c244c4
|
add flatbuffer serde for message
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
aa4a22ad47
|
segment serde
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
9f479aab98
|
allocate brokers to serve segments
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
94b8c42b2c
|
clean up
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
8060fdcac5
|
remove old code
|
2022-07-28 23:24:38 -07:00 |
|
chrislu
|
f25e273e32
|
display data center and rack in cluster.ps
|
2022-07-28 23:22:52 -07:00 |
|
chrislu
|
68065128b8
|
add dc and rack
|
2022-07-28 23:22:51 -07:00 |
|
chrislu
|
21b6b07dd8
|
renaming
|
2022-07-28 23:22:06 -07:00 |
|
chrislu
|
64f3d6fb6e
|
metadata subscription uses client epoch
|
2022-07-23 10:50:28 -07:00 |
|
chrislu
|
449ecc8890
|
remove max connection age
related to https://github.com/chrislusf/seaweedfs/pull/3226
|
2022-06-29 02:44:12 -07:00 |
|
chrislu
|
530da0cc89
|
remove unused code
|
2022-06-22 00:34:25 -07:00 |
|
chrislu
|
41dfe27102
|
Revert "remove max connection age"
This reverts commit b9b684194f.
|
2022-06-20 14:49:39 -07:00 |
|
chrislu
|
1d0c53ea56
|
remote storage: stop supporting hdfs as a remote storage
|
2022-06-20 14:15:59 -07:00 |
|
石昌林
|
3dd60529c5
|
some code optimizations
|
2022-06-17 19:07:39 +08:00 |
|
石昌林
|
78b3728169
|
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
|
2022-06-15 21:07:55 +08:00 |
|
chrislu
|
d12f431d98
|
collect volume server status
|
2022-06-12 11:56:23 -07:00 |
|
Chris Lu
|
f43ec9f363
|
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud
|
2022-06-07 08:56:55 -07:00 |
|
Konstantin Lebedev
|
c07820178f
|
fix s3 tests
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
|
2022-06-07 14:43:10 +05:00 |
|
chrislu
|
4fd5f96598
|
filer: remove replication, collection, disk_type info from entry metadata
these metadata can change and are not used
|
2022-06-06 00:39:35 -07:00 |
|
creeew
|
02ae102731
|
fix filer.sync missing source srv uploaded files to target when target down
|
2022-06-02 01:28:47 +08:00 |
|
chrislu
|
682382648e
|
collect cluster node start time
|
2022-05-30 16:23:52 -07:00 |
|
chrislu
|
a2b101a737
|
subscribe metadata between a range
|
2022-05-30 15:04:19 -07:00 |
|
chrislu
|
ca16fbf0ef
|
shell: cluster.ps display filer group
|
2022-05-29 16:37:14 -07:00 |
|
Chris Lu
|
8efe55f981
|
Merge pull request #3059 from guol-fnst/avoid_dup_vol
avoid loading duplicated volume directory
|
2022-05-17 23:48:57 -07:00 |
|
guol-fnst
|
8e2426a879
|
revert changes of pb file
orignal protoc-gen-go-grpc verion is 1.1.0
|
2022-05-18 11:52:14 +08:00 |
|
guol-fnst
|
b12944f9c6
|
fix naming convention
notify volume server of duplicate directoris
improve searching efficiency
|
2022-05-17 15:41:49 +08:00 |
|
guol-fnst
|
de6aa9cce8
|
avoid duplicated volume directory
|
2022-05-16 19:33:51 +08:00 |
|
chrislu
|
866fed1eab
|
avoid grpc name conflict
fix https://github.com/chrislusf/seaweedfs/issues/3055
|
2022-05-15 21:31:29 -07:00 |
|
chrislu
|
29198720f2
|
s3: add grpc server to accept configuration changes
|
2022-05-15 00:43:37 -07:00 |
|
chrislu
|
8bc16f1149
|
rename
|
2022-05-06 23:38:22 -07:00 |
|
chrislu
|
94635e9b5c
|
filer: add filer group
|
2022-05-01 21:59:16 -07:00 |
|
chrislu
|
4bd6bea429
|
fix test
|
2022-05-01 09:02:01 -07:00 |
|
Konstantin Lebedev
|
1e35b4929f
|
shell vacuum volume by collection and volume id
|
2022-04-18 18:40:58 +05:00 |
|
chrislu
|
b4be56bb3b
|
add timing info during ping operation
|
2022-04-16 12:45:49 -07:00 |
|
Konstantin Lebedev
|
35bc67f030
|
fix conflicts
|
2022-04-07 18:52:13 +05:00 |
|
Konstantin Lebedev
|
f5246b748d
|
Merge branch 'new_master' into hashicorp_raft
# Conflicts:
# weed/pb/master_pb/master.pb.go
|
2022-04-07 18:50:27 +05:00 |
|
Konstantin Lebedev
|
85d80fd36d
|
fix removing old raft server
|
2022-04-07 15:31:37 +05:00 |
|
Konstantin Lebedev
|
357aa818fe
|
add raft shell cmds
|
2022-04-06 15:23:53 +05:00 |
|
chrislu
|
bc888226fc
|
erasure coding: tracking encoded/decoded volumes
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
|
2022-04-05 19:03:02 -07:00 |
|
chrislu
|
958f880b70
|
mount: add grpc method to adjust quota
|
2022-04-02 15:14:37 -07:00 |
|
chrislu
|
daca2d22a5
|
use original server address string as map key
|
2022-04-01 17:34:42 -07:00 |
|
chrislu
|
bbbbbd70a4
|
master supports grpc ping
|
2022-04-01 16:50:58 -07:00 |
|
chrislu
|
743ad690b6
|
filer supports grpc ping
|
2022-04-01 16:44:58 -07:00 |
|
chrislu
|
800cbc004c
|
volume server adds ping function
|
2022-04-01 16:37:06 -07:00 |
|
chrislu
|
21e0898631
|
refactor: change masters from a slice to a map
|
2022-03-26 13:33:17 -07:00 |
|
chrislu
|
a0bad1c997
|
remove any go mod changes
This reverts commit 6c7f7d6887.
|
2022-03-21 23:04:00 -07:00 |
|
chrislu
|
6c7f7d6887
|
Revert "Merge pull request #2782 from SadmiB/upstream"
This reverts commit a644b7236a, reversing
changes made to 349257f822.
|
2022-03-21 23:00:50 -07:00 |
|
Chris Lu
|
0d692a6254
|
Merge branch 'master' into upstream
|
2022-03-21 11:02:04 -07:00 |
|
SadmiB
|
d12540c9f2
|
Add contabo api client
|
2022-03-21 17:16:49 +01:00 |
|
chrislu
|
4042fdf3bb
|
rename to skipCheckParentDir
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
|
2022-03-16 23:55:31 -07:00 |
|
chrislu
|
202a29d014
|
refactoring
|
2022-02-25 01:17:26 -08:00 |
|
chrislu
|
91d6785cf3
|
define metadata action types
|
2022-02-25 00:54:16 -08:00 |
|
chrislu
|
be3fc77391
|
mount2: use consistent inode
|
2022-02-25 00:53:27 -08:00 |
|
chrislu
|
ceaf993a27
|
mount2: add rdev
|
2022-02-24 14:51:25 -08:00 |
|
chrislu
|
b9b684194f
|
remove max connection age
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
|
2022-01-27 02:28:22 -08:00 |
|
chrislu
|
b1063162b6
|
display bucket quota
|
2022-01-21 00:55:04 -08:00 |
|
chrislu
|
fec8428fd8
|
POSIX: different inode for same named different file types
|
2022-01-12 11:51:13 -08:00 |
|
chrislu
|
6a12520a96
|
fix logging
|
2022-01-10 01:00:11 -08:00 |
|
chrislu
|
5c87fcc6d2
|
add client id for all metadata listening clients
|
2021-12-30 00:23:57 -08:00 |
|
chrislu
|
9f9ef1340c
|
use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
|
2021-12-26 00:15:03 -08:00 |
|
banjiaojuhao
|
083bf3a137
|
filer server: add "datacenter, rack and datanode" for path specific configuration
|
2021-12-23 23:25:05 +08:00 |
|
banjiaojuhao
|
08336be92e
|
filer server: allow upload file to specific dataNode
|
2021-12-22 21:57:26 +08:00 |
|
chrislu
|
a2d3f89c7b
|
add lock messages
|
2021-12-10 13:24:38 -08:00 |
|
chrislu
|
b70cb3e0b2
|
upgrade protoc to 3.17.3
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
|
2021-12-05 00:42:25 -08:00 |
|
Chris Lu
|
e0fc2898e9
|
auto updated filer peer list
|
2021-11-06 14:23:35 -07:00 |
|
Chris Lu
|
4b9c42996a
|
refactor grpc API
|
2021-11-05 18:11:40 -07:00 |
|
Chris Lu
|
5ea86ef1da
|
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
This reverts commit af71ae11aa.
|
2021-11-05 17:52:15 -07:00 |
|
Chris Lu
|
af71ae11aa
|
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
|
2021-11-03 01:09:48 -07:00 |
|
Chris Lu
|
5160eb08f7
|
shell: optionally read filer address from master
|
2021-11-02 23:38:45 -07:00 |
|
Chris Lu
|
24858507cc
|
rename API to avoid confusion
|
2021-10-30 19:27:25 -07:00 |
|
Chris Lu
|
5435027ff0
|
volume copy: stream out copying progress and avoid grpc request timeout
fix https://github.com/chrislusf/seaweedfs/issues/2386
|
2021-10-24 02:52:56 -07:00 |
|
Chris Lu
|
3be3c17f59
|
volume vacuum: avoid timeout with streaming progress report
fix https://github.com/chrislusf/seaweedfs/issues/2396
|
2021-10-24 01:55:34 -07:00 |
|
Chris Lu
|
cca62fdb30
|
mount: streaming renaming folders
|
2021-10-17 04:22:42 -07:00 |
|
Chris Lu
|
004e56c1a6
|
save updated lastTsNs
|
2021-10-17 02:01:20 -07:00 |
|
Chris Lu
|
0a856241fe
|
avoid int bigger than math.MaxInt32
fix https://github.com/chrislusf/seaweedfs/issues/2363
|
2021-10-07 21:12:57 -07:00 |
|
Chris Lu
|
225b019fe0
|
stream read multiple volumes in a volume server
|
2021-09-27 02:51:31 -07:00 |
|
Chris Lu
|
c4d7ee6c5c
|
volume server: read all files in a volume
|
2021-09-27 01:45:32 -07:00 |
|
Chris Lu
|
52fe86df45
|
use default 10000 for grpc port
|
2021-09-20 14:05:59 -07:00 |
|
Chris Lu
|
2789d10342
|
go fmt
|
2021-09-14 10:37:06 -07:00 |
|
Chris Lu
|
e5fc35ed0c
|
change server address from string to a type
|
2021-09-12 22:47:52 -07:00 |
|
Chris Lu
|
0128239c0f
|
handle ipv6 addresses
|
2021-09-07 16:43:54 -07:00 |
|
Chris Lu
|
889a39a24f
|
ipv6
|
2021-09-07 02:30:25 -07:00 |
|
Chris Lu
|
2b1feb732c
|
remote.cache supports replication
|
2021-09-06 18:30:44 -07:00 |
|
Chris Lu
|
c218ef20c7
|
filer.remote.sync: automatically detect the primary remote storage
|
2021-09-06 15:10:55 -07:00 |
|
Chris Lu
|
0cda61d539
|
return error if not found
|
2021-09-05 23:43:44 -07:00 |
|
Chris Lu
|
e93d4935e3
|
add other replica locations when assigning volumes
|
2021-09-05 23:32:25 -07:00 |
|
Chris Lu
|
0652805236
|
cloud drive: add createBucket() deleteBucket()
|
2021-09-03 22:30:55 -07:00 |
|
Chris Lu
|
d1a4e19a3f
|
volume: copy file also copies modification time
to ensure ttl can work well
|
2021-09-01 02:42:57 -07:00 |
|
Chris Lu
|
8e125339d5
|
line wrap
|
2021-08-31 23:30:28 -07:00 |
|
Chris Lu
|
68ea99190f
|
add additionalPathPrefixes
|
2021-08-31 23:26:26 -07:00 |
|
Chris Lu
|
43fd11278e
|
support follow additional path prefixes
|
2021-08-31 23:23:08 -07:00 |
|
Chris Lu
|
1dfcdc53b7
|
cloud drive: add storj, filebase
|
2021-08-30 17:43:14 -07:00 |
|
Chris Lu
|
701cc133f3
|
cloud drive: s3 add option for v4 signature
|
2021-08-30 17:28:33 -07:00 |
|
Chris Lu
|
05f32376eb
|
add cluster id in filer configuration response
|
2021-08-29 21:02:10 -07:00 |
|
Chris Lu
|
001a472057
|
cloud mount: remote storage support hdfs
|
2021-08-29 18:41:29 -07:00 |
|
Chris Lu
|
a22f37b01c
|
remove tencent region
|
2021-08-28 02:23:03 -07:00 |
|
Chris Lu
|
05a648bb96
|
refactor: separating out remote.proto
|
2021-08-26 15:18:34 -07:00 |
|
Chris Lu
|
c08ac536ed
|
cloud drive: add support for Wasabi
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
|
2021-08-25 17:34:29 -07:00 |
|
Chris Lu
|
a7a914f120
|
cloud drive: add support for Baidu BOS
|
2021-08-24 23:46:33 -07:00 |
|
Chris Lu
|
a19c728034
|
cloud drive: add support for Tencent COS
|
2021-08-24 23:19:45 -07:00 |
|
Chris Lu
|
19a81d25af
|
cloud drive: add support for Aliyun OSS
|
2021-08-24 23:14:24 -07:00 |
|
Chris Lu
|
47d775cf68
|
cloud drive: add support for BackBlaze
|
2021-08-24 22:30:06 -07:00 |
|
Chris Lu
|
e9ebe24f2e
|
cloud drive: add support for Azure
|
2021-08-24 01:18:30 -07:00 |
|
Chris Lu
|
00c4e06caa
|
cloud drive: s3 configurable force path style
|
2021-08-23 03:30:41 -07:00 |
|
Chris Lu
|
2836a58d87
|
cloud drive: S3 supports storage class
|
2021-08-23 02:18:59 -07:00 |
|
Chris Lu
|
258063de26
|
cloud drive: add google cloud storage
|
2021-08-23 00:29:27 -07:00 |
|
Chris Lu
|
3ada61c875
|
rename
|
2021-08-14 21:50:35 -07:00 |
|
Chris Lu
|
c34747c79d
|
rename, fix wrong logic.
|
2021-08-14 21:46:34 -07:00 |
|
Chris Lu
|
08258a819d
|
fix mistake
|
2021-08-14 05:10:30 -07:00 |
|
Chris Lu
|
4909bd9684
|
gRpc connection error on filer when no volume left #2243
fix https://github.com/chrislusf/seaweedfs/issues/2243
grpc do not cache connections only when connection problem happens.
Normal error results should not close the shared grpc connection.
|
2021-08-14 05:06:44 -07:00 |
|
Chris Lu
|
5a0f92423e
|
use grpc and jwt
|
2021-08-12 21:40:33 -07:00 |
|
Chris Lu
|
5571f4f70a
|
master: add master.follower to handle read file id lookup requests
|
2021-08-12 18:10:59 -07:00 |
|
Chris Lu
|
69655ba8e5
|
mount: cache on reading remote storage
|
2021-08-09 22:11:57 -07:00 |
|
Chris Lu
|
713c035a6e
|
shell: remote.cache remote.uncache
|
2021-08-09 14:35:18 -07:00 |
|
Chris Lu
|
c0b12da4ef
|
shell: add filer.remote.unmount
|
2021-08-08 22:26:37 -07:00 |
|
Chris Lu
|
eed26af266
|
Merge branch 'master' into add_remote_storage
|
2021-08-08 15:48:04 -07:00 |
|
Chris Lu
|
4370a4db63
|
use int64 for volume count in case of negative overflow
|
2021-08-08 15:19:39 -07:00 |
|
Chris Lu
|
270770d7d7
|
refactor
|
2021-08-07 14:18:53 -07:00 |
|
Chris Lu
|
6b743dbbf9
|
refactor client subscribe metadata
|
2021-08-04 16:25:46 -07:00 |
|
Chris Lu
|
9df7d16791
|
read <- remote_storage
|
2021-07-31 22:39:38 -07:00 |
|
Chris Lu
|
899963ac20
|
remote storage location changed to struct
|
2021-07-29 02:08:55 -07:00 |
|
Chris Lu
|
1752eeb538
|
remote.mount saves the mapping
|
2021-07-27 03:26:35 -07:00 |
|
Chris Lu
|
4b94b03d90
|
directory to remote storage mapping
|
2021-07-27 01:16:28 -07:00 |
|
Chris Lu
|
99b599aa8a
|
remote.mount
|
2021-07-26 22:53:44 -07:00 |
|
Chris Lu
|
70effac0d3
|
configure and store remote configurations
|
2021-07-21 02:24:34 -07:00 |
|
Chris Lu
|
450222dd64
|
add remote to filer.Entry and filer_pb entry, add RemoteConf
|
2021-07-19 02:47:27 -07:00 |
|
Chris Lu
|
8dc5def435
|
proto: add remote
|
2021-07-19 01:12:31 -07:00 |
|
Chris Lu
|
b194f91f47
|
add version to filer configuration response
|
2021-07-12 01:23:20 -07:00 |
|
lyg
|
1a4db87e19
|
fix s3 metadata error with multipart upload
|
2021-07-02 11:00:42 +08:00 |
|
Chris Lu
|
215b169562
|
mount: recursively rename locally
|
2021-07-01 01:19:31 -07:00 |
|
Chris Lu
|
ab606dec2a
|
filer: add path-specific option to enforce readonly
|
2021-06-04 01:03:41 -07:00 |
|
Chris Lu
|
c5de97ddbe
|
keep alive for streaming connections
fix https://github.com/chrislusf/seaweedfs/issues/2096
related to https://github.com/chrislusf/seaweedfs/issues/1277
|
2021-05-31 01:26:24 -07:00 |
|
Chris Lu
|
1456616a77
|
recreate grpc connections if too many errors
address https://github.com/chrislusf/seaweedfs/issues/2098
|
2021-05-30 00:07:43 -07:00 |
|
Chris Lu
|
3d3fa43542
|
filer: re-create grpc connections if having transport error
fix https://github.com/chrislusf/seaweedfs/issues/2070
|
2021-05-22 17:46:53 -07:00 |
|
Chris Lu
|
dc1309f084
|
FUSE mount: support multiple filers
fix https://github.com/chrislusf/seaweedfs/issues/2015
fix https://github.com/chrislusf/seaweedfs/issues/1531
|
2021-05-21 01:28:00 -07:00 |
|
Chris Lu
|
f0ad172e80
|
shell: show which server holds the lock
fix https://github.com/chrislusf/seaweedfs/issues/1983
|
2021-04-22 23:56:35 -07:00 |
|
Chris Lu
|
b465095db1
|
shell: add volume.check.disk to fix inconsistency for replicated volumes
fix https://github.com/chrislusf/seaweedfs/issues/1923
|
2021-03-22 00:03:16 -07:00 |
|
Chris Lu
|
b1a86cf808
|
s3: copy object to itself
fix https://github.com/chrislusf/seaweedfs/issues/1922
|
2021-03-19 01:31:49 -07:00 |
|
Chris Lu
|
35d939a1d2
|
go fmt
|
2021-03-14 13:21:02 -07:00 |
|
wuh-fnst
|
4c1d945e46
|
make List correctly judge whether it is the last file
|
2021-03-12 11:22:26 +08:00 |
|
Chris Lu
|
828f6e9f4d
|
volume: auto add missing vif files
fix https://github.com/chrislusf/seaweedfs/issues/1878
|
2021-03-09 12:09:32 -08:00 |
|
Chris Lu
|
9d402ebe9f
|
refactoring
|
2021-03-06 14:26:27 -08:00 |
|
Chris Lu
|
bd727b7b53
|
go fmt
|
2021-03-02 20:59:56 -08:00 |
|