Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount ( #3594 )
...
https://github.com/seaweedfs/seaweedfs/issues/3593
2022-09-07 09:50:17 -07:00
chrislu
570b488572
ensure memory alignment on different CPUs
2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount ( #3537 )
...
https://github.com/seaweedfs/seaweedfs/issues/3511
2022-08-29 13:23:02 -07:00
chrislu
6f483a4d36
atomic add
...
fix https://github.com/seaweedfs/seaweedfs/issues/3514
2022-08-25 22:20:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races ( #3503 )
2022-08-24 09:49:05 -07:00
Konstantin Lebedev
6199d9b923
rand start garbageThreshold and reset metric ReplicaPlacementMismatch ( #3396 )
...
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch
* vacuum through sleep
2022-08-12 13:11:09 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite
...
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 13:31:01 -07:00
Konstantin Lebedev
2a5629dd28
align
2022-08-02 01:29:44 +05:00
Konstantin Lebedev
df9cc31a0a
refactor
2022-08-02 01:18:26 +05:00
Konstantin Lebedev
0a5d3c3cf6
rename to localWrite and replicatedWrite
2022-08-01 23:09:09 +05:00
Konstantin Lebedev
fc65122766
rename to LoadAvg_1M
2022-08-01 21:32:21 +05:00
Konstantin Lebedev
ab50f21ad9
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 21:16:57 +05:00
Konstantin Lebedev
5209ebbeef
remove percent
2022-08-01 20:40:38 +05:00
Konstantin Lebedev
78cbd8002f
revert Sleep
2022-08-01 20:21:23 +05:00
Konstantin Lebedev
cd5c7ad052
move to github.com/seaweedfs/seaweedfs
2022-08-01 16:36:32 +05:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
Konstantin Lebedev
c0d92f61a1
comment
2022-08-01 14:40:42 +05:00
Konstantin Lebedev
1d29f67c02
revert disk stats
2022-08-01 14:29:41 +05:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
chrislu
3828b8ce87
"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"
2022-07-27 12:12:40 -07:00
ningfd
58f2dd6740
change topology mapping definition, improve spelling
2022-07-22 17:18:56 +08:00
ningfd
6f882eb354
optimiz: master ui will render data in order
2022-07-22 14:37:49 +08:00
chrislu
48382676d2
fix filtering by volume id
2022-07-08 10:29:24 -07:00
chrislu
a79f5bd0d5
adjust used size reporting
2022-07-04 13:46:32 -07:00
chrislu
3142e77eca
mount: stats report physical size instead of logical size
2022-06-26 12:14:34 -07:00
chrislu
96496d5286
master: broadcast new volume locations to clients to avoid possible racing condition
...
fix https://github.com/chrislusf/seaweedfs/issues/3220
2022-06-23 00:41:33 -07:00
zzq09494
9df5ad5309
fix: vacuum create a lot of connections quickly
2022-06-22 09:57:22 +08:00
Konstantin Lebedev
fb57aa431c
stats master_replica_placement_mismatch
2022-06-10 15:30:40 +05:00
guol-fnst
b12944f9c6
fix naming convention
...
notify volume server of duplicate directoris
improve searching efficiency
2022-05-17 15:41:49 +08:00
guol-fnst
de6aa9cce8
avoid duplicated volume directory
2022-05-16 19:33:51 +08:00
qiang.zeng
06cb337b82
fix typo
2022-05-09 14:21:16 +08:00
chrislu
00c1dfec4f
go fmt
2022-05-01 23:16:29 -07:00
Konstantin Lebedev
36c5a59ed8
add help
2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
chrislu
4524ceae6b
less verbose logs
...
fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085
2022-04-17 22:55:26 -07:00
Chris Lu
a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft
...
hashicorp raft
2022-04-10 23:00:05 -07:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
shibinbin
c20e1edd99
fix: master lose some volumes
2022-04-07 15:18:28 +08:00
chrislu
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
ae558fa073
log reasons volumes became unwritable
2022-03-21 00:41:44 -07:00
chrislu
57c6eddd22
avoid possible deadlock if volume layout is used in some logs
2022-03-21 00:04:01 -07:00
Konstantin Lebedev
9ea09cc41c
healthz check to avoid drain pod with last replicas
2022-02-16 14:18:36 +05:00
Konstantin Lebedev
0ed76a0556
clearly
2022-02-14 14:10:06 +05:00
Konstantin Lebedev
36013f63ed
https://github.com/chrislusf/seaweedfs/issues/2648
2022-02-14 13:59:12 +05:00
chrislu
433fde4b18
move error to a separate file
...
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Konstantin Lebedev
3f4e17aa24
error metrics for filer and store
2022-02-04 14:07:14 +05:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Chris Lu
b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode
2021-12-21 08:56:51 -08:00
banjiaojuhao
dda6b90d25
assign fileId according to DataNode with empty DataCenter and Rack
2021-12-21 17:28:33 +08:00
chrislu
5eacff9d4f
log message adds server name
...
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
2021-12-16 10:46:26 -08:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Chris Lu
e4830bd93d
go fmt
2021-10-07 21:13:31 -07:00
Chris Lu
332d49432d
reduce concurrent volume grow requests
2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00
refactor
2021-10-05 00:40:04 -07:00
Chris Lu
8a66306064
calculate disk usage in case of race condition
...
related to https://github.com/chrislusf/seaweedfs/issues/2357
2021-10-04 23:32:07 -07:00
Chris Lu
a067deaabc
avoid possible modified location list
...
fix issue 1 of https://github.com/chrislusf/seaweedfs/issues/2345
2021-09-28 16:54:18 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
574485ec69
better IP v6 support
2021-09-07 19:29:42 -07:00
Chris Lu
6923af7280
refactoring
2021-09-06 16:20:49 -07:00
Chris Lu
e93d4935e3
add other replica locations when assigning volumes
2021-09-05 23:32:25 -07:00
Chris Lu
7a13816e94
refactor
2021-09-05 23:17:15 -07:00
Chris Lu
65af3cf4df
master: disconnect only the phantom volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/2311
2021-09-05 15:20:03 -07:00
Chris Lu
78e8ddf910
Only when tailing volume, the zero-ed cookie should skip checking.
...
This only happens when checkCookie == false and fsync == false.
2021-08-13 02:09:35 -07:00
Chris Lu
d1d1fc772c
move some volume lookup operations to grpc
...
jwt related lookup will come in next commit
2021-08-12 20:33:00 -07:00
Chris Lu
01336d71eb
minor
2021-08-10 13:04:33 -07:00
Chris Lu
eed26af266
Merge branch 'master' into add_remote_storage
2021-08-08 15:48:04 -07:00
Chris Lu
4370a4db63
use int64 for volume count in case of negative overflow
2021-08-08 15:19:39 -07:00
Chris Lu
cb1dbd3135
refactor
2021-08-01 11:53:46 -07:00
Chris Lu
b624090398
go fmt
2021-07-01 01:21:14 -07:00
Chris Lu
d474ce6fe3
master: avoid repeated leader redirection
...
fix https://github.com/chrislusf/seaweedfs/issues/2146
2021-06-21 22:56:07 -07:00
Chris Lu
87a32bfef4
avoid possible nil when node is disconnected from its parent
...
fix https://github.com/chrislusf/seaweedfs/issues/2073
2021-05-19 10:02:01 -07:00
Chris Lu
d2d36a3f9d
master: avoid creating too many volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2062
2021-05-11 10:05:31 -07:00
Chris Lu
9a6aa00e9d
avoid nil locations
...
fix https://github.com/chrislusf/seaweedfs/issues/2059
2021-05-10 02:39:52 -07:00
qieqieplus
c4d32f6937
ahead of time volume assignment
2021-05-06 18:55:44 +08:00
Patrick Schmidt
7413d59750
Fix EC shard count logic
...
This fixes the calculation of the amount of EC shards a node holds.
Previously a global counter was increased, but also used inside the
loop to apply disk usage deltas. This led to wrong absolute numbers.
The fix is to apply only deltas of single EC shards per iteration.
2021-03-05 12:50:58 +01:00
Patrick Schmidt
5f7b024891
Show the real disk usage in stats calls
...
Currently the file size of only one volume location is taken into
account in the stats. This commit multiplies the disk usages by the
amount of nodes holding a replica of the volume.
This will yield the expected amount of disk usage and matches the
total size calculations from before.
2021-02-26 13:58:40 +01:00
Chris Lu
2270737344
volume: avoid fixed vacuum timeout for large volumes
...
1GB for 3 minutes, about 5.7MB/s
2021-02-22 12:52:37 -08:00
Chris Lu
565f7a6e72
Update data_node.go
2021-02-19 14:22:36 -08:00
Chris Lu
a37473ae60
add back volume ids
...
address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576
2021-02-19 14:22:12 -08:00
Chris Lu
c576ad04ac
fix volume server display for volumes
2021-02-19 01:38:56 -08:00
Chris Lu
73958e357d
add descriptive error if no free volumes
2021-02-18 19:10:20 -08:00
Chris Lu
3575d41009
go fmt
2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c
refactoring to get master function, instead of passing master values directly
...
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
68775d29e3
fix tests
2021-02-16 10:51:03 -08:00
Chris Lu
b314d78e97
fix print
2021-02-16 10:48:28 -08:00
Chris Lu
53ca7e66ef
avoid dead lock
2021-02-16 10:48:16 -08:00
Chris Lu
3097b9a9b7
fix existence checking
2021-02-16 05:59:43 -08:00
Chris Lu
cb9cc29518
volume.list display; fix updating maxVolumeCount for disk
2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
1102ae32c4
fix concurrent map reads
2021-01-31 18:26:26 -08:00
Chris Lu
9c9ba3c209
nil related
...
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
d9e8479c06
adjust UI max count
2020-12-17 13:47:51 -08:00
