石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
石昌林
78b3728169
add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations
...
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply
# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply
# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply
# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply
# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
2022-06-15 21:07:55 +08:00
chrislu
d12f431d98
collect volume server status
2022-06-12 11:56:23 -07:00
shichanglin5
a63559ee6c
fix: fix command prompt information
2022-06-10 12:46:10 +08:00
chrislu
4fd5f96598
filer: remove replication, collection, disk_type info from entry metadata
...
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
chrislu
f97acdd489
volume.fix.replication fix retry logic
...
fix https://github.com/chrislusf/seaweedfs/issues/3136
2022-06-03 08:45:29 -07:00
chrislu
6793bc853c
help message when in simulation mode
2022-05-31 14:48:46 -07:00
chrislu
89948a373b
fix error reporting for "Need to a hdd disk type"
...
related to https://github.com/chrislusf/seaweedfs/issues/3128
2022-05-31 12:43:55 -07:00
chrislu
5b9347c938
typo
2022-05-30 12:37:41 -07:00
chrislu
ca16fbf0ef
shell: cluster.ps display filer group
2022-05-29 16:37:14 -07:00
Konstantin Lebedev
44f53ceda6
fix collectionIsMismatch charset
2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2
fix skip loop
2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c
avoid delete volume replica if collection mismatch
2022-05-16 13:07:05 +05:00
chrislu
271b5aed96
shell: volume.fsck add a note for -reallyDeleteFromVolume option
2022-05-15 11:07:04 -07:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
guosj
1a5de36257
fix return value
2022-04-29 11:19:36 +08:00
Chris Lu
b557faf175
Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid
...
volume.check.disk add param volumeId
2022-04-26 22:09:06 -07:00
Konstantin Lebedev
d4343ab7da
forcePurging desc
2022-04-25 23:11:56 +05:00
Konstantin Lebedev
ae56b2c00f
change forcePurging to a pointer
2022-04-25 23:10:01 +05:00
Konstantin Lebedev
6d2fda27d2
delete missing data from volumes in one replica
2022-04-25 22:59:46 +05:00
Konstantin Lebedev
67476d830a
volume.check.disk add param volumeId
2022-04-25 18:27:14 +05:00
Chris Lu
88a669dd19
Merge pull request #2858 from kmlebedev/fsck_replicas
...
fsck replicas
2022-04-19 00:17:43 -07:00
Konstantin Lebedev
2364fab927
volume.list show only readonly, collectionPattern and volumeId
2022-04-19 11:44:41 +05:00
Konstantin Lebedev
36c5a59ed8
add help
2022-04-18 19:36:14 +05:00
Konstantin Lebedev
1e35b4929f
shell vacuum volume by collection and volume id
2022-04-18 18:40:58 +05:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc
2022-04-18 10:39:29 +08:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
chrislu
460d56d283
shell: cluster.check prints out clock delta and network latency
2022-04-16 13:24:17 -07:00
chrislu
1f03fcccb1
fix nil in cluster_check shell command
...
fix https://github.com/chrislusf/seaweedfs/issues/2905
2022-04-12 08:47:27 -07:00
Konstantin Lebedev
76b1c5ce67
cleanupUploads use jwt token
2022-04-12 13:46:11 +05:00
Konstantin Lebedev
2dfbe210e0
avoid breaking loop in cleanupUploads if error is empty
2022-04-11 11:16:20 +05:00
Konstantin Lebedev
931cb9e581
use "cluster.raft.{ps,add,remove}"
2022-04-11 10:50:01 +05:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
Konstantin Lebedev
85d80fd36d
fix removing old raft server
2022-04-07 15:31:37 +05:00
Konstantin Lebedev
357aa818fe
add raft shell cmds
2022-04-06 15:23:53 +05:00
chrislu
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
chrislu
4aae87f405
check missing hdd disk type
2022-04-04 14:48:00 -07:00
chrislu
6a2bcd03aa
configure mount quota
2022-04-02 21:34:26 -07:00
chrislu
105578a2f2
skip pinging self for master and volume server
2022-04-01 20:25:35 -07:00
chrislu
4ecba915f3
add check between peers
2022-04-01 17:40:25 -07:00
chrislu
4b5c0e3fa9
check cluster connectivities
2022-04-01 17:27:49 -07:00
Konstantin Lebedev
7f1383a41e
findExtraChunksInVolumeServers in consideration of replication
2022-04-01 14:45:41 +05:00
Konstantin Lebedev
3817e05dd0
fix collect filer files
2022-04-01 10:17:09 +05:00
Konstantin Lebedev
3cedb21bb7
skip new entities
2022-03-31 21:36:10 +05:00
Konstantin Lebedev
1d9e30d8c0
fsck replicas
2022-03-31 19:10:06 +05:00
Konstantin Lebedev
4f5b018a02
collecting only bucket volumes
2022-03-31 13:35:58 +05:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
banjiaojuhao
f7f2a597dd
minor
2022-03-08 16:22:55 +08:00
chrislu
ede6ce44c6
fix test
2022-03-06 17:09:55 -08:00
chrislu
6e49e75a5b
use logical number of files and sizes for statistics and quota
2022-03-04 18:47:44 -08:00
chrislu
95717d1006
simpler output
2022-02-24 13:50:08 -08:00
Chris Lu
6834df77a0
Merge pull request #2701 from guo-sj/fix_bugs_in_return_value
2022-02-23 19:23:37 -08:00
guosj
26f3ab8d4b
fix a return bug in func (c *commandVacuum) Do
2022-02-24 09:04:38 +08:00
Chris Lu
38dcaaa76e
Merge pull request #2683 from guo-sj/fix_fsconfigure_bug
...
fix minor bug in commandFsConfigure.Do
2022-02-16 18:16:47 -08:00
guosj
ca121ecd66
fix minor bug in commandFsConfigure.Do
2022-02-17 09:57:52 +08:00
Chris Lu
64afbc5235
Merge pull request #2640 from Radtoo/fsck_with_delete
...
Added basic deletion capabilities to fsck.
2022-02-14 23:00:01 -08:00
Radtoo
724ce04b1c
We picked the second flag's name.
2022-02-13 00:53:35 +01:00
Radtoo
fbe7ed7927
Revert the the flag to "reallyDeleteFromVolume".
2022-02-10 20:01:48 +01:00
chrislu
21aaa4c1f1
ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount)
...
fix https://github.com/chrislusf/seaweedfs/issues/2642
2022-02-08 01:51:13 -08:00
chrislu
13f6ec1c4e
test checking ec distribution
2022-02-08 01:50:05 -08:00
chrislu
f34c2ff7c5
use embed txt
2022-02-08 01:14:27 -08:00
chrislu
c8c7c10c3f
volume.tier.move: avoid double counting
...
related to https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:57:35 -08:00
chrislu
f18803424a
volume.balance: add delay during tight loop
...
fix https://github.com/chrislusf/seaweedfs/issues/2637
2022-02-08 00:53:55 -08:00
chrislu
85c1615b43
filer read empty file may cause OOM in some cases
...
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
Radtoo
fbb14e0ea8
Adding separate toggle to purge absent vols
...
While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected.
Also renamed forcePurge to just purge.
2022-02-06 23:46:52 +01:00
Radtoo
6b17f45da2
Also delete paths for which a volume is entirely absent.
2022-02-06 23:22:04 +01:00
Radtoo
fa0cfdfb7f
Added basic deletion capabilities to fsck.
2022-02-06 17:05:58 +01:00
chrislu
6bee1e9714
[volume.check.disk] fix wrong logic to compare
...
fix https://github.com/chrislusf/seaweedfs/issues/2626
minuend - subtrahend
2022-02-01 07:48:28 -08:00
chrislu
b8490fe427
adjust volume count even when not applying the changes
2022-01-28 19:11:46 -08:00
divanikus
67e3fe996a
async volumeTierMove
2022-01-26 18:22:31 +03:00
chrislu
9d0f58c329
skip printing fs configuration
2022-01-21 13:29:47 -08:00
chrislu
ce2049cdb6
refactoring, move genFn before saveFn
2022-01-21 12:08:58 -08:00
chrislu
e47f63d159
enforce bucket quota
2022-01-21 02:34:42 -08:00
chrislu
6e57d8d0de
s3: check bucket usage and adjust read only according to quota
2022-01-21 02:15:27 -08:00
chrislu
606667f205
able to configure the quota for a bucket
2022-01-21 01:42:20 -08:00
chrislu
b1063162b6
display bucket quota
2022-01-21 00:55:04 -08:00
chrislu
f103491912
s3: list bucket size from weed shell
2022-01-21 00:26:49 -08:00
chrislu
ea57654e34
refactoring
2022-01-21 00:16:50 -08:00
chrislu
15c01d8b7f
add some notes
2022-01-12 15:04:48 -08:00
chrislu
107a4884a8
shell: tighter memory allocation
2022-01-12 14:59:29 -08:00
chrislu
adfd54e7c4
fix compilation
2022-01-12 01:24:24 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu
60dc450091
skip fixing read only volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
chrislu
94e5c0a454
skip purging from ec volumes
2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b
add lock messages
2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d
rename file
2021-12-10 13:11:00 -08:00
chrislu
7f0a97c7b6
shell: volume.list supports different verbosity level
2021-12-05 21:54:40 -08:00
chrislu
e6c026db65
volume.fix.replication: fix misplaced volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2416
2021-12-05 16:56:25 -08:00
chrislu
53e2dee177
[volume.check.disk] was using the wrong source and target locations
...
fix https://github.com/chrislusf/seaweedfs/issues/2268
2021-12-05 14:32:04 -08:00
Chris Lu
f3a334965d
Update command_remote_uncache.go
...
minor
2021-11-28 23:10:32 -08:00
Chris Lu
9ccfc1cfee
Update command_volume_fsck.go
...
minor
2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9
Update command_volume_fsck.go
...
remove unused code
2021-11-28 22:37:11 -08:00
user
9668b15f38
Filtering by volume id is prioritized.
2021-11-11 18:18:56 +09:00
user
563a74a9eb
Volume filter by collection pattern added.
2021-11-11 18:01:47 +09:00
user
dbb8003ce3
Volume filter function added.
2021-11-11 17:45:17 +09:00
user
c387fe957b
Unused parameter removed.
2021-11-11 16:08:41 +09:00
Chris Lu
4729a57cc0
use constants
2021-11-08 17:47:56 -08:00
Chris Lu
a8b0f8864d
fix help message
2021-11-07 12:38:35 -08:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
2021-11-05 17:52:15 -07:00
Chris Lu
314c32514b
add command to list filers
2021-11-05 17:39:41 -07:00
Chris Lu
00ae965d8d
randomize a bit for ec shards distribution
2021-11-04 09:23:40 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62
better printout
2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc
add one unit test
2021-11-02 23:36:43 -07:00
Chris Lu
794375ca0a
adjust help message since both fullPercent and quietFor are needed.
2021-11-01 17:22:47 -07:00
Chris Lu
24858507cc
rename API to avoid confusion
2021-10-30 19:27:25 -07:00
Chris Lu
d774fa6c9a
rename variable
2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62
fix compilation
2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589
erasure coding: skip erasure coding if less than recommended 4 nodes
2021-10-25 14:38:11 -07:00
Chris Lu
5435027ff0
volume copy: stream out copying progress and avoid grpc request timeout
...
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-24 02:52:56 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Konstantin Lebedev
be4b3ed509
AclHandlers
2021-10-11 15:03:56 +05:00
Chris Lu
e862b2529a
refactor
2021-10-01 12:10:11 -07:00
Konstantin Lebedev
5e64b22b45
check that the topology has been updated
2021-10-01 18:51:22 +05:00
Konstantin Lebedev
2cecde89c3
rename opt volumesPerStep
2021-10-01 00:17:54 +05:00
Konstantin Lebedev
fc51ffce2b
https://github.com/chrislusf/seaweedfs/issues/1846
2021-09-30 20:24:24 +05:00
Chris Lu
ad5099e570
refactor
2021-09-19 12:02:23 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
119d5908dd
shell: do not need to lock to see volume -h
2021-09-13 22:13:34 -07:00
Chris Lu
6cd1ce8b74
erasure coding: add cleanup step if anything goes wrong
2021-09-13 01:55:49 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
0b4269b6a8
remember commands even if failed
2021-09-08 15:55:19 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
Chris Lu
8e4b43a017
minor
2021-09-06 15:13:38 -07:00
Chris Lu
57a95887d2
remote.cache remote.uncache supports all mounted directories
2021-09-05 14:47:06 -07:00
Chris Lu
c735608685
obfuscate secret keys on display
2021-09-05 14:23:49 -07:00
Chris Lu
32e94de86a
default auto trimming suffix to true
2021-09-05 13:27:35 -07:00
Chris Lu
60573fd3e2
option to map remote bucket to trimmed bucket name
2021-09-05 11:55:52 -07:00
Chris Lu
d57d4c5f8f
shell: add remote.mount.buckets
2021-09-04 21:37:25 -07:00
Chris Lu
d983aa4c7d
correct filtering
2021-09-04 13:58:14 -07:00
Chris Lu
49952ba905
fix cache/uncache filters
2021-09-04 13:45:39 -07:00
Chris Lu
df29281536
rename
2021-09-04 05:11:06 -07:00
Chris Lu
c3db389e42
refactor
2021-09-04 05:09:38 -07:00
Chris Lu
8707ef00a6
refactor
2021-09-04 05:02:20 -07:00
Chris Lu
63c84584bb
refactor
2021-09-04 04:53:36 -07:00
Chris Lu
da49d25950
auto list of storage types
2021-09-04 00:18:21 -07:00
Chris Lu
38f73c8324
add gcs project id
2021-09-03 23:38:33 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29
filer.remote.sync: exit when directory is unmounted
...
this will not propagate the deletions back to the cloud
2021-09-01 01:29:22 -07:00
Chris Lu
cca588429b
shell: fs.meta.save skip saving system logs
2021-08-31 23:05:56 -07:00
Chris Lu
0ee208c1a4
adjust example
2021-08-31 02:04:32 -07:00
Chris Lu
97b7948cd9
adjust error message
2021-08-31 01:07:34 -07:00
Chris Lu
c9205458a1
cloud drive: create mount directory if not exists
2021-08-31 01:07:19 -07:00
Chris Lu
1dfcdc53b7
cloud drive: add storj, filebase
2021-08-30 17:43:14 -07:00
Chris Lu
701cc133f3
cloud drive: s3 add option for v4 signature
2021-08-30 17:28:33 -07:00
Chris Lu
f117f7bf54
adjust comment
2021-08-30 15:56:14 -07:00
