Chris Lu
3097b9a9b7
fix existence checking
2021-02-16 05:59:43 -08:00
Chris Lu
cb9cc29518
volume.list display; fix updating maxVolumeCount for disk
2021-02-16 03:55:24 -08:00
Chris Lu
3fe628f04e
use hdd instead of empty string
2021-02-16 03:03:00 -08:00
Chris Lu
f8446b42ab
this can compile now!!!
2021-02-16 02:47:02 -08:00
Chris Lu
4bd8a692d8
disk type can be generic tags
2021-02-13 13:50:14 -08:00
Chris Lu
821c46edf1
Merge branch 'master' into support_ssd_volume
2021-02-09 11:37:07 -08:00
Chris Lu
1102ae32c4
fix concurrent map reads
2021-01-31 18:26:26 -08:00
Chris Lu
9c9ba3c209
nil related
...
related to https://github.com/chrislusf/seaweedfs/issues/1676
2021-01-03 12:25:58 -08:00
Chris Lu
d9e8479c06
adjust UI max count
2020-12-17 13:47:51 -08:00
Chris Lu
3cdf5945a2
adjust UI
2020-12-17 13:37:00 -08:00
Chris Lu
f696a2b2a7
assign volumes based on disk type
2020-12-17 13:25:05 -08:00
Chris Lu
1bf22c0b5b
go fmt
2020-12-16 09:14:05 -08:00
Chris Lu
94525aa0fd
allocate volume by disk type
2020-12-13 23:08:21 -08:00
Chris Lu
a9db24cd05
master allocate volumes if ssd type runs out
2020-12-13 19:44:57 -08:00
Chris Lu
0d2ec832e2
rename from volumeType to diskType
2020-12-13 11:59:32 -08:00
Chris Lu
715b199eeb
fix tests
2020-12-13 04:14:50 -08:00
Chris Lu
d156c74ec0
volume server set volume type and heartbeat to the master
2020-12-13 03:11:24 -08:00
Chris Lu
e9cd798bd3
adding volume type
2020-12-13 00:58:58 -08:00
Chris Lu
003b6245e7
fix nil
2020-12-02 00:09:19 -08:00
Chris Lu
965413c21b
shell: add volume.vacuum command
2020-11-28 23:18:02 -08:00
Chris Lu
c7ebadc25d
avoid possible concurrent access inside ensureCorrectWritables()
2020-11-22 17:15:59 -08:00
Chris Lu
8cb8cd4cc5
add locks
2020-11-17 16:59:48 -08:00
Chris Lu
c6bd244ebd
add TODO
2020-11-11 12:51:27 -08:00
Chris Lu
e0002f8dd7
check existing volumes for writable status
2020-10-24 01:34:31 -07:00
Chris Lu
720b1d9b88
adding locking to avoid nil VolumeLocationList
...
fix panic: runtime error: invalid memory address or nil pointer dereference
Oct 22 00:53:44 bedb-master1 weed[8055]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x17658da]
Oct 22 00:53:44 bedb-master1 weed[8055]: goroutine 310 [running]:
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLocationList).Length(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_location_list.go:35
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).enoughCopies(...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:376
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*VolumeLayout).ensureCorrectWritables(0xc000111d50, 0xc000b55438)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/volume_layout.go:202 +0x5a
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/topology.(*Topology).SyncDataNodeRegistration(0xc00042ac60, 0xc001454d30, 0x1, 0x1, 0xc0005fc000, 0xc00135de40, 0x4, 0xc00135de50, 0x10, 0x10d, ...)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/topology/topology.go:224 +0x616
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/server.(*MasterServer).SendHeartbeat(0xc000162700, 0x23b97c0, 0xc000ae2c90, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/server/master_grpc_server.go:106 +0x325
Oct 22 00:53:44 bedb-master1 weed[8055]: github.com/chrislusf/seaweedfs/weed/pb/master_pb._Seaweed_SendHeartbeat_Handler(0x1f8e7c0, 0xc000162700, 0x23b0a60, 0xc00024b440, 0x3172c38, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/seaweedfs/weed/pb/master_pb/master.pb.go:4250 +0xad
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).processStreamingRPC(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0xc0001fea80, 0x311fec0, 0x0, 0x0, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1329 +0xcd8
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).handleStream(0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100, 0x0)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:1409 +0xc5c
Oct 22 00:53:44 bedb-master1 weed[8055]: google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0001ce8b0, 0xc0001f31e0, 0x23bb800, 0xc000ac5500, 0xc000ab7100)
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:746 +0xa5
Oct 22 00:53:44 bedb-master1 weed[8055]: created by google.golang.org/grpc.(*Server).serveStreams.func1
Oct 22 00:53:44 bedb-master1 weed[8055]: #011/root/go/pkg/mod/google.golang.org/grpc@v1.29.1/server.go:744 +0xa5
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Oct 22 00:53:44 bedb-master1 systemd[1]: weedmaster.service: Failed with result 'exit-code'.
2020-10-21 23:15:48 -07:00
Chris Lu
410b818aa7
master: avoid timer leakage
2020-10-19 14:24:57 -07:00
Chris Lu
c7d7b1a0f6
Merge pull request #1485 from LIBA-S/fix_oversized
...
Correct the oversized state of volume after compaction
2020-09-23 19:24:30 -07:00
LIBA-S
eecd6b5d35
Fix a race condition when handle VolumeLocationList
2020-09-23 20:56:51 +08:00
LIBA-S
0157798ebf
Correct the oversized state of volume after compaction
2020-09-23 20:27:42 +08:00
Chris Lu
289e62a305
master: better locking of in memory volume data
...
related to https://github.com/chrislusf/seaweedfs/issues/1436#issuecomment-695880135
2020-09-20 23:07:55 -07:00
Chris Lu
6a92f0bc7a
refactoring to typed Size
...
Go is amazing with refactoring!
2020-08-18 17:04:28 -07:00
Chris Lu
152a6cbc2b
minor adjustments
2020-08-10 20:42:27 -07:00
cheng.li01
25fbff5d52
fix bug: two same volumeId in different collections
...
1, there will be two leader when master server startup in a few seconds
2, raft server will get a leader even there is only one master, so there is no need to do hard code to set the server to be leader
2020-08-10 16:37:47 +08:00
cheng.li01
dad1161c70
fix dn.volumes Iterate when write issue
2020-07-08 19:57:19 +08:00
Chris Lu
e912fd15e3
renaming
2020-06-19 22:45:27 -07:00
Evgenii Kozlov
0e0db70f55
Set volumes ReadOnly if low free disk space
2020-06-05 18:18:15 +03:00
bingoohuang
1f8782a1ed
try showing the first 100 volume ids and an extra ...
2020-05-29 16:15:33 +08:00
bingoohuang
1a642b9876
add Volume Ids column only for max 100 volumes for convenience in the master ui.
2020-05-29 15:37:58 +08:00
Chris Lu
e4af63a721
volume server: accept fsync=true in write requests
2020-04-11 21:39:16 -07:00
James Hartig
eae3f27c80
Added treat_replication_as_minimums master toml option
2020-04-01 19:08:48 -04:00
Chris Lu
e39e78ea8d
remove println
2020-03-22 18:37:12 -07:00
Chris Lu
35208711e5
logging
2020-03-22 18:32:49 -07:00
Chris Lu
c3cb6fa1d7
volume: compaction can cause readonly volumes
...
address https://github.com/chrislusf/seaweedfs/issues/1233
2020-03-17 09:43:57 -07:00
Chris Lu
560df51def
refactoring
2020-03-15 03:11:26 -07:00
Chris Lu
7edbee6f57
volume: proxy writes to remote volume server, with replication or not
...
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
2020-03-15 02:50:42 -07:00
Chris Lu
d022b6bc0e
fix compilation
2020-03-14 16:32:16 -07:00
Chris Lu
0576a27f44
protect against possible nil
...
which is unlikely to happen though
2020-03-14 16:06:06 -07:00
Chris Lu
3cc9e85895
volume: vacuum pass preallocate variable
2020-03-13 16:17:44 -07:00
Chris Lu
4ce23e8809
fix compilation error
2020-03-13 15:51:38 -07:00
Chris Lu
c90eb0da1f
volume: handling readonly volumes after compaction
...
ensure readonly volumes are not added as writable
2020-03-13 15:41:27 -07:00
Chris Lu
14ae33d642
adjust logging
2020-03-09 10:36:03 -07:00
Chris Lu
2e3f6ad3a9
filer: remember content is gzipped or not
2020-03-08 21:39:33 -07:00
Chris Lu
1ae83c2938
go fmt
2020-03-07 06:12:57 -08:00
Chris Lu
564629444b
format
2020-03-07 06:12:20 -08:00
Chris Lu
e3b8bf5588
Merge pull request #1219 from song-zhang/master
...
schedule new volume by free volume number of nodes
2020-03-07 05:54:06 -08:00
zhangsong
40f70481cd
schedule new volume by free volume number of nodes
2020-03-07 21:41:42 +08:00
Chris Lu
13e215ee5c
filer: option to encrypt data on volume server
2020-03-06 00:49:47 -08:00
Chris Lu
892e726eb9
avoid reusing context object
...
fix https://github.com/chrislusf/seaweedfs/issues/1182
2020-02-25 21:50:12 -08:00
Chris Lu
45156cc2fe
add test cases for replication=011
2020-02-20 01:21:11 -08:00
Chris Lu
dc786a63ac
master: add configurable volume growth toml setting
2020-02-03 18:15:16 -08:00
Chris Lu
72a64a5cf8
use the same context object in order to retry
2020-01-26 14:42:11 -08:00
Chris Lu
d1ab16b6e3
treat it as a single node cluster if empty raft server name
...
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
2020-01-10 00:37:44 -08:00
Chris Lu
09ca936c78
shell: add ec.decode command
2019-12-23 12:48:20 -08:00
Chris Lu
4cba32d1d0
go fmt
2019-12-11 22:22:09 -08:00
zhangsong
5eb86aad47
improve data consistency when replication write
2019-12-08 22:21:35 +08:00
Chris Lu
e426bd541e
grow volumes on volume servers with slots freed by cloud storage
2019-12-03 21:36:42 -08:00
zhangsong
e83c36e26f
fix the bug of volume never be vacuumed
2019-12-02 13:25:32 +08:00
divinerapier
5656d43264
can not break out of for-select block
...
Signed-off-by: divinerapier <poriter.coco@gmail.com>
2019-11-20 08:25:29 +08:00
zhangsong
61fa485700
add volume number param in assign operation
2019-11-10 20:11:03 +08:00
Lei Liu
1bcef02a6c
fix dir/lookup and col/delete api
...
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-31 16:55:45 +08:00
Lei Liu
f2f90436ef
fix leader master /dir/lookup api
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-30 16:38:40 +08:00
Lei Liu
c2884cace2
misc updated
...
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
2019-10-29 21:28:28 +08:00
Chris Lu
faec9076a4
adjust parameter names
2019-10-21 22:57:01 -07:00
j.laycock
6fc6322c90
Change joeslay paths to chrislusf paths
2019-09-12 14:18:21 +01:00
Tom Maxwell
4a878c0006
Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve
2019-09-04 15:27:14 +01:00
j.laycock
cbd0a98fa1
Add InMemory to Volume Grow structure
2019-09-03 18:19:02 +01:00
j.laycock
595a1beff0
Swap imports to use joeslay
2019-09-02 11:28:40 +01:00
Chris Lu
862c2cb8e6
go fmt
2019-07-21 21:51:38 -07:00
Chris Lu
79762385bd
master: ensure only one exclusive vacuum process
...
fix https://github.com/chrislusf/seaweedfs/issues/1011
2019-07-21 21:49:10 -07:00
Chris Lu
4b15c8f0c4
volume: lock writables changes
2019-07-21 13:49:09 -07:00
Chris Lu
898d943b25
refactoring
2019-07-17 23:43:48 -07:00
Chris Lu
57092d69ac
go fmt
2019-06-10 21:33:32 -07:00
Chris Lu
d344e0a035
fix ec related bugs
2019-06-05 23:20:26 -07:00
Chris Lu
6b08db65b0
fix shard count reporting
2019-06-05 01:58:37 -07:00
Chris Lu
eaa76f11b7
free volume slots factor in ec shard counts
2019-06-04 23:41:56 -07:00
Chris Lu
ca8a2bb534
go fmt
2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7
ui add ec shard statuses
2019-06-04 21:52:37 -07:00
Chris Lu
f53024d79d
delete ec collection
2019-05-30 10:40:22 -07:00
Chris Lu
1d111d6ce8
collection list normal and ec volumes
2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4
list ec collections also
2019-05-30 09:17:58 -07:00
Chris Lu
3f9ecee40f
working with reading remote intervals
2019-05-28 21:29:07 -07:00
Chris Lu
2858a954b3
read ec volume shards locations from master
2019-05-28 00:13:13 -07:00
Chris Lu
ac14cd5ad0
wdclient need to have ec shard info on initial dump
2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588
broadcast vid->location map to master clients for ec chards
2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4
incrementally update master ec shards state
2019-05-26 00:21:17 -07:00
Chris Lu
f0e6574d5e
allocate ec shards to volume servers
2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4
pb shard info uses ShardBits instead one message for one shard
2019-05-24 13:28:44 -07:00
Chris Lu
228850d588
shard id starts from zero
2019-05-24 11:52:23 -07:00
Chris Lu
8838176d82
register ec shards to topology
2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40
register ec shards to each data node
2019-05-23 22:51:18 -07:00