mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-24 01:17:56 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,959 Commits 37 Branches 296 Tags 152 MiB
15965f7c54
Commit Graph

337 Commits

Author SHA1 Message Date
Chris Lu
7edbee6f57 volume: proxy writes to remote volume server, with replication or not 
the panic is triggered by uploading a file to a volume server not holding the designated replica.
2020-03-15 10:20:14.365488 I | http: panic serving 127.0.0.1:57124: runtime error: invalid memory address or nil pointer dereference
goroutine 119 [running]:
net/http.(*conn).serve.func1(0xc0001a8000)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1772 +0x139
panic(0x2316fe0, 0x3662900)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/runtime/panic.go:973 +0x396
github.com/chrislusf/seaweedfs/weed/topology.getWritableRemoteReplications(0xc00009c000, 0x2, 0x7ffeefbffbd2, 0xe, 0x0, 0xa, 0x0, 0x0, 0xbb4bf1f7)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:157 +0x53
github.com/chrislusf/seaweedfs/weed/topology.ReplicatedWrite(0x7ffeefbffbd2, 0xe, 0xc00009c000, 0xc000000002, 0xc000472750, 0xc0001b2200, 0x0, 0x1, 0x0)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/topology/store_replicate.go:29 +0xc7
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).PostHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers_write.go:52 +0x56f
github.com/chrislusf/seaweedfs/weed/server.(*VolumeServer).privateStoreHandler(0xc0001513f0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/gopath/src/github.com/chrislusf/seaweedfs/weed/server/volume_server_handlers.go:37 +0x21f
net/http.HandlerFunc.ServeHTTP(0xc0004420e0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2012 +0x44
net/http.(*ServeMux).ServeHTTP(0xc0001fc800, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2387 +0x1a5
net/http.serverHandler.ServeHTTP(0xc0001781c0, 0x292bde0, 0xc0001fe2a0, 0xc0001b2200)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0001a8000, 0x2934420, 0xc000212400)
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
	/home/travis/.gimme/versions/go1.14.linux.amd64/src/net/http/server.go:2933 +0x35c
Eg:
server A (datacenter 1) and server B (datacenter 2) hold replica (100) for volume 1.
If you upload a file with a key 1,xxxxx to server C (datacenter 3) will trigger the panic on server C.
The server C should either proxy upload file to the correct volume server or should return an HTTP error code and not panic.
 2020-03-15 02:50:42 -07:00
Chris Lu
d022b6bc0e fix compilation 2020-03-14 16:32:16 -07:00
Chris Lu
0576a27f44 protect against possible nil 
which is unlikely to happen though
 2020-03-14 16:06:06 -07:00
Chris Lu
3cc9e85895 volume: vacuum pass preallocate variable 2020-03-13 16:17:44 -07:00
Chris Lu
4ce23e8809 fix compilation error 2020-03-13 15:51:38 -07:00
Chris Lu
c90eb0da1f volume: handling readonly volumes after compaction 
ensure readonly volumes are not added as writable
 2020-03-13 15:41:27 -07:00
Chris Lu
14ae33d642 adjust logging 2020-03-09 10:36:03 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
1ae83c2938 go fmt 2020-03-07 06:12:57 -08:00
Chris Lu
564629444b format 2020-03-07 06:12:20 -08:00
Chris Lu
e3b8bf5588
Merge pull request #1219 from song-zhang/master 
schedule new volume by free volume number of nodes
 2020-03-07 05:54:06 -08:00
zhangsong
40f70481cd schedule new volume by free volume number of nodes 2020-03-07 21:41:42 +08:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
892e726eb9 avoid reusing context object 
fix https://github.com/chrislusf/seaweedfs/issues/1182
 2020-02-25 21:50:12 -08:00
Chris Lu
45156cc2fe add test cases for replication=011 2020-02-20 01:21:11 -08:00
Chris Lu
dc786a63ac master: add configurable volume growth toml setting 2020-02-03 18:15:16 -08:00
Chris Lu
72a64a5cf8 use the same context object in order to retry 2020-01-26 14:42:11 -08:00
Chris Lu
d1ab16b6e3 treat it as a single node cluster if empty raft server name 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1118
 2020-01-10 00:37:44 -08:00
Chris Lu
09ca936c78 shell: add ec.decode command 2019-12-23 12:48:20 -08:00
Chris Lu
4cba32d1d0 go fmt 2019-12-11 22:22:09 -08:00
zhangsong
5eb86aad47 improve data consistency when replication write 2019-12-08 22:21:35 +08:00
Chris Lu
e426bd541e grow volumes on volume servers with slots freed by cloud storage 2019-12-03 21:36:42 -08:00
zhangsong
e83c36e26f fix the bug of volume never be vacuumed 2019-12-02 13:25:32 +08:00
divinerapier
5656d43264
can not break out of for-select block 
Signed-off-by: divinerapier <poriter.coco@gmail.com>
 2019-11-20 08:25:29 +08:00
zhangsong
61fa485700 add volume number param in assign operation 2019-11-10 20:11:03 +08:00
Lei Liu
1bcef02a6c fix dir/lookup and col/delete api 
1, Fix Layouts first letter capitalized
2, Return http 204 when delete a collection

Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-31 16:55:45 +08:00
Lei Liu
f2f90436ef fix leader master /dir/lookup api 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-30 16:38:40 +08:00
Lei Liu
c2884cace2 misc updated 
Signed-off-by: Lei Liu <lei01.liu@horizon.ai>
 2019-10-29 21:28:28 +08:00
Chris Lu
faec9076a4 adjust parameter names 2019-10-21 22:57:01 -07:00
j.laycock
6fc6322c90 Change joeslay paths to chrislusf paths 2019-09-12 14:18:21 +01:00
Tom Maxwell
4a878c0006 Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve 2019-09-04 15:27:14 +01:00
j.laycock
cbd0a98fa1 Add InMemory to Volume Grow structure 2019-09-03 18:19:02 +01:00
j.laycock
595a1beff0 Swap imports to use joeslay 2019-09-02 11:28:40 +01:00
Chris Lu
862c2cb8e6 go fmt 2019-07-21 21:51:38 -07:00
Chris Lu
79762385bd master: ensure only one exclusive vacuum process 
fix https://github.com/chrislusf/seaweedfs/issues/1011
 2019-07-21 21:49:10 -07:00
Chris Lu
4b15c8f0c4 volume: lock writables changes 2019-07-21 13:49:09 -07:00
Chris Lu
898d943b25 refactoring 2019-07-17 23:43:48 -07:00
Chris Lu
57092d69ac go fmt 2019-06-10 21:33:32 -07:00
Chris Lu
d344e0a035 fix ec related bugs 2019-06-05 23:20:26 -07:00
Chris Lu
6b08db65b0 fix shard count reporting 2019-06-05 01:58:37 -07:00
Chris Lu
eaa76f11b7 free volume slots factor in ec shard counts 2019-06-04 23:41:56 -07:00
Chris Lu
ca8a2bb534 go fmt 2019-06-04 22:04:10 -07:00
Chris Lu
2215e81be7 ui add ec shard statuses 2019-06-04 21:52:37 -07:00
Chris Lu
f53024d79d delete ec collection 2019-05-30 10:40:22 -07:00
Chris Lu
1d111d6ce8 collection list normal and ec volumes 2019-05-30 09:27:23 -07:00
Chris Lu
e941d0c2f4 list ec collections also 2019-05-30 09:17:58 -07:00
Chris Lu
3f9ecee40f working with reading remote intervals 2019-05-28 21:29:07 -07:00
Chris Lu
2858a954b3 read ec volume shards locations from master 2019-05-28 00:13:13 -07:00
Chris Lu
ac14cd5ad0 wdclient need to have ec shard info on initial dump 2019-05-26 01:01:04 -07:00
Chris Lu
b58e25e588 broadcast vid->location map to master clients for ec chards 2019-05-26 00:49:15 -07:00
Chris Lu
af67d99ca4 incrementally update master ec shards state 2019-05-26 00:21:17 -07:00
Chris Lu
f0e6574d5e allocate ec shards to volume servers 2019-05-25 02:02:44 -07:00
Chris Lu
6f4b09b6a4 pb shard info uses ShardBits instead one message for one shard 2019-05-24 13:28:44 -07:00
Chris Lu
228850d588 shard id starts from zero 2019-05-24 11:52:23 -07:00
Chris Lu
8838176d82 register ec shards to topology 2019-05-23 23:34:29 -07:00
Chris Lu
8a96445f40 register ec shards to each data node 2019-05-23 22:51:18 -07:00
Chris Lu
4659d80035 prepare to register ec shard info in master 2019-05-23 00:42:28 -07:00
Chris Lu
e913ee380a add comments 2019-05-23 00:04:24 -07:00
Chris Lu
7180520889 ec shard info can be queried via VolumeList() 2019-05-22 22:44:28 -07:00
Wine93
32f93fb09e style: go fmt for all 2019-04-30 03:22:19 +00:00
Chris Lu
440111a349 volume: support http status 304 for the same file id 2019-04-21 13:33:23 -07:00
Chris Lu
ebe94be1a1 maintain layout correctness when changing volumes 2019-04-21 13:32:36 -07:00
Chris Lu
132921ad41 adjust log or error 2019-04-21 10:14:17 -07:00
Chris Lu
316bd27f75 add remove volumes with version info 2019-04-20 23:53:37 -07:00
Chris Lu
6fc1f53018 shell: add command volume.move 2019-04-20 11:35:23 -07:00
Chris Lu
e5506152c0 refactoring 2019-04-18 21:43:36 -07:00
Chris Lu
2f76681d62 refactor: adjust grpc API name 2019-04-10 21:41:17 -07:00
Chris Lu
0bd7ced7c2 fix test 2019-04-04 20:11:59 -07:00
Chris Lu
9cc73f4a9a fix test 2019-04-04 19:34:15 -07:00
Chris Lu
766396d249 weed master: atomic volume counting 
possible fix for https://github.com/chrislusf/seaweedfs/issues/913
 2019-04-04 19:27:00 -07:00
Chris Lu
95e0520182 weed volume: add grpc operation to relicate a volume to local 2019-03-23 11:33:34 -07:00
Chris Lu
aca653c08b weed shell: list volumes 2019-03-17 20:27:08 -07:00
Chris Lu
657dd2e6c9 add shell command to list all collections 2019-03-16 13:43:16 -07:00
chenwanli
900aad4935 Fix https://github.com/chrislusf/seaweedfs/issues/864 2019-02-28 11:37:44 +08:00
chenwanli
0bc3cdd26a Fix https://github.com/chrislusf/seaweedfs/issues/825 2019-02-25 10:47:41 +08:00
Chris Lu
e108688990 avoid grpc 5 seconds timeout 
some operations may take longer than 5 seconds.

only keep the timeout for raft operations
 2019-02-20 01:01:01 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
98a03b38e5 avoid util package depends on security package 2019-02-16 12:23:35 -08:00
Chris Lu
31aa60ce05 Merge branch 'master' into add_jwt 2019-02-16 08:55:38 -08:00
Chris Lu
ad257ae179 simplify isLeader() logic 2019-02-16 08:55:23 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
7a493bbefa better way to detect a master isLeader 2019-01-28 10:36:16 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
bingoohuang
219838db87
fail fast when failed to write local volume 2019-01-15 15:47:43 +08:00
Chris Lu
2d23d86fd3 no timeout for volume vacuum 
revert changes on volume vacuum timemout from https://github.com/chrislusf/seaweedfs/pull/829
 2019-01-10 09:07:40 -08:00
chenwanli
0a3e83a36a Set timeout for master and volume non-streaming rpc 2019-01-10 19:41:03 +08:00
Chris Lu
d67f7ddfaf adjust log 2019-01-02 12:58:06 -08:00
Chris Lu
03cfb4267f adjust vaccum logging 2018-12-31 00:06:52 -08:00
Chris Lu
871dee4674 fix test 2018-11-23 12:37:09 -08:00
Chris Lu
444dfded84 add fs.FSStatfser for SeaweedFS weed mount 2018-11-23 00:24:51 -08:00
Chris Lu
d64546c791 adjust logs 2018-11-22 14:31:10 -08:00
Chris Lu
4fcfc9410f cleanup 2018-11-18 11:51:38 -08:00
Chris Lu
9e98bba206 go fmt 2018-10-24 00:00:01 -07:00
Chris Lu
82a5d39778 add back volume vacuum cleanup 2018-10-18 23:02:58 -07:00
Chris Lu
201c24c110 add locking to volume layout vacuum 2018-10-18 20:34:43 -07:00
Chris Lu
fda771c83f migrate volume sync status to grpc API on volume server 2018-10-15 01:19:15 -07:00
Chris Lu
b423bb9e2d migrate assign volume to grpc API on volume server 2018-10-15 00:40:46 -07:00
Chris Lu
b1daede91b move volume vacuum to gRpc 2018-10-14 23:12:43 -07:00
Chris Lu
e8d4be579d send initial list of volume location 2018-07-28 01:17:35 -07:00
Chris Lu
a12c7b86b0 broadcast messages of new and deleted volumes 2018-07-27 23:09:55 -07:00
Chris Lu
2fd48df9d9 fix test 2018-07-11 12:52:48 -07:00
Chris Lu
f7afa80250 also remove volumeLayout if collection is empty 2018-07-11 02:01:33 -07:00
Chris Lu
0d83c1b91e refactoring 2018-06-25 00:01:53 -07:00
Chris Lu
018a9a20be add test for volume registeration at master 2018-06-24 23:55:27 -07:00
Chris Lu
cb19a7da99 add a note 2018-06-24 23:55:06 -07:00
Chris Lu
afc1532b24 file rename 2018-06-24 23:19:56 -07:00
Chris Lu
4433b4c006 treat 0 sized file as normal 
fix https://github.com/chrislusf/seaweedfs/issues/409
 2018-06-12 21:11:26 -07:00
Chris Lu
8b0718ac92 go vet 2018-05-27 11:56:49 -07:00
tengattack
371ad2e397 fix reserve volume on node 2018-01-17 19:53:41 +08:00
Chris Lu
9026b3e86e always remember the max volume id 2017-11-28 17:08:59 -08:00
Chris Lu
58344980e4 preallocate disk space during compaction also, add cleanup for failed compaction 2017-08-29 23:59:53 -07:00
Chris Lu
f7c22f0159 fix bug when collecting vacuum errors 2017-08-29 23:11:20 -07:00
Chris Lu
3a8746ca54 better error message 2017-08-10 10:26:39 -07:00
Xinran Xu
4a0939a4e4 Fix panic error when target collection is not exist. 
Fix: interface conversion: interface {} is nil, not *topology.Collection.
 2017-07-14 20:04:33 +08:00
Chris Lu
2c7dad589d stop when dn.parent is nil 
fix https://github.com/chrislusf/seaweedfs/issues/473
 2017-06-16 08:27:50 -07:00
Chris Lu
a7faefd4ce fix compilation error 2017-05-22 23:04:43 -07:00
Chris Lu
b3513d3167 skip only readonly volumes 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2017-05-22 17:05:27 -07:00
sparklxb
4049c72e6c fix bug: replication, metadata not replicated 2017-03-04 11:24:12 +08:00
Chris Lu
30f138a538 fix tests 2017-02-12 22:18:48 -08:00
Chris Lu
41f686ab9d remove weeds.conf 2017-02-12 21:58:44 -08:00
Chris Lu
e767c3ea4f disable master connection timeout 
temporarily disable master connection timeout due to heartbeat
connection timeout
 2017-01-10 01:30:00 -08:00
Chris Lu
e46c3415f7 gRpc for master~volume heartbeat 2017-01-10 01:01:12 -08:00
Chris Lu
ed44f12f6d support Fallocate on linux 2017-01-08 11:01:46 -08:00
sparklxb
da9b672d1b support additional header name-value pairs 2017-01-08 22:34:42 +08:00
sparklxb
86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
Chris Lu
9ba52db585 add writable checking 
fix https://github.com/chrislusf/seaweedfs/issues/405
 2016-11-26 12:14:06 -08:00
Chris Lu
0559aa9673 use Lock instead of RLock 
fix https://github.com/chrislusf/seaweedfs/issues/364
 2016-09-07 18:21:05 -07:00
霍晓栋
f04d8fcbcc if replicated volume has one copy in readonly mode at one node,it should be removed from writable list 2016-08-09 20:12:39 +08:00
霍晓栋
079fdbad13 fix volume_layout bug, add new feature for volume etag write,filer pass through query params' 2016-06-29 16:05:00 +08:00
Chris Lu
b617b13c43 remember oversized volumes 
fix https://github.com/chrislusf/seaweedfs/issues/331
 2016-06-27 15:28:23 -07:00
Chris Lu
d0dbf6d2ea adjust logging 2016-06-26 12:49:10 -07:00
Chris Lu
96d2f11f81 adjust log 2016-06-05 09:05:03 -07:00
Chris Lu
5ce6bbf076 directory structure change to work with glide 
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
 2016-06-02 18:09:14 -07:00
Chris Lu
5071f528f6 testing compilation with remove package 2013-02-10 03:49:51 -08:00
Chris Lu
55f2627fcf testing new directory structure 2013-02-10 03:31:40 -08:00
Chris Lu
ab6fb13ad7 avoid the "src" folder 2013-02-10 03:25:35 -08:00