mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-28 13:31:27 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,616 Commits 32 Branches 296 Tags 148 MiB
c57c79a0ab
Commit Graph

5647 Commits

Author SHA1 Message Date
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
chrislu
bd13a7968f remove unused symlink resolving 2022-08-04 01:35:18 -07:00
chrislu
88945d9954 mount: do not follow soft link with xattr 2022-08-04 01:21:15 -07:00
chrislu
90db4b5a44 minor 2022-08-03 00:10:47 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite 
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
 2022-08-01 13:31:01 -07:00
Konstantin Lebedev
3ffa4ba91d clean 2022-08-02 01:30:38 +05:00
Konstantin Lebedev
2a5629dd28 align 2022-08-02 01:29:44 +05:00
Konstantin Lebedev
df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev
0a5d3c3cf6 rename to localWrite and replicatedWrite 2022-08-01 23:09:09 +05:00
chrislu
bd5c9904be Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-08-01 09:40:05 -07:00
chrislu
e26d1cf4ef fix test 2022-08-01 09:40:01 -07:00
Konstantin Lebedev
fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev
ab50f21ad9 Detailed metrics VolumeServerRequestHistogram for writing to disk and replication 2022-08-01 21:16:57 +05:00
Konstantin Lebedev
2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
78cbd8002f revert Sleep 2022-08-01 20:21:23 +05:00
Konstantin Lebedev
cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00
Konstantin Lebedev
634bc97840 fix conflicts 2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
Konstantin Lebedev
c0d92f61a1 comment 2022-08-01 14:40:42 +05:00
Konstantin Lebedev
1d29f67c02 revert disk stats 2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
chrislu
036566629a filer.sync: fix synchronization logic in active-active mode 
fix https://github.com/seaweedfs/seaweedfs/issues/3328
 2022-08-01 00:06:18 -07:00
chrislu
303bd067b5 Revert "rename: delete source entry metadata only, skipping hard links" 
This reverts commit 03466f955e.

fix https://github.com/seaweedfs/seaweedfs/issues/3386
 2022-07-31 22:51:41 -07:00
chrislu
03c6f978f4 3.19 2022-07-31 17:16:51 -07:00
chrislu
afef014832 scaffold 2022-07-31 13:23:44 -07:00
Chris Lu
0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy 
Add healthy indicator for raft status
 2022-07-30 11:55:30 -07:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu
cb096d1f53 minor 2022-07-30 09:52:52 -07:00
chrislu
0423d50106 wip note 2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a moving 2022-07-29 23:39:11 -07:00
chrislu
13b9a52f80 fix deprecated functions 2022-07-29 01:34:39 -07:00
chrislu
be0ec7bc2f fix test 2022-07-29 01:15:26 -07:00
chrislu
1f161f53dc add back rack 2022-07-29 01:15:19 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
chrislu
de3afd4199 fix rebasing 2022-07-28 23:25:46 -07:00
chrislu
06cd491abc add message batch 2022-07-28 23:24:38 -07:00
chrislu
7576c244c4 add flatbuffer serde for message 2022-07-28 23:24:38 -07:00
chrislu
a3fae7cd16 add an empty placeholder for mq.topic.list 2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47 segment serde 2022-07-28 23:24:38 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0 refactor 2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c clean up 2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu
f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd mq broker adds data center and rack 2022-07-28 23:22:07 -07:00
chrislu
87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu
ac66227798 renaming 2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17 rename to SeaweedMQ 2022-07-28 23:22:06 -07:00
Chris Lu
3e7e922d16
Merge pull request #3378 from vanhtuan0409/change-put-policy 
PutUserAction should completely replace identity actions
 2022-07-28 19:43:25 -07:00
chrislu
89e276de64 fix loop 2022-07-28 18:06:49 -07:00
chrislu
a458bee9f2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-28 16:32:02 -07:00
chrislu
e0eda52c54 mount: ensure symlink parent directory is tracked 
fix https://github.com/chrislusf/seaweedfs/issues/3373
 2022-07-28 16:32:00 -07:00
Tuan Vuong
1d9a888be9 PutUserAction should completely replace identity actions 2022-07-28 17:02:56 +07:00
zzq09494
01b7aa79fa fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted 2022-07-28 12:04:37 +08:00
chrislu
ca836568ac Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Chris Lu
b7e4e26778
Merge pull request #3371 from askeipx/issues/3341 
delete disk_size metrics when collection deleted
 2022-07-27 11:25:42 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev
ac5ce31278 leave notion to refactor after prometheus upgrade 2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev
47c72e6f35 remove all (currently existing) collection volume metrics 2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c delete disk_size metrics when collection deleted 2022-07-27 15:02:41 +05:00
chrislu
f401b996eb mount: fix symlink size reporting 2022-07-26 23:12:25 -07:00
Konstantin Lebedev
c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
Konstantin Lebedev
da9d3e8f6c refactor 2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4 fix logic else brake 2022-07-26 11:47:11 +05:00
chrislu
475185fb72 3.18 2022-07-25 22:41:38 -07:00
chrislu
ec0edb1ac4 filer: fix wrong logic during read 2022-07-25 22:40:00 -07:00
guosj
cc7a4b0a6e correct comment 2022-07-25 11:46:41 +08:00
chrislu
33cff61992 fix glog test 2022-07-24 19:13:54 -07:00
chrislu
dbe4849ffd 3.17 2022-07-24 18:46:50 -07:00
chrislu
b98c0b4bf4 disable symlink translation for now 2022-07-24 18:46:32 -07:00
chrislu
2dc5d4adac disable symlink translation for now 2022-07-24 18:24:34 -07:00
chrislu
3d2bbf96d2 adjust glog format 2022-07-24 17:29:08 -07:00
chrislu
2807fe08fc adjust glog format 2022-07-24 16:44:58 -07:00
chrislu
2b64916d3e set status to fuse.OK 2022-07-24 15:30:55 -07:00
chrislu
7386cf6a06 mount: resolve symlink 2022-07-24 15:11:24 -07:00
chrislu
40089b6753 maintain node lookup counter 2022-07-24 13:43:11 -07:00
chrislu
d24db396cc fix removing path from inode2path 2022-07-23 18:20:29 -07:00
chrislu
41eeb4deef do not add new inode during link 2022-07-23 13:37:07 -07:00
chrislu
02c2d81cde fix removePathFromInode2Path 2022-07-23 13:21:42 -07:00
chrislu
4d5dc55766 mount: better support hard links 2022-07-23 12:57:00 -07:00
chrislu
ebda0689ab use linux default max name length 2022-07-23 10:51:06 -07:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
chrislu
2c8818351f mount: comment out stuck fuse inode notify during renaming 2022-07-22 18:17:34 -07:00
chrislu
a5c94acd4f mount: check possible nil entry case 2022-07-22 15:33:37 -07:00
Chris Lu
0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap 
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
 2022-07-22 02:57:50 -07:00
LHHDZ
994a2dec78 fix vid_map_test tests 2022-07-22 17:54:57 +08:00
LHHDZ
58c02d6429 Solve the problem that LookupFileId lookup urls is empty due to leader switching 
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
 2022-07-22 17:22:38 +08:00
ningfd
58f2dd6740 change topology mapping definition, improve spelling 2022-07-22 17:18:56 +08:00
ningfd
6f882eb354 optimiz: master ui will render data in order 2022-07-22 14:37:49 +08:00
chrislu
7a6c559ab4 fix Change replication via volume.configure.replication by collection 
fix https://github.com/chrislusf/seaweedfs/issues/3346
 2022-07-21 22:01:05 -07:00
chrislu
fc8241fb5e leveldb3: add instant dropping bucket 2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44 explicit bucket aware declaration 2022-07-21 18:23:53 -07:00
Konstantin Lebedev
3c42814b58 avoid deadlock 2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb use safe onPeerUpdateDoneCns 2022-07-21 15:51:14 +05:00
chrislu
6147b61b0a mount: fix concurrent map read and map write 
fix https://github.com/chrislusf/seaweedfs/issues/3344
 2022-07-21 01:38:26 -07:00
Chris Lu
9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume 2022-07-20 09:06:06 -07:00
Konstantin Lebedev
7875470e74 onPeerUpdateGoroutineCount use int32 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
c5189c343b remove ticker 
update the topology before each file
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
e2d991d8d0 ticker.Stop 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
de4fcc0e2c sync update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
fa88dff7ce update otherNodes 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
72dca31cfa fix update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
884ffbafee clouse background update 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
39eaf426f8 fix TestVolumeServerEvacuate 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b5e5f6f55a update topologyInfo 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
867269cdcf help rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6f764e1014 volume server evacuate from rack 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
ba0e3ce5fa volume server evacuate to target server 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b6471ecd75 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7 fix design 2022-07-20 18:08:12 +05:00
Tuan Vuong
312e13416b filter duplicated action 2022-07-20 16:43:50 +07:00
guol-fnst
ac694f0c8f rename parameter and reuse functions 
rename milestone to  watermark
 2022-07-20 17:00:40 +08:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer 
volume server evacuate to target server
 2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03 remove ticker 
update the topology before each file
 2022-07-20 00:54:23 +05:00
Konstantin Lebedev
11e393dbe7 err msg with duplicated local subscription detected move to log level 1 
https://github.com/chrislusf/seaweedfs/issues/3320
 2022-07-20 00:45:13 +05:00
Konstantin Lebedev
f6a966b4fc add waiting log message 2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849 check for ping before deleting raft server 
https://github.com/chrislusf/seaweedfs/issues/3083
 2022-07-20 00:04:12 +05:00
chrislu
2ae3f812f8 minor 2022-07-19 11:43:31 -07:00
Chris Lu
9667588af0
Merge pull request #3318 from kmlebedev/issues/3310 
Use fallback if urls are not found
 2022-07-19 11:38:43 -07:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove 
raftServerRemoval typo and logs
 2022-07-19 11:32:17 -07:00
guol-fnst
91285bb51d remove uncessary code 2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34 optimiz 2022-07-19 16:58:53 +08:00
Konstantin Lebedev
f419d5643a fix typo 
add remove logs
 2022-07-19 11:50:52 +05:00
guol-fnst
bec9f79659 use snapshot 2022-07-19 11:14:41 +08:00
Konstantin Lebedev
d422e7769c ticker.Stop 2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b sync update topologyInfo 2022-07-18 16:27:02 +05:00
chrislu
56ec89625a 3.16 2022-07-18 00:53:05 -07:00
Konstantin Lebedev
2b4112e462 update otherNodes 2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d fix update topologyInfo 2022-07-18 01:46:31 +05:00
Konstantin Lebedev
7b1497ee63 Use BackoffSchedule for getLookupFileId 2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8 Use fallback if urls are not found 2022-07-15 15:29:15 +05:00
Konstantin Lebedev
4d5144e50d clouse background update 2022-07-15 13:51:08 +05:00
chrislu
1db012485f mount: fix bug during busy writes 
fix https://github.com/chrislusf/seaweedfs/issues/3315
 2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag 
File upload should succeed should return `ETag` instead of `Etag`
 2022-07-14 15:35:41 -07:00
chrislu
fa61074513 add clientId logging 2022-07-14 12:27:34 -07:00
chrislu
fbd8f868a1 filer may have trouble to re-connect 
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
 2022-07-14 12:15:31 -07:00
LHHDZ
c08f6d0ff7 File upload should succeed should return ETag instead of Etag 
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
 2022-07-14 19:45:01 +08:00
Chris Lu
957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume 
optimiz concurrency
 2022-07-13 23:12:24 -07:00
guol-fnst
300b383cdf use 10 or numCPU workers if env is not found 2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2 optimiz concurrency 
user can customize number of workers via env "GOMAXPROCS"
 2022-07-14 09:57:25 +08:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak 2022-07-13 09:23:37 -07:00