mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-25 10:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,966 Commits 32 Branches 296 Tags 153 MiB
3b840c20e3
Commit Graph

1627 Commits

Author SHA1 Message Date
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting (#3577) 2022-09-06 08:19:13 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597) 2022-09-05 07:47:43 -07:00
famosss
7f56a40553
Turn down the 404 log level (#3585) 2022-09-04 18:17:01 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize (#3570) 
https://github.com/seaweedfs/seaweedfs/issues/3557
 2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log (#3568) 
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567

* format message
 2022-09-01 09:04:15 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt (#3554) 2022-08-30 10:12:02 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr (#3550) 
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
 2022-08-30 00:03:01 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug (#3534) 2022-08-28 07:51:47 -07:00
chrislu
3741779144 minor 2022-08-27 00:20:58 -07:00
chrislu
580a7fa685 adjust for data race 
fix https://github.com/seaweedfs/seaweedfs/issues/3512
 2022-08-27 00:19:13 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children (#3530) 2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate (#3525) 
https://github.com/seaweedfs/seaweedfs/issues/3524
 2022-08-26 10:18:49 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks (#3505) 2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created (#3498) 
* replace mkdir to mkFile

* ContentLength must be zero

* revert mkDir

* Seaweedfs-Is-Directory-Key return metadata
 2022-08-24 00:15:44 -07:00
chrislu
10414fd81c ping timeout at 15 seconds 
this 72 minute timeout setting seems unreasonably long

15 seconds is around the time when a new raft leader should be elected.
 2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398) 
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
 2022-08-23 23:18:21 -07:00
chrislu
9fce75607d s3: report http.StatusOK if the directory is explicitly created 
fix https://github.com/seaweedfs/seaweedfs/issues/3457
 2022-08-23 01:16:46 -07:00
chrislu
601ba5fb68 master: disable locking if not leader 2022-08-22 14:11:48 -07:00
chrislu
f4b52d4c52 fix format 2022-08-22 00:20:21 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files (#3478) 
* feat(weed.move): add a speed limit parameter of moving files

* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
 2022-08-21 23:08:31 -07:00
chrislu
fef9c6a520 filer: do not always save files in "/etc" folder to filer store 
fix https://github.com/seaweedfs/seaweedfs/issues/3476
 2022-08-21 17:02:16 -07:00
chrislu
2762154130 fix compilation 2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3 EC: with multiple volume locations, the ec rebuilding may fail 2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations 
related to https://github.com/seaweedfs/seaweedfs/issues/3459
 2022-08-21 14:52:37 -07:00
chrislu
65ff7198fe adjust type 2022-08-20 22:37:24 -07:00
chrislu
28b862f45f use util.Retry to retry 2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8 webdav: retryable data chunk upload 2022-08-20 19:18:12 -07:00
chrislu
f8fa430257 filer: retryable data chunk upload 2022-08-20 19:15:44 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
chrislu
fdd8c5d5e0 Filer if-modified-since check doesn't fire on "exact" #3467 
fix https://github.com/seaweedfs/seaweedfs/issues/3467
 2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89 filer: retryable when error is not found 2022-08-19 03:36:15 -07:00
Eric Yang
b6f99c4d83
fix naming typo (#3465) 
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-18 23:25:32 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
e259052a70 volume: fix metrics for volume request time 2022-08-17 04:29:31 -07:00
chrislu
789d9dbfe1 webdav: fix nil 
fix https://github.com/seaweedfs/seaweedfs/issues/3440
 2022-08-14 23:33:30 -07:00
Konstantin Lebedev
4d4cd0948d
avoid infinite loop WaitUntilConnected() (#3431) 
https://github.com/seaweedfs/seaweedfs/issues/3421
 2022-08-11 15:03:26 -07:00
Eric Yang
986daecbfe
add upload response (#3428) 
* add upload response

* fixing comments

Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
 2022-08-10 22:43:09 -07:00
chrislu
ae93c966d9 ensure memory is aligned 
fix https://github.com/seaweedfs/seaweedfs/issues/3427
 2022-08-10 22:27:13 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405) 
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
 2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics (#3402) 
* refactor FilerRequest metrics

* avoid double count proxy

* defer to
 2022-08-04 01:44:54 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite 
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
 2022-08-01 13:31:01 -07:00
Konstantin Lebedev
df9cc31a0a refactor 2022-08-02 01:18:26 +05:00
Konstantin Lebedev
fc65122766 rename to LoadAvg_1M 2022-08-01 21:32:21 +05:00
Konstantin Lebedev
2f85e31f5c Merge branch 'master' into gentle_vacuum 2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef remove percent 2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083 
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
 2022-08-01 08:23:10 -07:00
Konstantin Lebedev
cd5c7ad052 move to github.com/seaweedfs/seaweedfs 2022-08-01 16:36:32 +05:00
Konstantin Lebedev
3c75479e2b Merge branch 'master' into gentle_vacuum 
# Conflicts:
#	weed/pb/messaging_pb/messaging.pb.go
#	weed/pb/messaging_pb/messaging_grpc.pb.go
#	weed/pb/s3_pb/s3.pb.go
#	weed/pb/volume_server_pb/volume_server.pb.go
#	weed/server/volume_grpc_vacuum.go
 2022-08-01 14:45:22 +05:00
Konstantin Lebedev
1d29f67c02 revert disk stats 2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3 rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping 2022-08-01 12:51:41 +05:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d vacuum show LA 2022-07-29 11:59:33 +05:00
chrislu
9f479aab98 allocate brokers to serve segments 2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0 refactor 2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5 remove old code 2022-07-28 23:24:38 -07:00
chrislu
f25e273e32 display data center and rack in cluster.ps 2022-07-28 23:22:52 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Konstantin Lebedev
c88ea31f62 fix RUnlock of unlocked RWMutex 2022-07-26 12:57:07 +05:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
ningfd
58f2dd6740 change topology mapping definition, improve spelling 2022-07-22 17:18:56 +08:00
Konstantin Lebedev
3c42814b58 avoid deadlock 2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb use safe onPeerUpdateDoneCns 2022-07-21 15:51:14 +05:00
Konstantin Lebedev
7875470e74 onPeerUpdateGoroutineCount use int32 2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7 fix design 2022-07-20 18:08:12 +05:00
Konstantin Lebedev
f6a966b4fc add waiting log message 2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849 check for ping before deleting raft server 
https://github.com/chrislusf/seaweedfs/issues/3083
 2022-07-20 00:04:12 +05:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove 
raftServerRemoval typo and logs
 2022-07-19 11:32:17 -07:00
Konstantin Lebedev
f419d5643a fix typo 
add remove logs
 2022-07-19 11:50:52 +05:00
chrislu
aca20cd9f4 more logging related to filer metadata subscription 2022-07-15 00:42:13 -07:00
chrislu
fbd8f868a1 filer may have trouble to re-connect 
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
 2022-07-14 12:15:31 -07:00
liubaojiang
4e83e92156 add subscriber clientId if it is the first time 2022-07-08 20:27:25 +08:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files 
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
 2022-06-27 20:19:21 +02:00
Konstantin Lebedev
6c20a3b622 avoid set currentMaster k8s svc.local discoveruy service domains 
https://github.com/chrislusf/seaweedfs/issues/2589
 2022-06-27 21:47:05 +05:00
chrislu
9dfea44aeb fix unit 2022-06-26 12:21:38 -07:00
chrislu
3142e77eca mount: stats report physical size instead of logical size 2022-06-26 12:14:34 -07:00
chrislu
0c13fc873c fix compilation 2022-06-23 00:45:11 -07:00
chrislu
96496d5286 master: broadcast new volume locations to clients to avoid possible racing condition 
fix https://github.com/chrislusf/seaweedfs/issues/3220
 2022-06-23 00:41:33 -07:00
Konstantin Lebedev
c1c8dad677 avoid no such raft date directory 
https://github.com/chrislusf/seaweedfs/issues/3214
 2022-06-21 13:47:51 +05:00
石昌林
81f7f08708 Determine whether to preallocate according to the master configuration before executing copy volume 2022-06-20 21:12:44 +08:00
chrislu
aaa9938353 Revert "adjust conditions" 
This reverts commit e024586ff1.
 2022-06-16 01:26:36 -07:00
chrislu
e024586ff1 adjust conditions 2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix 
exclude replication from the concurrentUploadLimitMB
 2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e add condition when inFlightUploadDataLimitCond signal 2022-06-16 14:07:11 +08:00
liubaojiang
3076ac101e move vs.concurrentUploadLimit != 0 out of the lock 2022-06-16 09:58:44 +08:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req 2022-06-15 06:33:33 -07:00
Konstantin Lebedev
31a929f7f2 logging processRangeRequest errors 2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
zhihao.qu
42d04c581b feat(filer.sync): add metricsServer in filer.sync. 
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
 2022-06-15 11:33:18 +08:00
yulai.li
28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
chrislu
d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
chrislu
444ac21050 go fmt 2022-06-11 09:51:11 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix 2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb Bug Fix: force reload filer page when upload/delete/rename/create directory 2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64 fix typo 2022-06-10 22:03:11 +08:00
chrislu
9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" 
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
 2022-06-10 06:38:17 -07:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID 
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
 2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal 
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
 2022-06-09 20:41:16 +08:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
chrislu
8402af0882 filer: detect ttl based on fs configure setting 2022-06-06 00:15:37 -07:00
chrislu
ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f remove unused logic 2022-06-04 19:11:06 -07:00
chrislu
376b005ad3 fix wrong logic 2022-05-31 11:55:58 -07:00
chrislu
27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu
596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
liubaojiang
f0ee3e6f21 reduce the scope of inFlightUploadDataLimitCond lock 2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
chrislu
7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
liubaojiang
076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e add inFlightUploadDataLimitCond signal 2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c exclude replication from the concurrentUploadLimitMB 2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync 2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9 hotfix_fsync fix fsync 2022-05-19 23:17:17 +08:00
guol-fnst
b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775 rename UUID file 
fix typo
move locationUUID  into DiskLocation
 2022-05-17 15:41:47 +08:00
guol-fnst
de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
Chris Lu
a69bf53405
Merge pull request #2996 from kmlebedev/ydb 
ydb filer store
 2022-05-05 15:34:37 -07:00
chrislu
b201edb9df fix wrong assignment 2022-05-03 07:12:33 -07:00
chrislu
9271866d1e fix segmentation violation 
fix https://github.com/chrislusf/seaweedfs/issues/3000
 2022-05-02 23:56:11 -07:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb 
# Conflicts:
#	go.mod
#	go.sum
 2022-05-03 00:13:57 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
chrislu
5d081d5c3c skip if already copied to remote 
fix https://github.com/chrislusf/seaweedfs/issues/2993
 2022-04-30 20:03:04 -07:00
Konstantin Lebedev
03aec9ce7e avoid panic if hashicorp raft not initialized 2022-04-30 13:56:35 +05:00
chrislu
94f824e1ce volume: sync to disk before copying volume files 
address https://github.com/chrislusf/seaweedfs/issues/2976
 2022-04-26 13:03:43 -07:00
guol-fnst
8b2552f944 fix http response error code 2022-04-24 11:07:27 +08:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
Konstantin Lebedev
1e35b4929f shell vacuum volume by collection and volume id 2022-04-18 18:40:58 +05:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu
b4be56bb3b add timing info during ping operation 2022-04-16 12:45:49 -07:00
a
846858fb43 merge master 2022-04-12 14:42:29 -07:00
Chris Lu
a87f57e47c
Merge pull request #2868 from kmlebedev/hashicorp_raft 
hashicorp raft
 2022-04-10 23:00:05 -07:00
Konstantin Lebedev
d2fe0fae33 use the full ServerAddress as the identifier 2022-04-11 10:44:44 +05:00
Chris Lu
9f4ecd50ac
Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style 
Change filer UI icon from picture to icon font.
 2022-04-08 00:50:32 -07:00
yulai.li
ceb058ddd9 Change filer UI icon from picture to icon font. 2022-04-08 15:07:56 +08:00
chrislu
7f26676f57 adjust td width 2022-04-07 22:34:32 -07:00
yulai.li
dff63e9024 Make operation buttons show when table row hovered 2022-04-08 13:10:12 +08:00
chrislu
4dba102137 use icon instead of button text 2022-04-07 11:15:01 -07:00
Chris Lu
bbc95dac47
Merge pull request #2885 from binbinshi/master 
fix: master lose some volumes
 2022-04-07 10:16:19 -07:00
Konstantin Lebedev
104ea7b029 master ui add raft stats and Max Volume Id 2022-04-07 20:52:01 +05:00
a
41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev
a2fdb3e277 fix master ui 2022-04-07 19:37:40 +05:00
Konstantin Lebedev
35bc67f030 fix conflicts 2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	weed/pb/master_pb/master.pb.go
 2022-04-07 18:50:27 +05:00
yulai.li
11a9f993a8 Make filer-ui bootstrap style 2022-04-07 20:17:00 +08:00
Konstantin Lebedev
7ff248d5cd refactor OnPeerUpdate 2022-04-07 16:23:22 +05:00
Konstantin Lebedev
85d80fd36d fix removing old raft server 2022-04-07 15:31:37 +05:00
chrislu
abe3cc6df2 filer UI touch up 2022-04-07 01:25:55 -07:00
yulai.li
2347c21cdd Tune filer UI add rename feature 2022-04-07 15:19:41 +08:00
yulai.li
2454020a92 Add upload progress support 2022-04-07 15:19:37 +08:00
yulai.li
4f87ee7755 Add create directory and delete web UI features for filer 2022-04-07 15:19:33 +08:00
shibinbin
c20e1edd99 fix: master lose some volumes 2022-04-07 15:18:28 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
Konstantin Lebedev
14a2cc83bf raft update peers via OnPeerUpdate 2022-04-06 21:17:04 +05:00
Konstantin Lebedev
357aa818fe add raft shell cmds 2022-04-06 15:23:53 +05:00
chrislu
79b8e6a8c3 add a place holder for later merge consecutive chunks 2022-04-06 00:28:48 -07:00
Konstantin Lebedev
c1aeded2dd force raft bootstrap 
avoid err bootstrap only works on new clusters
 2022-04-06 12:18:43 +05:00
chrislu
bc888226fc erasure coding: tracking encoded/decoded volumes 
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
 2022-04-05 19:03:02 -07:00
Konstantin Lebedev
0e796a5582 rm set NoSnapshotRestoreOnStart 2022-04-05 18:28:42 +05:00
Konstantin Lebedev
68f11b9687 sleep bootstraping 2022-04-05 15:42:19 +05:00
Konstantin Lebedev
1ae7f509f5 LocalID from raw string 2022-04-05 13:50:39 +05:00
Konstantin Lebedev
89d32a0670 fix glog Info 2022-04-05 13:43:34 +05:00
Konstantin Lebedev
17c6e8e39f Merge branch 'new_master' into hashicorp_raft 
# Conflicts:
#	go.mod
#	go.sum
 2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae auto bootstraping and update peers 2022-04-05 13:17:53 +05:00
Konstantin Lebedev
622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3 fix cluster status 2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu
6d55716fc3 use constants 2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4 master supports grpc ping 2022-04-01 16:50:58 -07:00
chrislu
743ad690b6 filer supports grpc ping 2022-04-01 16:44:58 -07:00
chrislu
2305508b65 refactor: separate into two files 2022-04-01 16:40:49 -07:00
chrislu
800cbc004c volume server adds ping function 2022-04-01 16:37:06 -07:00
a
549e341c6f resolv 2022-03-30 06:31:19 -05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517 fix permission mkdir snapshot 
avoid open file operation not permitted
 2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error 
Fix delete all "Seaweed-" prefixed tagging error
 2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc update code according to PR's comment 2022-03-28 15:23:31 +08:00
guosj
02076f01a6 fix tagging error 2022-03-28 14:50:48 +08:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
chrislu
fba1cfc2d6 simplify a bit 2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da refactor 2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers 
Fix remove deleted peers
 2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26 fix remove deleted peers of raft server 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe always clear previous log to avoid server is promotable 
https://github.com/chrislusf/seaweedfs/issues/2804
 2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request 
Need to exit waiting if request is was canceled
 2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf exclude the replication from the concurrentUploadLimitMB 2022-03-24 13:54:42 +05:00
chrislu
9b04f17555 remove dead code 2022-03-23 23:18:32 -07:00
chrislu
0b5faef612 fix 2022-03-23 23:16:54 -07:00
chrislu
0562fceb99 volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. 
fix https://github.com/chrislusf/seaweedfs/issues/2755
 2022-03-23 22:53:58 -07:00
zzq09494
9f16df736e fix-filer: calculation error of the method skipCheckParentDirEntry 2022-03-19 09:15:42 +08:00
elee
b25d03340b Merge branch 'a' of github.com:gfxlabs/seaweedfs into a 2022-03-17 04:51:59 -05:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
chrislu
4042fdf3bb rename to skipCheckParentDir 
related to https://github.com/chrislusf/seaweedfs/pull/2761

It's better to default to false.
 2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7 go fmt 2022-03-17 14:19:48 +08:00
zzq09494
81cce4b4c3 filer: support uploading file without needEnsureParentDir 2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255 Revert "filer: support uploading file without needEnsureParentDir" 
This reverts commit a93c4947ba.
 2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba filer: support uploading file without needEnsureParentDir 2022-03-17 10:18:23 +08:00
Konstantin Lebedev
f43c6daeda Need to exit waiting if request is was canceled 2022-03-15 19:55:22 +05:00
banjiaojuhao
f28dbbe5c5 [bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user. 2022-03-10 11:40:39 +08:00
chrislu
6d3db4445b buffer for all range requests 2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a a little optimization 2022-03-07 00:24:59 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest 
filer: support get metadata with resolved manifest chunk
 2022-03-07 00:05:46 -08:00
chrislu
bb0b784544 minor 2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808 filer: support metadata with resolved manifest chunk 2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841 filer: add back isAppend function 2022-03-07 15:41:07 +08:00
Konstantin Lebedev
cf444ebd07 Set default leveldb2 enabled 
avoid Filer store is enabled for both leveldb2 and mysql
 2022-03-06 18:27:25 +05:00
chrislu
f1713c96ae avoid possible runtime error: index out of range [0] with length 0 2022-03-05 21:14:31 -08:00
chrislu
a96d4254e9 filer, s3, volume server: a bit memory optimization 2022-03-02 20:15:28 -08:00
chrislu
b2a148cb4c use file size as max range 2022-02-26 03:00:08 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
e423548673 rename: pass along entry metadata 2022-02-25 02:53:37 -08:00
chrislu
320637dc7a use "mv.from" for moving files 2022-02-23 15:34:42 -08:00
banjiaojuhao
6ab09e9071 filer_http: support uploading file with offset 2022-02-22 00:15:00 +08:00
banjiaojuhao
e6126cef62 filer_web: support moving entry 2022-02-20 23:56:23 +08:00
banjiaojuhao
4c30934cd9 filer: support get file entry 2022-02-18 22:52:26 +08:00
Konstantin Lebedev
9ea09cc41c healthz check to avoid drain pod with last replicas 2022-02-16 14:18:36 +05:00
garenchan
bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
root
7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu
433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
chrislu
affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
Konstantin Lebedev
9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Konstantin Lebedev
c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu
4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu
e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu
826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu
b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
Sebastian Kurfuerst
fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
chrislu
67b723f74e Filer Server API support fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2528
 2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node 
filer server: allow upload file to specific dataNode
 2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu
4b8dcff448 reverting default admin scripts 
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
 2021-12-20 13:34:57 -08:00
chrislu
a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu
7210558c7b s3: pass through s3 presigned headers 
fix https://github.com/chrislusf/seaweedfs/discussions/2502
 2021-12-15 13:18:53 -08:00
chrislu
7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu
2d6fcdf83a add delay if need to resume from disk 2021-12-09 13:13:05 -08:00
liubaojiang
c7abc34dbb rename operation returns the correct old parent path 2021-12-10 00:02:57 +08:00
chrislu
40ee9b4d7b sleep between the gap of each execution 
related to https://github.com/chrislusf/seaweedfs/issues/2476
 2021-12-07 11:44:08 -08:00
chrislu
b70cb3e0b2 upgrade protoc to 3.17.3 
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
 2021-12-05 00:42:25 -08:00
Chris Lu
3d7390302d add s3.clean.uploads -timeAgo=24h 2021-11-29 00:49:49 -08:00
Chris Lu
88ff8fc27b ensure uploaded chunks are deleted on error 2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2 avoid empty extended attribute value 2021-11-28 23:50:52 -08:00
Chris Lu
a0ef6e3611 prevent nil response 
fix https://github.com/chrislusf/seaweedfs/issues/2452
 2021-11-15 08:46:01 -08:00
Chris Lu
4729a57cc0 use constants 2021-11-08 17:47:56 -08:00
Chris Lu
d9dd72ea56 rename pacakge 2021-11-08 17:47:56 -08:00
Chris Lu
c4e22b5a9a filer: deprecate "-peers" option 2021-11-06 14:36:45 -07:00
Chris Lu
e0fc2898e9 auto updated filer peer list 2021-11-06 14:23:35 -07:00
Chris Lu
84bb8e7365 send peers info to filers 2021-11-06 04:07:38 -07:00
Chris Lu
4b9c42996a refactor grpc API 2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
77f90ae288 add leader election in master 2021-11-04 00:54:38 -07:00
Chris Lu
3c245c69d3 handle possible race condition 2021-11-03 01:36:39 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
0c8dea9de8 go fmt 2021-11-02 23:39:16 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
24858507cc rename API to avoid confusion 2021-10-30 19:27:25 -07:00
Chris Lu
c857cc7286 cloud tier: remove tagging since not all s3 vendors support this 2021-10-29 12:39:19 -07:00
Chris Lu
5435027ff0 volume copy: stream out copying progress and avoid grpc request timeout 
fix https://github.com/chrislusf/seaweedfs/issues/2386
 2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Konstantin Lebedev
c78220a7f2 fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389 2021-10-20 16:01:06 +05:00
Chris Lu
29fa1b9fdc add logs 2021-10-17 13:50:34 -07:00
Chris Lu
cca62fdb30 mount: streaming renaming folders 2021-10-17 04:22:42 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
de1ec050e8 fier/s3: save "Content-Disposition" to extended properties 
related to https://github.com/chrislusf/seaweedfs/issues/2371
 2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f escape file name only when necessary 2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99 s3: use "response-content-disposition" to overwrite default content-disposition 
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371

see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
 2021-10-11 23:29:52 -07:00
Chris Lu
bf218cd59d removing etcd sequencer 
causing go mod tidy problem. If anyone wants this, please help to resolve this first.

github.com/chrislusf/seaweedfs/weed/sequence imports
	go.etcd.io/etcd/client tested by
	go.etcd.io/etcd/client.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:10:46 -07:00
Chris Lu
332d49432d reduce concurrent volume grow requests 2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00 refactor 2021-10-05 00:40:04 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
2e9372dcf7 volume stream read skips deleted content 2021-09-27 03:07:44 -07:00
Chris Lu
225b019fe0 stream read multiple volumes in a volume server 2021-09-27 02:51:31 -07:00
Chris Lu
1904448d4e adjust starting offset 2021-09-27 02:01:29 -07:00
Chris Lu
aa64f2ac4c send needle data instead of raw needle body 2021-09-27 01:53:41 -07:00
Chris Lu
c4d7ee6c5c volume server: read all files in a volume 2021-09-27 01:45:32 -07:00