a
41d396edc4
Merge branch 'master' into a
2022-04-07 08:22:59 -07:00
Konstantin Lebedev
a2fdb3e277
fix master ui
2022-04-07 19:37:40 +05:00
Konstantin Lebedev
35bc67f030
fix conflicts
2022-04-07 18:52:13 +05:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
yulai.li
11a9f993a8
Make filer-ui bootstrap style
2022-04-07 20:17:00 +08:00
Konstantin Lebedev
7ff248d5cd
refactor OnPeerUpdate
2022-04-07 16:23:22 +05:00
Konstantin Lebedev
85d80fd36d
fix removing old raft server
2022-04-07 15:31:37 +05:00
chrislu
abe3cc6df2
filer UI touch up
2022-04-07 01:25:55 -07:00
yulai.li
2347c21cdd
Tune filer UI add rename feature
2022-04-07 15:19:41 +08:00
yulai.li
2454020a92
Add upload progress support
2022-04-07 15:19:37 +08:00
yulai.li
4f87ee7755
Add create directory and delete web UI features for filer
2022-04-07 15:19:33 +08:00
shibinbin
c20e1edd99
fix: master lose some volumes
2022-04-07 15:18:28 +08:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a
2022-04-06 18:45:55 -05:00
Konstantin Lebedev
14a2cc83bf
raft update peers via OnPeerUpdate
2022-04-06 21:17:04 +05:00
Konstantin Lebedev
357aa818fe
add raft shell cmds
2022-04-06 15:23:53 +05:00
chrislu
79b8e6a8c3
add a place holder for later merge consecutive chunks
2022-04-06 00:28:48 -07:00
Konstantin Lebedev
c1aeded2dd
force raft bootstrap
...
avoid err bootstrap only works on new clusters
2022-04-06 12:18:43 +05:00
chrislu
bc888226fc
erasure coding: tracking encoded/decoded volumes
...
If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.
2022-04-05 19:03:02 -07:00
Konstantin Lebedev
0e796a5582
rm set NoSnapshotRestoreOnStart
2022-04-05 18:28:42 +05:00
Konstantin Lebedev
68f11b9687
sleep bootstraping
2022-04-05 15:42:19 +05:00
Konstantin Lebedev
1ae7f509f5
LocalID from raw string
2022-04-05 13:50:39 +05:00
Konstantin Lebedev
89d32a0670
fix glog Info
2022-04-05 13:43:34 +05:00
Konstantin Lebedev
17c6e8e39f
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# go.mod
# go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae
auto bootstraping and update peers
2022-04-05 13:17:53 +05:00
Konstantin Lebedev
622297f1a7
add stats raft handler
2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3
fix cluster status
2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
6d55716fc3
use constants
2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4
master supports grpc ping
2022-04-01 16:50:58 -07:00
chrislu
743ad690b6
filer supports grpc ping
2022-04-01 16:44:58 -07:00
chrislu
2305508b65
refactor: separate into two files
2022-04-01 16:40:49 -07:00
chrislu
800cbc004c
volume server adds ping function
2022-04-01 16:37:06 -07:00
a
549e341c6f
resolv
2022-03-30 06:31:19 -05:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517
fix permission mkdir snapshot
...
avoid open file operation not permitted
2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error
...
Fix delete all "Seaweed-" prefixed tagging error
2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc
update code according to PR's comment
2022-03-28 15:23:31 +08:00
guosj
02076f01a6
fix tagging error
2022-03-28 14:50:48 +08:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1
refactor
2022-03-26 13:13:19 -07:00
chrislu
fba1cfc2d6
simplify a bit
2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da
refactor
2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers
...
Fix remove deleted peers
2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26
fix remove deleted peers of raft server
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe
always clear previous log to avoid server is promotable
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request
...
Need to exit waiting if request is was canceled
2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf
exclude the replication from the concurrentUploadLimitMB
2022-03-24 13:54:42 +05:00
chrislu
9b04f17555
remove dead code
2022-03-23 23:18:32 -07:00
chrislu
0b5faef612
fix
2022-03-23 23:16:54 -07:00
chrislu
0562fceb99
volume: fail fast if too many concurrent requests, to avoid dead lock due to replication.
...
fix https://github.com/chrislusf/seaweedfs/issues/2755
2022-03-23 22:53:58 -07:00
zzq09494
9f16df736e
fix-filer: calculation error of the method skipCheckParentDirEntry
2022-03-19 09:15:42 +08:00
elee
b25d03340b
Merge branch 'a' of github.com:gfxlabs/seaweedfs into a
2022-03-17 04:51:59 -05:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
chrislu
4042fdf3bb
rename to skipCheckParentDir
...
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7
go fmt
2022-03-17 14:19:48 +08:00
zzq09494
81cce4b4c3
filer: support uploading file without needEnsureParentDir
2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255
Revert "filer: support uploading file without needEnsureParentDir"
...
This reverts commit a93c4947ba
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba
filer: support uploading file without needEnsureParentDir
2022-03-17 10:18:23 +08:00
Konstantin Lebedev
f43c6daeda
Need to exit waiting if request is was canceled
2022-03-15 19:55:22 +05:00
banjiaojuhao
f28dbbe5c5
[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.
2022-03-10 11:40:39 +08:00
chrislu
6d3db4445b
buffer for all range requests
2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a
a little optimization
2022-03-07 00:24:59 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest
...
filer: support get metadata with resolved manifest chunk
2022-03-07 00:05:46 -08:00
chrislu
bb0b784544
minor
2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808
filer: support metadata with resolved manifest chunk
2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841
filer: add back isAppend function
2022-03-07 15:41:07 +08:00
Konstantin Lebedev
cf444ebd07
Set default leveldb2 enabled
...
avoid Filer store is enabled for both leveldb2 and mysql
2022-03-06 18:27:25 +05:00
chrislu
f1713c96ae
avoid possible runtime error: index out of range [0] with length 0
2022-03-05 21:14:31 -08:00
chrislu
a96d4254e9
filer, s3, volume server: a bit memory optimization
2022-03-02 20:15:28 -08:00
chrislu
b2a148cb4c
use file size as max range
2022-02-26 03:00:08 -08:00
chrislu
28b395bef4
better control for reader caching
2022-02-26 02:16:47 -08:00
chrislu
e423548673
rename: pass along entry metadata
2022-02-25 02:53:37 -08:00
chrislu
320637dc7a
use "mv.from" for moving files
2022-02-23 15:34:42 -08:00
banjiaojuhao
6ab09e9071
filer_http: support uploading file with offset
2022-02-22 00:15:00 +08:00
banjiaojuhao
e6126cef62
filer_web: support moving entry
2022-02-20 23:56:23 +08:00
banjiaojuhao
4c30934cd9
filer: support get file entry
2022-02-18 22:52:26 +08:00
Konstantin Lebedev
9ea09cc41c
healthz check to avoid drain pod with last replicas
2022-02-16 14:18:36 +05:00
garenchan
bd032eabe7
[UPDATE] Make heartbeat interval and election timeout of masters configurable.
2022-02-14 21:09:07 +08:00
root
7f0c793083
fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions
2022-02-08 10:13:19 +08:00
chrislu
433fde4b18
move error to a separate file
...
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
...
error metrics for filer and store
2022-02-04 22:35:13 -08:00
chrislu
affe3c2c12
change to util.WriteFile
2022-02-04 21:32:27 -08:00
Konstantin Lebedev
9978f54acf
fix metric names
2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24
error metrics for filer and store
2022-02-04 14:07:14 +05:00
Konstantin Lebedev
c9952759c4
metrics master is leader
2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524
new master metrics
2022-01-24 19:09:43 +05:00
chrislu
4a311c7f5e
dedup local metadata subscribers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu
e69c374956
minor
2022-01-13 02:01:53 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu
b8fbf19e9a
mount: rename follow POSIX
2022-01-11 03:23:03 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
Sebastian Kurfuerst
fcc09cef6f
Refactor: pass in claim type into security.DecodeJwt
2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef
rename security.GenJwt to security.GenJwtForVolumeServer
2021-12-29 12:39:41 +01:00
chrislu
67b723f74e
Filer Server API support fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
