Konstantin Lebedev
17c6e8e39f
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# go.mod
# go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
b7cdde14ae
auto bootstraping and update peers
2022-04-05 13:17:53 +05:00
Konstantin Lebedev
622297f1a7
add stats raft handler
2022-04-04 19:16:06 +05:00
Konstantin Lebedev
1a1e5778c3
fix cluster status
2022-04-04 18:52:08 +05:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
6d55716fc3
use constants
2022-04-01 16:55:26 -07:00
chrislu
bbbbbd70a4
master supports grpc ping
2022-04-01 16:50:58 -07:00
chrislu
743ad690b6
filer supports grpc ping
2022-04-01 16:44:58 -07:00
chrislu
2305508b65
refactor: separate into two files
2022-04-01 16:40:49 -07:00
chrislu
800cbc004c
volume server adds ping function
2022-04-01 16:37:06 -07:00
a
549e341c6f
resolv
2022-03-30 06:31:19 -05:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
Konstantin Lebedev
84b7b83517
fix permission mkdir snapshot
...
avoid open file operation not permitted
2022-03-28 18:41:52 +05:00
Chris Lu
93615b2a49
Merge pull request #2828 from guo-sj/fix-tagging-error
...
Fix delete all "Seaweed-" prefixed tagging error
2022-03-28 00:35:22 -07:00
guosj
dd2b9d93cc
update code according to PR's comment
2022-03-28 15:23:31 +08:00
guosj
02076f01a6
fix tagging error
2022-03-28 14:50:48 +08:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1
refactor
2022-03-26 13:13:19 -07:00
chrislu
fba1cfc2d6
simplify a bit
2022-03-26 10:24:05 -07:00
chrislu
a3411dd9da
refactor
2022-03-26 10:21:26 -07:00
Chris Lu
1b27f96669
Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers
...
Fix remove deleted peers
2022-03-25 11:36:15 -07:00
Konstantin Lebedev
ddd3945c26
fix remove deleted peers of raft server
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 15:09:38 +05:00
Konstantin Lebedev
c1450bf9fe
always clear previous log to avoid server is promotable
...
https://github.com/chrislusf/seaweedfs/issues/2804
2022-03-25 13:40:19 +05:00
Chris Lu
89d84e275b
Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request
...
Need to exit waiting if request is was canceled
2022-03-24 12:21:44 -07:00
Konstantin Lebedev
0b790d2bbf
exclude the replication from the concurrentUploadLimitMB
2022-03-24 13:54:42 +05:00
chrislu
9b04f17555
remove dead code
2022-03-23 23:18:32 -07:00
chrislu
0b5faef612
fix
2022-03-23 23:16:54 -07:00
chrislu
0562fceb99
volume: fail fast if too many concurrent requests, to avoid dead lock due to replication.
...
fix https://github.com/chrislusf/seaweedfs/issues/2755
2022-03-23 22:53:58 -07:00
zzq09494
9f16df736e
fix-filer: calculation error of the method skipCheckParentDirEntry
2022-03-19 09:15:42 +08:00
elee
b25d03340b
Merge branch 'a' of github.com:gfxlabs/seaweedfs into a
2022-03-17 04:51:59 -05:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
chrislu
4042fdf3bb
rename to skipCheckParentDir
...
related to https://github.com/chrislusf/seaweedfs/pull/2761
It's better to default to false.
2022-03-16 23:55:31 -07:00
zzq09494
40b0033fa7
go fmt
2022-03-17 14:19:48 +08:00
zzq09494
81cce4b4c3
filer: support uploading file without needEnsureParentDir
2022-03-17 10:53:47 +08:00
zzq09494
a6a8892255
Revert "filer: support uploading file without needEnsureParentDir"
...
This reverts commit a93c4947ba
2022-03-17 10:27:17 +08:00
zzq09494
a93c4947ba
filer: support uploading file without needEnsureParentDir
2022-03-17 10:18:23 +08:00
Konstantin Lebedev
f43c6daeda
Need to exit waiting if request is was canceled
2022-03-15 19:55:22 +05:00
banjiaojuhao
f28dbbe5c5
[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.
2022-03-10 11:40:39 +08:00
chrislu
6d3db4445b
buffer for all range requests
2022-03-07 01:56:47 -08:00
chrislu
f3bcbeb60a
a little optimization
2022-03-07 00:24:59 -08:00
Chris Lu
1e7fcef581
Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest
...
filer: support get metadata with resolved manifest chunk
2022-03-07 00:05:46 -08:00
chrislu
bb0b784544
minor
2022-03-07 00:04:59 -08:00
banjiaojuhao
bfcc9ca808
filer: support metadata with resolved manifest chunk
2022-03-07 15:47:51 +08:00
banjiaojuhao
71f3046841
filer: add back isAppend function
2022-03-07 15:41:07 +08:00
Konstantin Lebedev
cf444ebd07
Set default leveldb2 enabled
...
avoid Filer store is enabled for both leveldb2 and mysql
2022-03-06 18:27:25 +05:00
chrislu
f1713c96ae
avoid possible runtime error: index out of range [0] with length 0
2022-03-05 21:14:31 -08:00
chrislu
a96d4254e9
filer, s3, volume server: a bit memory optimization
2022-03-02 20:15:28 -08:00
chrislu
b2a148cb4c
use file size as max range
2022-02-26 03:00:08 -08:00
chrislu
28b395bef4
better control for reader caching
2022-02-26 02:16:47 -08:00
chrislu
e423548673
rename: pass along entry metadata
2022-02-25 02:53:37 -08:00
chrislu
320637dc7a
use "mv.from" for moving files
2022-02-23 15:34:42 -08:00
banjiaojuhao
6ab09e9071
filer_http: support uploading file with offset
2022-02-22 00:15:00 +08:00
banjiaojuhao
e6126cef62
filer_web: support moving entry
2022-02-20 23:56:23 +08:00
banjiaojuhao
4c30934cd9
filer: support get file entry
2022-02-18 22:52:26 +08:00
Konstantin Lebedev
9ea09cc41c
healthz check to avoid drain pod with last replicas
2022-02-16 14:18:36 +05:00
garenchan
bd032eabe7
[UPDATE] Make heartbeat interval and election timeout of masters configurable.
2022-02-14 21:09:07 +08:00
root
7f0c793083
fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions
2022-02-08 10:13:19 +08:00
chrislu
433fde4b18
move error to a separate file
...
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics
...
error metrics for filer and store
2022-02-04 22:35:13 -08:00
chrislu
affe3c2c12
change to util.WriteFile
2022-02-04 21:32:27 -08:00
Konstantin Lebedev
9978f54acf
fix metric names
2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24
error metrics for filer and store
2022-02-04 14:07:14 +05:00
Konstantin Lebedev
c9952759c4
metrics master is leader
2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524
new master metrics
2022-01-24 19:09:43 +05:00
chrislu
4a311c7f5e
dedup local metadata subscribers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2542
2022-01-23 16:14:22 -08:00
chrislu
e69c374956
minor
2022-01-13 02:01:53 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu
b8fbf19e9a
mount: rename follow POSIX
2022-01-11 03:23:03 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
Sebastian Kurfuerst
fcc09cef6f
Refactor: pass in claim type into security.DecodeJwt
2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef
rename security.GenJwt to security.GenJwtForVolumeServer
2021-12-29 12:39:41 +01:00
chrislu
67b723f74e
Filer Server API support fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node
...
filer server: allow upload file to specific dataNode
2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e
filer server: allow upload file to specific dataNode
2021-12-22 21:57:26 +08:00
chrislu
4b8dcff448
reverting default admin scripts
...
fix https://github.com/chrislusf/seaweedfs/issues/2525
this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
2021-12-20 13:34:57 -08:00
chrislu
a152f17937
mount: improve read performance on random reads
2021-12-19 22:43:14 -08:00
chrislu
7210558c7b
s3: pass through s3 presigned headers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
chrislu
7bf14452f7
s3: handle Response-Content-Disposition
2021-12-12 23:05:40 -08:00
chrislu
a2d3f89c7b
add lock messages
2021-12-10 13:24:38 -08:00
chrislu
2d6fcdf83a
add delay if need to resume from disk
2021-12-09 13:13:05 -08:00
liubaojiang
c7abc34dbb
rename operation returns the correct old parent path
2021-12-10 00:02:57 +08:00
chrislu
40ee9b4d7b
sleep between the gap of each execution
...
related to https://github.com/chrislusf/seaweedfs/issues/2476
2021-12-07 11:44:08 -08:00
chrislu
b70cb3e0b2
upgrade protoc to 3.17.3
...
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
2021-12-05 00:42:25 -08:00
Chris Lu
3d7390302d
add s3.clean.uploads -timeAgo=24h
2021-11-29 00:49:49 -08:00
Chris Lu
88ff8fc27b
ensure uploaded chunks are deleted on error
2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2
avoid empty extended attribute value
2021-11-28 23:50:52 -08:00
Chris Lu
a0ef6e3611
prevent nil response
...
fix https://github.com/chrislusf/seaweedfs/issues/2452
2021-11-15 08:46:01 -08:00
Chris Lu
4729a57cc0
use constants
2021-11-08 17:47:56 -08:00
Chris Lu
d9dd72ea56
rename pacakge
2021-11-08 17:47:56 -08:00
Chris Lu
c4e22b5a9a
filer: deprecate "-peers" option
2021-11-06 14:36:45 -07:00
Chris Lu
e0fc2898e9
auto updated filer peer list
2021-11-06 14:23:35 -07:00
Chris Lu
84bb8e7365
send peers info to filers
2021-11-06 04:07:38 -07:00
Chris Lu
4b9c42996a
refactor grpc API
2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
2021-11-05 17:52:15 -07:00
Chris Lu
77f90ae288
add leader election in master
2021-11-04 00:54:38 -07:00
Chris Lu
3c245c69d3
handle possible race condition
2021-11-03 01:36:39 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
0c8dea9de8
go fmt
2021-11-02 23:39:16 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
24858507cc
rename API to avoid confusion
2021-10-30 19:27:25 -07:00
Chris Lu
c857cc7286
cloud tier: remove tagging since not all s3 vendors support this
2021-10-29 12:39:19 -07:00
Chris Lu
5435027ff0
volume copy: stream out copying progress and avoid grpc request timeout
...
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Konstantin Lebedev
c78220a7f2
fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389
2021-10-20 16:01:06 +05:00
Chris Lu
29fa1b9fdc
add logs
2021-10-17 13:50:34 -07:00
Chris Lu
cca62fdb30
mount: streaming renaming folders
2021-10-17 04:22:42 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
de1ec050e8
fier/s3: save "Content-Disposition" to extended properties
...
related to https://github.com/chrislusf/seaweedfs/issues/2371
2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f
escape file name only when necessary
2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99
s3: use "response-content-disposition" to overwrite default content-disposition
...
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371
see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
2021-10-11 23:29:52 -07:00
Chris Lu
bf218cd59d
removing etcd sequencer
...
causing go mod tidy problem. If anyone wants this, please help to resolve this first.
github.com/chrislusf/seaweedfs/weed/sequence imports
go.etcd.io/etcd/client tested by
go.etcd.io/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
332d49432d
reduce concurrent volume grow requests
2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00
refactor
2021-10-05 00:40:04 -07:00
Chris Lu
366f522a2d
add redis3
2021-10-04 01:01:31 -07:00
Chris Lu
2e9372dcf7
volume stream read skips deleted content
2021-09-27 03:07:44 -07:00
Chris Lu
225b019fe0
stream read multiple volumes in a volume server
2021-09-27 02:51:31 -07:00
Chris Lu
1904448d4e
adjust starting offset
2021-09-27 02:01:29 -07:00
Chris Lu
aa64f2ac4c
send needle data instead of raw needle body
2021-09-27 01:53:41 -07:00
Chris Lu
c4d7ee6c5c
volume server: read all files in a volume
2021-09-27 01:45:32 -07:00
Chris Lu
603ea2db73
avoid looping forever if there are no more metadata updates
2021-09-26 11:55:27 -07:00
Chris Lu
ede7a65a50
update log
2021-09-19 01:33:50 -07:00
Chris Lu
c1255d30ad
add cache error log
2021-09-19 01:33:26 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
7504be58f9
Avoid xattr printed out as HTTP headers
...
fix https://github.com/chrislusf/seaweedfs/issues/2336
2021-09-13 04:00:57 -07:00
Chris Lu
f74b29416a
better etag matching
2021-09-13 00:31:46 -07:00
Chris Lu
e9760f261e
minor
2021-09-13 00:31:06 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
7591336a22
log format
2021-09-11 14:27:57 -07:00
Chris Lu
7e600bff5f
adjust log level
2021-09-11 12:59:15 -07:00
Chris Lu
a243d7e047
EC: clean up if failed in the middle
2021-09-11 02:07:14 -07:00
Chris Lu
f7f01ac209
return if error
2021-09-11 02:06:43 -07:00
Chris Lu
59cb6146a3
refactor
2021-09-11 02:06:30 -07:00
Chris Lu
5496d68f6a
increase counter only if not early terminated
2021-09-11 02:05:55 -07:00
Chris Lu
0207f5fe9b
replicated remote.cache
2021-09-08 15:54:55 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
Chris Lu
2b1feb732c
remote.cache supports replication
2021-09-06 18:30:44 -07:00
Chris Lu
6923af7280
refactoring
2021-09-06 16:20:49 -07:00
Chris Lu
e93d4935e3
add other replica locations when assigning volumes
2021-09-05 23:32:25 -07:00
Chris Lu
7a13816e94
refactor
2021-09-05 23:17:15 -07:00
Chris Lu
006c01a519
fix format
2021-09-05 16:18:50 -07:00
Chris Lu
65af3cf4df
master: disconnect only the phantom volume server
...
fix https://github.com/chrislusf/seaweedfs/issues/2311
2021-09-05 15:20:03 -07:00
Chris Lu
24fda6f47f
remove QR code for mobile upload
2021-09-01 16:59:37 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
d1a4e19a3f
volume: copy file also copies modification time
...
to ensure ttl can work well
2021-09-01 02:42:57 -07:00
