Commit Graph

686 Commits

Author SHA1 Message Date
chrislu
ef78631a7c just a bit safer 2022-08-31 00:10:17 -07:00
chrislu
9c944377dc minor 2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6 s3: close response body with copying object 2022-08-31 00:09:23 -07:00
chrislu
b7a887fea1 3.24 2022-08-28 21:53:13 -07:00
chrislu
dbf0de4ce1 minor clean up 2022-08-25 00:19:08 -07:00
chrislu
c4e862e908 3.23 2022-08-21 19:13:26 -07:00
chrislu
f7e0a65e75 retry for all errors 2022-08-20 23:34:45 -07:00
chrislu
aac45f3e89 filer: retryable when error is not found 2022-08-19 03:36:15 -07:00
chrislu
58dd880322 minor 2022-08-18 23:51:08 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
chrislu
fa4d0093e1 3.22 2022-08-15 16:48:23 -07:00
chrislu
cb476a53ff remove logs 2022-08-15 01:05:35 -07:00
chrislu
7c029b2183 3.21 2022-08-15 00:32:15 -07:00
chrislu
ae93c966d9 ensure memory is aligned
fix https://github.com/seaweedfs/seaweedfs/issues/3427
2022-08-10 22:27:13 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code (#3423)
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-09 01:24:43 -07:00
chrislu
0854171d22 3.20 2022-08-07 14:42:28 -07:00
chrislu
67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
chrislu
1a4bf0dcb5 filer.sync: parallelize the filer.sync 2022-08-07 00:56:15 -07:00
chrislu
0e9478488d filer.sync: fix when excluded paths is empty 2022-08-07 00:55:34 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405)
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
2022-08-04 17:35:00 -07:00
chrislu
03c6f978f4 3.19 2022-07-31 17:16:51 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu
475185fb72 3.18 2022-07-25 22:41:38 -07:00
chrislu
dbe4849ffd 3.17 2022-07-24 18:46:50 -07:00
chrislu
2ae3f812f8 minor 2022-07-19 11:43:31 -07:00
chrislu
56ec89625a 3.16 2022-07-18 00:53:05 -07:00
chrislu
93ccc6e05f 3.15 2022-07-11 00:50:07 -07:00
chrislu
3c79c77056 3.14 2022-07-04 13:15:49 -07:00
chrislu
40a9634c3d 3.13 2022-06-26 20:13:52 -07:00
chrislu
4dc27e1ed5 3.12 2022-06-20 19:07:00 -07:00
chrislu
d4ef06cdcf 3.11 2022-06-14 12:14:52 -07:00
chrislu
e437ec5a68 3.10 2022-06-12 19:30:27 -07:00
chrislu
4a046e4de7 3.09 2022-06-05 19:00:24 -07:00
chrislu
8a49240d64 3.08 2022-05-31 11:57:41 -07:00
chrislu
fbd99d53c1 3.07 2022-05-31 00:36:24 -07:00
chrislu
968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu
7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
chrislu
2f846777bb 3.06 2022-05-23 01:21:48 -07:00
chrislu
acc318e12b 3.05 2022-05-22 21:43:59 -07:00
chrislu
9ff0d99002 3.04 2022-05-15 21:32:21 -07:00
chrislu
cd2d10118c 3.03 2022-05-15 18:17:07 -07:00
chrislu
a4ca3ed1f0 3.02 2022-05-08 22:28:32 -07:00
chrislu
ef6c6c450e avoid fatal error if port is already in use 2022-05-04 14:55:14 -07:00
chrislu
73961e24d8 3.01 2022-05-01 23:33:45 -07:00
chrislu
b2a6111090 skip ipv6 all interfaces and localhost
fix https://github.com/chrislusf/seaweedfs/issues/2983
2022-04-27 17:18:09 -07:00
chrislu
2aef5b8b32 3.00 2022-04-24 21:46:03 -07:00
chrislu
9ae66f55fc 2.99 2022-04-17 23:01:42 -07:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu
c6ec5269f4 2.98 2022-04-10 18:56:40 -07:00
chrislu
d310711de0 avoid possible deadlock: move metadata log flush channel out of lock scope 2022-04-03 21:53:30 -07:00
chrislu
77a7d7253f 2.97 2022-04-03 19:08:01 -07:00
chrislu
0490ee87ef 2.96 2022-03-27 16:11:17 -07:00
chrislu
8f0410af2c 2.95 2022-03-21 01:47:03 -07:00
chrislu
3da2b83b38 Added a "-conf_dir" option to customize *.toml configuration file directory.
fix https://github.com/chrislusf/seaweedfs/issues/2753
2022-03-19 00:22:47 -07:00
chrislu
f247cab5cd skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1 2022-03-17 16:54:29 -07:00
chrislu
3639cad69c master, filer, s3: also listen to "localhost" in addition to specific ip address
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
chrislu
2eda3a686f 2.94 2022-03-14 00:55:01 -07:00
chrislu
bd5c5586b5 generate inode via path and time 2022-03-14 00:03:29 -07:00
chrislu
0ba4e4cd23 2.93 2022-03-06 18:54:12 -08:00
chrislu
784583afc6 avoid pool memory allocation if too large 2022-03-02 13:50:28 -08:00
chrislu
ba14307319 2.92 2022-02-28 15:22:19 -08:00
chrislu
09cd00f356 2.91 2022-02-27 04:03:39 -08:00
chrislu
d602d68fd1 remove dead code 2022-02-27 03:41:32 -08:00
chrislu
708e14fcfa avoid possible too big memory allocation 2022-02-26 03:22:41 -08:00
chrislu
2ab0ad24a3 use memory pool 2022-02-26 02:59:19 -08:00
chrislu
28b395bef4 better control for reader caching 2022-02-26 02:16:47 -08:00
chrislu
3ad5fa6f6f chunk cache adds function ReadChunkAt 2022-02-25 21:55:04 -08:00
chrislu
497ebbbd45 2.90 2022-02-20 22:00:13 -08:00
chrislu
6a40fd1c65 2.89 2022-02-14 01:52:16 -08:00
Eng Zer Jun
b92df1654c
test: use T.TempDir to create temporary test directory
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-02-14 10:38:13 +08:00
root
7f0c793083 fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions 2022-02-08 10:13:19 +08:00
chrislu
76e297d64f sync call to write file, avoid vif loading error
fix https://github.com/chrislusf/seaweedfs/issues/2633
2022-02-04 11:14:04 -08:00
chrislu
7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu
e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu
b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu
e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu
9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
chrislu
de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu
fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu
2dcb8cb93b POSIX: ensure file and directory inodes are different
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
2022-01-11 23:44:48 -08:00
chrislu
19555385f7 2.85 2022-01-09 19:30:23 -08:00
chrislu
5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
2021-12-30 14:45:27 +01:00
chrislu
fb434318e3 dynamically adjust connection timeout
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu
5788bf2270 s3: increase timeout limit
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
chrislu
c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu
c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu
5ea9715721 2.81
also sync java client version to SeaweedFS version
2021-12-05 18:05:24 -08:00
Tanmoy Majumdar
ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu
7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu
3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
Chris Lu
f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Chris Lu
100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu
5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu
fc9e246592 2.76 2021-10-31 18:08:28 -07:00