mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 15:28:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,126 Commits 32 Branches 296 Tags 152 MiB
7b3c0e937f
Commit Graph

803 Commits

Author SHA1 Message Date
chrislu
19555385f7 2.85 2022-01-09 19:30:23 -08:00
chrislu
5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu
fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu
5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
chrislu
c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu
c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu
5ea9715721 2.81 
also sync java client version to SeaweedFS version
 2021-12-05 18:05:24 -08:00
Tanmoy Majumdar
ea09fb477a return ' shouldRetry=true' so that filer can retry the failed chunk 2021-12-03 11:54:20 +06:00
Chris Lu
7227cfddf5 2.80 2021-11-29 00:57:08 -08:00
Chris Lu
3a19eea97c allocate memory by slabs 2021-11-27 12:13:00 -08:00
Chris Lu
f3c789d662 2.79 2021-11-21 18:40:24 -08:00
Chris Lu
100c654ec3 2.78 2021-11-14 23:29:59 -08:00
Chris Lu
5cf332357b 2.77 2021-11-07 13:52:45 -08:00
Chris Lu
fc9e246592 2.76 2021-10-31 18:08:28 -07:00
Chris Lu
c9d3fb4a30 2.75 2021-10-24 18:15:59 -07:00
Chris Lu
182f43ae5f 2.74 2021-10-18 14:23:54 -07:00
Chris Lu
cd4fa7561b 2.73 2021-10-18 10:47:48 -07:00
Chris Lu
97c963bac9 2.72 2021-10-17 17:40:27 -07:00
Chris Lu
3833dac3f7 continue to read from memory if there is no flush 2021-10-17 13:53:04 -07:00
Chris Lu
8965a53c4d add warning error 2021-10-16 15:57:30 -07:00
Chris Lu
5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil 
refactor: move from io/ioutil to io and os package
 2021-10-13 22:38:58 -07:00
Chris Lu
46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu
3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu
e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu
f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu
371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu
893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu
4ed2994555 use tsMemory to determine whether read from disk or memory 
remove lastFlushTime
 2021-10-04 16:02:56 -07:00
Chris Lu
513fed323a SkipListElementReference can be an empty object 2021-10-04 02:30:44 -07:00
Chris Lu
280ab7f95c add test 2021-10-04 02:30:24 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
ba7fbac07f rename 2021-10-03 19:23:34 -07:00
Chris Lu
e6196cdc50 add name list 2021-10-03 17:54:25 -07:00
Chris Lu
a481c4a45e return previous element if visited 2021-10-03 13:50:52 -07:00
Chris Lu
22d8684e88 refactor out listStore 2021-10-03 02:19:21 -07:00
Chris Lu
d343b0db57 update value 2021-10-03 01:15:14 -07:00
Chris Lu
4f50f8c2ca insert key and value 2021-10-03 01:07:35 -07:00
Chris Lu
69b84bb771 TestFindGreaterOrEqual 2021-10-02 14:15:49 -07:00
Chris Lu
57e2fd3f9b remove bptree 2021-10-02 14:03:54 -07:00
Chris Lu
4c1741fdbb working skiplist 2021-10-02 14:02:56 -07:00
Chris Lu
b6694279d7 Merge branch 'master' into bptree 2021-10-01 16:55:44 -07:00
Chris Lu
1e3fdf366f go fmt 2021-10-01 12:10:24 -07:00
Chris Lu
cee4d20bc1 2.70 2021-09-26 17:37:46 -07:00
Chris Lu
603ea2db73 avoid looping forever if there are no more metadata updates 2021-09-26 11:55:27 -07:00
Chris Lu
9887610b54 log tsNs should be processing time 2021-09-26 11:54:13 -07:00
Chris Lu
2baed2e1e9 avoid possible metadata subscription data loss 
Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.
 2021-09-25 01:18:44 -07:00
Chris Lu
b3d88180ca Merge branch 'master' into bptree 2021-09-19 23:56:59 -07:00
Chris Lu
fa7c65bd4b 2.69 2021-09-19 21:44:06 -07:00
Chris Lu
e066e2642c add NodeStore 2021-09-18 15:32:17 -07:00
Chris Lu
8f2e4be074 wip 2021-09-18 14:04:30 -07:00
Chris Lu
b751debd31 split node based on the last inserted key 2021-09-18 01:29:47 -07:00
Chris Lu
2226c3c8b6 Merge branch 'master' into bptree 2021-09-17 10:35:21 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
20ac710ceb 2.68 2021-09-13 02:16:09 -07:00
Chris Lu
574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu
9fdf02bcda remove detecting ipv6 
Got this error on my local:

transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host

related to https://github.com/chrislusf/seaweedfs/pull/2310
 2021-09-07 02:31:34 -07:00
Chris Lu
8c6d706328 2.67 2021-09-07 00:08:03 -07:00
Chris Lu
6022db6d6a 2.66 2021-09-05 16:21:14 -07:00
Chris Lu
2348e8d8da
Merge pull request #2310 from nivekuil/ipv6 
Detect ipv6 addresses
 2021-09-05 10:56:44 -07:00
nivekuil
0fe9d2997b Detect ipv6 addresses 2021-09-05 06:21:40 -07:00
Chris Lu
c5ee03d6af format 2021-09-04 13:57:55 -07:00
Chris Lu
8ec357b3d3 go mod 2021-09-03 23:25:33 -07:00
Chris Lu
03a31587ce go fmt 2021-09-03 20:42:28 -07:00
Chris Lu
bca4a9de78 simplify 2021-09-02 23:09:24 -07:00
Chris Lu
958125bd02 conforming to http user agent common practice 2021-09-02 22:55:35 -07:00
Chris Lu
11a496404b reset wait time 2021-09-02 19:55:01 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
bec3f63298 2.65 2021-08-28 05:27:33 -07:00
Chris Lu
ff7dc3b44c 2.64 2021-08-23 00:39:15 -07:00
Chris Lu
df1d6133a8 bptree does not work well for auto-increasing keys 2021-08-22 18:19:26 -07:00
Chris Lu
51c8f2518f change key type to ItemKey 2021-08-21 15:54:42 -07:00
Chris Lu
b3e49d2758 change value type to ItemValue 2021-08-21 15:52:17 -07:00
Chris Lu
38c8470d1d add back non_dedup 2021-08-21 15:13:13 -07:00
Chris Lu
849f185a20 add memory kv store 2021-08-21 15:00:44 -07:00
Chris Lu
5f6cc9a814 make proto node 2021-08-21 13:36:52 -07:00
Chris Lu
172da83449 bpnode use get prev and next 2021-08-20 18:50:16 -07:00
Chris Lu
01661ec6a7 move to getter setter file 2021-08-20 18:37:34 -07:00
Chris Lu
0c360eb6b2 add getter and setter for root of tree and map 2021-08-20 18:34:50 -07:00
Chris Lu
88d68cad87 remove dedup 2021-08-20 04:14:52 -07:00
qieqieplus
7720533f84 reduce gzip allocation 2021-08-20 18:38:18 +08:00
Chris Lu
2d237da74a remove size since each put/get will have to update the root node 2021-08-20 01:19:11 -07:00
Chris Lu
ec72547c8d started by copying from https://sourcegraph.com/github.com/timtadh/data-structures@master/-/tree/tree/bptree 2021-08-20 01:12:52 -07:00
Chris Lu
1f35d32be0 2.63 2021-08-15 23:14:59 -07:00
Chris Lu
bb94930196 add some delays if error 2021-08-15 20:06:47 -07:00
Chris Lu
c3ffd457ef fix compilation error 2021-08-15 12:40:22 -07:00
Chris Lu
fda2fc47b1 add RetryForever 2021-08-15 12:37:35 -07:00
Chris Lu
9462f5129a shell: add "remote.meta.sync" 2021-08-15 01:53:46 -07:00
Chris Lu
5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu
8cfd487608 2.62 2021-08-08 23:33:12 -07:00
Chris Lu
56ee1d5ef1 2.61 2021-08-01 15:50:19 -07:00
Chris Lu
5dede5d38d 2.60 2021-07-25 22:09:09 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
a45bbc0b75 2.59 2021-07-15 15:52:22 -07:00
Chris Lu
297b41266b 2.58 2021-07-12 01:33:47 -07:00
bingoohuang
ed57a55eae show RemoteVolumes/EcVolumes only if it is not empty 2021-07-06 15:20:18 +08:00
Chris Lu
f5fa0b08fd 2.57 2021-07-03 15:10:57 -07:00
Chris Lu
2420c60fc4 log reading adds delay between retries 2021-07-01 14:01:25 -07:00
Chris Lu
b624090398 go fmt 2021-07-01 01:21:14 -07:00
Chris Lu
a2979aa051 2.56 2021-06-27 23:33:45 -07:00
Chris Lu
c764596f96 filer: slow metadata topic read may lose some change events 
fix https://github.com/chrislusf/seaweedfs/issues/2117
 2021-06-27 05:54:16 -07:00
Chris Lu
cc7714fdbe logging changes to debug 2021-06-27 05:54:16 -07:00
Chris Lu
05af54ad10 2.55 
trigger migrated travis build
 2021-06-23 00:41:04 -07:00
Chris Lu
b3eb4fecc7 2.54 2021-06-19 03:48:15 -07:00
Chris Lu
7a81caa31e 2.53 2021-06-13 17:12:34 -07:00
Chris Lu
7225cb4ac5 add block and mutex profiling 2021-06-13 16:15:54 -07:00
Chris Lu
76f24af79a 2.52 2021-06-07 12:13:23 -07:00
Chris Lu
c8dea3dd89 2.51 2021-06-06 21:54:00 -07:00
Chris Lu
556cc3a4ca mount: avoid exception if disk cache is not initialized 
related to https://github.com/chrislusf/seaweedfs/issues/2102
 2021-05-31 16:42:55 -07:00
Chris Lu
4233ad3f07 2.50 2021-05-30 20:40:30 -07:00
Chris Lu
dce1f02c9e filer.backup: backup to local directory optionally is incremental 
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
 2021-05-25 17:19:20 -07:00
Chris Lu
42fb03a66e 2.49 2021-05-23 00:51:47 -07:00
Chris Lu
45a762223a 2.48 2021-05-14 10:26:42 -07:00
Chris Lu
0f7b43af99 2.47 2021-05-11 10:12:15 -07:00
Chris Lu
4596e64710 2.46 2021-05-10 21:58:37 -07:00
Chris Lu
fe2edd4b50 2.45 2021-05-10 13:53:03 -07:00
Chris Lu
aa6949ef25 2.44 2021-05-09 23:28:48 -07:00
Chris Lu
3942e3b2ef a better fix 2021-05-09 01:42:19 -07:00
Chris Lu
957e1a1bc1 fuse: important: if filer -filer.path is not root, directory listing will fail 2021-05-09 00:32:21 -07:00
Chris Lu
c48ef78670 2.43 2021-05-01 00:39:04 -07:00
Chris Lu
3a86d4dbfd mount: fix directory invalidation 
fix https://github.com/chrislusf/seaweedfs/issues/2038
 2021-04-30 22:51:06 -07:00
Chris Lu
84312e6799 2.42 2021-04-30 03:14:07 -07:00
Chris Lu
fa0d973113
Merge pull request #2037 from utsl42/reader_at 
make reader_at handle random reads more efficiently for FUSE
 2021-04-28 17:31:46 -07:00
Nathan Hawkins
042de9359c make reader_at handle random reads more efficiently for FUSE 2021-04-28 19:13:37 -04:00
Chris Lu
a26a37dfa3 fix compilation 
fix related to #2032
 2021-04-28 13:36:53 -07:00
Chris Lu
9dca75aea8
Merge pull request #2032 from tobiasmuehl/patch-2 
Detect rar archives by mime type
 2021-04-28 10:31:19 -07:00
Tobias Mühl
a8864e2abd
Detect rar archives by mime type 
RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format))
 2021-04-28 15:54:19 +07:00
Tobias Mühl
12a7e87007
Do not compress brotli archives 2021-04-28 15:51:49 +07:00
bingoohuang
cf552417a7 minFreeSpace refactored 2021-04-27 10:37:24 +08:00
bingoohuang
31f1cdeac2 minFreeSpace argument allows size like 10GiB 2021-04-26 18:48:34 +08:00
Chris Lu
86185262bb 2.41 2021-04-24 16:54:36 -07:00
Chris Lu
e983f91b03 2.40 2021-04-18 13:58:01 -07:00
Chris Lu
6bc09b18c4 truncate is a bit faster to reuse the storage 2021-04-14 20:26:56 -07:00
Chris Lu
742ab1ec81 2.39 2021-04-11 19:47:11 -07:00
Chris Lu
f62c153274 go fmt 2021-04-10 23:48:18 -07:00
Chris Lu
af313dff58 add gateway for easier POST and DELETE blobs 2021-04-10 23:47:47 -07:00
Chris Lu
a37eca78cd 2.38 2021-04-05 19:41:54 -07:00
Chris Lu
6eee200c13 2.37 2021-04-04 18:45:48 -07:00
Chris Lu
fbb82a5c9c skip limiting if limit is zero 2021-04-04 18:38:33 -07:00
Chris Lu
bdf2ddddfd revert to same implementation as before 
This reverts commit 7e8edc3c4a.
 2021-04-02 02:21:38 -07:00
Chris Lu
7e8edc3c4a refactoring 2021-04-02 01:10:24 -07:00
Chris Lu
6b7aa9633f 2.36 2021-03-28 19:09:06 -07:00
Chris Lu
4abb511db3 make a local copy of the in memory cached data 2021-03-22 22:33:07 -07:00
Chris Lu
1dd5bc134c 2.35 2021-03-22 00:05:09 -07:00
Chris Lu
9672f9e1b2 2.34 2021-03-16 03:01:15 -07:00
Chris Lu
91a3ac9731 2.33 2021-03-16 00:36:06 -07:00
Chris Lu
4b1ed227d1 revert fasthttp changes 
related to https://github.com/chrislusf/seaweedfs/issues/1907
 2021-03-16 00:33:14 -07:00
Chris Lu
9f00f95bfb 2.32 2021-03-14 21:32:00 -07:00
Chris Lu
f5b5e4da2d 2.31 2021-03-09 12:52:16 -08:00
Chris Lu
d888e9a90f 2.30 2021-03-07 15:01:19 -08:00
Chris Lu
40ff30b83f adjust help message 2021-03-03 02:12:46 -08:00
Chris Lu
e982b06bcd support IPv6 2021-03-02 20:59:39 -08:00
Chris Lu
f29ae0db76 2.29 2021-02-28 18:08:26 -08:00
bingoohuang
5cdff56731 set default env prefix to WEED_ 2021-03-01 09:35:23 +08:00
Chris Lu
3b76a51f5f add help message 2021-02-27 12:15:49 -08:00
Chris Lu
70434df105 use "options" 2021-02-27 12:12:53 -08:00
Chris Lu
d3c31c69a7 avoid confusion: conf and config are too similar 2021-02-27 12:08:09 -08:00
Chris Lu
37f104f88f 2.28 2021-02-22 22:54:34 -08:00
Chris Lu
90cdf9dcac avoid conflict with "weed scaffold -config=xxx" 2021-02-22 16:57:27 -08:00
Chris Lu
72b0d9d8c4 avoid unnecessary user home checking 2021-02-22 16:50:56 -08:00
Chris Lu
151c281f36 2.27 2021-02-21 19:29:27 -08:00
bingoohuang
ee7cdf3668 revert ErrXyz to ErrorXyz temporarily. 2021-02-20 16:58:08 +08:00
bingoohuang
50df484d86 support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc. 2021-02-20 16:52:57 +08:00
bingoohuang
7ffe736d20 renaming to ErrXyz 2021-02-19 15:59:55 +08:00
Chris Lu
71f0c19515 2.26 2021-02-15 13:38:38 -08:00
Chris Lu
8726ef34c1 2.25 2021-02-14 22:23:43 -08:00
Chris Lu
4ce56bac08 fix format error 2021-02-12 06:44:26 -08:00
Chris Lu
7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu
487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
c959aaa19b mount: load directory based on correct filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1799
 2021-02-10 23:14:18 -08:00
Chris Lu
5e4b5109dd 2.24 2021-02-07 22:29:43 -08:00
bingoohuang
7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
Chris Lu
318a3d2efc 2.23 2021-01-31 20:18:06 -08:00
Chris Lu
2c5eac5705 2.22 2021-01-24 14:06:09 -08:00
Chris Lu
52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu
c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu
e2c7e3fe6d better locks 2021-01-13 13:20:33 -08:00
Chris Lu
819a85f59a avoid viper concurrent access 2021-01-13 13:14:52 -08:00
Chris Lu
0ef43a23a7 go fmt 2021-01-12 02:30:12 -08:00
Chris Lu
cfb9342a15 avoid concurrent map updates to viper 2021-01-12 02:28:13 -08:00
Chris Lu
394513f598 filer: ensure seamless meta data updates 2021-01-11 02:08:55 -08:00
Chris Lu
f0d3b3bf93 2.20 2021-01-09 22:55:22 -08:00
Chris Lu
90b117acf1 update ccache version 2021-01-08 02:17:43 -08:00
Chris Lu
2ce86f308e 2.19 2021-01-03 17:24:02 -08:00
Chris Lu
d2ead72918 2.18 2020-12-28 12:37:10 -08:00
Chris Lu
515b7632d7 2.17 2020-12-25 23:58:22 -08:00
Chris Lu
4f31c1bb94 go fmt 2020-12-22 02:34:08 -08:00
Chris Lu
6912bf94ae 2.16 2020-12-20 15:30:12 -08:00
Chris Lu
021358749b better detect IP address 
following advice from https://github.com/chrislusf/seaweedfs/issues/1671
 2020-12-14 00:30:20 -08:00
Chris Lu
813453f3e1 2.15 2020-12-13 03:45:35 -08:00
Chris Lu
06bb7bf6c0 adding /usr/local/etc/seaweedfs for bsd style config search path 
related to https://reviews.freebsd.org/D27391
 2020-12-07 16:46:48 -08:00
Chris Lu
46b91228af 2.14 2020-12-06 18:56:58 -08:00
Chris Lu
04062c56c7 webdav: improve webdav upload speed 2020-12-01 15:32:27 -08:00
Chris Lu
141ce67c09 close http request body 2020-11-30 02:45:00 -08:00
Chris Lu
0e99531dbf 2.13 2020-11-29 17:00:03 -08:00
Chris Lu
0da7ecfd29 go fmt 2020-11-26 11:22:30 -08:00
Chris Lu
a102157020 do not retry if failed with a non transport related error 2020-11-26 01:29:30 -08:00
Chris Lu
d3be2ceedd 2.12 2020-11-22 17:17:33 -08:00
Chris Lu
92f906b6fc remove zstd 
fix https://github.com/chrislusf/seaweedfs/issues/1629
 2020-11-21 13:06:45 -08:00
Chris Lu
98827d68d9 2.11 2020-11-15 20:25:56 -08:00
Chris Lu
2bd6fd3bbe remove unused function 2020-11-15 20:15:07 -08:00
Chris Lu
95c0de285d refactoring 2020-11-15 16:58:48 -08:00
Chris Lu
73f934d5de s3: do not close reader too early 
fix https://github.com/chrislusf/seaweedfs/issues/1609
 2020-11-11 01:49:39 -08:00
Chris Lu
15956d8a8e 2.10 2020-11-10 23:17:13 -08:00
Chris Lu
9b10d1ac33 2.08 2020-11-01 12:45:20 -08:00
Chris Lu
8750cac090 move to util.RetryWaitTime 2020-11-01 02:36:43 -08:00
Chris Lu
7192a378cc add retry to master clients also 2020-11-01 01:40:16 -07:00
Chris Lu
004aa9ca5d only retry transport related errors 2020-11-01 01:21:44 -07:00
Chris Lu
f7b21973ac mount: add retry for all operations with filer 
fix https://github.com/chrislusf/seaweedfs/issues/1589
 2020-11-01 01:11:08 -07:00
Chris Lu
44921220b0 2.07 2020-10-25 22:03:46 -07:00
Chris Lu
27ba0a2be3 2.06 2020-10-25 12:36:47 -07:00
Chris Lu
5b92dfe302 adjust log level 2020-10-23 23:58:54 -07:00
Chris Lu
11716fbf6f make reading error more obvious 2020-10-22 13:33:45 -07:00
Chris Lu
3bf0116de1 mount: less channel waiting 2020-10-21 02:16:21 -07:00
Chris Lu
c31b254248 mount: shortcut when there is only one chunk 2020-10-20 23:48:29 -07:00
Chris Lu
898db14729 2.05 2020-10-17 02:42:33 -07:00
Chris Lu
0542911e29 go fmt 2020-10-13 19:50:46 -07:00
Chris Lu
c127da1219 filer: linearize timeout for large chunk of data 2020-10-13 14:04:46 -07:00
Chris Lu
9b4f7fed14 mount: report filer IO error 
related to https://github.com/chrislusf/seaweedfs/issues/1530
 2020-10-13 11:21:13 -07:00
Chris Lu
3f7d1d1bf1 Only wait on retryable requests 2020-10-13 00:29:46 -07:00
Chris Lu
b18f21cce1 mount: fix bound tree with filer.path 
fix https://github.com/chrislusf/seaweedfs/issues/1528
 2020-10-12 21:58:37 -07:00
Chris Lu
bbd0afd37e 2.04 2020-10-11 21:25:30 -07:00
Chris Lu
4fc673341f 2.03 2020-10-04 21:58:22 -07:00
Chris Lu
707936f482 re-enable caching larger than 16MB 
revert 62ce85610e
 2020-10-03 14:12:38 -07:00
Chris Lu
f1e879fe7a 2.02 2020-10-01 07:15:49 -07:00
Chris Lu
b9887504e8 fix test 2020-09-27 23:19:50 -07:00
Chris Lu
f46eae284e adjust for test 2020-09-27 23:08:11 -07:00
Chris Lu
75aca5e13d 2.01 2020-09-27 23:02:41 -07:00
Chris Lu
c49e2bb9a3 adjust 2020-09-27 12:07:45 -07:00
Chris Lu
62ce85610e skip caching too large chunks 2020-09-27 11:58:48 -07:00
Chris Lu
9ad2dcca2b more tests 2020-09-27 11:42:51 -07:00
Chris Lu
e43d86c796 fix pre allocated volume size 2020-09-27 10:58:19 -07:00
Chris Lu
31fc7bb2e1 refactor 
adjust for faster test
 2020-09-27 10:41:29 -07:00
Chris Lu
a37535cd9f avoid non utf-8 in filename 
fix https://github.com/chrislusf/seaweedfs/issues/1493
 2020-09-26 00:13:39 -07:00
Chris Lu
1012df7bb5 switch hardlink id from int64 to bytes 2020-09-24 11:11:42 -07:00
Chris Lu
5e239afdfc hardlink works now 2020-09-24 03:06:48 -07:00
Chris Lu
62563a895a refactoring 2020-09-20 16:00:01 -07:00
Chris Lu
9cac9c4001 2.00 2020-09-20 00:47:04 -07:00
Chris Lu
f8fea19669 1.99 2020-09-12 13:46:33 -07:00
Chris Lu
ba984a4e29 1.98 2020-09-12 04:13:02 -07:00
Chris Lu
446e476a11 go fmt 2020-09-12 04:08:03 -07:00
Chris Lu
ab201c2798 1.97 2020-09-11 13:47:50 -07:00
Chris Lu
19537c9d21 1.96 2020-09-10 19:52:07 -07:00
Chris Lu
660d7c0edd 1.95 2020-09-10 01:32:05 -07:00
Chris Lu
387ab6796f filer: cross cluster synchronization 2020-09-09 11:21:23 -07:00
Chris Lu
4fc0bd1a81 return http response directly 2020-09-09 03:53:09 -07:00
Chris Lu
c18ea21f7a 1.94 2020-09-07 13:13:53 -07:00
Chris Lu
432fe3d926 1.93 2020-09-06 12:12:09 -07:00
Chris Lu
b8f32bcab9 filer: compress stored metadata 2020-09-03 11:00:20 -07:00
Chris Lu
dd52037c27 1.92 2020-08-30 22:56:35 -07:00
Chris Lu
a41588279a change log level 5 to 4 2020-08-30 20:12:04 -07:00
Chris Lu
b69cb74c03 read meta logs by timestamp 
pass in event ts when moving logs
meta aggregator reads in memory logs only
 2020-08-29 17:37:19 -07:00
Chris Lu
ca658a97c5 add signatures to messages to avoid double processing 2020-08-28 23:48:48 -07:00
Chris Lu
0be6863c87 rename 2020-08-26 22:40:15 -07:00
Chris Lu
5f55a87101 close http response 2020-08-24 00:32:44 -07:00
Chris Lu
81a8aa6581 1.91 2020-08-23 17:28:40 -07:00
Chris Lu
aee27ccbe1 multiple fixes 
* adjust isOpen count
* move ContinuousDirtyPages lock to filehandle
* fix problem with MergeIntoVisibles, avoid reusing slices
* let filer delete the garbage
 2020-08-23 15:48:02 -07:00
Chris Lu
99d05f758c adjust logs 2020-08-18 23:39:18 -07:00
Chris Lu
6a92f0bc7a refactoring to typed Size 
Go is amazing with refactoring!
 2020-08-18 17:04:28 -07:00
Chris Lu
09e126bae5 refactoring: use interface 2020-08-17 20:20:08 -07:00
Chris Lu
be4d42b8e2 rename 2020-08-17 20:15:53 -07:00
Chris Lu
97e54a80d4 rename variables 2020-08-17 16:05:13 -07:00
Chris Lu
ff200398bb 1.90 2020-08-16 21:18:50 -07:00
Chris Lu
b71df82292 1.89 2020-08-16 00:57:28 -07:00
Chris Lu
003d48da21 adjust logs 2020-08-15 19:55:28 -07:00
Chris Lu
e74dc4e4bc add back fs node cache for renaming 2020-08-09 21:56:09 -07:00
Chris Lu
b056707770 1.88 2020-08-08 22:55:12 -07:00
Chris Lu
20e2ac1add filer: store md5 metadata for files uploaded by filer 
fix https://github.com/chrislusf/seaweedfs/issues/1412
 2020-08-06 10:04:17 -07:00
Chris Lu
1b3a80dd3d non-fatal error 2020-08-01 13:46:52 -07:00
Chris Lu
bbbea8159c http request use gzip if possible 2020-08-01 13:20:52 -07:00
Chris Lu
2f03481cb2 in case when content is not compressed 2020-08-01 13:08:10 -07:00
Chris Lu
a6b59d50f7 1.87 2020-07-30 13:52:04 -07:00
Chris Lu
7a7e70b6cc 1.86 2020-07-20 23:39:58 -07:00
Chris Lu
befb396892 mount: resolve home directory 
fix https://github.com/chrislusf/seaweedfs/issues/1391
 2020-07-16 22:38:55 -07:00
Chris Lu
ddec7b2bb9 go fmt 2020-07-14 07:34:16 -07:00
Chris Lu
5a4f258a8c 1.85 2020-07-12 17:34:09 -07:00
Chris Lu
224103e13b aggregated logs has empty flushFn 2020-07-11 09:12:03 -07:00
Chris Lu
b813fac4a3 1.84 2020-06-28 07:28:21 -07:00
Chris Lu
f0f884edd5 1.83 2020-06-27 15:46:59 -07:00
Chris Lu
bef356ce4c since we already know the chunk size, no need to iterate 2020-06-27 12:51:04 -07:00
Chris Lu
a808b3b5df incase the memory data is too small 2020-06-27 11:59:15 -07:00
Chris Lu
3dbd51c3c2 a little bit more efficient 2020-06-26 10:02:37 -07:00
Chris Lu
212b6e7d42 error logging 2020-06-26 10:01:55 -07:00
Chris Lu
48b23f2fdd FUSE mount: prevent concurrent modification 2020-06-26 10:00:48 -07:00
Chris Lu
3cec4b3c49 FUSE mount: fix file id written twice? 
fix https://github.com/chrislusf/seaweedfs/issues/1373
 2020-06-25 20:11:49 -07:00
Chris Lu
5be12eea37 zstd fix 2020-06-24 11:39:12 -07:00
Chris Lu
fe60db404a prepare for zstd 2020-06-23 09:12:02 -07:00
Chris Lu
3080c197e3 rename UnCompressData to DecompressData 2020-06-20 08:16:16 -07:00
Chris Lu
a735fd767c adjust uncompress data 2020-06-20 08:01:00 -07:00
Chris Lu
e912fd15e3 renaming 2020-06-19 22:45:27 -07:00
Chris Lu
c48b407053 1.82 2020-06-19 13:47:27 -07:00
Chris Lu
c0283eee1a more test cases 2020-06-19 09:45:42 -07:00
Chris Lu
55b6efb755 fix checking visited nodes 2020-06-19 09:45:42 -07:00
Chris Lu
546709c1de check has visited by the boundary 2020-06-18 22:28:17 -07:00