mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-11 15:59:06 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,106 Commits 32 Branches 296 Tags 150 MiB
e423548673
Commit Graph

4758 Commits

Author SHA1 Message Date
Radtoo
fa0cfdfb7f Added basic deletion capabilities to fsck. 2022-02-06 17:05:58 +01:00
chrislu
433fde4b18 move error to a separate file 
This file contains metric names for all errors
The naming convention is ErrorSomeThing = "error.some.thing"
 2022-02-04 22:57:51 -08:00
Chris Lu
a23fcb9a7c
Merge pull request #2634 from kmlebedev/errorMetrics 
error metrics for filer and store
 2022-02-04 22:35:13 -08:00
Chris Lu
247bbabda5
Merge pull request #2632 from lapshin-vitaly/s3api_errors 
add s3api error for copy in file, not directory
 2022-02-04 22:34:34 -08:00
chrislu
ced3b89395 add util package 2022-02-04 21:34:58 -08:00
chrislu
affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu
76e297d64f sync call to write file, avoid vif loading error 
fix https://github.com/chrislusf/seaweedfs/issues/2633
 2022-02-04 11:14:04 -08:00
zerospiel
f3364fec99 weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting 
Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject).

Added generic comment rule of traversing methods
 2022-02-04 15:14:48 +03:00
Konstantin Lebedev
9978f54acf fix metric names 2022-02-04 16:45:16 +05:00
Konstantin Lebedev
3f4e17aa24 error metrics for filer and store 2022-02-04 14:07:14 +05:00
Lapshinn Vitaly
6bdc274d4d add s3api error for copy in file, not directory 2022-02-04 03:28:37 +03:00
chrislu
6cf2e7d493 s3: listObjectParts return ErrNoSuchUpload if does not exist 
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc
{
    "Initiator": null,
    "Owner": null,
    "StorageClass": "STANDARD"
}

If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.
 2022-02-03 12:34:16 -08:00
zerospiel
b54a65ba5a weed/s3api: added new bucket handlers for more compatibility with AWS S3 
Protocol

Otherwise any requests to the underlying handlers results in calls to
ListObjects (v1) that may intensively load gateway and volume servers.

Added the following handlers with default responses:
- GetBucketLocation
- GetBucketRequestPayment

Added the following handlers with NotFound and NotImplemented responses:
- PutBucketAcl
- GetBucketPolicy
- PutBucketPolicy
- DeleteBucketPolicy
- GetBucketCors
- PutBucketCors
- DeleteBucketCors
 2022-02-03 17:17:05 +03:00
chrislu
6bee1e9714 [volume.check.disk] fix wrong logic to compare 
fix https://github.com/chrislusf/seaweedfs/issues/2626

minuend - subtrahend
 2022-02-01 07:48:28 -08:00
chrislu
7270067289 2.88 2022-01-30 20:25:26 -08:00
chrislu
84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
chrislu
b8490fe427 adjust volume count even when not applying the changes 2022-01-28 19:11:46 -08:00
Chris Lu
b3f0f170b4
Merge pull request #2618 from divanikus/master 2022-01-28 03:34:52 -08:00
chrislu
b9b684194f remove max connection age 
following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779
 2022-01-27 02:28:22 -08:00
divanikus
67e3fe996a async volumeTierMove 2022-01-26 18:22:31 +03:00
chrislu
62d815d1ca use limited in memory buffer instead of swap file 2022-01-26 00:36:19 -08:00
Chris Lu
9596fce562
Merge pull request #2613 from kmlebedev/masterPromMetrics 
Master prom metrics
 2022-01-25 08:13:14 -08:00
Konstantin Lebedev
5c9259fa3c fix metrics master name 2022-01-25 14:42:47 +05:00
chrislu
7328962009 revert "POSIX: should not delete if a directory is not empty" 
revert 0c75f15062

Reported:

Hi, about commit: 0c75f15062
POSIX: should not delete if a directory is not empty
It should still delete with a command like rm -rf "${path}"/  because it is a forced delete, but now it gets fail to delete non-empty folder: [...]
Can you enable the delete if it is forced?
 2022-01-24 14:21:50 -08:00
Konstantin Lebedev
c9952759c4 metrics master is leader 2022-01-24 20:13:07 +05:00
Konstantin Lebedev
28efe31524 new master metrics 2022-01-24 19:09:43 +05:00
chrislu
3bba2124ef use a sliding window of in-memory writable chunks 2022-01-23 23:02:05 -08:00
chrislu
520591e6ea reset swap file chunk after uploading 2022-01-23 18:30:53 -08:00
chrislu
e185d90d24 2.87 2022-01-23 16:18:55 -08:00
chrislu
4a311c7f5e dedup local metadata subscribers 
fix https://github.com/chrislusf/seaweedfs/discussions/2542
 2022-01-23 16:14:22 -08:00
chrislu
bb6854b972 adjust log 2022-01-22 12:35:09 -08:00
chrislu
2bdd737971 rename file 2022-01-22 08:28:35 -08:00
chrislu
643bbbeb49 rename 2022-01-22 08:27:40 -08:00
chrislu
18ed06b420 use memory when under 16 chunks 2022-01-22 08:11:01 -08:00
chrislu
8e80f3cd65 move upload pipeline locking to a different file 2022-01-22 08:09:55 -08:00
chrislu
dde34fa99d rename 2022-01-22 08:08:01 -08:00
chrislu
a2aa542370 rename 2022-01-22 08:07:11 -08:00
chrislu
c376ccc5a5 swap file based random write large file upload 2022-01-22 08:06:53 -08:00
chrislu
b9ae16fbc5 fix memory allocation 2022-01-22 08:05:04 -08:00
chrislu
1ee828b768 refactor 
do not expose internal offset
 2022-01-22 06:34:29 -08:00
chrislu
02d0c12cdd rename 2022-01-22 06:00:10 -08:00
chrislu
5dea5c0449 refactor 2022-01-22 05:59:07 -08:00
chrislu
8aa6bf0bb9 refactoring 2022-01-22 05:40:10 -08:00
chrislu
d97bd54e63 just refactoring 2022-01-22 04:18:54 -08:00
chrislu
482014f9da rename file 2022-01-22 03:50:18 -08:00
chrislu
3b4a9addaf rename 2022-01-22 01:46:10 -08:00
chrislu
4acfc098e9 re-order 2022-01-22 01:43:14 -08:00
chrislu
e71dcfb3a6 add logging for memory allocation 2022-01-22 01:35:12 -08:00
chrislu
9d0f58c329 skip printing fs configuration 2022-01-21 13:29:47 -08:00
chrislu
ce2049cdb6 refactoring, move genFn before saveFn 2022-01-21 12:08:58 -08:00
chrislu
e47f63d159 enforce bucket quota 2022-01-21 02:34:42 -08:00
chrislu
6e57d8d0de s3: check bucket usage and adjust read only according to quota 2022-01-21 02:15:27 -08:00
chrislu
606667f205 able to configure the quota for a bucket 2022-01-21 01:42:20 -08:00
chrislu
b1063162b6 display bucket quota 2022-01-21 00:55:04 -08:00
chrislu
f103491912 s3: list bucket size from weed shell 2022-01-21 00:26:49 -08:00
chrislu
6c7135d77e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-21 00:17:06 -08:00
chrislu
ea57654e34 refactoring 2022-01-21 00:16:50 -08:00
Chris Lu
7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics 
master metricsHttpPort
 2022-01-20 09:26:25 -08:00
chrislu
b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev
77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
chrislu
77362700e1 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix #2593
 2022-01-18 12:04:40 -08:00
chrislu
05c3c3f56b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-17 23:38:13 -08:00
chrislu
9b77f0054e 2.86 2022-01-17 23:38:03 -08:00
Chris Lu
ec254d8a89
Merge pull request #2597 from guol-fnst/gocql_to 
add gocql timeout setting
 2022-01-17 23:35:13 -08:00
chrislu
9274557552 keep dirty pages based on temp file 2022-01-17 23:23:49 -08:00
guol-fnst
da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu
c87b8f4c30 S3: fail fast when "X-Amz-Copy-Source" is a folder 
fix https://github.com/chrislusf/seaweedfs/issues/2593
 2022-01-17 23:09:37 -08:00
chrislu
b2acfd75e9 ensure entry view cache is invalidated 2022-01-17 23:02:30 -08:00
chrislu
f4ad63528a wait for reading threads to complete before dropping sealed chunks 2022-01-17 22:24:44 -08:00
chrislu
0a3f95ca01 more logs 2022-01-17 20:41:00 -08:00
chrislu
b068bc291d testing with always resetting entry view cache 2022-01-17 20:07:01 -08:00
chrislu
047446d5ca remove extra async execution 2022-01-17 15:50:11 -08:00
chrislu
7bf7af971b more logs 2022-01-17 14:15:10 -08:00
chrislu
fc22071a2f more logs 2022-01-17 14:02:37 -08:00
chrislu
381f4e73a0 delete actual reference first 2022-01-17 13:56:47 -08:00
chrislu
0ba88596e8 invalidate filehandle entry view cache 2022-01-17 13:53:30 -08:00
chrislu
1734017ba1 add test 2022-01-17 13:40:41 -08:00
chrislu
da7f13e73e Revert "testing skip memory management" 
This reverts commit 6c908352cb.
 2022-01-17 03:21:31 -08:00
chrislu
6c908352cb testing skip memory management 2022-01-17 03:19:24 -08:00
chrislu
77d9993f38 remove unused variables 2022-01-17 03:19:11 -08:00
chrislu
f710d5ffca a little speed up 2022-01-17 03:19:00 -08:00
chrislu
fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu
1bd6d289d4 better locking on file handle 2022-01-15 05:45:29 -08:00
chrislu
2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu
b17c426e99 weed server: optionally start IAM service 
related to https://github.com/chrislusf/seaweedfs/issues/2560
 2022-01-13 22:49:49 -08:00
chrislu
3c8b74318e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-13 13:03:07 -08:00
chrislu
8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
banjiaojuhao
45e9c83421 padding zero for sparse file 2022-01-13 22:21:22 +08:00
chrislu
fe5b9e39cc POSIX: check permission when removing items 2022-01-13 02:07:39 -08:00
chrislu
1453263b63 remove dead code 2022-01-13 02:02:04 -08:00
chrislu
e69c374956 minor 2022-01-13 02:01:53 -08:00
chrislu
9b954dc0d4 adjust make file 2022-01-13 01:33:13 -08:00
chrislu
f2847f1266 POSIX: check deletion permission 2022-01-12 23:58:11 -08:00
chrislu
0c75f15062 POSIX: should not delete if a directory is not empty 2022-01-12 23:57:54 -08:00
chrislu
de27058d0b POSIX: differentiate device and char device 2022-01-12 21:45:38 -08:00
chrislu
d400a11832 POSIX: adjust source file ctime 
SeaweedFS uses mtime as ctime
 2022-01-12 21:45:18 -08:00
chrislu
b44f05a2d0 POSIX: change timestamp on each attribute change 2022-01-12 19:31:25 -08:00
chrislu
15c01d8b7f add some notes 2022-01-12 15:04:48 -08:00
chrislu
107a4884a8 shell: tighter memory allocation 2022-01-12 14:59:29 -08:00
chrislu
fec8428fd8 POSIX: different inode for same named different file types 2022-01-12 11:51:13 -08:00
chrislu
e82ad60122 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-12 11:07:49 -08:00
chrislu
caf0a3486b POSIX: adjust ctime for file truncate 2022-01-12 11:07:39 -08:00
Konstantin Lebedev
edb753ab4d https://github.com/chrislusf/seaweedfs/issues/2583 2022-01-12 16:04:59 +05:00
chrislu
adfd54e7c4 fix compilation 2022-01-12 01:24:24 -08:00
chrislu
6cc92817dc add logs for request mode 2022-01-12 01:13:19 -08:00
chrislu
826a7b307e master: remove hard coded filer settings in master.toml 
fix https://github.com/chrislusf/seaweedfs/issues/2529
 2022-01-12 01:11:25 -08:00
chrislu
cd1ad88f30 POSIX: check name is too long ENAMETOOLONG 2022-01-12 00:16:00 -08:00
chrislu
2dcb8cb93b POSIX: ensure file and directory inodes are different 
this is just an in memory representation.

POSIX wants different inode numbers for the same named file or directory.
 2022-01-11 23:44:48 -08:00
chrislu
5bb37d5905 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 23:29:12 -08:00
chrislu
10ecf80ca1 add a debug capability to list all metadata keys 2022-01-11 23:25:04 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails 
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
 2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu
41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
chrislu
2d0ccc4d34 add logs 2022-01-11 12:23:01 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2 
Fix paths2
 2022-01-11 12:04:30 -08:00
chrislu
b8fbf19e9a mount: rename follow POSIX 2022-01-11 03:23:03 -08:00
chrislu
6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
chrislu
cbc055dc2b mount: file fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2561
 2022-01-10 00:52:16 -08:00
chrislu
19555385f7 2.85 2022-01-09 19:30:23 -08:00
Radtoo
389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a Now works with a single file too 
Parsing removed from doFixOneVolume

Needle init removed from runFix
 2022-01-08 16:31:53 +01:00
chrislu
110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
chrislu
60dc450091 skip fixing read only volumes 
fix https://github.com/chrislusf/seaweedfs/issues/2562
 2022-01-06 09:52:28 -08:00
chrislu
3df8f96117 avoid changing inode 2022-01-06 01:36:11 -08:00
chrislu
67b0645808 mount: need to change entry name after renaming 2022-01-05 21:27:41 -08:00
chrislu
4de060daa6 mount: skip special character in the filenames 
fix https://github.com/chrislusf/seaweedfs/issues/2559
 2022-01-05 03:57:24 -08:00
chrislu
e76105e2ab fix auth permission checking 2022-01-03 21:05:20 -08:00
chrislu
a7887166cf wildcard prefix to restrict access to directories in s3 bucket 
https://github.com/chrislusf/seaweedfs/discussions/2551
 2022-01-03 15:39:36 -08:00
chrislu
5799a20f71 2.84 2022-01-02 17:05:19 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158 
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
 2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d BUGFIX: ensure Authorization header is only added once 2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst
10404c4275 FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client 
- one JWT for reading and one for writing, analogous to how the JWT
  between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer

Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.

## Docs to be adjusted after a release

Page `Amazon-S3-API`:

```
# Authentication with Filer

You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.

Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.

With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```

Page `Security Overview`:

```
The following items are not covered, yet:

- master server http REST services

Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.

...

Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**

...

# Securing Filer HTTP with JWT

To enable JWT-based access control for the Filer,

1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.

If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.

If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.

The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```

Page `Security Configuration`:

```
(update scaffold file)

...

[filer_jwt.signing]
key = "blahblahblahblah"

[filer_jwt.signing.read]
key = "blahblahblahblah"
```

Resolves: #158
 2021-12-30 14:45:27 +01:00
chrislu
34742be029 remove duplicated metadata subscription in filer 
https://github.com/chrislusf/seaweedfs/issues/2545
 2021-12-30 01:51:52 -08:00
chrislu
5c87fcc6d2 add client id for all metadata listening clients 2021-12-30 00:23:57 -08:00
chrislu
fb434318e3 dynamically adjust connection timeout 
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:44:39 -08:00
chrislu
5788bf2270 s3: increase timeout limit 
https://github.com/chrislusf/seaweedfs/issues/2541
 2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst
fcc09cef6f Refactor: pass in claim type into security.DecodeJwt 2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef rename security.GenJwt to security.GenJwtForVolumeServer 2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst
eda4c43a08 fix typo in error message 2021-12-29 12:38:14 +01:00
chrislu
498661e3bb mount: remove limits on number of parallel requests 2021-12-28 17:41:01 -08:00
chrislu
9a00c17555 reader: avoid wrong pattern detection due to lock waiting 2021-12-28 16:30:33 -08:00
chrislu
f7a6f6b4c0 if this is enabled, there are some "bus error" with git clone 2021-12-28 13:44:52 -08:00
chrislu
0da2dfd640 fuse: change to direct io mode 
before and after:

chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192  0.01s user 3.22s system 83% cpu 3.857 total
 2021-12-28 12:22:56 -08:00
chrislu
3fd4da34a4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-28 01:10:38 -08:00
chrislu
2422556456 monitor write pattern: avoid timing due to locking 2021-12-28 01:10:35 -08:00
chenkai
47c30e3add filer list entries use context to break job 2021-12-28 15:03:41 +08:00
chrislu
80db8b13d8 bug: cleanup function was called twice 2021-12-27 20:53:02 -08:00
chrislu
67b723f74e Filer Server API support fsync 
fix https://github.com/chrislusf/seaweedfs/issues/2528
 2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls 
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
 2021-12-26 00:15:03 -08:00
chrislu
c935b9669e 2.83 2021-12-25 01:01:34 -08:00
chrislu
eb4ad2546f use proper chunk size limit option 2021-12-24 22:52:18 -08:00
chrislu
41bbf320bb use 2MB chunk size. cache size is the wrong option 2021-12-24 22:50:19 -08:00
chrislu
982ea85d81 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-24 22:40:07 -08:00
chrislu
083d8e9ece add stream writer 
this should improve streaming write performance, which is common in many cases, e.g., copying large files.

This is additional to improved random read write operations: 3e69d19380...19084d8791
 2021-12-24 22:38:22 -08:00
Sebastian Kurfürst
6db49100d6 BUGFIX: add access.ui setting to scaffolded security.toml 
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
 2021-12-24 13:59:04 +01:00
chrislu
255a1c7dcd refactor type names 2021-12-23 18:23:18 -08:00
chrislu
f77ca41769 refactor 2021-12-23 17:48:34 -08:00
chrislu
1d36884845 rename files 2021-12-23 17:47:58 -08:00
chrislu
2d1a1f5e03 rename variables and functions 2021-12-23 17:35:57 -08:00
chrislu
7bf48ee135 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-23 17:23:26 -08:00
chrislu
6de331b014 clean up 2021-12-23 17:23:21 -08:00
chrislu
032df784ed chunked file works now 2021-12-23 17:17:32 -08:00
banjiaojuhao
083bf3a137 filer server: add "datacenter, rack and datanode" for path specific configuration 2021-12-23 23:25:05 +08:00
chrislu
c2aad1c7ff detect non streaming mode on first write request 2021-12-22 17:20:44 -08:00
chrislu
b541e39a2c fix tests 2021-12-22 16:17:30 -08:00
chrislu
2bc6fa90ff Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-22 16:05:43 -08:00
chrislu
0ec7bc6710 detect non streaming mode on the first read 2021-12-22 16:05:38 -08:00
chrislu
4c1368d621 fix test 2021-12-22 16:05:08 -08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node 
filer server: allow upload file to specific dataNode
 2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e filer server: allow upload file to specific dataNode 2021-12-22 21:57:26 +08:00
chrislu
7b78fc72b0 add page chunk interval list 2021-12-22 02:53:33 -08:00
chrislu
b7cd52636b Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2021-12-21 22:24:43 -08:00
chrislu
3981d65b68 remove println 2021-12-21 22:24:38 -08:00
chrislu
9a73319b45 mount: different write strategy for streaming write and random write 2021-12-21 17:28:55 -08:00
Chris Lu
b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode 2021-12-21 08:56:51 -08:00
banjiaojuhao
dda6b90d25 assign fileId according to DataNode with empty DataCenter and Rack 2021-12-21 17:28:33 +08:00
chrislu
4b8dcff448 reverting default admin scripts 
fix https://github.com/chrislusf/seaweedfs/issues/2525

this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
 2021-12-20 13:34:57 -08:00
chrislu
b21a67bbe6 add writer pattern object for later use 2021-12-20 11:53:48 -08:00
chrislu
4fd29dad86 remove writeOnly flag 2021-12-20 01:11:43 -08:00
chrislu
bc96682760 refactor, change file locations 2021-12-20 01:02:23 -08:00
chrislu
866c2657f0 avoid FUSE cache only for the first 512 bytes 2021-12-19 23:13:36 -08:00
chrislu
0cb9036f66 mount: only cache the first chunk on stream read 2021-12-19 23:06:03 -08:00
chrislu
a152f17937 mount: improve read performance on random reads 2021-12-19 22:43:14 -08:00
chrislu
85c526c583 s3: bind to a specific IP 
fix https://github.com/chrislusf/seaweedfs/issues/2516
 2021-12-17 11:34:37 -08:00
chrislu
5eacff9d4f log message adds server name 
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
 2021-12-16 10:46:26 -08:00
chrislu
50ddd8c8e2 remove debug messages 
fix https://github.com/chrislusf/seaweedfs/issues/2514
 2021-12-16 00:58:15 -08:00
chrislu
7210558c7b s3: pass through s3 presigned headers 
fix https://github.com/chrislusf/seaweedfs/discussions/2502
 2021-12-15 13:18:53 -08:00
chrislu
bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files 
fix https://github.com/chrislusf/seaweedfs/issues/2509
 2021-12-14 19:48:31 -08:00
chrislu
316f326464 add more help message 2021-12-13 13:14:36 -08:00
Konstantin Lebedev
969f513265 disable audit log for s3 statusHandler 2021-12-13 16:08:19 +05:00
Chris Lu
ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking 
Audit log force async
 2021-12-13 00:47:16 -08:00
Konstantin Lebedev
34779e8f38 force enable asynchronous I/O sending events to Fluentd 2021-12-13 13:39:39 +05:00
chrislu
c3b73ec23b 2.82 2021-12-12 23:25:24 -08:00
chrislu
7bf14452f7 s3: handle Response-Content-Disposition 2021-12-12 23:05:40 -08:00
chrislu
94e5c0a454 skip purging from ec volumes 2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b add lock messages 2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d rename file 2021-12-10 13:11:00 -08:00
kmlebedev
4f98553ba9 audit log SignatureVersion 2021-12-10 19:40:32 +05:00