chrislu
d400a11832
POSIX: adjust source file ctime
...
SeaweedFS uses mtime as ctime
2022-01-12 21:45:18 -08:00
chrislu
b44f05a2d0
POSIX: change timestamp on each attribute change
2022-01-12 19:31:25 -08:00
chrislu
15c01d8b7f
add some notes
2022-01-12 15:04:48 -08:00
chrislu
107a4884a8
shell: tighter memory allocation
2022-01-12 14:59:29 -08:00
chrislu
fec8428fd8
POSIX: different inode for same named different file types
2022-01-12 11:51:13 -08:00
chrislu
e82ad60122
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-12 11:07:49 -08:00
chrislu
caf0a3486b
POSIX: adjust ctime for file truncate
2022-01-12 11:07:39 -08:00
Konstantin Lebedev
edb753ab4d
https://github.com/chrislusf/seaweedfs/issues/2583
2022-01-12 16:04:59 +05:00
chrislu
adfd54e7c4
fix compilation
2022-01-12 01:24:24 -08:00
chrislu
6cc92817dc
add logs for request mode
2022-01-12 01:13:19 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
chrislu
cd1ad88f30
POSIX: check name is too long ENAMETOOLONG
2022-01-12 00:16:00 -08:00
chrislu
2dcb8cb93b
POSIX: ensure file and directory inodes are different
...
this is just an in memory representation.
POSIX wants different inode numbers for the same named file or directory.
2022-01-11 23:44:48 -08:00
chrislu
5bb37d5905
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-11 23:29:12 -08:00
chrislu
10ecf80ca1
add a debug capability to list all metadata keys
2022-01-11 23:25:04 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-11 12:24:56 -08:00
chrislu
41daecfdca
Update mount_std.go
2022-01-11 12:23:12 -08:00
chrislu
2d0ccc4d34
add logs
2022-01-11 12:23:01 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
...
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu
b8fbf19e9a
mount: rename follow POSIX
2022-01-11 03:23:03 -08:00
chrislu
6a12520a96
fix logging
2022-01-10 01:00:11 -08:00
chrislu
cbc055dc2b
mount: file fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2561
2022-01-10 00:52:16 -08:00
chrislu
19555385f7
2.85
2022-01-09 19:30:23 -08:00
Radtoo
389002f195
Using positional arguments rather than option flag to enable better shell usage
2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a
Now works with a single file too
...
Parsing removed from doFixOneVolume
Needle init removed from runFix
2022-01-08 16:31:53 +01:00
chrislu
110d5a5233
support fixing a collection of volumes, or volumes under one directory
2022-01-07 14:52:16 -08:00
chrislu
60dc450091
skip fixing read only volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2562
2022-01-06 09:52:28 -08:00
chrislu
3df8f96117
avoid changing inode
2022-01-06 01:36:11 -08:00
chrislu
67b0645808
mount: need to change entry name after renaming
2022-01-05 21:27:41 -08:00
chrislu
4de060daa6
mount: skip special character in the filenames
...
fix https://github.com/chrislusf/seaweedfs/issues/2559
2022-01-05 03:57:24 -08:00
chrislu
e76105e2ab
fix auth permission checking
2022-01-03 21:05:20 -08:00
chrislu
a7887166cf
wildcard prefix to restrict access to directories in s3 bucket
...
https://github.com/chrislusf/seaweedfs/discussions/2551
2022-01-03 15:39:36 -08:00
chrislu
5799a20f71
2.84
2022-01-02 17:05:19 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
c35660175d
BUGFIX: ensure Authorization header is only added once
2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst
1cd3b6b4e1
BUGFIX: security.toml contained wrong keys
2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
34742be029
remove duplicated metadata subscription in filer
...
https://github.com/chrislusf/seaweedfs/issues/2545
2021-12-30 01:51:52 -08:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
chrislu
fb434318e3
dynamically adjust connection timeout
...
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu
5788bf2270
s3: increase timeout limit
...
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst
fcc09cef6f
Refactor: pass in claim type into security.DecodeJwt
2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst
d156d410ef
rename security.GenJwt to security.GenJwtForVolumeServer
2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst
eda4c43a08
fix typo in error message
2021-12-29 12:38:14 +01:00
chrislu
498661e3bb
mount: remove limits on number of parallel requests
2021-12-28 17:41:01 -08:00
chrislu
9a00c17555
reader: avoid wrong pattern detection due to lock waiting
2021-12-28 16:30:33 -08:00
chrislu
f7a6f6b4c0
if this is enabled, there are some "bus error" with git clone
2021-12-28 13:44:52 -08:00
chrislu
0da2dfd640
fuse: change to direct io mode
...
before and after:
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu
3fd4da34a4
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-28 01:10:38 -08:00
chrislu
2422556456
monitor write pattern: avoid timing due to locking
2021-12-28 01:10:35 -08:00
chenkai
47c30e3add
filer list entries use context to break job
2021-12-28 15:03:41 +08:00
chrislu
80db8b13d8
bug: cleanup function was called twice
2021-12-27 20:53:02 -08:00
chrislu
67b723f74e
Filer Server API support fsync
...
fix https://github.com/chrislusf/seaweedfs/issues/2528
2021-12-26 17:28:47 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
chrislu
c935b9669e
2.83
2021-12-25 01:01:34 -08:00
chrislu
eb4ad2546f
use proper chunk size limit option
2021-12-24 22:52:18 -08:00
chrislu
41bbf320bb
use 2MB chunk size. cache size is the wrong option
2021-12-24 22:50:19 -08:00
chrislu
982ea85d81
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-24 22:40:07 -08:00
chrislu
083d8e9ece
add stream writer
...
this should improve streaming write performance, which is common in many cases, e.g., copying large files.
This is additional to improved random read write operations: 3e69d19380...19084d8791
2021-12-24 22:38:22 -08:00
Sebastian Kurfürst
6db49100d6
BUGFIX: add access.ui setting to scaffolded security.toml
...
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu
255a1c7dcd
refactor type names
2021-12-23 18:23:18 -08:00
chrislu
f77ca41769
refactor
2021-12-23 17:48:34 -08:00
chrislu
1d36884845
rename files
2021-12-23 17:47:58 -08:00
chrislu
2d1a1f5e03
rename variables and functions
2021-12-23 17:35:57 -08:00
chrislu
7bf48ee135
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-23 17:23:26 -08:00
chrislu
6de331b014
clean up
2021-12-23 17:23:21 -08:00
chrislu
032df784ed
chunked file works now
2021-12-23 17:17:32 -08:00
banjiaojuhao
083bf3a137
filer server: add "datacenter, rack and datanode" for path specific configuration
2021-12-23 23:25:05 +08:00
chrislu
c2aad1c7ff
detect non streaming mode on first write request
2021-12-22 17:20:44 -08:00
chrislu
b541e39a2c
fix tests
2021-12-22 16:17:30 -08:00
chrislu
2bc6fa90ff
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-22 16:05:43 -08:00
chrislu
0ec7bc6710
detect non streaming mode on the first read
2021-12-22 16:05:38 -08:00
chrislu
4c1368d621
fix test
2021-12-22 16:05:08 -08:00
Chris Lu
4e73705533
Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node
...
filer server: allow upload file to specific dataNode
2021-12-22 12:49:15 -08:00
banjiaojuhao
08336be92e
filer server: allow upload file to specific dataNode
2021-12-22 21:57:26 +08:00
chrislu
7b78fc72b0
add page chunk interval list
2021-12-22 02:53:33 -08:00
chrislu
b7cd52636b
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2021-12-21 22:24:43 -08:00
chrislu
3981d65b68
remove println
2021-12-21 22:24:38 -08:00
chrislu
9a73319b45
mount: different write strategy for streaming write and random write
2021-12-21 17:28:55 -08:00
Chris Lu
b0665a15f4
Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode
2021-12-21 08:56:51 -08:00
banjiaojuhao
dda6b90d25
assign fileId according to DataNode with empty DataCenter and Rack
2021-12-21 17:28:33 +08:00
chrislu
4b8dcff448
reverting default admin scripts
...
fix https://github.com/chrislusf/seaweedfs/issues/2525
this new default value was introduced in 2.80
this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.
2021-12-20 13:34:57 -08:00
chrislu
b21a67bbe6
add writer pattern object for later use
2021-12-20 11:53:48 -08:00
chrislu
4fd29dad86
remove writeOnly flag
2021-12-20 01:11:43 -08:00
chrislu
bc96682760
refactor, change file locations
2021-12-20 01:02:23 -08:00
chrislu
866c2657f0
avoid FUSE cache only for the first 512 bytes
2021-12-19 23:13:36 -08:00
chrislu
0cb9036f66
mount: only cache the first chunk on stream read
2021-12-19 23:06:03 -08:00
chrislu
a152f17937
mount: improve read performance on random reads
2021-12-19 22:43:14 -08:00
chrislu
85c526c583
s3: bind to a specific IP
...
fix https://github.com/chrislusf/seaweedfs/issues/2516
2021-12-17 11:34:37 -08:00
chrislu
5eacff9d4f
log message adds server name
...
address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733
2021-12-16 10:46:26 -08:00
chrislu
50ddd8c8e2
remove debug messages
...
fix https://github.com/chrislusf/seaweedfs/issues/2514
2021-12-16 00:58:15 -08:00
chrislu
7210558c7b
s3: pass through s3 presigned headers
...
fix https://github.com/chrislusf/seaweedfs/discussions/2502
2021-12-15 13:18:53 -08:00
chrislu
bf4d7affc0
gateway to remote object store: skip replicating multipart upload part files
...
fix https://github.com/chrislusf/seaweedfs/issues/2509
2021-12-14 19:48:31 -08:00
chrislu
316f326464
add more help message
2021-12-13 13:14:36 -08:00
Konstantin Lebedev
969f513265
disable audit log for s3 statusHandler
2021-12-13 16:08:19 +05:00
Chris Lu
ab0c6a84bf
Merge pull request #2503 from kmlebedev/audit_log_nonblocking
...
Audit log force async
2021-12-13 00:47:16 -08:00
Konstantin Lebedev
34779e8f38
force enable asynchronous I/O sending events to Fluentd
2021-12-13 13:39:39 +05:00
chrislu
c3b73ec23b
2.82
2021-12-12 23:25:24 -08:00
chrislu
7bf14452f7
s3: handle Response-Content-Disposition
2021-12-12 23:05:40 -08:00
chrislu
94e5c0a454
skip purging from ec volumes
2021-12-12 22:55:27 -08:00
chrislu
a2d3f89c7b
add lock messages
2021-12-10 13:24:38 -08:00
chrislu
991a3dca0d
rename file
2021-12-10 13:11:00 -08:00
kmlebedev
4f98553ba9
audit log SignatureVersion
2021-12-10 19:40:32 +05:00
chrislu
2d6fcdf83a
add delay if need to resume from disk
2021-12-09 13:13:05 -08:00
liubaojiang
c7abc34dbb
rename operation returns the correct old parent path
2021-12-10 00:02:57 +08:00
Konstantin Lebedev
98251fe16a
non blocking audit log
2021-12-09 19:47:16 +05:00
chrislu
40ee9b4d7b
sleep between the gap of each execution
...
related to https://github.com/chrislusf/seaweedfs/issues/2476
2021-12-07 11:44:08 -08:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log
2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81
audit log config
2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20
audit log
2021-12-07 12:15:48 +05:00
Konstantin Lebedev
3ac48cd540
audit log
2021-12-07 12:13:21 +05:00
chrislu
7f0a97c7b6
shell: volume.list supports different verbosity level
2021-12-05 21:54:40 -08:00
chrislu
5ea9715721
2.81
...
also sync java client version to SeaweedFS version
2021-12-05 18:05:24 -08:00
chrislu
e6c026db65
volume.fix.replication: fix misplaced volumes
...
fix https://github.com/chrislusf/seaweedfs/issues/2416
2021-12-05 16:56:25 -08:00
chrislu
53e2dee177
[volume.check.disk] was using the wrong source and target locations
...
fix https://github.com/chrislusf/seaweedfs/issues/2268
2021-12-05 14:32:04 -08:00
chrislu
c146c76d10
avoid creating the same bucket with a different randomized name
...
related to https://github.com/chrislusf/seaweedfs/issues/2492
2021-12-05 13:06:41 -08:00
chrislu
42d97a3442
adjust randomized bucket name
2021-12-05 12:36:58 -08:00
chrislu
488afa5002
volume: load volume can optionally be skipped, if ec volume exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2489
2021-12-05 02:28:52 -08:00
chrislu
59e58c4b23
volume: fix loading old volume format
...
fix https://github.com/chrislusf/seaweedfs/issues/2487
2021-12-05 01:06:01 -08:00
chrislu
b70cb3e0b2
upgrade protoc to 3.17.3
...
$brew install protobuf
$ protoc --version
libprotoc 3.17.3
$ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26
$ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1
2021-12-05 00:42:25 -08:00
chrislu
5c3b783310
s3: copy object API needs to escape special characters
...
fix https://github.com/chrislusf/seaweedfs/issues/2482
2021-12-04 23:24:53 -08:00
Bl1tz23
83bffca7e6
fix MongodbStore.ListDirectoryEntries panic on find failure
2021-12-03 13:58:37 +03:00
Tanmoy Majumdar
ea09fb477a
return ' shouldRetry=true' so that filer can retry the failed chunk
2021-12-03 11:54:20 +06:00
Chris Lu
689f5513a9
redis3 supports sentinel
2021-11-29 01:09:51 -08:00
Chris Lu
7227cfddf5
2.80
2021-11-29 00:57:08 -08:00
Chris Lu
3d7390302d
add s3.clean.uploads -timeAgo=24h
2021-11-29 00:49:49 -08:00
Chris Lu
2f72c24498
skip the rest logic
2021-11-29 00:32:21 -08:00
Chris Lu
88ff8fc27b
ensure uploaded chunks are deleted on error
2021-11-29 00:28:26 -08:00
Chris Lu
bea26549f2
avoid empty extended attribute value
2021-11-28 23:50:52 -08:00
Chris Lu
ce2af0082e
revert
2021-11-28 23:35:22 -08:00
Chris Lu
1c9f3c7ac0
read deleted chunks when replcating data
2021-11-28 23:34:34 -08:00
Chris Lu
96c66ca2aa
read deleted chunks when replicating data
2021-11-28 23:33:03 -08:00
Chris Lu
f3a334965d
Update command_remote_uncache.go
...
minor
2021-11-28 23:10:32 -08:00
Chris Lu
9ccfc1cfee
Update command_volume_fsck.go
...
minor
2021-11-28 22:39:24 -08:00
Chris Lu
71a94267c9
Update command_volume_fsck.go
...
remove unused code
2021-11-28 22:37:11 -08:00
Chris Lu
ad16221a35
adjust error log
2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d
add logs for writing to remote file
2021-11-27 22:09:23 -08:00
Chris Lu
3a19eea97c
allocate memory by slabs
2021-11-27 12:13:00 -08:00
limd
8805c04128
fix redis2 sentinel config example
2021-11-25 19:20:02 +08:00
limd
ec03f22cc3
Merge remote-tracking branch 'origin/master'
2021-11-25 16:07:14 +08:00
limd
220797bd71
support redis sentinel
2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0
add retries when writing to remote s3
...
fix https://github.com/chrislusf/seaweedfs/issues/2465
2021-11-22 21:48:04 -08:00
Chris Lu
f3c789d662
2.79
2021-11-21 18:40:24 -08:00
Konstantin Lebedev
40abae7caa
Avoid forbiddening for HeadBucketHandler if owners are specified
...
https://github.com/chrislusf/seaweedfs/issues/2434
2021-11-17 14:42:24 +05:00
Chris Lu
3d87aa767d
fix same dc and other dc
2021-11-16 09:14:01 -08:00
Chris Lu
a0ef6e3611
prevent nil response
...
fix https://github.com/chrislusf/seaweedfs/issues/2452
2021-11-15 08:46:01 -08:00
Chris Lu
100c654ec3
2.78
2021-11-14 23:29:59 -08:00
Chris Lu
7bf891c00a
randomize same-dc servers and other-dc servers
2021-11-12 11:30:11 -08:00
Chris Lu
1f75f1f9dc
filer: fix mysql2 SQL template
2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70
filer: fix mysql command to upsert
2021-11-11 22:27:13 -08:00
user
9668b15f38
Filtering by volume id is prioritized.
2021-11-11 18:18:56 +09:00
user
563a74a9eb
Volume filter by collection pattern added.
2021-11-11 18:01:47 +09:00
user
dbb8003ce3
Volume filter function added.
2021-11-11 17:45:17 +09:00
user
c387fe957b
Unused parameter removed.
2021-11-11 16:08:41 +09:00
Konstantin Lebedev
42f4aa824b
force overwrite s3-is-admin
...
https://github.com/chrislusf/seaweedfs/issues/2433
2021-11-11 11:34:01 +05:00
Chris Lu
c77c0de418
make install
2021-11-09 12:34:57 -08:00
Chris Lu
a7eee8b14b
Merge pull request #2427 from Bl1tz23/master
...
Fix: potencial iam identities data race
2021-11-09 08:53:48 -08:00
Chris Lu
1d4a61af5e
add brokers
2021-11-09 08:50:55 -08:00
Bl1tz23
c683409e92
s3: add RWMutex to iam, use RLock for concurrent reading
2021-11-09 18:11:06 +03:00
Bl1tz23
5018b22f36
s3: fix potencial iam identities data race
2021-11-09 12:19:50 +03:00
Chris Lu
59d1435d65
s3: avoid possible attacks by version
2021-11-08 17:47:56 -08:00
Chris Lu
5c99b8acc3
rename variables
2021-11-08 17:47:56 -08:00
Chris Lu
00fdab1284
adjust client name
2021-11-08 17:47:56 -08:00
Chris Lu
4729a57cc0
use constants
2021-11-08 17:47:56 -08:00
Chris Lu
d9dd72ea56
rename pacakge
2021-11-08 17:47:56 -08:00
Chris Lu
5cf332357b
2.77
2021-11-07 13:52:45 -08:00
Chris Lu
244841f080
adjust logs
2021-11-07 13:27:57 -08:00
Chris Lu
6e999f1176
s3: fix ListObject if more than 10000 objects
2021-11-07 12:39:36 -08:00
Chris Lu
a8b0f8864d
fix help message
2021-11-07 12:38:35 -08:00
Chris Lu
55973e8572
log unknown access key
2021-11-07 12:37:46 -08:00
Chris Lu
c4e22b5a9a
filer: deprecate "-peers" option
2021-11-06 14:36:45 -07:00
Chris Lu
751a7073e3
Merge branch 'flexible_filer_addition'
2021-11-06 14:26:26 -07:00
Chris Lu
e0fc2898e9
auto updated filer peer list
2021-11-06 14:23:35 -07:00
Chris Lu
04663c3611
remote.mount: print out metadata sync errors
2021-11-06 11:29:50 -07:00
Chris Lu
330d1fde7f
send peers info to filers
2021-11-06 11:29:50 -07:00
Chris Lu
84bb8e7365
send peers info to filers
2021-11-06 04:07:38 -07:00
Chris Lu
4b9c42996a
refactor grpc API
2021-11-05 18:11:40 -07:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
.
2021-11-05 17:52:15 -07:00
Chris Lu
314c32514b
add command to list filers
2021-11-05 17:39:41 -07:00
Chris Lu
7348d2ae63
randomize traffic if replicated
2021-11-05 00:03:13 -07:00
Chris Lu
00ae965d8d
randomize a bit for ec shards distribution
2021-11-04 09:23:40 -07:00
Chris Lu
77f90ae288
add leader election in master
2021-11-04 00:54:38 -07:00
Chris Lu
35c37562bc
check auth only when enabled
2021-11-03 12:11:36 -07:00
Chris Lu
3c245c69d3
handle possible race condition
2021-11-03 01:36:39 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62
better printout
2021-11-02 23:45:47 -07:00
Chris Lu
e66865a8c5
adjust master client log level
2021-11-02 23:45:28 -07:00
Chris Lu
0c8dea9de8
go fmt
2021-11-02 23:39:16 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
18bfbf62fc
add one unit test
2021-11-02 23:36:43 -07:00
Chris Lu
e1ab8b01d0
s3: adjust permission for HEAD bucket operation
...
fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856
2021-11-02 16:28:55 -07:00
Chris Lu
b25661c6df
s3: skip permission checking for creating bucket if the bucket already exists
...
fix https://github.com/chrislusf/seaweedfs/issues/2417
Rclone was trying to create the bucket even though the bucket already exists.
2021-11-02 13:13:36 -07:00
Chris Lu
1b90d60765
filer: paginate large directories to delete file chunks
2021-11-02 01:04:50 -07:00
Chris Lu
794375ca0a
adjust help message since both fullPercent and quietFor are needed.
2021-11-01 17:22:47 -07:00
Chris Lu
fc9e246592
2.76
2021-10-31 18:08:28 -07:00
Chris Lu
9cf756b188
S3: support CORS
...
related to https://github.com/chrislusf/seaweedfs/issues/1271
2021-10-31 18:06:43 -07:00
Chris Lu
f17fa400d5
refactoring
2021-10-31 18:05:34 -07:00
Chris Lu
006b4ec0f8
refactoring
2021-10-31 18:02:08 -07:00
Chris Lu
edbf6d297b
filer.meta.tail: add example to send metadata to elastic search
2021-10-31 18:01:33 -07:00
Chris Lu
24858507cc
rename API to avoid confusion
2021-10-30 19:27:25 -07:00
Chris Lu
f5824f369c
avoid reporting error when it is a user request error
2021-10-29 14:14:29 -07:00
Chris Lu
c857cc7286
cloud tier: remove tagging since not all s3 vendors support this
2021-10-29 12:39:19 -07:00
Chris Lu
d04cdcf40d
s3 header add user agent
2021-10-29 12:28:24 -07:00
Konstantin Lebedev
dc3ae5e8a6
DeleteBucketLifecycleConfiguration return 204
2021-10-28 19:34:37 +05:00
Konstantin Lebedev
2afb5a13af
fix DeleteBucketLifecycleConfiguration
2021-10-28 18:30:33 +05:00
Chris Lu
2e76834e4d
filer store: redis2 fix wrong pagination
2021-10-27 23:46:25 -07:00
Chris Lu
54b6e0f3fd
adjust logs
2021-10-27 23:46:07 -07:00
Chris Lu
900e888695
filer: avoid possible prefixed pagination loop on unsupported filer stores
2021-10-27 23:45:48 -07:00
Chris Lu
d774fa6c9a
rename variable
2021-10-25 14:39:20 -07:00
Chris Lu
2539ba0b62
fix compilation
2021-10-25 14:38:48 -07:00
Chris Lu
5f2d7c1589
erasure coding: skip erasure coding if less than recommended 4 nodes
2021-10-25 14:38:11 -07:00
Chris Lu
c9d3fb4a30
2.75
2021-10-24 18:15:59 -07:00
Chris Lu
5435027ff0
volume copy: stream out copying progress and avoid grpc request timeout
...
fix https://github.com/chrislusf/seaweedfs/issues/2386
2021-10-24 02:52:56 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Konstantin Lebedev
235329a92a
fix Tagging add xmlns
2021-10-20 19:40:22 +05:00
Konstantin Lebedev
dc7e525cb9
fix Tagging test
2021-10-20 19:12:00 +05:00
Konstantin Lebedev
0b834600d5
fix PutObjectTaggingHandler Unmarshal
2021-10-20 17:58:06 +05:00
Konstantin Lebedev
c78220a7f2
fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389
2021-10-20 16:01:06 +05:00
Chris Lu
182f43ae5f
2.74
2021-10-18 14:23:54 -07:00
Chris Lu
309f46e3af
S3: fix upload limit if the size is more than 8GB
2021-10-18 14:14:18 -07:00
Chris Lu
cd4fa7561b
2.73
2021-10-18 10:47:48 -07:00
Chris Lu
a6bb509cb7
adjust mime detection
2021-10-18 10:47:39 -07:00
Konstantin Lebedev
dd2cc1acaf
fix https://github.com/chrislusf/seaweedfs/issues/2387
2021-10-18 16:27:57 +05:00
Chris Lu
97c963bac9
2.72
2021-10-17 17:40:27 -07:00
Chris Lu
3833dac3f7
continue to read from memory if there is no flush
2021-10-17 13:53:04 -07:00
Chris Lu
29fa1b9fdc
add logs
2021-10-17 13:50:34 -07:00
Chris Lu
cca62fdb30
mount: streaming renaming folders
2021-10-17 04:22:42 -07:00
Chris Lu
004e56c1a6
save updated lastTsNs
2021-10-17 02:01:20 -07:00
Chris Lu
71a8c80f6f
s3: continuously listen to file updates
...
fix https://github.com/chrislusf/seaweedfs/issues/2342
2021-10-17 01:04:19 -07:00
Chris Lu
93bb7869b8
Revert "mount: fix renaming a deep directory with unvisited directories"
...
This reverts commit 0ccdb937bb
.
2021-10-16 23:52:30 -07:00
Chris Lu
8e2c9713a3
turn on new faster algorithm to translate into visible chunks
2021-10-16 23:35:20 -07:00
Chris Lu
e7d1f183cc
ensure file id is populated
2021-10-16 23:34:27 -07:00
Chris Lu
fc1694b321
add tests
2021-10-16 23:34:12 -07:00
Chris Lu
0ccdb937bb
mount: fix renaming a deep directory with unvisited directories
2021-10-16 23:33:45 -07:00
Chris Lu
b9a2efd69b
temporarily reverting
2021-10-16 16:29:10 -07:00
Chris Lu
8a124ef9fc
Revert "remove deprecated code"
...
This reverts commit de7688c539
.
2021-10-16 16:09:33 -07:00
Chris Lu
93624635b4
Revert "go fmt"
...
This reverts commit 6c704eb2ba
.
2021-10-16 16:09:30 -07:00
Chris Lu
6c704eb2ba
go fmt
2021-10-16 16:06:18 -07:00
Chris Lu
de7688c539
remove deprecated code
2021-10-16 16:06:12 -07:00
Chris Lu
7336990639
faster file read for large files
2021-10-16 16:03:16 -07:00
Chris Lu
8965a53c4d
add warning error
2021-10-16 15:57:30 -07:00
Chris Lu
8e3075d051
ensure file entry chunks have populated the file id
2021-10-16 01:05:48 -07:00
Chris Lu
822ee2475b
remove unnecessary file
2021-10-16 01:05:06 -07:00
Konstantin Lebedev
8de7915950
s3 mime detect
2021-10-14 15:03:11 +05:00
Chris Lu
5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil
...
refactor: move from io/ioutil to io and os package
2021-10-13 22:38:58 -07:00
Chris Lu
46a09c6074
adjust test
2021-10-13 22:38:47 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
4cbd390fbe
test: add fail message
2021-10-13 20:42:20 -07:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy
...
Bucket policy
2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1
GetBucketLifecycleConfigurationHandler
2021-10-14 01:35:33 +05:00
Konstantin Lebedev
458145425e
S3 MultipartUpload pass contentType to meta
2021-10-12 17:14:54 +05:00
Chris Lu
de1ec050e8
fier/s3: save "Content-Disposition" to extended properties
...
related to https://github.com/chrislusf/seaweedfs/issues/2371
2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f
escape file name only when necessary
2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99
s3: use "response-content-disposition" to overwrite default content-disposition
...
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371
see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
2021-10-11 23:29:52 -07:00
Chris Lu
f0b928ff5e
go fmt
2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc
volume: find a non-empty offset when binary searching by timestamp
2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327
volume: find a non-empty offset when binary searching by timestamp
...
fix https://github.com/chrislusf/seaweedfs/issues/2364
2021-10-11 16:11:50 -07:00
Konstantin Lebedev
be4b3ed509
AclHandlers
2021-10-11 15:03:56 +05:00
Chris Lu
e6ef7b2387
return error early
...
fix https://github.com/chrislusf/seaweedfs/issues/2370
2021-10-11 01:24:30 -07:00
Chris Lu
3d586be552
2.71
2021-10-10 22:40:44 -07:00
Chris Lu
f4676824a7
fix test code compilation
2021-10-10 22:14:13 -07:00
Chris Lu
8205166009
disable testing with redis server
...
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
panic: exec: "redis-server": executable file not found in $PATH
goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
2021-10-10 21:57:55 -07:00
Chris Lu
3e2acf677c
removing tikv to resolve "go mod tidy" problem
...
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.
go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
github.com/tikv/client-go/v2/tikv imports
go.etcd.io/etcd/clientv3 tested by
go.etcd.io/etcd/clientv3.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d
removing etcd sequencer
...
causing go mod tidy problem. If anyone wants this, please help to resolve this first.
github.com/chrislusf/seaweedfs/weed/sequence imports
go.etcd.io/etcd/client tested by
go.etcd.io/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
17e08a16f4
fix redis3 deletion
2021-10-09 04:54:48 -07:00
Chris Lu
ce1efeb0eb
adds more error message
2021-10-09 04:54:14 -07:00
Chris Lu
e4830bd93d
go fmt
2021-10-07 21:13:31 -07:00
Chris Lu
0a856241fe
avoid int bigger than math.MaxInt32
...
fix https://github.com/chrislusf/seaweedfs/issues/2363
2021-10-07 21:12:57 -07:00
Chris Lu
d688e10ed1
do not try to compress if not sure about the file content
2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8
revert temporary changes that disabled compression
...
fix https://github.com/chrislusf/seaweedfs/issues/2362
2021-10-07 13:24:16 -07:00
Chris Lu
f3d8232e14
reduce one redis lookup on hot path
2021-10-06 22:01:19 -07:00
Chris Lu
2336a397dc
use pipeline to save some time
2021-10-06 20:40:18 -07:00
Chris Lu
371fead8a5
redis3 using redis native sorted set
2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d
test with real redis
...
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12 1789 760599 ns/op
BenchmarkRedis-12 17539 64122 ns/op
PASS
2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a
add benchmark test
2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1
redis3 adds distributed locking
2021-10-06 00:03:54 -07:00
Chris Lu
f0d1e7bd05
skip ec volumes when loading normal volumes
2021-10-05 02:31:44 -07:00
Chris Lu
332d49432d
reduce concurrent volume grow requests
2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00
refactor
2021-10-05 00:40:04 -07:00
Chris Lu
8a66306064
calculate disk usage in case of race condition
...
related to https://github.com/chrislusf/seaweedfs/issues/2357
2021-10-04 23:32:07 -07:00
Chris Lu
4ed2994555
use tsMemory to determine whether read from disk or memory
...
remove lastFlushTime
2021-10-04 16:02:56 -07:00
Chris Lu
6a030547a2
server: remove peer check if not starting master
...
more fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1
add back logic to check master peers
...
fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree
...
Add Redis3
2021-10-04 02:43:15 -07:00
Chris Lu
947add39e6
clean up *SkipListElementReference loaded from Redis
2021-10-04 02:31:38 -07:00
Chris Lu
513fed323a
SkipListElementReference can be an empty object
2021-10-04 02:30:44 -07:00
Chris Lu
280ab7f95c
add test
2021-10-04 02:30:24 -07:00
Chris Lu
04662126bb
add redis3
2021-10-04 01:04:27 -07:00
Chris Lu
2b9aab3442
use 1000 per batch
2021-10-04 01:03:40 -07:00
Chris Lu
366f522a2d
add redis3
2021-10-04 01:01:31 -07:00
Chris Lu
ba7fbac07f
rename
2021-10-03 19:23:34 -07:00
Chris Lu
e6196cdc50
add name list
2021-10-03 17:54:25 -07:00
Chris Lu
a481c4a45e
return previous element if visited
2021-10-03 13:50:52 -07:00
Chris Lu
22d8684e88
refactor out listStore
2021-10-03 02:19:21 -07:00
Chris Lu
d343b0db57
update value
2021-10-03 01:15:14 -07:00
Chris Lu
4f50f8c2ca
insert key and value
2021-10-03 01:07:35 -07:00
Chris Lu
69b84bb771
TestFindGreaterOrEqual
2021-10-02 14:15:49 -07:00
Chris Lu
57e2fd3f9b
remove bptree
2021-10-02 14:03:54 -07:00