chrislu
735038b2c1
backup do not need to use preallocation
...
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
Konstantin Lebedev
f127b326bf
add options to scaffold
2022-05-03 22:54:31 +05:00
Konstantin Lebedev
d8925b4e83
Merge branch 'new_master' into ydb
...
# Conflicts:
# go.mod
# go.sum
2022-05-03 00:13:57 +05:00
chrislu
94635e9b5c
filer: add filer group
2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4
ydb-go-sdk move to v3
2022-05-02 02:07:47 +05:00
chrislu
192983b464
s3 backend support customizing storage class
2022-04-30 17:36:40 -07:00
chrislu
3885374edf
conditionally build elastic, gocdk to reduce binary size
2022-04-21 01:10:46 -07:00
chrislu
6994e88192
fix typo
2022-04-21 00:02:18 -07:00
a
1d6a9e66b6
Merge branch 'master' into a
2022-04-20 14:01:42 -07:00
chrislu
a69f0b62c2
start filer local socket only on non windows
...
fix https://github.com/chrislusf/seaweedfs/issues/2930
2022-04-18 00:08:31 -07:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc
2022-04-18 10:39:29 +08:00
justin
3551ca2fcf
enhancement: replace sort.Slice with slices.SortFunc to reduce reflection
2022-04-18 10:35:43 +08:00
chrislu
1384529eb7
Fix filer.backup deletes files in backup folder in incremental mode
...
fix https://github.com/chrislusf/seaweedfs/issues/2919
2022-04-14 13:35:01 -07:00
a
846858fb43
merge master
2022-04-12 14:42:29 -07:00
a
41d396edc4
Merge branch 'master' into a
2022-04-07 08:22:59 -07:00
Konstantin Lebedev
f5246b748d
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
chrislu
86ed27f602
mount: remove leftover socket file
2022-04-07 00:33:13 -07:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a
2022-04-06 18:45:55 -05:00
Konstantin Lebedev
17c6e8e39f
Merge branch 'new_master' into hashicorp_raft
...
# Conflicts:
# go.mod
# go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
622297f1a7
add stats raft handler
2022-04-04 19:16:06 +05:00
Konstantin Lebedev
14dd971890
hashicorp raft with state machine
2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b
initial add hashicorp raft
2022-04-04 13:50:56 +05:00
chrislu
6a2bcd03aa
configure mount quota
2022-04-02 21:34:26 -07:00
chrislu
958f880b70
mount: add grpc method to adjust quota
2022-04-02 15:14:37 -07:00
chrislu
daca2d22a5
use original server address string as map key
2022-04-01 17:34:42 -07:00
a
7e92517571
change user and pass to username and password
2022-04-01 14:09:25 -05:00
Konstantin Lebedev
b6a2b43efe
allowDeleteBucketNotEmpty
2022-03-30 22:46:13 +05:00
a
bc603e534f
diff
2022-03-28 16:50:28 +00:00
chrislu
21e0898631
refactor: change masters from a slice to a map
2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1
refactor
2022-03-26 13:13:19 -07:00
Konstantin Lebedev
849a2c1f68
fix cmdFiler
2022-03-23 14:30:46 +05:00
Konstantin Lebedev
105bcfdbd4
set iam ip from filer
2022-03-23 13:45:51 +05:00
elee
423ce57cde
prefix search, bucket implemented
2022-03-17 21:12:25 -05:00
Berck Nash
7ee38fa3a4
The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems:
...
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified. Instead, it bound to localhost which broke the default configuration. This change implements the documented behavior instead.
(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP. I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
2022-03-17 15:30:23 -06:00
elee
921535001a
arangodb adapter
2022-03-17 04:49:26 -05:00
Berck Nash
9b14f0c81a
Add mTLS support for both master and volume http server.
2022-03-16 09:52:17 -06:00
chrislu
3639cad69c
master, filer, s3: also listen to "localhost" in addition to specific ip address
...
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
chrislu
3a6eb8ca5f
default bind to one ip address
...
fix https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-11 14:02:39 -08:00
banjiaojuhao
b9ff7723dd
[bugfix] filer: nil pointer dereference
2022-03-07 23:26:25 +08:00
chrislu
da3d330616
s3 and filer transport using unix domain socket instead of tcp
2022-03-07 02:00:14 -08:00
chrislu
c7e8ac18f0
mount: quota for one mounted collection
...
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
2022-03-06 02:44:40 -08:00
chrislu
f51e20028a
mount: avoid comma in mount options
...
fix https://github.com/chrislusf/seaweedfs/issues/2719
2022-03-03 03:42:29 -08:00
chrislu
18543c6e8b
minor
2022-02-27 23:11:09 -08:00
chrislu
c3792c8352
remove dead code
2022-02-27 03:03:19 -08:00
chrislu
aa9eef81e6
retire mount v1
2022-02-27 02:57:27 -08:00
chrislu
6e1ab97988
use debug option to see operations
2022-02-27 02:02:30 -08:00
chrislu
2112d99140
mount2: add back readonly mode
2022-02-27 01:13:32 -08:00
chrislu
fc7a4957ea
fix mount2 options
2022-02-25 21:22:44 -08:00
chrislu
202a29d014
refactoring
2022-02-25 01:17:26 -08:00
Tuan Vuong
d2ec62656d
initialize master address in iam options
2022-02-23 12:01:54 +07:00
banjiaojuhao
b5ec346700
FilerStore: add redis_lua
2022-02-15 20:54:57 +08:00
chrislu
4e72863ba5
mount2 add debug mode
2022-02-15 00:26:30 -08:00
chrislu
4e181db21a
mount: default disable cache
...
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
2022-02-14 20:42:33 -08:00
chrislu
377bf31445
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-02-14 13:48:51 -08:00
chrislu
e8420aaed7
fix building for windows freebsd
2022-02-14 13:48:48 -08:00
Chris Lu
d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1
2022-02-14 10:08:37 -08:00
Konstantin Lebedev
526094d2da
StopTimeout 30 sec
2022-02-14 21:42:27 +05:00
Konstantin Lebedev
275e9a4e86
reduce to default http server KillTimeout and StopTimeout
2022-02-14 21:38:24 +05:00
garenchan
bd032eabe7
[UPDATE] Make heartbeat interval and election timeout of masters configurable.
2022-02-14 21:09:07 +08:00
chrislu
05724a68d4
skip other OS
2022-02-14 02:59:51 -08:00
chrislu
b9c2bff931
clean up
2022-02-14 02:14:26 -08:00
chrislu
dbeeda8123
listen for metadata updates
2022-02-14 01:09:31 -08:00
chrislu
180445f5a8
change to use fuse file system
2022-02-11 21:35:09 -08:00
chrislu
f87da798a4
to be re-written following fuse virtual file system
2022-02-11 03:09:30 -08:00
chrislu
b6143de52a
mount with name
2022-02-10 22:43:55 -08:00
chrislu
7a0c35674c
clean up previously mounted folder
2022-02-10 20:46:53 -08:00
chrislu
c3f9d9fa2e
initial setup
2022-02-10 20:32:13 -08:00
chrislu
85c1615b43
filer read empty file may cause OOM in some cases
...
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
chrislu
affe3c2c12
change to util.WriteFile
2022-02-04 21:32:27 -08:00
chrislu
84c9bc4389
edge case: old entry was not replicated to remote storage
2022-01-30 20:23:24 -08:00
Chris Lu
7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics
...
master metricsHttpPort
2022-01-20 09:26:25 -08:00
chrislu
b3e526ba95
url should be always using forward slash
2022-01-19 22:16:26 -08:00
Konstantin Lebedev
77c98b657e
master metricsHttpPort
2022-01-19 21:43:22 +05:00
guol-fnst
da9540e666
add gocql timeout setting
2022-01-18 15:21:13 +08:00
chrislu
fc0628c038
working
2022-01-17 01:53:56 -08:00
chrislu
2bfeb5d1c8
add filer to iam option
2022-01-15 03:37:52 -08:00
chrislu
b17c426e99
weed server: optionally start IAM service
...
related to https://github.com/chrislusf/seaweedfs/issues/2560
2022-01-13 22:49:49 -08:00
chrislu
8907e6a40a
add more help messages
2022-01-13 13:03:04 -08:00
chrislu
826a7b307e
master: remove hard coded filer settings in master.toml
...
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails
...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]
goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
/go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-01-11 12:24:56 -08:00
chrislu
41daecfdca
Update mount_std.go
2022-01-11 12:23:12 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
...
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu
6a12520a96
fix logging
2022-01-10 01:00:11 -08:00
Radtoo
389002f195
Using positional arguments rather than option flag to enable better shell usage
2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a
Now works with a single file too
...
Parsing removed from doFixOneVolume
Needle init removed from runFix
2022-01-08 16:31:53 +01:00
chrislu
110d5a5233
support fixing a collection of volumes, or volumes under one directory
2022-01-07 14:52:16 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id
2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
...
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
1cd3b6b4e1
BUGFIX: security.toml contained wrong keys
2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst
10404c4275
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
...
- one JWT for reading and one for writing, analogous to how the JWT
between Master and Volume Server works
- I did not implement IP `whiteList` parameter on the filer
Additionally, because http_util.DownloadFile now sets the JWT,
the `download` command should now work when `jwt.signing.read` is
configured. By looking at the code, I think this case did not work
before.
## Docs to be adjusted after a release
Page `Amazon-S3-API`:
```
# Authentication with Filer
You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as
explained in [Security-Configuration](Security-Configuration) -
controlled by the `grpc.*` configuration in `security.toml`.
Starting with version XX, it is also possible to authenticate the HTTP
operations between the S3-API-Proxy and the Filer (especially
uploading new files). This is configured by setting
`filer_jwt.signing.key` and `filer_jwt.signing.read.key` in
`security.toml`.
With both configurations (gRPC and JWT), it is possible to have Filer
and S3 communicate in fully authenticated fashion; so Filer will reject
any unauthenticated communication.
```
Page `Security Overview`:
```
The following items are not covered, yet:
- master server http REST services
Starting with version XX, the Filer HTTP REST services can be secured
with a JWT, by setting `filer_jwt.signing.key` and
`filer_jwt.signing.read.key` in `security.toml`.
...
Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer.
Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).**
...
# Securing Filer HTTP with JWT
To enable JWT-based access control for the Filer,
1. generate `security.toml` file by `weed scaffold -config=security`
2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string
3. copy the same `security.toml` file to the filers and all S3 proxies.
If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`.
If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`.
The S3 API Gateway reads the above JWT keys and sends authenticated
HTTP requests to the filer.
```
Page `Security Configuration`:
```
(update scaffold file)
...
[filer_jwt.signing]
key = "blahblahblahblah"
[filer_jwt.signing.read]
key = "blahblahblahblah"
```
Resolves : #158
2021-12-30 14:45:27 +01:00
chrislu
5c87fcc6d2
add client id for all metadata listening clients
2021-12-30 00:23:57 -08:00
chrislu
fb434318e3
dynamically adjust connection timeout
...
better fix for https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:44:39 -08:00
chrislu
5788bf2270
s3: increase timeout limit
...
https://github.com/chrislusf/seaweedfs/issues/2541
2021-12-29 22:21:02 -08:00
chrislu
498661e3bb
mount: remove limits on number of parallel requests
2021-12-28 17:41:01 -08:00
chrislu
0da2dfd640
fuse: change to direct io mode
...
before and after:
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.86s system 84% cpu 4.561 total
chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192
8192+0 records in
8192+0 records out
1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec)
dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.22s system 83% cpu 3.857 total
2021-12-28 12:22:56 -08:00
chrislu
9f9ef1340c
use streaming mode for long poll grpc calls
...
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Sebastian Kurfürst
6db49100d6
BUGFIX: add access.ui setting to scaffolded security.toml
...
... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)
2021-12-24 13:59:04 +01:00
chrislu
85c526c583
s3: bind to a specific IP
...
fix https://github.com/chrislusf/seaweedfs/issues/2516
2021-12-17 11:34:37 -08:00
chrislu
bf4d7affc0
gateway to remote object store: skip replicating multipart upload part files
...
fix https://github.com/chrislusf/seaweedfs/issues/2509
2021-12-14 19:48:31 -08:00
chrislu
316f326464
add more help message
2021-12-13 13:14:36 -08:00
kmlebedev
4f98553ba9
audit log SignatureVersion
2021-12-10 19:40:32 +05:00
Konstantin Lebedev
98251fe16a
non blocking audit log
2021-12-09 19:47:16 +05:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log
2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81
audit log config
2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20
audit log
2021-12-07 12:15:48 +05:00
chrislu
c146c76d10
avoid creating the same bucket with a different randomized name
...
related to https://github.com/chrislusf/seaweedfs/issues/2492
2021-12-05 13:06:41 -08:00
chrislu
42d97a3442
adjust randomized bucket name
2021-12-05 12:36:58 -08:00
Chris Lu
689f5513a9
redis3 supports sentinel
2021-11-29 01:09:51 -08:00
Chris Lu
3d7390302d
add s3.clean.uploads -timeAgo=24h
2021-11-29 00:49:49 -08:00
Chris Lu
ad16221a35
adjust error log
2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d
add logs for writing to remote file
2021-11-27 22:09:23 -08:00
limd
8805c04128
fix redis2 sentinel config example
2021-11-25 19:20:02 +08:00
limd
ec03f22cc3
Merge remote-tracking branch 'origin/master'
2021-11-25 16:07:14 +08:00
limd
220797bd71
support redis sentinel
2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0
add retries when writing to remote s3
...
fix https://github.com/chrislusf/seaweedfs/issues/2465
2021-11-22 21:48:04 -08:00
Chris Lu
1f75f1f9dc
filer: fix mysql2 SQL template
2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70
filer: fix mysql command to upsert
2021-11-11 22:27:13 -08:00
Chris Lu
c4e22b5a9a
filer: deprecate "-peers" option
2021-11-06 14:36:45 -07:00
Chris Lu
5ea86ef1da
Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()"
...
This reverts commit af71ae11aa
2021-11-05 17:52:15 -07:00
Chris Lu
af71ae11aa
master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()
2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62
better printout
2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7
shell: optionally read filer address from master
2021-11-02 23:38:45 -07:00
Chris Lu
edbf6d297b
filer.meta.tail: add example to send metadata to elastic search
2021-10-31 18:01:33 -07:00
Chris Lu
3be3c17f59
volume vacuum: avoid timeout with streaming progress report
...
fix https://github.com/chrislusf/seaweedfs/issues/2396
2021-10-24 01:55:34 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
...
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil . This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
3e2acf677c
removing tikv to resolve "go mod tidy" problem
...
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.
go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
github.com/tikv/client-go/v2/tikv imports
go.etcd.io/etcd/clientv3 tested by
go.etcd.io/etcd/clientv3.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d
removing etcd sequencer
...
causing go mod tidy problem. If anyone wants this, please help to resolve this first.
github.com/chrislusf/seaweedfs/weed/sequence imports
go.etcd.io/etcd/client tested by
go.etcd.io/etcd/client.test imports
github.com/coreos/etcd/integration imports
github.com/coreos/etcd/proxy/grpcproxy imports
google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
6a030547a2
server: remove peer check if not starting master
...
more fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1
add back logic to check master peers
...
fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree
...
Add Redis3
2021-10-04 02:43:15 -07:00
Chris Lu
04662126bb
add redis3
2021-10-04 01:04:27 -07:00
Chris Lu
366f522a2d
add redis3
2021-10-04 01:01:31 -07:00
Chris Lu
b297849147
typo
2021-10-01 23:24:54 -07:00
Chris Lu
af207bbaf0
retry both assign volume and uploading data
...
fix https://github.com/chrislusf/seaweedfs/issues/2351
2021-10-01 23:23:39 -07:00
Chris Lu
4a1d4d7462
s3: default to allow empty folder for better performance
2021-09-26 22:34:14 -07:00
Chris Lu
a814f3f0a8
adjust metadata tail output
2021-09-25 01:04:51 -07:00
Chris Lu
52fe86df45
use default 10000 for grpc port
2021-09-20 14:05:59 -07:00
Chris Lu
84fdda85e7
go fmt
2021-09-19 12:06:15 -07:00
Chris Lu
94a01fcfcb
filer.remote.gateway: add options to include or exclude new bucket names to mirror
2021-09-15 23:04:16 -07:00
Chris Lu
b5f4910412
adjust help messages
2021-09-15 22:53:10 -07:00
Chris Lu
f0907eb83c
add command filre.remote.gateway
2021-09-15 22:48:04 -07:00
Chris Lu
63da4bbb54
separate filer.remote.gateway command to avoid confusion
2021-09-15 22:47:17 -07:00
Chris Lu
2789d10342
go fmt
2021-09-14 10:37:06 -07:00
Chris Lu
e5fc35ed0c
change server address from string to a type
2021-09-12 22:47:52 -07:00
Chris Lu
2c9d4c8f43
custom grpc port: filer
2021-09-12 02:28:37 -07:00
Chris Lu
e690a2be16
custom grpc port: volume server
2021-09-12 02:25:15 -07:00
Chris Lu
232ad2fe65
custom grpc port: master follower
2021-09-12 02:25:00 -07:00
Chris Lu
bd1f800efe
custom grpc port: master
2021-09-12 02:19:10 -07:00
Chris Lu
2b8ab1534a
mysql table DDL: minor, not necessary, but a little more strict
...
the directory is already hashed into dirhash. So this change is not strictly required.
2021-09-08 23:55:18 -07:00
Chris Lu
574485ec69
better IP v6 support
2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f
handle ipv6 addresses
2021-09-07 16:43:54 -07:00
Chris Lu
6923af7280
refactoring
2021-09-06 16:20:49 -07:00
Chris Lu
64f6532fbe
update help message
2021-09-06 16:16:22 -07:00
Chris Lu
308d4b2dae
filer.remote.sync: createBucketWithRandomSuffix defaults to true
2021-09-06 15:14:26 -07:00
Chris Lu
c218ef20c7
filer.remote.sync: automatically detect the primary remote storage
2021-09-06 15:10:55 -07:00
Chris Lu
1702ce5395
remove unused code
2021-09-06 12:01:44 -07:00
Chris Lu
60573fd3e2
option to map remote bucket to trimmed bucket name
2021-09-05 11:55:52 -07:00
Chris Lu
bdefdee4e6
filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts
2021-09-04 22:46:28 -07:00
Chris Lu
98c68d7a79
filer.remote.sync fix upload logic
2021-09-04 18:46:28 -07:00
Chris Lu
16a342eb8a
re-ordering
2021-09-04 13:46:44 -07:00
Chris Lu
796b2eb929
remember sync time
2021-09-04 13:46:22 -07:00
Chris Lu
99d4594ee8
update remote storage mount mapping
2021-09-04 05:22:34 -07:00
Chris Lu
13cc2ef723
when creating bucket, use remote storage client for default storage
2021-09-04 04:54:55 -07:00
Chris Lu
4b28c5f6c3
filer.remote.sync: split into buckets mode and single directory mode
2021-09-04 04:35:46 -07:00
Chris Lu
03a31587ce
go fmt
2021-09-03 20:42:28 -07:00
Chris Lu
7f2a8246a9
minor
2021-09-02 12:32:59 -07:00
Chris Lu
9a73b0e3c9
refactor
2021-09-02 07:07:16 -07:00
Chris Lu
e281f0fa82
refactor
2021-09-02 06:53:21 -07:00
Chris Lu
7ce97b59d8
go fmt
2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29
filer.remote.sync: exit when directory is unmounted
...
this will not propagate the deletions back to the cloud
2021-09-01 01:29:22 -07:00
Chris Lu
8e125339d5
line wrap
2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f
add additionalPathPrefixes
2021-08-31 23:26:26 -07:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv
2021-08-30 15:52:54 +08:00
Chris Lu
6deee4c0b9
import hdfs
2021-08-29 18:49:43 -07:00
Chris Lu
a31f2907f0
cloud drive: filer.remote.sync supports remove folder
2021-08-29 18:46:28 -07:00
Chris Lu
05d2774117
refactoring
2021-08-28 22:49:38 -07:00
Chris Lu
a9b41900d8
handle cases when mounted directory does not exist
2021-08-28 01:56:56 -07:00
Chris Lu
05a648bb96
refactor: separating out remote.proto
2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac
Add deleterange_concurrency to filer configuration file
2021-08-26 18:25:08 +08:00
yulai.li
de8ef28460
Update filer.toml scaffold to support tikv store
2021-08-26 16:26:27 +08:00
yulai.li
546efeba8f
Fix build bug
2021-08-26 16:20:35 +08:00
Chris Lu
6bab20d862
cat entry content
2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e
cloud drive: add support for Azure
2021-08-24 01:18:30 -07:00
Chris Lu
258063de26
cloud drive: add google cloud storage
2021-08-23 00:29:27 -07:00
Chris Lu
05fc7db755
filer.copy: large files pack file ids into chunks
2021-08-18 23:46:54 -07:00
Chris Lu
72eb6d5b9d
ensure no writes to remote storage if content is not changed
2021-08-15 20:23:41 -07:00
Chris Lu
5a7c40510f
format output
2021-08-15 20:07:13 -07:00
Chris Lu
5d5a21ba2d
adjust log format
2021-08-15 19:46:45 -07:00
Chris Lu
c45c12aedc
metadata updates
2021-08-15 19:27:30 -07:00
Chris Lu
49b5e47bd1
retry forever with filer.remote.sync, and some refactoring
2021-08-15 12:38:26 -07:00
Chris Lu
c34747c79d
rename, fix wrong logic.
2021-08-14 21:46:34 -07:00
Chris Lu
0db2517994
go fmt
2021-08-14 02:55:44 -07:00
Chris Lu
7937db52e1
Filer locationPrefix configure does not exec replication #2257
...
fix https://github.com/chrislusf/seaweedfs/issues/2257
2021-08-14 02:54:13 -07:00
Chris Lu
5a0f92423e
use grpc and jwt
2021-08-12 21:40:33 -07:00
Chris Lu
6238644c35
remove gateway command
2021-08-12 20:52:04 -07:00
Chris Lu
d1d1fc772c
move some volume lookup operations to grpc
...
jwt related lookup will come in next commit
2021-08-12 20:33:00 -07:00
Chris Lu
5571f4f70a
master: add master.follower to handle read file id lookup requests
2021-08-12 18:10:59 -07:00
Chris Lu
5469019852
adjust data type
2021-08-12 17:54:34 -07:00
Chris Lu
713c035a6e
shell: remote.cache remote.uncache
2021-08-09 14:35:18 -07:00
