mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-13 01:19:07 +08:00
Code Issues Packages Projects Releases Wiki Activity
7,087 Commits 32 Branches 296 Tags 150 MiB
320637dc7a
Commit Graph

907 Commits

Author SHA1 Message Date
chrislu
bf4d7affc0 gateway to remote object store: skip replicating multipart upload part files 
fix https://github.com/chrislusf/seaweedfs/issues/2509
 2021-12-14 19:48:31 -08:00
chrislu
316f326464 add more help message 2021-12-13 13:14:36 -08:00
kmlebedev
4f98553ba9 audit log SignatureVersion 2021-12-10 19:40:32 +05:00
Konstantin Lebedev
98251fe16a non blocking audit log 2021-12-09 19:47:16 +05:00
Chris Lu
2ba08afed1
Merge pull request #2498 from kmlebedev/s3_audit_log 2021-12-07 09:35:48 -08:00
Konstantin Lebedev
10678cde81 audit log config 2021-12-07 18:20:52 +05:00
Konstantin Lebedev
4ec8715f20 audit log 2021-12-07 12:15:48 +05:00
chrislu
c146c76d10 avoid creating the same bucket with a different randomized name 
related to https://github.com/chrislusf/seaweedfs/issues/2492
 2021-12-05 13:06:41 -08:00
chrislu
42d97a3442 adjust randomized bucket name 2021-12-05 12:36:58 -08:00
Chris Lu
689f5513a9 redis3 supports sentinel 2021-11-29 01:09:51 -08:00
Chris Lu
3d7390302d add s3.clean.uploads -timeAgo=24h 2021-11-29 00:49:49 -08:00
Chris Lu
ad16221a35 adjust error log 2021-11-28 22:06:17 -08:00
Chris Lu
cf1586a34d add logs for writing to remote file 2021-11-27 22:09:23 -08:00
limd
8805c04128 fix redis2 sentinel config example 2021-11-25 19:20:02 +08:00
limd
ec03f22cc3 Merge remote-tracking branch 'origin/master' 2021-11-25 16:07:14 +08:00
limd
220797bd71 support redis sentinel 2021-11-25 15:57:03 +08:00
Chris Lu
6c27845be0 add retries when writing to remote s3 
fix https://github.com/chrislusf/seaweedfs/issues/2465
 2021-11-22 21:48:04 -08:00
Chris Lu
1f75f1f9dc filer: fix mysql2 SQL template 2021-11-11 22:28:28 -08:00
Chris Lu
3abbaccb70 filer: fix mysql command to upsert 2021-11-11 22:27:13 -08:00
Chris Lu
c4e22b5a9a filer: deprecate "-peers" option 2021-11-06 14:36:45 -07:00
Chris Lu
5ea86ef1da Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" 
This reverts commit af71ae11aa.
 2021-11-05 17:52:15 -07:00
Chris Lu
af71ae11aa master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates() 2021-11-03 01:09:48 -07:00
Chris Lu
ab97b17e62 better printout 2021-11-02 23:45:47 -07:00
Chris Lu
5160eb08f7 shell: optionally read filer address from master 2021-11-02 23:38:45 -07:00
Chris Lu
edbf6d297b filer.meta.tail: add example to send metadata to elastic search 2021-10-31 18:01:33 -07:00
Chris Lu
3be3c17f59 volume vacuum: avoid timeout with streaming progress report 
fix https://github.com/chrislusf/seaweedfs/issues/2396
 2021-10-24 01:55:34 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package 
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
 2021-10-14 12:27:58 +08:00
Chris Lu
3e2acf677c removing tikv to resolve "go mod tidy" problem 
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.

go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
	github.com/tikv/client-go/v2/tikv imports
	go.etcd.io/etcd/clientv3 tested by
	go.etcd.io/etcd/clientv3.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d removing etcd sequencer 
causing go mod tidy problem. If anyone wants this, please help to resolve this first.

github.com/chrislusf/seaweedfs/weed/sequence imports
	go.etcd.io/etcd/client tested by
	go.etcd.io/etcd/client.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
 2021-10-10 19:10:46 -07:00
Chris Lu
6a030547a2 server: remove peer check if not starting master 
more fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1 add back logic to check master peers 
fix https://github.com/chrislusf/seaweedfs/issues/2352
 2021-10-04 02:51:26 -07:00
Chris Lu
d4bb16e20e
Merge pull request #2354 from chrislusf/bptree 
Add Redis3
 2021-10-04 02:43:15 -07:00
Chris Lu
04662126bb add redis3 2021-10-04 01:04:27 -07:00
Chris Lu
366f522a2d add redis3 2021-10-04 01:01:31 -07:00
Chris Lu
b297849147 typo 2021-10-01 23:24:54 -07:00
Chris Lu
af207bbaf0 retry both assign volume and uploading data 
fix https://github.com/chrislusf/seaweedfs/issues/2351
 2021-10-01 23:23:39 -07:00
Chris Lu
4a1d4d7462 s3: default to allow empty folder for better performance 2021-09-26 22:34:14 -07:00
Chris Lu
a814f3f0a8 adjust metadata tail output 2021-09-25 01:04:51 -07:00
Chris Lu
52fe86df45 use default 10000 for grpc port 2021-09-20 14:05:59 -07:00
Chris Lu
84fdda85e7 go fmt 2021-09-19 12:06:15 -07:00
Chris Lu
94a01fcfcb filer.remote.gateway: add options to include or exclude new bucket names to mirror 2021-09-15 23:04:16 -07:00
Chris Lu
b5f4910412 adjust help messages 2021-09-15 22:53:10 -07:00
Chris Lu
f0907eb83c add command filre.remote.gateway 2021-09-15 22:48:04 -07:00
Chris Lu
63da4bbb54 separate filer.remote.gateway command to avoid confusion 2021-09-15 22:47:17 -07:00
Chris Lu
2789d10342 go fmt 2021-09-14 10:37:06 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
2c9d4c8f43 custom grpc port: filer 2021-09-12 02:28:37 -07:00
Chris Lu
e690a2be16 custom grpc port: volume server 2021-09-12 02:25:15 -07:00
Chris Lu
232ad2fe65 custom grpc port: master follower 2021-09-12 02:25:00 -07:00
Chris Lu
bd1f800efe custom grpc port: master 2021-09-12 02:19:10 -07:00
Chris Lu
2b8ab1534a mysql table DDL: minor, not necessary, but a little more strict 
the directory is already hashed into dirhash. So this change is not strictly required.
 2021-09-08 23:55:18 -07:00
Chris Lu
574485ec69 better IP v6 support 2021-09-07 19:29:42 -07:00
Chris Lu
0128239c0f handle ipv6 addresses 2021-09-07 16:43:54 -07:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
64f6532fbe update help message 2021-09-06 16:16:22 -07:00
Chris Lu
308d4b2dae filer.remote.sync: createBucketWithRandomSuffix defaults to true 2021-09-06 15:14:26 -07:00
Chris Lu
c218ef20c7 filer.remote.sync: automatically detect the primary remote storage 2021-09-06 15:10:55 -07:00
Chris Lu
1702ce5395 remove unused code 2021-09-06 12:01:44 -07:00
Chris Lu
60573fd3e2 option to map remote bucket to trimmed bucket name 2021-09-05 11:55:52 -07:00
Chris Lu
bdefdee4e6 filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts 2021-09-04 22:46:28 -07:00
Chris Lu
98c68d7a79 filer.remote.sync fix upload logic 2021-09-04 18:46:28 -07:00
Chris Lu
16a342eb8a re-ordering 2021-09-04 13:46:44 -07:00
Chris Lu
796b2eb929 remember sync time 2021-09-04 13:46:22 -07:00
Chris Lu
99d4594ee8 update remote storage mount mapping 2021-09-04 05:22:34 -07:00
Chris Lu
13cc2ef723 when creating bucket, use remote storage client for default storage 2021-09-04 04:54:55 -07:00
Chris Lu
4b28c5f6c3 filer.remote.sync: split into buckets mode and single directory mode 2021-09-04 04:35:46 -07:00
Chris Lu
03a31587ce go fmt 2021-09-03 20:42:28 -07:00
Chris Lu
7f2a8246a9 minor 2021-09-02 12:32:59 -07:00
Chris Lu
9a73b0e3c9 refactor 2021-09-02 07:07:16 -07:00
Chris Lu
e281f0fa82 refactor 2021-09-02 06:53:21 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
3bd48c4f29 filer.remote.sync: exit when directory is unmounted 
this will not propagate the deletions back to the cloud
 2021-09-01 01:29:22 -07:00
Chris Lu
8e125339d5 line wrap 2021-08-31 23:30:28 -07:00
Chris Lu
68ea99190f add additionalPathPrefixes 2021-08-31 23:26:26 -07:00
Rain Li
e5f9ff983d
Merge branch 'master' into filerstore-tikv 2021-08-30 15:52:54 +08:00
Chris Lu
6deee4c0b9 import hdfs 2021-08-29 18:49:43 -07:00
Chris Lu
a31f2907f0 cloud drive: filer.remote.sync supports remove folder 2021-08-29 18:46:28 -07:00
Chris Lu
05d2774117 refactoring 2021-08-28 22:49:38 -07:00
Chris Lu
a9b41900d8 handle cases when mounted directory does not exist 2021-08-28 01:56:56 -07:00
Chris Lu
05a648bb96 refactor: separating out remote.proto 2021-08-26 15:18:34 -07:00
yulai.li
c1dc5ab4ac Add deleterange_concurrency to filer configuration file 2021-08-26 18:25:08 +08:00
yulai.li
de8ef28460 Update filer.toml scaffold to support tikv store 2021-08-26 16:26:27 +08:00
yulai.li
546efeba8f Fix build bug 2021-08-26 16:20:35 +08:00
Chris Lu
6bab20d862 cat entry content 2021-08-24 02:38:32 -07:00
Chris Lu
e9ebe24f2e cloud drive: add support for Azure 2021-08-24 01:18:30 -07:00
Chris Lu
258063de26 cloud drive: add google cloud storage 2021-08-23 00:29:27 -07:00
Chris Lu
05fc7db755 filer.copy: large files pack file ids into chunks 2021-08-18 23:46:54 -07:00
Chris Lu
72eb6d5b9d ensure no writes to remote storage if content is not changed 2021-08-15 20:23:41 -07:00
Chris Lu
5a7c40510f format output 2021-08-15 20:07:13 -07:00
Chris Lu
5d5a21ba2d adjust log format 2021-08-15 19:46:45 -07:00
Chris Lu
c45c12aedc metadata updates 2021-08-15 19:27:30 -07:00
Chris Lu
49b5e47bd1 retry forever with filer.remote.sync, and some refactoring 2021-08-15 12:38:26 -07:00
Chris Lu
c34747c79d rename, fix wrong logic. 2021-08-14 21:46:34 -07:00
Chris Lu
0db2517994 go fmt 2021-08-14 02:55:44 -07:00
Chris Lu
7937db52e1 Filer locationPrefix configure does not exec replication #2257 
fix https://github.com/chrislusf/seaweedfs/issues/2257
 2021-08-14 02:54:13 -07:00
Chris Lu
5a0f92423e use grpc and jwt 2021-08-12 21:40:33 -07:00
Chris Lu
6238644c35 remove gateway command 2021-08-12 20:52:04 -07:00
Chris Lu
d1d1fc772c move some volume lookup operations to grpc 
jwt related lookup will come in next commit
 2021-08-12 20:33:00 -07:00
Chris Lu
5571f4f70a master: add master.follower to handle read file id lookup requests 2021-08-12 18:10:59 -07:00
Chris Lu
5469019852 adjust data type 2021-08-12 17:54:34 -07:00