Commit Graph

6506 Commits

Author SHA1 Message Date
Konstantin Lebedev
8de7915950 s3 mime detect 2021-10-14 15:03:11 +05:00
Chris Lu
5fd4b05c5e
Merge pull request #2381 from Juneezee/deprecate-ioutil
refactor: move from io/ioutil to io and os package
2021-10-13 22:38:58 -07:00
Chris Lu
46a09c6074 adjust test 2021-10-13 22:38:47 -07:00
Eng Zer Jun
a23bcbb7ec
refactor: move from io/ioutil to io and os package
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-14 12:27:58 +08:00
Chris Lu
4cbd390fbe test: add fail message 2021-10-13 20:42:20 -07:00
Chris Lu
b693a8d6b9
Merge pull request #2380 from kmlebedev/bucket_policy
Bucket policy
2021-10-13 15:29:53 -07:00
Konstantin Lebedev
9d6ffa0ea1 GetBucketLifecycleConfigurationHandler 2021-10-14 01:35:33 +05:00
Chris Lu
7700bf62f0
Merge pull request #2376 from kmlebedev/S3MultipartUploadPassMime 2021-10-12 08:54:59 -07:00
Konstantin Lebedev
458145425e S3 MultipartUpload pass contentType to meta 2021-10-12 17:14:54 +05:00
Chris Lu
de1ec050e8 fier/s3: save "Content-Disposition" to extended properties
related to https://github.com/chrislusf/seaweedfs/issues/2371
2021-10-12 00:04:28 -07:00
Chris Lu
34f764007f escape file name only when necessary 2021-10-11 23:33:45 -07:00
Chris Lu
8e9273db99 s3: use "response-content-disposition" to overwrite default content-disposition
fix one part of https://github.com/chrislusf/seaweedfs/issues/2371

see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html
2021-10-11 23:29:52 -07:00
Chris Lu
f0b928ff5e go fmt 2021-10-11 23:23:46 -07:00
Chris Lu
3afa451cdc volume: find a non-empty offset when binary searching by timestamp 2021-10-11 22:00:41 -07:00
Chris Lu
b530f12327 volume: find a non-empty offset when binary searching by timestamp
fix https://github.com/chrislusf/seaweedfs/issues/2364
2021-10-11 16:11:50 -07:00
Konstantin Lebedev
be4b3ed509 AclHandlers 2021-10-11 15:03:56 +05:00
Chris Lu
84d2e1bdd0 fix container building, make it same as rocksdb image 2021-10-11 01:29:05 -07:00
Chris Lu
e6ef7b2387 return error early
fix https://github.com/chrislusf/seaweedfs/issues/2370
2021-10-11 01:24:30 -07:00
Chris Lu
4800d0ce26 separate into multiple actions 2021-10-11 00:56:35 -07:00
Chris Lu
bde2fc65b0 separate into multiple jobs for better execution 2021-10-11 00:35:22 -07:00
Chris Lu
3d586be552 2.71 2021-10-10 22:40:44 -07:00
Chris Lu
f4676824a7 fix test code compilation 2021-10-10 22:14:13 -07:00
Chris Lu
8205166009 disable testing with redis server
--- FAIL: TestNameList (0.00s)
panic: exec: "redis-server": executable file not found in $PATH [recovered]
	panic: exec: "redis-server": executable file not found in $PATH

goroutine 37 [running]:
testing.tRunner.func1.2({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e
testing.tRunner.func1()
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218
panic({0xde2f80, 0xc0003da160})
	/opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215
github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59)
	/home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376
testing.tRunner(0xc0003e2680, 0xf57f48)
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102
created by testing.(*T).Run
	/opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a
2021-10-10 21:57:55 -07:00
Chris Lu
3e2acf677c removing tikv to resolve "go mod tidy" problem
tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back.

go mod tidy
go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker
go: finding module for package cloud.google.com/go/kms/apiv1
go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint
go: finding module for package google.golang.org/grpc/naming
go: finding module for package github.com/coreos/etcd/clientv3/credentials
go: finding module for package github.com/coreos/etcd/clientv3/balancer
go: finding module for package github.com/d4l3k/messagediff
go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible
go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible
go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0
go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1
go: finding module for package google.golang.org/grpc/naming
github.com/chrislusf/seaweedfs/weed/filer/tikv imports
	github.com/tikv/client-go/v2/tikv imports
	go.etcd.io/etcd/clientv3 tested by
	go.etcd.io/etcd/clientv3.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:27:02 -07:00
Chris Lu
bf218cd59d removing etcd sequencer
causing go mod tidy problem. If anyone wants this, please help to resolve this first.

github.com/chrislusf/seaweedfs/weed/sequence imports
	go.etcd.io/etcd/client tested by
	go.etcd.io/etcd/client.test imports
	github.com/coreos/etcd/integration imports
	github.com/coreos/etcd/proxy/grpcproxy imports
	google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming
2021-10-10 19:10:46 -07:00
Chris Lu
5ca0a551ac java: adjust cache expiration policy for long running java processes 2021-10-09 05:38:15 -07:00
Chris Lu
17e08a16f4 fix redis3 deletion 2021-10-09 04:54:48 -07:00
Chris Lu
ce1efeb0eb adds more error message 2021-10-09 04:54:14 -07:00
Chris Lu
e4830bd93d go fmt 2021-10-07 21:13:31 -07:00
Chris Lu
0a856241fe avoid int bigger than math.MaxInt32
fix https://github.com/chrislusf/seaweedfs/issues/2363
2021-10-07 21:12:57 -07:00
Chris Lu
d688e10ed1 do not try to compress if not sure about the file content 2021-10-07 13:29:00 -07:00
Chris Lu
332f5ad3a8 revert temporary changes that disabled compression
fix https://github.com/chrislusf/seaweedfs/issues/2362
2021-10-07 13:24:16 -07:00
Chris Lu
f3d8232e14 reduce one redis lookup on hot path 2021-10-06 22:01:19 -07:00
Chris Lu
03bb82043c Update SeaweedFS_Gateway_RemoteObjectStore.png 2021-10-06 20:43:02 -07:00
Chris Lu
2336a397dc use pipeline to save some time 2021-10-06 20:40:18 -07:00
Chris Lu
7584f758b8 Update SeaweedFS_Gateway_RemoteObjectStore.png 2021-10-06 20:37:37 -07:00
Chris Lu
ed456253d2 Update SeaweedFS_Gateway_RemoteObjectStore.png 2021-10-06 20:36:55 -07:00
Chris Lu
f14be41439 Update SeaweedFS_Gateway_RemoteObjectStore.png 2021-10-06 20:35:04 -07:00
Chris Lu
b08fac0014 Added SeaweedFS_Gateway_RemoteObjectStore.png 2021-10-06 20:28:08 -07:00
Chris Lu
371fead8a5 redis3 using redis native sorted set 2021-10-06 18:18:24 -07:00
Chris Lu
8668d49c9d test with real redis
cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
BenchmarkNameList-12    	    1789	    760599 ns/op
BenchmarkRedis-12       	   17539	     64122 ns/op
PASS
2021-10-06 01:25:37 -07:00
Chris Lu
6b31f3c97a add benchmark test 2021-10-06 00:37:57 -07:00
Chris Lu
893f0587b1 redis3 adds distributed locking 2021-10-06 00:03:54 -07:00
Chris Lu
f0d1e7bd05 skip ec volumes when loading normal volumes 2021-10-05 02:31:44 -07:00
Chris Lu
332d49432d reduce concurrent volume grow requests 2021-10-05 01:58:30 -07:00
Chris Lu
96119eab00 refactor 2021-10-05 00:40:04 -07:00
Chris Lu
8a66306064 calculate disk usage in case of race condition
related to https://github.com/chrislusf/seaweedfs/issues/2357
2021-10-04 23:32:07 -07:00
Chris Lu
4ed2994555 use tsMemory to determine whether read from disk or memory
remove lastFlushTime
2021-10-04 16:02:56 -07:00
Chris Lu
6a030547a2 server: remove peer check if not starting master
more fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 03:27:10 -07:00
Chris Lu
62c2732fd1 add back logic to check master peers
fix https://github.com/chrislusf/seaweedfs/issues/2352
2021-10-04 02:51:26 -07:00