mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-03 00:49:13 +08:00
Code Issues Packages Projects Releases Wiki Activity
8,513 Commits 32 Branches 296 Tags 149 MiB
9b006250b2
Commit Graph

5562 Commits

Author SHA1 Message Date
chrislu
2ea18cdcc8 remove dead code 2022-07-07 22:26:03 -07:00
chrislu
b7de5c6c43 shell: remove unused filer parameter from shell.toml 2022-07-07 15:01:23 -07:00
chrislu
f2f0482dd3 mount: random read also try to use the local cache first 2022-07-07 11:50:28 -07:00
chrislu
05c53820b9 mount: file handler release memory 2022-07-07 11:49:40 -07:00
chrislu
69ef6459c3 S3 ListObjectsV2 fix pagination under bucket root 
fix https://github.com/chrislusf/seaweedfs/issues/3166
 2022-07-06 02:28:34 -07:00
Konstantin Lebedev
785223e587 rabbitpubsub enable durable 2022-07-06 10:05:29 +05:00
chrislu
57975b93ff adjust used size reporting 2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5 adjust used size reporting 2022-07-04 13:46:32 -07:00
chrislu
3c79c77056 3.14 2022-07-04 13:15:49 -07:00
guol-fnst
5bc895673e add getting bucket quota 2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610 fix usage 2022-07-04 11:31:39 +08:00
chrislu
141f662734 edge case checking when volume server does not have capacity to balance 
fix https://github.com/chrislusf/seaweedfs/issues/3257
 2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798 fix tests 2022-07-01 12:09:42 -07:00
Chris Lu
ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255 
volumeServer.evacuate fix select empty node
 2022-07-01 02:36:30 -07:00
Konstantin Lebedev
3d5f1cbc5e fix pagination 2022-07-01 14:17:49 +05:00
chrislu
06a8b174b5 also remove Sync() for idx file 2022-06-30 13:50:53 -07:00
chrislu
52580743b9 stuck with file.Sync() 
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s

goroutine 61 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
	/usr/local/go/src/time/sleep.go:176 +0x32

goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
	/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
	/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
	/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
	_testmain.go:57 +0x1aa

goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf

goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
	/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
	/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
	/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
	/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
	/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
	/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
	/usr/local/go/src/testing/testing.go:1486 +0x35f

goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
	/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL	github.com/chrislusf/seaweedfs/weed/storage	600.194s
?   	github.com/chrislusf/seaweedfs/weed/storage/backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/erasure_coding	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/idx	[no test files]
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/needle_map	(cached)
ok  	github.com/chrislusf/seaweedfs/weed/storage/super_block	(cached)
?   	github.com/chrislusf/seaweedfs/weed/storage/types	[no test files]
?   	github.com/chrislusf/seaweedfs/weed/storage/volume_info	[no test files]
FAIL
 2022-06-30 12:32:55 -07:00
Konstantin Lebedev
5ed8165161 fix logic 
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
 2022-06-30 17:58:24 +05:00
石昌林
200a862701 fixed volume xx not found caused by missing VolumeLocation events 
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
 2022-06-30 13:41:56 +08:00
Chris Lu
4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta 
[Critical] Avoid synchronizing from scratch due to err
 2022-06-29 12:36:28 -07:00
Chris Lu
b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder 
use const multipart uploads folder
 2022-06-29 12:09:03 -07:00
石昌林
2b5b37231b Make sure that onChangeEvent is called successfully before updating lastTsNs 2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542 Avoid synchronizing from scratch due to err 2022-06-29 23:31:04 +08:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder 
avoid error bucket NotEmpty if multipart uploads folder exist
 2022-06-29 16:21:16 +05:00
chrislu
a8abab2412 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-29 02:44:14 -07:00
chrislu
449ecc8890 remove max connection age 
related to https://github.com/chrislusf/seaweedfs/pull/3226
 2022-06-29 02:44:12 -07:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy 
validate tags on copy object and add regex for validating tags
 2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535 remove println 2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2 fix return 2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c fix return 2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913 fix parseTagging function and add test for validate tags 2022-06-28 14:48:55 +03:00
Chris Lu
a79bf9d92a
Merge pull request #3252 from guol-fnst/update_tbk 
remove hdfs, add tikv for full version update
 2022-06-27 22:37:41 -07:00
guol-fnst
74c9f39d48 remove hdfs, add tikv for full version update 2022-06-28 12:35:07 +08:00
石昌林
e1b94eb6b9 fix error: concurrent map writes when add or remove cluster node 2022-06-28 10:34:59 +08:00
chrislu
4b1f48a399 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-27 12:40:50 -07:00
chrislu
4a65159250 fix reading time 2022-06-27 12:40:47 -07:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files 
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
 2022-06-27 20:19:21 +02:00
Chris Lu
4be406a095
Merge pull request #3249 from kmlebedev/issues_2589 
avoid set currentMaster svc.local k8s discovery service domains
 2022-06-27 11:05:07 -07:00
Konstantin Lebedev
9bff097b01 currentMaster redirected 2022-06-27 21:55:16 +05:00
Konstantin Lebedev
6c20a3b622 avoid set currentMaster k8s svc.local discoveruy service domains 
https://github.com/chrislusf/seaweedfs/issues/2589
 2022-06-27 21:47:05 +05:00
Lapshin Vitaliy
1a5981d583 fix test 2022-06-27 18:46:21 +03:00
garenchan
8aa19577f4 fix 3238: handle errors for GenerateDirUuid method 2022-06-27 22:04:50 +08:00
Lapshin Vitaliy
d7c3493d15 Merge branch 'master' into bufix/validate-tags-on-copy 2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a adjust log message 2022-06-26 23:12:16 -07:00
Chris Lu
fdd880b111
Merge pull request #3237 from shichanglin5/remove_query_collection 
Remove the collection query param of s3api and let the collection be …
 2022-06-26 23:02:11 -07:00
石昌林
15bfc60a88 Remove the collection query param of s3api and let the collection be determined by the filer 2022-06-27 13:47:53 +08:00
yulai.li
af23e63e3f Improve filer command help, add supported filer store list 2022-06-27 12:09:16 +08:00
chrislu
40a9634c3d 3.13 2022-06-26 20:13:52 -07:00
Chris Lu
cc01f71752
Merge pull request #3233 from blacktear23/filerstore-tikv 
Add TiKV Filerstore back
 2022-06-26 20:12:18 -07:00
yulai.li
824cbe32be Make tikv filer enable/disable by build tags 2022-06-27 10:57:24 +08:00
chrislu
9dfea44aeb fix unit 2022-06-26 12:21:38 -07:00
chrislu
3142e77eca mount: stats report physical size instead of logical size 2022-06-26 12:14:34 -07:00
yulai.li
13159ad0d8 Merge branch 'master' into filerstore-tikv 2022-06-26 22:44:01 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
chrislu
f77eda4b4f Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-25 09:22:52 -07:00
chrislu
3ce46c3566 fix entry full url lookup 2022-06-25 09:22:49 -07:00
Chris Lu
dc59ccd110
Merge pull request #3228 from shichanglin5/fix_volumeNotFound 
When the connection with the leader is disconnected, the vidMap shoul…
 2022-06-24 23:06:13 -07:00
石昌林
f1ea906c13 Code comment optimization 2022-06-25 11:56:09 +08:00
石昌林
0c8e033e68 When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader 2022-06-24 17:41:46 +08:00
chrislu
9c517d2b35 masterclient: fallback to directly querying master in case of missing volume id location 2022-06-24 02:08:57 -07:00
Chris Lu
280e33092c
Merge pull request #3226 from kmlebedev/advancedtls 
Advancedtls
 2022-06-24 01:34:58 -07:00
chrislu
509a9047db test compact map with snowflake sequencer 2022-06-23 21:45:51 -07:00
chrislu
217d7dc377 verify seq is always unique 2022-06-23 21:22:05 -07:00
Konstantin Lebedev
2e782dfdcb rm defer comments 2022-06-24 01:50:13 +05:00
Konstantin Lebedev
7efaafb0ee fix Authenticate 2022-06-24 01:42:04 +05:00
Konstantin Lebedev
b0aa51d7ef enable require client cert 2022-06-24 00:29:23 +05:00
Konstantin Lebedev
ea7cdb8b0e seperate option 2022-06-24 00:26:56 +05:00
Konstantin Lebedev
5b388ed6c1 initial advancedtls 2022-06-23 23:32:15 +05:00
chrislu
0c13fc873c fix compilation 2022-06-23 00:45:11 -07:00
chrislu
96496d5286 master: broadcast new volume locations to clients to avoid possible racing condition 
fix https://github.com/chrislusf/seaweedfs/issues/3220
 2022-06-23 00:41:33 -07:00
chrislu
52c44d646e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-22 00:34:27 -07:00
chrislu
530da0cc89 remove unused code 2022-06-22 00:34:25 -07:00
famosss
8a6e7a7df8
Merge branch 'chrislusf:master' into master 2022-06-22 09:58:56 +08:00
zzq09494
9df5ad5309 fix: vacuum create a lot of connections quickly 2022-06-22 09:57:22 +08:00
Chris Lu
d652ce1a36
Merge pull request #3216 from kmlebedev/mk_raft_basedir 
avoid no such raft date directory
 2022-06-21 12:21:30 -07:00
chrislu
6b3beb709d fix format 2022-06-21 12:21:02 -07:00
Konstantin Lebedev
c1c8dad677 avoid no such raft date directory 
https://github.com/chrislusf/seaweedfs/issues/3214
 2022-06-21 13:47:51 +05:00
chrislu
4dc27e1ed5 3.12 2022-06-20 19:07:00 -07:00
chrislu
492da3dbce master: put metadata under instance specific folder 2022-06-20 19:04:49 -07:00
chrislu
41dfe27102 Revert "remove max connection age" 
This reverts commit b9b684194f.
 2022-06-20 14:49:39 -07:00
chrislu
2295766e2c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page 2022-06-20 22:36:07 +02:00
chrislu
bf5a5abfb1 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-20 13:18:11 -07:00
chrislu
69a2e503ab non-recursive directory deletion 
fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.
 2022-06-20 13:18:08 -07:00
chrislu
76f26c4b99 mount: delete target entry first during move 2022-06-20 13:15:02 -07:00
Chris Lu
460c7cf68e
Merge pull request #3203 from shichanglin5/volume_copy_preallocate 2022-06-20 07:41:45 -07:00
石昌林
81f7f08708 Determine whether to preallocate according to the master configuration before executing copy volume 2022-06-20 21:12:44 +08:00
石昌林
be5c901bea Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems 2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48 validate tags on copy object and add regex for validating tags 2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode 
AbortMultipartUploadHandler should return 204 instead of 200
 2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker 2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0 AbortMultipartUploadHandler should return 204 instead of 200 
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
 2022-06-20 13:26:22 +08:00
石昌林
9e036df356 remove go.uber.org/atomic 2022-06-20 12:35:29 +08:00
geekboood
fdacd94af5 fix: invalid chunk data when failed to read manifests 2022-06-19 16:54:04 +08:00
石昌林
3dd60529c5 some code optimizations 2022-06-17 19:07:39 +08:00
石昌林
37df209195 add some unit tests and some code optimizes 2022-06-17 17:11:18 +08:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update 
remove "weed" from UsageLine, or weed will not show usage of weed update
 2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5 remove "weed" from UsageLine, or weed will not show usage of weed update 2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e reduce upfront memory usage for low density volume 2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update 
Weed update
 2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4 fix path bug on windows 
fix usage
 2022-06-16 17:24:56 +08:00
chrislu
aaa9938353 Revert "adjust conditions" 
This reverts commit e024586ff1.
 2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97 remove -output, add -dir,-name for more flexibility 2022-06-16 16:07:32 +08:00
chrislu
e024586ff1 adjust conditions 2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix 
exclude replication from the concurrentUploadLimitMB
 2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e add condition when inFlightUploadDataLimitCond signal 2022-06-16 14:07:11 +08:00
zzq09494
62f74f5e3c add bucket label to s3 prometheus metrics 2022-06-16 13:21:25 +08:00
guol-fnst
4bc8792643 support specific version of weed 2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29 remove unused function 
remove log when weed is up to date

add check whether weed is full version
 2022-06-16 11:40:20 +08:00
liubaojiang
3076ac101e move vs.concurrentUploadLimit != 0 out of the lock 2022-06-16 09:58:44 +08:00
Chris Lu
097a12dfb4
Merge pull request #3187 from ningfdx/remote 2022-06-15 06:34:32 -07:00
Chris Lu
55b9f281d1
Merge pull request #3185 from kmlebedev/logging_err_process_range_req 2022-06-15 06:33:33 -07:00
石昌林
78b3728169 add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations 
configure s3 circuit breaker by 'command_s3_circuitbreaker.go':
usage eg:
# Configure the number of simultaneous global (current s3api node) requests
s3.circuit.breaker -global -type count -actions Write -values 1000 -apply

# Configure the number of simultaneous requests for bucket x read and write
s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply

# Configure the total bytes of simultaneous requests for bucket write
s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply

# Disable circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -enable false -apply

# Delete circuit breaker config of bucket 'x'
s3.circuit.breaker -buckets x -delete -apply
 2022-06-15 21:07:55 +08:00
ningfd
338705f375 fix(wdclient): GetLocations return 2022-06-15 19:20:13 +08:00
Konstantin Lebedev
31a929f7f2 logging processRangeRequest errors 2022-06-15 15:36:43 +05:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37 Fix: provide filer.ui.deleteDir options for server command 2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update 2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555 fix minor bug 2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6 fix(filer.sync): modify clientName format : from -> to 2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87 Change ui.deleteDir default to true 2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b feat(filer.sync): add metricsServer in filer.sync. 
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
 2022-06-15 11:33:18 +08:00
chrislu
d4ef06cdcf 3.11 2022-06-14 12:14:52 -07:00
chrislu
b347b2fb54 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-14 11:16:51 -07:00
chrislu
36d2756978 volume: avoid writing too much data for range requests in large files 
related to https://github.com/chrislusf/seaweedfs/issues/3178
 2022-06-14 11:16:47 -07:00
yulai.li
28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea feat(filer.sync): add offset to path. 2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0 add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate" 2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372 add update feature 2022-06-14 19:01:57 +08:00
Chris Lu
b22ca85fbb
Merge pull request #3169 from shichanglin5/fix_auth 
fix: When there is no access permission configured before startup, th…
 2022-06-12 23:40:42 -07:00
shichanglin5
f01dd27752 fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup 2022-06-13 13:23:26 +08:00
chrislu
e437ec5a68 3.10 2022-06-12 19:30:27 -07:00
chrislu
d12f431d98 collect volume server status 2022-06-12 11:56:23 -07:00
chrislu
444ac21050 go fmt 2022-06-11 09:51:11 -07:00
chrislu
a74e926063 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-11 09:51:02 -07:00
chrislu
61b8c9c361 remote object store gateway: disable tagging for backblaze 2022-06-11 09:50:59 -07:00
Chris Lu
0a8b3948db
Merge pull request #3165 from blacktear23/reload-bug-fix 2022-06-10 07:47:43 -07:00
yulai.li
83f8cdd5cb Bug Fix: force reload filer page when upload/delete/rename/create directory 2022-06-10 22:39:21 +08:00
guol-fnst
eabf4d4e64 fix typo 2022-06-10 22:03:11 +08:00
Chris Lu
36757d28a8
Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch 
stats master_replica_placement_mismatch
 2022-06-10 06:44:16 -07:00
chrislu
9f8b72a54d Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" 
This reverts commit 37da689319, reversing
changes made to 00d53c34c4.
 2022-06-10 06:38:17 -07:00
Konstantin Lebedev
fb57aa431c stats master_replica_placement_mismatch 2022-06-10 15:30:40 +05:00
shichanglin5
a63559ee6c fix: fix command prompt information 2022-06-10 12:46:10 +08:00
Chris Lu
37da689319
Merge pull request #3159 from shichanglin5/_duplicateUUID 
perf: Optimized volume handling duplicateUUID logic to avoid quitting…
 2022-06-09 18:01:02 -07:00
shichanglin5
f5b0c04b14 perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal 
Under normal circumstances, there will be no problems, but when the
master is debugged in the local environment, the volume client cannot
communicate with the master normally, so the sendHeartBeat logic is
restarted, and a new connection is created to report the heartbeat. If
the master has not cleared the uuid of the volume at this time, then The
master will respond to volume duplicateUUIDS, and the volume service
will exit, but in fact the uuid of the volume is not duplicated
 2022-06-09 20:41:16 +08:00
zhihao.qu
cd5cca36a4 feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges 2022-06-09 10:53:19 +08:00
Chris Lu
4a5135961f
Merge pull request #3154 from kmlebedev/s3_test_invalid_range 2022-06-08 01:45:40 -07:00
Konstantin Lebedev
49f058da85 fix s3 tests: 
ranged_request_invalid_range
ranged_request_empty_object
 2022-06-08 13:25:13 +05:00
chrislu
8a1166c021 auto remove previous leftover socket 2022-06-08 00:32:09 -07:00
chrislu
633a4aed18 filer: fix customizable local socket file name 
fix https://github.com/chrislusf/seaweedfs/issues/3147
 2022-06-08 00:15:28 -07:00
chrislu
acbc0286e2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-07 22:33:38 -07:00
chrislu
ebd6097e6f fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 
related to https://github.com/chrislusf/seaweedfs/pull/3074
 2022-06-07 22:33:35 -07:00
Chris Lu
f43ec9f363
Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud 2022-06-07 08:56:55 -07:00
Konstantin Lebedev
c07820178f fix s3 tests 
bucket_list_delimiter_prefix
bucket_list_delimiter_prefix_underscore
bucket_list_delimiter_prefix_ends_with_delimiter
 2022-06-07 14:43:10 +05:00
chrislu
78be3b0f6c fix compilation 2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata 
these metadata can change and are not used
 2022-06-06 00:39:35 -07:00
chrislu
cbf46de5f4 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-06 00:15:40 -07:00
chrislu
8402af0882 filer: detect ttl based on fs configure setting 2022-06-06 00:15:37 -07:00
ningfd
f32142f6f5 add disableXAttr in mount option 2022-06-06 14:09:01 +08:00
chrislu
4a046e4de7 3.09 2022-06-05 19:00:24 -07:00
chrislu
11f1e7996b fix test 2022-06-05 18:50:41 -07:00
chrislu
d65bb2c6df mount: file handle locks entry better 
related to https://github.com/chrislusf/seaweedfs/issues/2952
 2022-06-05 18:15:06 -07:00
chrislu
746092a60b add a comment 2022-06-05 16:44:07 -07:00
chrislu
056c480eb0 volume: checksum remove the hashing step in Value() 2022-06-05 15:24:02 -07:00
chrislu
ecef844dfc stream read large files 2022-06-05 11:54:04 -07:00
chrislu
48cca4e54f remove unused logic 2022-06-04 19:11:06 -07:00
chrislu
685643d506 needle read into a writer 2022-06-04 19:10:52 -07:00
chrislu
663bc5dc23 add needle reading in chunks 2022-06-04 18:15:39 -07:00
chrislu
18b29f709c refactoring, split file 2022-06-03 23:31:14 -07:00
chrislu
39c3cb04af Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-03 08:45:32 -07:00
chrislu
f97acdd489 volume.fix.replication fix retry logic 
fix https://github.com/chrislusf/seaweedfs/issues/3136
 2022-06-03 08:45:29 -07:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
Konstantin Lebedev
b86628f85d fix show empty dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-06-01 20:10:52 +05:00
chrislu
6793bc853c help message when in simulation mode 2022-05-31 14:48:46 -07:00
chrislu
89948a373b fix error reporting for "Need to a hdd disk type" 
related to https://github.com/chrislusf/seaweedfs/issues/3128
 2022-05-31 12:43:55 -07:00
chrislu
8a49240d64 3.08 2022-05-31 11:57:41 -07:00
chrislu
376b005ad3 fix wrong logic 2022-05-31 11:55:58 -07:00
chrislu
fbd99d53c1 3.07 2022-05-31 00:36:24 -07:00
chrislu
27732ecfa4 move s3 related constants from package http to s3_constants 2022-05-30 22:57:41 -07:00
chrislu
f4a6da6cb2 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 22:47:29 -07:00
chrislu
596c3860ca use final destination to resolve fs configuration 
related to https://github.com/chrislusf/seaweedfs/issues/3075
 2022-05-30 22:47:26 -07:00
Chris Lu
ca01ce0524
Merge pull request #3122 from chrislusf/filer-sync-with-peers 
Filer bootstrap from peers
 2022-05-30 21:38:31 -07:00
chrislu
6adc42147f fresh filer store bootstrap from the oldest peer 2022-05-30 21:27:48 -07:00
chrislu
490e0850bf filer.meta.tail add one example of using untilTimeAgo 2022-05-30 20:24:59 -07:00
liubaojiang
f0ee3e6f21 reduce the scope of inFlightUploadDataLimitCond lock 2022-05-31 09:49:07 +08:00
chrislu
c59068d0f3 refactor 2022-05-30 16:28:36 -07:00
chrislu
682382648e collect cluster node start time 2022-05-30 16:23:52 -07:00
chrislu
968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu
f214dfb1f5 stop when in memory log is done 2022-05-30 15:25:21 -07:00
chrislu
aece35a64f stop when on disk log is done 2022-05-30 15:20:51 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
chrislu
8902fa6ff6 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-30 13:28:32 -07:00
Chris Lu
730c9cf601
Merge pull request #3114 from kmlebedev/nextcloud 
Create folders in s3 via nextcloud
 2022-05-30 13:10:10 -07:00
chrislu
5b9347c938 typo 2022-05-30 12:37:41 -07:00
Konstantin Lebedev
af562e1a20 rm log info 2022-05-30 23:26:26 +05:00
Konstantin Lebedev
62e5e3822c list self dir 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 18:09:08 +05:00
Konstantin Lebedev
26f3646961 show fake dirs 
https://github.com/chrislusf/seaweedfs/issues/3086
 2022-05-30 15:03:27 +05:00
chrislu
b8f3db0d46 s3: keep auth enabled in case identities are set to empty 
fix https://github.com/chrislusf/seaweedfs/issues/3084
 2022-05-29 19:00:03 -07:00
chrislu
81d6159290 volume: report error if a volume has nil data backend 
fix https://github.com/chrislusf/seaweedfs/issues/3105
 2022-05-29 16:59:30 -07:00
chrislu
ca16fbf0ef shell: cluster.ps display filer group 2022-05-29 16:37:14 -07:00
chrislu
9219664505 reset sync offset if peer filer resets 
fix https://github.com/chrislusf/seaweedfs/issues/3089
 2022-05-29 13:11:24 -07:00
Konstantin Lebedev
54356211c2 list empty folder 2022-05-27 13:17:18 +05:00
Kaiwalya Joshi
4598581741
fix: Minor spelling fixes on FilerStorePathTranslator. 2022-05-25 01:11:02 -07:00
chrislu
5b8b022985 remove unused parameter 2022-05-24 18:52:04 -07:00
Robert Coelho
0e6e72d462 mount: ReadDir return EIO on EnsureVisited err 2022-05-24 13:02:59 -04:00
Robert Coelho
1fabbe8a25 mount: cleanup ReadDir rewrite's branches to not assume offsets 2022-05-24 12:48:32 -04:00
Robert Coelho
cb422d96f7 mount: rewrite ReadDir to respect input.Offset to fix partial results 2022-05-24 12:16:58 -04:00
Robert Coelho
05602167f4 mount: do not mark meta_cache when there is an err 2022-05-24 12:13:15 -04:00
chrislu
7e25a2d416 reduce busy waiting when reading metadata logs 2022-05-24 00:23:53 -07:00
chrislu
2f846777bb 3.06 2022-05-23 01:21:48 -07:00
chrislu
a5cdcc7e63 fix reading files larger than 20GB due to manifest resolving bug 2022-05-23 01:16:10 -07:00
chrislu
59cc3c0bfd fix reading files larger than 20GB due to manifest resolving bug 
this is introduced since Mar 08 2022
 2022-05-23 01:14:56 -07:00
chrislu
acc318e12b 3.05 2022-05-22 21:43:59 -07:00
liubaojiang
076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
liubaojiang
71b2e6223e add inFlightUploadDataLimitCond signal 2022-05-20 15:27:06 +08:00
liubaojiang
1a41691b4c exclude replication from the concurrentUploadLimitMB 2022-05-20 14:40:05 +08:00
Chris Lu
cb64983c6b
Merge pull request #3079 from ztinpn/hotfix_fsync 2022-05-19 10:33:42 -07:00
tianzhang
66747ee9c9 hotfix_fsync fix fsync 2022-05-19 23:17:17 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol 
avoid loading duplicated volume directory
 2022-05-17 23:48:57 -07:00
guol-fnst
8e2426a879 revert changes of pb file 
orignal protoc-gen-go-grpc verion is 1.1.0
 2022-05-18 11:52:14 +08:00
guol-fnst
b12944f9c6 fix naming convention 
notify volume server of duplicate directoris
improve searching efficiency
 2022-05-17 15:41:49 +08:00
guol-fnst
076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
guol-fnst
8fab39e775 rename UUID file 
fix typo
move locationUUID  into DiskLocation
 2022-05-17 15:41:47 +08:00
Chris Lu
72e7dcde51
Merge pull request #3060 from natmaka/master 2022-05-16 07:42:58 -07:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around 2022-05-16 22:11:33 +08:00
guol-fnst
de6aa9cce8 avoid duplicated volume directory 2022-05-16 19:33:51 +08:00
Konstantin Lebedev
44f53ceda6 fix collectionIsMismatch charset 2022-05-16 13:23:23 +05:00
Konstantin Lebedev
10d435f2c2 fix skip loop 2022-05-16 13:16:27 +05:00
Konstantin Lebedev
279053572c avoid delete volume replica if collection mismatch 2022-05-16 13:07:05 +05:00
chrislu
9ff0d99002 3.04 2022-05-15 21:32:21 -07:00
chrislu
866fed1eab avoid grpc name conflict 
fix https://github.com/chrislusf/seaweedfs/issues/3055
 2022-05-15 21:31:29 -07:00
chrislu
cd2d10118c 3.03 2022-05-15 18:17:07 -07:00
chrislu
271b5aed96 shell: volume.fsck add a note for -reallyDeleteFromVolume option 2022-05-15 11:07:04 -07:00
chrislu
4076d74813 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-15 00:43:39 -07:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
Chris Lu
ea4918394c
Merge pull request #3047 from shichanglin5/master 
fix the problem of metadata and tagging loss when files are copied
 2022-05-13 22:52:48 -07:00
shichanglin5
688d55488c test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go 2022-05-14 10:40:29 +08:00
chrislu
735038b2c1 backup do not need to use preallocation 
fix https://github.com/chrislusf/seaweedfs/issues/3044
 2022-05-13 13:46:52 -07:00
shichanglin5
1166dead00 fix the problem of metadata and tagging loss when files are copied 
by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)
 2022-05-13 19:46:20 +08:00
Konstantin Lebedev
d12a423aa0 refactor 
add some tests

https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-12 12:23:02 +05:00
Konstantin Lebedev
f523aed3c9 fixes from comments 
https://github.com/chrislusf/seaweedfs/pull/2996
 2022-05-11 21:50:09 +05:00
chrislu
bff1ccc1de fix compilation 2022-05-11 00:52:15 -07:00
chrislu
76da688176 adjust log level 2022-05-11 00:45:19 -07:00
chrislu
3e20336bde fix atomic add int64 
fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831
 2022-05-11 00:44:35 -07:00
chrislu
a85720f0f6 remove deprecated function 2022-05-10 00:02:23 -07:00
chrislu
25559eeb92 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-05-10 00:01:46 -07:00
chrislu
791c29ae93 remove deprecated function 
WithQueryCachePolicyKeepInCache is deprecated and already default behavior
 2022-05-10 00:01:43 -07:00
qiang.zeng
06cb337b82 fix typo 2022-05-09 14:21:16 +08:00
chrislu
a4ca3ed1f0 3.02 2022-05-08 22:28:32 -07:00