chrislu
b64fc44e8d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-18 09:01:17 -07:00
chrislu
340aea9b34
add broker leader
2023-09-18 09:01:14 -07:00
chrislu
0d390ec882
fix nil
2023-09-18 08:52:35 -07:00
Lars Lehtonen
d83107efe5
fix: weed/storage: dropped error
2023-09-18 08:18:57 -07:00
zemul
91cbaef02b
fix: mount proxyByFiler write error: wrong jwt
2023-09-18 07:44:50 -07:00
chrislu
4e17d399a6
refactor
2023-09-17 00:04:20 -07:00
chrislu
7a49d3541a
avoid nil
2023-09-17 00:04:13 -07:00
chrislu
a65088d739
broker report stats to balancer
2023-09-16 23:00:51 -07:00
chrislu
a1cb49d078
adjust log
2023-09-16 23:00:28 -07:00
chrislu
b590a68e75
revert to non-streaming mode to assign file id
...
fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685
2023-09-16 16:13:08 -07:00
chrislu
4650c4c65f
Revert "turn on streaming assign file id"
...
This reverts commit 733db2bc88
2023-09-16 16:09:20 -07:00
chrislu
fa62926347
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-16 15:06:22 -07:00
chrislu
89a1fd1751
Squashed commit of the following:
...
commit 48274251463b50139f687f685ce7ba436d99443bb771fefa370a851ec008908810370a851ec00b39941edc0b875f562779984b6c54cf2492a45499ba67e6ca299e4f985698cb470d44df1eb2da46d5504ae8383adbcba75271c9caf331194d6c18d86f4eb8e8624d1990456670905911853d
2023-09-16 15:06:16 -07:00
chrislu
4827425146
balancer works
2023-09-16 15:05:38 -07:00
chrislu
3b50139f68
comments
2023-09-15 22:22:32 -07:00
chrislu
7f685ce7ba
adjust APIs
2023-09-15 22:20:05 -07:00
chrislu
436d99443b
receive broker stats
2023-09-14 23:49:05 -07:00
wang wusong
9e5b1eef9b
avoid passe lock by value
2023-09-14 06:51:12 -07:00
chrislu
b771fefa37
Merge branch 'master' into sub
2023-09-13 00:03:47 -07:00
annluda
8908810376
Fix s3 api object list with params max-keys and prefix
2023-09-11 21:35:15 -07:00
chrislu
5407b86766
3.56
2023-09-10 22:05:41 -07:00
chrislu
07c2d75ac7
ensure single threaded access
2023-09-10 22:03:44 -07:00
chrislu
0a851ec00b
Create balancer.go
2023-09-10 22:01:25 -07:00
chrislu
39941edc0b
add publisher shutdown
2023-09-07 23:55:19 -07:00
chrislu
875f562779
server side send response at least once per second
2023-09-06 23:16:41 -07:00
chrislu
984b6c54cf
ack interval 128
2023-09-06 23:15:29 -07:00
chrislu
2492a45499
ack interval
2023-09-06 22:39:46 -07:00
Lars Lehtonen
f24c7e803f
weed/storage/erasure_coding: Close() after error handling
2023-09-06 07:35:54 -07:00
Lars Lehtonen
d762b12e11
weed/storage/erasure_coding: fix dropped test error
2023-09-06 07:35:54 -07:00
Lars Lehtonen
a879e1bbb5
weed/storage/erasure_coding: remove unused err from encodeDatFile() signature
2023-09-06 07:35:54 -07:00
Patrick Schmidt
98dcec0ee2
Clean up old signature hash pools
2023-09-05 10:33:27 -07:00
Patrick Schmidt
cdd817edf9
Improve S3 request signing performance
...
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
2023-09-05 10:33:27 -07:00
chrislu
ba67e6ca29
api for sub
2023-09-04 21:43:50 -07:00
chrislu
9e4f985698
publish, benchmark
2023-09-04 21:43:30 -07:00
Lars Lehtonen
e6385e5e8b
weed/util: fix dropped errors ( #4803 )
2023-09-01 13:10:45 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5
connect and publish
2023-08-28 09:02:12 -07:00
chrislu
504ae8383a
protoc version
2023-08-28 09:01:25 -07:00
chrislu
dbcba75271
rename to lookup
2023-08-27 18:59:04 -07:00
chrislu
c9caf33119
move functions
2023-08-27 18:33:46 -07:00
chrislu
4d6c18d86f
pub sub initial tests
2023-08-27 17:50:59 -07:00
chrislu
4eb8e8624d
rename
2023-08-27 13:14:39 -07:00
chrislu
1990456670
sub
2023-08-27 13:13:14 -07:00
chrislu
905911853d
adjust proto
2023-08-26 13:39:21 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer ( #4767 )
2023-08-24 07:08:56 -07:00
chrislu
733db2bc88
turn on streaming assign file id
2023-08-23 00:34:21 -07:00
chrislu
99f037b958
streaming assign file ids
2023-08-23 00:31:33 -07:00
chrislu
94b7e2a37c
add stream assign server side implementation
2023-08-22 09:59:04 -07:00
chrislu
ccedad5196
refactor files
2023-08-22 09:54:06 -07:00
chrislu
010c5e91e3
add stream assign proto
2023-08-22 09:53:54 -07:00
chrislu
a1149f060b
upgrade jwt
2023-08-22 08:26:31 -07:00
chrislu
c49caff5a1
change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims
2023-08-21 23:42:39 -07:00
chrislu
c45665eb97
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-08-21 09:47:36 -07:00
SmsS4
f61490966f
Add time to first byte metric for s3 ( #4768 )
...
* Add time to first byte metric for s3
* Change second to millisecond
2023-08-21 00:42:39 -07:00
chrislu
01d70c21f3
Squashed commit of the following:
...
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
2023-08-20 22:53:05 -07:00
chrislu
16e3f2d528
fix log buffer test
2023-08-20 12:46:15 -07:00
chrislu
3e9c32a3f0
refactoring to use atomic bool
...
fix compilation
2023-08-20 12:13:06 -07:00
SmsS4
1e72579ac7
Fix s3.clean.uploads ( #4766 )
2023-08-19 06:30:18 -07:00
chrislu
6792db21dc
ErrInvalidPart when completed parts has extra parts
...
fix https://github.com/seaweedfs/seaweedfs/discussions/4746
2023-08-17 00:51:42 -07:00
chrislu
f5c4b642bb
re-ordering
2023-08-16 23:54:23 -07:00
chrislu
3852307e94
renaming
2023-08-16 23:47:43 -07:00
chrislu
6c7fa567d4
add separate cache directory for write buffers
2023-08-16 23:39:21 -07:00
SmsS4
ea1b259c12
Update weed fix description ( #4745 )
2023-08-13 03:50:53 -07:00
Nikita Mochalov
4b1ba7f5b2
Save disk space metrics immediately ( #4740 )
2023-08-10 06:52:32 -07:00
Nikita Mochalov
e6a49dc533
Fix resource leaks ( #4737 )
...
* Fix division by zero
* Fix file handle leak
* Fix file handle leak
* Fix file handle leak
* Fix goroutine leak
2023-08-09 15:30:36 -07:00
chrislu
3365468d0d
added an error message
2023-08-08 20:35:21 -07:00
wusong
3ec977cd6c
minor ( #4723 )
2023-08-01 02:34:59 -07:00
chrislu
c09100f928
fix building mongo index
2023-07-24 11:09:51 -07:00
chrislu
7eafa3420b
3.55
2023-07-23 23:11:37 -07:00
SmoothDenis
51bcc219ea
s3api should return 500 code from filer ( #4699 )
2023-07-22 07:22:38 -07:00
wusong
33b9192536
Fix large file chunks deleted by mistake ( #4678 )
2023-07-17 07:52:14 -07:00
chrislu
358b3a4894
3.54
2023-07-16 22:47:57 -07:00
chrislu
6f588b5b18
fix refactoring mistake
...
fix https://github.com/seaweedfs/seaweedfs/issues/4639
2023-07-16 11:22:48 -07:00
mervynzhang
df400e6c71
Concurrency works better ( #4663 )
...
Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>
2023-07-12 23:04:54 -07:00
wusong
0cb9ddd8ec
Fix data loss: add lock for metacache ( #4664 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-07-11 22:23:32 -07:00
Henco Appel
89542db2d0
Fix S3 API multipart upload on Windows ( #4666 )
...
The file paths were being having '/' changed to '\' by filepath.Dir()
resulting in a file being created with '\' separators, but when trying
to read the files, the same wasn't happening.
Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2023-07-11 09:05:14 -07:00
Henco Appel
680c2e76f1
Fix the segfault from not setting s3 https port ( #4667 )
2023-07-11 08:03:20 -07:00
Konstantin Lebedev
2c076ebd96
avoid logging nil error ( #4659 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-07-10 23:38:52 -07:00
LHHDZ
3730772e05
refresh max volume count after adjust ( #4662 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-07-10 23:37:33 -07:00
chrislu
c278bac263
avoid nil needle map
...
fix https://github.com/seaweedfs/seaweedfs/issues/4640
2023-07-07 22:16:58 -07:00
chrislu
0606b59a3e
mount: create mount root on filer
...
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/127
2023-07-07 09:30:32 -07:00
chrislu
14546f0ad8
minor
2023-07-06 09:12:28 -07:00
chrislu
ab30f489c3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-07-06 00:33:02 -07:00
chrislu
31b2751aff
clone volume locations in case they are changed
...
fix https://github.com/seaweedfs/seaweedfs/issues/4642
2023-07-06 00:32:58 -07:00
wusong
61553beba2
Fix DataBackend nil pointer ( #4641 )
2023-07-04 23:22:10 -07:00
chrislu
a315490f7d
proxy to master uses http address
...
fix https://github.com/seaweedfs/seaweedfs/issues/4607
2023-07-04 11:45:21 -07:00
chrislu
2aa59ab37c
fix copying level db files during commi
...
fix https://github.com/seaweedfs/seaweedfs/issues/4635
2023-07-04 11:28:12 -07:00
wusong
c44a6552d2
Fix buff use ( #4628 )
2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr
f0d3c874b6
Unix Socket listener for S3 server ( #4621 )
...
Add localSocket option to s3 server
2023-06-26 16:22:45 -07:00
chrislu
2c4c2f0994
3.53
2023-06-25 22:51:14 -07:00
chrislu
f193942adc
lock on append metadata operation
2023-06-25 22:07:23 -07:00
chrislu
61c42f9991
adjust lock APIs
2023-06-25 20:30:20 -07:00
chrislu
aafb874ad2
lock until cancelled
2023-06-25 19:31:25 -07:00
chrislu
c4f44d56f7
add distributed lock client
2023-06-25 17:38:34 -07:00
chrislu
f8aa5ea844
adjust filer.proto
2023-06-25 17:38:19 -07:00
chrislu
13a8593ab9
adjust errors
2023-06-25 17:37:54 -07:00
chrislu
06471dac9d
init lock ring
2023-06-25 15:28:16 -07:00
chrislu
5149b3d07b
filer can proxy to peer filer holding the lock
2023-06-25 15:28:01 -07:00
chrislu
868f7875d7
refactor
2023-06-25 14:30:58 -07:00
chrislu
26eff062c8
refactor
2023-06-25 14:17:58 -07:00
chrislu
464a71a373
add distributed lock manager
2023-06-25 14:14:40 -07:00
chrislu
ee4f7cd636
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-06-25 00:58:24 -07:00
chrislu
3fd659df2a
add distributed lock manager
2023-06-25 00:58:21 -07:00
Damiano Albani
0d9ba8c612
Support templating name of files stored via Rclone backend ( #4606 )
2023-06-24 07:08:24 -07:00
chrislu
a82c44972b
refactor
2023-06-20 00:12:59 -07:00
chrislu
14a19170e2
refactor
2023-06-19 23:51:04 -07:00
chrislu
84ced2abf5
add LockRing
2023-06-19 23:48:33 -07:00
chrislu
fe1716ccdf
remove deprecated tests
2023-06-19 18:43:51 -07:00
chrislu
1ce725728d
minor
2023-06-19 18:43:41 -07:00
chrislu
11549706ef
refactor
2023-06-19 18:21:07 -07:00
chrislu
8ec1bc2c99
remove unused cluster node leader
2023-06-19 18:19:13 -07:00
Bai Jie
3b88ab42aa
remove duplicate fileCount query ( #4588 )
2023-06-18 00:14:14 -07:00
Bai Jie
44b9d72ef0
doIsEmpty() return error if v.DataBackend is nil ( #4587 )
2023-06-18 00:13:40 -07:00
chrislu
94fbf02ea2
generate jwt token when looking up needle id
...
fix https://github.com/seaweedfs/seaweedfs/issues/4577
2023-06-17 00:11:23 -07:00
柏杰
0b0fb9b9e4
avoid data race read volume.IsEmpty ( #4574 )
...
* avoid data race read volume.IsEmpty
- avoid phantom read isEmpty for onlyEmpty
- use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope
* add Destroy(onlyEmpty: true) test
* add Destroy(onlyEmpty: false) test
* remove unused `IsEmpty()`
* change literal `8` to `SuperBlockSize`
2023-06-14 14:39:58 -07:00
Konstantin Lebedev
1e22d5caf2
fix get file stats for IsEmpty ( #4576 )
2023-06-14 01:43:30 -07:00
Konstantin Lebedev
4dd890d4a2
optional https port for s3 ( #4482 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-13 23:58:49 -07:00
chrislu
3fbf4f6189
Revert "Initial patch for redis version 7 support ( #4572 )"
...
This reverts commit a25bca0692
2023-06-12 22:27:38 -07:00
Konstantin Lebedev
4527ead295
fix from comment delete volume is empty ( #4573 )
...
* fix from coments
https://github.com/seaweedfs/seaweedfs/pull/4561
* fix tests
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 22:22:46 -07:00
Aaron Gipson
a25bca0692
Initial patch for redis version 7 support ( #4572 )
2023-06-12 13:30:03 -07:00
Konstantin Lebedev
25535e9c36
Delete volume is empty ( #4561 )
...
* use onlyEmpty for deleteVolume
https://github.com/seaweedfs/seaweedfs/issues/4559
* fix IsEmpty
* fix test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-12 10:42:44 -07:00
柏杰
68feffa4e6
avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value ( #4560 )
2023-06-10 23:15:27 -07:00
Konstantin Lebedev
5ee04d20fa
Healthz check for deadlocks ( #4558 )
2023-06-09 09:42:48 -07:00
wusong
d6af3a8eb6
fix 404 not found ( #4536 )
...
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 09:14:49 -07:00
Konstantin Lebedev
5caff2c0f0
fix: deletes newly created volumes ( #4551 )
...
https://github.com/seaweedfs/seaweedfs/issues/4531
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-06 09:08:27 -07:00
wusong
9bdbf9c880
revert #4491 ( #4550 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-06 00:17:51 -07:00
chrislu
0bffb3770f
support TTL when creating folders
...
related to https://github.com/seaweedfs/seaweedfs/discussions/4549
2023-06-05 23:29:25 -07:00
wusong
26f15d0079
Fix no more writable volumes by delay judgment ( #4548 )
...
* fix nomore writables volumes while disk free space is sufficient by time delay
* reset
---------
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-05 10:17:21 -07:00
chrislu
fb4b61036c
3.52
2023-06-04 22:34:26 -07:00
Konstantin Lebedev
a0931be0c0
S3 TLS credentials Refreshing ( #4506 )
...
* S3 TLS credentials Refreshing
* fix: logging
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-06-04 14:27:56 -07:00
wusong
5aec6da8a3
fix deadlock for filer upload ( #4527 )
...
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-06-03 13:38:27 -07:00
chrislu
e23f3d6eca
set df.File to nil after it is closed
...
possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530
2023-06-02 17:16:45 -07:00
chrislu
ca7cc61319
reduce the window size between unregistering a volume server and creating volumes on that server
...
fix https://github.com/seaweedfs/seaweedfs/issues/4467
2023-05-30 08:20:00 -07:00
chrislu
68c3cf767c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-05-29 16:45:28 -07:00
chrislu
d1d1757520
adjust error message
2023-05-29 16:45:25 -07:00
wusong
e7f4418855
add return for error handle ( #4502 )
2023-05-24 07:45:06 -07:00
chrislu
4310e1fac4
3.51
2023-05-21 23:43:39 -07:00
wusong
8fffe3e822
fix no more writables volumes while disk free space is sufficient ( #4491 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-05-21 22:18:50 -07:00
mervynzhang
1ebb549f77
support swift ( #4480 )
2023-05-19 06:39:25 -07:00
Mesar Hameed
a90b777ff4
Feat: etcd filer store keys should have customizable prefix ( #4484 )
...
An etcd cluster is not necessarily only dedicated to seaweedfs.
This security enhancement adds a customizable key_prefix option to the etcd filer store.
This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the
key_prefix, instead of all keys on the etcd cluster.
2023-05-18 23:08:56 -07:00
SmsS4
17e91d2917
Use filerGroup for s3 buckets collection prefix ( #4465 )
...
* Use filerGroup for s3 buckets collection prefix
* Fix templates
* Remove flags
* Remove s3CollectionPrefix
2023-05-16 09:39:43 -07:00
chrislu
b7f011f777
fix test
2023-05-15 17:46:56 -07:00
Mesar Hameed
65484e80ef
Feat: support username/password authentication for etcd filer store s… ( #4477 )
...
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262
Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
2023-05-15 17:45:14 -07:00
chrislu
9204ee2d2d
3.50
2023-05-11 08:03:11 -07:00
chrislu
d999f1f0e2
update raft version
...
fix #4460
2023-05-09 22:54:23 -07:00
chrislu
25b237b83c
reset wait time on a successful retry
2023-05-07 23:39:33 -07:00
chrislu
59f55c1a66
3.49
2023-05-07 14:00:19 -07:00
chrislu
b2a3860290
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-05-07 13:33:47 -07:00
chrislu
4511edc871
update raft
2023-05-07 13:33:44 -07:00
themarkchen
7592d013fe
fix shell volume.balance bug ( #4447 )
2023-05-04 07:35:37 -07:00
chrislu
42766bccc8
3.48
2023-04-30 22:20:18 -07:00
zemul
e9fda774f4
[Filer] post add param:saveInside ( #4434 )
...
* fix:mount deadlock
* feat: filer http upload to metadata
* feat: /etc save inside
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-29 08:31:05 -07:00
wusong
4867aa03ab
Fix: http rename move dir to subdir ( #4432 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-04-26 20:54:36 -07:00
Konstantin Lebedev
d75a7b7f62
allow deleting only older empty dir without recursion ( #4430 )
2023-04-25 08:31:14 -07:00
zemul
f352616b7d
Volume range read use sync.pool ( #4422 )
2023-04-24 07:00:48 -07:00
chrislu
18686b7375
3.47
2023-04-23 22:45:41 -07:00
zemul
68f1cc34a5
[mount]Fix copy file range ( #4407 )
...
* fix:mount deadlock
* fix
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-23 22:34:38 -07:00
chrislu
e1b92bc701
reset local file content if remote file is updated
...
fix https://github.com/seaweedfs/seaweedfs/issues/4289
2023-04-22 22:32:21 -07:00
chrislu
ff7b6d779e
avoid overwriting variables
...
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-22 11:49:59 -07:00
wusong
2e240704ab
Writables inconsistency ( #4417 )
...
fix: inconsistent read and write permissions between master and volume server
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:14:41 -07:00
wusong
19245dde50
mount: add retry for read only case ( #4416 )
...
* mount: add retry for read only case
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
* add new util retry function for mount
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
* change error list param
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
---------
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-21 00:09:31 -07:00
SmsS4
4131874fa8
use env for flags wtih dot in name ( #4415 )
2023-04-20 09:33:27 -07:00
Damiano Albani
9f55c7c90e
Rclone storage backend ( #4402 )
...
* Add Rclone storage backend
* Support templating the name of files stored via Rclone
* Enable Rclone accounting
* Remove redundant type conversion
* Provide progress information for Rclone download/upload operations
* Log error when Rclone can't instantiate filesystem
* Remove filename templating functionality for Rclone storage
To (maybe) be later reintroduced as a generic functionality for all
storage backends.
* Remove S3 specific check
* Move Rclone config initialisation to init() method
2023-04-18 14:12:12 -07:00
wusong
a408b46d95
compilation fail ( #4414 )
...
Signed-off-by: Wusong Wang <wangwusong@virtaitech.com>
Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>
2023-04-18 00:36:55 -07:00
chrislu
0a22eea55d
collect ec shard from multiple locations
...
fix https://github.com/seaweedfs/seaweedfs/issues/4365
2023-04-17 22:56:21 -07:00
chrislu
ae9388723f
adjust error message
2023-04-17 22:01:39 -07:00
chrislu
8ecdf958ab
3.46
2023-04-16 21:57:43 -07:00
Chengyu Liu
0f653d2153
update fuse client log which should output the fh of FileHandle. ( #4404 )
2023-04-15 22:15:16 -07:00
zemul
0122e022ea
Mount concurrent read ( #4400 )
...
* fix:mount deadlock
* feat: concurrent read
* fix
* Remove useless code
* fix
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-04-13 22:32:45 -07:00
Konstantin Lebedev
5614ad0000
fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter ( #4399 )
...
* fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter
* fix list with delimiter and start token
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 16:53:49 -07:00
Konstantin Lebedev
88b2bbadd1
try fix s3test test_bucket_listv2_delimiter_prefix ( #4396 )
2023-04-12 08:47:20 -07:00
Konstantin Lebedev
095cdb0c7f
fix test prefix is a directory ( #4393 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-12 00:08:13 -07:00
Konstantin Lebedev
44ad07276c
s3 fix get list of dir object key with slash suffix ( #4391 )
...
* s3 fix get list of dir object key with slash suffix
https://github.com/seaweedfs/seaweedfs/issues/3086
* list only entry dir eq prefix
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-04-11 23:52:35 -07:00
Konstantin Lebedev
130bc3e668
s3 fix get fake dir object key ( #4390 )
2023-04-11 07:36:22 -07:00
Matt
da1d3b5a33
adjusted regex to be from 1 to 255 for the value ( #4377 )
...
* compatibility patch for csi driver
* added namespace to all component parameters
* added namespace to all component parameters
* dereference in range
* added namespace to values.yml defaults
* added namespace to s3 component
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* changed helm directory structure
* update charts location
* fixed dereference
* updated permissions
* updated permissions
* match current action schema
* added helm chart liniting CI
* modified chart list changed
* fixed nodejs warning
* standardized a few defaults
* added chart dirs
* lowerd period seconds for volume startup to test chart lint changes
* lowerd period seconds for volume startup to test chart lint changes
* test
* changed back
* adjustment
* debug ls statment
* removed change detection
* remvoed change detection
* always lint the charts
* added missing )
* fixed typo
* added spaces in front of all comments
* pdated values.yaml
* pdated values.yaml
* don't validate maintainers for now
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* - adds helm chart lint on changes
- adds test helm chart install on k8s
* updated helm chart readme.md
* added artifact hub
* added artifact hub
* added ttl validation
* adjusted regex to be from 1 to 255 for the value
* added better error message
* fixed regex
2023-04-10 12:24:38 -07:00
Matt
2d213d87e1
File Path Configuration TTL Validation ( #4376 )
...
* compatibility patch for csi driver
* added namespace to all component parameters
* added namespace to all component parameters
* dereference in range
* added namespace to values.yml defaults
* added namespace to s3 component
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* added helm chart to github pages
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* push on all tags
* changed helm directory structure
* update charts location
* fixed dereference
* updated permissions
* updated permissions
* match current action schema
* added helm chart liniting CI
* modified chart list changed
* fixed nodejs warning
* standardized a few defaults
* added chart dirs
* lowerd period seconds for volume startup to test chart lint changes
* lowerd period seconds for volume startup to test chart lint changes
* test
* changed back
* adjustment
* debug ls statment
* removed change detection
* remvoed change detection
* always lint the charts
* added missing )
* fixed typo
* added spaces in front of all comments
* pdated values.yaml
* pdated values.yaml
* don't validate maintainers for now
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* update helm_ci.yml
* - adds helm chart lint on changes
- adds test helm chart install on k8s
* updated helm chart readme.md
* added artifact hub
* added artifact hub
* added ttl validation
2023-04-04 22:07:17 -07:00
chrislu
efef6e94bf
use UPSERT for postgres style databases
2023-04-04 11:52:36 -07:00
chrislu
422bfaed69
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-04-02 22:53:02 -07:00
chrislu
b9ab65dba5
3.45
2023-04-02 22:52:58 -07:00
wusong
2b82d0f871
style: clerical error ( #4361 )
2023-03-31 07:36:21 -07:00
LHHDZ
8e80ce9476
fix key corrupt when fs.configure copy path trie ( #4353 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-29 11:29:44 -07:00
chrislu
3cc7b90414
print wait in progress
2023-03-26 12:22:48 -07:00
chrislu
f1f14e28bc
adjust name
2023-03-26 12:17:23 -07:00
chrislu
d54798c3ae
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-03-25 22:03:00 -07:00
chrislu
6bfb91b539
update entry Attributes.Mtime = time.Now().Unix()
...
fix https://github.com/seaweedfs/seaweedfs/issues/4337
2023-03-25 22:02:57 -07:00
renweijun
bbd461b2c6
weed mount default EnableACL，Support chmod chown ( #4335 )
2023-03-25 04:47:53 -07:00
chrislu
5db9fcccd4
refactoring
2023-03-21 23:01:49 -07:00
Konstantin Lebedev
de4545c28b
add validate config for raft ( #4332 )
2023-03-21 06:36:13 -07:00
chrislu
adb90bd252
avoid lower casing the command
...
fix https://github.com/seaweedfs/seaweedfs/pull/4321
2023-03-19 21:20:46 -07:00
chrislu
5b43c4bb98
3.44
2023-03-19 20:27:55 -07:00
chrislu
50dc2fe96b
cleaning variables
2023-03-19 18:48:40 -07:00
Konstantin Lebedev
29d1312625
Fs.verify.concurrency ( #4293 )
...
* init
* refactor
* refactor print msg
* param just concurrency
* fix race
* next fix race
* reset variables before each run
* clean
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-19 18:38:57 -07:00
chrislu
784daea1fa
fix volume not found if marked as read only
...
fix https://github.com/seaweedfs/seaweedfs/issues/4088
2023-03-19 18:30:13 -07:00
Stewart Miles
264be0d2d4
Retry until a leader is selected. ( #4318 )
...
Fixes regression introduced in
https://github.com/seaweedfs/seaweedfs/pull/4313
Related to #4307
2023-03-16 20:50:38 -07:00
LHHDZ
d695119073
The filer does not set defaultReplication to the defaultReplication o… ( #4315 )
...
The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master)
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-03-15 22:10:39 -07:00
Stewart Miles
57ab1f8516
Use exponential backoff to query leader. ( #4313 )
...
`topology.Leader()` was using a backoff that typically
resulted in at least a 5s delay when initially starting
a master and raft server. This changes the backoff
algorithm to use exponential backoff starting with 100ms
and waiting up to 20s for leader selection.
Related to #4307
2023-03-15 17:49:46 -07:00
Stewart Miles
dd71f54c6b
Fix -raftHashicorp and -raftBootstrap flag propagation. ( #4309 )
...
`weed server` was not correctly propagating
`-master.raftHashicorp` and `-master.raftBootstrap` flags when
starting the master server.
Related to #4307
2023-03-15 13:03:20 -07:00
Patrick Schmidt
71b33faef0
Skip parent directory creation in mount ( #4310 )
...
A POSIX filesystem does not implicitly create parent directories when
they do not exist. Directories must be explicitly created and
permissions be set.
This also fixes a bug where asynchronous operations would create a
file in the filer before the parent directory was created. If the
file was a symlink or another special type of file the directory
would inherit that type and become unusable in the mounted FS.
2023-03-15 12:37:25 -07:00
chrislu
f5854d13df
fix test
2023-03-14 22:05:16 -07:00
chrislu
0454bb2d88
fix test
...
when size is zero, the "LastModified" is not written
2023-03-14 22:02:49 -07:00
chrislu
2268d38263
fix naming
...
fix https://github.com/seaweedfs/seaweedfs/issues/4305
2023-03-14 08:38:02 -07:00
chrislu
98bc93cf31
fix tests
2023-03-13 10:00:27 -07:00
Konstantin Lebedev
e17429223e
shell script unclean variables ( #4298 )
2023-03-13 07:16:31 -07:00
Thomas Anderson
bd70683871
fix(weed/topology/node.go): typo in error message ( #4292 )
2023-03-08 23:58:03 -08:00
Rohit Chormale
59706c89fb
#4270 set http status code to 409 if dir already exists ( #4287 )
2023-03-08 07:45:08 -08:00
chrislu
7111c08289
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-03-08 01:48:30 -08:00
chrislu
c589e9837c
fix tests
2023-03-08 01:48:28 -08:00
Kevin Liu
244385bf0d
Fix binding metrics to ipv6 ( #4286 )
...
* Fix binding metrics to ipv6
* Update weed/stats/metrics.go
---------
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-03-08 01:04:02 -08:00
zemul
bd6f2716d3
fix_read_tomb_needle_meta ( #4263 )
...
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:26:55 -08:00
zemul
0bf56298d5
fix chunk.ModifiedTsNs ( #4264 )
...
* fix
* fix mtime s > ns
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-03-02 08:24:36 -08:00
chrislu
8241c9a829
adjust ui pagination size
...
fix https://github.com/seaweedfs/seaweedfs/issues/4258
2023-02-28 08:31:26 -08:00
lfhy
1976ca9160
add -disk to filer command ( #4247 )
...
* add -disk to filer command
* add diskType to filer.grpc
* use filer.disk when filerWebDavOptions.disk is empty
* add filer.disk to weed server command.
---------
Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
2023-02-25 09:48:59 -08:00
chrislu
214b7cd286
volume.fix.replication: adjust the retry checking times
2023-02-22 10:47:52 -08:00
LHHDZ
db5515eada
Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists ( #4240 )
...
Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2023-02-21 23:08:52 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency ( #4239 )
2023-02-21 19:47:33 -08:00
chrislu
3227e4175e
3.43
2023-02-20 16:56:05 -08:00
chrislu
9bb0a9e306
clean comments
2023-02-17 01:31:44 -08:00
chrislu
545d5d0cc3
fix for io.ReaderAt used in filer.remote.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/4194
2023-02-17 01:30:55 -08:00
wusong
79af6b41d4
[mount] Fix read file-data in entry.content ( #4207 )
2023-02-15 07:18:56 -08:00
chrislu
5869945f16
avoid infinite loop
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904
2023-02-10 13:05:04 -08:00
chrislu
e037c71ec3
adjust text
2023-02-10 13:04:29 -08:00
chrislu
67b8c2853a
add line return
2023-02-10 12:53:43 -08:00
chrislu
302155a6fa
fix compilation
2023-02-10 12:32:59 -08:00
chrislu
5520b81551
add some error messages for args
...
related to https://github.com/seaweedfs/seaweedfs/issues/4115
2023-02-10 12:30:53 -08:00
chrislu
dc4ed2cd9b
do not move cloud tier volumes
...
fix https://github.com/seaweedfs/seaweedfs/issues/4195
2023-02-09 22:17:54 -08:00
chrislu
31bb91583f
fix bug when vid not found
...
fix https://github.com/seaweedfs/seaweedfs/issues/4193
2023-02-09 17:30:44 -08:00
chrislu
79d68a81fd
text change
2023-02-07 15:00:41 -08:00
chrislu
999a126a3f
typo
2023-02-07 14:59:47 -08:00
chrislu
1c2364936a
detect invalid bucket name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4143
2023-02-07 14:57:29 -08:00
chrislu
5de93fe442
refactoring
2023-02-07 14:45:20 -08:00
Eric Yang
5083429704
ADHOC: s3 bucket name verification ( #4189 )
...
* ADHOC: s3 bucket name verification
* add test
2023-02-07 14:37:29 -08:00
Neo
d5f77706a8
volume:fix return if Replicate write error ( #4188 )
2023-02-06 22:34:56 -08:00
chrislu
8821d6b161
3.42
2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region ( #4177 )
...
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu
84e9934bf9
fix filer.remote.sync on a S3 cloud mount
...
fix https://github.com/seaweedfs/seaweedfs/issues/4175
2023-02-01 20:44:00 -08:00
chrislu
0932437a1c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-02-01 02:38:09 -08:00
chrislu
c47f75eb5b
fix null content type caused exception in hdfs client
...
fix https://github.com/seaweedfs/seaweedfs/issues/4170
2023-02-01 02:38:05 -08:00
wusong
fe573f26a8
memchunk save content twice ( #4174 )
2023-01-31 19:36:59 -08:00
Neo
b9b613a78e
filter system log dir does not make subscribe event ( #4172 )
2023-01-31 18:56:11 -08:00
wusong
665917fdf1
[mount]: delete redundant code ( #4173 )
2023-01-31 18:54:46 -08:00
a
0d085ec175
make directory lister trigger if mime type is httpd/unix-directory ( #4169 )
2023-01-31 06:54:40 -08:00
chrislu
4d71af87f3
3.41
2023-01-29 22:56:01 -08:00
chrislu
ef70415c70
skipping manifest chunk volume id change
2023-01-29 22:55:06 -08:00
chrislu
6e8dbf8868
add fs.meta.changeVolumeId
2023-01-29 22:51:58 -08:00
wusong
de081c0d64
[mount] fix metacache update ( #4161 )
2023-01-29 07:55:27 -08:00
Guo Lei
b63d1e9c6b
fix index ( #4158 )
2023-01-28 06:55:27 -08:00
chrislu
81fdf3651b
grpc connection to filer add sw-client-id header
2023-01-20 01:48:12 -08:00
chrislu
bfe5d910c6
use one readerCache for the whole file
2023-01-16 22:43:02 -08:00
chrislu
4614e85efa
adjust help message
2023-01-15 21:28:36 -08:00
chrislu
c24678153c
use memory chunks only for sequential writes
2023-01-15 21:10:08 -08:00
chrislu
2885ba0e50
3.40
2023-01-15 20:12:05 -08:00
chrislu
12f8251d32
avoid index out of slice range
2023-01-14 13:31:22 -08:00
chrislu
313fd17ae4
allow random writes for mem chunk
2023-01-13 20:32:18 -08:00
Chris Lu
1cd2e64aac
merge chunks during upload ( #4130 )
...
* merge chunks during upload
* fix test
2023-01-12 23:07:36 -08:00
chrislu
2452f3b2f7
sort chunks by offset
2023-01-12 20:29:07 -08:00
chrislu
361d97941b
mysql: Missed escaping column name
...
fix https://github.com/seaweedfs/seaweedfs/issues/4129
2023-01-11 08:46:56 -08:00
chrislu
efcffe4f77
no need to set chunks during flush
2023-01-10 10:01:52 -08:00
chrislu
85f1b70207
reduce group size from 256MB to 64MB
2023-01-10 01:35:44 -08:00
chrislu
86657ed3a8
avoid repeatedly adding manifest chunks
2023-01-10 01:35:10 -08:00
chrislu
28fe578944
avoid possible nil entry
2023-01-10 01:07:34 -08:00
chrislu
47fa5ef979
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-10 00:52:50 -08:00
chrislu
9e0d526163
fix compilation
2023-01-10 00:52:05 -08:00
chrislu
7f49c59c14
cluster.ps add filer meta sync progress
2023-01-10 00:51:25 -08:00
chrislu
340e7c3a2e
chunk group remove manifestChunks and reset sections in SetChunks()
2023-01-10 00:49:31 -08:00
chrislu
e650c8397d
ensure section.chunks is always garbage free
2023-01-10 00:48:01 -08:00
chrislu
75bdd4a0d1
refactor
2023-01-10 00:46:46 -08:00
Guo Lei
83e29faa55
use time.NewTicker instead of time.Tick. ( #4119 )
2023-01-09 06:47:08 -08:00
chrislu
51d4a4b28d
3.39
2023-01-08 18:20:22 -08:00
chrislu
0b52f08226
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-01-08 18:03:25 -08:00
chrislu
469c959e6a
add missing options for webdav
2023-01-08 18:03:22 -08:00
monchickey
3e2c9ea73d
Add image cropping. ( #4117 )
2023-01-06 09:28:07 -08:00
chrislu
296fdc296c
mount: faster add chunks
2023-01-06 01:03:29 -08:00
chrislu
2abf817580
fix for stream reader
...
fix https://github.com/seaweedfs/seaweedfs/issues/4112
2023-01-05 11:19:21 -08:00
chrislu
e367444586
add notes for full version
2023-01-04 09:52:25 -08:00
chrislu
5423790b2c
add back previous chunk upload selection algo
2023-01-03 22:50:39 -08:00
chrislu
7bdae5172e
batch delete EC needles
...
fix https://github.com/seaweedfs/seaweedfs/issues/4107
2023-01-03 22:05:26 -08:00
zemul
6b4c033431
add mount log ( #4101 )
...
* filer.backup use replication.source.filer
* add mount log
* Revert "filer.backup use replication.source.filer"
This reverts commit 07bf6f956c
2023-01-03 00:00:45 -08:00
chrislu
48f2edc065
3.38
2023-01-02 23:23:02 -08:00
Chris Lu
d4566d4aaa
more solid weed mount ( #4089 )
...
* compare chunks by timestamp
* fix slab clearing error
* fix test compilation
* move oldest chunk to sealed, instead of by fullness
* lock on fh.entryViewCache
* remove verbose logs
* revert slat clearing
* less logs
* less logs
* track write and read by timestamp
* remove useless logic
* add entry lock on file handle release
* use mem chunk only, swap file chunk has problems
* comment out code that maybe used later
* add debug mode to compare data read and write
* more efficient readResolvedChunks with linked list
* small optimization
* fix test compilation
* minor fix on writer
* add SeparateGarbageChunks
* group chunks into sections
* turn off debug mode
* fix tests
* fix tests
* tmp enable swap file chunk
* Revert "tmp enable swap file chunk"
This reverts commit 985137ec47
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 ( #4094 )
2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax ( #4096 )
2023-01-01 05:06:57 -08:00
CommanderRoot
265a56630b
filer.store.mysql: Escape table columns in SQL query ( #4095 )
2023-01-01 05:06:41 -08:00
Guo Lei
ed5f3f073b
add more help message, in case of misunderstanding ( #4092 )
2022-12-29 07:05:05 -08:00
Guo Lei
d8cfa1552b
support enable/disable vacuum ( #4087 )
...
* stop vacuum
* suspend/resume vacuum
* remove unused code
* rename
* rename param
2022-12-28 01:36:44 -08:00
Guo Lei
5f993a5f20
Fix ide warning ( #4082 )
...
fix warning message from IDE
"redundant type from array, slice, or map composite literal"
2022-12-26 00:03:47 -08:00
Konstantin Lebedev
569bc7b54d
fs.verify needles status of file ( #4060 )
...
* fs.verify needles status of file
* fix parse args
* refactor logging
* set modifyTimeAgoAtSec
2022-12-21 15:54:17 -08:00
chrislu
77b7c9f7cc
lock and unlock chunks for read
2022-12-21 14:15:35 -08:00
chrislu
d1797deccb
remove duplicated logic of removing from writableChunks
2022-12-21 14:15:11 -08:00
chrislu
6ede19e825
add a simple file replication progress bar
2022-12-20 19:47:21 -08:00
chrislu
f7beba8515
unlock before submitting the uploading jobs
2022-12-20 12:52:30 -08:00
chrislu
28d479e5c0
mount: adjust locking for upload pipeline
2022-12-19 15:07:22 -08:00
chrislu
f9fb3cd1fc
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-19 11:32:00 -08:00
chrislu
6c7fe40305
filer sink retries reading file chunks, skipping missing chunks
...
if the file chunk is not available during replication time, the file is skipped
2022-12-19 11:31:58 -08:00
chrislu
3fc3e7083c
metadata skip reading un-available logs
...
this means the volumes for metadata histories have been lost.
2022-12-19 11:30:50 -08:00
lfhy
038bab3726
add -filer.path to webdav command ( #4061 )
2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993
more detailed logs
2022-12-17 13:18:35 -08:00
LHHDZ
f84c13dabd
fix response not allowed ( #4057 )
2022-12-13 05:48:50 -08:00
LHHDZ
bb289d7d69
sort tags by key ( #4047 )
...
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-12-11 22:49:57 -08:00
chrislu
438146249f
3.37
2022-12-11 21:08:06 -08:00
chrislu
ac9dea0ad9
rotate log files
2022-12-11 21:05:58 -08:00
chrislu
fc6b9e6e0c
volume server logs add url info
...
debug https://github.com/seaweedfs/seaweedfs/issues/3964
2022-12-11 13:15:38 -08:00
chrislu
38c63c429b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-12-10 21:49:09 -08:00
chrislu
e1ca6308cb
add chunk etag when downloading from remote storage
...
fix https://github.com/seaweedfs/seaweedfs/issues/3987
2022-12-10 21:49:07 -08:00
wusong
549354e324
Fix hardlink counting ( #4042 )
...
Signed-off-by: wusong <wangwusong@virtaitech.com>
Signed-off-by: wusong <wangwusong@virtaitech.com>
Co-authored-by: wusong <wangwusong@virtaitech.com>
2022-12-08 10:50:57 -08:00
Jiffs Maverick
5f59d43c88
[filer] Fix uri escape during renaming ( #4039 )
2022-12-07 07:47:40 -08:00
chrislu
659f28b855
simple refactoring
2022-12-05 23:19:38 -08:00
chrislu
ca49ef8ed8
duplicated logic
2022-12-05 23:19:22 -08:00
Chris Lu
dac9c28d05
Revert "refactor: moved to locked entry" ( #4035 )
...
* Revert "refactor: moved to locked entry"
This reverts commit 94bc9afd9d
2022-12-05 12:32:27 -08:00
chrislu
fad7e1f7cb
3.36
2022-12-04 23:34:29 -08:00
chrislu
94bc9afd9d
refactor: moved to locked entry
2022-12-04 23:33:05 -08:00
aronneagu
2b783738d6
Update Makefile - use binary in current directory ( #4027 )
2022-12-01 14:09:14 -08:00
aronneagu
77699855a7
Return ETag from remote when file doesn't exist on Filer ( #4025 )
2022-11-30 07:43:30 -08:00
Konstantin Lebedev
f59c3acd5f
volume.list show volumes only from the specified dc/rack/dn ( #4024 )
2022-11-30 06:30:55 -08:00
Konstantin Lebedev
2b910d1cf8
avoid recursive deleting newly created empty directories ( #4016 )
2022-11-25 08:45:47 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip ( #4012 )
2022-11-24 10:22:59 -08:00
famosss
15284e2bee
fix: Lost Content-Encoding metadata ( #4010 )
2022-11-23 22:52:34 -08:00
James Hartig
81624de27b
Include name/mime in ReadAllNeedles ( #4005 )
2022-11-23 15:59:38 -08:00
Eric Yang
26d9b662b1
ADHOC: fix fs.meta.save typo ( #3993 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-11-20 21:05:00 -08:00
James Hartig
4c85da7844
Include meta in ReadAllNeedles ( #3991 )
...
This is useful for doing backups on the data so we can accurately store the
last modified time, the compression state, and verify the crc.
Previously we were doing VolumeNeedleStatus and then an HTTP request which
needlessly read from the dat file twice.
2022-11-20 20:19:41 -08:00
chrislu
70a4c98b00
refactor filer_pb.Entry and filer.Entry to use GetChunks()
...
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
chrislu
b1d7963e57
3.35
2022-11-14 00:21:00 -08:00
Guo Lei
5b905fb2b7
Lazy loading ( #3958 )
...
* types packages is imported more than onece
* lazy-loading
* fix bugs
* fix bugs
* fix unit tests
* fix test error
* rename function
* unload ldb after initial startup
* Don't load ldb when starting volume server if ldbtimeout is set.
* remove uncessary unloadldb
* Update weed/command/server.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* Update weed/command/volume.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-11-14 00:19:27 -08:00
chrislu
d90aa31d5f
filer store: skip disabled location specific filer store initialization
...
fix https://github.com/seaweedfs/seaweedfs/issues/3971
2022-11-13 23:30:23 -08:00
chrislu
7ac40ae3d4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-11-13 23:01:58 -08:00
chrislu
aae82dca6f
file name escape uri
...
fix https://github.com/seaweedfs/seaweedfs/issues/3969
2022-11-13 23:01:56 -08:00
LHHDZ
39f0246deb
fix 'NoSuchUpload' when upload part ( #3968 )
2022-11-11 04:53:53 -08:00
famosss
df0662038c
feat:add a uuid after uploadid ( #3963 )
2022-11-10 07:17:29 -08:00
aronneagu
180853a2c9
Replace dashes with underscores in x-amz-meta headers ( #3965 )
2022-11-10 07:09:53 -08:00
chrislu
47db75a695
3.34
2022-11-06 23:06:22 -08:00
Konstantin Lebedev
5431c445cd
fix filer.remote.sync to azure with ContentType ( #3949 )
...
* fix filer.remote.sync to azure with ContentType
* fix pass X-Amz-Meta to X-Ms-Meta
2022-11-04 09:10:33 -07:00
zemul
cec55feeb0
[filer.backup] add retention_duration ( #3941 )
2022-11-04 00:34:40 -07:00
chrislu
4193dafce1
azure metadata: skip metadata prefixed with "X-"
...
fix https://github.com/seaweedfs/seaweedfs/issues/3875
2022-11-02 21:42:02 -07:00
pavelzhurov
abe4a61659
Bug fix: empty key in DeleteMultipleObjects request caused bucket delete ( #3939 )
2022-11-02 05:51:13 -07:00
Konstantin Lebedev
0999f9b7ff
[volume.fsck] collect ids without cut off time for finding missing data from volumes ( #3934 )
...
collect all file ids from the file without cut off time for finding missing data from volumes
2022-10-31 11:38:12 -07:00
Konstantin Lebedev
a322ba042e
[volume.fsck] param volumeId is comma separated the volume id ( #3933 )
...
volume.fsck param volumeId is comma separated the volume id
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-31 11:36:26 -07:00
Konstantin Lebedev
4a48332248
refactor error contains already deleted ( #3932 )
2022-10-31 11:34:45 -07:00
Konstantin Lebedev
c0deaa4948
[volume.fsck] check needles status from volume server ( #3926 )
...
check needles status from volume server
2022-10-31 11:33:04 -07:00
chrislu
0f360862bf
3.33
2022-10-30 20:40:18 -07:00
Konstantin Lebedev
409c9328de
[master] avoid vacuum if not enough replica copies ( #3924 )
...
avoid vacuum if not enough replica copies
2022-10-30 20:34:19 -07:00
Konstantin Lebedev
bf8a9d2db1
[volume.chek.disk] sync of deletions the fix ( #3923 )
...
* sync of deletions the fix
* avoid return if only partiallyDeletedNeedles
* refactor sync deletions
2022-10-30 20:32:46 -07:00
chrislu
8b9957d461
add back "/" prefix if it is missing in object
...
fix https://github.com/seaweedfs/seaweedfs/issues/3737
2022-10-29 17:54:30 -07:00
chrislu
0d9f2f9e7a
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-28 12:53:23 -07:00
chrislu
ea2637734a
refactor filer proto chunk variable from mtime to modified_ts_ns
2022-10-28 12:53:19 -07:00
Konstantin Lebedev
764d9cb105
[voluche.chek.disk] needles older than the cutoff time are not missing yet ( #3922 )
...
needles older than the cutoff time are not missing yet
https://github.com/seaweedfs/seaweedfs/issues/3919
2022-10-28 12:12:20 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error ( #3918 )
2022-10-28 06:59:39 -07:00
chrislu
1e0d64c048
master: delete partially created replicated volumes during volume growth
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120
2022-10-26 12:32:24 -07:00
Konstantin Lebedev
c6f2899375
avoid fix missing of deleted needles ( #3909 )
2022-10-25 21:35:54 -07:00
Eric Yang
51d462f204
ADHOC: volume fsck using append at ns ( #3906 )
...
* ADHOC: volume fsck using append at ns
* nit
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-10-24 22:09:38 -07:00
famosss
34132b2c9f
fix: isTruncated returns problematic ( #3907 )
2022-10-24 21:15:37 -07:00
famosss
874fd197b5
feat: simplify a bit ( #3905 )
2022-10-24 17:58:26 -07:00
famosss
25e012d30b
fix: set user metadata key to lowercase ( #3894 )
...
* fix: set user metadata key to lowercase
* feat: simplify a bit
2022-10-24 17:29:52 -07:00
chrislu
091f999e8c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-24 16:50:41 -07:00
chrislu
377870f4a9
keep system log data
2022-10-24 16:50:39 -07:00
Guo Lei
97edb40275
Fix errinfo ( #3893 )
...
* types packages is imported more than onece
* Fix error response when format of --expires is wrong.
It MUST be in RFC 1123 date format.
2022-10-24 09:25:48 -07:00
chrislu
d5364218b2
adjust help message
2022-10-24 09:25:08 -07:00
chrislu
d880fc2bb3
fix merge
2022-10-23 18:26:22 -07:00
Konstantin Lebedev
6253058d9d
ensure monotonic n.AppendAtNs in each place ( #3880 )
...
https://github.com/seaweedfs/seaweedfs/issues/3852
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-10-23 18:24:52 -07:00
Konstantin Lebedev
452202f35b
ReadNeedleMeta recovery panic occurred ( #3883 )
...
ReadNeedleMeta panic occurred
https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:28 -07:00
chrislu
e3ea544d40
skip reading needle if size is invalid
...
related to https://github.com/seaweedfs/seaweedfs/issues/3882
2022-10-23 14:00:15 -07:00
chrislu
5294b6dd4d
remove volume server tcp client
2022-10-23 13:15:03 -07:00
chrislu
f2d9049e6a
fix size variable
2022-10-23 13:05:37 -07:00
chrislu
184fbb6c50
volume server: remote tier volumes only soft delete in local index
...
fix https://github.com/seaweedfs/seaweedfs/issues/3889
2022-10-23 13:04:38 -07:00
liubaojiang
25471d579a
add md5 header when UploadData to replication in ReplicatedWrite ( #3881 )
2022-10-19 01:34:14 -07:00
chrislu
758d70bdc0
3.32
2022-10-16 20:43:37 -07:00
chrislu
de286fe662
shell: volume.move handles volume moved to cloud tier
...
fix https://github.com/seaweedfs/seaweedfs/issues/3803
2022-10-16 17:52:22 -07:00
chrislu
52e0a88a15
shell: volume.tier.upload clean up replicated copies
...
fix https://github.com/seaweedfs/seaweedfs/issues/3804
2022-10-16 17:02:56 -07:00
Konstantin Lebedev
e20f0dbd2d
avoid data race of TraverseBfs ( #3856 )
...
* avoid data race of TraverseBfs
* close is enough
avoid panic
I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255
I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c]
goroutine 287 [running]:
github.com/seaweedfs/seaweedfs/weed/storage.(*SortedFileNeedleMap).Close(0x0)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load.func1()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e
github.com/seaweedfs/seaweedfs/weed/storage.(*Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c
github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1)
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da
github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes.func2()
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9
created by github.com/seaweedfs/seaweedfs/weed/storage.(*DiskLocation).concurrentLoadingVolumes
/Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150
2022-10-16 11:30:41 -07:00
Konstantin Lebedev
ee38ab8581
avoid data race of LogBuffer isStopping ( #3859 )
2022-10-16 11:28:56 -07:00
Konstantin Lebedev
7836f7574e
[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 ( #3861 )
...
* fix apply purging and add verifyNeedle
* common readSourceNeedleBlob
* use consts
2022-10-15 20:38:46 -07:00
Konstantin Lebedev
f476cf3403
fix volume_server.proto conflict ( #3858 )
...
fix conflict
https://github.com/seaweedfs/seaweedfs/pull/3841
2022-10-15 20:01:07 -07:00
chrislu
5c2eb602c9
Revert "mount: remove mount directory from generating cache directory id"
...
This reverts commit 109dc7fdf6
2022-10-14 22:24:14 -07:00
Konstantin Lebedev
2f72103c83
avoid load volume file with BytesOffset mismatch ( #3841 )
...
* avoid load volume file with BytesOffset mismatch
https://github.com/seaweedfs/seaweedfs/issues/2966
* set BytesOffset if has not VolumeInfoFile
* typos fail => failed
* exit if bytesOffset mismatch
2022-10-14 00:18:09 -07:00
Konstantin Lebedev
f19c9e3d9d
Volume fsck by volume ( #3851 )
...
* refactor
* refactor args verbose and writer
* refactor readFilerFileIdFile
* fix filter by collectMtime
* skip system log collection
2022-10-13 23:30:30 -07:00
chrislu
9c8678ded9
ensure monotonic n.AppendAtNs
...
fix https://github.com/seaweedfs/seaweedfs/issues/3852
2022-10-13 23:15:00 -07:00
chrislu
e05637c42c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-13 23:10:49 -07:00
chrislu
dcd0743a35
remove unused ReadNeedleBlobRequest.needle_id
...
fix https://github.com/seaweedfs/seaweedfs/issues/3853
2022-10-13 23:10:46 -07:00
LHHDZ
a5b867af69
don't skip empty value at AssembleEntryWithAcp ( #3855 )
...
* add acl helper functionalities
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* add tests
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* remove 0 when create map
Signed-off-by: changlin.shi <changlin.shi@ly.com>
* delete when empty at `AssembleEntryWithAcp`
`PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-13 22:50:44 -07:00
Konstantin Lebedev
1f7e52c63e
vacuum metrics and force sync dst files ( #3832 )
2022-10-13 00:51:20 -07:00
chrislu
f5d4952d73
filer: redis store reduce from 2 redis operations to 1 for updates.
2022-10-12 23:50:09 -07:00
Guo Lei
f95c25e113
types packages is imported more than onece ( #3838 )
2022-10-12 22:59:07 -07:00
chrislu
9a339a9cfb
default hasSlowRead to true
2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true ( #3710 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
LHHDZ
bf5e45b66a
add acl helper functionalities ( #3831 )
2022-10-12 20:15:16 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c ( #3834 )
2022-10-12 07:18:40 -07:00
zemul
a05725aea6
filer: get directory metadata ( #3833 )
2022-10-12 07:14:49 -07:00
chrislu
e55076c46f
cloud tier: add retry when copying data file
...
fix https://github.com/seaweedfs/seaweedfs/issues/3828
2022-10-12 00:38:32 -07:00
chrislu
cea73ac008
serialize sqlite operations
...
fix https://github.com/seaweedfs/seaweedfs/issues/3827
2022-10-12 00:03:27 -07:00
chrislu
dff85e9c71
fix error handling
2022-10-12 00:03:14 -07:00
chrislu
8aec786a6d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-11 21:58:19 -07:00
chrislu
0d817bc347
fix invalid memory address or nil pointer dereference on filer.sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3826
2022-10-11 21:58:17 -07:00
Guo Lei
84c401e693
Optimiz leveldb metric ( #3830 )
...
* optimiz updating mapmetric for leveldb
* import loading leveldb
* add comments
2022-10-11 21:13:25 -07:00
LHHDZ
d21e2f523d
split ExtAcpKey to ExtAmzOwnerKey and ExtAmzAclKey to avoid unn… ( #3824 )
...
split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call
Signed-off-by: changlin.shi <changlin.shi@ly.com>
Signed-off-by: changlin.shi <changlin.shi@ly.com>
2022-10-11 20:14:14 -07:00
Konstantin Lebedev
5b28c3f728
revert disabling FSync for non Mac ( #3814 )
2022-10-10 07:28:02 -07:00
LHHDZ
3550692afc
change s3_account.go package to avoid cycle dependency ( #3813 )
2022-10-10 06:44:29 -07:00
chrislu
0711870f43
3.31
2022-10-10 00:15:30 -07:00
chrislu
109dc7fdf6
mount: remove mount directory from generating cache directory id
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3811
2022-10-10 00:12:10 -07:00
chrislu
fc54afdba9
tests add parsing ec shard info
2022-10-09 22:47:55 -07:00
chrislu
0623bf582e
include ec shard for capacityByFreeVolumeCount
2022-10-09 22:31:30 -07:00
chrislu
98dc1e5c15
move volume: find target volume server by exiting/max ratio
2022-10-09 22:30:59 -07:00
chrislu
049f040c3c
refactor
2022-10-09 22:28:39 -07:00
chrislu
124b97f6d7
parsing ec volumes
2022-10-09 19:21:23 -07:00
chrislu
1e27d36576
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-10-09 18:58:14 -07:00
chrislu
f9383aa726
refactor to change capacity data type
2022-10-09 18:58:10 -07:00
binbinshi
b7de4a967e
fix: compact_map get error mismatching cokie ( #3748 )
...
* fix: compact_map get error
* fix: CompactSection delete lock and move test to compact_map
Co-authored-by: shibinbin <shibinbin@megvii.com>
2022-10-09 16:56:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options ( #3807 )
...
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
5db25a8f2a
metric shows who is currently blocking the cluster or not ( #3799 )
...
* master_admin_lock Shows whether cluster is locked now or not
https://github.com/seaweedfs/seaweedfs/issues/3452
* fix metric MasterAdminLock
2022-10-07 13:26:29 -07:00
Viktor Kuzmin
f8d3ff466d
Allow parallel volume loading from different dirs during startup. ( #3802 )
2022-10-07 11:27:15 -07:00
Konstantin Lebedev
a522507f95
configure raft metrics ( #3798 )
2022-10-07 04:20:34 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop ( #3797 )
2022-10-06 09:30:30 -07:00
Konstantin Lebedev
36daa7709d
show raft leader via shell ( #3796 )
2022-10-06 07:10:41 -07:00
chrislu
ea271600ec
fix parameters
2022-10-04 12:36:05 -07:00
chrislu
0452ae6a6c
filer.sync: limit concurrency when fetching file chunks
...
fix https://github.com/seaweedfs/seaweedfs/issues/3787
2022-10-04 11:35:07 -07:00
Konstantin Lebedev
ec46a34f33
avoid increasing the coordinates of metric labels ( #3785 )
2022-10-04 01:20:01 -07:00
chrislu
29632d5a34
3.30
2022-10-02 18:13:44 -07:00
LHHDZ
e9584d9661
add ownership rest apis ( #3765 )
2022-10-01 19:18:00 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir ( #3770 )
2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7
better fix for option format
2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb
mount: fix extra option format
2022-09-30 08:44:41 -07:00
chrislu
8e81619d02
mount: accept all extra mount options
...
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Eric Yang
56c94cc08e
ADHOC: filter deleted files from idx file binary search ( #3763 )
...
* ADHOC: filter deleted files from idx file binary search
* remove unwanted check
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-29 12:48:36 -07:00
LHHDZ
3de1e19780
s3: sync bucket info from filer ( #3759 )
2022-09-29 12:29:01 -07:00
LHHDZ
e00a12b099
associate Account and Identity by accountId ( #3754 )
2022-09-28 13:25:59 -07:00
LHHDZ
aacdcc4cad
s3: add account ( #3753 )
...
associate `Account` and `Identity` by accountId
2022-09-28 12:45:18 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal ( #3755 )
...
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
chrislu
47fd5d3fa1
skip debug message
2022-09-27 13:12:31 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server ( #3708 )
...
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
jerebear12
190afcc3fd
Allow CORS on Filer ( #3750 )
...
Adjusted filer response headers
2022-09-27 11:47:52 -07:00
Konstantin Lebedev
faa6167b6b
fs.meta.load load any dirs with prefix "important" ( #3747 )
...
* fs.meta.load load any dirs with prefix "important"
* replace dirPattern to dirPrefix
* help dirPrefix
2022-09-27 09:58:46 -07:00
chrislu
b6d7556dda
skip truncation on error
...
fix https://github.com/seaweedfs/seaweedfs/issues/3746
2022-09-27 09:48:23 -07:00
chrislu
1ffb1e696e
3.29
2022-09-25 19:20:45 -07:00
chrislu
640c53cda8
add future list
2022-09-25 17:43:42 -07:00
chrislu
cc570a4477
assert results are sorted
2022-09-25 13:49:28 -07:00
chrislu
5c8f1467a1
ordered execution async wait
2022-09-25 13:45:55 -07:00
chrislu
973b8ff067
rename
2022-09-25 13:34:55 -07:00
chrislu
e31fdbc89b
rename file
2022-09-25 12:59:38 -07:00
chrislu
31922b2bf2
s3 to watch specific directories
2022-09-20 09:30:05 -07:00
chrislu
b90d92fca9
grpc: watch metadata changes by directory
2022-09-20 09:25:18 -07:00
chrislu
b463ca1a2f
filer replication: compare content changes directly
...
Fix https://github.com/seaweedfs/seaweedfs/issues/3714
The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks.
For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.
2022-09-20 08:35:10 -07:00
chrislu
b2a171632b
mount: fix 0 file size in linux
2022-09-18 21:42:13 -07:00
chrislu
23f3aea4cf
adjust logs
2022-09-18 18:14:24 -07:00
chrislu
901e8157de
3.28
2022-09-18 17:46:04 -07:00
chrislu
9d230521a4
report write error
2022-09-18 16:49:48 -07:00
chrislu
10d5b4b32b
volume server: rename readBufferSize to readBufferSizeMB
2022-09-17 10:56:28 -07:00
chrislu
3fc261d27c
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-16 23:54:39 -07:00
chrislu
06748f2771
remove unused code
2022-09-16 23:54:37 -07:00
Konstantin Lebedev
5d87ad72d8
mute log filer: no entry is found in filer store ( #3707 )
2022-09-16 09:05:38 -07:00
Ryan Russell
824f7ad9e1
refactor(shell): readability improvements ( #3704 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:49 -07:00
Ryan Russell
277976bd76
refactor(storage): readability improvements ( #3703 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-16 02:43:17 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization ( #3702 )
...
* simplify a bit
* feat: volume: add "readBufSize" option to customize read optimization
* refactor : redbufSIze -> readBufferSize
* simplify a bit
* simplify a bit
2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35
mark "hasSlowRead" as experimental
2022-09-15 23:33:46 -07:00
Ryan Russell
6f7ef8711a
docs(s3api): readability improvements ( #3696 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:13:21 -07:00
Ryan Russell
ad3a3c8782
refactor(s3api_object_handlers): deleteMultipleObjectsLimmit -> `de… ( #3695 )
...
refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:48 -07:00
Ryan Russell
c4fc007630
refactor(raft_hashicorp): timeSpeep -> timeSleep ( #3694 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:30 -07:00
Ryan Russell
c202d4c1b0
refactor(master_grpc_server_cluster): seletedSet -> selectedSet ( #3693 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 03:12:13 -07:00
chrislu
f9e3e7d1c4
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4
volume: add "hasSlowRead" option to customize read optimization
2022-09-15 03:11:32 -07:00
Ryan Russell
e0064ae097
docs: update fileIdsToDelete var ( #3692 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:09:06 -07:00
Ryan Russell
d65bdeef08
docs(meta_cache): readability fixes ( #3691 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:57 -07:00
Ryan Russell
92658b05a8
refactor(volume_grpc_client_to_master): grpcConection -> `grpcConne… ( #3690 )
...
refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:04:29 -07:00
Ryan Russell
a635973847
docs(topology): readability improvements ( #3688 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 02:03:10 -07:00
Ryan Russell
0fc242b084
docs: panicing -> panicking ( #3687 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:38:04 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes ( #3686 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:33:04 -07:00
Ryan Russell
150ce9f3f6
docs(wdclient): readability fix ( #3685 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:29:26 -07:00
ImFantuan
b64411bda8
fix 0 size ( #3683 )
2022-09-15 01:27:02 -07:00
famosss
bb880d6253
IsTruncated returned by the ListMultipartUploads method is set to false by default ( #3682 )
...
* simplify a bit
* fix: IsTruncated returned by the ListMultipartUploads method is set to false by default
2022-09-15 00:27:15 -07:00
chrislu
21c0587900
go fmt
2022-09-14 23:06:44 -07:00
chrislu
c8645fd232
master: implement grpc VolumeMarkWritable
...
fix https://github.com/seaweedfs/seaweedfs/issues/3657
2022-09-14 23:05:30 -07:00
Ryan Russell
c30f6abb11
refactor(command_ec_rebuild): rebultErr -> rebuildErr ( #3679 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:10:27 -07:00
Ryan Russell
12914af4d8
Character readability ( #3678 )
...
* refactor(pb): `quote_charactoer` -> `quote_character`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(volume_server): `quoteCharactoer` -> `quoteCharacter`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 13:09:53 -07:00
Ryan Russell
3429cbe3fc
refactor(webdav_server): modifiledTime -> modifiedTime ( #3676 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:20:27 -07:00
Ryan Russell
d5239ab983
refactor(server): suppported -> supported ( #3677 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:16:39 -07:00
Ryan Russell
bd2dc6d641
refactor(shell): Decending -> Descending ( #3675 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:06:48 -07:00
Ryan Russell
dfbd8efd26
refactor(command_ec_decode): exisitngEcIndexBits -> `existingEcInde… ( #3674 )
...
refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:02:33 -07:00
Ryan Russell
f2dde2b1da
refactor(ItemList): Incluseive -> Inclusive ( #3673 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 12:00:16 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements ( #3672 )
...
* refactor(net_timeout): `listner` -> `listener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(filer): `localPublicListner` -> `localPublicListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(command): `masterLocalListner` -> `masterLocalListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor(net_timeout): `ipListner` -> `ipListener`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:59:55 -07:00
Ryan Russell
a8d7615eec
refactor(filer_client): EachEntryFunciton -> EachEntryFunction ( #3671 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:45 -07:00
Ryan Russell
c4363c4b5e
refactor(command_fs_rm): entiries -> entries ( #3670 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:27 -07:00
Ryan Russell
a36d7b77be
refactor(cipher_test): plantext -> plaintext ( #3669 )
...
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:24:09 -07:00
Ryan Russell
72d8a9f9a8
refactor(exclusive_locker): Interval readability batch of updates ( #3668 )
...
* refactor(filechunk_manifest): `localProcesed` -> `localProcessed`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `SafeRenewInteval` -> `SafeRenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `InitLockInteval` -> `InitLockInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
* refactor: `RenewInteval` -> `RenewInterval`
Signed-off-by: Ryan Russell <git@ryanrussell.org>
Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 10:29:55 -07:00
Ryan Russell
e22335ba78
refactor(queue_unbounded): inbountLen -> inboundLen ( #3666 )
2022-09-14 10:12:45 -07:00
Ryan Russell
d54eb9966f
refactor: Directory readability ( #3665 )
2022-09-14 10:11:31 -07:00
Ryan Russell
d734fff322
docs: replicte -> replicate ( #3664 )
2022-09-14 10:01:18 -07:00
Ryan Russell
dfaa602661
refactor(notification_kafka): parition -> partition ( #3663 )
2022-09-14 09:15:21 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… ( #3662 )
2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original ( #3661 )
2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer ( #3654 )
2022-09-14 06:33:00 -07:00
Ryan Russell
c9f573eed8
docs: indicates readability comment fix ( #3659 )
2022-09-14 06:31:07 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename ( #3658 )
2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix ( #3655 )
2022-09-14 04:53:04 -07:00
chrislu
58d18b68d8
rename variable
2022-09-14 00:06:54 -07:00
Konstantin Lebedev
f8ef25099c
Removing chunks on failed to write to replicas ( #3591 )
...
* Removing chunks on failed to write to replicas
https://github.com/seaweedfs/seaweedfs/issues/3578
* put with in the util.Retry
* just purge on any errors
2022-09-13 22:33:35 -07:00
chrislu
0baf3d38c2
adjust logs
2022-09-13 10:33:28 -07:00
chrislu
ce55922884
fixing
2022-09-12 09:43:54 -07:00
Konstantin Lebedev
721c6197f9
skip deltaBeat if dn is zero ( #3630 )
...
* skip deltaBeat
https://github.com/seaweedfs/seaweedfs/issues/3629
* fix GrpcPort
* skip url :0
* skip empty DataCenter or Rack
* skip empty heartbeat Ip
* dell msg add DataCenter
* comment todo
* fix
2022-09-11 22:31:53 -07:00
chrislu
b834027c5a
refactor
2022-09-11 21:53:15 -07:00
chrislu
d8ca7d34fe
3.27
2022-09-11 19:47:53 -07:00
chrislu
22064c3425
mount: ensure ordered file handle lock and unlock
2022-09-11 19:44:34 -07:00
chrislu
b9112747b5
volume server: synchronously report volume readonly status to master
...
fix https://github.com/seaweedfs/seaweedfs/issues/3628
2022-09-11 19:29:10 -07:00
chrislu
6690236754
rename variables
2022-09-11 19:18:42 -07:00
chrislu
4af8c427eb
refactor
2022-09-10 16:56:03 -07:00
chrislu
4957d8eec6
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-10 15:36:02 -07:00
chrislu
b1501f61dd
Revert "a new test case"
...
This reverts commit eb7cf3de81
2022-09-10 15:35:42 -07:00
Eric Yang
ddd6bee970
ADHOC: Volume fsck use a time cutoff param ( #3626 )
...
* ADHOC: cut off volumn fsck
* more
* fix typo
* add test
* modify name
* fix comment
* fix comments
* nit
* fix typo
* Update weed/shell/command_volume_fsck.go
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-10 15:29:17 -07:00
chrislu
2c6b68b40e
simplify a bit
2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a
[sync] override amz storage class, None to delete ( #3639 )
...
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636
* use empty string to delete
* without nil check
2022-09-10 14:15:42 -07:00
chrislu
3cb914f7e1
avoid dead lock
2022-09-10 11:26:19 -07:00
chrislu
ebef579ebb
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-10 11:22:18 -07:00
chrislu
576c113c59
replace PR https://github.com/seaweedfs/seaweedfs/pull/3621
...
replace https://github.com/seaweedfs/seaweedfs/pull/3621
2022-09-10 11:22:16 -07:00
Brian
4e3e2b1b82
Add option in volume.fix.replication to only fix under-replication and not delete volumes ( #3640 )
2022-09-10 08:05:28 -07:00
chrislu
205ecb5d03
skip directories under ".uploads" directory
...
related to https://github.com/seaweedfs/seaweedfs/issues/3636
skipping all directories under ".uploads" directory.
2022-09-09 22:57:38 -07:00
chrislu
fc4208d128
volume.balance: default to balance ALL_COLLECTIONS
2022-09-09 16:06:05 -07:00
gfx
48db56ddad
arangodb s3 bucket name compatibility ( #3588 )
...
* Update arangodb_store.go
* update readme, properly escape queries, add name patching
* use underscore
* use underscore
* better comment
* fix readme
Co-authored-by: a <a@a.a>
2022-09-09 09:43:42 -07:00
VictorSFF
10d545060f
mount: fix new inode allocation ( #3633 )
2022-09-09 06:32:19 -07:00
famosss
41af5a6ee5
simplify a bit ( #3631 )
2022-09-08 21:01:21 -07:00
chrislu
64e75a286e
simplify a bit
2022-09-08 19:07:57 -07:00
chrislu
d14d029c73
reduce v.dataFileAccessLock lock scope
2022-09-08 19:05:31 -07:00
chrislu
406a80da4b
adjust variable names
2022-09-08 18:54:16 -07:00
chrislu
2bfc8970d2
refactor: move ReadNeedleDataInto into volume_read.go
2022-09-08 18:54:02 -07:00
chrislu
9b084d4c88
purge tcp implementation
2022-09-08 18:03:43 -07:00
chrislu
eb7cf3de81
a new test case
2022-09-08 14:04:42 -07:00
chrislu
4260804613
volume.balance: avoid moving out volume with max=1
2022-09-08 14:04:21 -07:00
chrislu
d653c5f811
unused
2022-09-08 12:27:53 -07:00
chrislu
03e22ddb43
rename
2022-09-08 12:06:03 -07:00
Konstantin Lebedev
31d2f77ceb
refactor https://github.com/seaweedfs/seaweedfs/pull/3616 ( #3625 )
2022-09-07 23:23:33 -07:00
chrislu
7de112943b
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-07 20:14:11 -07:00
chrislu
7e52b1e195
return err for processRangeRequest()
2022-09-07 20:14:09 -07:00
famosss
449582343f
fix:Sometimes a nil pointer exception is thrown ( #3618 )
2022-09-07 18:57:13 -07:00
Konstantin Lebedev
695e3a909c
avoid data race on currentDiskUsage.maxVolumeCount ( #3592 )
...
https://github.com/seaweedfs/seaweedfs/issues/3590
2022-09-07 09:51:15 -07:00
Konstantin Lebedev
cca45b02a2
avoid data race on calc freeVolumeSlotCount ( #3594 )
...
https://github.com/seaweedfs/seaweedfs/issues/3593
2022-09-07 09:50:17 -07:00
famosss
9678fc2106
fix: volume heartbeat processing error ( #3616 )
2022-09-07 09:48:51 -07:00
famosss
5ff33eb558
fix: volume Deadlock when exception is thrown ( #3613 )
...
* fix: volume Deadlock when exception is thrown
* adjust log level
2022-09-06 23:56:29 -07:00
Eric Yang
b324a6536c
ADHOC: add read needle meta grpc ( #3581 )
...
* ADHOC: add read needle meta grpc
* add test
* nit
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-09-06 23:51:27 -07:00
chrislu
c07ab9c060
3.26
2022-09-06 08:26:20 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket ( #3609 )
...
https://github.com/seaweedfs/seaweedfs/issues/3607
2022-09-06 08:20:23 -07:00
Konstantin Lebedev
3c3682fcce
more log detail for upload err and deleting ( #3577 )
2022-09-06 08:19:13 -07:00
Konstantin Lebedev
01a25755c5
[filer] masterClient updates volume adjust log level ( #3610 )
2022-09-06 07:26:59 -07:00
LHHDZ
bc629665de
fix bug due to data racing on VidMap ( #3606 )
2022-09-05 20:00:16 -07:00
Konstantin Lebedev
9b2b7d4f5a
avoid data race on GoCDKPubSub.topic ( #3596 )
2022-09-05 07:57:44 -07:00
Konstantin Lebedev
916673ae24
do not print diskInfos with doVolumeCheckDisk in error log ( #3598 )
2022-09-05 07:49:26 -07:00
Konstantin Lebedev
5307cadb32
[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF ( #3597 )
2022-09-05 07:47:43 -07:00
chrislu
5b38f22e6e
3.25
2022-09-04 22:45:55 -07:00
famosss
131d389fc4
adjust log level ( #3589 )
2022-09-04 22:37:32 -07:00
chrislu
f4abefc109
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 22:23:31 -07:00
chrislu
39340f7e42
cloud tier: s3 consume all read response body
...
fix https://github.com/seaweedfs/seaweedfs/issues/3584
2022-09-04 22:23:28 -07:00
chrislu
7c6324b114
adjust log level
2022-09-04 22:21:24 -07:00
gfx
bf844d8e46
Fix crash in arangodb filer when attempting to access a deleted bucket ( #3587 )
...
* Update helpers.go
* Update arangodb_store_bucket.go
2022-09-04 20:32:10 -07:00
gfx
3621987727
Correctly pass arangodb driver Transaction ID into the context ( #3586 )
...
Update arangodb_store.go
2022-09-04 19:29:52 -07:00
chrislu
8c5759a06d
minor optimization
2022-09-04 18:50:45 -07:00
chrislu
70777d9b9f
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-09-04 18:26:51 -07:00
chrislu
cb6cf331ca
filer.backup and filer.sync: include headers during backup and sync
...
fix https://github.com/seaweedfs/seaweedfs/issues/3532
2022-09-04 18:26:36 -07:00
famosss
7f56a40553
Turn down the 404 log level ( #3585 )
2022-09-04 18:17:01 -07:00
Brian
a28b668647
Added ability to change replication settings upon volume.tier.move ( #3583 )
2022-09-04 16:47:21 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync ( #3579 )
...
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
2022-09-02 23:03:23 -07:00
Konstantin Lebedev
853880bd83
[filer] DATA RACE on signal_handling fixed ( #3575 )
2022-09-02 06:20:19 -07:00
chrislu
a112cbe44c
filer: rocksdb store report not found correctly
...
fix https://github.com/seaweedfs/seaweedfs/discussions/3555
2022-09-01 15:42:59 -07:00
chrislu
d1bb23645e
Revert "avoid data race on grace.hooks ( #3572 )"
...
This reverts commit c37d6fc01a
2022-09-01 13:08:34 -07:00
Konstantin Lebedev
c37d6fc01a
avoid data race on grace.hooks ( #3572 )
...
https://github.com/seaweedfs/seaweedfs/issues/3564
2022-09-01 10:34:06 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket ( #3571 )
...
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552
* copy localSocket
2022-09-01 10:33:23 -07:00
Konstantin Lebedev
90d55cd179
avoid data race read fileChunksSize ( #3570 )
...
https://github.com/seaweedfs/seaweedfs/issues/3557
2022-09-01 10:31:57 -07:00
Konstantin Lebedev
82c2ce74db
more error details in the log ( #3568 )
...
* more error details in the log
https://github.com/seaweedfs/seaweedfs/issues/3567
* format message
2022-09-01 09:04:15 -07:00
chrislu
10efdc7aab
align memory
2022-09-01 09:02:44 -07:00
Konstantin Lebedev
803ca3c958
avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch ( #3566 )
...
https://github.com/seaweedfs/seaweedfs/issues/3565
2022-09-01 09:01:47 -07:00
chrislu
bcf35876d1
add more logs
2022-08-31 23:16:05 -07:00
dependabot[bot]
97d69d5336
Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 ( #3541 )
...
* Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0
Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud ) from 0.25.0 to 0.26.0.
- [Release notes](https://github.com/google/go-cloud/releases )
- [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0 )
---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update code
* more code fix
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2022-08-31 10:16:49 -07:00
Patrick Schmidt
a73e177ecf
Add an End-to-End workflow for FUSE mount ( #3562 )
...
* Add an e2e workflow to test FUSE mount
* Fix deadlocks during concurrent r/w
2022-08-31 09:27:53 -07:00
chrislu
4a4ef3cc3c
upload_content: upload close response.Body
2022-08-31 00:25:28 -07:00
chrislu
3d8b8ffced
refactor
2022-08-31 00:24:29 -07:00
chrislu
ef78631a7c
just a bit safer
2022-08-31 00:10:17 -07:00
chrislu
9c944377dc
minor
2022-08-31 00:09:32 -07:00
chrislu
a54f30b6c6
s3: close response body with copying object
2022-08-31 00:09:23 -07:00
chrislu
7eb121d5ae
print process before do the work
2022-08-30 13:37:05 -07:00
Konstantin Lebedev
77098de69a
migrate from boltdb to bbolt ( #3554 )
2022-08-30 10:12:02 -07:00
chrislu
de5263364b
master client: use GetMaster() for latest master address
2022-08-30 09:52:06 -07:00
chrislu
e583dbdd2a
more detailed error
2022-08-30 09:43:17 -07:00
chrislu
d81db3c703
s3: fix configuring IAM for the same user
...
hi, how can I add bucket permission to a user now?
Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name.
Now I am getting error:
duplicate accessKey[ХХХХ], already configured in user[YYYY]
s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1
s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1
2022-08-30 09:37:52 -07:00
Konstantin Lebedev
ade94b0d0a
avoid race conditions access to SuperBlock.Version ( #3539 )
...
* avoid race conditions access to SuperBlock.Version
https://github.com/seaweedfs/seaweedfs/issues/3515
* superBlockAccessLock replace to sync.Mutex
2022-08-30 00:08:00 -07:00
chrislu
ae6292f9f0
rename variable
2022-08-30 00:07:15 -07:00
Konstantin Lebedev
e16dda88e4
avoid race conditions access to MasterClient.currentMaster ( #3538 )
...
https://github.com/seaweedfs/seaweedfs/issues/3510
2022-08-30 00:05:38 -07:00
Konstantin Lebedev
105702ebe0
avoid data race access to uploadReaderToChunks.uploadErr ( #3550 )
...
avoid data race access to uploadErr
https://github.com/seaweedfs/seaweedfs/issues/3549
2022-08-30 00:03:01 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay ( #3548 )
2022-08-30 00:02:09 -07:00
chrislu
570b488572
ensure memory alignment on different CPUs
2022-08-29 13:24:12 -07:00
Konstantin Lebedev
4966a3abc7
avoid race conditions access to growRequestCount ( #3537 )
...
https://github.com/seaweedfs/seaweedfs/issues/3511
2022-08-29 13:23:02 -07:00
Patrick Schmidt
00041f6e08
Fix corrupted mount reads after changes on fh ( #3536 )
2022-08-29 08:27:13 -07:00
chrislu
b7a887fea1
3.24
2022-08-28 21:53:13 -07:00
Rain Li
089446f5b1
Fix reload page before all file uploaded bug ( #3534 )
2022-08-28 07:51:47 -07:00
chrislu
cc0c8c5f81
simplify
2022-08-27 00:21:57 -07:00
chrislu
3741779144
minor
2022-08-27 00:20:58 -07:00
chrislu
580a7fa685
adjust for data race
...
fix https://github.com/seaweedfs/seaweedfs/issues/3512
2022-08-27 00:19:13 -07:00
chrislu
87b70a6809
clean up
2022-08-27 00:09:04 -07:00
chrislu
27c7611ebf
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 23:47:39 -07:00
chrislu
9920d65bc0
gateway to remote object store: adjust upload concurrency
2022-08-26 23:47:37 -07:00
chrislu
c839ce1b19
s3 sink use s3 upload manager
...
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-26 23:47:12 -07:00
famosss
dc4037925d
fix: Build DeletedVids before reset dn's children ( #3530 )
2022-08-26 22:52:08 -07:00
chrislu
3f3a1341d8
make CodeQL happy
2022-08-26 17:09:11 -07:00
chrislu
301b49b63f
atomic operation
2022-08-26 17:04:11 -07:00
chrislu
57a46f46a0
nano level precision
2022-08-26 16:55:15 -07:00
chrislu
8dae81c5ed
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-26 16:49:51 -07:00
chrislu
e0f4366f4c
more detailed error
...
related to https://github.com/seaweedfs/seaweedfs/issues/3528
2022-08-26 16:49:48 -07:00
Patrick Schmidt
5df105b1f9
Fix a few data races when reading files in mount ( #3527 )
2022-08-26 16:41:37 -07:00
chrislu
f5156cf3a8
rename varaible
2022-08-26 10:23:42 -07:00
Konstantin Lebedev
e90ab4ac60
avoid race conditions for OnPeerUpdate ( #3525 )
...
https://github.com/seaweedfs/seaweedfs/issues/3524
2022-08-26 10:18:49 -07:00
Konstantin Lebedev
4f7a1f67cd
avoid race conditions for diskLocation.MaxVolumeCount ( #3526 )
2022-08-26 08:41:42 -07:00
chrislu
6f483a4d36
atomic add
...
fix https://github.com/seaweedfs/seaweedfs/issues/3514
2022-08-25 22:20:34 -07:00
chrislu
7394f7feee
minor: adjust ordering
2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1
minor clean up
2022-08-25 00:19:08 -07:00
Patrick Schmidt
2930263dfd
Fix race conditions during in-flight size checks ( #3505 )
2022-08-24 20:03:34 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races ( #3503 )
2022-08-24 09:49:05 -07:00
Konstantin Lebedev
f7aeb06544
s3: report metadata if the directory is explicitly created ( #3498 )
...
* replace mkdir to mkFile
* ContentLength must be zero
* revert mkDir
* Seaweedfs-Is-Directory-Key return metadata
2022-08-24 00:15:44 -07:00
Guo Lei
c57c79a0ab
optimiz commitig compact ( #3388 )
...
* optimiz vacuuming volume
* fix bugx
* rename parameters
* fix conflict
* change copyDataBasedOnIndexFile to an instance method
* close needlemap
* optimiz commiting Vacuum volume for leveldb index
* fix bugs
* fix leveldb loading bugs
* refactor
* fix leveldb loading bug
* add leveldb recovery
* add test case for levelDB
* modify test case to cover all the new branches
* use one tmpNm instead of two instances
* refactor
* refactor
* move setWatermark to the end
* add test for watermark and updating leveldb
* fix error logic
* refactor, add test
* check nil before close needlemapeer
add test case
fix metric bug
* add tests, fix bugs
* adjust log level
remove wrong test case
refactor
* avoid duplicate updating metric for leveldb index
2022-08-23 23:53:35 -07:00
chrislu
10414fd81c
ping timeout at 15 seconds
...
this 72 minute timeout setting seems unreasonably long
15 seconds is around the time when a new raft leader should be elected.
2022-08-23 23:28:16 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long ( #3398 )
...
* remove old raft servers if they don't answer to pings for too long
add ping durations as options
rename ping fields
fix some todos
get masters through masterclient
raft remove server from leader
use raft servers to ping them
CheckMastersAlive for hashicorp raft only
* prepare blocking ping
* pass waitForReady as param
* pass waitForReady through all functions
* waitForReady works
* refactor
* remove unneeded params
* rollback unneeded changes
* fix
2022-08-23 23:18:21 -07:00
Guo Lei
762dc219a6
fix deletecount bug ( #3499 )
2022-08-23 08:53:24 -07:00
chrislu
21a933c932
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-23 01:52:32 -07:00
chrislu
f0b4a7659a
fix test
2022-08-23 01:52:29 -07:00
Patrick Schmidt
cda8cc22bc
Implement lseek syscall in FUSE ( #3491 )
...
See the man page of lseek:
https://man7.org/linux/man-pages/man2/lseek.2.html
2022-08-23 01:47:18 -07:00
chrislu
bf88006037
format
2022-08-23 01:20:45 -07:00
chrislu
9fce75607d
s3: report http.StatusOK if the directory is explicitly created
...
fix https://github.com/seaweedfs/seaweedfs/issues/3457
2022-08-23 01:16:46 -07:00
famosss
911475526c
fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil ( #3497 )
2022-08-22 23:54:51 -07:00
famosss
7eb15b1969
fix:Handle preflight cors requests ( #3496 )
2022-08-22 23:39:34 -07:00
chrislu
676e27c589
shell: stop long running jobs if lock is lost
2022-08-22 14:12:23 -07:00
chrislu
601ba5fb68
master: disable locking if not leader
2022-08-22 14:11:48 -07:00
chrislu
57e7582c36
refactoring
2022-08-22 14:11:13 -07:00
chrislu
7ea2c24c5d
metadata processing use retry instead of retryForEver
2022-08-22 11:25:40 -07:00
Patrick Schmidt
f875031f06
Reuse readDataByFileHandle in Read call ( #3482 )
2022-08-22 08:24:06 -07:00
famosss
bdba3da2e4
Handle preflight cors requests ( #3481 )
2022-08-22 08:21:38 -07:00
chrislu
f4b52d4c52
fix format
2022-08-22 00:20:21 -07:00
chrislu
b4131a3a72
filer.remote.sync: parallelize uploading to remove object storage
2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561
refactor
2022-08-22 00:00:51 -07:00
chrislu
3c8e95eeb3
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-21 23:26:09 -07:00
chrislu
8bac903d29
mount: add locking to prevent nil pointer
...
fix https://github.com/seaweedfs/seaweedfs/issues/3479
2022-08-21 23:26:06 -07:00
qzh
74b53729e1
feat(weed.move): add a speed limit parameter of moving files ( #3478 )
...
* feat(weed.move): add a speed limit parameter of moving files
* fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-21 23:08:31 -07:00
chrislu
7807f6641a
minor
2022-08-21 22:54:56 -07:00
chrislu
c4e862e908
3.23
2022-08-21 19:13:26 -07:00
Guo Lei
81e7e6ceb3
the statistical methord for fileCount should stay same ( #3477 )
...
the statistical methord for fileCount between should stay same
They all equal to the entry count in .idx file.
relate to commit(c7892bc
2022-08-21 19:11:18 -07:00
chrislu
63fbf281c7
remove same file copying rage limitation
2022-08-21 17:16:21 -07:00
Patrick Schmidt
3a75d7f7aa
Implement copy_file_range syscall in FUSE ( #3475 )
...
See the man page of copy_file_range:
https://man7.org/linux/man-pages/man2/copy_file_range.2.html
2022-08-21 17:11:43 -07:00
chrislu
fef9c6a520
filer: do not always save files in "/etc" folder to filer store
...
fix https://github.com/seaweedfs/seaweedfs/issues/3476
2022-08-21 17:02:16 -07:00
chrislu
2762154130
fix compilation
2022-08-21 16:09:23 -07:00
chrislu
fb5808e0c3
EC: with multiple volume locations, the ec rebuilding may fail
2022-08-21 15:04:50 -07:00
chrislu
5790d01c6f
EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations
...
related to https://github.com/seaweedfs/seaweedfs/issues/3459
2022-08-21 14:52:37 -07:00
chrislu
8b3429858d
align memory for atomic read/write
2022-08-21 12:20:27 -07:00
Patrick Schmidt
2ef6ab998c
Avoid race conditions with current filer address ( #3474 )
...
When multiple filer requests are in-flight and the current filer
disappears and a new one is selected by the first goroutine, then
there can be a lot of race conditions while retrieving the current
filer.
Therefore, load/save the current filer index atomically.
2022-08-21 12:18:13 -07:00
Patrick Schmidt
f49a9297c2
Fix hanging reads in chunk cacher ( #3473 )
...
Sometimes when an unexpected error occurs the cacher would set an
error and return. However, it would not broadcast the condition
signal in that case, therefore leaving the goroutine that runs
readChunkAt stuck forever.
I figured that the condition is unnecessary because readChunkAt is
acquiring a lock that is still held by the cacher goroutine anyway.
Callees of startCaching have to wait for a WaitGroup which makes sure
that readChunkAt can't acquire the lock before startCaching.
This way readChunkAt can execute normally and check for the error.
2022-08-21 11:54:02 -07:00
chrislu
388f82f322
minor
2022-08-21 11:49:29 -07:00
chrislu
77e4b1376e
refactoring
2022-08-21 11:35:54 -07:00
Patrick Schmidt
3f758820c1
Fix FUSE server buffer leaks in file gaps ( #3472 )
...
* Fix FUSE server buffer leaks in file gaps
This change zeros read buffers when encountering file gaps during
file/chunk reads in FUSE mounts.
It prevents leaking internal buffers of the FUSE server which could
otherwise reveal metadata, directory listings, file contents and
other data related to FUSE API calls.
The issue was that buffers are reused, but when a file gap was found
the buffer was not zeroed accordingly and the existing data of the
buffer was kept and returned.
* Move zero logic into its own method
2022-08-21 11:33:58 -07:00
chrislu
c7892bc7c4
volume: file counter should be all files
...
address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228
2022-08-20 23:35:31 -07:00
chrislu
f7e0a65e75
retry for all errors
2022-08-20 23:34:45 -07:00
chrislu
649555b223
adjust log
2022-08-20 23:34:07 -07:00
chrislu
973f6dd162
refactoring
2022-08-20 22:38:15 -07:00
chrislu
65ff7198fe
adjust type
2022-08-20 22:37:24 -07:00
chrislu
28b862f45f
use util.Retry to retry
2022-08-20 22:03:27 -07:00
chrislu
3bf8e772f8
webdav: retryable data chunk upload
2022-08-20 19:18:12 -07:00
chrislu
f8fa430257
filer: retryable data chunk upload
2022-08-20 19:15:44 -07:00
chrislu
4081d50607
filer sink: retryable data chunk uploading
2022-08-20 19:09:15 -07:00
chrislu
409f39390d
mount: retryable data chunk uploading
2022-08-20 19:04:36 -07:00
chrislu
6c8822f269
filer.copy: retryable file part upload
2022-08-20 18:59:57 -07:00
chrislu
a3553da7f7
add fileId to UploadWithRetry return result
2022-08-20 18:54:59 -07:00
chrislu
d49d0a9fc2
filer.copy: retryable upload
2022-08-20 18:50:57 -07:00
chrislu
689b4ecdcc
remove unused collection and replication from upload result
2022-08-20 18:14:57 -07:00
chrislu
e3f40d538d
cleaner code
2022-08-20 17:51:30 -07:00
chrislu
9f50b310a0
remove unused code
2022-08-20 17:51:12 -07:00
chrislu
11f99836c3
filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
...
fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-19 23:00:56 -07:00
chrislu
fdd8c5d5e0
Filer if-modified-since check doesn't fire on "exact" #3467
...
fix https://github.com/seaweedfs/seaweedfs/issues/3467
2022-08-19 22:21:51 -07:00
chrislu
aac45f3e89
filer: retryable when error is not found
2022-08-19 03:36:15 -07:00
chrislu
58dd880322
minor
2022-08-18 23:51:08 -07:00
Eric Yang
b6f99c4d83
fix naming typo ( #3465 )
...
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-18 23:25:32 -07:00
chrislu
5702727db5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-18 02:39:27 -07:00
chrislu
4d72a1ce6e
s3: remove deleted collection during metrics reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3453
2022-08-18 02:39:07 -07:00
Guo Lei
3172c33fa6
fix redundant type from array ( #3462 )
2022-08-18 02:25:25 -07:00
chrislu
42c6e52513
s3: fix regression on HEAD directory operation
2022-08-18 02:13:58 -07:00
chrislu
4573c99ae3
fix tests
2022-08-18 00:15:46 -07:00
chrislu
2b580a7566
also migrate jsonpb
2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09
move proto package
2022-08-17 12:05:07 -07:00
chrislu
e259052a70
volume: fix metrics for volume request time
2022-08-17 04:29:31 -07:00
qzh
7fcfaf7bc9
fix(filer.sync): offset may be set to 0 ( #3451 )
...
* fix(filer.sync): initializing the offset is related to the path
* fix(filer.sync): the offset maybe to be set to 0.
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 23:43:52 -07:00
qzh
400f0c3e5d
fix(filer.sync): initializing the offset is related to the path ( #3450 )
...
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 21:56:47 -07:00
chrislu
fa4d0093e1
3.22
2022-08-15 16:48:23 -07:00
chrislu
93261f5199
Revert "Refactor for Sync method ( #3426 )"
...
This reverts commit 670cb759f8
2022-08-15 15:35:31 -07:00
chrislu
9c588d4010
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-15 10:03:56 -07:00
chrislu
96caf21d09
less verbose log
2022-08-15 10:03:52 -07:00
Andrey Triumfov
31faa6d43d
Remove duplicate slashes in object path to prevent 500 errors ( #3442 )
2022-08-15 08:19:28 -07:00
chrislu
cb476a53ff
remove logs
2022-08-15 01:05:35 -07:00
chrislu
7c029b2183
3.21
2022-08-15 00:32:15 -07:00
Chris Lu
e88392b50f
Fix s3 pagination ( #3436 )
...
* Revert previous changes
* s3: use cursor to track tree traversal
fix https://github.com/seaweedfs/seaweedfs/issues/3166
* special cases for empty prefix and empty directory
* use constants
* address empty folder
* undo local changes
* fix IsTruncated
* adjust counting directories
* fix cases when prefix is a directory
* s3: handle directory object
works for
aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"
2022-08-15 00:30:19 -07:00
chrislu
7457c746f0
s3: fix aws s3api head-object
2022-08-14 23:52:35 -07:00
chrislu
789d9dbfe1
webdav: fix nil
...
fix https://github.com/seaweedfs/seaweedfs/issues/3440
2022-08-14 23:33:30 -07:00
Konstantin Lebedev
6199d9b923
rand start garbageThreshold and reset metric ReplicaPlacementMismatch ( #3396 )
...
* rand start garbageThreshold
reset metrics MasterReplicaPlacementMismatch
* vacuum through sleep
2022-08-12 13:11:09 -07:00
Konstantin Lebedev
4d4cd0948d
avoid infinite loop WaitUntilConnected() ( #3431 )
...
https://github.com/seaweedfs/seaweedfs/issues/3421
2022-08-11 15:03:26 -07:00
Eric Yang
986daecbfe
add upload response ( #3428 )
...
* add upload response
* fixing comments
Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>
2022-08-10 22:43:09 -07:00
chrislu
ae93c966d9
ensure memory is aligned
...
fix https://github.com/seaweedfs/seaweedfs/issues/3427
2022-08-10 22:27:13 -07:00
Rain Li
670cb759f8
Refactor for Sync method ( #3426 )
2022-08-10 08:21:57 -07:00
chrislu
662ec97602
mount: adjust on du stats reporting
...
fix https://github.com/seaweedfs/seaweedfs/issues/3424
2022-08-09 20:21:01 -07:00
Abirdcfly
b0633716b7
delete minor unreachable code ( #3423 )
...
Signed-off-by: Abirdcfly <fp544037857@gmail.com>
2022-08-09 01:24:43 -07:00
chrislu
0854171d22
3.20
2022-08-07 14:42:28 -07:00
chrislu
d8e69df295
adjust comments
2022-08-07 11:35:33 -07:00
chrislu
0aeec04c31
quicker to adapt to pattern change
2022-08-07 10:14:01 -07:00
chrislu
928d29af9e
fix wrong logic about reader isRandomMode()
2022-08-07 10:13:04 -07:00
chrislu
d3d52b1818
shell: fs.meta.load add quieter mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3414
2022-08-07 01:42:15 -07:00
chrislu
358ccb5e0e
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-07 01:34:36 -07:00
chrislu
67814a5c79
refactor and fix strings.Split
2022-08-07 01:34:32 -07:00
John W Higgins
3afda0c89c
Allow postgresql to use standard environment variables for connection ( #3413 )
2022-08-07 00:58:53 -07:00
chrislu
1a4bf0dcb5
filer.sync: parallelize the filer.sync
2022-08-07 00:56:15 -07:00
chrislu
02b38f894b
debug
2022-08-07 00:55:49 -07:00
chrislu
0e9478488d
filer.sync: fix when excluded paths is empty
2022-08-07 00:55:34 -07:00
chrislu
8a880a139d
mount: fix truncate operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/2609
2022-08-06 13:58:45 -07:00
LHHDZ
84ec68e11a
Add download speed limit support ( #3408 )
2022-08-05 01:16:42 -07:00
chrislu
b278bb24d3
mount: adjust df stats reporting when close to the limit
...
fix https://github.com/seaweedfs/seaweedfs/issues/3407
2022-08-05 00:31:41 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center ( #3405 )
...
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404
* GetDataCenter
* prefer same data center for ReplicationSource
* GetDataCenterId
* remove glog
2022-08-04 17:35:00 -07:00
Konstantin Lebedev
22181dd018
refactor FilerRequest metrics ( #3402 )
...
* refactor FilerRequest metrics
* avoid double count proxy
* defer to
2022-08-04 01:44:54 -07:00
chrislu
bd13a7968f
remove unused symlink resolving
2022-08-04 01:35:18 -07:00
chrislu
88945d9954
mount: do not follow soft link with xattr
2022-08-04 01:21:15 -07:00
chrislu
90db4b5a44
minor
2022-08-03 00:10:47 -07:00
Chris Lu
fc8035d672
Merge pull request #3394 from kmlebedev/metricsReplicatedWrite
...
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 13:31:01 -07:00
Konstantin Lebedev
3ffa4ba91d
clean
2022-08-02 01:30:38 +05:00
Konstantin Lebedev
2a5629dd28
align
2022-08-02 01:29:44 +05:00
Konstantin Lebedev
df9cc31a0a
refactor
2022-08-02 01:18:26 +05:00
Konstantin Lebedev
0a5d3c3cf6
rename to localWrite and replicatedWrite
2022-08-01 23:09:09 +05:00
chrislu
bd5c9904be
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2022-08-01 09:40:05 -07:00
chrislu
e26d1cf4ef
fix test
2022-08-01 09:40:01 -07:00
Konstantin Lebedev
fc65122766
rename to LoadAvg_1M
2022-08-01 21:32:21 +05:00
Konstantin Lebedev
ab50f21ad9
Detailed metrics VolumeServerRequestHistogram for writing to disk and replication
2022-08-01 21:16:57 +05:00
Konstantin Lebedev
2f85e31f5c
Merge branch 'master' into gentle_vacuum
2022-08-01 20:51:58 +05:00
Konstantin Lebedev
5209ebbeef
remove percent
2022-08-01 20:40:38 +05:00
Chris Lu
b59bc607bf
Merge pull request #3338 from kmlebedev/issues/3083
...
rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 08:23:10 -07:00
Konstantin Lebedev
78cbd8002f
revert Sleep
2022-08-01 20:21:23 +05:00
Konstantin Lebedev
cd5c7ad052
move to github.com/seaweedfs/seaweedfs
2022-08-01 16:36:32 +05:00
Konstantin Lebedev
634bc97840
fix conflicts
2022-08-01 14:47:00 +05:00
Konstantin Lebedev
3c75479e2b
Merge branch 'master' into gentle_vacuum
...
# Conflicts:
# weed/pb/messaging_pb/messaging.pb.go
# weed/pb/messaging_pb/messaging_grpc.pb.go
# weed/pb/s3_pb/s3.pb.go
# weed/pb/volume_server_pb/volume_server.pb.go
# weed/server/volume_grpc_vacuum.go
2022-08-01 14:45:22 +05:00
Konstantin Lebedev
c0d92f61a1
comment
2022-08-01 14:40:42 +05:00
Konstantin Lebedev
1d29f67c02
revert disk stats
2022-08-01 14:29:41 +05:00
Konstantin Lebedev
a98f6d66a3
rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping
2022-08-01 12:51:41 +05:00
chrislu
036566629a
filer.sync: fix synchronization logic in active-active mode
...
fix https://github.com/seaweedfs/seaweedfs/issues/3328
2022-08-01 00:06:18 -07:00
chrislu
303bd067b5
Revert "rename: delete source entry metadata only, skipping hard links"
...
This reverts commit 03466f955ehttps://github.com/seaweedfs/seaweedfs/issues/3386
2022-07-31 22:51:41 -07:00
chrislu
03c6f978f4
3.19
2022-07-31 17:16:51 -07:00
chrislu
afef014832
scaffold
2022-07-31 13:23:44 -07:00
Chris Lu
0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy
...
Add healthy indicator for raft status
2022-07-30 11:55:30 -07:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status
2022-07-30 19:34:26 +02:00
chrislu
cb096d1f53
minor
2022-07-30 09:52:52 -07:00
chrislu
0423d50106
wip note
2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a
moving
2022-07-29 23:39:11 -07:00
chrislu
13b9a52f80
fix deprecated functions
2022-07-29 01:34:39 -07:00
chrislu
be0ec7bc2f
fix test
2022-07-29 01:15:26 -07:00
chrislu
1f161f53dc
add back rack
2022-07-29 01:15:19 -07:00
chrislu
26dbc6c905
move to https://github.com/seaweedfs/seaweedfs
2022-07-29 00:17:28 -07:00
Konstantin Lebedev
2f0dda384d
vacuum show LA
2022-07-29 11:59:33 +05:00
chrislu
de3afd4199
fix rebasing
2022-07-28 23:25:46 -07:00
chrislu
06cd491abc
add message batch
2022-07-28 23:24:38 -07:00
chrislu
7576c244c4
add flatbuffer serde for message
2022-07-28 23:24:38 -07:00
chrislu
a3fae7cd16
add an empty placeholder for mq.topic.list
2022-07-28 23:24:38 -07:00
chrislu
aa4a22ad47
segment serde
2022-07-28 23:24:38 -07:00
chrislu
74f60f246f
dynamically connect to a filer
2022-07-28 23:24:38 -07:00
chrislu
9f479aab98
allocate brokers to serve segments
2022-07-28 23:24:38 -07:00
chrislu
bb01b68fa0
refactor
2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c
clean up
2022-07-28 23:24:38 -07:00
chrislu
8060fdcac5
remove old code
2022-07-28 23:24:38 -07:00
chrislu
f25e273e32
display data center and rack in cluster.ps
2022-07-28 23:22:52 -07:00
chrislu
68065128b8
add dc and rack
2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd
mq broker adds data center and rack
2022-07-28 23:22:07 -07:00
chrislu
87ccef72db
cluster.ps can list brokers
2022-07-28 23:22:07 -07:00
chrislu
ac66227798
renaming
2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8
renaming
2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17
rename to SeaweedMQ
2022-07-28 23:22:06 -07:00
Chris Lu
3e7e922d16
Merge pull request #3378 from vanhtuan0409/change-put-policy
...
PutUserAction should completely replace identity actions
2022-07-28 19:43:25 -07:00
chrislu
89e276de64
fix loop
2022-07-28 18:06:49 -07:00
chrislu
a458bee9f2
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-28 16:32:02 -07:00
chrislu
e0eda52c54
mount: ensure symlink parent directory is tracked
...
fix https://github.com/chrislusf/seaweedfs/issues/3373
2022-07-28 16:32:00 -07:00
Tuan Vuong
1d9a888be9
PutUserAction should completely replace identity actions
2022-07-28 17:02:56 +07:00
zzq09494
01b7aa79fa
fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted
2022-07-28 12:04:37 +08:00
chrislu
ca836568ac
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87
"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"
2022-07-27 12:12:40 -07:00
Chris Lu
b7e4e26778
Merge pull request #3371 from askeipx/issues/3341
...
delete disk_size metrics when collection deleted
2022-07-27 11:25:42 -07:00
Konstantin Lebedev
7e09a548a6
exclude directories to sync on filer
2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev
ac5ce31278
leave notion to refactor after prometheus upgrade
2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev
47c72e6f35
remove all (currently existing) collection volume metrics
2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev
709c83716c
delete disk_size metrics when collection deleted
2022-07-27 15:02:41 +05:00
chrislu
f401b996eb
mount: fix symlink size reporting
2022-07-26 23:12:25 -07:00
Konstantin Lebedev
c88ea31f62
fix RUnlock of unlocked RWMutex
2022-07-26 12:57:07 +05:00
Konstantin Lebedev
da9d3e8f6c
refactor
2022-07-26 11:56:45 +05:00
Konstantin Lebedev
046c3d5ad4
fix logic else brake
2022-07-26 11:47:11 +05:00
chrislu
475185fb72
3.18
2022-07-25 22:41:38 -07:00
chrislu
ec0edb1ac4
filer: fix wrong logic during read
2022-07-25 22:40:00 -07:00
guosj
cc7a4b0a6e
correct comment
2022-07-25 11:46:41 +08:00
chrislu
33cff61992
fix glog test
2022-07-24 19:13:54 -07:00
chrislu
dbe4849ffd
3.17
2022-07-24 18:46:50 -07:00
chrislu
b98c0b4bf4
disable symlink translation for now
2022-07-24 18:46:32 -07:00
chrislu
2dc5d4adac
disable symlink translation for now
2022-07-24 18:24:34 -07:00
chrislu
3d2bbf96d2
adjust glog format
2022-07-24 17:29:08 -07:00
chrislu
2807fe08fc
adjust glog format
2022-07-24 16:44:58 -07:00
chrislu
2b64916d3e
set status to fuse.OK
2022-07-24 15:30:55 -07:00
chrislu
7386cf6a06
mount: resolve symlink
2022-07-24 15:11:24 -07:00
chrislu
40089b6753
maintain node lookup counter
2022-07-24 13:43:11 -07:00
chrislu
d24db396cc
fix removing path from inode2path
2022-07-23 18:20:29 -07:00
chrislu
41eeb4deef
do not add new inode during link
2022-07-23 13:37:07 -07:00
chrislu
02c2d81cde
fix removePathFromInode2Path
2022-07-23 13:21:42 -07:00
chrislu
4d5dc55766
mount: better support hard links
2022-07-23 12:57:00 -07:00
chrislu
ebda0689ab
use linux default max name length
2022-07-23 10:51:06 -07:00
chrislu
64f3d6fb6e
metadata subscription uses client epoch
2022-07-23 10:50:28 -07:00
chrislu
2c8818351f
mount: comment out stuck fuse inode notify during renaming
2022-07-22 18:17:34 -07:00
chrislu
a5c94acd4f
mount: check possible nil entry case
2022-07-22 15:33:37 -07:00
Chris Lu
0716092b39
Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap
...
Solve the problem that `LookupFileId` lookup urls is empty due to lea…
2022-07-22 02:57:50 -07:00
LHHDZ
994a2dec78
fix vid_map_test tests
2022-07-22 17:54:57 +08:00
LHHDZ
58c02d6429
Solve the problem that LookupFileId lookup urls is empty due to leader switching
...
The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.
2022-07-22 17:22:38 +08:00
ningfd
58f2dd6740
change topology mapping definition, improve spelling
2022-07-22 17:18:56 +08:00
ningfd
6f882eb354
optimiz: master ui will render data in order
2022-07-22 14:37:49 +08:00
chrislu
7a6c559ab4
fix Change replication via volume.configure.replication by collection
...
fix https://github.com/chrislusf/seaweedfs/issues/3346
2022-07-21 22:01:05 -07:00
chrislu
fc8241fb5e
leveldb3: add instant dropping bucket
2022-07-21 18:48:51 -07:00
chrislu
c93f7ffa44
explicit bucket aware declaration
2022-07-21 18:23:53 -07:00
Konstantin Lebedev
3c42814b58
avoid deadlock
2022-07-21 17:15:10 +05:00
Konstantin Lebedev
93ca87b7cb
use safe onPeerUpdateDoneCns
2022-07-21 15:51:14 +05:00
chrislu
6147b61b0a
mount: fix concurrent map read and map write
...
fix https://github.com/chrislusf/seaweedfs/issues/3344
2022-07-21 01:38:26 -07:00
Chris Lu
9ec0d1caaa
Merge pull request #3337 from guol-fnst/loading_volume
2022-07-20 09:06:06 -07:00
Konstantin Lebedev
7875470e74
onPeerUpdateGoroutineCount use int32
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
c5189c343b
remove ticker
...
update the topology before each file
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
e2d991d8d0
ticker.Stop
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
de4fcc0e2c
sync update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
fa88dff7ce
update otherNodes
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
72dca31cfa
fix update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
884ffbafee
clouse background update
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
39eaf426f8
fix TestVolumeServerEvacuate
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b5e5f6f55a
update topologyInfo
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
867269cdcf
help rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6f764e1014
volume server evacuate from rack
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
ba0e3ce5fa
volume server evacuate to target server
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
b6471ecd75
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 18:40:50 +05:00
Konstantin Lebedev
6c390851e7
fix design
2022-07-20 18:08:12 +05:00
Tuan Vuong
312e13416b
filter duplicated action
2022-07-20 16:43:50 +07:00
guol-fnst
ac694f0c8f
rename parameter and reuse functions
...
rename milestone to watermark
2022-07-20 17:00:40 +08:00
Chris Lu
4820b594a7
Merge pull request #3298 from kmlebedev/evacuateTargetServer
...
volume server evacuate to target server
2022-07-19 13:56:49 -07:00
Konstantin Lebedev
d3f7c09c03
remove ticker
...
update the topology before each file
2022-07-20 00:54:23 +05:00
Konstantin Lebedev
11e393dbe7
err msg with duplicated local subscription detected move to log level 1
...
https://github.com/chrislusf/seaweedfs/issues/3320
2022-07-20 00:45:13 +05:00
Konstantin Lebedev
f6a966b4fc
add waiting log message
2022-07-20 00:31:57 +05:00
Konstantin Lebedev
6cfbfb0849
check for ping before deleting raft server
...
https://github.com/chrislusf/seaweedfs/issues/3083
2022-07-20 00:04:12 +05:00
chrislu
2ae3f812f8
minor
2022-07-19 11:43:31 -07:00
Chris Lu
9667588af0
Merge pull request #3318 from kmlebedev/issues/3310
...
Use fallback if urls are not found
2022-07-19 11:38:43 -07:00
Chris Lu
f0d560060f
Merge pull request #3336 from kmlebedev/raftSrvRemove
...
raftServerRemoval typo and logs
2022-07-19 11:32:17 -07:00
guol-fnst
91285bb51d
remove uncessary code
2022-07-19 20:00:44 +08:00
guol-fnst
b9256e0b34
optimiz
2022-07-19 16:58:53 +08:00
Konstantin Lebedev
f419d5643a
fix typo
...
add remove logs
2022-07-19 11:50:52 +05:00
guol-fnst
bec9f79659
use snapshot
2022-07-19 11:14:41 +08:00
Konstantin Lebedev
d422e7769c
ticker.Stop
2022-07-18 16:38:19 +05:00
Konstantin Lebedev
73a0dea16b
sync update topologyInfo
2022-07-18 16:27:02 +05:00
chrislu
56ec89625a
3.16
2022-07-18 00:53:05 -07:00
Konstantin Lebedev
2b4112e462
update otherNodes
2022-07-18 11:32:28 +05:00
Konstantin Lebedev
3c2774ec3d
fix update topologyInfo
2022-07-18 01:46:31 +05:00
Konstantin Lebedev
7b1497ee63
Use BackoffSchedule for getLookupFileId
2022-07-15 16:05:35 +05:00
Konstantin Lebedev
01996bccf8
Use fallback if urls are not found
2022-07-15 15:29:15 +05:00
Konstantin Lebedev
4d5144e50d
clouse background update
2022-07-15 13:51:08 +05:00
chrislu
1db012485f
mount: fix bug during busy writes
...
fix https://github.com/chrislusf/seaweedfs/issues/3315
2022-07-15 01:03:17 -07:00
chrislu
aca20cd9f4
more logging related to filer metadata subscription
2022-07-15 00:42:13 -07:00
Chris Lu
abeb7f65f8
Merge pull request #3314 from shichanglin5/fix_ETag
...
File upload should succeed should return `ETag` instead of `Etag`
2022-07-14 15:35:41 -07:00
chrislu
fa61074513
add clientId logging
2022-07-14 12:27:34 -07:00
chrislu
fbd8f868a1
filer may have trouble to re-connect
...
clientId is used twice: one for local metadata subscription, one for combined metadata subscription.
2022-07-14 12:15:31 -07:00
LHHDZ
c08f6d0ff7
File upload should succeed should return ETag instead of Etag
...
at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`
2022-07-14 19:45:01 +08:00
Chris Lu
957c9a2eb2
Merge pull request #3309 from guol-fnst/loading_volume
...
optimiz concurrency
2022-07-13 23:12:24 -07:00
guol-fnst
300b383cdf
use 10 or numCPU workers if env is not found
2022-07-14 14:06:46 +08:00
guol-fnst
308a48c0c2
optimiz concurrency
...
user can customize number of workers via env "GOMAXPROCS"
2022-07-14 09:57:25 +08:00
Chris Lu
1a130125de
Merge pull request #3308 from guo-sj/check_duplicate_ak
2022-07-13 09:23:37 -07:00
guosj
354b7bdff0
replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()
2022-07-13 22:49:03 +08:00
chrislu
289402a741
writer pattern: similar changes to reader pattern
2022-07-13 02:30:44 -07:00
guosj
ab1b9697e6
supplement check duplicate accesskey
2022-07-13 17:28:20 +08:00
chrislu
6e90f7bdd0
detect sequential mode with a counter
2022-07-13 02:20:03 -07:00
chrislu
578c8aea0e
typo
2022-07-13 02:18:31 -07:00
chrislu
1451b389a4
Avoid fatal error: sync: Unlock of unlocked RWMutex
...
fix https://github.com/chrislusf/seaweedfs/issues/3306
2022-07-13 00:58:15 -07:00
石昌林
392da4e038
Make the prompt information clearer
2022-07-13 11:39:45 +08:00
石昌林
ad5e8f68ec
Check whether there is a duplicate accessKey when modifying iam
2022-07-13 11:36:11 +08:00
Chris Lu
fba5219dab
Merge pull request #3303 from guol-fnst/loading_volume
...
speeding up loading volumes
2022-07-12 03:01:24 -07:00
Konstantin Lebedev
6622240df7
fix TestVolumeServerEvacuate
2022-07-12 14:56:34 +05:00
chrislu
860c207c94
typo
2022-07-12 02:56:19 -07:00
Konstantin Lebedev
8372721a62
update topologyInfo
2022-07-12 13:47:21 +05:00
guol-fnst
26313060a3
speeding up loading volumes
2022-07-12 16:19:33 +08:00
Konstantin Lebedev
ee95d23a22
help rack
2022-07-12 11:56:58 +05:00
Konstantin Lebedev
087fa1347f
volume server evacuate from rack
2022-07-12 11:33:08 +05:00
Chris Lu
de97ba94c3
Merge pull request #3302 from guo-sj/fix_put_user_policy
2022-07-11 21:15:37 -07:00
guosj
b3076909d1
refactor code
2022-07-12 11:25:36 +08:00
Konstantin Lebedev
4236c36599
volume server evacuate to target server
2022-07-11 16:58:15 +05:00
Chris Lu
7929a50327
Merge pull request #3296 from guo-sj/fix_put_user_policy
...
refine PutUserPolicy
2022-07-11 01:49:25 -07:00
guosj
7dbf19d09a
refine PutUserPolicy
2022-07-11 16:38:41 +08:00
chrislu
37578929d4
fix compilation
2022-07-11 01:30:00 -07:00
chrislu
93ccc6e05f
3.15
2022-07-11 00:50:07 -07:00
chrislu
f87952a411
remove locks
2022-07-11 00:25:20 -07:00
guosj
5469efa336
close SkipCheckParentDirectory flag
2022-07-11 09:44:48 +08:00
chrislu
3d0defa663
fix compilation
2022-07-08 19:16:58 -07:00
chrislu
6f00c7f061
comments
2022-07-08 19:07:31 -07:00
chrislu
d685b94102
mount: rename also invalidate source inode
2022-07-08 19:07:03 -07:00
chrislu
48382676d2
fix filtering by volume id
2022-07-08 10:29:24 -07:00
liubaojiang
4e83e92156
add subscriber clientId if it is the first time
2022-07-08 20:27:25 +08:00
guosj
5b3192344d
fix iam CreatePolicy error
2022-07-08 17:42:21 +08:00
chrislu
e2ecf137a2
fix resetting fh.reader
2022-07-08 01:04:15 -07:00
chrislu
185332b91f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-07-08 00:29:41 -07:00
chrislu
28add5a534
mount: fix racing conditions
...
prevent wrong reading when the SingleChunkCacher is started, but not finished yet
2022-07-08 00:29:39 -07:00
duanhongyi
1ceab96aba
filer tikv support tls
2022-07-08 14:23:06 +08:00
chrislu
a85ed3fe8f
minor
2022-07-07 22:27:24 -07:00
chrislu
2ea18cdcc8
remove dead code
2022-07-07 22:26:03 -07:00
chrislu
b7de5c6c43
shell: remove unused filer parameter from shell.toml
2022-07-07 15:01:23 -07:00
chrislu
f2f0482dd3
mount: random read also try to use the local cache first
2022-07-07 11:50:28 -07:00
chrislu
05c53820b9
mount: file handler release memory
2022-07-07 11:49:40 -07:00
chrislu
69ef6459c3
S3 ListObjectsV2 fix pagination under bucket root
...
fix https://github.com/chrislusf/seaweedfs/issues/3166
2022-07-06 02:28:34 -07:00
Konstantin Lebedev
785223e587
rabbitpubsub enable durable
2022-07-06 10:05:29 +05:00
chrislu
57975b93ff
adjust used size reporting
2022-07-04 13:47:42 -07:00
chrislu
a79f5bd0d5
adjust used size reporting
2022-07-04 13:46:32 -07:00
chrislu
3c79c77056
3.14
2022-07-04 13:15:49 -07:00
guol-fnst
5bc895673e
add getting bucket quota
2022-07-04 15:36:44 +08:00
guol-fnst
dd4664e610
fix usage
2022-07-04 11:31:39 +08:00
chrislu
141f662734
edge case checking when volume server does not have capacity to balance
...
fix https://github.com/chrislusf/seaweedfs/issues/3257
2022-07-01 12:16:18 -07:00
chrislu
bee3a7c798
fix tests
2022-07-01 12:09:42 -07:00
Chris Lu
ddb0f6bc74
Merge pull request #3258 from kmlebedev/issues_3255
...
volumeServer.evacuate fix select empty node
2022-07-01 02:36:30 -07:00
Konstantin Lebedev
3d5f1cbc5e
fix pagination
2022-07-01 14:17:49 +05:00
chrislu
06a8b174b5
also remove Sync() for idx file
2022-06-30 13:50:53 -07:00
chrislu
52580743b9
stuck with file.Sync()
...
starting to get weed/storage () - (master) > go test ./...
I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648
I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000
I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959
I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000
I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory
I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming...
panic: test timed out after 10m0s
goroutine 61 [running]:
testing.(*M).startAlarm.func1()
/usr/local/go/src/testing/testing.go:2029 +0x8e
created by time.goFunc
/usr/local/go/src/time/sleep.go:176 +0x32
goroutine 1 [chan receive, 10 minutes]:
testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1487 +0x37a
testing.runTests.func1(0xc0000cd520?)
/usr/local/go/src/testing/testing.go:1839 +0x6e
testing.tRunner(0xc0000cd520, 0xc00039fcd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?})
/usr/local/go/src/testing/testing.go:1837 +0x457
testing.(*M).Run(0xc0000dadc0)
/usr/local/go/src/testing/testing.go:1719 +0x5d9
main.main()
_testmain.go:57 +0x1aa
goroutine 21 [chan receive]:
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a
created by github.com/chrislusf/seaweedfs/weed/glog.init.0
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf
goroutine 55 [syscall]:
syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?)
/usr/local/go/src/runtime/sys_darwin.go:22 +0x4e
syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?)
/usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30
internal/poll.(*FD).Fsync.func1(...)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:18
internal/poll.ignoringEINTR(...)
/usr/local/go/src/internal/poll/fd_posix.go:74
internal/poll.(*FD).Fsync(0xc00038f1e0?)
/usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc
os.(*File).Sync(0xc00012a030)
/usr/local/go/src/os/file_posix.go:168 +0x4e
github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52})
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8
github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000)
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305
testing.tRunner(0xc00025a000, 0x1c67bd8)
/usr/local/go/src/testing/testing.go:1439 +0x102
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1486 +0x35f
goroutine 56 [chan receive, 10 minutes]:
github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1()
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a
created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker
/Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56
FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s
? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached)
? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached)
? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files]
ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached)
ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached)
? github.com/chrislusf/seaweedfs/weed/storage/types [no test files]
? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files]
FAIL
2022-06-30 12:32:55 -07:00
Konstantin Lebedev
5ed8165161
fix logic
...
add option targetServer
https://github.com/chrislusf/seaweedfs/issues/3255
2022-06-30 17:58:24 +05:00
石昌林
200a862701
fixed volume xx not found caused by missing VolumeLocation events
...
When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost
2022-06-30 13:41:56 +08:00
Chris Lu
4ae9884a6e
Merge pull request #3254 from shichanglin5/fix_subscribe_meta
...
[Critical] Avoid synchronizing from scratch due to err
2022-06-29 12:36:28 -07:00
Chris Lu
b98556cff5
Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder
...
use const multipart uploads folder
2022-06-29 12:09:03 -07:00
石昌林
2b5b37231b
Make sure that onChangeEvent is called successfully before updating lastTsNs
2022-06-29 23:57:07 +08:00
石昌林
e17e6ed542
Avoid synchronizing from scratch due to err
2022-06-29 23:31:04 +08:00
Konstantin Lebedev
bcbdc4cb37
use const multipart uploads folder
...
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu
a8abab2412
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-29 02:44:14 -07:00
chrislu
449ecc8890
remove max connection age
...
related to https://github.com/chrislusf/seaweedfs/pull/3226
2022-06-29 02:44:12 -07:00
Chris Lu
c6e6e303db
Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy
...
validate tags on copy object and add regex for validating tags
2022-06-29 01:37:14 -07:00
Lapshin Vitaliy
7c3d9d0535
remove println
2022-06-29 10:43:02 +03:00
Lapshin Vitaliy
61b2e3f9a2
fix return
2022-06-28 18:31:46 +03:00
Lapshin Vitaliy
606252472c
fix return
2022-06-28 18:24:03 +03:00
Lapshin Vitaliy
e969370913
fix parseTagging function and add test for validate tags
2022-06-28 14:48:55 +03:00
Chris Lu
a79bf9d92a
Merge pull request #3252 from guol-fnst/update_tbk
...
remove hdfs, add tikv for full version update
2022-06-27 22:37:41 -07:00
guol-fnst
74c9f39d48
remove hdfs, add tikv for full version update
2022-06-28 12:35:07 +08:00
石昌林
e1b94eb6b9
fix error: concurrent map writes when add or remove cluster node
2022-06-28 10:34:59 +08:00
chrislu
4b1f48a399
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-27 12:40:50 -07:00
chrislu
4a65159250
fix reading time
2022-06-27 12:40:47 -07:00
xdadrm
3a42269da8
Fixes WebDAV 0-bytes files
...
Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.
2022-06-27 20:19:21 +02:00
Chris Lu
4be406a095
Merge pull request #3249 from kmlebedev/issues_2589
...
avoid set currentMaster svc.local k8s discovery service domains
2022-06-27 11:05:07 -07:00
Konstantin Lebedev
9bff097b01
currentMaster redirected
2022-06-27 21:55:16 +05:00
Konstantin Lebedev
6c20a3b622
avoid set currentMaster k8s svc.local discoveruy service domains
...
https://github.com/chrislusf/seaweedfs/issues/2589
2022-06-27 21:47:05 +05:00
Lapshin Vitaliy
1a5981d583
fix test
2022-06-27 18:46:21 +03:00
garenchan
8aa19577f4
fix 3238: handle errors for GenerateDirUuid method
2022-06-27 22:04:50 +08:00
Lapshin Vitaliy
d7c3493d15
Merge branch 'master' into bufix/validate-tags-on-copy
2022-06-27 13:53:57 +03:00
chrislu
b9f7b6fb9a
adjust log message
2022-06-26 23:12:16 -07:00
Chris Lu
fdd880b111
Merge pull request #3237 from shichanglin5/remove_query_collection
...
Remove the collection query param of s3api and let the collection be …
2022-06-26 23:02:11 -07:00
石昌林
15bfc60a88
Remove the collection query param of s3api and let the collection be determined by the filer
2022-06-27 13:47:53 +08:00
yulai.li
af23e63e3f
Improve filer command help, add supported filer store list
2022-06-27 12:09:16 +08:00
chrislu
40a9634c3d
3.13
2022-06-26 20:13:52 -07:00
Chris Lu
cc01f71752
Merge pull request #3233 from blacktear23/filerstore-tikv
...
Add TiKV Filerstore back
2022-06-26 20:12:18 -07:00
yulai.li
824cbe32be
Make tikv filer enable/disable by build tags
2022-06-27 10:57:24 +08:00
chrislu
9dfea44aeb
fix unit
2022-06-26 12:21:38 -07:00
chrislu
3142e77eca
mount: stats report physical size instead of logical size
2022-06-26 12:14:34 -07:00
yulai.li
13159ad0d8
Merge branch 'master' into filerstore-tikv
2022-06-26 22:44:01 +08:00
yulai.li
46e0b629e5
Update tikv client version and add one PC support
2022-06-26 22:43:37 +08:00
chrislu
f77eda4b4f
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-25 09:22:52 -07:00
chrislu
3ce46c3566
fix entry full url lookup
2022-06-25 09:22:49 -07:00
Chris Lu
dc59ccd110
Merge pull request #3228 from shichanglin5/fix_volumeNotFound
...
When the connection with the leader is disconnected, the vidMap shoul…
2022-06-24 23:06:13 -07:00
石昌林
f1ea906c13
Code comment optimization
2022-06-25 11:56:09 +08:00
石昌林
0c8e033e68
When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader
2022-06-24 17:41:46 +08:00
chrislu
9c517d2b35
masterclient: fallback to directly querying master in case of missing volume id location
2022-06-24 02:08:57 -07:00
Chris Lu
280e33092c
Merge pull request #3226 from kmlebedev/advancedtls
...
Advancedtls
2022-06-24 01:34:58 -07:00
chrislu
509a9047db
test compact map with snowflake sequencer
2022-06-23 21:45:51 -07:00
chrislu
217d7dc377
verify seq is always unique
2022-06-23 21:22:05 -07:00
Konstantin Lebedev
2e782dfdcb
rm defer comments
2022-06-24 01:50:13 +05:00
Konstantin Lebedev
7efaafb0ee
fix Authenticate
2022-06-24 01:42:04 +05:00
Konstantin Lebedev
b0aa51d7ef
enable require client cert
2022-06-24 00:29:23 +05:00
Konstantin Lebedev
ea7cdb8b0e
seperate option
2022-06-24 00:26:56 +05:00
Konstantin Lebedev
5b388ed6c1
initial advancedtls
2022-06-23 23:32:15 +05:00
chrislu
0c13fc873c
fix compilation
2022-06-23 00:45:11 -07:00
chrislu
96496d5286
master: broadcast new volume locations to clients to avoid possible racing condition
...
fix https://github.com/chrislusf/seaweedfs/issues/3220
2022-06-23 00:41:33 -07:00
chrislu
52c44d646e
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-22 00:34:27 -07:00
chrislu
530da0cc89
remove unused code
2022-06-22 00:34:25 -07:00
famosss
8a6e7a7df8
Merge branch 'chrislusf:master' into master
2022-06-22 09:58:56 +08:00
zzq09494
9df5ad5309
fix: vacuum create a lot of connections quickly
2022-06-22 09:57:22 +08:00
Chris Lu
d652ce1a36
Merge pull request #3216 from kmlebedev/mk_raft_basedir
...
avoid no such raft date directory
2022-06-21 12:21:30 -07:00
chrislu
6b3beb709d
fix format
2022-06-21 12:21:02 -07:00
Konstantin Lebedev
c1c8dad677
avoid no such raft date directory
...
https://github.com/chrislusf/seaweedfs/issues/3214
2022-06-21 13:47:51 +05:00
chrislu
4dc27e1ed5
3.12
2022-06-20 19:07:00 -07:00
chrislu
492da3dbce
master: put metadata under instance specific folder
2022-06-20 19:04:49 -07:00
chrislu
41dfe27102
Revert "remove max connection age"
...
This reverts commit b9b684194f
2022-06-20 14:49:39 -07:00
chrislu
2295766e2c
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56
remote storage: stop supporting hdfs as a remote storage
2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page
2022-06-20 22:36:07 +02:00
chrislu
bf5a5abfb1
Merge branch 'master' of https://github.com/chrislusf/seaweedfs
2022-06-20 13:18:11 -07:00
chrislu
69a2e503ab
non-recursive directory deletion
...
fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.
2022-06-20 13:18:08 -07:00
chrislu
76f26c4b99
mount: delete target entry first during move
2022-06-20 13:15:02 -07:00
Chris Lu
460c7cf68e
Merge pull request #3203 from shichanglin5/volume_copy_preallocate
2022-06-20 07:41:45 -07:00
石昌林
81f7f08708
Determine whether to preallocate according to the master configuration before executing copy volume
2022-06-20 21:12:44 +08:00
石昌林
be5c901bea
Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems
2022-06-20 19:16:53 +08:00
Lapshin Vitaliy
5f5fd0bc48
validate tags on copy object and add regex for validating tags
2022-06-20 11:32:58 +03:00
Chris Lu
9e2d6e897e
Merge pull request #3198 from guol-fnst/fix_statuscode
...
AbortMultipartUploadHandler should return 204 instead of 200
2022-06-19 22:57:13 -07:00
LHHDZ
8db9f13bc6
Merge branch 'master' into circuit_breaker
2022-06-20 13:42:18 +08:00
guol-fnst
1d77deccd0
AbortMultipartUploadHandler should return 204 instead of 200
...
https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html
2022-06-20 13:26:22 +08:00
石昌林
9e036df356
remove go.uber.org/atomic
2022-06-20 12:35:29 +08:00
geekboood
fdacd94af5
fix: invalid chunk data when failed to read manifests
2022-06-19 16:54:04 +08:00
石昌林
3dd60529c5
some code optimizations
2022-06-17 19:07:39 +08:00
石昌林
37df209195
add some unit tests and some code optimizes
2022-06-17 17:11:18 +08:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
...
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-17 12:47:04 +08:00
chrislu
625fd16a2e
reduce upfront memory usage for low density volume
2022-06-16 15:39:29 -07:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
...
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4
fix path bug on windows
...
fix usage
2022-06-16 17:24:56 +08:00
chrislu
aaa9938353
Revert "adjust conditions"
...
This reverts commit e024586ff1
2022-06-16 01:26:36 -07:00
guol-fnst
995e221e97
remove -output, add -dir,-name for more flexibility
2022-06-16 16:07:32 +08:00
chrislu
e024586ff1
adjust conditions
2022-06-15 23:19:39 -07:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
...
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
liubaojiang
df0ce31a2e
add condition when inFlightUploadDataLimitCond signal
2022-06-16 14:07:11 +08:00
