chrislu
6ebe26a765
Revert "Revert "Revert "Add disk type to prometheus metrics" ( #4777 )""
...
This reverts commit 567d788928
2023-10-03 08:28:52 -07:00
chrislu
7540d43ee9
Revert "Revert "fix compilation""
...
This reverts commit f9abfd0b03
2023-10-03 08:28:02 -07:00
chrislu
249c0e06ef
Revert "fix compilation"
...
This reverts commit 451ec6504d
2023-10-03 08:27:50 -07:00
chrislu
451ec6504d
fix compilation
2023-10-03 08:15:18 -07:00
chrislu
f9abfd0b03
Revert "fix compilation"
...
This reverts commit 0483ba3889
2023-10-03 07:58:11 -07:00
chrislu
0483ba3889
fix compilation
2023-10-02 11:55:50 -07:00
chrislu
567d788928
Revert "Revert "Add disk type to prometheus metrics" ( #4777 )"
...
This reverts commit 9215ba24be
2023-10-02 11:49:54 -07:00
chrislu
b5070d6280
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-10-02 10:52:40 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg ( #4880 )
2023-10-02 08:29:09 -07:00
chrislu
bf13f3ced7
add subscriber coordinator
2023-10-02 01:02:27 -07:00
chrislu
734178093e
refactor TopicPartition struct
2023-10-02 01:01:45 -07:00
chrislu
2a578b9033
refactor
2023-10-01 11:59:19 -07:00
chrislu
321fda552c
comment
2023-10-01 11:36:27 -07:00
chrislu
b02fdeabff
RetryForever => RetryUntil
2023-10-01 11:33:56 -07:00
chrislu
ebd6f96d35
adjust mq.proto
2023-09-30 21:47:57 -07:00
chrislu
02cb799481
add a test framework for grpc
2023-09-30 21:36:58 -07:00
chrislu
c4e6545197
fix compilation
2023-09-30 21:26:09 -07:00
chrislu
358cba43ef
update proto generated files
2023-09-30 13:19:25 -07:00
chrislu
597ab1e4bf
add design for mq balancer
2023-09-30 13:19:05 -07:00
chrislu
d7c211fe8c
adjust mq proto
2023-09-30 13:18:49 -07:00
chrislu
40888b5612
align 64bit atomic operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/4878
2023-09-28 07:58:43 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync ( #4875 )
...
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647
* Update weed/shell/command_volume_fix_replication.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix: revert this existing do option to positive
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-27 23:12:10 -07:00
Konstantin Lebedev
44906f1f3b
fix: avoid error file name too long when writing a file ( #4876 )
2023-09-27 05:40:51 -07:00
chrislu
9d589b48e6
rename function
2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica ( #4873 )
...
https://github.com/seaweedfs/seaweedfs/issues/4647
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4
describe a topic
2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance ( #4864 )
...
* fix(ec): ignore 0 byte data files
refers to parts of #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
* fix(ec): ignore volumes not from the current collection during balance
fixes #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
---------
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b
Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs ""
...
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu
3d07895518
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 09:16:35 -07:00
chrislu
a6e35e4f68
fix compilation
2023-09-25 09:16:33 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration ( #4859 )
...
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface
https://github.com/seaweedfs/seaweedfs/issues/4519
* fix: test bucket acl default and
adjust the variable names
* fix: s3 api config test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
chrislu
c9177c92e5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 08:20:16 -07:00
chrislu
81e57bd45e
fix test
2023-09-25 08:20:03 -07:00
Lars Lehtonen
28a3a31b27
weed/replication/sub: fix dropped error ( #4865 )
2023-09-25 07:33:18 -07:00
chrislu
23f334dd13
comment
2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71
ensure latest stats are reported
2023-09-24 23:34:31 -07:00
chrislu
dff2ce5d2f
read existing broker stats
2023-09-24 23:10:33 -07:00
chrislu
1492bf7552
fix listing topics
2023-09-24 23:05:41 -07:00
chrislu
3cf9b8d621
create topic and report topic
2023-09-24 22:00:43 -07:00
chrislu
b3f94feede
do create topic
2023-09-24 22:00:43 -07:00
chrislu
d74348048a
implement create topic
2023-09-24 22:00:43 -07:00
chrislu
0361c321b4
add CreateTopic API
2023-09-24 22:00:43 -07:00
chrislu
0f8168c0c9
3.57
2023-09-24 21:29:31 -07:00
chrislu
1df4223195
minor
2023-09-24 09:54:57 -07:00
chrislu
10ac25f857
avoid deleting the entry too early
2023-09-23 12:43:00 -07:00
chrislu
c43238b30a
fix waiting condition
2023-09-23 12:28:02 -07:00
chrislu
322af1fc04
fix compilation
2023-09-23 12:13:32 -07:00
chrislu
0414b68a4f
if shared lock, still need to wait for exclusive lock to release
2023-09-23 12:08:23 -07:00
chrislu
9ba4639a27
adjust for better logs
2023-09-23 11:53:44 -07:00
chrislu
186b1f029b
read source uses shared lock
2023-09-23 10:33:44 -07:00
chrislu
2df70ce95f
adjust logs
2023-09-21 23:51:32 -07:00
chrislu
3e23421608
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-21 20:08:30 -07:00
chrislu
49a470bcdf
reduce logs
2023-09-21 20:08:20 -07:00
Konstantin Lebedev
750a0ba1b2
[iam] Replace action read/write to readAcp/writeAcp for handlers with acl ( #4858 )
...
Replace action read/write to readAcp/writeAcp for handlers with acl query
https://github.com/seaweedfs/seaweedfs/issues/4519
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-21 14:07:04 -07:00
chrislu
8b2c39f2c0
reduce binary size
2023-09-21 13:22:08 -07:00
chrislu
b388957340
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-21 11:08:30 -07:00
chrislu
31fc165715
mount switch to ordered lock requests
2023-09-21 11:08:26 -07:00
chrislu
7669852241
adjust logs
2023-09-21 11:04:12 -07:00
chrislu
e3b1bacf3f
add lock table to ensure ordered locks
2023-09-21 10:24:34 -07:00
Konstantin Lebedev
a46f873edd
[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. ( #4856 )
...
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private
https://github.com/seaweedfs/seaweedfs/issues/4519
* Update weed/s3api/s3api_bucket_handlers.go
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-21 08:20:05 -07:00
Konstantin Lebedev
d8b424d123
[s3] optimization iam lookup for reducing algorithm complexity ( #4857 )
...
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-21 08:19:11 -07:00
chrislu
411bdda08d
fix compilation
2023-09-19 16:22:41 -07:00
chrislu
d531b7cbbf
merge some previous missing changes
2023-09-19 14:10:54 -07:00
chrislu
b18112ae10
fix compilation bugs during merge
2023-09-19 14:08:17 -07:00
chrislu
6c6655de9d
wip
2023-09-19 14:03:42 -07:00
chrislu
cf28108ea4
fix grpd dial option
2023-09-19 14:02:51 -07:00
chrislu
27af11f1e8
Revert "Revert "Merge branch 'master' into sub""
...
This reverts commit 0bb97709d4
2023-09-18 18:47:34 -07:00
chrislu
0bb97709d4
Revert "Merge branch 'master' into sub"
...
This reverts commit 4d414f54a24827425146
2023-09-18 16:13:20 -07:00
chrislu
8cb42c39ad
Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs "
...
This reverts commit 2e5aa060264d414f54a2
2023-09-18 16:12:50 -07:00
chrislu
2e5aa06026
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-18 16:06:41 -07:00
chrislu
4d414f54a2
Merge branch 'master' into sub
2023-09-18 16:04:59 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 ( #4850 )
...
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone ) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases )
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md )
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0 )
---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* API changes
* go mod
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2023-09-18 14:43:05 -07:00
chrislu
b64fc44e8d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-18 09:01:17 -07:00
chrislu
340aea9b34
add broker leader
2023-09-18 09:01:14 -07:00
chrislu
0d390ec882
fix nil
2023-09-18 08:52:35 -07:00
Lars Lehtonen
d83107efe5
fix: weed/storage: dropped error
2023-09-18 08:18:57 -07:00
zemul
91cbaef02b
fix: mount proxyByFiler write error: wrong jwt
2023-09-18 07:44:50 -07:00
chrislu
4e17d399a6
refactor
2023-09-17 00:04:20 -07:00
chrislu
7a49d3541a
avoid nil
2023-09-17 00:04:13 -07:00
chrislu
a65088d739
broker report stats to balancer
2023-09-16 23:00:51 -07:00
chrislu
a1cb49d078
adjust log
2023-09-16 23:00:28 -07:00
chrislu
b590a68e75
revert to non-streaming mode to assign file id
...
fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685
2023-09-16 16:13:08 -07:00
chrislu
4650c4c65f
Revert "turn on streaming assign file id"
...
This reverts commit 733db2bc88
2023-09-16 16:09:20 -07:00
chrislu
fa62926347
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-16 15:06:22 -07:00
chrislu
89a1fd1751
Squashed commit of the following:
...
commit 48274251463b50139f687f685ce7ba436d99443bb771fefa370a851ec008908810370a851ec00b39941edc0b875f562779984b6c54cf2492a45499ba67e6ca299e4f985698cb470d44df1eb2da46d5504ae8383adbcba75271c9caf331194d6c18d86f4eb8e8624d1990456670905911853d
2023-09-16 15:06:16 -07:00
chrislu
4827425146
balancer works
2023-09-16 15:05:38 -07:00
chrislu
3b50139f68
comments
2023-09-15 22:22:32 -07:00
chrislu
7f685ce7ba
adjust APIs
2023-09-15 22:20:05 -07:00
chrislu
436d99443b
receive broker stats
2023-09-14 23:49:05 -07:00
wang wusong
9e5b1eef9b
avoid passe lock by value
2023-09-14 06:51:12 -07:00
chrislu
b771fefa37
Merge branch 'master' into sub
2023-09-13 00:03:47 -07:00
annluda
8908810376
Fix s3 api object list with params max-keys and prefix
2023-09-11 21:35:15 -07:00
chrislu
5407b86766
3.56
2023-09-10 22:05:41 -07:00
chrislu
07c2d75ac7
ensure single threaded access
2023-09-10 22:03:44 -07:00
chrislu
0a851ec00b
Create balancer.go
2023-09-10 22:01:25 -07:00
chrislu
39941edc0b
add publisher shutdown
2023-09-07 23:55:19 -07:00
chrislu
875f562779
server side send response at least once per second
2023-09-06 23:16:41 -07:00
chrislu
984b6c54cf
ack interval 128
2023-09-06 23:15:29 -07:00
chrislu
2492a45499
ack interval
2023-09-06 22:39:46 -07:00
Lars Lehtonen
f24c7e803f
weed/storage/erasure_coding: Close() after error handling
2023-09-06 07:35:54 -07:00
Lars Lehtonen
d762b12e11
weed/storage/erasure_coding: fix dropped test error
2023-09-06 07:35:54 -07:00
Lars Lehtonen
a879e1bbb5
weed/storage/erasure_coding: remove unused err from encodeDatFile() signature
2023-09-06 07:35:54 -07:00
Patrick Schmidt
98dcec0ee2
Clean up old signature hash pools
2023-09-05 10:33:27 -07:00
Patrick Schmidt
cdd817edf9
Improve S3 request signing performance
...
This change is caching HMAC hashers for repeated use in subsequent
requests and chunks, so they don't have to be initialized from
scratch every time.
On my local computer this gives me ~5-6 times faster signature
calculation and ~5-6.5% more throughput in S3 requests. The smaller
the payload the better the throughput gets.
2023-09-05 10:33:27 -07:00
chrislu
ba67e6ca29
api for sub
2023-09-04 21:43:50 -07:00
chrislu
9e4f985698
publish, benchmark
2023-09-04 21:43:30 -07:00
Lars Lehtonen
e6385e5e8b
weed/util: fix dropped errors ( #4803 )
2023-09-01 13:10:45 -07:00
chrislu
cb470d44df
can pub and sub
2023-09-01 00:36:51 -07:00
chrislu
1eb2da46d5
connect and publish
2023-08-28 09:02:12 -07:00
chrislu
504ae8383a
protoc version
2023-08-28 09:01:25 -07:00
chrislu
dbcba75271
rename to lookup
2023-08-27 18:59:04 -07:00
chrislu
c9caf33119
move functions
2023-08-27 18:33:46 -07:00
chrislu
4d6c18d86f
pub sub initial tests
2023-08-27 17:50:59 -07:00
chrislu
4eb8e8624d
rename
2023-08-27 13:14:39 -07:00
chrislu
1990456670
sub
2023-08-27 13:13:14 -07:00
chrislu
905911853d
adjust proto
2023-08-26 13:39:21 -07:00
Nico D'Cotta
796b7508f3
Implement SRV lookups for filer ( #4767 )
2023-08-24 07:08:56 -07:00
chrislu
733db2bc88
turn on streaming assign file id
2023-08-23 00:34:21 -07:00
chrislu
99f037b958
streaming assign file ids
2023-08-23 00:31:33 -07:00
chrislu
94b7e2a37c
add stream assign server side implementation
2023-08-22 09:59:04 -07:00
chrislu
ccedad5196
refactor files
2023-08-22 09:54:06 -07:00
chrislu
010c5e91e3
add stream assign proto
2023-08-22 09:53:54 -07:00
chrislu
a1149f060b
upgrade jwt
2023-08-22 08:26:31 -07:00
chrislu
c49caff5a1
change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims
2023-08-21 23:42:39 -07:00
chrislu
c45665eb97
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-08-21 09:47:36 -07:00
SmsS4
f61490966f
Add time to first byte metric for s3 ( #4768 )
...
* Add time to first byte metric for s3
* Change second to millisecond
2023-08-21 00:42:39 -07:00
chrislu
01d70c21f3
Squashed commit of the following:
...
commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:52:19 2023 -0700
fix compilation
commit e77ad33b7ca0423138fbae26a4433b60923a9588
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 22:46:44 2023 -0700
pub
commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3
Author: chrislu <chris.lu@gmail.com>
Date: Sun Aug 20 13:27:39 2023 -0700
fix generic type
commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d
Merge: 30f942580 16e3f2d52
2023-08-20 22:53:05 -07:00
chrislu
16e3f2d528
fix log buffer test
2023-08-20 12:46:15 -07:00
chrislu
3e9c32a3f0
refactoring to use atomic bool
...
fix compilation
2023-08-20 12:13:06 -07:00
SmsS4
1e72579ac7
Fix s3.clean.uploads ( #4766 )
2023-08-19 06:30:18 -07:00
chrislu
6792db21dc
ErrInvalidPart when completed parts has extra parts
...
fix https://github.com/seaweedfs/seaweedfs/discussions/4746
2023-08-17 00:51:42 -07:00
chrislu
f5c4b642bb
re-ordering
2023-08-16 23:54:23 -07:00
chrislu
3852307e94
renaming
2023-08-16 23:47:43 -07:00
chrislu
6c7fa567d4
add separate cache directory for write buffers
2023-08-16 23:39:21 -07:00
SmsS4
ea1b259c12
Update weed fix description ( #4745 )
2023-08-13 03:50:53 -07:00
Nikita Mochalov
4b1ba7f5b2
Save disk space metrics immediately ( #4740 )
2023-08-10 06:52:32 -07:00
Nikita Mochalov
e6a49dc533
Fix resource leaks ( #4737 )
...
* Fix division by zero
* Fix file handle leak
* Fix file handle leak
* Fix file handle leak
* Fix goroutine leak
2023-08-09 15:30:36 -07:00
chrislu
3365468d0d
added an error message
2023-08-08 20:35:21 -07:00
wusong
3ec977cd6c
minor ( #4723 )
2023-08-01 02:34:59 -07:00
chrislu
c09100f928
fix building mongo index
2023-07-24 11:09:51 -07:00
chrislu
7eafa3420b
3.55
2023-07-23 23:11:37 -07:00
SmoothDenis
51bcc219ea
s3api should return 500 code from filer ( #4699 )
2023-07-22 07:22:38 -07:00
wusong
33b9192536
Fix large file chunks deleted by mistake ( #4678 )
2023-07-17 07:52:14 -07:00
chrislu
358b3a4894
3.54
2023-07-16 22:47:57 -07:00
chrislu
6f588b5b18
fix refactoring mistake
...
fix https://github.com/seaweedfs/seaweedfs/issues/4639
2023-07-16 11:22:48 -07:00
mervynzhang
df400e6c71
Concurrency works better ( #4663 )
...
Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>
2023-07-12 23:04:54 -07:00
wusong
0cb9ddd8ec
Fix data loss: add lock for metacache ( #4664 )
...
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2023-07-11 22:23:32 -07:00
Henco Appel
89542db2d0
Fix S3 API multipart upload on Windows ( #4666 )
...
The file paths were being having '/' changed to '\' by filepath.Dir()
resulting in a file being created with '\' separators, but when trying
to read the files, the same wasn't happening.
Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2023-07-11 09:05:14 -07:00
Henco Appel
680c2e76f1
Fix the segfault from not setting s3 https port ( #4667 )
2023-07-11 08:03:20 -07:00
Konstantin Lebedev
2c076ebd96
avoid logging nil error ( #4659 )
...
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-07-10 23:38:52 -07:00
