1
0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-25 18:27:54 +08:00
Commit Graph

6924 Commits

Author SHA1 Message Date
chrislu
48fe50df81 minor 2024-04-12 13:31:54 -07:00
chrislu
0cbf364a3d all fields are optional 2024-04-12 13:29:35 -07:00
chrislu
7a72677f98 add test 2024-04-12 01:31:23 -07:00
chrislu
d45a372615 add schema.proto 2024-04-12 01:30:29 -07:00
chrislu
d5b2cee34a log possible racing condition
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:54 -07:00
chrislu
2a4a9c6343 log possible racing condition
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:18 -07:00
chrislu
e53a972483 Merge branch 'master' into mq-subscribe 2024-04-11 19:48:00 -07:00
Konstantin Lebedev
5189a09de0
[volume] Reduce the number of buffers for uploading one chunk () 2024-04-11 04:47:21 -07:00
skycope
6dae685f9c
fix append disorder () 2024-04-10 19:21:44 -07:00
chrislu
cc1c69f312 Merge branch 'master' into mq-subscribe 2024-04-08 11:03:14 -07:00
steve.wei
67ead9b18f
fix(volume.fix.replication): adjust volume count, not free volume count () 2024-04-08 07:30:04 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server () 2024-04-08 07:27:00 -07:00
clonefetch
9e07a87fcb
chore: fix function names in comment () 2024-04-08 07:19:02 -07:00
Konstantin Lebedev
3e25ed1b11
[s3] add s3 pass test_multipart_upload_size_too_small ()
* add s3 pass test_multipart_upload_size_too_small

* refactor metric names

* return ErrNoSuchUpload if empty parts

* fix test
2024-04-07 11:52:35 -07:00
Konstantin Lebedev
35cba720a5
[s3] add s3 pass test_multipart_upload ()
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev
d42a04cceb
[s3] fix s3 test_multipart_resend_first_finishes_last ()
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466

* add error handler metrics

* refactor

* refactor multipartExt

* delete bad entry parts
2024-04-06 10:56:39 -07:00
Nikita Borzykh
7aa25c1137
fix: correctly form partNumber from new entry.Name format in "listObjectParts" () 2024-04-04 09:16:16 -07:00
skycope
b19c9847c6
fix completed multiupload lost data ()
If there are putObjectPart requests with the same uploadId during
completeMultiPart, it can result in data loss. putObjectPart requests
might be due to timeout retries.

Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-04-02 20:18:42 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
chrislu
f37c0d0d7a comment out println 2024-04-02 15:31:31 -07:00
Konstantin Lebedev
d5d8b8e2ae
fix panic at isAllWritable ()
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
2024-04-02 09:06:19 -07:00
Henco Appel
2a88da4de7
fix: always close volume file () ()
If sync fails then close is never called. We should always be calling
close on the file.

Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2024-04-02 08:31:40 -07:00
chrislu
d935f70e3c shutdown follower when leader stops 2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1 send flush message to follower before shutting down logBuffer 2024-04-01 16:01:26 -07:00
Nikita Borzykh
df9031e19a
fix: ensure complete gRPC stream consumption in doListFilerEntries to prevent context canceled errors. () 2024-04-01 15:55:16 -07:00
chrislu
e568e742c9 renaming 2024-04-01 15:02:52 -07:00
chrislu
47975e75bf purge old code 2024-03-31 17:49:34 -07:00
chrislu
237e68059d shutdown follower if no clients, not only publishers
to avoid overhead when a publisher repeatedly send one message and close
2024-03-31 17:45:26 -07:00
chrislu
66a878af39 add control message to data message 2024-03-31 16:35:46 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6 simplify 2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75 logs 2024-03-30 20:37:46 -07:00
Nikita Borzykh
829c0da381
fix: adjust condition in prefixFilterEntries prevent infinite loop () 2024-03-30 14:47:00 -07:00
Konstantin Lebedev
9c1e0f5811
[master] grow volumes if no writable volumes in current dataCenter ()
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
2024-03-29 00:38:27 -07:00
Konstantin Lebedev
df40908e57
fix panic 5435 () 2024-03-28 16:17:59 -07:00
chrislu
13b7c7b366 Merge branch 'master' into mq-subscribe 2024-03-27 23:57:00 -07:00
chrislu
e3c8be7f89 LocalPartition shutdown 2024-03-27 23:55:06 -07:00
chrislu
ab9fc0ad70 simplify 2024-03-27 23:42:19 -07:00
chrislu
7463058299 change visibility 2024-03-27 23:11:35 -07:00
chrislu
dab5454332 publish and send to follower 2024-03-27 23:10:24 -07:00
chrislu
f249f831b1 MaybeShutdownLocalPartition refactor 2024-03-27 23:02:51 -07:00
chrislu
2a0ab7e90d fix 2024-03-27 22:59:50 -07:00
chrislu
059a120708 refactor 2024-03-27 22:58:20 -07:00
chrislu
7d87c1d2bb refactor 2024-03-27 22:48:37 -07:00
chrislu
6aa804b368 lock instead of RLock, to prevent racing condition
https://github.com/seaweedfs/seaweedfs/discussions/5432#discussioncomment-8933608
2024-03-27 16:05:11 -07:00
chrislu
5cc94a05b9 separate goroutine to send ack to publisher 2024-03-27 10:27:08 -07:00
chrislu
17806cde2a separate context
If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.
2024-03-27 10:24:57 -07:00
chrislu
f67c8309d8 rename 2024-03-26 21:52:12 -07:00
chrislu
7d69552974 log server list 2024-03-26 21:51:28 -07:00
chrislu
36fec34c47 print only adapted url
fix https://github.com/seaweedfs/seaweedfs/issues/5424
2024-03-25 12:50:43 -07:00
chrislu
707ff536a3 Merge branch 'master' into mq-subscribe 2024-03-25 11:21:19 -07:00
chrislu
aed6fdffd2 fix nil volume data backend
fix https://github.com/seaweedfs/seaweedfs/issues/5417
2024-03-25 08:19:31 -07:00
chrislu
d6a066495b adjust logs 2024-03-25 08:15:47 -07:00
chrislu
1b1b96ea1e close local partition if no more publishers and subscribers 2024-03-24 21:10:02 -07:00
Nikita Borzykh
9ca871af53
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries in etcd meta storage backend ()
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries
2024-03-24 14:08:54 -07:00
chrislu
4f5c4c3388 refactor 2024-03-24 13:07:51 -07:00
chrislu
d558a58946 refactor 2024-03-24 13:04:59 -07:00
chrislu
205d2285f3 setup follower by publisher
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
2024-03-24 12:57:09 -07:00
Nikita Korolev
2dd2bb3e16
fix filer address parsing () 2024-03-24 11:20:33 -07:00
Nikita Borzykh
1f3742850d
Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage () 2024-03-24 08:36:21 -07:00
Konstantin Lebedev
dc9568fc0d
[master] add test for PickForWrite add metrics for volume layout () 2024-03-22 07:39:11 -07:00
Nikita Korolev
953f571349
add basic support for sentry error reporting () 2024-03-22 07:10:57 -07:00
adasauce
61f4e40ad9
fix s3api: delimeter properly takes prefixes into account () 2024-03-22 07:03:34 -07:00
wusong
a5582cc921
[mount] fix GetAttr blocks count () 2024-03-22 06:11:16 -07:00
chrislu
ac13644fc9 Merge branch 'master' into mq-subscribe 2024-03-21 23:31:49 -07:00
chrislu
03f18e43df clean up logs 2024-03-21 23:30:49 -07:00
chrislu
b5a6f10612 Merge branch 'master' into mq-subscribe 2024-03-21 23:27:41 -07:00
chrislu
26e24db721 adjust log level 2024-03-21 23:27:19 -07:00
chrislu
36a3301e0c Merge branch 'master' into mq-subscribe 2024-03-21 23:24:28 -07:00
chrislu
5c0fcae039 clean up logs 2024-03-21 23:24:18 -07:00
chrislu
407a52fc93 adjust log level 2024-03-21 23:23:26 -07:00
chrislu
da31e9b939 correctly wait for the publisher to finish 2024-03-21 09:08:53 -07:00
zuzuviewer
211171e669
Fix iam service response 500 () 2024-03-21 08:43:54 -07:00
chrislu
e641d49f9f Merge branch 'master' into mq-subscribe 2024-03-21 08:22:14 -07:00
Konstantin Lebedev
25643cfbc6
fix: panic: assignment to entry in nil map on S3Sink.CreateEntry () 2024-03-21 08:12:31 -07:00
chrislu
8b18dd940f Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-21 08:07:58 -07:00
chrislu
50023fe1cf adjust log level 2024-03-21 08:07:56 -07:00
Konstantin Lebedev
d821cb3b18
fix: sync without dir /buckets/some/.uploads/hash_hash () 2024-03-20 12:54:29 -07:00
chrislu
7003ce7425 publisher is able to fully send data and receive acks
still needs to close the pipes cleanly
2024-03-20 12:25:40 -07:00
chrislu
96149cbf7f Merge branch 'master' into mq-subscribe 2024-03-19 09:14:27 -07:00
chrislu
a1816d21e9 fix compilation 2024-03-19 09:09:14 -07:00
chrislu
ad408a7d38 Merge branch 'master' into mq-subscribe 2024-03-19 08:58:09 -07:00
dependabot[bot]
359b00596f
chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 ()
* chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.65.2 to 1.66.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.65.2...v1.66.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix compilation error with rclone update ()

Fix compilation error

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Damiano Albani <damiano.albani@gmail.com>
2024-03-18 17:09:59 -07:00
jerebear12
7091269e65
fix get file metadata bug ()
* Removed problematic if statement

This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.

* Update filer_server_handers_read.go

* Updated filer responses to invalid requests
2024-03-18 11:08:16 -07:00
chrislu
516cba7053 remove FollowInMemoryMessages 2024-03-17 22:27:13 -07:00
chrislu
6ef3e6bcad logging 2024-03-17 22:24:20 -07:00
chrislu
a375b2815e Merge branch 'master' into mq-subscribe 2024-03-17 11:46:55 -07:00
chrislu
ca042bd067 simplify 2024-03-17 01:01:45 -07:00
chrislu
4dce1e9d29 Revert "remove github.com/golang/protobuf/proto"
This reverts commit 1516e0249f.
2024-03-17 01:00:43 -07:00
chrislu
a8b73296d7 fix test3 2024-03-16 23:53:46 -07:00
chrislu
3c559dc294 fix test2 2024-03-16 23:48:31 -07:00
chrislu
ea2601799c fix test 2024-03-16 23:22:17 -07:00
chrislu
1516e0249f remove github.com/golang/protobuf/proto 2024-03-16 23:18:11 -07:00
chrislu
8e5068fd2f notify 2024-03-16 23:16:33 -07:00
chrislu
aba934f0b5 adjust log level 2024-03-16 23:15:06 -07:00
chrislu
44ab42d03c unnecessary 2024-03-16 17:11:42 -07:00
chrislu
3f2dd8cc3e add follower id info 2024-03-16 17:11:18 -07:00
chrislu
b74e8082ba 3.64 2024-03-16 11:46:40 -07:00
chrislu
6a61b54f29 Merge branch 'mq-subscribe' 2024-03-16 11:42:23 -07:00
chrislu
27bb38228b only broad cast when there are waiting threads 2024-03-16 11:38:27 -07:00
chrislu
054d0b3487 wait for data 2024-03-16 11:14:14 -07:00
chrislu
f8957b84c1 rename to LastTsNs 2024-03-16 10:52:47 -07:00
chrislu
529fa8c4c8 go fmt 2024-03-16 10:51:47 -07:00
chrislu
059df3957d rename to lastFlushDataTime 2024-03-16 10:51:16 -07:00
chrislu
1b5c547cfe logs 2024-03-15 21:08:57 -07:00
chrislu
55714f54ab local partition is generated or not 2024-03-15 21:08:42 -07:00
chrislu
56ce040ab6 log errors 2024-03-15 21:05:40 -07:00
chrislu
549455c30d streaming mode 2024-03-15 21:05:24 -07:00
chrislu
f653838f34 follower receive a hello message before actual data
the leader can ensure the follower has connected
2024-03-15 20:36:01 -07:00
chrislu
64a32730a2 add size() 2024-03-15 20:34:39 -07:00
Michał Frąckiewicz
205829fa22
Document default method used by compact () 2024-03-15 12:16:44 -07:00
Konstantin Lebedev
54ee73287d
[filer] replace an invalid methods in a metric with an invalid one ()
fix: replace an invalid methods in a metric with an invalid one for filer
2024-03-14 08:19:09 -07:00
chrislu
6f615c62ae Merge branch 'master' into mq-subscribe 2024-03-12 09:19:06 -07:00
chrislu
604091a480 use stopChan to close previous filer peer meta subscription instances 2024-03-12 09:18:54 -07:00
chrislu
1efa502dbb go fmt 2024-03-12 09:17:20 -07:00
chrislu
6f75a0af55 ask follower to follow 2024-03-10 14:34:28 -07:00
chrislu
c1a5952009 rename 2024-03-09 12:57:01 -08:00
chrislu
e7823ee967 retry connecting to broker leader 2024-03-09 12:56:49 -08:00
rustrover
ab70aa92da
remove repetitive words () 2024-03-09 07:35:47 -08:00
chrislu
3b28433cb1 change LogBuffer visibility in LocalPartition 2024-03-07 10:57:04 -08:00
chrislu
28f8f33d6a include key in LogEntry 2024-03-07 10:53:30 -08:00
chrislu
34f2b600ac each log function adds a "done" return parameter 2024-03-07 10:50:09 -08:00
chrislu
62397f2371 adjust error 2024-03-07 10:47:50 -08:00
chrislu
832b60c13f pass errors 2024-03-07 10:47:06 -08:00
chrislu
49869eec83 edge cases for broker assignment 2024-03-07 10:45:38 -08:00
chrislu
33ab6cfcf7 adjust error 2024-03-07 10:44:09 -08:00
chrislu
8f79bb3987 Merge branch 'master' into mq-subscribe 2024-03-07 10:42:29 -08:00
chrislu
b544a69550 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-07 10:42:15 -08:00
chrislu
5c0f01c0eb purge cluster id 2024-03-07 10:42:12 -08:00
Konstantin Lebedev
170b63d6f8
[filer.backup] add param uploader_part_size for S3sink ()
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri
ecc154fa9e
Add deleted bytes to total_disk_size () 2024-03-06 07:28:46 -08:00
adasauce
3e3e0fa240
feat: add support for S3 ListObjects and ListObjectsV2 () 2024-03-06 07:24:44 -08:00
Konstantin Lebedev
872b5161e6
[shell] fs log purge ()
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
2024-03-05 09:09:08 -08:00
chrislu
fe03b1b522 handle single broker case 2024-03-01 05:59:40 -08:00
chrislu
50c5dd7313 test low active brokers 2024-02-29 20:50:57 -08:00
chrislu
7598922b41 assign followers 2024-02-29 14:51:06 -08:00
chrislu
7e443ef0a1 latest protoc-gen-go 2024-02-29 10:06:23 -08:00
chrislu
ead27a4f59 publish init add followers 2024-02-29 09:39:12 -08:00
chrislu
1b4484bf0a go fmt 2024-02-29 09:38:52 -08:00
chrislu
2a7028373d Merge branch 'master' into mq-subscribe 2024-02-29 06:22:39 -08:00
Sean Ross
0833057503
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… ()
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.
2024-02-29 02:16:28 -08:00
jerebear12
85d6d5371b
Disable filer UI in configuration ()
* Add filer.ui.enabled configuration property

* Add filer.expose_directory_metadata to config

* Ammend commit

* Remove ShowUI reference

* Update all routes that allow directory metadata

* Add cmd flag to server.go
2024-02-27 08:38:55 -08:00
chrislu
76ae39a363 Merge branch 'master' into mq-subscribe 2024-02-26 07:03:09 -08:00
chrislu
54d7748a4a 3.63 2024-02-25 22:17:28 -08:00
chrislu
a52ce076fc Merge branch 'master' into mq-subscribe 2024-02-25 08:00:37 -08:00
Konstantin Lebedev
7187346cc1
avoid unexpected compact size ()
https://github.com/seaweedfs/seaweedfs/issues/5215
2024-02-24 05:27:35 -08:00
chrislu
7c45992c79 when verifyNeedleIntegrity, do not truncate files
related to https://github.com/seaweedfs/seaweedfs/pull/5259
2024-02-24 02:37:03 -08:00
Chris Lu
32cb9b3276
fix unaligned 64-bit atomic operation in weed mount on arm when readi… ()
fix unaligned 64-bit atomic operation in weed mount on arm when reading a file

fix 
2024-02-19 18:51:45 -08:00
7y-9
41ec9d7759
refactor: put the auth outside () 2024-02-19 03:07:48 -08:00
7y-9
f037c09c11
fix: only admin auth can delete S3 bucket () 2024-02-19 00:38:05 -08:00
Konstantin Lebedev
6181aa7594
fix: fs verify error counter () 2024-02-15 01:31:51 -08:00
Sébastien
c694cb4e7d
filer: reduce delete entry ErrNotFound logs () 2024-02-14 14:22:08 -08:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) () 2024-02-14 14:18:56 -08:00
Justin Dhillon
a48e2ec45b
Fix Broken Links ()
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize

* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize

* remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css

* https://github.com/AShiou/hof
2024-02-14 08:26:38 -08:00
Chris Lu
16b4f9d9ab subscriber find broker leader first 2024-02-05 23:14:25 -08:00
Chris Lu
e27deed4bc upgrade protoc 2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af proxy to broker leader 2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26 report EOF error 2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864 minor 2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8 fix on finding lock owner 2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header ()
* Add s3 postpolicy support header

* optimized code
2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224 fix divide by zero 2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f fix lock TTL 2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99 less verbose 2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c Merge branch 'master' into mq-subscribe 2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f fix when two consecutive chunks with the same modified_ts_ns
fix https://github.com/seaweedfs/seaweedfs/issues/5276
2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d avoid too large expiration time 2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType () 2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c avoid timestamp overflow 2024-02-03 13:45:59 -08:00
chrislu
d41792461c lock returns host and owner 2024-02-02 15:54:57 -08:00
chrislu
d30150dde1 use a short-lived lock 2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709 fix nil error 2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822 re-lock if the lock owner is not found 2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types ()
* week/shell: Cluster check other disk types

The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.

This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Replace loop that copies slice

Use the following construct instead of a `for` loop:

```golang
x = append(x, y...)
```

See https://staticcheck.dev/docs/checks#S1011.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Check disk types when filer is in use

Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

---------

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2024-01-29 10:36:37 -08:00
Sébastien
0775d05a23
fix: http range request return status 500 ()
When volume server unavailable for at least one chunk; was returning status 206.

Split `StreamContent` in two parts,
- first prepare, to get chunk info and return stream function
- then write chunk, with that stream function

That allow to catch error in first step before setting response status code in `processRangeRequest`
2024-01-29 10:35:52 -08:00
chrislu
0b2e5ddc7c wait 3 seconds before shutting down publish client, to wait for all messages to be received 2024-01-28 22:04:42 -08:00
chrislu
545d5fbdf6 unused code 2024-01-28 22:03:13 -08:00
chrislu
a507069d19 lock for creating local partition 2024-01-28 15:55:26 -08:00
chrislu
9e6ea80dff read existing topic conf 2024-01-28 14:27:52 -08:00
chrislu
a297247137 wait for each publish clients 2024-01-28 14:27:32 -08:00
chrislu
b6c5e57c30 read will block if no items 2024-01-28 13:10:34 -08:00
chrislu
0bf5424a2e adjust visibility 2024-01-28 13:09:30 -08:00
chrislu
dedfd31dfb refactor 2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f refactor 2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff wait for publishing clients 2024-01-28 12:06:58 -08:00
chrislu
f8c55f101e clean up 2024-01-28 12:06:30 -08:00
chrislu
d40225ecd2 return the lookup results 2024-01-28 12:05:44 -08:00
chrislu
841fafd0a8 publish to input buffer
currently, the input buffer may not exist when start to publish
2024-01-27 23:43:22 -08:00
chrislu
5dc1362bdc close the input 2024-01-27 16:12:49 -08:00
chrislu
fa835c9319 Update buffered_queue_test.go 2024-01-27 14:46:12 -08:00
Chris Lu
b6de35cdb2 moved. there are some deadlock. WIP 2024-01-27 13:51:19 -08:00
chrislu
91af1f3069 schedule jobs 2024-01-26 14:09:57 -08:00
Benoît Knecht
f6e39139d8
weed/storage: Fix volume info file permissions () 2024-01-26 08:31:27 -08:00
chrislu
08c5fba825 adds a buffered queue 2024-01-25 20:22:41 -08:00
chrislu
cb50b720fd Revert "Fix weed fix"
This reverts commit 6c1ce18541.
2024-01-25 08:09:36 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic () 2024-01-25 06:19:58 -08:00
Konstantin Lebedev
0f8e76bbd6
fix: clean metric MasterReplicaPlacementMismatch for unregister volume () 2024-01-25 00:23:24 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic () 2024-01-25 00:17:32 -08:00
Konstantin Lebedev
0b71fae2e3
fix: s3tests test_get_object_ifnonematch_good () 2024-01-24 11:05:55 -08:00
chrislu
025f8eb37b return dir assign counter
fix https://github.com/seaweedfs/seaweedfs/issues/5231
2024-01-23 19:48:42 -08:00
chrislu
d4e66cc140 return dir assign counter
fix https://github.com/seaweedfs/seaweedfs/issues/5231
2024-01-23 19:47:31 -08:00
chrislu
a8ddbf9ed2 Merge branch 'master' into mq-subscribe 2024-01-23 19:41:16 -08:00
chrislu
49fcb48e04 same logic as grpc assign
fix https://github.com/seaweedfs/seaweedfs/issues/5213
2024-01-23 19:39:59 -08:00
chrislu
d268fbe18a when configure, cancel existing assignments 2024-01-22 11:57:17 -08:00
chrislu
3ff6b31d94 sub broker wait for partition 2024-01-22 11:56:55 -08:00
chrislu
2beaa2d0b3 pub/sub brokers check filer for assigned partitions 2024-01-22 11:44:21 -08:00
chrislu
c3f8530f97 refactor 2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0 no need to assign partitions explicitly 2024-01-22 11:04:47 -08:00
chrislu
428fea45f3 updated and added assignments 2024-01-22 11:04:24 -08:00
chrislu
e8b05ecc91 add/remove assigned partitions 2024-01-22 10:47:39 -08:00
chrislu
861ad732ca refactor 2024-01-22 09:00:22 -08:00
chrislu
47924afa1c refactor 2024-01-22 08:52:44 -08:00
chrislu
581cf6644c lookup reads from filer instead of local memory 2024-01-22 00:52:55 -08:00
chrislu
7121ae9617 separate out read topic conf and ensure topic active assignments 2024-01-22 00:51:31 -08:00
chrislu
c77d35313e pub/sub broker only check local assigned partitions 2024-01-22 00:49:57 -08:00
chrislu
b0a2e9aea3 fix assignments if brokers changed 2024-01-21 01:27:22 -08:00
chrislu
9ce7c482b3 refactor 2024-01-21 01:24:12 -08:00
chrislu
a5b906c0dd refactor 2024-01-21 01:23:43 -08:00
chrislu
34839237ab refactor 2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30 refactor 2024-01-20 11:41:11 -08:00
chrislu
541140f735 rename 2024-01-19 11:43:40 -08:00
chrislu
bb867694d8 skip updating multipart .upload files 2024-01-18 09:24:52 -08:00
chrislu
4ca6b1be3c skip updating the .upload multipart files 2024-01-18 09:24:52 -08:00
chrislu
d848efed1b skip updating multipart .upload files 2024-01-18 09:24:26 -08:00
chrislu
aca3c33bab skip updating the .upload multipart files 2024-01-18 09:23:24 -08:00
chrislu
314e25aad6 Merge branch 'master' into mq-subscribe 2024-01-18 09:17:16 -08:00
chrislu
5c90b27b0a go fmt 2024-01-18 09:17:08 -08:00
chrislu
263f1f3d04 Merge branch 'master' into mq-subscribe 2024-01-18 09:16:46 -08:00
chrislu
83e4b02517 fix 2024-01-18 09:16:20 -08:00
chrislu
2fa519bcb5 Merge branch 'pr/5216' 2024-01-18 09:14:01 -08:00
chrislu
15b66a6633 refactor 2024-01-18 09:13:14 -08:00
chrislu
3002087541 http assign logic should be the same as grpc assign
similar logic as bebbc9fe44
2024-01-18 08:58:53 -08:00
XIAOYQ
be166b434f fix: skip s3 .uploads 2024-01-18 22:13:46 +08:00
chrislu
c8090b1f99 send assignments to newly connected subscribers 2024-01-17 22:44:39 -08:00
chrislu
c616081ab9 refactor 2024-01-17 22:29:59 -08:00
Konstantin Lebedev
4e9ea1e628
chore: filer healthz handler check filer store () 2024-01-17 07:46:37 -08:00
Konstantin Lebedev
f9cf13fada
[filer] avoid return http ok for not allowed methods () 2024-01-17 07:17:07 -08:00
Andrew Azores
2eb82778bc
chore: fix typos in scaffold help output () 2024-01-17 07:10:17 -08:00
chrislu
bc8d2a01cc create local topic partition from config on filer 2024-01-17 00:15:52 -08:00
chrislu
fdf0ea8e11 passing partition info 2024-01-16 09:30:57 -08:00
chrislu
be0c426dc7 simplify to LookupTopicPartitions(topic) 2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5 simplify api 2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0 remove isForPublish from LookupTopicBrokers
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
chrislu
3795d8dca8 release local topic partition if no publisher and subscribers 2024-01-16 08:43:07 -08:00
chrislu
f782165638 rename 2024-01-16 08:40:45 -08:00
chrislu
dd1ec70e62 rename 2024-01-16 08:40:23 -08:00
chrislu
1877ce5126 rename 2024-01-15 21:31:21 -08:00
chrislu
ba73199174 reuse local partition 2024-01-15 21:22:41 -08:00