mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 00:27:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,002 Commits 32 Branches 296 Tags 152 MiB
3ef716409e
Commit Graph

7098 Commits

Author SHA1 Message Date
chrislu
47975e75bf purge old code 2024-03-31 17:49:34 -07:00
chrislu
237e68059d shutdown follower if no clients, not only publishers 
to avoid overhead when a publisher repeatedly send one message and close
 2024-03-31 17:45:26 -07:00
chrislu
66a878af39 add control message to data message 2024-03-31 16:35:46 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6 simplify 2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75 logs 2024-03-30 20:37:46 -07:00
Nikita Borzykh
829c0da381
fix: adjust condition in prefixFilterEntries prevent infinite loop (#5440) 2024-03-30 14:47:00 -07:00
Konstantin Lebedev
9c1e0f5811
[master] grow volumes if no writable volumes in current dataCenter (#5434) 
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
 2024-03-29 00:38:27 -07:00
Konstantin Lebedev
df40908e57
fix panic 5435 (#5436) 2024-03-28 16:17:59 -07:00
chrislu
13b7c7b366 Merge branch 'master' into mq-subscribe 2024-03-27 23:57:00 -07:00
chrislu
e3c8be7f89 LocalPartition shutdown 2024-03-27 23:55:06 -07:00
chrislu
ab9fc0ad70 simplify 2024-03-27 23:42:19 -07:00
chrislu
7463058299 change visibility 2024-03-27 23:11:35 -07:00
chrislu
dab5454332 publish and send to follower 2024-03-27 23:10:24 -07:00
chrislu
f249f831b1 MaybeShutdownLocalPartition refactor 2024-03-27 23:02:51 -07:00
chrislu
2a0ab7e90d fix 2024-03-27 22:59:50 -07:00
chrislu
059a120708 refactor 2024-03-27 22:58:20 -07:00
chrislu
7d87c1d2bb refactor 2024-03-27 22:48:37 -07:00
chrislu
6aa804b368 lock instead of RLock, to prevent racing condition 
https://github.com/seaweedfs/seaweedfs/discussions/5432#discussioncomment-8933608
 2024-03-27 16:05:11 -07:00
chrislu
5cc94a05b9 separate goroutine to send ack to publisher 2024-03-27 10:27:08 -07:00
chrislu
17806cde2a separate context 
If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.
 2024-03-27 10:24:57 -07:00
chrislu
f67c8309d8 rename 2024-03-26 21:52:12 -07:00
chrislu
7d69552974 log server list 2024-03-26 21:51:28 -07:00
chrislu
36fec34c47 print only adapted url 
fix https://github.com/seaweedfs/seaweedfs/issues/5424
 2024-03-25 12:50:43 -07:00
chrislu
707ff536a3 Merge branch 'master' into mq-subscribe 2024-03-25 11:21:19 -07:00
chrislu
aed6fdffd2 fix nil volume data backend 
fix https://github.com/seaweedfs/seaweedfs/issues/5417
 2024-03-25 08:19:31 -07:00
chrislu
d6a066495b adjust logs 2024-03-25 08:15:47 -07:00
chrislu
1b1b96ea1e close local partition if no more publishers and subscribers 2024-03-24 21:10:02 -07:00
Nikita Borzykh
9ca871af53
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries in etcd meta storage backend (#5416) 
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries
 2024-03-24 14:08:54 -07:00
chrislu
4f5c4c3388 refactor 2024-03-24 13:07:51 -07:00
chrislu
d558a58946 refactor 2024-03-24 13:04:59 -07:00
chrislu
205d2285f3 setup follower by publisher 
* the subscriber would getOrGen a local partition and wait
* the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.
 2024-03-24 12:57:09 -07:00
Nikita Korolev
2dd2bb3e16
fix filer address parsing (#5415) 2024-03-24 11:20:33 -07:00
Nikita Borzykh
1f3742850d
Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage (#5403) 2024-03-24 08:36:21 -07:00
Konstantin Lebedev
dc9568fc0d
[master] add test for PickForWrite add metrics for volume layout (#5413) 2024-03-22 07:39:11 -07:00
Nikita Korolev
953f571349
add basic support for sentry error reporting (#5410) 2024-03-22 07:10:57 -07:00
adasauce
61f4e40ad9
fix s3api: delimeter properly takes prefixes into account (#5411) 2024-03-22 07:03:34 -07:00
wusong
a5582cc921
[mount] fix GetAttr blocks count (#5409) 2024-03-22 06:11:16 -07:00
chrislu
ac13644fc9 Merge branch 'master' into mq-subscribe 2024-03-21 23:31:49 -07:00
chrislu
03f18e43df clean up logs 2024-03-21 23:30:49 -07:00
chrislu
b5a6f10612 Merge branch 'master' into mq-subscribe 2024-03-21 23:27:41 -07:00
chrislu
26e24db721 adjust log level 2024-03-21 23:27:19 -07:00
chrislu
36a3301e0c Merge branch 'master' into mq-subscribe 2024-03-21 23:24:28 -07:00
chrislu
5c0fcae039 clean up logs 2024-03-21 23:24:18 -07:00
chrislu
407a52fc93 adjust log level 2024-03-21 23:23:26 -07:00
chrislu
da31e9b939 correctly wait for the publisher to finish 2024-03-21 09:08:53 -07:00
zuzuviewer
211171e669
Fix iam service response 500 (#5405) 2024-03-21 08:43:54 -07:00
chrislu
e641d49f9f Merge branch 'master' into mq-subscribe 2024-03-21 08:22:14 -07:00
Konstantin Lebedev
25643cfbc6
fix: panic: assignment to entry in nil map on S3Sink.CreateEntry (#5406) 2024-03-21 08:12:31 -07:00
chrislu
8b18dd940f Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-21 08:07:58 -07:00
chrislu
50023fe1cf adjust log level 2024-03-21 08:07:56 -07:00
Konstantin Lebedev
d821cb3b18
fix: sync without dir /buckets/some/.uploads/hash_hash (#5402) 2024-03-20 12:54:29 -07:00
chrislu
7003ce7425 publisher is able to fully send data and receive acks 
still needs to close the pipes cleanly
 2024-03-20 12:25:40 -07:00
chrislu
96149cbf7f Merge branch 'master' into mq-subscribe 2024-03-19 09:14:27 -07:00
chrislu
a1816d21e9 fix compilation 2024-03-19 09:09:14 -07:00
chrislu
ad408a7d38 Merge branch 'master' into mq-subscribe 2024-03-19 08:58:09 -07:00
dependabot[bot]
359b00596f
chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 (#5387) 
* chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.65.2 to 1.66.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.65.2...v1.66.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix compilation error with rclone update (#5395)

Fix compilation error

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Damiano Albani <damiano.albani@gmail.com>
 2024-03-18 17:09:59 -07:00
jerebear12
7091269e65
fix get file metadata bug (#5394) 
* Removed problematic if statement

This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.

* Update filer_server_handers_read.go

* Updated filer responses to invalid requests
 2024-03-18 11:08:16 -07:00
chrislu
516cba7053 remove FollowInMemoryMessages 2024-03-17 22:27:13 -07:00
chrislu
6ef3e6bcad logging 2024-03-17 22:24:20 -07:00
chrislu
a375b2815e Merge branch 'master' into mq-subscribe 2024-03-17 11:46:55 -07:00
chrislu
ca042bd067 simplify 2024-03-17 01:01:45 -07:00
chrislu
4dce1e9d29 Revert "remove github.com/golang/protobuf/proto" 
This reverts commit 1516e0249f.
 2024-03-17 01:00:43 -07:00
chrislu
a8b73296d7 fix test3 2024-03-16 23:53:46 -07:00
chrislu
3c559dc294 fix test2 2024-03-16 23:48:31 -07:00
chrislu
ea2601799c fix test 2024-03-16 23:22:17 -07:00
chrislu
1516e0249f remove github.com/golang/protobuf/proto 2024-03-16 23:18:11 -07:00
chrislu
8e5068fd2f notify 2024-03-16 23:16:33 -07:00
chrislu
aba934f0b5 adjust log level 2024-03-16 23:15:06 -07:00
chrislu
44ab42d03c unnecessary 2024-03-16 17:11:42 -07:00
chrislu
3f2dd8cc3e add follower id info 2024-03-16 17:11:18 -07:00
chrislu
b74e8082ba 3.64 2024-03-16 11:46:40 -07:00
chrislu
6a61b54f29 Merge branch 'mq-subscribe' 2024-03-16 11:42:23 -07:00
chrislu
27bb38228b only broad cast when there are waiting threads 2024-03-16 11:38:27 -07:00
chrislu
054d0b3487 wait for data 2024-03-16 11:14:14 -07:00
chrislu
f8957b84c1 rename to LastTsNs 2024-03-16 10:52:47 -07:00
chrislu
529fa8c4c8 go fmt 2024-03-16 10:51:47 -07:00
chrislu
059df3957d rename to lastFlushDataTime 2024-03-16 10:51:16 -07:00
chrislu
1b5c547cfe logs 2024-03-15 21:08:57 -07:00
chrislu
55714f54ab local partition is generated or not 2024-03-15 21:08:42 -07:00
chrislu
56ce040ab6 log errors 2024-03-15 21:05:40 -07:00
chrislu
549455c30d streaming mode 2024-03-15 21:05:24 -07:00
chrislu
f653838f34 follower receive a hello message before actual data 
the leader can ensure the follower has connected
 2024-03-15 20:36:01 -07:00
chrislu
64a32730a2 add size() 2024-03-15 20:34:39 -07:00
Michał Frąckiewicz
205829fa22
Document default method used by compact (#5379) 2024-03-15 12:16:44 -07:00
Konstantin Lebedev
54ee73287d
[filer] replace an invalid methods in a metric with an invalid one (#5378) 
fix: replace an invalid methods in a metric with an invalid one for filer
 2024-03-14 08:19:09 -07:00
chrislu
6f615c62ae Merge branch 'master' into mq-subscribe 2024-03-12 09:19:06 -07:00
chrislu
604091a480 use stopChan to close previous filer peer meta subscription instances 2024-03-12 09:18:54 -07:00
chrislu
1efa502dbb go fmt 2024-03-12 09:17:20 -07:00
chrislu
6f75a0af55 ask follower to follow 2024-03-10 14:34:28 -07:00
chrislu
c1a5952009 rename 2024-03-09 12:57:01 -08:00
chrislu
e7823ee967 retry connecting to broker leader 2024-03-09 12:56:49 -08:00
rustrover
ab70aa92da
remove repetitive words (#5364) 2024-03-09 07:35:47 -08:00
chrislu
3b28433cb1 change LogBuffer visibility in LocalPartition 2024-03-07 10:57:04 -08:00
chrislu
28f8f33d6a include key in LogEntry 2024-03-07 10:53:30 -08:00
chrislu
34f2b600ac each log function adds a "done" return parameter 2024-03-07 10:50:09 -08:00
chrislu
62397f2371 adjust error 2024-03-07 10:47:50 -08:00
chrislu
832b60c13f pass errors 2024-03-07 10:47:06 -08:00
chrislu
49869eec83 edge cases for broker assignment 2024-03-07 10:45:38 -08:00
chrislu
33ab6cfcf7 adjust error 2024-03-07 10:44:09 -08:00
chrislu
8f79bb3987 Merge branch 'master' into mq-subscribe 2024-03-07 10:42:29 -08:00
chrislu
b544a69550 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-07 10:42:15 -08:00
chrislu
5c0f01c0eb purge cluster id 2024-03-07 10:42:12 -08:00
Konstantin Lebedev
170b63d6f8
[filer.backup] add param uploader_part_size for S3sink (#5352) 
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
 2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri
ecc154fa9e
Add deleted bytes to total_disk_size (#5161) 2024-03-06 07:28:46 -08:00
adasauce
3e3e0fa240
feat: add support for S3 ListObjects and ListObjectsV2 (#5350) 2024-03-06 07:24:44 -08:00
Konstantin Lebedev
872b5161e6
[shell] fs log purge (#5344) 
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
 2024-03-05 09:09:08 -08:00
chrislu
fe03b1b522 handle single broker case 2024-03-01 05:59:40 -08:00
chrislu
50c5dd7313 test low active brokers 2024-02-29 20:50:57 -08:00
chrislu
7598922b41 assign followers 2024-02-29 14:51:06 -08:00
chrislu
7e443ef0a1 latest protoc-gen-go 2024-02-29 10:06:23 -08:00
chrislu
ead27a4f59 publish init add followers 2024-02-29 09:39:12 -08:00
chrislu
1b4484bf0a go fmt 2024-02-29 09:38:52 -08:00
chrislu
2a7028373d Merge branch 'master' into mq-subscribe 2024-02-29 06:22:39 -08:00
Sean Ross
0833057503
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… (#5351) 
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.
 2024-02-29 02:16:28 -08:00
jerebear12
85d6d5371b
Disable filer UI in configuration (#5297) 
* Add filer.ui.enabled configuration property

* Add filer.expose_directory_metadata to config

* Ammend commit

* Remove ShowUI reference

* Update all routes that allow directory metadata

* Add cmd flag to server.go
 2024-02-27 08:38:55 -08:00
chrislu
76ae39a363 Merge branch 'master' into mq-subscribe 2024-02-26 07:03:09 -08:00
chrislu
54d7748a4a 3.63 2024-02-25 22:17:28 -08:00
chrislu
a52ce076fc Merge branch 'master' into mq-subscribe 2024-02-25 08:00:37 -08:00
Konstantin Lebedev
7187346cc1
avoid unexpected compact size (#5272) 
https://github.com/seaweedfs/seaweedfs/issues/5215
 2024-02-24 05:27:35 -08:00
chrislu
7c45992c79 when verifyNeedleIntegrity, do not truncate files 
related to https://github.com/seaweedfs/seaweedfs/pull/5259
 2024-02-24 02:37:03 -08:00
Chris Lu
32cb9b3276
fix unaligned 64-bit atomic operation in weed mount on arm when readi… (#5308) 
fix unaligned 64-bit atomic operation in weed mount on arm when reading a file

fix #5306
 2024-02-19 18:51:45 -08:00
7y-9
41ec9d7759
refactor: put the auth outside (#5313) 2024-02-19 03:07:48 -08:00
7y-9
f037c09c11
fix: only admin auth can delete S3 bucket (#5312) 2024-02-19 00:38:05 -08:00
Konstantin Lebedev
6181aa7594
fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Sébastien
c694cb4e7d
filer: reduce delete entry ErrNotFound logs (#5301) 2024-02-14 14:22:08 -08:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
Justin Dhillon
a48e2ec45b
Fix Broken Links (#5287) 
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize

* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize

* remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css

* https://github.com/AShiou/hof
 2024-02-14 08:26:38 -08:00
Chris Lu
16b4f9d9ab subscriber find broker leader first 2024-02-05 23:14:25 -08:00
Chris Lu
e27deed4bc upgrade protoc 2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af proxy to broker leader 2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26 report EOF error 2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864 minor 2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8 fix on finding lock owner 2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header (#5280) 
* Add s3 postpolicy support header

* optimized code
 2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224 fix divide by zero 2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f fix lock TTL 2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99 less verbose 2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c Merge branch 'master' into mq-subscribe 2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f fix when two consecutive chunks with the same modified_ts_ns 
fix https://github.com/seaweedfs/seaweedfs/issues/5276
 2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d avoid too large expiration time 2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType (#5275) 2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c avoid timestamp overflow 2024-02-03 13:45:59 -08:00
chrislu
d41792461c lock returns host and owner 2024-02-02 15:54:57 -08:00
chrislu
d30150dde1 use a short-lived lock 2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709 fix nil error 2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822 re-lock if the lock owner is not found 2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types (#5245) 
* week/shell: Cluster check other disk types

The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.

This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Replace loop that copies slice

Use the following construct instead of a `for` loop:

```golang
x = append(x, y...)
```

See https://staticcheck.dev/docs/checks#S1011.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Check disk types when filer is in use

Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

---------

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
 2024-01-29 10:36:37 -08:00
Sébastien
0775d05a23
fix: http range request return status 500 (#5251) 
When volume server unavailable for at least one chunk; was returning status 206.

Split `StreamContent` in two parts,
- first prepare, to get chunk info and return stream function
- then write chunk, with that stream function

That allow to catch error in first step before setting response status code in `processRangeRequest`
 2024-01-29 10:35:52 -08:00
chrislu
0b2e5ddc7c wait 3 seconds before shutting down publish client, to wait for all messages to be received 2024-01-28 22:04:42 -08:00
chrislu
545d5fbdf6 unused code 2024-01-28 22:03:13 -08:00
chrislu
a507069d19 lock for creating local partition 2024-01-28 15:55:26 -08:00
chrislu
9e6ea80dff read existing topic conf 2024-01-28 14:27:52 -08:00
chrislu
a297247137 wait for each publish clients 2024-01-28 14:27:32 -08:00
chrislu
b6c5e57c30 read will block if no items 2024-01-28 13:10:34 -08:00
chrislu
0bf5424a2e adjust visibility 2024-01-28 13:09:30 -08:00
chrislu
dedfd31dfb refactor 2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f refactor 2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff wait for publishing clients 2024-01-28 12:06:58 -08:00
chrislu
f8c55f101e clean up 2024-01-28 12:06:30 -08:00
chrislu
d40225ecd2 return the lookup results 2024-01-28 12:05:44 -08:00
chrislu
841fafd0a8 publish to input buffer 
currently, the input buffer may not exist when start to publish
 2024-01-27 23:43:22 -08:00
chrislu
5dc1362bdc close the input 2024-01-27 16:12:49 -08:00
chrislu
fa835c9319 Update buffered_queue_test.go 2024-01-27 14:46:12 -08:00
Chris Lu
b6de35cdb2 moved. there are some deadlock. WIP 2024-01-27 13:51:19 -08:00
chrislu
91af1f3069 schedule jobs 2024-01-26 14:09:57 -08:00
Benoît Knecht
f6e39139d8
weed/storage: Fix volume info file permissions (#5246) 2024-01-26 08:31:27 -08:00
chrislu
08c5fba825 adds a buffered queue 2024-01-25 20:22:41 -08:00
chrislu
cb50b720fd Revert "Fix weed fix" 
This reverts commit 6c1ce18541.
 2024-01-25 08:09:36 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic (#5241) 2024-01-25 06:19:58 -08:00
Konstantin Lebedev
0f8e76bbd6
fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic (#5238) 2024-01-25 00:17:32 -08:00
Konstantin Lebedev
0b71fae2e3
fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
chrislu
025f8eb37b return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:48:42 -08:00
chrislu
d4e66cc140 return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:47:31 -08:00
chrislu
a8ddbf9ed2 Merge branch 'master' into mq-subscribe 2024-01-23 19:41:16 -08:00
chrislu
49fcb48e04 same logic as grpc assign 
fix https://github.com/seaweedfs/seaweedfs/issues/5213
 2024-01-23 19:39:59 -08:00
chrislu
d268fbe18a when configure, cancel existing assignments 2024-01-22 11:57:17 -08:00
chrislu
3ff6b31d94 sub broker wait for partition 2024-01-22 11:56:55 -08:00
chrislu
2beaa2d0b3 pub/sub brokers check filer for assigned partitions 2024-01-22 11:44:21 -08:00
chrislu
c3f8530f97 refactor 2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0 no need to assign partitions explicitly 2024-01-22 11:04:47 -08:00
chrislu
428fea45f3 updated and added assignments 2024-01-22 11:04:24 -08:00
chrislu
e8b05ecc91 add/remove assigned partitions 2024-01-22 10:47:39 -08:00
chrislu
861ad732ca refactor 2024-01-22 09:00:22 -08:00
chrislu
47924afa1c refactor 2024-01-22 08:52:44 -08:00
chrislu
581cf6644c lookup reads from filer instead of local memory 2024-01-22 00:52:55 -08:00
chrislu
7121ae9617 separate out read topic conf and ensure topic active assignments 2024-01-22 00:51:31 -08:00
chrislu
c77d35313e pub/sub broker only check local assigned partitions 2024-01-22 00:49:57 -08:00
chrislu
b0a2e9aea3 fix assignments if brokers changed 2024-01-21 01:27:22 -08:00
chrislu
9ce7c482b3 refactor 2024-01-21 01:24:12 -08:00
chrislu
a5b906c0dd refactor 2024-01-21 01:23:43 -08:00
chrislu
34839237ab refactor 2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30 refactor 2024-01-20 11:41:11 -08:00
chrislu
541140f735 rename 2024-01-19 11:43:40 -08:00
chrislu
bb867694d8 skip updating multipart .upload files 2024-01-18 09:24:52 -08:00
chrislu
4ca6b1be3c skip updating the .upload multipart files 2024-01-18 09:24:52 -08:00
chrislu
d848efed1b skip updating multipart .upload files 2024-01-18 09:24:26 -08:00
chrislu
aca3c33bab skip updating the .upload multipart files 2024-01-18 09:23:24 -08:00
chrislu
314e25aad6 Merge branch 'master' into mq-subscribe 2024-01-18 09:17:16 -08:00
chrislu
5c90b27b0a go fmt 2024-01-18 09:17:08 -08:00
chrislu
263f1f3d04 Merge branch 'master' into mq-subscribe 2024-01-18 09:16:46 -08:00
chrislu
83e4b02517 fix 2024-01-18 09:16:20 -08:00
chrislu
2fa519bcb5 Merge branch 'pr/5216' 2024-01-18 09:14:01 -08:00
chrislu
15b66a6633 refactor 2024-01-18 09:13:14 -08:00
chrislu
3002087541 http assign logic should be the same as grpc assign 
similar logic as bebbc9fe44
 2024-01-18 08:58:53 -08:00
XIAOYQ
be166b434f fix: skip s3 .uploads 2024-01-18 22:13:46 +08:00
chrislu
c8090b1f99 send assignments to newly connected subscribers 2024-01-17 22:44:39 -08:00
chrislu
c616081ab9 refactor 2024-01-17 22:29:59 -08:00
Konstantin Lebedev
4e9ea1e628
chore: filer healthz handler check filer store (#5208) 2024-01-17 07:46:37 -08:00
Konstantin Lebedev
f9cf13fada
[filer] avoid return http ok for not allowed methods (#5209) 2024-01-17 07:17:07 -08:00
Andrew Azores
2eb82778bc
chore: fix typos in scaffold help output (#5211) 2024-01-17 07:10:17 -08:00
chrislu
bc8d2a01cc create local topic partition from config on filer 2024-01-17 00:15:52 -08:00
chrislu
fdf0ea8e11 passing partition info 2024-01-16 09:30:57 -08:00
chrislu
be0c426dc7 simplify to LookupTopicPartitions(topic) 2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5 simplify api 2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0 remove isForPublish from LookupTopicBrokers 
also adds a return parameter: whether the topic exists or not
 2024-01-16 08:52:42 -08:00
chrislu
3795d8dca8 release local topic partition if no publisher and subscribers 2024-01-16 08:43:07 -08:00
chrislu
f782165638 rename 2024-01-16 08:40:45 -08:00
chrislu
dd1ec70e62 rename 2024-01-16 08:40:23 -08:00
chrislu
1877ce5126 rename 2024-01-15 21:31:21 -08:00
chrislu
ba73199174 reuse local partition 2024-01-15 21:22:41 -08:00
chrislu
33d5343ea1 Merge branch 'master' into mq-subscribe 2024-01-15 20:56:21 -08:00
chrislu
59b8af99b0 3.62 2024-01-15 20:44:00 -08:00
chrislu
6d45bb3038 Merge branch 'master' into mq-subscribe 2024-01-15 20:43:03 -08:00
chrislu
7e6497cc1c adjust publisher subscriber 2024-01-15 20:42:46 -08:00
skycope
316ae45795
can vacuum volume when size mismatch (#5200) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-01-15 16:16:46 -08:00
chrislu
026c54a9bb fix publisher 2024-01-15 00:20:21 -08:00
chrislu
fa59a5d67e read from disk if not in memory 2024-01-15 00:20:12 -08:00
chrislu
49f7de9daa Merge branch 'master' into mq-subscribe 2024-01-13 17:52:18 -08:00
chrislu
d6ba97219b refactoring 2024-01-13 17:51:53 -08:00
Konstantin Lebedev
8d23e36c45
fix: doDeleteFiles deletes files (#5198) 2024-01-12 11:04:29 -08:00
Konstantin Lebedev
1169f94310
Fix filer sync set offset (#5197) 
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
 2024-01-12 10:57:18 -08:00
chrislu
3aa3991f0f adjust client side logs 2024-01-11 23:08:53 -08:00
chrislu
7afaad31a4 wait on local topic partition creation 2024-01-11 23:08:02 -08:00
chrislu
61dbdd0ff6 log ctrl messages 2024-01-11 23:07:32 -08:00
chrislu
f750a5e03b passing timestamp 2024-01-11 23:03:55 -08:00
chrislu
45994641e9 lint 2024-01-11 23:03:35 -08:00
chrislu
70a011a224 Merge branch 'master' into mq-subscribe 2024-01-11 20:58:57 -08:00
skycope
0e8a54f6f6
fix write volume over size MaxPossibleVolumeSize (#5190) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-01-11 20:23:46 -08:00
chrislu
050efe6c08 factor in existing ec volume count when estimating max volume count 
related to https://github.com/seaweedfs/seaweedfs/issues/5191
 2024-01-11 08:19:09 -08:00
chrislu
188f471a24 logging 2024-01-11 07:55:26 -08:00
chrislu
6a7a679137 passing partition generation timestamp 2024-01-10 22:36:17 -08:00
Konstantin Lebedev
a7fc723ae0
chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
chrislu
8af61dec91 avoid empty spaces in file name 2024-01-09 09:00:54 -08:00
chrislu
aed54eda61 refactor 2024-01-09 08:01:01 -08:00
chrislu
d51efddf5c flush to disk 
Need to do: read from disk
 2024-01-08 23:27:02 -08:00
jerebear12
8c966ac23b
Removed problematic if statement (#5180) 
This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.
 2024-01-08 11:35:20 -08:00
chrislu
fbb4917e2f Merge branch 'master' into mq-subscribe 2024-01-08 00:08:49 -08:00
chrislu
8ae00e47a6 3.61 2024-01-08 00:04:49 -08:00
chrislu
49428a303b add batch index for each memory buffer 2024-01-08 00:03:08 -08:00
Konstantin Lebedev
b9d32d32e1
chore: filer sync add doDeleteFiles option for create only mode (#5166) 2024-01-06 10:02:16 -08:00
chrislu
d0d24f1e40 rename functions 2024-01-05 17:10:43 -08:00
chrislu
d9bd343055 Merge branch 'master' into mq-subscribe 2024-01-05 15:36:00 -08:00
chrislu
1206313952 add logs on flush 2024-01-05 15:35:42 -08:00
chrislu
aa5b6a8187 adjust logs 2024-01-05 15:35:32 -08:00
chrislu
496fc8fbbf refactor 2024-01-05 15:35:19 -08:00
chrislu
e8611ed85d subscribe with partition offset 2024-01-05 15:24:14 -08:00
chrislu
531f854af2 rename functions 2024-01-05 15:16:53 -08:00
chrislu
f8787a9761 rename functions 2024-01-05 15:16:41 -08:00
chrislu
add16fb1a1 logs 2024-01-05 15:16:22 -08:00
chrislu
ddd0fde094 rename functions 2024-01-05 15:16:01 -08:00
chrislu
ee41dbb7fc rename functions 2024-01-05 15:14:25 -08:00
chrislu
1fed37bed0 add offset 2024-01-05 15:03:16 -08:00
Konstantin Lebedev
3b62b338a5
chore: add maxMB option for webdav (#5165) 2024-01-05 11:10:20 -08:00
chrislu
47a4963d7a subscription start from specified timestamp 2024-01-03 15:57:36 -08:00
chrislu
35869b5c80 subscriber can be notified of the assignment change when topic is just configured 
Next: Subscriber needs to read by the timestamp offset.
 2024-01-03 13:30:30 -08:00
Konstantin Lebedev
4f6172f369
Fix missing VersionConfiguration node in get-bucket-versioning response (#5162) 
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155

* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
 2024-01-03 12:27:13 -08:00
Konstantin Lebedev
3e43778869
fix: webdav avoid create empty files (#5160) 2024-01-03 09:52:51 -08:00
Konstantin Lebedev
b832ddd1ef
fix: return etag with md5 in webdav responses (#5158) 2024-01-03 08:50:32 -08:00
chrislu
efb695fd93 lookup existing topic partitions 2024-01-02 17:29:35 -08:00
chrislu
54e9877093 Merge branch 'master' into mq-subscribe 2024-01-02 13:25:00 -08:00
chrislu
d3688938d9 avoid nil 
fix https://github.com/seaweedfs/seaweedfs/issues/5153

introduced from #5109
 2024-01-02 12:55:34 -08:00
chrislu
923b1e5dad Merge branch 'master' into mq-subscribe 2024-01-01 20:56:12 -08:00
chrislu
d4e91b6ad9 3.60 2024-01-01 19:11:23 -08:00
chrislu
8b4cd50aac log errors 2024-01-01 19:09:17 -08:00
chrislu
bf9ae74384 add logs 2024-01-01 19:09:05 -08:00
chrislu
32bc8d6a38 adjust wait time 2023-12-31 17:42:44 -08:00
chrislu
458ddbf919 clean up dead code 2023-12-31 13:25:33 -08:00
chrislu
2845230329 passing broker into the assignments 2023-12-28 21:13:49 -08:00
chrislu
093fdc1621 subscriber can get assignments 2023-12-28 20:35:15 -08:00
Varun Upadhyay
77626666c5
Minor cleanup & gitignore update (#5144) 2023-12-28 20:25:43 -08:00
chrislu
9ed26cd7b0 Merge branch 'master' into mq-subscribe 2023-12-28 11:57:30 -08:00
chrislu
c950a40aad subscriber keep connected to the balancer 2023-12-28 11:56:37 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency (#4529) 
* fix: increase speed cmd fs meta load

* fix: add wg
 2023-12-28 11:54:37 -08:00
Lars Lehtonen
edc0532b1d
weed/server: remove unneeded err from streamReadOneVolume() signature (#5143) 2023-12-28 11:45:13 -08:00
chrislu
bebbc9fe44 create volume grow request if the selected volume is close to full 2023-12-27 11:45:44 -08:00
chrislu
bb9269f3f7 Revert "Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134)" 
This reverts commit 838578b55f.
 2023-12-24 12:31:08 -08:00
chrislu
c6b1dc7058 remove unused code 2023-12-24 11:11:41 -08:00
vivekkoya
838578b55f
Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134) 2023-12-23 15:01:57 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck (#5133) 
* Add modifyTimeAgo to volume.fsck

* Fix AppendAtNs
 2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler (#5132) 2023-12-23 09:18:16 -08:00
chrislu
03c4b2e988 correct lock accounting 
possibly fix https://github.com/seaweedfs/seaweedfs/issues/5082
 2023-12-22 14:16:23 -08:00
chrislu
c4badf7396 adjust options 2023-12-22 11:41:10 -08:00
chrislu
6324062acc fix 2023-12-22 11:37:15 -08:00
chrislu
e0727071c8 go fmt 2023-12-22 11:33:50 -08:00
chrislu
ee1c9bc314 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-22 11:33:02 -08:00
chrislu
034db049a0 comment fix 2023-12-22 11:33:00 -08:00
chrislu
6df44d3c95 option to filer.copy with public volume server url 
fix https://github.com/seaweedfs/seaweedfs/issues/5126
 2023-12-22 11:32:43 -08:00
chrislu
7a0d1e7fc1 minor 2023-12-22 11:29:54 -08:00
Farbod
c278f49bca
Using filer.remote.sync concurrency in filer.remote.gateway (#5123) 
- Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync

Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>
 2023-12-20 16:25:54 -08:00
jerebear12
06343f8976
Set allowed origins in config (#5109) 
* Add a way to use a JWT in an HTTP only cookie

If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.

* Added a way to specify allowed origins header from config

* Removed unecessary log

* Check list of domains from config or command flag

* Handle default wildcard and change name of config value to cors
 2023-12-20 16:21:11 -08:00
Konstantin Lebedev
3c9bcfb864
chore: add dsn for connection to mysql (#5060) 
* chore: add dsn for connection to mysql

* add comment

* new comment

* fix: validate dsn and adapt password
 2023-12-20 16:20:58 -08:00
chrislu
915416a277 fix wrong parameter 
fix https://github.com/seaweedfs/seaweedfs/pull/5102
 2023-12-13 12:02:57 -08:00
Farbod
d5c01e5ceb
Remote gateway Bug fix (#5100) 2023-12-12 07:46:45 -08:00
chrislu
61b05fad95 fix test 2023-12-11 20:57:29 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue (#5098) 
* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* balance partitions on brokers

* prepare topic partition first and then publish, move partition

* purge unused APIs

* clean up

* adjust logs

* add BalanceTopics() grpc API

* configure topic

* configure topic command

* refactor

* repair missing partitions

* sequence of operations to ensure ordering

* proto to close publishers and consumers

* rename file

* topic partition versioned by unixTimeNs

* create local topic partition

* close publishers

* randomize the client name

* wait until no publishers

* logs

* close stop publisher channel

* send last ack

* comments

* comment

* comments

* support list of brokers

* add cli options

* Update .gitignore

* logs

* return io.eof directly

* refactor

* optionally create topic

* refactoring

* detect consumer disconnection

* sub client wait for more messages

* subscribe by time stamp

* rename

* rename to sub_balancer

* rename

* adjust comments

* rename

* fix compilation

* rename

* rename

* SubscriberToSubCoordinator

* sticky rebalance

* go fmt

* add tests

* tracking topic=>broker

* merge

* comment
 2023-12-11 12:05:54 -08:00
Konstantin Lebedev
125ad8fe63 falls back to update only if error contains msg "duplicate entry" 
https://github.com/seaweedfs/seaweedfs/issues/5062
 2023-12-09 10:18:18 -08:00
chrislu
c1a6e624e5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-12-07 08:13:46 -08:00
chrislu
5fe4c04a1a add missing error in the log 
related to https://github.com/seaweedfs/seaweedfs/issues/5084
 2023-12-07 08:13:42 -08:00
Lars Lehtonen
43a5973a66 weed/storage: fix dropped test errors 2023-12-06 12:05:22 -08:00
jerebear12
d5d9fbb8aa Add a way to use a JWT in an HTTP only cookie 
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
 2023-12-05 08:57:01 -08:00
chrislu
756bcc032d adjust logs 2023-11-27 12:57:29 -08:00
Konstantin Lebedev
dd78397fea logging PickForWrite error 
https://github.com/seaweedfs/seaweedfs/issues/3886
 2023-11-27 12:56:15 -08:00
chrislu
81f11883e3 go fmt 2023-11-26 11:47:20 -08:00
chrislu
7a50762b9e reduce binary size by skipping rclone 2023-11-23 00:48:27 -08:00
zehweh
d818ca9631 fix TestMisplacedChecking() and add test-case 2023-11-21 08:24:34 -08:00
julor
2fa37e0287 Update filer_server_handlers_write_autochunk.go 2023-11-20 11:17:14 -08:00
julor
9e03b916d8 verify content-md5 in header 
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
 2023-11-20 11:17:14 -08:00
SmsS4
6c1ce18541 Fix weed fix 2023-11-20 11:14:19 -08:00
Lars Lehtonen
57801687de weed/command: fix dropped error 2023-11-20 08:05:24 -08:00
Viktor Kuzmin
06e723f629 Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952 2023-11-17 07:38:14 -08:00
Konstantin Lebedev
2b229e98ce fix: add doDeleteFile option for filer backup 2023-11-17 07:37:28 -08:00
chrislu
17710e1ecb adjust logs 2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr
6f559fe838 fs.mergeVolumes: change dir to a named parameter 2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be Fix typo (dist => dest) 2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3 fs.mergeVolumes: Make a plan based on volumes size 2023-11-13 06:41:24 -08:00
Konstantin Lebedev
e55fa68848 fix: excludeFileName is empty 2023-11-13 06:23:46 -08:00
Konstantin Lebedev
3c5295a1a6 filer backup add option for exclude file names that match the regexp to sync on filer 2023-11-13 06:23:46 -08:00
Konstantin Lebedev
5e41ab1370 s3 api add not implemented response for PutBucketVersioning 2023-11-13 06:22:23 -08:00
Konstantin Lebedev
7f6152f130 s3 api add default response for GetBucketVersioning 2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr
b05d682c48 make help command in shell case insensitive 2023-11-13 06:06:45 -08:00
chrislu
27b34f3793 3.59 2023-11-12 22:12:08 -08:00
jsh
47112917ff ec.decode: mount the collected ec shards 2023-11-10 00:04:42 -08:00
chrislu
85bed4d92e Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-09 08:02:57 -08:00
chrislu
c563386675 rename parameters 2023-11-09 08:02:39 -08:00
Lars Lehtonen
5db82e594a weed/server: fix dropped webdav error 2023-11-07 19:05:21 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go (#4988) 2023-11-07 07:58:19 -08:00
Konstantin Lebedev
d693beea87
Replace an invalid method in a metric with an invalid one (#4987) 2023-11-06 10:51:47 -08:00
chrislu
a667f0ad99 fix error reporting 
related to https://github.com/seaweedfs/seaweedfs/issues/4978
 2023-11-05 07:17:16 -08:00
chrislu
b40ba7c7fe Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-11-01 16:28:05 -07:00
chrislu
310eec9a9c skip if path changed 
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
 2023-11-01 16:27:59 -07:00
Nico D'Cotta
3834ad5c48
refactor webdav subdirectory, fixes #4967 (#4969) 
* refactor webdav subdirectory, fixes #4967

* fix bug where Name() was not called on delegate wrappedFileInfo
 2023-11-01 13:12:49 -07:00
tired-engineer
4ed06e9ba5
Pass correct Filer's gRPC port to S3 server. (#4972) 
Fixes seaweedfs/seaweedfs#4971
 2023-11-01 10:11:41 -07:00
SmsS4
ba11331cc1
Fix twice increase metric counter (#4961) 2023-10-30 08:05:27 -07:00
SmsS4
f95848ba7d
Add ErrorGetNotFound and ErrorGetInternal to volume server metrics (#4960) 2023-10-30 07:38:03 -07:00
chrislu
d1e83a3b4d 3.58 2023-10-29 23:04:20 -07:00
duanhongyi
bf839651ec
Fix: issues #4950 (#4952) 2023-10-25 22:09:39 -07:00
zehweh
8b39bbbe2f
fix copying .vif file in VolumeCopy (#4943) 
closes #4934
fixes #2633
might fix #3528
 2023-10-24 09:54:24 -07:00
chrislu
2bc05f70e7 log full percentage 2023-10-22 12:59:34 -07:00
chrislu
612fd7206e fix 2023-10-19 08:10:33 -07:00
Konstantin Lebedev
6c82d7ddfc
[s3] do reload s3 static config (#4923) 
* do reload s3 config

* print error on reload s3 config

* print success msg

* Update weed/s3api/s3api_server.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-10-18 23:26:49 -07:00
lakeland1990
3df0e11ad3
remove compare to fix #4668 issue (#4924) 2023-10-18 09:06:56 -07:00
chrislu
cbc24c7b24 disconnect from old subscribers 2023-10-13 16:48:14 -07:00
James Hedley
6acb43bbbb
Add optional flags to enable mTLS with verification of client certificate (#4910) 2023-10-13 09:02:24 -07:00
chrislu
b62066d38f adding DiskType to /dir/status output 2023-10-12 21:39:11 -07:00
chrislu
deb86ff4a6 upgrading grpc 2023-10-12 21:38:34 -07:00
chrislu
de0b969b36 Revert "rename" 
This reverts commit 35b5264ab7.
 2023-10-12 20:28:11 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file (#4881) 
* fix: disallow file name too long when writing a file

* bool LongerName to MaxFilenameLength

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero (#4909) 
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171

* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-12 08:15:20 -07:00
chrislu
35b5264ab7 rename 2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr
3e8604a4c5
add fs.mergeVolumes command into shell (#4907) 
* add fs.compact command into shell

* rename fs.compact to fs.mergeVolumes
 2023-10-10 12:47:18 -07:00
Konstantin Lebedev
2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment (#4893) 
* fix: skipping checking active volumes with the same number of files at the moment
 https://github.com/seaweedfs/seaweedfs/issues/4140

* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140

* add TestShouldSkipVolume

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-10-09 09:57:26 -07:00
Yuval Yacoby
3fe00996b2
added healthz endpoint to filer (#4899) 2023-10-08 07:03:18 -07:00
Andrew Garrett
082f67bfe9
Fix filer.backup local sink to propagate file mode changes (#4896) 2023-10-06 05:40:20 -07:00
chrislu
0fd7222d65 default to skip if less than 4 nodes 2023-10-05 11:13:48 -07:00
chrislu
7b3c63a8af handle wrapped errors 2023-10-05 08:15:11 -07:00
chrislu
4e493d596c minor 2023-10-05 08:13:53 -07:00
Lars Lehtonen
530bdedf76
weed/operation: fix dropped error (#4888) 2023-10-03 10:55:20 -07:00
chrislu
6ebe26a765 Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)"" 
This reverts commit 567d788928.
 2023-10-03 08:28:52 -07:00
chrislu
7540d43ee9 Revert "Revert "fix compilation"" 
This reverts commit f9abfd0b03.
 2023-10-03 08:28:02 -07:00
chrislu
249c0e06ef Revert "fix compilation" 
This reverts commit 451ec6504d.
 2023-10-03 08:27:50 -07:00
chrislu
451ec6504d fix compilation 2023-10-03 08:15:18 -07:00
chrislu
f9abfd0b03 Revert "fix compilation" 
This reverts commit 0483ba3889.
 2023-10-03 07:58:11 -07:00
chrislu
0483ba3889 fix compilation 2023-10-02 11:55:50 -07:00
chrislu
567d788928 Revert "Revert "Add disk type to prometheus metrics" (#4777)" 
This reverts commit 9215ba24be.
 2023-10-02 11:49:54 -07:00
chrislu
b5070d6280 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-10-02 10:52:40 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg (#4880) 2023-10-02 08:29:09 -07:00
chrislu
bf13f3ced7 add subscriber coordinator 2023-10-02 01:02:27 -07:00
chrislu
734178093e refactor TopicPartition struct 2023-10-02 01:01:45 -07:00
chrislu
2a578b9033 refactor 2023-10-01 11:59:19 -07:00
chrislu
321fda552c comment 2023-10-01 11:36:27 -07:00
chrislu
b02fdeabff RetryForever => RetryUntil 2023-10-01 11:33:56 -07:00
chrislu
ebd6f96d35 adjust mq.proto 2023-09-30 21:47:57 -07:00
chrislu
02cb799481 add a test framework for grpc 2023-09-30 21:36:58 -07:00
chrislu
c4e6545197 fix compilation 2023-09-30 21:26:09 -07:00
chrislu
358cba43ef update proto generated files 2023-09-30 13:19:25 -07:00
chrislu
597ab1e4bf add design for mq balancer 2023-09-30 13:19:05 -07:00
chrislu
d7c211fe8c adjust mq proto 2023-09-30 13:18:49 -07:00
chrislu
40888b5612 align 64bit atomic operation 
fix https://github.com/seaweedfs/seaweedfs/issues/4878
 2023-09-28 07:58:43 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync (#4875) 
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647

* Update weed/shell/command_volume_fix_replication.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* fix: revert this existing do option to positive

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-27 23:12:10 -07:00
Konstantin Lebedev
44906f1f3b
fix: avoid error file name too long when writing a file (#4876) 2023-09-27 05:40:51 -07:00
chrislu
9d589b48e6 rename function 2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica (#4873) 
https://github.com/seaweedfs/seaweedfs/issues/4647

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4 describe a topic 2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance (#4864) 
* fix(ec): ignore 0 byte data files

refers to parts of #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

* fix(ec): ignore volumes not from the current collection during balance

fixes #4861

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>

---------

Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
 2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"" 
This reverts commit 8cb42c39
 2023-09-25 09:35:16 -07:00
chrislu
3d07895518 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-25 09:16:35 -07:00
chrislu
a6e35e4f68 fix compilation 2023-09-25 09:16:33 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) 
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface

https://github.com/seaweedfs/seaweedfs/issues/4519

* fix: test bucket acl default and
adjust the variable names

* fix: s3 api config test

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-25 08:34:12 -07:00
chrislu
c9177c92e5 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-25 08:20:16 -07:00
chrislu
81e57bd45e fix test 2023-09-25 08:20:03 -07:00
Lars Lehtonen
28a3a31b27
weed/replication/sub: fix dropped error (#4865) 2023-09-25 07:33:18 -07:00
chrislu
23f334dd13 comment 2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71 ensure latest stats are reported 2023-09-24 23:34:31 -07:00
chrislu
dff2ce5d2f read existing broker stats 2023-09-24 23:10:33 -07:00
chrislu
1492bf7552 fix listing topics 2023-09-24 23:05:41 -07:00
chrislu
3cf9b8d621 create topic and report topic 2023-09-24 22:00:43 -07:00
chrislu
b3f94feede do create topic 2023-09-24 22:00:43 -07:00
chrislu
d74348048a implement create topic 2023-09-24 22:00:43 -07:00
chrislu
0361c321b4 add CreateTopic API 2023-09-24 22:00:43 -07:00
chrislu
0f8168c0c9 3.57 2023-09-24 21:29:31 -07:00
chrislu
1df4223195 minor 2023-09-24 09:54:57 -07:00
chrislu
10ac25f857 avoid deleting the entry too early 2023-09-23 12:43:00 -07:00
chrislu
c43238b30a fix waiting condition 2023-09-23 12:28:02 -07:00
chrislu
322af1fc04 fix compilation 2023-09-23 12:13:32 -07:00
chrislu
0414b68a4f if shared lock, still need to wait for exclusive lock to release 2023-09-23 12:08:23 -07:00
chrislu
9ba4639a27 adjust for better logs 2023-09-23 11:53:44 -07:00
chrislu
186b1f029b read source uses shared lock 2023-09-23 10:33:44 -07:00
chrislu
2df70ce95f adjust logs 2023-09-21 23:51:32 -07:00
chrislu
3e23421608 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-21 20:08:30 -07:00
chrislu
49a470bcdf reduce logs 2023-09-21 20:08:20 -07:00
Konstantin Lebedev
750a0ba1b2
[iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) 
Replace action read/write to readAcp/writeAcp for handlers with acl query
 https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 14:07:04 -07:00
chrislu
8b2c39f2c0 reduce binary size 2023-09-21 13:22:08 -07:00
chrislu
b388957340 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-21 11:08:30 -07:00
chrislu
31fc165715 mount switch to ordered lock requests 2023-09-21 11:08:26 -07:00
chrislu
7669852241 adjust logs 2023-09-21 11:04:12 -07:00
chrislu
e3b1bacf3f add lock table to ensure ordered locks 2023-09-21 10:24:34 -07:00
Konstantin Lebedev
a46f873edd
[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) 
* Passing test:
test_bucket_acl_default
test_bucket_acl_canned_private_to_private

https://github.com/seaweedfs/seaweedfs/issues/4519

* Update weed/s3api/s3api_bucket_handlers.go

---------

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
 2023-09-21 08:20:05 -07:00
Konstantin Lebedev
d8b424d123
[s3] optimization iam lookup for reducing algorithm complexity (#4857) 
optimization iam lookup for reducing algorithm complexity
https://github.com/seaweedfs/seaweedfs/issues/4519

Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
 2023-09-21 08:19:11 -07:00
chrislu
411bdda08d fix compilation 2023-09-19 16:22:41 -07:00
chrislu
d531b7cbbf merge some previous missing changes 2023-09-19 14:10:54 -07:00
chrislu
b18112ae10 fix compilation bugs during merge 2023-09-19 14:08:17 -07:00
chrislu
6c6655de9d wip 2023-09-19 14:03:42 -07:00
chrislu
cf28108ea4 fix grpd dial option 2023-09-19 14:02:51 -07:00
chrislu
27af11f1e8 Revert "Revert "Merge branch 'master' into sub"" 
This reverts commit 0bb97709d4.
 2023-09-18 18:47:34 -07:00
chrislu
0bb97709d4 Revert "Merge branch 'master' into sub" 
This reverts commit 4d414f54a2, reversing
changes made to 4827425146.
 2023-09-18 16:13:20 -07:00
chrislu
8cb42c39ad Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs" 
This reverts commit 2e5aa06026, reversing
changes made to 4d414f54a2.
 2023-09-18 16:12:50 -07:00
chrislu
2e5aa06026 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-18 16:06:41 -07:00
chrislu
4d414f54a2 Merge branch 'master' into sub 2023-09-18 16:04:59 -07:00
dependabot[bot]
a04bd4d26f
Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850) 
* Bump github.com/rclone/rclone from 1.63.1 to 1.64.0

Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* API changes

* go mod

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2023-09-18 14:43:05 -07:00
chrislu
b64fc44e8d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2023-09-18 09:01:17 -07:00
chrislu
340aea9b34 add broker leader 2023-09-18 09:01:14 -07:00
chrislu
0d390ec882 fix nil 2023-09-18 08:52:35 -07:00
Lars Lehtonen
d83107efe5 fix: weed/storage: dropped error 2023-09-18 08:18:57 -07:00
zemul
91cbaef02b fix: mount proxyByFiler write error: wrong jwt 2023-09-18 07:44:50 -07:00
chrislu
4e17d399a6 refactor 2023-09-17 00:04:20 -07:00
chrislu
7a49d3541a avoid nil 2023-09-17 00:04:13 -07:00