mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-22 16:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,377 Commits 32 Branches 296 Tags 152 MiB
f08f95ac80
Commit Graph

6668 Commits

Author SHA1 Message Date
chrislu
55714f54ab local partition is generated or not 2024-03-15 21:08:42 -07:00
chrislu
56ce040ab6 log errors 2024-03-15 21:05:40 -07:00
chrislu
549455c30d streaming mode 2024-03-15 21:05:24 -07:00
chrislu
f653838f34 follower receive a hello message before actual data 
the leader can ensure the follower has connected
 2024-03-15 20:36:01 -07:00
chrislu
64a32730a2 add size() 2024-03-15 20:34:39 -07:00
Michał Frąckiewicz
205829fa22
Document default method used by compact (#5379) 2024-03-15 12:16:44 -07:00
Konstantin Lebedev
54ee73287d
[filer] replace an invalid methods in a metric with an invalid one (#5378) 
fix: replace an invalid methods in a metric with an invalid one for filer
 2024-03-14 08:19:09 -07:00
chrislu
6f615c62ae Merge branch 'master' into mq-subscribe 2024-03-12 09:19:06 -07:00
chrislu
604091a480 use stopChan to close previous filer peer meta subscription instances 2024-03-12 09:18:54 -07:00
chrislu
1efa502dbb go fmt 2024-03-12 09:17:20 -07:00
chrislu
6f75a0af55 ask follower to follow 2024-03-10 14:34:28 -07:00
chrislu
c1a5952009 rename 2024-03-09 12:57:01 -08:00
chrislu
e7823ee967 retry connecting to broker leader 2024-03-09 12:56:49 -08:00
rustrover
ab70aa92da
remove repetitive words (#5364) 2024-03-09 07:35:47 -08:00
chrislu
3b28433cb1 change LogBuffer visibility in LocalPartition 2024-03-07 10:57:04 -08:00
chrislu
28f8f33d6a include key in LogEntry 2024-03-07 10:53:30 -08:00
chrislu
34f2b600ac each log function adds a "done" return parameter 2024-03-07 10:50:09 -08:00
chrislu
62397f2371 adjust error 2024-03-07 10:47:50 -08:00
chrislu
832b60c13f pass errors 2024-03-07 10:47:06 -08:00
chrislu
49869eec83 edge cases for broker assignment 2024-03-07 10:45:38 -08:00
chrislu
33ab6cfcf7 adjust error 2024-03-07 10:44:09 -08:00
chrislu
8f79bb3987 Merge branch 'master' into mq-subscribe 2024-03-07 10:42:29 -08:00
chrislu
b544a69550 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-03-07 10:42:15 -08:00
chrislu
5c0f01c0eb purge cluster id 2024-03-07 10:42:12 -08:00
Konstantin Lebedev
170b63d6f8
[filer.backup] add param uploader_part_size for S3sink (#5352) 
* fix: install cronie

* chore: refactor configure S3Sink

* chore: refactor cinfig

* add filer-backup compose file

* fix: X-Amz-Meta-Mtime and resolve with comments

* fix: attr mtime

* fix: MaxUploadPartst is reduced to the maximum allowable

* fix: env and force set max MaxUploadParts

* fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB
 2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri
ecc154fa9e
Add deleted bytes to total_disk_size (#5161) 2024-03-06 07:28:46 -08:00
adasauce
3e3e0fa240
feat: add support for S3 ListObjects and ListObjectsV2 (#5350) 2024-03-06 07:24:44 -08:00
Konstantin Lebedev
872b5161e6
[shell] fs log purge (#5344) 
* fix: install cronie

* chore: shell delete old logs

* Revert "fix: install cronie"

This reverts commit 920832b04b.

* fix: dayAgo

* fix: Nit: dayAgo => daysAgo
 2024-03-05 09:09:08 -08:00
chrislu
fe03b1b522 handle single broker case 2024-03-01 05:59:40 -08:00
chrislu
50c5dd7313 test low active brokers 2024-02-29 20:50:57 -08:00
chrislu
7598922b41 assign followers 2024-02-29 14:51:06 -08:00
chrislu
7e443ef0a1 latest protoc-gen-go 2024-02-29 10:06:23 -08:00
chrislu
ead27a4f59 publish init add followers 2024-02-29 09:39:12 -08:00
chrislu
1b4484bf0a go fmt 2024-02-29 09:38:52 -08:00
chrislu
2a7028373d Merge branch 'master' into mq-subscribe 2024-02-29 06:22:39 -08:00
Sean Ross
0833057503
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… (#5351) 
Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.
 2024-02-29 02:16:28 -08:00
jerebear12
85d6d5371b
Disable filer UI in configuration (#5297) 
* Add filer.ui.enabled configuration property

* Add filer.expose_directory_metadata to config

* Ammend commit

* Remove ShowUI reference

* Update all routes that allow directory metadata

* Add cmd flag to server.go
 2024-02-27 08:38:55 -08:00
chrislu
76ae39a363 Merge branch 'master' into mq-subscribe 2024-02-26 07:03:09 -08:00
chrislu
54d7748a4a 3.63 2024-02-25 22:17:28 -08:00
chrislu
a52ce076fc Merge branch 'master' into mq-subscribe 2024-02-25 08:00:37 -08:00
Konstantin Lebedev
7187346cc1
avoid unexpected compact size (#5272) 
https://github.com/seaweedfs/seaweedfs/issues/5215
 2024-02-24 05:27:35 -08:00
chrislu
7c45992c79 when verifyNeedleIntegrity, do not truncate files 
related to https://github.com/seaweedfs/seaweedfs/pull/5259
 2024-02-24 02:37:03 -08:00
Chris Lu
32cb9b3276
fix unaligned 64-bit atomic operation in weed mount on arm when readi… (#5308) 
fix unaligned 64-bit atomic operation in weed mount on arm when reading a file

fix #5306
 2024-02-19 18:51:45 -08:00
7y-9
41ec9d7759
refactor: put the auth outside (#5313) 2024-02-19 03:07:48 -08:00
7y-9
f037c09c11
fix: only admin auth can delete S3 bucket (#5312) 2024-02-19 00:38:05 -08:00
Konstantin Lebedev
6181aa7594
fix: fs verify error counter (#5261) 2024-02-15 01:31:51 -08:00
Sébastien
c694cb4e7d
filer: reduce delete entry ErrNotFound logs (#5301) 2024-02-14 14:22:08 -08:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304) 2024-02-14 14:18:56 -08:00
Justin Dhillon
a48e2ec45b
Fix Broken Links (#5287) 
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize

* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize

* remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css

* https://github.com/AShiou/hof
 2024-02-14 08:26:38 -08:00
Chris Lu
16b4f9d9ab subscriber find broker leader first 2024-02-05 23:14:25 -08:00
Chris Lu
e27deed4bc upgrade protoc 2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af proxy to broker leader 2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26 report EOF error 2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864 minor 2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8 fix on finding lock owner 2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header (#5280) 
* Add s3 postpolicy support header

* optimized code
 2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224 fix divide by zero 2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f fix lock TTL 2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99 less verbose 2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c Merge branch 'master' into mq-subscribe 2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f fix when two consecutive chunks with the same modified_ts_ns 
fix https://github.com/seaweedfs/seaweedfs/issues/5276
 2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d avoid too large expiration time 2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType (#5275) 2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c avoid timestamp overflow 2024-02-03 13:45:59 -08:00
chrislu
d41792461c lock returns host and owner 2024-02-02 15:54:57 -08:00
chrislu
d30150dde1 use a short-lived lock 2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709 fix nil error 2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822 re-lock if the lock owner is not found 2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types (#5245) 
* week/shell: Cluster check other disk types

The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.

This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Replace loop that copies slice

Use the following construct instead of a `for` loop:

```golang
x = append(x, y...)
```

See https://staticcheck.dev/docs/checks#S1011.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Check disk types when filer is in use

Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

---------

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
 2024-01-29 10:36:37 -08:00
Sébastien
0775d05a23
fix: http range request return status 500 (#5251) 
When volume server unavailable for at least one chunk; was returning status 206.

Split `StreamContent` in two parts,
- first prepare, to get chunk info and return stream function
- then write chunk, with that stream function

That allow to catch error in first step before setting response status code in `processRangeRequest`
 2024-01-29 10:35:52 -08:00
chrislu
0b2e5ddc7c wait 3 seconds before shutting down publish client, to wait for all messages to be received 2024-01-28 22:04:42 -08:00
chrislu
545d5fbdf6 unused code 2024-01-28 22:03:13 -08:00
chrislu
a507069d19 lock for creating local partition 2024-01-28 15:55:26 -08:00
chrislu
9e6ea80dff read existing topic conf 2024-01-28 14:27:52 -08:00
chrislu
a297247137 wait for each publish clients 2024-01-28 14:27:32 -08:00
chrislu
b6c5e57c30 read will block if no items 2024-01-28 13:10:34 -08:00
chrislu
0bf5424a2e adjust visibility 2024-01-28 13:09:30 -08:00
chrislu
dedfd31dfb refactor 2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f refactor 2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff wait for publishing clients 2024-01-28 12:06:58 -08:00
chrislu
f8c55f101e clean up 2024-01-28 12:06:30 -08:00
chrislu
d40225ecd2 return the lookup results 2024-01-28 12:05:44 -08:00
chrislu
841fafd0a8 publish to input buffer 
currently, the input buffer may not exist when start to publish
 2024-01-27 23:43:22 -08:00
chrislu
5dc1362bdc close the input 2024-01-27 16:12:49 -08:00
chrislu
fa835c9319 Update buffered_queue_test.go 2024-01-27 14:46:12 -08:00
Chris Lu
b6de35cdb2 moved. there are some deadlock. WIP 2024-01-27 13:51:19 -08:00
chrislu
91af1f3069 schedule jobs 2024-01-26 14:09:57 -08:00
Benoît Knecht
f6e39139d8
weed/storage: Fix volume info file permissions (#5246) 2024-01-26 08:31:27 -08:00
chrislu
08c5fba825 adds a buffered queue 2024-01-25 20:22:41 -08:00
chrislu
cb50b720fd Revert "Fix weed fix" 
This reverts commit 6c1ce18541.
 2024-01-25 08:09:36 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic (#5241) 2024-01-25 06:19:58 -08:00
Konstantin Lebedev
0f8e76bbd6
fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic (#5238) 2024-01-25 00:17:32 -08:00
Konstantin Lebedev
0b71fae2e3
fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
chrislu
025f8eb37b return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:48:42 -08:00
chrislu
d4e66cc140 return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:47:31 -08:00
chrislu
a8ddbf9ed2 Merge branch 'master' into mq-subscribe 2024-01-23 19:41:16 -08:00
chrislu
49fcb48e04 same logic as grpc assign 
fix https://github.com/seaweedfs/seaweedfs/issues/5213
 2024-01-23 19:39:59 -08:00
chrislu
d268fbe18a when configure, cancel existing assignments 2024-01-22 11:57:17 -08:00