mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-22 16:17:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,377 Commits 32 Branches 296 Tags 152 MiB
f08f95ac80
Commit Graph

6668 Commits

Author SHA1 Message Date
Chris Lu
e27deed4bc upgrade protoc 2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af proxy to broker leader 2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26 report EOF error 2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864 minor 2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8 fix on finding lock owner 2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833 proxy requests to lock owner 2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header (#5280) 
* Add s3 postpolicy support header

* optimized code
 2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224 fix divide by zero 2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f fix lock TTL 2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99 less verbose 2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c Merge branch 'master' into mq-subscribe 2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f fix when two consecutive chunks with the same modified_ts_ns 
fix https://github.com/seaweedfs/seaweedfs/issues/5276
 2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d avoid too large expiration time 2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType (#5275) 2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c avoid timestamp overflow 2024-02-03 13:45:59 -08:00
chrislu
d41792461c lock returns host and owner 2024-02-02 15:54:57 -08:00
chrislu
d30150dde1 use a short-lived lock 2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709 fix nil error 2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822 re-lock if the lock owner is not found 2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types (#5245) 
* week/shell: Cluster check other disk types

The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.

This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Replace loop that copies slice

Use the following construct instead of a `for` loop:

```golang
x = append(x, y...)
```

See https://staticcheck.dev/docs/checks#S1011.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

* weed/shell: Check disk types when filer is in use

Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>

---------

Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
 2024-01-29 10:36:37 -08:00
Sébastien
0775d05a23
fix: http range request return status 500 (#5251) 
When volume server unavailable for at least one chunk; was returning status 206.

Split `StreamContent` in two parts,
- first prepare, to get chunk info and return stream function
- then write chunk, with that stream function

That allow to catch error in first step before setting response status code in `processRangeRequest`
 2024-01-29 10:35:52 -08:00
chrislu
0b2e5ddc7c wait 3 seconds before shutting down publish client, to wait for all messages to be received 2024-01-28 22:04:42 -08:00
chrislu
545d5fbdf6 unused code 2024-01-28 22:03:13 -08:00
chrislu
a507069d19 lock for creating local partition 2024-01-28 15:55:26 -08:00
chrislu
9e6ea80dff read existing topic conf 2024-01-28 14:27:52 -08:00
chrislu
a297247137 wait for each publish clients 2024-01-28 14:27:32 -08:00
chrislu
b6c5e57c30 read will block if no items 2024-01-28 13:10:34 -08:00
chrislu
0bf5424a2e adjust visibility 2024-01-28 13:09:30 -08:00
chrislu
dedfd31dfb refactor 2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f refactor 2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff wait for publishing clients 2024-01-28 12:06:58 -08:00
chrislu
f8c55f101e clean up 2024-01-28 12:06:30 -08:00
chrislu
d40225ecd2 return the lookup results 2024-01-28 12:05:44 -08:00
chrislu
841fafd0a8 publish to input buffer 
currently, the input buffer may not exist when start to publish
 2024-01-27 23:43:22 -08:00
chrislu
5dc1362bdc close the input 2024-01-27 16:12:49 -08:00
chrislu
fa835c9319 Update buffered_queue_test.go 2024-01-27 14:46:12 -08:00
Chris Lu
b6de35cdb2 moved. there are some deadlock. WIP 2024-01-27 13:51:19 -08:00
chrislu
91af1f3069 schedule jobs 2024-01-26 14:09:57 -08:00
Benoît Knecht
f6e39139d8
weed/storage: Fix volume info file permissions (#5246) 2024-01-26 08:31:27 -08:00
chrislu
08c5fba825 adds a buffered queue 2024-01-25 20:22:41 -08:00
chrislu
cb50b720fd Revert "Fix weed fix" 
This reverts commit 6c1ce18541.
 2024-01-25 08:09:36 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic (#5241) 2024-01-25 06:19:58 -08:00
Konstantin Lebedev
0f8e76bbd6
fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239) 2024-01-25 00:23:24 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic (#5238) 2024-01-25 00:17:32 -08:00
Konstantin Lebedev
0b71fae2e3
fix: s3tests test_get_object_ifnonematch_good (#5237) 2024-01-24 11:05:55 -08:00
chrislu
025f8eb37b return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:48:42 -08:00
chrislu
d4e66cc140 return dir assign counter 
fix https://github.com/seaweedfs/seaweedfs/issues/5231
 2024-01-23 19:47:31 -08:00
chrislu
a8ddbf9ed2 Merge branch 'master' into mq-subscribe 2024-01-23 19:41:16 -08:00
chrislu
49fcb48e04 same logic as grpc assign 
fix https://github.com/seaweedfs/seaweedfs/issues/5213
 2024-01-23 19:39:59 -08:00
chrislu
d268fbe18a when configure, cancel existing assignments 2024-01-22 11:57:17 -08:00
chrislu
3ff6b31d94 sub broker wait for partition 2024-01-22 11:56:55 -08:00
chrislu
2beaa2d0b3 pub/sub brokers check filer for assigned partitions 2024-01-22 11:44:21 -08:00
chrislu
c3f8530f97 refactor 2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0 no need to assign partitions explicitly 2024-01-22 11:04:47 -08:00
chrislu
428fea45f3 updated and added assignments 2024-01-22 11:04:24 -08:00
chrislu
e8b05ecc91 add/remove assigned partitions 2024-01-22 10:47:39 -08:00
chrislu
861ad732ca refactor 2024-01-22 09:00:22 -08:00
chrislu
47924afa1c refactor 2024-01-22 08:52:44 -08:00
chrislu
581cf6644c lookup reads from filer instead of local memory 2024-01-22 00:52:55 -08:00
chrislu
7121ae9617 separate out read topic conf and ensure topic active assignments 2024-01-22 00:51:31 -08:00
chrislu
c77d35313e pub/sub broker only check local assigned partitions 2024-01-22 00:49:57 -08:00
chrislu
b0a2e9aea3 fix assignments if brokers changed 2024-01-21 01:27:22 -08:00
chrislu
9ce7c482b3 refactor 2024-01-21 01:24:12 -08:00
chrislu
a5b906c0dd refactor 2024-01-21 01:23:43 -08:00
chrislu
34839237ab refactor 2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30 refactor 2024-01-20 11:41:11 -08:00
chrislu
541140f735 rename 2024-01-19 11:43:40 -08:00
chrislu
bb867694d8 skip updating multipart .upload files 2024-01-18 09:24:52 -08:00
chrislu
4ca6b1be3c skip updating the .upload multipart files 2024-01-18 09:24:52 -08:00
chrislu
d848efed1b skip updating multipart .upload files 2024-01-18 09:24:26 -08:00
chrislu
aca3c33bab skip updating the .upload multipart files 2024-01-18 09:23:24 -08:00
chrislu
314e25aad6 Merge branch 'master' into mq-subscribe 2024-01-18 09:17:16 -08:00
chrislu
5c90b27b0a go fmt 2024-01-18 09:17:08 -08:00
chrislu
263f1f3d04 Merge branch 'master' into mq-subscribe 2024-01-18 09:16:46 -08:00
chrislu
83e4b02517 fix 2024-01-18 09:16:20 -08:00
chrislu
2fa519bcb5 Merge branch 'pr/5216' 2024-01-18 09:14:01 -08:00
chrislu
15b66a6633 refactor 2024-01-18 09:13:14 -08:00
chrislu
3002087541 http assign logic should be the same as grpc assign 
similar logic as bebbc9fe44
 2024-01-18 08:58:53 -08:00
XIAOYQ
be166b434f fix: skip s3 .uploads 2024-01-18 22:13:46 +08:00
chrislu
c8090b1f99 send assignments to newly connected subscribers 2024-01-17 22:44:39 -08:00
chrislu
c616081ab9 refactor 2024-01-17 22:29:59 -08:00
Konstantin Lebedev
4e9ea1e628
chore: filer healthz handler check filer store (#5208) 2024-01-17 07:46:37 -08:00
Konstantin Lebedev
f9cf13fada
[filer] avoid return http ok for not allowed methods (#5209) 2024-01-17 07:17:07 -08:00
Andrew Azores
2eb82778bc
chore: fix typos in scaffold help output (#5211) 2024-01-17 07:10:17 -08:00
chrislu
bc8d2a01cc create local topic partition from config on filer 2024-01-17 00:15:52 -08:00
chrislu
fdf0ea8e11 passing partition info 2024-01-16 09:30:57 -08:00
chrislu
be0c426dc7 simplify to LookupTopicPartitions(topic) 2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5 simplify api 2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0 remove isForPublish from LookupTopicBrokers 
also adds a return parameter: whether the topic exists or not
 2024-01-16 08:52:42 -08:00
chrislu
3795d8dca8 release local topic partition if no publisher and subscribers 2024-01-16 08:43:07 -08:00
chrislu
f782165638 rename 2024-01-16 08:40:45 -08:00
chrislu
dd1ec70e62 rename 2024-01-16 08:40:23 -08:00
chrislu
1877ce5126 rename 2024-01-15 21:31:21 -08:00
chrislu
ba73199174 reuse local partition 2024-01-15 21:22:41 -08:00
chrislu
33d5343ea1 Merge branch 'master' into mq-subscribe 2024-01-15 20:56:21 -08:00
chrislu
59b8af99b0 3.62 2024-01-15 20:44:00 -08:00
chrislu
6d45bb3038 Merge branch 'master' into mq-subscribe 2024-01-15 20:43:03 -08:00
chrislu
7e6497cc1c adjust publisher subscriber 2024-01-15 20:42:46 -08:00
skycope
316ae45795
can vacuum volume when size mismatch (#5200) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-01-15 16:16:46 -08:00
chrislu
026c54a9bb fix publisher 2024-01-15 00:20:21 -08:00
chrislu
fa59a5d67e read from disk if not in memory 2024-01-15 00:20:12 -08:00
chrislu
49f7de9daa Merge branch 'master' into mq-subscribe 2024-01-13 17:52:18 -08:00
chrislu
d6ba97219b refactoring 2024-01-13 17:51:53 -08:00
Konstantin Lebedev
8d23e36c45
fix: doDeleteFiles deletes files (#5198) 2024-01-12 11:04:29 -08:00
Konstantin Lebedev
1169f94310
Fix filer sync set offset (#5197) 
* fix: compose 2mount with sync

* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
 2024-01-12 10:57:18 -08:00
chrislu
3aa3991f0f adjust client side logs 2024-01-11 23:08:53 -08:00
chrislu
7afaad31a4 wait on local topic partition creation 2024-01-11 23:08:02 -08:00
chrislu
61dbdd0ff6 log ctrl messages 2024-01-11 23:07:32 -08:00
chrislu
f750a5e03b passing timestamp 2024-01-11 23:03:55 -08:00
chrislu
45994641e9 lint 2024-01-11 23:03:35 -08:00
chrislu
70a011a224 Merge branch 'master' into mq-subscribe 2024-01-11 20:58:57 -08:00
skycope
0e8a54f6f6
fix write volume over size MaxPossibleVolumeSize (#5190) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-01-11 20:23:46 -08:00
chrislu
050efe6c08 factor in existing ec volume count when estimating max volume count 
related to https://github.com/seaweedfs/seaweedfs/issues/5191
 2024-01-11 08:19:09 -08:00
chrislu
188f471a24 logging 2024-01-11 07:55:26 -08:00
chrislu
6a7a679137 passing partition generation timestamp 2024-01-10 22:36:17 -08:00
Konstantin Lebedev
a7fc723ae0
chore: add status code for request_total metrics (#5188) 2024-01-10 10:05:27 -08:00
chrislu
8af61dec91 avoid empty spaces in file name 2024-01-09 09:00:54 -08:00
chrislu
aed54eda61 refactor 2024-01-09 08:01:01 -08:00
chrislu
d51efddf5c flush to disk 
Need to do: read from disk
 2024-01-08 23:27:02 -08:00
jerebear12
8c966ac23b
Removed problematic if statement (#5180) 
This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.
 2024-01-08 11:35:20 -08:00
chrislu
fbb4917e2f Merge branch 'master' into mq-subscribe 2024-01-08 00:08:49 -08:00
chrislu
8ae00e47a6 3.61 2024-01-08 00:04:49 -08:00
chrislu
49428a303b add batch index for each memory buffer 2024-01-08 00:03:08 -08:00
Konstantin Lebedev
b9d32d32e1
chore: filer sync add doDeleteFiles option for create only mode (#5166) 2024-01-06 10:02:16 -08:00
chrislu
d0d24f1e40 rename functions 2024-01-05 17:10:43 -08:00
chrislu
d9bd343055 Merge branch 'master' into mq-subscribe 2024-01-05 15:36:00 -08:00
chrislu
1206313952 add logs on flush 2024-01-05 15:35:42 -08:00
chrislu
aa5b6a8187 adjust logs 2024-01-05 15:35:32 -08:00
chrislu
496fc8fbbf refactor 2024-01-05 15:35:19 -08:00
chrislu
e8611ed85d subscribe with partition offset 2024-01-05 15:24:14 -08:00
chrislu
531f854af2 rename functions 2024-01-05 15:16:53 -08:00
chrislu
f8787a9761 rename functions 2024-01-05 15:16:41 -08:00
chrislu
add16fb1a1 logs 2024-01-05 15:16:22 -08:00
chrislu
ddd0fde094 rename functions 2024-01-05 15:16:01 -08:00
chrislu
ee41dbb7fc rename functions 2024-01-05 15:14:25 -08:00
chrislu
1fed37bed0 add offset 2024-01-05 15:03:16 -08:00
Konstantin Lebedev
3b62b338a5
chore: add maxMB option for webdav (#5165) 2024-01-05 11:10:20 -08:00
chrislu
47a4963d7a subscription start from specified timestamp 2024-01-03 15:57:36 -08:00
chrislu
35869b5c80 subscriber can be notified of the assignment change when topic is just configured 
Next: Subscriber needs to read by the timestamp offset.
 2024-01-03 13:30:30 -08:00
Konstantin Lebedev
4f6172f369
Fix missing VersionConfiguration node in get-bucket-versioning response (#5162) 
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155

* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
 2024-01-03 12:27:13 -08:00
Konstantin Lebedev
3e43778869
fix: webdav avoid create empty files (#5160) 2024-01-03 09:52:51 -08:00
Konstantin Lebedev
b832ddd1ef
fix: return etag with md5 in webdav responses (#5158) 2024-01-03 08:50:32 -08:00
chrislu
efb695fd93 lookup existing topic partitions 2024-01-02 17:29:35 -08:00
chrislu
54e9877093 Merge branch 'master' into mq-subscribe 2024-01-02 13:25:00 -08:00
chrislu
d3688938d9 avoid nil 
fix https://github.com/seaweedfs/seaweedfs/issues/5153

introduced from #5109
 2024-01-02 12:55:34 -08:00
chrislu
923b1e5dad Merge branch 'master' into mq-subscribe 2024-01-01 20:56:12 -08:00
chrislu
d4e91b6ad9 3.60 2024-01-01 19:11:23 -08:00
chrislu
8b4cd50aac log errors 2024-01-01 19:09:17 -08:00
chrislu
bf9ae74384 add logs 2024-01-01 19:09:05 -08:00
chrislu
32bc8d6a38 adjust wait time 2023-12-31 17:42:44 -08:00