Sébastien
c694cb4e7d
filer: reduce delete entry ErrNotFound logs ( #5301 )
2024-02-14 14:22:08 -08:00
Konstantin Lebedev
8fb978d6f6
[volume] requests that were canceled by the client are not an internal server problem(500 => 499) ( #5304 )
2024-02-14 14:18:56 -08:00
Justin Dhillon
a48e2ec45b
Fix Broken Links ( #5287 )
...
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize
* https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize
* remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css
* https://github.com/AShiou/hof
2024-02-14 08:26:38 -08:00
Chris Lu
16b4f9d9ab
subscriber find broker leader first
2024-02-05 23:14:25 -08:00
Chris Lu
e27deed4bc
upgrade protoc
2024-02-05 18:39:08 -08:00
Chris Lu
656b78d1af
proxy to broker leader
2024-02-05 18:00:50 -08:00
Chris Lu
90fcde0e26
report EOF error
2024-02-05 17:47:34 -08:00
Chris Lu
07028a4864
minor
2024-02-05 17:30:20 -08:00
Chris Lu
fa72761bb8
fix on finding lock owner
2024-02-05 17:30:14 -08:00
Chris Lu
4dc6681833
proxy requests to lock owner
2024-02-05 16:46:40 -08:00
sxlehua
ffe5f19aa0
Add s3 postpolicy support header ( #5280 )
...
* Add s3 postpolicy support header
* optimized code
2024-02-05 07:55:07 -08:00
Chris Lu
0d74ac9224
fix divide by zero
2024-02-04 13:38:00 -08:00
Chris Lu
f9131de43f
fix lock TTL
2024-02-04 12:47:21 -08:00
Chris Lu
44b49bec99
less verbose
2024-02-04 12:47:07 -08:00
Chris Lu
53612b770c
Merge branch 'master' into mq-subscribe
2024-02-04 10:44:01 -08:00
Chris Lu
56df44845f
fix when two consecutive chunks with the same modified_ts_ns
...
fix https://github.com/seaweedfs/seaweedfs/issues/5276
2024-02-04 09:22:24 -08:00
Chris Lu
0a12301b3d
avoid too large expiration time
2024-02-04 09:20:21 -08:00
sxlehua
0b49c16361
Adapt S3 POST ContentType ( #5275 )
2024-02-03 22:33:34 -08:00
Chris Lu
1b5ba4190c
avoid timestamp overflow
2024-02-03 13:45:59 -08:00
chrislu
d41792461c
lock returns host and owner
2024-02-02 15:54:57 -08:00
chrislu
d30150dde1
use a short-lived lock
2024-02-01 23:01:44 -08:00
chrislu
c1bb76d709
fix nil error
2024-01-30 08:21:42 -08:00
chrislu
ac50d8a822
re-lock if the lock owner is not found
2024-01-29 22:46:23 -08:00
Benoît Knecht
56287bd07d
weed/shell: Cluster check other disk types ( #5245 )
...
* week/shell: Cluster check other disk types
The `cluster.check` command only took the empty (`""`) and `hdd` disk types
into consideration, but a cluster with only `ssd` or `nvme` disk types would be
equally valid.
This commit simply checks that _any_ disk type is defined, and that some
volumes are available for it.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
* weed/shell: Replace loop that copies slice
Use the following construct instead of a `for` loop:
```golang
x = append(x, y...)
```
See https://staticcheck.dev/docs/checks#S1011 .
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
* weed/shell: Check disk types when filer is in use
Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes,
so make sure those disk types exist and have volumes associated with them when
Filer is deployed in the cluster.
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
---------
Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>
2024-01-29 10:36:37 -08:00
Sébastien
0775d05a23
fix: http range request return status 500 ( #5251 )
...
When volume server unavailable for at least one chunk; was returning status 206.
Split `StreamContent` in two parts,
- first prepare, to get chunk info and return stream function
- then write chunk, with that stream function
That allow to catch error in first step before setting response status code in `processRangeRequest`
2024-01-29 10:35:52 -08:00
chrislu
0b2e5ddc7c
wait 3 seconds before shutting down publish client, to wait for all messages to be received
2024-01-28 22:04:42 -08:00
chrislu
545d5fbdf6
unused code
2024-01-28 22:03:13 -08:00
chrislu
a507069d19
lock for creating local partition
2024-01-28 15:55:26 -08:00
chrislu
9e6ea80dff
read existing topic conf
2024-01-28 14:27:52 -08:00
chrislu
a297247137
wait for each publish clients
2024-01-28 14:27:32 -08:00
chrislu
b6c5e57c30
read will block if no items
2024-01-28 13:10:34 -08:00
chrislu
0bf5424a2e
adjust visibility
2024-01-28 13:09:30 -08:00
chrislu
dedfd31dfb
refactor
2024-01-28 12:30:08 -08:00
chrislu
cbf750a31f
refactor
2024-01-28 12:23:20 -08:00
chrislu
b51dfe2bff
wait for publishing clients
2024-01-28 12:06:58 -08:00
chrislu
f8c55f101e
clean up
2024-01-28 12:06:30 -08:00
chrislu
d40225ecd2
return the lookup results
2024-01-28 12:05:44 -08:00
chrislu
841fafd0a8
publish to input buffer
...
currently, the input buffer may not exist when start to publish
2024-01-27 23:43:22 -08:00
chrislu
5dc1362bdc
close the input
2024-01-27 16:12:49 -08:00
chrislu
fa835c9319
Update buffered_queue_test.go
2024-01-27 14:46:12 -08:00
Chris Lu
b6de35cdb2
moved. there are some deadlock. WIP
2024-01-27 13:51:19 -08:00
chrislu
91af1f3069
schedule jobs
2024-01-26 14:09:57 -08:00
Benoît Knecht
f6e39139d8
weed/storage: Fix volume info file permissions ( #5246 )
2024-01-26 08:31:27 -08:00
chrislu
08c5fba825
adds a buffered queue
2024-01-25 20:22:41 -08:00
chrislu
cb50b720fd
Revert "Fix weed fix"
...
This reverts commit 6c1ce18541
2024-01-25 08:09:36 -08:00
Benoît Knecht
1f08010ef0
weed/shell: Cleanup volume balance logic ( #5241 )
2024-01-25 06:19:58 -08:00
Konstantin Lebedev
0f8e76bbd6
fix: clean metric MasterReplicaPlacementMismatch for unregister volume ( #5239 )
2024-01-25 00:23:24 -08:00
Benoît Knecht
a6aee847b9
weed/shell: Fix volume.balance logic ( #5238 )
2024-01-25 00:17:32 -08:00
Konstantin Lebedev
0b71fae2e3
fix: s3tests test_get_object_ifnonematch_good ( #5237 )
2024-01-24 11:05:55 -08:00
chrislu
025f8eb37b
return dir assign counter
...
fix https://github.com/seaweedfs/seaweedfs/issues/5231
2024-01-23 19:48:42 -08:00
chrislu
d4e66cc140
return dir assign counter
...
fix https://github.com/seaweedfs/seaweedfs/issues/5231
2024-01-23 19:47:31 -08:00
chrislu
a8ddbf9ed2
Merge branch 'master' into mq-subscribe
2024-01-23 19:41:16 -08:00
chrislu
49fcb48e04
same logic as grpc assign
...
fix https://github.com/seaweedfs/seaweedfs/issues/5213
2024-01-23 19:39:59 -08:00
chrislu
d268fbe18a
when configure, cancel existing assignments
2024-01-22 11:57:17 -08:00
chrislu
3ff6b31d94
sub broker wait for partition
2024-01-22 11:56:55 -08:00
chrislu
2beaa2d0b3
pub/sub brokers check filer for assigned partitions
2024-01-22 11:44:21 -08:00
chrislu
c3f8530f97
refactor
2024-01-22 11:06:53 -08:00
chrislu
5f35857cd0
no need to assign partitions explicitly
2024-01-22 11:04:47 -08:00
chrislu
428fea45f3
updated and added assignments
2024-01-22 11:04:24 -08:00
chrislu
e8b05ecc91
add/remove assigned partitions
2024-01-22 10:47:39 -08:00
chrislu
861ad732ca
refactor
2024-01-22 09:00:22 -08:00
chrislu
47924afa1c
refactor
2024-01-22 08:52:44 -08:00
chrislu
581cf6644c
lookup reads from filer instead of local memory
2024-01-22 00:52:55 -08:00
chrislu
7121ae9617
separate out read topic conf and ensure topic active assignments
2024-01-22 00:51:31 -08:00
chrislu
c77d35313e
pub/sub broker only check local assigned partitions
2024-01-22 00:49:57 -08:00
chrislu
b0a2e9aea3
fix assignments if brokers changed
2024-01-21 01:27:22 -08:00
chrislu
9ce7c482b3
refactor
2024-01-21 01:24:12 -08:00
chrislu
a5b906c0dd
refactor
2024-01-21 01:23:43 -08:00
chrislu
34839237ab
refactor
2024-01-20 12:16:40 -08:00
chrislu
2828ccbb30
refactor
2024-01-20 11:41:11 -08:00
chrislu
541140f735
rename
2024-01-19 11:43:40 -08:00
chrislu
bb867694d8
skip updating multipart .upload files
2024-01-18 09:24:52 -08:00
chrislu
4ca6b1be3c
skip updating the .upload multipart files
2024-01-18 09:24:52 -08:00
chrislu
d848efed1b
skip updating multipart .upload files
2024-01-18 09:24:26 -08:00
chrislu
aca3c33bab
skip updating the .upload multipart files
2024-01-18 09:23:24 -08:00
chrislu
314e25aad6
Merge branch 'master' into mq-subscribe
2024-01-18 09:17:16 -08:00
chrislu
5c90b27b0a
go fmt
2024-01-18 09:17:08 -08:00
chrislu
263f1f3d04
Merge branch 'master' into mq-subscribe
2024-01-18 09:16:46 -08:00
chrislu
83e4b02517
fix
2024-01-18 09:16:20 -08:00
chrislu
2fa519bcb5
Merge branch 'pr/5216'
2024-01-18 09:14:01 -08:00
chrislu
15b66a6633
refactor
2024-01-18 09:13:14 -08:00
chrislu
3002087541
http assign logic should be the same as grpc assign
...
similar logic as bebbc9fe44
2024-01-18 08:58:53 -08:00
XIAOYQ
be166b434f
fix: skip s3 .uploads
2024-01-18 22:13:46 +08:00
chrislu
c8090b1f99
send assignments to newly connected subscribers
2024-01-17 22:44:39 -08:00
chrislu
c616081ab9
refactor
2024-01-17 22:29:59 -08:00
Konstantin Lebedev
4e9ea1e628
chore: filer healthz handler check filer store ( #5208 )
2024-01-17 07:46:37 -08:00
Konstantin Lebedev
f9cf13fada
[filer] avoid return http ok for not allowed methods ( #5209 )
2024-01-17 07:17:07 -08:00
Andrew Azores
2eb82778bc
chore: fix typos in scaffold help output ( #5211 )
2024-01-17 07:10:17 -08:00
chrislu
bc8d2a01cc
create local topic partition from config on filer
2024-01-17 00:15:52 -08:00
chrislu
fdf0ea8e11
passing partition info
2024-01-16 09:30:57 -08:00
chrislu
be0c426dc7
simplify to LookupTopicPartitions(topic)
2024-01-16 09:30:46 -08:00
chrislu
db3670a3a5
simplify api
2024-01-16 08:55:47 -08:00
chrislu
34a78ffad0
remove isForPublish from LookupTopicBrokers
...
also adds a return parameter: whether the topic exists or not
2024-01-16 08:52:42 -08:00
chrislu
3795d8dca8
release local topic partition if no publisher and subscribers
2024-01-16 08:43:07 -08:00
chrislu
f782165638
rename
2024-01-16 08:40:45 -08:00
chrislu
dd1ec70e62
rename
2024-01-16 08:40:23 -08:00
chrislu
1877ce5126
rename
2024-01-15 21:31:21 -08:00
chrislu
ba73199174
reuse local partition
2024-01-15 21:22:41 -08:00
chrislu
33d5343ea1
Merge branch 'master' into mq-subscribe
2024-01-15 20:56:21 -08:00
chrislu
59b8af99b0
3.62
2024-01-15 20:44:00 -08:00
chrislu
6d45bb3038
Merge branch 'master' into mq-subscribe
2024-01-15 20:43:03 -08:00
chrislu
7e6497cc1c
adjust publisher subscriber
2024-01-15 20:42:46 -08:00
skycope
316ae45795
can vacuum volume when size mismatch ( #5200 )
...
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-15 16:16:46 -08:00
chrislu
026c54a9bb
fix publisher
2024-01-15 00:20:21 -08:00
chrislu
fa59a5d67e
read from disk if not in memory
2024-01-15 00:20:12 -08:00
chrislu
49f7de9daa
Merge branch 'master' into mq-subscribe
2024-01-13 17:52:18 -08:00
chrislu
d6ba97219b
refactoring
2024-01-13 17:51:53 -08:00
Konstantin Lebedev
8d23e36c45
fix: doDeleteFiles deletes files ( #5198 )
2024-01-12 11:04:29 -08:00
Konstantin Lebedev
1169f94310
Fix filer sync set offset ( #5197 )
...
* fix: compose 2mount with sync
* fix: DATA RACE
https://github.com/seaweedfs/seaweedfs/issues/5194
https://github.com/seaweedfs/seaweedfs/issues/5195
2024-01-12 10:57:18 -08:00
chrislu
3aa3991f0f
adjust client side logs
2024-01-11 23:08:53 -08:00
chrislu
7afaad31a4
wait on local topic partition creation
2024-01-11 23:08:02 -08:00
chrislu
61dbdd0ff6
log ctrl messages
2024-01-11 23:07:32 -08:00
chrislu
f750a5e03b
passing timestamp
2024-01-11 23:03:55 -08:00
chrislu
45994641e9
lint
2024-01-11 23:03:35 -08:00
chrislu
70a011a224
Merge branch 'master' into mq-subscribe
2024-01-11 20:58:57 -08:00
skycope
0e8a54f6f6
fix write volume over size MaxPossibleVolumeSize ( #5190 )
...
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-01-11 20:23:46 -08:00
chrislu
050efe6c08
factor in existing ec volume count when estimating max volume count
...
related to https://github.com/seaweedfs/seaweedfs/issues/5191
2024-01-11 08:19:09 -08:00
chrislu
188f471a24
logging
2024-01-11 07:55:26 -08:00
chrislu
6a7a679137
passing partition generation timestamp
2024-01-10 22:36:17 -08:00
Konstantin Lebedev
a7fc723ae0
chore: add status code for request_total metrics ( #5188 )
2024-01-10 10:05:27 -08:00
chrislu
8af61dec91
avoid empty spaces in file name
2024-01-09 09:00:54 -08:00
chrislu
aed54eda61
refactor
2024-01-09 08:01:01 -08:00
chrislu
d51efddf5c
flush to disk
...
Need to do: read from disk
2024-01-08 23:27:02 -08:00
jerebear12
8c966ac23b
Removed problematic if statement ( #5180 )
...
This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.
2024-01-08 11:35:20 -08:00
chrislu
fbb4917e2f
Merge branch 'master' into mq-subscribe
2024-01-08 00:08:49 -08:00
chrislu
8ae00e47a6
3.61
2024-01-08 00:04:49 -08:00
chrislu
49428a303b
add batch index for each memory buffer
2024-01-08 00:03:08 -08:00
Konstantin Lebedev
b9d32d32e1
chore: filer sync add doDeleteFiles option for create only mode ( #5166 )
2024-01-06 10:02:16 -08:00
chrislu
d0d24f1e40
rename functions
2024-01-05 17:10:43 -08:00
chrislu
d9bd343055
Merge branch 'master' into mq-subscribe
2024-01-05 15:36:00 -08:00
chrislu
1206313952
add logs on flush
2024-01-05 15:35:42 -08:00
chrislu
aa5b6a8187
adjust logs
2024-01-05 15:35:32 -08:00
chrislu
496fc8fbbf
refactor
2024-01-05 15:35:19 -08:00
chrislu
e8611ed85d
subscribe with partition offset
2024-01-05 15:24:14 -08:00
chrislu
531f854af2
rename functions
2024-01-05 15:16:53 -08:00
chrislu
f8787a9761
rename functions
2024-01-05 15:16:41 -08:00
chrislu
add16fb1a1
logs
2024-01-05 15:16:22 -08:00
chrislu
ddd0fde094
rename functions
2024-01-05 15:16:01 -08:00
chrislu
ee41dbb7fc
rename functions
2024-01-05 15:14:25 -08:00
chrislu
1fed37bed0
add offset
2024-01-05 15:03:16 -08:00
Konstantin Lebedev
3b62b338a5
chore: add maxMB option for webdav ( #5165 )
2024-01-05 11:10:20 -08:00
chrislu
47a4963d7a
subscription start from specified timestamp
2024-01-03 15:57:36 -08:00
chrislu
35869b5c80
subscriber can be notified of the assignment change when topic is just configured
...
Next: Subscriber needs to read by the timestamp offset.
2024-01-03 13:30:30 -08:00
Konstantin Lebedev
4f6172f369
Fix missing VersionConfiguration node in get-bucket-versioning response ( #5162 )
...
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
* fix: s3 response for get bucket version
https://github.com/seaweedfs/seaweedfs/issues/5155
2024-01-03 12:27:13 -08:00
Konstantin Lebedev
3e43778869
fix: webdav avoid create empty files ( #5160 )
2024-01-03 09:52:51 -08:00
Konstantin Lebedev
b832ddd1ef
fix: return etag with md5 in webdav responses ( #5158 )
2024-01-03 08:50:32 -08:00
chrislu
efb695fd93
lookup existing topic partitions
2024-01-02 17:29:35 -08:00
chrislu
54e9877093
Merge branch 'master' into mq-subscribe
2024-01-02 13:25:00 -08:00
chrislu
d3688938d9
avoid nil
...
fix https://github.com/seaweedfs/seaweedfs/issues/5153
introduced from #5109
2024-01-02 12:55:34 -08:00
chrislu
923b1e5dad
Merge branch 'master' into mq-subscribe
2024-01-01 20:56:12 -08:00
chrislu
d4e91b6ad9
3.60
2024-01-01 19:11:23 -08:00
chrislu
8b4cd50aac
log errors
2024-01-01 19:09:17 -08:00
chrislu
bf9ae74384
add logs
2024-01-01 19:09:05 -08:00
chrislu
32bc8d6a38
adjust wait time
2023-12-31 17:42:44 -08:00
chrislu
458ddbf919
clean up dead code
2023-12-31 13:25:33 -08:00
chrislu
2845230329
passing broker into the assignments
2023-12-28 21:13:49 -08:00
chrislu
093fdc1621
subscriber can get assignments
2023-12-28 20:35:15 -08:00
Varun Upadhyay
77626666c5
Minor cleanup & gitignore update ( #5144 )
2023-12-28 20:25:43 -08:00
chrislu
9ed26cd7b0
Merge branch 'master' into mq-subscribe
2023-12-28 11:57:30 -08:00
chrislu
c950a40aad
subscriber keep connected to the balancer
2023-12-28 11:56:37 -08:00
Konstantin Lebedev
183352c796
shell meta load add concurrency ( #4529 )
...
* fix: increase speed cmd fs meta load
* fix: add wg
2023-12-28 11:54:37 -08:00
Lars Lehtonen
edc0532b1d
weed/server: remove unneeded err from streamReadOneVolume() signature ( #5143 )
2023-12-28 11:45:13 -08:00
chrislu
bebbc9fe44
create volume grow request if the selected volume is close to full
2023-12-27 11:45:44 -08:00
chrislu
bb9269f3f7
Revert "Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. ( #5134 )"
...
This reverts commit 838578b55f
2023-12-24 12:31:08 -08:00
chrislu
c6b1dc7058
remove unused code
2023-12-24 11:11:41 -08:00
vivekkoya
838578b55f
Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. ( #5134 )
2023-12-23 15:01:57 -08:00
Seyed Mahdi Sadegh Shobeiri
97236389e8
Add modifyTimeAgo to volume.fsck ( #5133 )
...
* Add modifyTimeAgo to volume.fsck
* Fix AppendAtNs
2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri
54ba2c8868
Fix cutoffTimeAgo in findMissingChunksInFiler ( #5132 )
2023-12-23 09:18:16 -08:00
chrislu
03c4b2e988
correct lock accounting
...
possibly fix https://github.com/seaweedfs/seaweedfs/issues/5082
2023-12-22 14:16:23 -08:00
chrislu
c4badf7396
adjust options
2023-12-22 11:41:10 -08:00
chrislu
6324062acc
fix
2023-12-22 11:37:15 -08:00
chrislu
e0727071c8
go fmt
2023-12-22 11:33:50 -08:00
chrislu
ee1c9bc314
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-12-22 11:33:02 -08:00
chrislu
034db049a0
comment fix
2023-12-22 11:33:00 -08:00
chrislu
6df44d3c95
option to filer.copy with public volume server url
...
fix https://github.com/seaweedfs/seaweedfs/issues/5126
2023-12-22 11:32:43 -08:00
chrislu
7a0d1e7fc1
minor
2023-12-22 11:29:54 -08:00
Farbod
c278f49bca
Using filer.remote.sync concurrency in filer.remote.gateway ( #5123 )
...
- Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync
Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>
2023-12-20 16:25:54 -08:00
jerebear12
06343f8976
Set allowed origins in config ( #5109 )
...
* Add a way to use a JWT in an HTTP only cookie
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
* Added a way to specify allowed origins header from config
* Removed unecessary log
* Check list of domains from config or command flag
* Handle default wildcard and change name of config value to cors
2023-12-20 16:21:11 -08:00
Konstantin Lebedev
3c9bcfb864
chore: add dsn for connection to mysql ( #5060 )
...
* chore: add dsn for connection to mysql
* add comment
* new comment
* fix: validate dsn and adapt password
2023-12-20 16:20:58 -08:00
chrislu
915416a277
fix wrong parameter
...
fix https://github.com/seaweedfs/seaweedfs/pull/5102
2023-12-13 12:02:57 -08:00
Farbod
d5c01e5ceb
Remote gateway Bug fix ( #5100 )
2023-12-12 07:46:45 -08:00
chrislu
61b05fad95
fix test
2023-12-11 20:57:29 -08:00
Chris Lu
580940bf82
Merge accumulated changes related to message queue ( #5098 )
...
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* balance partitions on brokers
* prepare topic partition first and then publish, move partition
* purge unused APIs
* clean up
* adjust logs
* add BalanceTopics() grpc API
* configure topic
* configure topic command
* refactor
* repair missing partitions
* sequence of operations to ensure ordering
* proto to close publishers and consumers
* rename file
* topic partition versioned by unixTimeNs
* create local topic partition
* close publishers
* randomize the client name
* wait until no publishers
* logs
* close stop publisher channel
* send last ack
* comments
* comment
* comments
* support list of brokers
* add cli options
* Update .gitignore
* logs
* return io.eof directly
* refactor
* optionally create topic
* refactoring
* detect consumer disconnection
* sub client wait for more messages
* subscribe by time stamp
* rename
* rename to sub_balancer
* rename
* adjust comments
* rename
* fix compilation
* rename
* rename
* SubscriberToSubCoordinator
* sticky rebalance
* go fmt
* add tests
* tracking topic=>broker
* merge
* comment
2023-12-11 12:05:54 -08:00
Konstantin Lebedev
125ad8fe63
falls back to update only if error contains msg "duplicate entry"
...
https://github.com/seaweedfs/seaweedfs/issues/5062
2023-12-09 10:18:18 -08:00
chrislu
c1a6e624e5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-12-07 08:13:46 -08:00
chrislu
5fe4c04a1a
add missing error in the log
...
related to https://github.com/seaweedfs/seaweedfs/issues/5084
2023-12-07 08:13:42 -08:00
Lars Lehtonen
43a5973a66
weed/storage: fix dropped test errors
2023-12-06 12:05:22 -08:00
jerebear12
d5d9fbb8aa
Add a way to use a JWT in an HTTP only cookie
...
If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.
2023-12-05 08:57:01 -08:00
chrislu
756bcc032d
adjust logs
2023-11-27 12:57:29 -08:00
Konstantin Lebedev
dd78397fea
logging PickForWrite error
...
https://github.com/seaweedfs/seaweedfs/issues/3886
2023-11-27 12:56:15 -08:00
chrislu
81f11883e3
go fmt
2023-11-26 11:47:20 -08:00
chrislu
7a50762b9e
reduce binary size by skipping rclone
2023-11-23 00:48:27 -08:00
zehweh
d818ca9631
fix TestMisplacedChecking() and add test-case
2023-11-21 08:24:34 -08:00
julor
2fa37e0287
Update filer_server_handlers_write_autochunk.go
2023-11-20 11:17:14 -08:00
julor
9e03b916d8
verify content-md5 in header
...
if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks
2023-11-20 11:17:14 -08:00
SmsS4
6c1ce18541
Fix weed fix
2023-11-20 11:14:19 -08:00
Lars Lehtonen
57801687de
weed/command: fix dropped error
2023-11-20 08:05:24 -08:00
Viktor Kuzmin
06e723f629
Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952
2023-11-17 07:38:14 -08:00
Konstantin Lebedev
2b229e98ce
fix: add doDeleteFile option for filer backup
2023-11-17 07:37:28 -08:00
chrislu
17710e1ecb
adjust logs
2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr
6f559fe838
fs.mergeVolumes: change dir to a named parameter
2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr
d3556479be
Fix typo (dist => dest)
2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr
bb274ea8a3
fs.mergeVolumes: Make a plan based on volumes size
2023-11-13 06:41:24 -08:00
Konstantin Lebedev
e55fa68848
fix: excludeFileName is empty
2023-11-13 06:23:46 -08:00
Konstantin Lebedev
3c5295a1a6
filer backup add option for exclude file names that match the regexp to sync on filer
2023-11-13 06:23:46 -08:00
Konstantin Lebedev
5e41ab1370
s3 api add not implemented response for PutBucketVersioning
2023-11-13 06:22:23 -08:00
Konstantin Lebedev
7f6152f130
s3 api add default response for GetBucketVersioning
2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr
b05d682c48
make help command in shell case insensitive
2023-11-13 06:06:45 -08:00
chrislu
27b34f3793
3.59
2023-11-12 22:12:08 -08:00
jsh
47112917ff
ec.decode: mount the collected ec shards
2023-11-10 00:04:42 -08:00
chrislu
85bed4d92e
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-11-09 08:02:57 -08:00
chrislu
c563386675
rename parameters
2023-11-09 08:02:39 -08:00
Lars Lehtonen
5db82e594a
weed/server: fix dropped webdav error
2023-11-07 19:05:21 -08:00
zehweh
2b9dda7d2e
fix isMisplaced() in command_volume_fix_replication.go ( #4988 )
2023-11-07 07:58:19 -08:00
Konstantin Lebedev
d693beea87
Replace an invalid method in a metric with an invalid one ( #4987 )
2023-11-06 10:51:47 -08:00
chrislu
a667f0ad99
fix error reporting
...
related to https://github.com/seaweedfs/seaweedfs/issues/4978
2023-11-05 07:17:16 -08:00
chrislu
b40ba7c7fe
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-11-01 16:28:05 -07:00
chrislu
310eec9a9c
skip if path changed
...
mitigate https://github.com/seaweedfs/seaweedfs/issues/4968
2023-11-01 16:27:59 -07:00
Nico D'Cotta
3834ad5c48
refactor webdav subdirectory, fixes #4967 ( #4969 )
...
* refactor webdav subdirectory, fixes #4967
* fix bug where Name() was not called on delegate wrappedFileInfo
2023-11-01 13:12:49 -07:00
tired-engineer
4ed06e9ba5
Pass correct Filer's gRPC port to S3 server. ( #4972 )
...
Fixes seaweedfs/seaweedfs#4971
2023-11-01 10:11:41 -07:00
SmsS4
ba11331cc1
Fix twice increase metric counter ( #4961 )
2023-10-30 08:05:27 -07:00
SmsS4
f95848ba7d
Add ErrorGetNotFound and ErrorGetInternal to volume server metrics ( #4960 )
2023-10-30 07:38:03 -07:00
chrislu
d1e83a3b4d
3.58
2023-10-29 23:04:20 -07:00
duanhongyi
bf839651ec
Fix: issues #4950 ( #4952 )
2023-10-25 22:09:39 -07:00
zehweh
8b39bbbe2f
fix copying .vif file in VolumeCopy ( #4943 )
...
closes #4934
fixes #2633
might fix #3528
2023-10-24 09:54:24 -07:00
chrislu
2bc05f70e7
log full percentage
2023-10-22 12:59:34 -07:00
chrislu
612fd7206e
fix
2023-10-19 08:10:33 -07:00
Konstantin Lebedev
6c82d7ddfc
[s3] do reload s3 static config ( #4923 )
...
* do reload s3 config
* print error on reload s3 config
* print success msg
* Update weed/s3api/s3api_server.go
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-10-18 23:26:49 -07:00
lakeland1990
3df0e11ad3
remove compare to fix #4668 issue ( #4924 )
2023-10-18 09:06:56 -07:00
chrislu
cbc24c7b24
disconnect from old subscribers
2023-10-13 16:48:14 -07:00
James Hedley
6acb43bbbb
Add optional flags to enable mTLS with verification of client certificate ( #4910 )
2023-10-13 09:02:24 -07:00
chrislu
b62066d38f
adding DiskType to /dir/status output
2023-10-12 21:39:11 -07:00
chrislu
deb86ff4a6
upgrading grpc
2023-10-12 21:38:34 -07:00
chrislu
de0b969b36
Revert "rename"
...
This reverts commit 35b5264ab7
2023-10-12 20:28:11 -07:00
Konstantin Lebedev
1cac5d983d
fix: disallow file name too long when writing a file ( #4881 )
...
* fix: disallow file name too long when writing a file
* bool LongerName to MaxFilenameLength
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 14:29:55 -07:00
Konstantin Lebedev
edee91ef0e
fix has changes false if deleted result size eq zero ( #4909 )
...
* fix has changes false if deleted result size eq zero
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix change respos status 202 to 304 if the file has already been deleted
https://github.com/seaweedfs/seaweedfs/issues/4171
* fix delete if set apply changes
https://github.com/seaweedfs/seaweedfs/issues/4171
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-12 08:15:20 -07:00
chrislu
35b5264ab7
rename
2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr
3e8604a4c5
add fs.mergeVolumes command into shell ( #4907 )
...
* add fs.compact command into shell
* rename fs.compact to fs.mergeVolumes
2023-10-10 12:47:18 -07:00
Konstantin Lebedev
2b3e39397e
fix: skipping checking active volumes with the same number of files at the moment ( #4893 )
...
* fix: skipping checking active volumes with the same number of files at the moment
https://github.com/seaweedfs/seaweedfs/issues/4140
* refactor with comments
https://github.com/seaweedfs/seaweedfs/issues/4140
* add TestShouldSkipVolume
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-10-09 09:57:26 -07:00
Yuval Yacoby
3fe00996b2
added healthz endpoint to filer ( #4899 )
2023-10-08 07:03:18 -07:00
Andrew Garrett
082f67bfe9
Fix filer.backup local sink to propagate file mode changes ( #4896 )
2023-10-06 05:40:20 -07:00
chrislu
0fd7222d65
default to skip if less than 4 nodes
2023-10-05 11:13:48 -07:00
chrislu
7b3c63a8af
handle wrapped errors
2023-10-05 08:15:11 -07:00
chrislu
4e493d596c
minor
2023-10-05 08:13:53 -07:00
Lars Lehtonen
530bdedf76
weed/operation: fix dropped error ( #4888 )
2023-10-03 10:55:20 -07:00
chrislu
6ebe26a765
Revert "Revert "Revert "Add disk type to prometheus metrics" ( #4777 )""
...
This reverts commit 567d788928
2023-10-03 08:28:52 -07:00
chrislu
7540d43ee9
Revert "Revert "fix compilation""
...
This reverts commit f9abfd0b03
2023-10-03 08:28:02 -07:00
chrislu
249c0e06ef
Revert "fix compilation"
...
This reverts commit 451ec6504d
2023-10-03 08:27:50 -07:00
chrislu
451ec6504d
fix compilation
2023-10-03 08:15:18 -07:00
chrislu
f9abfd0b03
Revert "fix compilation"
...
This reverts commit 0483ba3889
2023-10-03 07:58:11 -07:00
chrislu
0483ba3889
fix compilation
2023-10-02 11:55:50 -07:00
chrislu
567d788928
Revert "Revert "Add disk type to prometheus metrics" ( #4777 )"
...
This reverts commit 9215ba24be
2023-10-02 11:49:54 -07:00
chrislu
b5070d6280
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-10-02 10:52:40 -07:00
Konstantin Lebedev
dffe00a822
fix: logger place msg ( #4880 )
2023-10-02 08:29:09 -07:00
chrislu
bf13f3ced7
add subscriber coordinator
2023-10-02 01:02:27 -07:00
chrislu
734178093e
refactor TopicPartition struct
2023-10-02 01:01:45 -07:00
chrislu
2a578b9033
refactor
2023-10-01 11:59:19 -07:00
chrislu
321fda552c
comment
2023-10-01 11:36:27 -07:00
chrislu
b02fdeabff
RetryForever => RetryUntil
2023-10-01 11:33:56 -07:00
chrislu
ebd6f96d35
adjust mq.proto
2023-09-30 21:47:57 -07:00
chrislu
02cb799481
add a test framework for grpc
2023-09-30 21:36:58 -07:00
chrislu
c4e6545197
fix compilation
2023-09-30 21:26:09 -07:00
chrislu
358cba43ef
update proto generated files
2023-09-30 13:19:25 -07:00
chrislu
597ab1e4bf
add design for mq balancer
2023-09-30 13:19:05 -07:00
chrislu
d7c211fe8c
adjust mq proto
2023-09-30 13:18:49 -07:00
chrislu
40888b5612
align 64bit atomic operation
...
fix https://github.com/seaweedfs/seaweedfs/issues/4878
2023-09-28 07:58:43 -07:00
Konstantin Lebedev
dd580190b4
fix: avoid deleting one replica without sync ( #4875 )
...
* fix: avoid deleting one replica without sync
https://github.com/seaweedfs/seaweedfs/issues/4647
* Update weed/shell/command_volume_fix_replication.go
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
* fix: revert this existing do option to positive
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-27 23:12:10 -07:00
Konstantin Lebedev
44906f1f3b
fix: avoid error file name too long when writing a file ( #4876 )
2023-09-27 05:40:51 -07:00
chrislu
9d589b48e6
rename function
2023-09-26 15:17:33 -07:00
Konstantin Lebedev
df4ded758e
fix: avoid deleting more than one replica ( #4873 )
...
https://github.com/seaweedfs/seaweedfs/issues/4647
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-26 00:20:48 -07:00
chrislu
19505c1cf4
describe a topic
2023-09-25 20:46:00 -07:00
Tobias Gurtzick
78dbac7702
fix(ec): volumes created by foreign collection due to bug in ec balance ( #4864 )
...
* fix(ec): ignore 0 byte data files
refers to parts of #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
* fix(ec): ignore volumes not from the current collection during balance
fixes #4861
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
---------
Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>
2023-09-25 10:35:43 -07:00
chrislu
645ae8c57b
Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs ""
...
This reverts commit 8cb42c39
2023-09-25 09:35:16 -07:00
chrislu
3d07895518
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 09:16:35 -07:00
chrislu
a6e35e4f68
fix compilation
2023-09-25 09:16:33 -07:00
Konstantin Lebedev
f8b94cac0e
[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration ( #4859 )
...
* move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface
https://github.com/seaweedfs/seaweedfs/issues/4519
* fix: test bucket acl default and
adjust the variable names
* fix: s3 api config test
---------
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2023-09-25 08:34:12 -07:00
chrislu
c9177c92e5
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-25 08:20:16 -07:00
chrislu
81e57bd45e
fix test
2023-09-25 08:20:03 -07:00
Lars Lehtonen
28a3a31b27
weed/replication/sub: fix dropped error ( #4865 )
2023-09-25 07:33:18 -07:00
chrislu
23f334dd13
comment
2023-09-24 23:36:43 -07:00
chrislu
c7e05e4e71
ensure latest stats are reported
2023-09-24 23:34:31 -07:00
chrislu
dff2ce5d2f
read existing broker stats
2023-09-24 23:10:33 -07:00
chrislu
1492bf7552
fix listing topics
2023-09-24 23:05:41 -07:00
chrislu
3cf9b8d621
create topic and report topic
2023-09-24 22:00:43 -07:00
chrislu
b3f94feede
do create topic
2023-09-24 22:00:43 -07:00
chrislu
d74348048a
implement create topic
2023-09-24 22:00:43 -07:00
chrislu
0361c321b4
add CreateTopic API
2023-09-24 22:00:43 -07:00
chrislu
0f8168c0c9
3.57
2023-09-24 21:29:31 -07:00
chrislu
1df4223195
minor
2023-09-24 09:54:57 -07:00
chrislu
10ac25f857
avoid deleting the entry too early
2023-09-23 12:43:00 -07:00
chrislu
c43238b30a
fix waiting condition
2023-09-23 12:28:02 -07:00
chrislu
322af1fc04
fix compilation
2023-09-23 12:13:32 -07:00
chrislu
0414b68a4f
if shared lock, still need to wait for exclusive lock to release
2023-09-23 12:08:23 -07:00
chrislu
9ba4639a27
adjust for better logs
2023-09-23 11:53:44 -07:00
chrislu
186b1f029b
read source uses shared lock
2023-09-23 10:33:44 -07:00
chrislu
2df70ce95f
adjust logs
2023-09-21 23:51:32 -07:00
chrislu
3e23421608
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2023-09-21 20:08:30 -07:00
chrislu
49a470bcdf
reduce logs
2023-09-21 20:08:20 -07:00
Konstantin Lebedev
750a0ba1b2
[iam] Replace action read/write to readAcp/writeAcp for handlers with acl ( #4858 )
...
Replace action read/write to readAcp/writeAcp for handlers with acl query
https://github.com/seaweedfs/seaweedfs/issues/4519
Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>
2023-09-21 14:07:04 -07:00
chrislu
8b2c39f2c0
reduce binary size
2023-09-21 13:22:08 -07:00
