Commit Graph

6838 Commits

Author SHA1 Message Date
shenxingwuying
ee25ada732
reduce ambiguity about use memory_sequencer (#5555) 2024-04-29 21:51:00 -07:00
dependabot[bot]
36a1cf0361
chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 (#5552)
* chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1

Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie) from 0.3.1 to 1.0.1.
- [Release notes](https://github.com/viant/ptrie/releases)
- [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/viant/ptrie
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix compilation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-04-29 18:42:31 -07:00
chrislu
5e8d788311 Merge branch 'master' into mq-subscribe 2024-04-29 06:44:03 -07:00
chrislu
e6bfd061c7 AllowEmptyFolder checks during object deletion 2024-04-29 06:25:06 -07:00
chrislu
d521466a37 split file 2024-04-29 06:23:42 -07:00
chrislu
a1b59948cc rename files 2024-04-29 05:33:56 -07:00
chrislu
b570e015ce Update write_parquet_test.go 2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01 record value API 2024-04-28 13:42:45 -07:00
chrislu
b166597c83 fix test 2024-04-28 13:03:01 -07:00
chrislu
189a7fc90e record type begin and end 2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9 improve api 2024-04-28 12:42:58 -07:00
chrislu
1a2332c257 rename api 2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546 Delete schema_builder_test.go 2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54 struct to schema correctly 2024-04-28 12:23:54 -07:00
chrislu
f37862bffe compile test 2024-04-28 12:23:33 -07:00
chrislu
7310d022ad recursive struct to schema 2024-04-28 11:14:13 -07:00
Konstantin Lebedev
6f9267ebdf
fix: skip file size checking for volume with TTL (#5541)
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
2024-04-28 00:21:57 -07:00
chrislu
a2a872ca03 rename 2024-04-27 23:44:11 -07:00
chrislu
60103171b6 rename 2024-04-27 23:30:15 -07:00
chrislu
111a4e1126 add field index 2024-04-27 23:30:06 -07:00
Konstantin Lebedev
3d3ee04eb9
[s3] Put bucket lifecycle configuration (#5510) 2024-04-27 07:39:22 -07:00
Kevin Bulteel
a3dab1fcd0
feat: upload files directly without multipart/form-data (#5539) 2024-04-26 16:03:40 -07:00
Nikita Borzykh
0d04264494
Move file deletion logging to v2 verbisity (#5538) 2024-04-26 06:41:06 -07:00
chrislu
72b50980f4 a little bit more efficient 2024-04-25 23:59:30 -07:00
chrislu
8948fb51e3 Merge branch 'master' into mq-subscribe 2024-04-25 23:42:19 -07:00
chrislu
63036133ff minor 2024-04-25 22:02:03 -07:00
chrislu
2a9b0a9332 use parquet levels to construct row 2024-04-25 21:29:45 -07:00
M@
abf01a0eb7
Fixes unlocked read from logBuffer.LastTsNs that is racey. (#5536) 2024-04-25 15:46:12 -07:00
chrislu
977e7988e6 toRow conversion with levels info 2024-04-25 09:14:37 -07:00
chrislu
9cb9d27b5b file rename 2024-04-25 08:56:31 -07:00
chrislu
313097aca0 printout test values 2024-04-25 08:54:57 -07:00
chrislu
827e293010 add visitor 2024-04-25 08:54:41 -07:00
chrislu
138a027a2e add field types but without value 2024-04-24 23:33:11 -07:00
chrislu
30b5aef62e record type is also optional 2024-04-24 23:32:35 -07:00
chrislu
ce4597c623 delete the generated file 2024-04-24 23:29:55 -07:00
chrislu
f91484ca07 add some notes 2024-04-24 23:28:20 -07:00
chrislu
f93c4cf48c adjust test 2024-04-24 23:24:53 -07:00
M@
855607c536
Fixes race with unlocked setter. (#5532) 2024-04-24 23:18:28 -07:00
M@
607927da60
Migrates from disintegration/imaging c2019 to cognusion/imaging c2024. (#5533) 2024-04-24 23:16:04 -07:00
chrislu
d88c1872ac works with single level of list 2024-04-24 23:04:47 -07:00
Konstantin Lebedev
2f3fee9bb9
filer read chunk retry if status code 499 (#5528) 2024-04-24 06:51:05 -07:00
chrislu
d7e5f6b2a5 fix ordering 2024-04-24 01:32:02 -07:00
skycope
6e4b9181f5
fix "volume.fix.replication" move many replications only to one volumeServer (#5522) 2024-04-23 06:33:50 -07:00
chrislu
9ef238b610 has a bug 2024-04-22 00:42:18 -07:00
chrislu
cc1ce3de87 all record fields are optional by default 2024-04-18 23:17:52 -07:00
chrislu
384ed7a381 add test 2024-04-18 22:41:20 -07:00
chrislu
57949f9959 support list type 2024-04-18 22:41:12 -07:00
chrislu
05d18130b6 add schema builder 2024-04-18 22:27:18 -07:00
chrislu
20666bdf81 Merge branch 'master' into mq-subscribe 2024-04-18 21:33:06 -07:00
chrislu
55976ae04a avoid repeated calls to heavy-weighted viper 2024-04-18 09:09:45 -07:00
chrislu
31f1f96038 improve perf a bit 2024-04-18 08:47:55 -07:00
chrislu
d9490c5e1f rename 2024-04-18 08:47:45 -07:00
chrislu
0847a17484 convert to parquet schema and value 2024-04-17 23:49:21 -07:00
steve.wei
0bdf121e51
rename VolumeServerVolumeGauge (#5504) 2024-04-17 04:49:50 -07:00
chrislu
ce2b2fa9b2 Merge branch 'master' into mq-subscribe 2024-04-15 20:09:14 -07:00
chrislu
0edc5aad9a 3.65 2024-04-14 20:15:46 -07:00
Konstantin Lebedev
33537ae29f
[s3] fix s3 test_multipart_get_part (#5476)
* try fix s3  test_multipart_get_part

* add passed s3 tests

* fix SeaweedFSUploadId

* rm spaces

* convert part request to range

* add passed s3 tests of multipart
2024-04-14 10:41:32 -07:00
yudrywet
8833745fbc
chore: fix some typos in comments (#5497) 2024-04-14 07:42:40 -07:00
chrislu
68ade92d4f publisher send schema 2024-04-12 23:36:15 -07:00
chrislu
8415476c2d fix test 2024-04-12 23:35:54 -07:00
chrislu
5324e4288a rename parameter 2024-04-12 22:33:00 -07:00
chrislu
6f80dbdfe5 Create schema.go 2024-04-12 22:31:28 -07:00
chrislu
53d1d2b78a save schema when configuring topic 2024-04-12 22:29:53 -07:00
chrislu
04fb4c34e3 add proto for value 2024-04-12 22:27:16 -07:00
chrislu
48fe50df81 minor 2024-04-12 13:31:54 -07:00
chrislu
0cbf364a3d all fields are optional 2024-04-12 13:29:35 -07:00
chrislu
7a72677f98 add test 2024-04-12 01:31:23 -07:00
chrislu
d45a372615 add schema.proto 2024-04-12 01:30:29 -07:00
chrislu
d5b2cee34a log possible racing condition
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:54 -07:00
chrislu
2a4a9c6343 log possible racing condition
fix https://github.com/seaweedfs/seaweedfs/issues/5412
2024-04-11 23:44:18 -07:00
chrislu
e53a972483 Merge branch 'master' into mq-subscribe 2024-04-11 19:48:00 -07:00
Konstantin Lebedev
5189a09de0
[volume] Reduce the number of buffers for uploading one chunk (#5458) 2024-04-11 04:47:21 -07:00
skycope
6dae685f9c
fix append disorder (#5492) 2024-04-10 19:21:44 -07:00
chrislu
cc1c69f312 Merge branch 'master' into mq-subscribe 2024-04-08 11:03:14 -07:00
steve.wei
67ead9b18f
fix(volume.fix.replication): adjust volume count, not free volume count (#5479) 2024-04-08 07:30:04 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server (#5480) 2024-04-08 07:27:00 -07:00
clonefetch
9e07a87fcb
chore: fix function names in comment (#5478) 2024-04-08 07:19:02 -07:00
Konstantin Lebedev
3e25ed1b11
[s3] add s3 pass test_multipart_upload_size_too_small (#5475)
* add s3 pass test_multipart_upload_size_too_small

* refactor metric names

* return ErrNoSuchUpload if empty parts

* fix test
2024-04-07 11:52:35 -07:00
Konstantin Lebedev
35cba720a5
[s3] add s3 pass test_multipart_upload (#5474)
add s3 pass test_multipart_upload
2024-04-07 11:51:22 -07:00
Konstantin Lebedev
d42a04cceb
[s3] fix s3 test_multipart_resend_first_finishes_last (#5471)
* try fix s3 test
https://github.com/seaweedfs/seaweedfs/pull/5466

* add error handler metrics

* refactor

* refactor multipartExt

* delete bad entry parts
2024-04-06 10:56:39 -07:00
Nikita Borzykh
7aa25c1137
fix: correctly form partNumber from new entry.Name format in "listObjectParts" (#5470) 2024-04-04 09:16:16 -07:00
skycope
b19c9847c6
fix completed multiupload lost data (#5460)
If there are putObjectPart requests with the same uploadId during
completeMultiPart, it can result in data loss. putObjectPart requests
might be due to timeout retries.

Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-04-02 20:18:42 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
chrislu
f37c0d0d7a comment out println 2024-04-02 15:31:31 -07:00
Konstantin Lebedev
d5d8b8e2ae
fix panic at isAllWritable (#5457)
fix panic
https://github.com/seaweedfs/seaweedfs/issues/5456
2024-04-02 09:06:19 -07:00
Henco Appel
2a88da4de7
fix: always close volume file (#4530) (#5459)
If sync fails then close is never called. We should always be calling
close on the file.

Co-authored-by: Hendrik Appel <happel@europe.altair.com>
2024-04-02 08:31:40 -07:00
chrislu
d935f70e3c shutdown follower when leader stops 2024-04-01 22:32:30 -07:00
chrislu
f07875e8e1 send flush message to follower before shutting down logBuffer 2024-04-01 16:01:26 -07:00
Nikita Borzykh
df9031e19a
fix: ensure complete gRPC stream consumption in doListFilerEntries to prevent context canceled errors. (#5441) 2024-04-01 15:55:16 -07:00
chrislu
e568e742c9 renaming 2024-04-01 15:02:52 -07:00
chrislu
47975e75bf purge old code 2024-03-31 17:49:34 -07:00
chrislu
237e68059d shutdown follower if no clients, not only publishers
to avoid overhead when a publisher repeatedly send one message and close
2024-03-31 17:45:26 -07:00
chrislu
66a878af39 add control message to data message 2024-03-31 16:35:46 -07:00
chrislu
546ae87c39 send is_close message to broker 2024-03-31 01:28:40 -07:00
chrislu
c9df613b6b add publisher name for debugging 2024-03-31 00:19:16 -07:00
chrislu
ca4f89a6f6 simplify 2024-03-30 20:37:57 -07:00
chrislu
0d55bddc75 logs 2024-03-30 20:37:46 -07:00
Nikita Borzykh
829c0da381
fix: adjust condition in prefixFilterEntries prevent infinite loop (#5440) 2024-03-30 14:47:00 -07:00
Konstantin Lebedev
9c1e0f5811
[master] grow volumes if no writable volumes in current dataCenter (#5434)
* grow volumes if no writable volumes in current dataCenter
https://github.com/seaweedfs/seaweedfs/issues/3886

* fix tests with volume grow

* automatic volume grow one volume

* add ErrorChunkAssign metrics
2024-03-29 00:38:27 -07:00
Konstantin Lebedev
df40908e57
fix panic 5435 (#5436) 2024-03-28 16:17:59 -07:00