Commit Graph

36 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
chrislu
a4b25a642d math/rand => math/rand/v2 2024-08-29 09:52:21 -07:00
chrislu
bebc385607 randomly pick one replica to write 2024-08-27 09:02:39 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766)
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628)
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
4b5adc60b4 upload result changes to the right output variable 2021-01-03 01:44:22 -08:00
Chris Lu
3433accb1b weed upload: add TTL 2020-12-30 15:46:07 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
35f6518c36 weed upload: usePublicUrl was not being used on big files
fix https://github.com/chrislusf/seaweedfs/issues/1492
2020-09-25 02:41:38 -07:00
Chris Lu
39e72fb23c go fmt 2020-05-10 03:50:30 -07:00
zhangsong
d819865fe7 add fsync option for benchmark 2020-04-30 23:31:08 +08:00
Chris Lu
eedd33dda3 refactoring 2020-03-28 13:41:58 -07:00
Chris Lu
d848d08944 use public url also for deletions 2020-03-18 11:16:45 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option
fix https://github.com/chrislusf/seaweedfs/issues/1236
2020-03-18 10:50:53 -07:00
Chris Lu
2e3f6ad3a9 filer: remember content is gzipped or not 2020-03-08 21:39:33 -07:00
Chris Lu
13e215ee5c filer: option to encrypt data on volume server 2020-03-06 00:49:47 -08:00
Chris Lu
392678f8f3 upload skipping mimetype if not needed 2019-12-15 20:57:08 -08:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
7103c1ab7e go fmt 2019-02-15 00:09:48 -08:00
Chris Lu
74fb237727 benchmark can work in secure mode 2019-02-15 00:09:19 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
bingoohuang
ab6be025d7 go fmt and fix some typo 2019-01-17 09:17:19 +08:00
Chris Lu
852ee21835 avoid .gz auto decompression 2018-12-22 13:58:16 -08:00
Chris Lu
fc3924ac70 set file name to base file name
fixing https://github.com/chrislusf/seaweedfs/issues/632
2018-02-26 08:01:23 -08:00
sparklxb
66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
sparklxb
86a7c56275 support additional header name-value pairs 2017-01-08 09:16:40 +08:00
lixianbin
d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
霍晓栋
1f63094542 refactor Volume Assign function 2016-06-26 10:50:18 +08:00
Chris Lu
5ce6bbf076 directory structure change to work with glide
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00