Commit Graph

36 Commits

Author SHA1 Message Date
Chris Lu
dc784bf217
merge current message queue code changes (#6201)
* listing files to convert to parquet

* write parquet files

* save logs into parquet files

* pass by value

* compact logs into parquet format

* can skip existing files

* refactor

* refactor

* fix compilation

* when no partition found

* refactor

* add untested parquet file read

* rename package

* refactor

* rename files

* remove unused

* add merged log read func

* parquet wants to know the file size

* rewind by time

* pass in stop ts

* add stop ts

* adjust log

* minor

* adjust log

* skip .parquet files when reading message logs

* skip non message files

* Update subscriber_record.go

* send messages

* skip message data with only ts

* skip non log files

* update parquet-go package

* ensure a valid record type

* add new field to a record type

* Update read_parquet_to_log.go

* fix parquet file name generation

* separating reading parquet and logs

* add key field

* add skipped logs

* use in memory cache

* refactor

* refactor

* refactor

* refactor, and change compact log

* refactor

* rename

* refactor

* fix format

* prefix v to version directory
2024-11-04 12:08:25 -08:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628)
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398)
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
2022-08-23 23:18:21 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
d2ddf1dbdb typo 2021-08-03 09:32:31 -07:00
Chris Lu
b624090398 go fmt 2021-07-01 01:21:14 -07:00
Daniel Nagy
0e3adde47f
Return correct exitcode when wheed upload fails 2021-06-07 23:38:14 +02:00
Chris Lu
cc0df36a9e smaller file chunks are faster 2021-04-01 02:21:59 -07:00
Chris Lu
62191b08ea disk type support custom tags 2021-02-22 02:03:12 -08:00
Chris Lu
3575d41009 go fmt 2021-02-17 20:57:08 -08:00
Chris Lu
6daa932f5c refactoring to get master function, instead of passing master values directly
this will enable retrying later
2021-02-17 20:55:55 -08:00
Chris Lu
0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
Chris Lu
ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
4250a3a675 upload: add read default collection from master
fix https://github.com/chrislusf/seaweedfs/issues/1724#issuecomment-753576071
2021-01-02 23:43:01 -08:00
Chris Lu
1bf22c0b5b go fmt 2020-12-16 09:14:05 -08:00
Chris Lu
8baba93fce rename parameter name to "disk" 2020-12-13 12:06:21 -08:00
Chris Lu
0d2ec832e2 rename from volumeType to diskType 2020-12-13 11:59:32 -08:00
Chris Lu
e9cd798bd3 adding volume type 2020-12-13 00:58:58 -08:00
Chris Lu
f43146b237 resolve directories if containing home directory 2020-07-16 22:50:14 -07:00
Chris Lu
6566c8e114 weed upload: add usePublicUrl option
fix https://github.com/chrislusf/seaweedfs/issues/1236
2020-03-18 10:50:53 -07:00
Chris Lu
d335f04de6 support env variables to overwrite toml file 2020-01-29 09:09:55 -08:00
Chris Lu
ede876cfdb periodic scripts exeuction from leader master 2019-06-05 01:30:24 -07:00
Chris Lu
4f714cef22 default weed filer.copy and weed upload chunk size to 32MB 2019-04-04 23:14:32 -07:00
Chris Lu
77b9af531d adding grpc mutual tls 2019-02-18 12:11:52 -08:00
Chris Lu
4ff4a147b2 cleanup security.Secret 2019-02-09 21:56:32 -08:00
Sergey
aa5ccff6d2
fixing of typos 2019-02-06 18:59:15 +05:00
Chris Lu
49f40cd145 adjust help message 2018-12-22 15:05:15 -08:00
Chris Lu
dbee816205 fix typo
fix https://github.com/chrislusf/seaweedfs/issues/549
2017-08-10 10:26:39 -07:00
sparklxb
66e7013dfe suport uploading files to specific dataCenter 2017-01-09 23:34:47 +08:00
lixianbin
d96d0a87cf fix bug: upload big .gz file more than maxMB 2017-01-04 11:23:40 +08:00
Chris Lu
cdae9fc680 add "weed copy" command to copy files to filer 2016-07-20 23:45:55 -07:00
Chris Lu
5ce6bbf076 directory structure change to work with glide
glide has its own requirements. My previous workaround caused me some
code checkin errors. Need to fix this.
2016-06-02 18:09:14 -07:00