7221 Commits

Author	SHA1	Message	Date
chrislu	a829f36d17	adjust error handling ... fix https://github.com/seaweedfs/seaweedfs/issues/5677	2024-06-17 10:59:41 -07:00
chrislu	49893267e9	Merge branch 'master' into mq	2024-06-15 11:53:19 -07:00
chrislu	464611f614	optionally skip deleting file chunks	2024-06-15 11:39:48 -07:00
chrislu	64fd233a59	DeleteUncommittedChunks	2024-06-15 09:55:22 -07:00
chrislu	843e778875	refactor	2024-06-15 09:41:08 -07:00
chrislu	677cfb8ad1	refactor	2024-06-15 09:39:49 -07:00
chrislu	cb1ba0275a	sort imports	2024-06-14 11:44:42 -07:00
chrislu	32dbb6a2ce	Merge branch 'master' into mq	2024-06-14 11:42:05 -07:00
vadimartynov	8aae82dd71	Added context for the MasterClient's methods to avoid endless loops (#5628) ... * Added context for the MasterClient's methods to avoid endless loops * Returned WithClient function. Added WithClientCustomGetMaster function * Hid unused ctx arguments * Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions * Changed the context termination check in the tryConnectToMaster function * Added a child context to the tryConnectToMaster function * Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark	2024-06-14 11:40:34 -07:00
chrislu	70b31e3a28	Merge branch 'master' into mq	2024-06-14 09:17:58 -07:00
chrislu	2a169dde9a	minor	2024-06-14 09:17:46 -07:00
chrislu	8ffda89ed6	Merge branch 'master' into mq	2024-06-13 21:48:57 -07:00
chrislu	b03c831ad2	more logs	2024-06-13 21:48:44 -07:00
chrislu	458dda56ae	Merge branch 'master' into mq	2024-06-12 00:32:04 -07:00
Taehyung Lim	4744889973	fix issue: sometimes volume.fsck report 'volume not found' (#5537) ... * fix issue: sometimes volume.fsck report 'volume not found' when a volume server has multiple disk types * rename variable * adjust counters --------- Co-authored-by: chrislu <chris.lu@gmail.com>	2024-06-11 22:22:57 -07:00
MeenakshiSachdeva	d2516d99cc	Meenakshi fix filer sync (#5672) ... * Fix Filer Sync Issue: 5455 * Fix Filer Sync Issue: 5455	2024-06-10 11:16:29 -07:00
MeenakshiSachdeva	1caa012a76	Fix Filer Sync Issue: 5455 (#5663)	2024-06-10 09:19:38 -07:00
chrislu	0cd1c4be48	Merge branch 'master' into mq	2024-06-09 21:21:43 -07:00
chrislu	a9cd9b0542	3.68	2024-06-09 21:10:46 -07:00
chrislu	b5d36541d9	Merge branch 'master' into mq	2024-06-09 12:14:57 -07:00
chrislu	dbfbabac55	simplify	2024-06-09 12:14:10 -07:00
steve.wei	2150289442	fix: Ensure that the clientAddress is unique (#5655)	2024-06-07 09:13:03 -07:00
chrislu	25b2850e7d	refactor out FilerClientAccessor	2024-06-06 19:44:19 -07:00
chrislu	4093115ca9	Merge branch 'master' into mq	2024-06-06 19:27:07 -07:00
Gaspare Iengo	dc6b750424	Fix panic (#5654)	2024-06-06 18:59:50 -07:00
Gaspare Iengo	fce8fc1e16	Fix Issue #5649 (#5652)	2024-06-06 12:49:33 -07:00
steve.wei	d8da4bbaa7	Set the capacity of clientChan to 10000 (#5647)	2024-06-05 05:41:46 -07:00
chrislu	9f02bf4ede	Merge branch 'master' into mq	2024-06-04 21:12:30 -07:00
Lê Hoàng Nam	1f2dc54647	fix(s3api): make post api upload object default return 204 (#5648)	2024-06-04 06:18:47 -07:00
LHHDZ	36b5b713ba	fix deadlock caused by message chan blocked (#5639)	2024-06-03 07:42:40 -07:00
Dominic Pearson	d8bde9b96e	Solaris disk support (#5638)	2024-06-02 22:10:28 -07:00
NyaMisty	579ebbdf60	Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed (#5636)	2024-06-02 14:25:42 -07:00
NyaMisty	0c62d591e2	Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) (#5635)	2024-06-02 14:16:05 -07:00
NyaMisty	e1c671c028	Fix volume.tier.upload nil pointer panic (#5634)	2024-06-02 14:15:39 -07:00
NyaMisty	f2a37933b1	Fix sorted needle file's write mode for (*SortedFileNeedleMap).Delete() (#5633)	2024-06-02 14:12:25 -07:00
chrislu	41cc825ddb	Merge branch 'master' into mq	2024-05-30 12:03:22 -07:00
chrislu	d1f0c40402	remove per-message debug logs	2024-05-30 09:49:08 -07:00
chrislu	df9d889489	fix size calculation	2024-05-30 09:41:04 -07:00
chrislu	ff14dfa0ee	debug	2024-05-30 09:15:50 -07:00
chrislu	b485b9695c	fix passing sub follower offset	2024-05-30 09:15:23 -07:00
chrislu	f7b818483a	fix cases when buffer wraps around	2024-05-30 09:10:30 -07:00
chrislu	9a5cc065b0	fix	2024-05-30 08:29:06 -07:00
chrislu	23a72db1df	stop partitionOffsetChan if closed	2024-05-30 00:27:44 -07:00
chrislu	af19256dbf	skip control messages	2024-05-29 23:34:39 -07:00
chrislu	ad2b878130	fix inflight message tracker	2024-05-29 23:33:37 -07:00
chrislu	f8a22038c5	passing along publisher name	2024-05-29 23:31:48 -07:00
chrislu	f0b9e163a0	avoid nil stopChan	2024-05-29 16:18:02 -07:00
chrislu	a5874954dc	less logs	2024-05-29 16:17:37 -07:00
chrislu	3ddfd15ec1	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2024-05-29 10:41:38 -07:00
chrislu	3e7a92061b	pass along volume server grpc port ... fix https://github.com/seaweedfs/seaweedfs/issues/5617	2024-05-29 10:41:33 -07:00
chrislu	edc87c16f9	balance subscribers ... need to ensure the offsets are persisted	2024-05-27 17:30:16 -07:00
dependabot[bot]	722a14ef57	chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.2.0 to 0.4.2 (#5621) ... * chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.2.0 to 0.4.2. - [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md) - [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.2.0...v0.4.2) --- updated-dependencies: - dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix compilation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>	2024-05-27 12:36:00 -07:00
Vlad	72f7b4e9b1	Replaces volume.unmount in the help of the volumeServer.leave (#5619)	2024-05-27 01:13:06 -07:00
chrislu	6366898af2	add market	2024-05-26 14:35:22 -07:00
chrislu	96b326a304	use type ConsumerGroupInstanceId	2024-05-26 14:09:51 -07:00
chrislu	ba778539d9	Merge branch 'master' into mq	2024-05-24 17:15:27 -07:00
chrislu	364bb6c7b4	avoid ticker leak	2024-05-24 17:15:12 -07:00
chrislu	37d1ee562d	refactor	2024-05-23 09:21:48 -07:00
chrislu	cdeaaf95b4	go fmt	2024-05-23 08:25:53 -07:00
chrislu	77a7c5c2a0	Merge branch 'master' into mq	2024-05-23 08:25:27 -07:00
chrislu	ba98f02d02	go fmt	2024-05-23 08:25:16 -07:00
chrislu	d40b350f54	coordinator receives unassignment ack	2024-05-23 08:23:35 -07:00
chrislu	73d008fafa	Merge branch 'master' into mq	2024-05-22 20:39:31 -07:00
Chris Lu	cfcf383ee7	fix wrong prefix example	2024-05-21 21:45:29 -07:00
Vlad	a9b057389f	chore: add missing concurrency help in fs.meta.load command (#5609)	2024-05-21 21:42:48 -07:00
chrislu	554ae09f82	rename	2024-05-21 10:02:07 -07:00
chrislu	6ef2d010aa	rename file	2024-05-21 09:57:59 -07:00
chrislu	67e6051585	rename Coordinator to SubCoordinator	2024-05-21 09:57:45 -07:00
chrislu	b5099263a4	rename file	2024-05-21 09:56:46 -07:00
chrislu	d5abffa42c	rename Balancer to PubBalancer	2024-05-21 09:56:30 -07:00
chrislu	fa98ecf71e	client side stop partition subscribing if unassigned	2024-05-21 08:42:04 -07:00
chrislu	6634b42981	fix compilation	2024-05-21 08:26:55 -07:00
chrislu	2190d59bc3	refactor ... ToDo 1. handle unassignment on client side 2. get notified of completed unassignment 3. in consumer_group.go, balance and send unassignment	2024-05-21 08:05:56 -07:00
chrislu	3148dec1f6	avoid concurrent processing for the same key	2024-05-20 13:29:37 -07:00
chrislu	a7d30d0705	refactor	2024-05-20 12:32:12 -07:00
chrislu	33aecb7bb4	remove ProcessorConfiguration	2024-05-20 12:28:01 -07:00
chrislu	65dd5ac6fb	fix	2024-05-20 12:27:45 -07:00
chrislu	3b7f24b1fc	unused	2024-05-20 12:26:54 -07:00
chrislu	f13475ac2e	sending keyed offset	2024-05-20 11:09:48 -07:00
chrislu	d45b1d058d	minor	2024-05-20 11:07:54 -07:00
chrislu	3622196881	go fmt	2024-05-20 11:05:18 -07:00
chrislu	5038577f7e	Merge branch 'master' into mq	2024-05-20 11:04:53 -07:00
chrislu	aba212dedd	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2024-05-20 11:03:58 -07:00
chrislu	d218fe54fa	go fmt	2024-05-20 11:03:56 -07:00
chrislu	7a4ace5dfb	adjust logs	2024-05-20 09:34:07 -07:00
chrislu	d8ab22012a	track offset	2024-05-20 09:33:37 -07:00
chrislu	2b07a40da5	add InflightMessageTracker	2024-05-20 09:19:39 -07:00
chrislu	a2885512e1	fix	2024-05-19 15:06:33 -07:00
chrislu	8d02712c63	ConcurrentPartitionLimit	2024-05-19 14:55:03 -07:00
chrislu	8ce2cced47	less logs	2024-05-19 14:54:46 -07:00
chrislu	c6db3f31a1	start consuming ASAP	2024-05-19 14:52:38 -07:00
chrislu	c21b32dd20	logs	2024-05-19 11:02:17 -07:00
chrislu	b1871427c3	persist consumer group offset ... 1. use one follower 2. read write consumer group offset	2024-05-19 00:46:12 -07:00
sb	a3a00d9499	Feature/mongodb security (#5602)	2024-05-18 12:19:10 -07:00
sb	3fae87632f	add filer configuration options for username/password and tls (#5601)	2024-05-18 09:02:03 -07:00
chrislu	8d5bb7420d	rename	2024-05-17 18:21:21 -07:00
chrislu	94742b8ace	add subscribe follower	2024-05-17 18:17:23 -07:00
Riccardo Bertossa	f6e8a9bf9c	added s3 iam DeleteBucket permission management (#5599)	2024-05-17 04:54:09 -07:00
chrislu	279cb2b85e	consumer acks received messages	2024-05-16 20:28:19 -07:00
chrislu	faffb2973c	adjust errors	2024-05-16 11:02:48 -07:00
chrislu	759a8e50cf	remove logs	2024-05-14 23:22:51 -07:00
chrislu	1f20178ded	subscriber receives partitions and dispatch to processors	2024-05-14 23:22:43 -07:00
chrislu	972e9faaa2	move initial assignment to rebalance logic	2024-05-14 09:09:36 -07:00
chrislu	6e5075e14e	move read write topic config into filer client accessor	2024-05-14 08:50:17 -07:00
Martin Stiborský	54f3913bed	[s3] Fixed s3 replication by sending content-md as base64 (#5596)	2024-05-14 06:48:24 -07:00
chrislu	ac63f2b5a1	refactor	2024-05-13 22:03:57 -07:00
chrislu	844a1a0192	rename	2024-05-13 19:24:47 -07:00
chrislu	372bd8d71d	consumer instance passing MaxPartitionCount to coordinator	2024-05-13 18:30:10 -07:00
chrislu	2142842f82	refactor	2024-05-13 18:25:38 -07:00
Konstantin Lebedev	d389c5b27e	fix: recreate index include deleted files (#5579) ... * fix: recreate index include deleted files https://github.com/seaweedfs/seaweedfs/issues/5508 * fix: counting the number of files * fix: log	2024-05-12 11:31:34 -07:00
Dave St.Germain	731b3aadbe	Add support for OpenBSD (#5578) ... Co-authored-by: Dave St.Germain <dcs@adullmoment.com>	2024-05-10 14:35:41 -07:00
chrislu	8cc5298a08	clean up unused variables	2024-05-10 13:10:45 -07:00
chrislu	d3032d1e80	3.67	2024-05-07 08:47:30 -07:00
chrislu	1917f70f78	also delete parent folder if empty ... fix https://github.com/seaweedfs/seaweedfs/issues/5567	2024-05-07 08:46:19 -07:00
Patrick Schmidt	96c48bd0da	Fix deadlock in lock table locks (#5566)	2024-05-07 05:44:53 -07:00
chrislu	3682eb929f	3.66	2024-05-05 22:23:18 -07:00
chrislu	fcedf97e94	fix test	2024-05-05 22:19:59 -07:00
chrislu	664b704e30	add delay	2024-05-05 14:17:41 -07:00
chrislu	4e3701df77	reset last error	2024-05-05 14:16:54 -07:00
chrislu	658d302aa4	init candidate servers when setting a starting snapshot	2024-05-05 14:16:26 -07:00
chrislu	a5645d50a7	ensure head index is within range	2024-05-05 12:20:08 -07:00
chrislu	253ec73a17	log	2024-05-05 12:19:31 -07:00
chrislu	30f8600a24	working	2024-05-03 21:27:06 -07:00
chrislu	7a9b115cc2	rename	2024-05-02 11:14:58 -07:00
chrislu	4a3e8869bb	subscriber can unmarshal the record	2024-05-02 09:56:11 -07:00
chrislu	1c5043baf7	rename	2024-05-02 09:47:29 -07:00
chrislu	f2f4630029	rename field types	2024-05-02 08:59:22 -07:00
chrislu	b8af997e41	consistent with protobuf data types	2024-05-02 08:52:10 -07:00
chrislu	0f35b3a4ea	add example to publish a record	2024-05-02 08:48:51 -07:00
chrislu	928a4e8dff	rename	2024-05-02 08:35:06 -07:00
chrislu	5bc1b70a4c	rename to WithField()	2024-05-02 08:32:15 -07:00
chrislu	a568d128fb	Merge branch 'master' into mq-subscribe	2024-05-02 08:13:53 -07:00
chrislu	f80fb1c02e	Merge branch 'master' into mq-subscribe	2024-04-29 21:57:31 -07:00
shenxingwuying	ee25ada732	reduce ambiguity about use memory_sequencer (#5555)	2024-04-29 21:51:00 -07:00
dependabot[bot]	36a1cf0361	chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 (#5552) ... * chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie) from 0.3.1 to 1.0.1. - [Release notes](https://github.com/viant/ptrie/releases) - [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md) - [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1) --- updated-dependencies: - dependency-name: github.com/viant/ptrie dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * fix compilation --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>	2024-04-29 18:42:31 -07:00
chrislu	5e8d788311	Merge branch 'master' into mq-subscribe	2024-04-29 06:44:03 -07:00
chrislu	e6bfd061c7	AllowEmptyFolder checks during object deletion	2024-04-29 06:25:06 -07:00
chrislu	d521466a37	split file	2024-04-29 06:23:42 -07:00
chrislu	a1b59948cc	rename files	2024-04-29 05:33:56 -07:00
chrislu	b570e015ce	Update write_parquet_test.go	2024-04-28 13:56:55 -07:00
chrislu	5dc186ef01	record value API	2024-04-28 13:42:45 -07:00
chrislu	b166597c83	fix test	2024-04-28 13:03:01 -07:00
chrislu	189a7fc90e	record type begin and end	2024-04-28 13:00:52 -07:00
chrislu	0da1b7b0f9	improve api	2024-04-28 12:42:58 -07:00
chrislu	1a2332c257	rename api	2024-04-28 12:40:00 -07:00
chrislu	5ce8f50546	Delete schema_builder_test.go	2024-04-28 12:39:27 -07:00
chrislu	8a0e5b6d54	struct to schema correctly	2024-04-28 12:23:54 -07:00
chrislu	f37862bffe	compile test	2024-04-28 12:23:33 -07:00
chrislu	7310d022ad	recursive struct to schema	2024-04-28 11:14:13 -07:00
Konstantin Lebedev	6f9267ebdf	fix: skip file size checking for volume with TTL (#5541) ... fix: skip file size checking for volume with tll https://github.com/seaweedfs/seaweedfs/issues/5508	2024-04-28 00:21:57 -07:00
chrislu	a2a872ca03	rename	2024-04-27 23:44:11 -07:00
chrislu	60103171b6	rename	2024-04-27 23:30:15 -07:00
chrislu	111a4e1126	add field index	2024-04-27 23:30:06 -07:00
Konstantin Lebedev	3d3ee04eb9	[s3] Put bucket lifecycle configuration (#5510)	2024-04-27 07:39:22 -07:00
Kevin Bulteel	a3dab1fcd0	feat: upload files directly without multipart/form-data (#5539)	2024-04-26 16:03:40 -07:00
Nikita Borzykh	0d04264494	Move file deletion logging to v2 verbisity (#5538)	2024-04-26 06:41:06 -07:00
chrislu	72b50980f4	a little bit more efficient	2024-04-25 23:59:30 -07:00
chrislu	8948fb51e3	Merge branch 'master' into mq-subscribe	2024-04-25 23:42:19 -07:00
chrislu	63036133ff	minor	2024-04-25 22:02:03 -07:00
chrislu	2a9b0a9332	use parquet levels to construct row	2024-04-25 21:29:45 -07:00
M@	abf01a0eb7	Fixes unlocked read from logBuffer.LastTsNs that is racey. (#5536)	2024-04-25 15:46:12 -07:00
chrislu	977e7988e6	toRow conversion with levels info	2024-04-25 09:14:37 -07:00
chrislu	9cb9d27b5b	file rename	2024-04-25 08:56:31 -07:00
chrislu	313097aca0	printout test values	2024-04-25 08:54:57 -07:00
chrislu	827e293010	add visitor	2024-04-25 08:54:41 -07:00
chrislu	138a027a2e	add field types but without value	2024-04-24 23:33:11 -07:00
chrislu	30b5aef62e	record type is also optional	2024-04-24 23:32:35 -07:00
chrislu	ce4597c623	delete the generated file	2024-04-24 23:29:55 -07:00
chrislu	f91484ca07	add some notes	2024-04-24 23:28:20 -07:00
chrislu	f93c4cf48c	adjust test	2024-04-24 23:24:53 -07:00
M@	855607c536	Fixes race with unlocked setter. (#5532)	2024-04-24 23:18:28 -07:00
M@	607927da60	Migrates from disintegration/imaging c2019 to cognusion/imaging c2024. (#5533)	2024-04-24 23:16:04 -07:00
chrislu	d88c1872ac	works with single level of list	2024-04-24 23:04:47 -07:00
Konstantin Lebedev	2f3fee9bb9	filer read chunk retry if status code 499 (#5528)	2024-04-24 06:51:05 -07:00
chrislu	d7e5f6b2a5	fix ordering	2024-04-24 01:32:02 -07:00
skycope	6e4b9181f5	fix "volume.fix.replication" move many replications only to one volumeServer (#5522)	2024-04-23 06:33:50 -07:00
chrislu	9ef238b610	has a bug	2024-04-22 00:42:18 -07:00
chrislu	cc1ce3de87	all record fields are optional by default	2024-04-18 23:17:52 -07:00
chrislu	384ed7a381	add test	2024-04-18 22:41:20 -07:00
chrislu	57949f9959	support list type	2024-04-18 22:41:12 -07:00
chrislu	05d18130b6	add schema builder	2024-04-18 22:27:18 -07:00
chrislu	20666bdf81	Merge branch 'master' into mq-subscribe	2024-04-18 21:33:06 -07:00
chrislu	55976ae04a	avoid repeated calls to heavy-weighted viper	2024-04-18 09:09:45 -07:00
chrislu	31f1f96038	improve perf a bit	2024-04-18 08:47:55 -07:00
chrislu	d9490c5e1f	rename	2024-04-18 08:47:45 -07:00
chrislu	0847a17484	convert to parquet schema and value	2024-04-17 23:49:21 -07:00
steve.wei	0bdf121e51	rename VolumeServerVolumeGauge (#5504)	2024-04-17 04:49:50 -07:00
chrislu	ce2b2fa9b2	Merge branch 'master' into mq-subscribe	2024-04-15 20:09:14 -07:00
chrislu	0edc5aad9a	3.65	2024-04-14 20:15:46 -07:00
Konstantin Lebedev	33537ae29f	[s3] fix s3 test_multipart_get_part (#5476) ... * try fix s3 test_multipart_get_part * add passed s3 tests * fix SeaweedFSUploadId * rm spaces * convert part request to range * add passed s3 tests of multipart	2024-04-14 10:41:32 -07:00
yudrywet	8833745fbc	chore: fix some typos in comments (#5497)	2024-04-14 07:42:40 -07:00
chrislu	68ade92d4f	publisher send schema	2024-04-12 23:36:15 -07:00
chrislu	8415476c2d	fix test	2024-04-12 23:35:54 -07:00
chrislu	5324e4288a	rename parameter	2024-04-12 22:33:00 -07:00
chrislu	6f80dbdfe5	Create schema.go	2024-04-12 22:31:28 -07:00
chrislu	53d1d2b78a	save schema when configuring topic	2024-04-12 22:29:53 -07:00
chrislu	04fb4c34e3	add proto for value	2024-04-12 22:27:16 -07:00
chrislu	48fe50df81	minor	2024-04-12 13:31:54 -07:00
chrislu	0cbf364a3d	all fields are optional	2024-04-12 13:29:35 -07:00
chrislu	7a72677f98	add test	2024-04-12 01:31:23 -07:00
chrislu	d45a372615	add schema.proto	2024-04-12 01:30:29 -07:00
chrislu	d5b2cee34a	log possible racing condition ... fix https://github.com/seaweedfs/seaweedfs/issues/5412	2024-04-11 23:44:54 -07:00
chrislu	2a4a9c6343	log possible racing condition ... fix https://github.com/seaweedfs/seaweedfs/issues/5412	2024-04-11 23:44:18 -07:00
chrislu	e53a972483	Merge branch 'master' into mq-subscribe	2024-04-11 19:48:00 -07:00
Konstantin Lebedev	5189a09de0	[volume] Reduce the number of buffers for uploading one chunk (#5458)	2024-04-11 04:47:21 -07:00
skycope	6dae685f9c	fix append disorder (#5492)	2024-04-10 19:21:44 -07:00
chrislu	cc1c69f312	Merge branch 'master' into mq-subscribe	2024-04-08 11:03:14 -07:00
steve.wei	67ead9b18f	fix(volume.fix.replication): adjust volume count, not free volume count (#5479)	2024-04-08 07:30:04 -07:00
Henco Appel	5c8e6014ba	fix: filer authenticate with with volume server (#5480)	2024-04-08 07:27:00 -07:00
clonefetch	9e07a87fcb	chore: fix function names in comment (#5478)	2024-04-08 07:19:02 -07:00
Konstantin Lebedev	3e25ed1b11	[s3] add s3 pass test_multipart_upload_size_too_small (#5475) ... * add s3 pass test_multipart_upload_size_too_small * refactor metric names * return ErrNoSuchUpload if empty parts * fix test	2024-04-07 11:52:35 -07:00
Konstantin Lebedev	35cba720a5	[s3] add s3 pass test_multipart_upload (#5474) ... add s3 pass test_multipart_upload	2024-04-07 11:51:22 -07:00
Konstantin Lebedev	d42a04cceb	[s3] fix s3 test_multipart_resend_first_finishes_last (#5471) ... * try fix s3 test https://github.com/seaweedfs/seaweedfs/pull/5466 * add error handler metrics * refactor * refactor multipartExt * delete bad entry parts	2024-04-06 10:56:39 -07:00
Nikita Borzykh	7aa25c1137	fix: correctly form partNumber from new entry.Name format in "listObjectParts" (#5470)	2024-04-04 09:16:16 -07:00
skycope	b19c9847c6	fix completed multiupload lost data (#5460) ... If there are putObjectPart requests with the same uploadId during completeMultiPart, it can result in data loss. putObjectPart requests might be due to timeout retries. Co-authored-by: Yang Wang <yangwang@weride.ai>	2024-04-02 20:18:42 -07:00
chrislu	ccdd9cd8de	refactor	2024-04-02 16:25:43 -07:00
chrislu	f37c0d0d7a	comment out println	2024-04-02 15:31:31 -07:00
Konstantin Lebedev	d5d8b8e2ae	fix panic at isAllWritable (#5457) ... fix panic https://github.com/seaweedfs/seaweedfs/issues/5456	2024-04-02 09:06:19 -07:00
Henco Appel	2a88da4de7	fix: always close volume file (#4530) (#5459) ... If sync fails then close is never called. We should always be calling close on the file. Co-authored-by: Hendrik Appel <happel@europe.altair.com>	2024-04-02 08:31:40 -07:00
chrislu	d935f70e3c	shutdown follower when leader stops	2024-04-01 22:32:30 -07:00
chrislu	f07875e8e1	send flush message to follower before shutting down logBuffer	2024-04-01 16:01:26 -07:00
Nikita Borzykh	df9031e19a	fix: ensure complete gRPC stream consumption in doListFilerEntries to prevent context canceled errors. (#5441)	2024-04-01 15:55:16 -07:00
chrislu	e568e742c9	renaming	2024-04-01 15:02:52 -07:00
chrislu	47975e75bf	purge old code	2024-03-31 17:49:34 -07:00
chrislu	237e68059d	shutdown follower if no clients, not only publishers ... to avoid overhead when a publisher repeatedly send one message and close	2024-03-31 17:45:26 -07:00
chrislu	66a878af39	add control message to data message	2024-03-31 16:35:46 -07:00
chrislu	546ae87c39	send is_close message to broker	2024-03-31 01:28:40 -07:00
chrislu	c9df613b6b	add publisher name for debugging	2024-03-31 00:19:16 -07:00
chrislu	ca4f89a6f6	simplify	2024-03-30 20:37:57 -07:00
chrislu	0d55bddc75	logs	2024-03-30 20:37:46 -07:00
Nikita Borzykh	829c0da381	fix: adjust condition in prefixFilterEntries prevent infinite loop (#5440)	2024-03-30 14:47:00 -07:00
Konstantin Lebedev	9c1e0f5811	[master] grow volumes if no writable volumes in current dataCenter (#5434) ... * grow volumes if no writable volumes in current dataCenter https://github.com/seaweedfs/seaweedfs/issues/3886 * fix tests with volume grow * automatic volume grow one volume * add ErrorChunkAssign metrics	2024-03-29 00:38:27 -07:00
Konstantin Lebedev	df40908e57	fix panic 5435 (#5436)	2024-03-28 16:17:59 -07:00
chrislu	13b7c7b366	Merge branch 'master' into mq-subscribe	2024-03-27 23:57:00 -07:00
chrislu	e3c8be7f89	LocalPartition shutdown	2024-03-27 23:55:06 -07:00
chrislu	ab9fc0ad70	simplify	2024-03-27 23:42:19 -07:00
chrislu	7463058299	change visibility	2024-03-27 23:11:35 -07:00
chrislu	dab5454332	publish and send to follower	2024-03-27 23:10:24 -07:00
chrislu	f249f831b1	MaybeShutdownLocalPartition refactor	2024-03-27 23:02:51 -07:00
chrislu	2a0ab7e90d	fix	2024-03-27 22:59:50 -07:00
chrislu	059a120708	refactor	2024-03-27 22:58:20 -07:00
chrislu	7d87c1d2bb	refactor	2024-03-27 22:48:37 -07:00
chrislu	6aa804b368	lock instead of RLock, to prevent racing condition ... https://github.com/seaweedfs/seaweedfs/discussions/5432#discussioncomment-8933608	2024-03-27 16:05:11 -07:00
chrislu	5cc94a05b9	separate goroutine to send ack to publisher	2024-03-27 10:27:08 -07:00
chrislu	17806cde2a	separate context ... If using the same context, the downstream grpc would be cancelled when the parent grpc is cancelled.	2024-03-27 10:24:57 -07:00
chrislu	f67c8309d8	rename	2024-03-26 21:52:12 -07:00
chrislu	7d69552974	log server list	2024-03-26 21:51:28 -07:00
chrislu	36fec34c47	print only adapted url ... fix https://github.com/seaweedfs/seaweedfs/issues/5424	2024-03-25 12:50:43 -07:00
chrislu	707ff536a3	Merge branch 'master' into mq-subscribe	2024-03-25 11:21:19 -07:00
chrislu	aed6fdffd2	fix nil volume data backend ... fix https://github.com/seaweedfs/seaweedfs/issues/5417	2024-03-25 08:19:31 -07:00
chrislu	d6a066495b	adjust logs	2024-03-25 08:15:47 -07:00
chrislu	1b1b96ea1e	close local partition if no more publishers and subscribers	2024-03-24 21:10:02 -07:00
Nikita Borzykh	9ca871af53	Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries in etcd meta storage backend (#5416) ... Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries	2024-03-24 14:08:54 -07:00
chrislu	4f5c4c3388	refactor	2024-03-24 13:07:51 -07:00
chrislu	d558a58946	refactor	2024-03-24 13:04:59 -07:00
chrislu	205d2285f3	setup follower by publisher ... * the subscriber would getOrGen a local partition and wait * the publisher would getOrGen a local partition. If localPartition follower is not setup, and init message has follower info, it would create followers based on init.Messages.	2024-03-24 12:57:09 -07:00
Nikita Korolev	2dd2bb3e16	fix filer address parsing (#5415)	2024-03-24 11:20:33 -07:00
Nikita Borzykh	1f3742850d	Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage (#5403)	2024-03-24 08:36:21 -07:00
Konstantin Lebedev	dc9568fc0d	[master] add test for PickForWrite add metrics for volume layout (#5413)	2024-03-22 07:39:11 -07:00
Nikita Korolev	953f571349	add basic support for sentry error reporting (#5410)	2024-03-22 07:10:57 -07:00
adasauce	61f4e40ad9	fix s3api: delimeter properly takes prefixes into account (#5411)	2024-03-22 07:03:34 -07:00
wusong	a5582cc921	[mount] fix GetAttr blocks count (#5409)	2024-03-22 06:11:16 -07:00
chrislu	ac13644fc9	Merge branch 'master' into mq-subscribe	2024-03-21 23:31:49 -07:00
chrislu	03f18e43df	clean up logs	2024-03-21 23:30:49 -07:00
chrislu	b5a6f10612	Merge branch 'master' into mq-subscribe	2024-03-21 23:27:41 -07:00
chrislu	26e24db721	adjust log level	2024-03-21 23:27:19 -07:00
chrislu	36a3301e0c	Merge branch 'master' into mq-subscribe	2024-03-21 23:24:28 -07:00
chrislu	5c0fcae039	clean up logs	2024-03-21 23:24:18 -07:00
chrislu	407a52fc93	adjust log level	2024-03-21 23:23:26 -07:00
chrislu	da31e9b939	correctly wait for the publisher to finish	2024-03-21 09:08:53 -07:00
zuzuviewer	211171e669	Fix iam service response 500 (#5405)	2024-03-21 08:43:54 -07:00
chrislu	e641d49f9f	Merge branch 'master' into mq-subscribe	2024-03-21 08:22:14 -07:00
Konstantin Lebedev	25643cfbc6	fix: panic: assignment to entry in nil map on S3Sink.CreateEntry (#5406)	2024-03-21 08:12:31 -07:00
chrislu	8b18dd940f	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2024-03-21 08:07:58 -07:00
chrislu	50023fe1cf	adjust log level	2024-03-21 08:07:56 -07:00
Konstantin Lebedev	d821cb3b18	fix: sync without dir /buckets/some/.uploads/hash_hash (#5402)	2024-03-20 12:54:29 -07:00
chrislu	7003ce7425	publisher is able to fully send data and receive acks ... still needs to close the pipes cleanly	2024-03-20 12:25:40 -07:00
chrislu	96149cbf7f	Merge branch 'master' into mq-subscribe	2024-03-19 09:14:27 -07:00
chrislu	a1816d21e9	fix compilation	2024-03-19 09:09:14 -07:00
chrislu	ad408a7d38	Merge branch 'master' into mq-subscribe	2024-03-19 08:58:09 -07:00
dependabot[bot]	359b00596f	chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 (#5387) ... * chore(deps): bump github.com/rclone/rclone from 1.65.2 to 1.66.0 Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.65.2 to 1.66.0. - [Release notes](https://github.com/rclone/rclone/releases) - [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md) - [Commits](https://github.com/rclone/rclone/compare/v1.65.2...v1.66.0) --- updated-dependencies: - dependency-name: github.com/rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * Fix compilation error with rclone update (#5395) Fix compilation error --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Damiano Albani <damiano.albani@gmail.com>	2024-03-18 17:09:59 -07:00
jerebear12	7091269e65	fix get file metadata bug (#5394) ... * Removed problematic if statement This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files. * Update filer_server_handers_read.go * Updated filer responses to invalid requests	2024-03-18 11:08:16 -07:00
chrislu	516cba7053	remove FollowInMemoryMessages	2024-03-17 22:27:13 -07:00
chrislu	6ef3e6bcad	logging	2024-03-17 22:24:20 -07:00
chrislu	a375b2815e	Merge branch 'master' into mq-subscribe	2024-03-17 11:46:55 -07:00
chrislu	ca042bd067	simplify	2024-03-17 01:01:45 -07:00
chrislu	4dce1e9d29	Revert "remove github.com/golang/protobuf/proto" ... This reverts commit 1516e0249f.	2024-03-17 01:00:43 -07:00
chrislu	a8b73296d7	fix test3	2024-03-16 23:53:46 -07:00
chrislu	3c559dc294	fix test2	2024-03-16 23:48:31 -07:00
chrislu	ea2601799c	fix test	2024-03-16 23:22:17 -07:00
chrislu	1516e0249f	remove github.com/golang/protobuf/proto	2024-03-16 23:18:11 -07:00
chrislu	8e5068fd2f	notify	2024-03-16 23:16:33 -07:00
chrislu	aba934f0b5	adjust log level	2024-03-16 23:15:06 -07:00
chrislu	44ab42d03c	unnecessary	2024-03-16 17:11:42 -07:00
chrislu	3f2dd8cc3e	add follower id info	2024-03-16 17:11:18 -07:00
chrislu	b74e8082ba	3.64	2024-03-16 11:46:40 -07:00
chrislu	6a61b54f29	Merge branch 'mq-subscribe'	2024-03-16 11:42:23 -07:00
chrislu	27bb38228b	only broad cast when there are waiting threads	2024-03-16 11:38:27 -07:00
chrislu	054d0b3487	wait for data	2024-03-16 11:14:14 -07:00
chrislu	f8957b84c1	rename to LastTsNs	2024-03-16 10:52:47 -07:00
chrislu	529fa8c4c8	go fmt	2024-03-16 10:51:47 -07:00
chrislu	059df3957d	rename to lastFlushDataTime	2024-03-16 10:51:16 -07:00
chrislu	1b5c547cfe	logs	2024-03-15 21:08:57 -07:00
chrislu	55714f54ab	local partition is generated or not	2024-03-15 21:08:42 -07:00
chrislu	56ce040ab6	log errors	2024-03-15 21:05:40 -07:00
chrislu	549455c30d	streaming mode	2024-03-15 21:05:24 -07:00
chrislu	f653838f34	follower receive a hello message before actual data ... the leader can ensure the follower has connected	2024-03-15 20:36:01 -07:00
chrislu	64a32730a2	add size()	2024-03-15 20:34:39 -07:00
Michał Frąckiewicz	205829fa22	Document default method used by compact (#5379)	2024-03-15 12:16:44 -07:00
Konstantin Lebedev	54ee73287d	[filer] replace an invalid methods in a metric with an invalid one (#5378) ... fix: replace an invalid methods in a metric with an invalid one for filer	2024-03-14 08:19:09 -07:00
chrislu	6f615c62ae	Merge branch 'master' into mq-subscribe	2024-03-12 09:19:06 -07:00
chrislu	604091a480	use stopChan to close previous filer peer meta subscription instances	2024-03-12 09:18:54 -07:00
chrislu	1efa502dbb	go fmt	2024-03-12 09:17:20 -07:00
chrislu	6f75a0af55	ask follower to follow	2024-03-10 14:34:28 -07:00
chrislu	c1a5952009	rename	2024-03-09 12:57:01 -08:00
chrislu	e7823ee967	retry connecting to broker leader	2024-03-09 12:56:49 -08:00
rustrover	ab70aa92da	remove repetitive words (#5364)	2024-03-09 07:35:47 -08:00
chrislu	3b28433cb1	change LogBuffer visibility in LocalPartition	2024-03-07 10:57:04 -08:00
chrislu	28f8f33d6a	include key in LogEntry	2024-03-07 10:53:30 -08:00
chrislu	34f2b600ac	each log function adds a "done" return parameter	2024-03-07 10:50:09 -08:00
chrislu	62397f2371	adjust error	2024-03-07 10:47:50 -08:00
chrislu	832b60c13f	pass errors	2024-03-07 10:47:06 -08:00
chrislu	49869eec83	edge cases for broker assignment	2024-03-07 10:45:38 -08:00
chrislu	33ab6cfcf7	adjust error	2024-03-07 10:44:09 -08:00
chrislu	8f79bb3987	Merge branch 'master' into mq-subscribe	2024-03-07 10:42:29 -08:00
chrislu	b544a69550	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2024-03-07 10:42:15 -08:00
chrislu	5c0f01c0eb	purge cluster id	2024-03-07 10:42:12 -08:00
Konstantin Lebedev	170b63d6f8	[filer.backup] add param uploader_part_size for S3sink (#5352) ... * fix: install cronie * chore: refactor configure S3Sink * chore: refactor cinfig * add filer-backup compose file * fix: X-Amz-Meta-Mtime and resolve with comments * fix: attr mtime * fix: MaxUploadPartst is reduced to the maximum allowable * fix: env and force set max MaxUploadParts * fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB	2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri	ecc154fa9e	Add deleted bytes to total_disk_size (#5161)	2024-03-06 07:28:46 -08:00
adasauce	3e3e0fa240	feat: add support for S3 ListObjects and ListObjectsV2 (#5350)	2024-03-06 07:24:44 -08:00
Konstantin Lebedev	872b5161e6	[shell] fs log purge (#5344) ... * fix: install cronie * chore: shell delete old logs * Revert "fix: install cronie" This reverts commit 920832b04b. * fix: dayAgo * fix: Nit: dayAgo => daysAgo	2024-03-05 09:09:08 -08:00
chrislu	fe03b1b522	handle single broker case	2024-03-01 05:59:40 -08:00
chrislu	50c5dd7313	test low active brokers	2024-02-29 20:50:57 -08:00
chrislu	7598922b41	assign followers	2024-02-29 14:51:06 -08:00
chrislu	7e443ef0a1	latest protoc-gen-go	2024-02-29 10:06:23 -08:00
chrislu	ead27a4f59	publish init add followers	2024-02-29 09:39:12 -08:00
chrislu	1b4484bf0a	go fmt	2024-02-29 09:38:52 -08:00
chrislu	2a7028373d	Merge branch 'master' into mq-subscribe	2024-02-29 06:22:39 -08:00
Sean Ross	0833057503	Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… (#5351) ... Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.	2024-02-29 02:16:28 -08:00
jerebear12	85d6d5371b	Disable filer UI in configuration (#5297) ... * Add filer.ui.enabled configuration property * Add filer.expose_directory_metadata to config * Ammend commit * Remove ShowUI reference * Update all routes that allow directory metadata * Add cmd flag to server.go	2024-02-27 08:38:55 -08:00
chrislu	76ae39a363	Merge branch 'master' into mq-subscribe	2024-02-26 07:03:09 -08:00
chrislu	54d7748a4a	3.63	2024-02-25 22:17:28 -08:00
chrislu	a52ce076fc	Merge branch 'master' into mq-subscribe	2024-02-25 08:00:37 -08:00
Konstantin Lebedev	7187346cc1	avoid unexpected compact size (#5272) ... https://github.com/seaweedfs/seaweedfs/issues/5215	2024-02-24 05:27:35 -08:00
chrislu	7c45992c79	when verifyNeedleIntegrity, do not truncate files ... related to https://github.com/seaweedfs/seaweedfs/pull/5259	2024-02-24 02:37:03 -08:00
Chris Lu	32cb9b3276	fix unaligned 64-bit atomic operation in weed mount on arm when readi… (#5308) ... fix unaligned 64-bit atomic operation in weed mount on arm when reading a file fix #5306	2024-02-19 18:51:45 -08:00
7y-9	41ec9d7759	refactor: put the auth outside (#5313)	2024-02-19 03:07:48 -08:00
7y-9	f037c09c11	fix: only admin auth can delete S3 bucket (#5312)	2024-02-19 00:38:05 -08:00
Konstantin Lebedev	6181aa7594	fix: fs verify error counter (#5261)	2024-02-15 01:31:51 -08:00
Sébastien	c694cb4e7d	filer: reduce delete entry ErrNotFound logs (#5301)	2024-02-14 14:22:08 -08:00