7222 Commits

Author	SHA1	Message	Date
chrislu	f8957b84c1	rename to LastTsNs	2024-03-16 10:52:47 -07:00
chrislu	529fa8c4c8	go fmt	2024-03-16 10:51:47 -07:00
chrislu	059df3957d	rename to lastFlushDataTime	2024-03-16 10:51:16 -07:00
chrislu	1b5c547cfe	logs	2024-03-15 21:08:57 -07:00
chrislu	55714f54ab	local partition is generated or not	2024-03-15 21:08:42 -07:00
chrislu	56ce040ab6	log errors	2024-03-15 21:05:40 -07:00
chrislu	549455c30d	streaming mode	2024-03-15 21:05:24 -07:00
chrislu	f653838f34	follower receive a hello message before actual data ... the leader can ensure the follower has connected	2024-03-15 20:36:01 -07:00
chrislu	64a32730a2	add size()	2024-03-15 20:34:39 -07:00
Michał Frąckiewicz	205829fa22	Document default method used by compact (#5379)	2024-03-15 12:16:44 -07:00
Konstantin Lebedev	54ee73287d	[filer] replace an invalid methods in a metric with an invalid one (#5378) ... fix: replace an invalid methods in a metric with an invalid one for filer	2024-03-14 08:19:09 -07:00
chrislu	6f615c62ae	Merge branch 'master' into mq-subscribe	2024-03-12 09:19:06 -07:00
chrislu	604091a480	use stopChan to close previous filer peer meta subscription instances	2024-03-12 09:18:54 -07:00
chrislu	1efa502dbb	go fmt	2024-03-12 09:17:20 -07:00
chrislu	6f75a0af55	ask follower to follow	2024-03-10 14:34:28 -07:00
chrislu	c1a5952009	rename	2024-03-09 12:57:01 -08:00
chrislu	e7823ee967	retry connecting to broker leader	2024-03-09 12:56:49 -08:00
rustrover	ab70aa92da	remove repetitive words (#5364)	2024-03-09 07:35:47 -08:00
chrislu	3b28433cb1	change LogBuffer visibility in LocalPartition	2024-03-07 10:57:04 -08:00
chrislu	28f8f33d6a	include key in LogEntry	2024-03-07 10:53:30 -08:00
chrislu	34f2b600ac	each log function adds a "done" return parameter	2024-03-07 10:50:09 -08:00
chrislu	62397f2371	adjust error	2024-03-07 10:47:50 -08:00
chrislu	832b60c13f	pass errors	2024-03-07 10:47:06 -08:00
chrislu	49869eec83	edge cases for broker assignment	2024-03-07 10:45:38 -08:00
chrislu	33ab6cfcf7	adjust error	2024-03-07 10:44:09 -08:00
chrislu	8f79bb3987	Merge branch 'master' into mq-subscribe	2024-03-07 10:42:29 -08:00
chrislu	b544a69550	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2024-03-07 10:42:15 -08:00
chrislu	5c0f01c0eb	purge cluster id	2024-03-07 10:42:12 -08:00
Konstantin Lebedev	170b63d6f8	[filer.backup] add param uploader_part_size for S3sink (#5352) ... * fix: install cronie * chore: refactor configure S3Sink * chore: refactor cinfig * add filer-backup compose file * fix: X-Amz-Meta-Mtime and resolve with comments * fix: attr mtime * fix: MaxUploadPartst is reduced to the maximum allowable * fix: env and force set max MaxUploadParts * fix: env WEED_SINK_S3_UPLOADER_PART_SIZE_MB	2024-03-07 08:35:51 -08:00
Seyed Mahdi Sadegh Shobeiri	ecc154fa9e	Add deleted bytes to total_disk_size (#5161)	2024-03-06 07:28:46 -08:00
adasauce	3e3e0fa240	feat: add support for S3 ListObjects and ListObjectsV2 (#5350)	2024-03-06 07:24:44 -08:00
Konstantin Lebedev	872b5161e6	[shell] fs log purge (#5344) ... * fix: install cronie * chore: shell delete old logs * Revert "fix: install cronie" This reverts commit 920832b04b. * fix: dayAgo * fix: Nit: dayAgo => daysAgo	2024-03-05 09:09:08 -08:00
chrislu	fe03b1b522	handle single broker case	2024-03-01 05:59:40 -08:00
chrislu	50c5dd7313	test low active brokers	2024-02-29 20:50:57 -08:00
chrislu	7598922b41	assign followers	2024-02-29 14:51:06 -08:00
chrislu	7e443ef0a1	latest protoc-gen-go	2024-02-29 10:06:23 -08:00
chrislu	ead27a4f59	publish init add followers	2024-02-29 09:39:12 -08:00
chrislu	1b4484bf0a	go fmt	2024-02-29 09:38:52 -08:00
chrislu	2a7028373d	Merge branch 'master' into mq-subscribe	2024-02-29 06:22:39 -08:00
Sean Ross	0833057503	Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the fin… (#5351) ... Moved noapplexattr under runtime.GOARCH == "amd64" to resolve the finder copy bug on arm64 macOS devices.	2024-02-29 02:16:28 -08:00
jerebear12	85d6d5371b	Disable filer UI in configuration (#5297) ... * Add filer.ui.enabled configuration property * Add filer.expose_directory_metadata to config * Ammend commit * Remove ShowUI reference * Update all routes that allow directory metadata * Add cmd flag to server.go	2024-02-27 08:38:55 -08:00
chrislu	76ae39a363	Merge branch 'master' into mq-subscribe	2024-02-26 07:03:09 -08:00
chrislu	54d7748a4a	3.63	2024-02-25 22:17:28 -08:00
chrislu	a52ce076fc	Merge branch 'master' into mq-subscribe	2024-02-25 08:00:37 -08:00
Konstantin Lebedev	7187346cc1	avoid unexpected compact size (#5272) ... https://github.com/seaweedfs/seaweedfs/issues/5215	2024-02-24 05:27:35 -08:00
chrislu	7c45992c79	when verifyNeedleIntegrity, do not truncate files ... related to https://github.com/seaweedfs/seaweedfs/pull/5259	2024-02-24 02:37:03 -08:00
Chris Lu	32cb9b3276	fix unaligned 64-bit atomic operation in weed mount on arm when readi… (#5308) ... fix unaligned 64-bit atomic operation in weed mount on arm when reading a file fix #5306	2024-02-19 18:51:45 -08:00
7y-9	41ec9d7759	refactor: put the auth outside (#5313)	2024-02-19 03:07:48 -08:00
7y-9	f037c09c11	fix: only admin auth can delete S3 bucket (#5312)	2024-02-19 00:38:05 -08:00
Konstantin Lebedev	6181aa7594	fix: fs verify error counter (#5261)	2024-02-15 01:31:51 -08:00
Sébastien	c694cb4e7d	filer: reduce delete entry ErrNotFound logs (#5301)	2024-02-14 14:22:08 -08:00
Konstantin Lebedev	8fb978d6f6	[volume] requests that were canceled by the client are not an internal server problem(500 => 499) (#5304)	2024-02-14 14:18:56 -08:00
Justin Dhillon	a48e2ec45b	Fix Broken Links (#5287) ... * https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-setprocessworkingsetsize * https://learn.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-getprocessworkingsetsize * remove https://github.com/h5bp/html5-boilerplate/blob/master/src/css/main.css * https://github.com/AShiou/hof	2024-02-14 08:26:38 -08:00
Chris Lu	16b4f9d9ab	subscriber find broker leader first	2024-02-05 23:14:25 -08:00
Chris Lu	e27deed4bc	upgrade protoc	2024-02-05 18:39:08 -08:00
Chris Lu	656b78d1af	proxy to broker leader	2024-02-05 18:00:50 -08:00
Chris Lu	90fcde0e26	report EOF error	2024-02-05 17:47:34 -08:00
Chris Lu	07028a4864	minor	2024-02-05 17:30:20 -08:00
Chris Lu	fa72761bb8	fix on finding lock owner	2024-02-05 17:30:14 -08:00
Chris Lu	4dc6681833	proxy requests to lock owner	2024-02-05 16:46:40 -08:00
sxlehua	ffe5f19aa0	Add s3 postpolicy support header (#5280) ... * Add s3 postpolicy support header * optimized code	2024-02-05 07:55:07 -08:00
Chris Lu	0d74ac9224	fix divide by zero	2024-02-04 13:38:00 -08:00
Chris Lu	f9131de43f	fix lock TTL	2024-02-04 12:47:21 -08:00
Chris Lu	44b49bec99	less verbose	2024-02-04 12:47:07 -08:00
Chris Lu	53612b770c	Merge branch 'master' into mq-subscribe	2024-02-04 10:44:01 -08:00
Chris Lu	56df44845f	fix when two consecutive chunks with the same modified_ts_ns ... fix https://github.com/seaweedfs/seaweedfs/issues/5276	2024-02-04 09:22:24 -08:00
Chris Lu	0a12301b3d	avoid too large expiration time	2024-02-04 09:20:21 -08:00
sxlehua	0b49c16361	Adapt S3 POST ContentType (#5275)	2024-02-03 22:33:34 -08:00
Chris Lu	1b5ba4190c	avoid timestamp overflow	2024-02-03 13:45:59 -08:00
chrislu	d41792461c	lock returns host and owner	2024-02-02 15:54:57 -08:00
chrislu	d30150dde1	use a short-lived lock	2024-02-01 23:01:44 -08:00
chrislu	c1bb76d709	fix nil error	2024-01-30 08:21:42 -08:00
chrislu	ac50d8a822	re-lock if the lock owner is not found	2024-01-29 22:46:23 -08:00
Benoît Knecht	56287bd07d	weed/shell: Cluster check other disk types (#5245) ... * week/shell: Cluster check other disk types The `cluster.check` command only took the empty (`""`) and `hdd` disk types into consideration, but a cluster with only `ssd` or `nvme` disk types would be equally valid. This commit simply checks that _any_ disk type is defined, and that some volumes are available for it. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch> * weed/shell: Replace loop that copies slice Use the following construct instead of a `for` loop: ```golang x = append(x, y...) ``` See https://staticcheck.dev/docs/checks#S1011. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch> * weed/shell: Check disk types when filer is in use Filer stores its metadata logs in generic (i.e. `""`) or HDD disk type volumes, so make sure those disk types exist and have volumes associated with them when Filer is deployed in the cluster. Signed-off-by: Benoît Knecht <bknecht@protonmail.ch> --------- Signed-off-by: Benoît Knecht <bknecht@protonmail.ch>	2024-01-29 10:36:37 -08:00
Sébastien	0775d05a23	fix: http range request return status 500 (#5251) ... When volume server unavailable for at least one chunk; was returning status 206. Split `StreamContent` in two parts, - first prepare, to get chunk info and return stream function - then write chunk, with that stream function That allow to catch error in first step before setting response status code in `processRangeRequest`	2024-01-29 10:35:52 -08:00
chrislu	0b2e5ddc7c	wait 3 seconds before shutting down publish client, to wait for all messages to be received	2024-01-28 22:04:42 -08:00
chrislu	545d5fbdf6	unused code	2024-01-28 22:03:13 -08:00
chrislu	a507069d19	lock for creating local partition	2024-01-28 15:55:26 -08:00
chrislu	9e6ea80dff	read existing topic conf	2024-01-28 14:27:52 -08:00
chrislu	a297247137	wait for each publish clients	2024-01-28 14:27:32 -08:00
chrislu	b6c5e57c30	read will block if no items	2024-01-28 13:10:34 -08:00
chrislu	0bf5424a2e	adjust visibility	2024-01-28 13:09:30 -08:00
chrislu	dedfd31dfb	refactor	2024-01-28 12:30:08 -08:00
chrislu	cbf750a31f	refactor	2024-01-28 12:23:20 -08:00
chrislu	b51dfe2bff	wait for publishing clients	2024-01-28 12:06:58 -08:00
chrislu	f8c55f101e	clean up	2024-01-28 12:06:30 -08:00
chrislu	d40225ecd2	return the lookup results	2024-01-28 12:05:44 -08:00
chrislu	841fafd0a8	publish to input buffer ... currently, the input buffer may not exist when start to publish	2024-01-27 23:43:22 -08:00
chrislu	5dc1362bdc	close the input	2024-01-27 16:12:49 -08:00
chrislu	fa835c9319	Update buffered_queue_test.go	2024-01-27 14:46:12 -08:00
Chris Lu	b6de35cdb2	moved. there are some deadlock. WIP	2024-01-27 13:51:19 -08:00
chrislu	91af1f3069	schedule jobs	2024-01-26 14:09:57 -08:00
Benoît Knecht	f6e39139d8	weed/storage: Fix volume info file permissions (#5246)	2024-01-26 08:31:27 -08:00
chrislu	08c5fba825	adds a buffered queue	2024-01-25 20:22:41 -08:00
chrislu	cb50b720fd	Revert "Fix weed fix" ... This reverts commit 6c1ce18541.	2024-01-25 08:09:36 -08:00
Benoît Knecht	1f08010ef0	weed/shell: Cleanup volume balance logic (#5241)	2024-01-25 06:19:58 -08:00
Konstantin Lebedev	0f8e76bbd6	fix: clean metric MasterReplicaPlacementMismatch for unregister volume (#5239)	2024-01-25 00:23:24 -08:00
Benoît Knecht	a6aee847b9	weed/shell: Fix volume.balance logic (#5238)	2024-01-25 00:17:32 -08:00
Konstantin Lebedev	0b71fae2e3	fix: s3tests test_get_object_ifnonematch_good (#5237)	2024-01-24 11:05:55 -08:00
chrislu	025f8eb37b	return dir assign counter ... fix https://github.com/seaweedfs/seaweedfs/issues/5231	2024-01-23 19:48:42 -08:00
chrislu	d4e66cc140	return dir assign counter ... fix https://github.com/seaweedfs/seaweedfs/issues/5231	2024-01-23 19:47:31 -08:00
chrislu	a8ddbf9ed2	Merge branch 'master' into mq-subscribe	2024-01-23 19:41:16 -08:00
chrislu	49fcb48e04	same logic as grpc assign ... fix https://github.com/seaweedfs/seaweedfs/issues/5213	2024-01-23 19:39:59 -08:00
chrislu	d268fbe18a	when configure, cancel existing assignments	2024-01-22 11:57:17 -08:00
chrislu	3ff6b31d94	sub broker wait for partition	2024-01-22 11:56:55 -08:00
chrislu	2beaa2d0b3	pub/sub brokers check filer for assigned partitions	2024-01-22 11:44:21 -08:00
chrislu	c3f8530f97	refactor	2024-01-22 11:06:53 -08:00
chrislu	5f35857cd0	no need to assign partitions explicitly	2024-01-22 11:04:47 -08:00
chrislu	428fea45f3	updated and added assignments	2024-01-22 11:04:24 -08:00
chrislu	e8b05ecc91	add/remove assigned partitions	2024-01-22 10:47:39 -08:00
chrislu	861ad732ca	refactor	2024-01-22 09:00:22 -08:00
chrislu	47924afa1c	refactor	2024-01-22 08:52:44 -08:00
chrislu	581cf6644c	lookup reads from filer instead of local memory	2024-01-22 00:52:55 -08:00
chrislu	7121ae9617	separate out read topic conf and ensure topic active assignments	2024-01-22 00:51:31 -08:00
chrislu	c77d35313e	pub/sub broker only check local assigned partitions	2024-01-22 00:49:57 -08:00
chrislu	b0a2e9aea3	fix assignments if brokers changed	2024-01-21 01:27:22 -08:00
chrislu	9ce7c482b3	refactor	2024-01-21 01:24:12 -08:00
chrislu	a5b906c0dd	refactor	2024-01-21 01:23:43 -08:00
chrislu	34839237ab	refactor	2024-01-20 12:16:40 -08:00
chrislu	2828ccbb30	refactor	2024-01-20 11:41:11 -08:00
chrislu	541140f735	rename	2024-01-19 11:43:40 -08:00
chrislu	bb867694d8	skip updating multipart .upload files	2024-01-18 09:24:52 -08:00
chrislu	4ca6b1be3c	skip updating the .upload multipart files	2024-01-18 09:24:52 -08:00
chrislu	d848efed1b	skip updating multipart .upload files	2024-01-18 09:24:26 -08:00
chrislu	aca3c33bab	skip updating the .upload multipart files	2024-01-18 09:23:24 -08:00
chrislu	314e25aad6	Merge branch 'master' into mq-subscribe	2024-01-18 09:17:16 -08:00
chrislu	5c90b27b0a	go fmt	2024-01-18 09:17:08 -08:00
chrislu	263f1f3d04	Merge branch 'master' into mq-subscribe	2024-01-18 09:16:46 -08:00
chrislu	83e4b02517	fix	2024-01-18 09:16:20 -08:00
chrislu	2fa519bcb5	Merge branch 'pr/5216'	2024-01-18 09:14:01 -08:00
chrislu	15b66a6633	refactor	2024-01-18 09:13:14 -08:00
chrislu	3002087541	http assign logic should be the same as grpc assign ... similar logic as bebbc9fe44	2024-01-18 08:58:53 -08:00
XIAOYQ	be166b434f	fix: skip s3 .uploads	2024-01-18 22:13:46 +08:00
chrislu	c8090b1f99	send assignments to newly connected subscribers	2024-01-17 22:44:39 -08:00
chrislu	c616081ab9	refactor	2024-01-17 22:29:59 -08:00
Konstantin Lebedev	4e9ea1e628	chore: filer healthz handler check filer store (#5208)	2024-01-17 07:46:37 -08:00
Konstantin Lebedev	f9cf13fada	[filer] avoid return http ok for not allowed methods (#5209)	2024-01-17 07:17:07 -08:00
Andrew Azores	2eb82778bc	chore: fix typos in scaffold help output (#5211)	2024-01-17 07:10:17 -08:00
chrislu	bc8d2a01cc	create local topic partition from config on filer	2024-01-17 00:15:52 -08:00
chrislu	fdf0ea8e11	passing partition info	2024-01-16 09:30:57 -08:00
chrislu	be0c426dc7	simplify to LookupTopicPartitions(topic)	2024-01-16 09:30:46 -08:00
chrislu	db3670a3a5	simplify api	2024-01-16 08:55:47 -08:00
chrislu	34a78ffad0	remove isForPublish from LookupTopicBrokers ... also adds a return parameter: whether the topic exists or not	2024-01-16 08:52:42 -08:00
chrislu	3795d8dca8	release local topic partition if no publisher and subscribers	2024-01-16 08:43:07 -08:00
chrislu	f782165638	rename	2024-01-16 08:40:45 -08:00
chrislu	dd1ec70e62	rename	2024-01-16 08:40:23 -08:00
chrislu	1877ce5126	rename	2024-01-15 21:31:21 -08:00
chrislu	ba73199174	reuse local partition	2024-01-15 21:22:41 -08:00
chrislu	33d5343ea1	Merge branch 'master' into mq-subscribe	2024-01-15 20:56:21 -08:00
chrislu	59b8af99b0	3.62	2024-01-15 20:44:00 -08:00
chrislu	6d45bb3038	Merge branch 'master' into mq-subscribe	2024-01-15 20:43:03 -08:00
chrislu	7e6497cc1c	adjust publisher subscriber	2024-01-15 20:42:46 -08:00
skycope	316ae45795	can vacuum volume when size mismatch (#5200) ... Co-authored-by: Yang Wang <yangwang@weride.ai>	2024-01-15 16:16:46 -08:00
chrislu	026c54a9bb	fix publisher	2024-01-15 00:20:21 -08:00
chrislu	fa59a5d67e	read from disk if not in memory	2024-01-15 00:20:12 -08:00
chrislu	49f7de9daa	Merge branch 'master' into mq-subscribe	2024-01-13 17:52:18 -08:00
chrislu	d6ba97219b	refactoring	2024-01-13 17:51:53 -08:00
Konstantin Lebedev	8d23e36c45	fix: doDeleteFiles deletes files (#5198)	2024-01-12 11:04:29 -08:00
Konstantin Lebedev	1169f94310	Fix filer sync set offset (#5197) ... * fix: compose 2mount with sync * fix: DATA RACE https://github.com/seaweedfs/seaweedfs/issues/5194 https://github.com/seaweedfs/seaweedfs/issues/5195	2024-01-12 10:57:18 -08:00
chrislu	3aa3991f0f	adjust client side logs	2024-01-11 23:08:53 -08:00
chrislu	7afaad31a4	wait on local topic partition creation	2024-01-11 23:08:02 -08:00
chrislu	61dbdd0ff6	log ctrl messages	2024-01-11 23:07:32 -08:00
chrislu	f750a5e03b	passing timestamp	2024-01-11 23:03:55 -08:00
chrislu	45994641e9	lint	2024-01-11 23:03:35 -08:00
chrislu	70a011a224	Merge branch 'master' into mq-subscribe	2024-01-11 20:58:57 -08:00
skycope	0e8a54f6f6	fix write volume over size MaxPossibleVolumeSize (#5190) ... Co-authored-by: Yang Wang <yangwang@weride.ai>	2024-01-11 20:23:46 -08:00
chrislu	050efe6c08	factor in existing ec volume count when estimating max volume count ... related to https://github.com/seaweedfs/seaweedfs/issues/5191	2024-01-11 08:19:09 -08:00
chrislu	188f471a24	logging	2024-01-11 07:55:26 -08:00
chrislu	6a7a679137	passing partition generation timestamp	2024-01-10 22:36:17 -08:00
Konstantin Lebedev	a7fc723ae0	chore: add status code for request_total metrics (#5188)	2024-01-10 10:05:27 -08:00
chrislu	8af61dec91	avoid empty spaces in file name	2024-01-09 09:00:54 -08:00
chrislu	aed54eda61	refactor	2024-01-09 08:01:01 -08:00
chrislu	d51efddf5c	flush to disk ... Need to do: read from disk	2024-01-08 23:27:02 -08:00
jerebear12	8c966ac23b	Removed problematic if statement (#5180) ... This if statement was causing the value of option.AllowedOrigins to be always equal to "*". Now the values in the config file will be used when present. This allows for people who don't need this feature to not update their security.toml files.	2024-01-08 11:35:20 -08:00
chrislu	fbb4917e2f	Merge branch 'master' into mq-subscribe	2024-01-08 00:08:49 -08:00
chrislu	8ae00e47a6	3.61	2024-01-08 00:04:49 -08:00
chrislu	49428a303b	add batch index for each memory buffer	2024-01-08 00:03:08 -08:00
Konstantin Lebedev	b9d32d32e1	chore: filer sync add doDeleteFiles option for create only mode (#5166)	2024-01-06 10:02:16 -08:00
chrislu	d0d24f1e40	rename functions	2024-01-05 17:10:43 -08:00
chrislu	d9bd343055	Merge branch 'master' into mq-subscribe	2024-01-05 15:36:00 -08:00
chrislu	1206313952	add logs on flush	2024-01-05 15:35:42 -08:00
chrislu	aa5b6a8187	adjust logs	2024-01-05 15:35:32 -08:00
chrislu	496fc8fbbf	refactor	2024-01-05 15:35:19 -08:00
chrislu	e8611ed85d	subscribe with partition offset	2024-01-05 15:24:14 -08:00
chrislu	531f854af2	rename functions	2024-01-05 15:16:53 -08:00
chrislu	f8787a9761	rename functions	2024-01-05 15:16:41 -08:00
chrislu	add16fb1a1	logs	2024-01-05 15:16:22 -08:00
chrislu	ddd0fde094	rename functions	2024-01-05 15:16:01 -08:00
chrislu	ee41dbb7fc	rename functions	2024-01-05 15:14:25 -08:00
chrislu	1fed37bed0	add offset	2024-01-05 15:03:16 -08:00
Konstantin Lebedev	3b62b338a5	chore: add maxMB option for webdav (#5165)	2024-01-05 11:10:20 -08:00
chrislu	47a4963d7a	subscription start from specified timestamp	2024-01-03 15:57:36 -08:00
chrislu	35869b5c80	subscriber can be notified of the assignment change when topic is just configured ... Next: Subscriber needs to read by the timestamp offset.	2024-01-03 13:30:30 -08:00
Konstantin Lebedev	4f6172f369	Fix missing VersionConfiguration node in get-bucket-versioning response (#5162) ... * fix: s3 response for get bucket version https://github.com/seaweedfs/seaweedfs/issues/5155 * fix: s3 response for get bucket version https://github.com/seaweedfs/seaweedfs/issues/5155	2024-01-03 12:27:13 -08:00
Konstantin Lebedev	3e43778869	fix: webdav avoid create empty files (#5160)	2024-01-03 09:52:51 -08:00
Konstantin Lebedev	b832ddd1ef	fix: return etag with md5 in webdav responses (#5158)	2024-01-03 08:50:32 -08:00
chrislu	efb695fd93	lookup existing topic partitions	2024-01-02 17:29:35 -08:00
chrislu	54e9877093	Merge branch 'master' into mq-subscribe	2024-01-02 13:25:00 -08:00
chrislu	d3688938d9	avoid nil ... fix https://github.com/seaweedfs/seaweedfs/issues/5153 introduced from #5109	2024-01-02 12:55:34 -08:00
chrislu	923b1e5dad	Merge branch 'master' into mq-subscribe	2024-01-01 20:56:12 -08:00
chrislu	d4e91b6ad9	3.60	2024-01-01 19:11:23 -08:00
chrislu	8b4cd50aac	log errors	2024-01-01 19:09:17 -08:00
chrislu	bf9ae74384	add logs	2024-01-01 19:09:05 -08:00
chrislu	32bc8d6a38	adjust wait time	2023-12-31 17:42:44 -08:00
chrislu	458ddbf919	clean up dead code	2023-12-31 13:25:33 -08:00
chrislu	2845230329	passing broker into the assignments	2023-12-28 21:13:49 -08:00
chrislu	093fdc1621	subscriber can get assignments	2023-12-28 20:35:15 -08:00
Varun Upadhyay	77626666c5	Minor cleanup & gitignore update (#5144)	2023-12-28 20:25:43 -08:00
chrislu	9ed26cd7b0	Merge branch 'master' into mq-subscribe	2023-12-28 11:57:30 -08:00
chrislu	c950a40aad	subscriber keep connected to the balancer	2023-12-28 11:56:37 -08:00
Konstantin Lebedev	183352c796	shell meta load add concurrency (#4529) ... * fix: increase speed cmd fs meta load * fix: add wg	2023-12-28 11:54:37 -08:00
Lars Lehtonen	edc0532b1d	weed/server: remove unneeded err from streamReadOneVolume() signature (#5143)	2023-12-28 11:45:13 -08:00
chrislu	bebbc9fe44	create volume grow request if the selected volume is close to full	2023-12-27 11:45:44 -08:00
chrislu	bb9269f3f7	Revert "Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134)" ... This reverts commit 838578b55f.	2023-12-24 12:31:08 -08:00
chrislu	c6b1dc7058	remove unused code	2023-12-24 11:11:41 -08:00
vivekkoya	838578b55f	Update network.go since To16 converts the IP address ip up to a 16-byte representation. If ip is not an IP address (it is the wrong length), To16 returns nil. (#5134)	2023-12-23 15:01:57 -08:00
Seyed Mahdi Sadegh Shobeiri	97236389e8	Add modifyTimeAgo to volume.fsck (#5133) ... * Add modifyTimeAgo to volume.fsck * Fix AppendAtNs	2023-12-23 12:17:30 -08:00
Seyed Mahdi Sadegh Shobeiri	54ba2c8868	Fix cutoffTimeAgo in findMissingChunksInFiler (#5132)	2023-12-23 09:18:16 -08:00
chrislu	03c4b2e988	correct lock accounting ... possibly fix https://github.com/seaweedfs/seaweedfs/issues/5082	2023-12-22 14:16:23 -08:00
chrislu	c4badf7396	adjust options	2023-12-22 11:41:10 -08:00
chrislu	6324062acc	fix	2023-12-22 11:37:15 -08:00
chrislu	e0727071c8	go fmt	2023-12-22 11:33:50 -08:00
chrislu	ee1c9bc314	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-12-22 11:33:02 -08:00
chrislu	034db049a0	comment fix	2023-12-22 11:33:00 -08:00
chrislu	6df44d3c95	option to filer.copy with public volume server url ... fix https://github.com/seaweedfs/seaweedfs/issues/5126	2023-12-22 11:32:43 -08:00
chrislu	7a0d1e7fc1	minor	2023-12-22 11:29:54 -08:00
Farbod	c278f49bca	Using filer.remote.sync concurrency in filer.remote.gateway (#5123) ... - Chnaged ProcessEvenFn to be concurrent just like filer.remote.sync Co-authored-by: itsfarbod <itsfarbod@hamravesh.com>	2023-12-20 16:25:54 -08:00
jerebear12	06343f8976	Set allowed origins in config (#5109) ... * Add a way to use a JWT in an HTTP only cookie If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie. * Added a way to specify allowed origins header from config * Removed unecessary log * Check list of domains from config or command flag * Handle default wildcard and change name of config value to cors	2023-12-20 16:21:11 -08:00
Konstantin Lebedev	3c9bcfb864	chore: add dsn for connection to mysql (#5060) ... * chore: add dsn for connection to mysql * add comment * new comment * fix: validate dsn and adapt password	2023-12-20 16:20:58 -08:00
chrislu	915416a277	fix wrong parameter ... fix https://github.com/seaweedfs/seaweedfs/pull/5102	2023-12-13 12:02:57 -08:00
Farbod	d5c01e5ceb	Remote gateway Bug fix (#5100)	2023-12-12 07:46:45 -08:00
chrislu	61b05fad95	fix test	2023-12-11 20:57:29 -08:00
Chris Lu	580940bf82	Merge accumulated changes related to message queue (#5098) ... * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * balance partitions on brokers * prepare topic partition first and then publish, move partition * purge unused APIs * clean up * adjust logs * add BalanceTopics() grpc API * configure topic * configure topic command * refactor * repair missing partitions * sequence of operations to ensure ordering * proto to close publishers and consumers * rename file * topic partition versioned by unixTimeNs * create local topic partition * close publishers * randomize the client name * wait until no publishers * logs * close stop publisher channel * send last ack * comments * comment * comments * support list of brokers * add cli options * Update .gitignore * logs * return io.eof directly * refactor * optionally create topic * refactoring * detect consumer disconnection * sub client wait for more messages * subscribe by time stamp * rename * rename to sub_balancer * rename * adjust comments * rename * fix compilation * rename * rename * SubscriberToSubCoordinator * sticky rebalance * go fmt * add tests * tracking topic=>broker * merge * comment	2023-12-11 12:05:54 -08:00
Konstantin Lebedev	125ad8fe63	falls back to update only if error contains msg "duplicate entry" ... https://github.com/seaweedfs/seaweedfs/issues/5062	2023-12-09 10:18:18 -08:00
chrislu	c1a6e624e5	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-12-07 08:13:46 -08:00
chrislu	5fe4c04a1a	add missing error in the log ... related to https://github.com/seaweedfs/seaweedfs/issues/5084	2023-12-07 08:13:42 -08:00
Lars Lehtonen	43a5973a66	weed/storage: fix dropped test errors	2023-12-06 12:05:22 -08:00
jerebear12	d5d9fbb8aa	Add a way to use a JWT in an HTTP only cookie ... If a JWT is not included in the Authorization header or a query string, attempt to get a JWT from an HTTP only cookie.	2023-12-05 08:57:01 -08:00
chrislu	756bcc032d	adjust logs	2023-11-27 12:57:29 -08:00
Konstantin Lebedev	dd78397fea	logging PickForWrite error ... https://github.com/seaweedfs/seaweedfs/issues/3886	2023-11-27 12:56:15 -08:00
chrislu	81f11883e3	go fmt	2023-11-26 11:47:20 -08:00
chrislu	7a50762b9e	reduce binary size by skipping rclone	2023-11-23 00:48:27 -08:00
zehweh	d818ca9631	fix TestMisplacedChecking() and add test-case	2023-11-21 08:24:34 -08:00
julor	2fa37e0287	Update filer_server_handlers_write_autochunk.go	2023-11-20 11:17:14 -08:00
julor	9e03b916d8	verify content-md5 in header ... if upload content by s3 client and set content-md5 in header,then will verify recived and saved content,if not equal delete thos chunks	2023-11-20 11:17:14 -08:00
SmsS4	6c1ce18541	Fix weed fix	2023-11-20 11:14:19 -08:00
Lars Lehtonen	57801687de	weed/command: fix dropped error	2023-11-20 08:05:24 -08:00
Viktor Kuzmin	06e723f629	Waiting for writes to complete with acquired locks may lead to deadlock, fixes #4952	2023-11-17 07:38:14 -08:00
Konstantin Lebedev	2b229e98ce	fix: add doDeleteFile option for filer backup	2023-11-17 07:37:28 -08:00
chrislu	17710e1ecb	adjust logs	2023-11-15 18:11:35 -08:00
Amir H. Yeganemehr	6f559fe838	fs.mergeVolumes: change dir to a named parameter	2023-11-13 08:25:44 -08:00
Amir H. Yeganemehr	d3556479be	Fix typo (dist => dest)	2023-11-13 06:41:24 -08:00
Amir H. Yeganemehr	bb274ea8a3	fs.mergeVolumes: Make a plan based on volumes size	2023-11-13 06:41:24 -08:00
Konstantin Lebedev	e55fa68848	fix: excludeFileName is empty	2023-11-13 06:23:46 -08:00
Konstantin Lebedev	3c5295a1a6	filer backup add option for exclude file names that match the regexp to sync on filer	2023-11-13 06:23:46 -08:00
Konstantin Lebedev	5e41ab1370	s3 api add not implemented response for PutBucketVersioning	2023-11-13 06:22:23 -08:00
Konstantin Lebedev	7f6152f130	s3 api add default response for GetBucketVersioning	2023-11-13 06:22:23 -08:00
Amir H. Yeganemehr	b05d682c48	make help command in shell case insensitive	2023-11-13 06:06:45 -08:00
chrislu	27b34f3793	3.59	2023-11-12 22:12:08 -08:00
jsh	47112917ff	ec.decode: mount the collected ec shards	2023-11-10 00:04:42 -08:00
chrislu	85bed4d92e	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-11-09 08:02:57 -08:00
chrislu	c563386675	rename parameters	2023-11-09 08:02:39 -08:00
Lars Lehtonen	5db82e594a	weed/server: fix dropped webdav error	2023-11-07 19:05:21 -08:00
zehweh	2b9dda7d2e	fix isMisplaced() in command_volume_fix_replication.go (#4988)	2023-11-07 07:58:19 -08:00
Konstantin Lebedev	d693beea87	Replace an invalid method in a metric with an invalid one (#4987)	2023-11-06 10:51:47 -08:00
chrislu	a667f0ad99	fix error reporting ... related to https://github.com/seaweedfs/seaweedfs/issues/4978	2023-11-05 07:17:16 -08:00
chrislu	b40ba7c7fe	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-11-01 16:28:05 -07:00
chrislu	310eec9a9c	skip if path changed ... mitigate https://github.com/seaweedfs/seaweedfs/issues/4968	2023-11-01 16:27:59 -07:00
Nico D'Cotta	3834ad5c48	refactor webdav subdirectory, fixes #4967 (#4969) ... * refactor webdav subdirectory, fixes #4967 * fix bug where Name() was not called on delegate wrappedFileInfo	2023-11-01 13:12:49 -07:00
tired-engineer	4ed06e9ba5	Pass correct Filer's gRPC port to S3 server. (#4972) ... Fixes seaweedfs/seaweedfs#4971	2023-11-01 10:11:41 -07:00
SmsS4	ba11331cc1	Fix twice increase metric counter (#4961)	2023-10-30 08:05:27 -07:00
SmsS4	f95848ba7d	Add ErrorGetNotFound and ErrorGetInternal to volume server metrics (#4960)	2023-10-30 07:38:03 -07:00
chrislu	d1e83a3b4d	3.58	2023-10-29 23:04:20 -07:00
duanhongyi	bf839651ec	Fix: issues #4950 (#4952)	2023-10-25 22:09:39 -07:00
zehweh	8b39bbbe2f	fix copying .vif file in VolumeCopy (#4943) ... closes #4934 fixes #2633 might fix #3528	2023-10-24 09:54:24 -07:00
chrislu	2bc05f70e7	log full percentage	2023-10-22 12:59:34 -07:00
chrislu	612fd7206e	fix	2023-10-19 08:10:33 -07:00
Konstantin Lebedev	6c82d7ddfc	[s3] do reload s3 static config (#4923) ... * do reload s3 config * print error on reload s3 config * print success msg * Update weed/s3api/s3api_server.go --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-10-18 23:26:49 -07:00
lakeland1990	3df0e11ad3	remove compare to fix #4668 issue (#4924)	2023-10-18 09:06:56 -07:00
chrislu	cbc24c7b24	disconnect from old subscribers	2023-10-13 16:48:14 -07:00
James Hedley	6acb43bbbb	Add optional flags to enable mTLS with verification of client certificate (#4910)	2023-10-13 09:02:24 -07:00
chrislu	b62066d38f	adding DiskType to /dir/status output	2023-10-12 21:39:11 -07:00
chrislu	deb86ff4a6	upgrading grpc	2023-10-12 21:38:34 -07:00
chrislu	de0b969b36	Revert "rename" ... This reverts commit 35b5264ab7.	2023-10-12 20:28:11 -07:00
Konstantin Lebedev	1cac5d983d	fix: disallow file name too long when writing a file (#4881) ... * fix: disallow file name too long when writing a file * bool LongerName to MaxFilenameLength --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-10-12 14:29:55 -07:00
Konstantin Lebedev	edee91ef0e	fix has changes false if deleted result size eq zero (#4909) ... * fix has changes false if deleted result size eq zero https://github.com/seaweedfs/seaweedfs/issues/4171 * fix change respos status 202 to 304 if the file has already been deleted https://github.com/seaweedfs/seaweedfs/issues/4171 * fix delete if set apply changes https://github.com/seaweedfs/seaweedfs/issues/4171 --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-10-12 08:15:20 -07:00
chrislu	35b5264ab7	rename	2023-10-11 21:44:56 -07:00
Amir H. Yeganemehr	3e8604a4c5	add fs.mergeVolumes command into shell (#4907) ... * add fs.compact command into shell * rename fs.compact to fs.mergeVolumes	2023-10-10 12:47:18 -07:00
Konstantin Lebedev	2b3e39397e	fix: skipping checking active volumes with the same number of files at the moment (#4893) ... * fix: skipping checking active volumes with the same number of files at the moment https://github.com/seaweedfs/seaweedfs/issues/4140 * refactor with comments https://github.com/seaweedfs/seaweedfs/issues/4140 * add TestShouldSkipVolume --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-10-09 09:57:26 -07:00
Yuval Yacoby	3fe00996b2	added healthz endpoint to filer (#4899)	2023-10-08 07:03:18 -07:00
Andrew Garrett	082f67bfe9	Fix filer.backup local sink to propagate file mode changes (#4896)	2023-10-06 05:40:20 -07:00
chrislu	0fd7222d65	default to skip if less than 4 nodes	2023-10-05 11:13:48 -07:00
chrislu	7b3c63a8af	handle wrapped errors	2023-10-05 08:15:11 -07:00
chrislu	4e493d596c	minor	2023-10-05 08:13:53 -07:00
Lars Lehtonen	530bdedf76	weed/operation: fix dropped error (#4888)	2023-10-03 10:55:20 -07:00
chrislu	6ebe26a765	Revert "Revert "Revert "Add disk type to prometheus metrics" (#4777)"" ... This reverts commit 567d788928.	2023-10-03 08:28:52 -07:00
chrislu	7540d43ee9	Revert "Revert "fix compilation"" ... This reverts commit f9abfd0b03.	2023-10-03 08:28:02 -07:00
chrislu	249c0e06ef	Revert "fix compilation" ... This reverts commit 451ec6504d.	2023-10-03 08:27:50 -07:00
chrislu	451ec6504d	fix compilation	2023-10-03 08:15:18 -07:00
chrislu	f9abfd0b03	Revert "fix compilation" ... This reverts commit 0483ba3889.	2023-10-03 07:58:11 -07:00
chrislu	0483ba3889	fix compilation	2023-10-02 11:55:50 -07:00
chrislu	567d788928	Revert "Revert "Add disk type to prometheus metrics" (#4777)" ... This reverts commit 9215ba24be.	2023-10-02 11:49:54 -07:00
chrislu	b5070d6280	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-10-02 10:52:40 -07:00
Konstantin Lebedev	dffe00a822	fix: logger place msg (#4880)	2023-10-02 08:29:09 -07:00
chrislu	bf13f3ced7	add subscriber coordinator	2023-10-02 01:02:27 -07:00
chrislu	734178093e	refactor TopicPartition struct	2023-10-02 01:01:45 -07:00
chrislu	2a578b9033	refactor	2023-10-01 11:59:19 -07:00
chrislu	321fda552c	comment	2023-10-01 11:36:27 -07:00
chrislu	b02fdeabff	RetryForever => RetryUntil	2023-10-01 11:33:56 -07:00
chrislu	ebd6f96d35	adjust mq.proto	2023-09-30 21:47:57 -07:00
chrislu	02cb799481	add a test framework for grpc	2023-09-30 21:36:58 -07:00
chrislu	c4e6545197	fix compilation	2023-09-30 21:26:09 -07:00
chrislu	358cba43ef	update proto generated files	2023-09-30 13:19:25 -07:00
chrislu	597ab1e4bf	add design for mq balancer	2023-09-30 13:19:05 -07:00
chrislu	d7c211fe8c	adjust mq proto	2023-09-30 13:18:49 -07:00
chrislu	40888b5612	align 64bit atomic operation ... fix https://github.com/seaweedfs/seaweedfs/issues/4878	2023-09-28 07:58:43 -07:00
Konstantin Lebedev	dd580190b4	fix: avoid deleting one replica without sync (#4875) ... * fix: avoid deleting one replica without sync https://github.com/seaweedfs/seaweedfs/issues/4647 * Update weed/shell/command_volume_fix_replication.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * fix: revert this existing do option to positive --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-09-27 23:12:10 -07:00
Konstantin Lebedev	44906f1f3b	fix: avoid error file name too long when writing a file (#4876)	2023-09-27 05:40:51 -07:00
chrislu	9d589b48e6	rename function	2023-09-26 15:17:33 -07:00
Konstantin Lebedev	df4ded758e	fix: avoid deleting more than one replica (#4873) ... https://github.com/seaweedfs/seaweedfs/issues/4647 Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-09-26 00:20:48 -07:00
chrislu	19505c1cf4	describe a topic	2023-09-25 20:46:00 -07:00
Tobias Gurtzick	78dbac7702	fix(ec): volumes created by foreign collection due to bug in ec balance (#4864) ... * fix(ec): ignore 0 byte data files refers to parts of #4861 Signed-off-by: Tobias Gurtzick <magic@wizardtales.com> * fix(ec): ignore volumes not from the current collection during balance fixes #4861 Signed-off-by: Tobias Gurtzick <magic@wizardtales.com> --------- Signed-off-by: Tobias Gurtzick <magic@wizardtales.com>	2023-09-25 10:35:43 -07:00
chrislu	645ae8c57b	Revert "Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs"" ... This reverts commit 8cb42c39	2023-09-25 09:35:16 -07:00
chrislu	3d07895518	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-25 09:16:35 -07:00
chrislu	a6e35e4f68	fix compilation	2023-09-25 09:16:33 -07:00
Konstantin Lebedev	f8b94cac0e	[s3acl] Step1: move s3account.AccountManager into to iam.S3ApiConfiguration (#4859) ... * move s3account.AccountManager into to iam.S3ApiConfiguration and switch to Interface https://github.com/seaweedfs/seaweedfs/issues/4519 * fix: test bucket acl default and adjust the variable names * fix: s3 api config test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-09-25 08:34:12 -07:00
chrislu	c9177c92e5	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-25 08:20:16 -07:00
chrislu	81e57bd45e	fix test	2023-09-25 08:20:03 -07:00
Lars Lehtonen	28a3a31b27	weed/replication/sub: fix dropped error (#4865)	2023-09-25 07:33:18 -07:00
chrislu	23f334dd13	comment	2023-09-24 23:36:43 -07:00
chrislu	c7e05e4e71	ensure latest stats are reported	2023-09-24 23:34:31 -07:00
chrislu	dff2ce5d2f	read existing broker stats	2023-09-24 23:10:33 -07:00
chrislu	1492bf7552	fix listing topics	2023-09-24 23:05:41 -07:00
chrislu	3cf9b8d621	create topic and report topic	2023-09-24 22:00:43 -07:00
chrislu	b3f94feede	do create topic	2023-09-24 22:00:43 -07:00
chrislu	d74348048a	implement create topic	2023-09-24 22:00:43 -07:00
chrislu	0361c321b4	add CreateTopic API	2023-09-24 22:00:43 -07:00
chrislu	0f8168c0c9	3.57	2023-09-24 21:29:31 -07:00
chrislu	1df4223195	minor	2023-09-24 09:54:57 -07:00
chrislu	10ac25f857	avoid deleting the entry too early	2023-09-23 12:43:00 -07:00
chrislu	c43238b30a	fix waiting condition	2023-09-23 12:28:02 -07:00
chrislu	322af1fc04	fix compilation	2023-09-23 12:13:32 -07:00
chrislu	0414b68a4f	if shared lock, still need to wait for exclusive lock to release	2023-09-23 12:08:23 -07:00
chrislu	9ba4639a27	adjust for better logs	2023-09-23 11:53:44 -07:00
chrislu	186b1f029b	read source uses shared lock	2023-09-23 10:33:44 -07:00
chrislu	2df70ce95f	adjust logs	2023-09-21 23:51:32 -07:00
chrislu	3e23421608	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-21 20:08:30 -07:00
chrislu	49a470bcdf	reduce logs	2023-09-21 20:08:20 -07:00
Konstantin Lebedev	750a0ba1b2	[iam] Replace action read/write to readAcp/writeAcp for handlers with acl (#4858) ... Replace action read/write to readAcp/writeAcp for handlers with acl query https://github.com/seaweedfs/seaweedfs/issues/4519 Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-09-21 14:07:04 -07:00
chrislu	8b2c39f2c0	reduce binary size	2023-09-21 13:22:08 -07:00
chrislu	b388957340	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-21 11:08:30 -07:00
chrislu	31fc165715	mount switch to ordered lock requests	2023-09-21 11:08:26 -07:00
chrislu	7669852241	adjust logs	2023-09-21 11:04:12 -07:00
chrislu	e3b1bacf3f	add lock table to ensure ordered locks	2023-09-21 10:24:34 -07:00
Konstantin Lebedev	a46f873edd	[s3acl] Step 0: Put bucket ACL only responds success if the ACL is private. (#4856) ... * Passing test: test_bucket_acl_default test_bucket_acl_canned_private_to_private https://github.com/seaweedfs/seaweedfs/issues/4519 * Update weed/s3api/s3api_bucket_handlers.go --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-09-21 08:20:05 -07:00
Konstantin Lebedev	d8b424d123	[s3] optimization iam lookup for reducing algorithm complexity (#4857) ... optimization iam lookup for reducing algorithm complexity https://github.com/seaweedfs/seaweedfs/issues/4519 Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-09-21 08:19:11 -07:00
chrislu	411bdda08d	fix compilation	2023-09-19 16:22:41 -07:00
chrislu	d531b7cbbf	merge some previous missing changes	2023-09-19 14:10:54 -07:00
chrislu	b18112ae10	fix compilation bugs during merge	2023-09-19 14:08:17 -07:00
chrislu	6c6655de9d	wip	2023-09-19 14:03:42 -07:00
chrislu	cf28108ea4	fix grpd dial option	2023-09-19 14:02:51 -07:00
chrislu	27af11f1e8	Revert "Revert "Merge branch 'master' into sub"" ... This reverts commit 0bb97709d4.	2023-09-18 18:47:34 -07:00
chrislu	0bb97709d4	Revert "Merge branch 'master' into sub" ... This reverts commit 4d414f54a2, reversing changes made to 4827425146.	2023-09-18 16:13:20 -07:00
chrislu	8cb42c39ad	Revert "Merge branch 'master' of https://github.com/seaweedfs/seaweedfs" ... This reverts commit 2e5aa06026, reversing changes made to 4d414f54a2.	2023-09-18 16:12:50 -07:00
chrislu	2e5aa06026	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-18 16:06:41 -07:00
chrislu	4d414f54a2	Merge branch 'master' into sub	2023-09-18 16:04:59 -07:00
dependabot[bot]	a04bd4d26f	Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 (#4850) ... * Bump github.com/rclone/rclone from 1.63.1 to 1.64.0 Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.63.1 to 1.64.0. - [Release notes](https://github.com/rclone/rclone/releases) - [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md) - [Commits](https://github.com/rclone/rclone/compare/v1.63.1...v1.64.0) --- updated-dependencies: - dependency-name: github.com/rclone/rclone dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * API changes * go mod --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>	2023-09-18 14:43:05 -07:00
chrislu	b64fc44e8d	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-18 09:01:17 -07:00
chrislu	340aea9b34	add broker leader	2023-09-18 09:01:14 -07:00
chrislu	0d390ec882	fix nil	2023-09-18 08:52:35 -07:00
Lars Lehtonen	d83107efe5	fix: weed/storage: dropped error	2023-09-18 08:18:57 -07:00
zemul	91cbaef02b	fix: mount proxyByFiler write error: wrong jwt	2023-09-18 07:44:50 -07:00
chrislu	4e17d399a6	refactor	2023-09-17 00:04:20 -07:00
chrislu	7a49d3541a	avoid nil	2023-09-17 00:04:13 -07:00
chrislu	a65088d739	broker report stats to balancer	2023-09-16 23:00:51 -07:00
chrislu	a1cb49d078	adjust log	2023-09-16 23:00:28 -07:00
chrislu	b590a68e75	revert to non-streaming mode to assign file id ... fix https://github.com/seaweedfs/seaweedfs/issues/4838#issuecomment-1722290685	2023-09-16 16:13:08 -07:00
chrislu	4650c4c65f	Revert "turn on streaming assign file id" ... This reverts commit 733db2bc88.	2023-09-16 16:09:20 -07:00
chrislu	fa62926347	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-09-16 15:06:22 -07:00
chrislu	89a1fd1751	Squashed commit of the following: ... commit 4827425146 Author: chrislu <chris.lu@gmail.com> Date: Sat Sep 16 15:05:38 2023 -0700 balancer works commit 3b50139f68 Author: chrislu <chris.lu@gmail.com> Date: Fri Sep 15 22:22:32 2023 -0700 comments commit 7f685ce7ba Author: chrislu <chris.lu@gmail.com> Date: Fri Sep 15 22:20:05 2023 -0700 adjust APIs commit 436d99443b Author: chrislu <chris.lu@gmail.com> Date: Thu Sep 14 23:49:05 2023 -0700 receive broker stats commit b771fefa37 Merge: 0a851ec00 890881037 Author: chrislu <chris.lu@gmail.com> Date: Wed Sep 13 00:03:47 2023 -0700 Merge branch 'master' into sub commit 0a851ec00b Author: chrislu <chris.lu@gmail.com> Date: Sun Sep 10 22:01:25 2023 -0700 Create balancer.go commit 39941edc0b Author: chrislu <chris.lu@gmail.com> Date: Thu Sep 7 23:55:19 2023 -0700 add publisher shutdown commit 875f562779 Author: chrislu <chris.lu@gmail.com> Date: Wed Sep 6 23:16:41 2023 -0700 server side send response at least once per second commit 984b6c54cf Author: chrislu <chris.lu@gmail.com> Date: Wed Sep 6 23:15:29 2023 -0700 ack interval 128 commit 2492a45499 Author: chrislu <chris.lu@gmail.com> Date: Wed Sep 6 22:39:46 2023 -0700 ack interval commit ba67e6ca29 Author: chrislu <chris.lu@gmail.com> Date: Mon Sep 4 21:43:50 2023 -0700 api for sub commit 9e4f985698 Author: chrislu <chris.lu@gmail.com> Date: Mon Sep 4 21:43:30 2023 -0700 publish, benchmark commit cb470d44df Author: chrislu <chris.lu@gmail.com> Date: Fri Sep 1 00:36:51 2023 -0700 can pub and sub commit 1eb2da46d5 Author: chrislu <chris.lu@gmail.com> Date: Mon Aug 28 09:02:12 2023 -0700 connect and publish commit 504ae8383a Author: chrislu <chris.lu@gmail.com> Date: Mon Aug 28 09:01:25 2023 -0700 protoc version commit dbcba75271 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 27 18:59:04 2023 -0700 rename to lookup commit c9caf33119 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 27 18:33:46 2023 -0700 move functions commit 4d6c18d86f Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 27 17:50:59 2023 -0700 pub sub initial tests commit 4eb8e8624d Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 27 13:14:39 2023 -0700 rename commit 1990456670 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 27 13:13:14 2023 -0700 sub commit 905911853d Author: chrislu <chris.lu@gmail.com> Date: Sat Aug 26 13:39:21 2023 -0700 adjust proto	2023-09-16 15:06:16 -07:00
chrislu	4827425146	balancer works	2023-09-16 15:05:38 -07:00
chrislu	3b50139f68	comments	2023-09-15 22:22:32 -07:00
chrislu	7f685ce7ba	adjust APIs	2023-09-15 22:20:05 -07:00
chrislu	436d99443b	receive broker stats	2023-09-14 23:49:05 -07:00
wang wusong	9e5b1eef9b	avoid passe lock by value	2023-09-14 06:51:12 -07:00
chrislu	b771fefa37	Merge branch 'master' into sub	2023-09-13 00:03:47 -07:00
annluda	8908810376	Fix s3 api object list with params max-keys and prefix	2023-09-11 21:35:15 -07:00
chrislu	5407b86766	3.56	2023-09-10 22:05:41 -07:00
chrislu	07c2d75ac7	ensure single threaded access	2023-09-10 22:03:44 -07:00
chrislu	0a851ec00b	Create balancer.go	2023-09-10 22:01:25 -07:00
chrislu	39941edc0b	add publisher shutdown	2023-09-07 23:55:19 -07:00
chrislu	875f562779	server side send response at least once per second	2023-09-06 23:16:41 -07:00
chrislu	984b6c54cf	ack interval 128	2023-09-06 23:15:29 -07:00
chrislu	2492a45499	ack interval	2023-09-06 22:39:46 -07:00
Lars Lehtonen	f24c7e803f	weed/storage/erasure_coding: Close() after error handling	2023-09-06 07:35:54 -07:00
Lars Lehtonen	d762b12e11	weed/storage/erasure_coding: fix dropped test error	2023-09-06 07:35:54 -07:00
Lars Lehtonen	a879e1bbb5	weed/storage/erasure_coding: remove unused err from encodeDatFile() signature	2023-09-06 07:35:54 -07:00
Patrick Schmidt	98dcec0ee2	Clean up old signature hash pools	2023-09-05 10:33:27 -07:00
Patrick Schmidt	cdd817edf9	Improve S3 request signing performance ... This change is caching HMAC hashers for repeated use in subsequent requests and chunks, so they don't have to be initialized from scratch every time. On my local computer this gives me ~5-6 times faster signature calculation and ~5-6.5% more throughput in S3 requests. The smaller the payload the better the throughput gets.	2023-09-05 10:33:27 -07:00
chrislu	ba67e6ca29	api for sub	2023-09-04 21:43:50 -07:00
chrislu	9e4f985698	publish, benchmark	2023-09-04 21:43:30 -07:00
Lars Lehtonen	e6385e5e8b	weed/util: fix dropped errors (#4803)	2023-09-01 13:10:45 -07:00
chrislu	cb470d44df	can pub and sub	2023-09-01 00:36:51 -07:00
chrislu	1eb2da46d5	connect and publish	2023-08-28 09:02:12 -07:00
chrislu	504ae8383a	protoc version	2023-08-28 09:01:25 -07:00
chrislu	dbcba75271	rename to lookup	2023-08-27 18:59:04 -07:00
chrislu	c9caf33119	move functions	2023-08-27 18:33:46 -07:00
chrislu	4d6c18d86f	pub sub initial tests	2023-08-27 17:50:59 -07:00
chrislu	4eb8e8624d	rename	2023-08-27 13:14:39 -07:00
chrislu	1990456670	sub	2023-08-27 13:13:14 -07:00
chrislu	905911853d	adjust proto	2023-08-26 13:39:21 -07:00
Nico D'Cotta	796b7508f3	Implement SRV lookups for filer (#4767)	2023-08-24 07:08:56 -07:00
chrislu	733db2bc88	turn on streaming assign file id	2023-08-23 00:34:21 -07:00
chrislu	99f037b958	streaming assign file ids	2023-08-23 00:31:33 -07:00
chrislu	94b7e2a37c	add stream assign server side implementation	2023-08-22 09:59:04 -07:00
chrislu	ccedad5196	refactor files	2023-08-22 09:54:06 -07:00
chrislu	010c5e91e3	add stream assign proto	2023-08-22 09:53:54 -07:00
chrislu	a1149f060b	upgrade jwt	2023-08-22 08:26:31 -07:00
chrislu	c49caff5a1	change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims	2023-08-21 23:42:39 -07:00
chrislu	c45665eb97	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-08-21 09:47:36 -07:00
SmsS4	f61490966f	Add time to first byte metric for s3 (#4768) ... * Add time to first byte metric for s3 * Change second to millisecond	2023-08-21 00:42:39 -07:00
chrislu	01d70c21f3	Squashed commit of the following: ... commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:52:19 2023 -0700 fix compilation commit e77ad33b7ca0423138fbae26a4433b60923a9588 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:46:44 2023 -0700 pub commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 13:27:39 2023 -0700 fix generic type commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d Merge: 30f942580 16e3f2d52 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 12:47:14 2023 -0700 Merge branch 'master' into pubsub commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 11:10:58 2023 -0700 wip commit f8b00980bc2f3879bb43decffd9a08d842f196f2 Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:35 2023 -0700 add design document commit 08d2bebe42a26ebc39f1542f54d99e73620727dd Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:06 2023 -0700 minor commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:13:49 2023 -0700 rename	2023-08-20 22:53:05 -07:00
chrislu	16e3f2d528	fix log buffer test	2023-08-20 12:46:15 -07:00
chrislu	3e9c32a3f0	refactoring to use atomic bool ... fix compilation	2023-08-20 12:13:06 -07:00
SmsS4	1e72579ac7	Fix s3.clean.uploads (#4766)	2023-08-19 06:30:18 -07:00
chrislu	6792db21dc	ErrInvalidPart when completed parts has extra parts ... fix https://github.com/seaweedfs/seaweedfs/discussions/4746	2023-08-17 00:51:42 -07:00
chrislu	f5c4b642bb	re-ordering	2023-08-16 23:54:23 -07:00
chrislu	3852307e94	renaming	2023-08-16 23:47:43 -07:00
chrislu	6c7fa567d4	add separate cache directory for write buffers	2023-08-16 23:39:21 -07:00
SmsS4	ea1b259c12	Update weed fix description (#4745)	2023-08-13 03:50:53 -07:00
Nikita Mochalov	4b1ba7f5b2	Save disk space metrics immediately (#4740)	2023-08-10 06:52:32 -07:00
Nikita Mochalov	e6a49dc533	Fix resource leaks (#4737) ... * Fix division by zero * Fix file handle leak * Fix file handle leak * Fix file handle leak * Fix goroutine leak	2023-08-09 15:30:36 -07:00
chrislu	3365468d0d	added an error message	2023-08-08 20:35:21 -07:00
wusong	3ec977cd6c	minor (#4723)	2023-08-01 02:34:59 -07:00
chrislu	c09100f928	fix building mongo index	2023-07-24 11:09:51 -07:00
chrislu	7eafa3420b	3.55	2023-07-23 23:11:37 -07:00
SmoothDenis	51bcc219ea	s3api should return 500 code from filer (#4699)	2023-07-22 07:22:38 -07:00
wusong	33b9192536	Fix large file chunks deleted by mistake (#4678)	2023-07-17 07:52:14 -07:00
chrislu	358b3a4894	3.54	2023-07-16 22:47:57 -07:00
chrislu	6f588b5b18	fix refactoring mistake ... fix https://github.com/seaweedfs/seaweedfs/issues/4639	2023-07-16 11:22:48 -07:00
mervynzhang	df400e6c71	Concurrency works better (#4663) ... Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>	2023-07-12 23:04:54 -07:00
wusong	0cb9ddd8ec	Fix data loss: add lock for metacache (#4664) ... Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-07-11 22:23:32 -07:00
Henco Appel	89542db2d0	Fix S3 API multipart upload on Windows (#4666) ... The file paths were being having '/' changed to '\' by filepath.Dir() resulting in a file being created with '\' separators, but when trying to read the files, the same wasn't happening. Co-authored-by: Hendrik Appel <happel@europe.altair.com>	2023-07-11 09:05:14 -07:00
Henco Appel	680c2e76f1	Fix the segfault from not setting s3 https port (#4667)	2023-07-11 08:03:20 -07:00
Konstantin Lebedev	2c076ebd96	avoid logging nil error (#4659) ... Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-07-10 23:38:52 -07:00
LHHDZ	3730772e05	refresh max volume count after adjust (#4662) ... Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-07-10 23:37:33 -07:00
chrislu	c278bac263	avoid nil needle map ... fix https://github.com/seaweedfs/seaweedfs/issues/4640	2023-07-07 22:16:58 -07:00
chrislu	0606b59a3e	mount: create mount root on filer ... fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/127	2023-07-07 09:30:32 -07:00
chrislu	14546f0ad8	minor	2023-07-06 09:12:28 -07:00
chrislu	ab30f489c3	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-07-06 00:33:02 -07:00
chrislu	31b2751aff	clone volume locations in case they are changed ... fix https://github.com/seaweedfs/seaweedfs/issues/4642	2023-07-06 00:32:58 -07:00
wusong	61553beba2	Fix DataBackend nil pointer (#4641)	2023-07-04 23:22:10 -07:00
chrislu	a315490f7d	proxy to master uses http address ... fix https://github.com/seaweedfs/seaweedfs/issues/4607	2023-07-04 11:45:21 -07:00
chrislu	2aa59ab37c	fix copying level db files during commi ... fix https://github.com/seaweedfs/seaweedfs/issues/4635	2023-07-04 11:28:12 -07:00
wusong	c44a6552d2	Fix buff use (#4628)	2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr	f0d3c874b6	Unix Socket listener for S3 server (#4621) ... Add localSocket option to s3 server	2023-06-26 16:22:45 -07:00
chrislu	2c4c2f0994	3.53	2023-06-25 22:51:14 -07:00
chrislu	f193942adc	lock on append metadata operation	2023-06-25 22:07:23 -07:00
chrislu	61c42f9991	adjust lock APIs	2023-06-25 20:30:20 -07:00
chrislu	aafb874ad2	lock until cancelled	2023-06-25 19:31:25 -07:00
chrislu	c4f44d56f7	add distributed lock client	2023-06-25 17:38:34 -07:00
chrislu	f8aa5ea844	adjust filer.proto	2023-06-25 17:38:19 -07:00
chrislu	13a8593ab9	adjust errors	2023-06-25 17:37:54 -07:00
chrislu	06471dac9d	init lock ring	2023-06-25 15:28:16 -07:00
chrislu	5149b3d07b	filer can proxy to peer filer holding the lock	2023-06-25 15:28:01 -07:00
chrislu	868f7875d7	refactor	2023-06-25 14:30:58 -07:00
chrislu	26eff062c8	refactor	2023-06-25 14:17:58 -07:00
chrislu	464a71a373	add distributed lock manager	2023-06-25 14:14:40 -07:00
chrislu	ee4f7cd636	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-06-25 00:58:24 -07:00
chrislu	3fd659df2a	add distributed lock manager	2023-06-25 00:58:21 -07:00
Damiano Albani	0d9ba8c612	Support templating name of files stored via Rclone backend (#4606)	2023-06-24 07:08:24 -07:00
chrislu	a82c44972b	refactor	2023-06-20 00:12:59 -07:00
chrislu	14a19170e2	refactor	2023-06-19 23:51:04 -07:00
chrislu	84ced2abf5	add LockRing	2023-06-19 23:48:33 -07:00
chrislu	fe1716ccdf	remove deprecated tests	2023-06-19 18:43:51 -07:00
chrislu	1ce725728d	minor	2023-06-19 18:43:41 -07:00
chrislu	11549706ef	refactor	2023-06-19 18:21:07 -07:00
chrislu	8ec1bc2c99	remove unused cluster node leader	2023-06-19 18:19:13 -07:00
Bai Jie	3b88ab42aa	remove duplicate fileCount query (#4588)	2023-06-18 00:14:14 -07:00
Bai Jie	44b9d72ef0	doIsEmpty() return error if v.DataBackend is nil (#4587)	2023-06-18 00:13:40 -07:00
chrislu	94fbf02ea2	generate jwt token when looking up needle id ... fix https://github.com/seaweedfs/seaweedfs/issues/4577	2023-06-17 00:11:23 -07:00
柏杰	0b0fb9b9e4	avoid data race read volume.IsEmpty (#4574) ... * avoid data race read volume.IsEmpty - avoid phantom read isEmpty for onlyEmpty - use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope * add Destroy(onlyEmpty: true) test * add Destroy(onlyEmpty: false) test * remove unused `IsEmpty()` * change literal `8` to `SuperBlockSize`	2023-06-14 14:39:58 -07:00
Konstantin Lebedev	1e22d5caf2	fix get file stats for IsEmpty (#4576)	2023-06-14 01:43:30 -07:00
Konstantin Lebedev	4dd890d4a2	optional https port for s3 (#4482) ... Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-13 23:58:49 -07:00
chrislu	3fbf4f6189	Revert "Initial patch for redis version 7 support (#4572)" ... This reverts commit a25bca0692.	2023-06-12 22:27:38 -07:00
Konstantin Lebedev	4527ead295	fix from comment delete volume is empty (#4573) ... * fix from coments https://github.com/seaweedfs/seaweedfs/pull/4561 * fix tests --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-12 22:22:46 -07:00
Aaron Gipson	a25bca0692	Initial patch for redis version 7 support (#4572)	2023-06-12 13:30:03 -07:00
Konstantin Lebedev	25535e9c36	Delete volume is empty (#4561) ... * use onlyEmpty for deleteVolume https://github.com/seaweedfs/seaweedfs/issues/4559 * fix IsEmpty * fix test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-12 10:42:44 -07:00
柏杰	68feffa4e6	avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560)	2023-06-10 23:15:27 -07:00
Konstantin Lebedev	5ee04d20fa	Healthz check for deadlocks (#4558)	2023-06-09 09:42:48 -07:00
wusong	d6af3a8eb6	fix 404 not found (#4536) ... Signed-off-by: wang wusong <wangwusong@virtaitech.com> Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-06 09:14:49 -07:00
Konstantin Lebedev	5caff2c0f0	fix: deletes newly created volumes (#4551) ... https://github.com/seaweedfs/seaweedfs/issues/4531 Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-06 09:08:27 -07:00
wusong	9bdbf9c880	revert #4491 (#4550) ... Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-06 00:17:51 -07:00
chrislu	0bffb3770f	support TTL when creating folders ... related to https://github.com/seaweedfs/seaweedfs/discussions/4549	2023-06-05 23:29:25 -07:00
wusong	26f15d0079	Fix no more writable volumes by delay judgment (#4548) ... * fix nomore writables volumes while disk free space is sufficient by time delay * reset --------- Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-05 10:17:21 -07:00
chrislu	fb4b61036c	3.52	2023-06-04 22:34:26 -07:00
Konstantin Lebedev	a0931be0c0	S3 TLS credentials Refreshing (#4506) ... * S3 TLS credentials Refreshing * fix: logging --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-04 14:27:56 -07:00
wusong	5aec6da8a3	fix deadlock for filer upload (#4527) ... Signed-off-by: wang wusong <wangwusong@virtaitech.com> Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-03 13:38:27 -07:00
chrislu	e23f3d6eca	set df.File to nil after it is closed ... possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530	2023-06-02 17:16:45 -07:00
chrislu	ca7cc61319	reduce the window size between unregistering a volume server and creating volumes on that server ... fix https://github.com/seaweedfs/seaweedfs/issues/4467	2023-05-30 08:20:00 -07:00
chrislu	68c3cf767c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-05-29 16:45:28 -07:00
chrislu	d1d1757520	adjust error message	2023-05-29 16:45:25 -07:00
wusong	e7f4418855	add return for error handle (#4502)	2023-05-24 07:45:06 -07:00
chrislu	4310e1fac4	3.51	2023-05-21 23:43:39 -07:00
wusong	8fffe3e822	fix no more writables volumes while disk free space is sufficient (#4491) ... Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-05-21 22:18:50 -07:00
mervynzhang	1ebb549f77	support swift (#4480)	2023-05-19 06:39:25 -07:00
Mesar Hameed	a90b777ff4	Feat: etcd filer store keys should have customizable prefix (#4484) ... An etcd cluster is not necessarily only dedicated to seaweedfs. This security enhancement adds a customizable key_prefix option to the etcd filer store. This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the key_prefix, instead of all keys on the etcd cluster.	2023-05-18 23:08:56 -07:00
SmsS4	17e91d2917	Use filerGroup for s3 buckets collection prefix (#4465) ... * Use filerGroup for s3 buckets collection prefix * Fix templates * Remove flags * Remove s3CollectionPrefix	2023-05-16 09:39:43 -07:00
chrislu	b7f011f777	fix test	2023-05-15 17:46:56 -07:00
Mesar Hameed	65484e80ef	Feat: support username/password authentication for etcd filer store s… (#4477) ... Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262 Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>	2023-05-15 17:45:14 -07:00
chrislu	9204ee2d2d	3.50	2023-05-11 08:03:11 -07:00
chrislu	d999f1f0e2	update raft version ... fix #4460	2023-05-09 22:54:23 -07:00
chrislu	25b237b83c	reset wait time on a successful retry	2023-05-07 23:39:33 -07:00
chrislu	59f55c1a66	3.49	2023-05-07 14:00:19 -07:00
chrislu	b2a3860290	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-05-07 13:33:47 -07:00
chrislu	4511edc871	update raft	2023-05-07 13:33:44 -07:00
themarkchen	7592d013fe	fix shell volume.balance bug (#4447)	2023-05-04 07:35:37 -07:00
chrislu	42766bccc8	3.48	2023-04-30 22:20:18 -07:00
zemul	e9fda774f4	[Filer] post add param:saveInside (#4434) ... * fix:mount deadlock * feat: filer http upload to metadata * feat: /etc save inside --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-29 08:31:05 -07:00
wusong	4867aa03ab	Fix: http rename move dir to subdir (#4432) ... Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-04-26 20:54:36 -07:00
Konstantin Lebedev	d75a7b7f62	allow deleting only older empty dir without recursion (#4430)	2023-04-25 08:31:14 -07:00
zemul	f352616b7d	Volume range read use sync.pool (#4422)	2023-04-24 07:00:48 -07:00
chrislu	18686b7375	3.47	2023-04-23 22:45:41 -07:00
zemul	68f1cc34a5	[mount]Fix copy file range (#4407) ... * fix:mount deadlock * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-23 22:34:38 -07:00
chrislu	e1b92bc701	reset local file content if remote file is updated ... fix https://github.com/seaweedfs/seaweedfs/issues/4289	2023-04-22 22:32:21 -07:00
chrislu	ff7b6d779e	avoid overwriting variables ... fix https://github.com/seaweedfs/seaweedfs/issues/4365	2023-04-22 11:49:59 -07:00
wusong	2e240704ab	Writables inconsistency (#4417) ... fix: inconsistent read and write permissions between master and volume server Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-21 00:14:41 -07:00
wusong	19245dde50	mount: add retry for read only case (#4416) ... * mount: add retry for read only case Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * add new util retry function for mount Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * change error list param Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> --------- Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-21 00:09:31 -07:00
SmsS4	4131874fa8	use env for flags wtih dot in name (#4415)	2023-04-20 09:33:27 -07:00
Damiano Albani	9f55c7c90e	Rclone storage backend (#4402) ... * Add Rclone storage backend * Support templating the name of files stored via Rclone * Enable Rclone accounting * Remove redundant type conversion * Provide progress information for Rclone download/upload operations * Log error when Rclone can't instantiate filesystem * Remove filename templating functionality for Rclone storage To (maybe) be later reintroduced as a generic functionality for all storage backends. * Remove S3 specific check * Move Rclone config initialisation to init() method	2023-04-18 14:12:12 -07:00
wusong	a408b46d95	compilation fail (#4414) ... Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-18 00:36:55 -07:00
chrislu	0a22eea55d	collect ec shard from multiple locations ... fix https://github.com/seaweedfs/seaweedfs/issues/4365	2023-04-17 22:56:21 -07:00
chrislu	ae9388723f	adjust error message	2023-04-17 22:01:39 -07:00
chrislu	8ecdf958ab	3.46	2023-04-16 21:57:43 -07:00
Chengyu Liu	0f653d2153	update fuse client log which should output the fh of FileHandle. (#4404)	2023-04-15 22:15:16 -07:00
zemul	0122e022ea	Mount concurrent read (#4400) ... * fix:mount deadlock * feat: concurrent read * fix * Remove useless code * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-13 22:32:45 -07:00
Konstantin Lebedev	5614ad0000	fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399) ... * fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter * fix list with delimiter and start token --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-12 16:53:49 -07:00
Konstantin Lebedev	88b2bbadd1	try fix s3test test_bucket_listv2_delimiter_prefix (#4396)	2023-04-12 08:47:20 -07:00
Konstantin Lebedev	095cdb0c7f	fix test prefix is a directory (#4393) ... Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-12 00:08:13 -07:00
Konstantin Lebedev	44ad07276c	s3 fix get list of dir object key with slash suffix (#4391) ... * s3 fix get list of dir object key with slash suffix https://github.com/seaweedfs/seaweedfs/issues/3086 * list only entry dir eq prefix --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-11 23:52:35 -07:00
Konstantin Lebedev	130bc3e668	s3 fix get fake dir object key (#4390)	2023-04-11 07:36:22 -07:00
Matt	da1d3b5a33	adjusted regex to be from 1 to 255 for the value (#4377) ... * compatibility patch for csi driver * added namespace to all component parameters * added namespace to all component parameters * dereference in range * added namespace to values.yml defaults * added namespace to s3 component * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * push on all tags * push on all tags * push on all tags * push on all tags * push on all tags * changed helm directory structure * update charts location * fixed dereference * updated permissions * updated permissions * match current action schema * added helm chart liniting CI * modified chart list changed * fixed nodejs warning * standardized a few defaults * added chart dirs * lowerd period seconds for volume startup to test chart lint changes * lowerd period seconds for volume startup to test chart lint changes * test * changed back * adjustment * debug ls statment * removed change detection * remvoed change detection * always lint the charts * added missing ) * fixed typo * added spaces in front of all comments * pdated values.yaml * pdated values.yaml * don't validate maintainers for now * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * - adds helm chart lint on changes - adds test helm chart install on k8s * updated helm chart readme.md * added artifact hub * added artifact hub * added ttl validation * adjusted regex to be from 1 to 255 for the value * added better error message * fixed regex	2023-04-10 12:24:38 -07:00
Matt	2d213d87e1	File Path Configuration TTL Validation (#4376) ... * compatibility patch for csi driver * added namespace to all component parameters * added namespace to all component parameters * dereference in range * added namespace to values.yml defaults * added namespace to s3 component * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * push on all tags * push on all tags * push on all tags * push on all tags * push on all tags * changed helm directory structure * update charts location * fixed dereference * updated permissions * updated permissions * match current action schema * added helm chart liniting CI * modified chart list changed * fixed nodejs warning * standardized a few defaults * added chart dirs * lowerd period seconds for volume startup to test chart lint changes * lowerd period seconds for volume startup to test chart lint changes * test * changed back * adjustment * debug ls statment * removed change detection * remvoed change detection * always lint the charts * added missing ) * fixed typo * added spaces in front of all comments * pdated values.yaml * pdated values.yaml * don't validate maintainers for now * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * - adds helm chart lint on changes - adds test helm chart install on k8s * updated helm chart readme.md * added artifact hub * added artifact hub * added ttl validation	2023-04-04 22:07:17 -07:00
chrislu	efef6e94bf	use UPSERT for postgres style databases	2023-04-04 11:52:36 -07:00
chrislu	422bfaed69	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-04-02 22:53:02 -07:00
chrislu	b9ab65dba5	3.45	2023-04-02 22:52:58 -07:00
wusong	2b82d0f871	style: clerical error (#4361)	2023-03-31 07:36:21 -07:00
LHHDZ	8e80ce9476	fix key corrupt when fs.configure copy path trie (#4353) ... Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-03-29 11:29:44 -07:00
chrislu	3cc7b90414	print wait in progress	2023-03-26 12:22:48 -07:00
chrislu	f1f14e28bc	adjust name	2023-03-26 12:17:23 -07:00
chrislu	d54798c3ae	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-03-25 22:03:00 -07:00
chrislu	6bfb91b539	update entry Attributes.Mtime = time.Now().Unix() ... fix https://github.com/seaweedfs/seaweedfs/issues/4337	2023-03-25 22:02:57 -07:00
renweijun	bbd461b2c6	weed mount default EnableACL，Support chmod chown (#4335)	2023-03-25 04:47:53 -07:00
chrislu	5db9fcccd4	refactoring	2023-03-21 23:01:49 -07:00
Konstantin Lebedev	de4545c28b	add validate config for raft (#4332)	2023-03-21 06:36:13 -07:00
chrislu	adb90bd252	avoid lower casing the command ... fix https://github.com/seaweedfs/seaweedfs/pull/4321	2023-03-19 21:20:46 -07:00
chrislu	5b43c4bb98	3.44	2023-03-19 20:27:55 -07:00
chrislu	50dc2fe96b	cleaning variables	2023-03-19 18:48:40 -07:00
Konstantin Lebedev	29d1312625	Fs.verify.concurrency (#4293) ... * init * refactor * refactor print msg * param just concurrency * fix race * next fix race * reset variables before each run * clean --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-03-19 18:38:57 -07:00
chrislu	784daea1fa	fix volume not found if marked as read only ... fix https://github.com/seaweedfs/seaweedfs/issues/4088	2023-03-19 18:30:13 -07:00
Stewart Miles	264be0d2d4	Retry until a leader is selected. (#4318) ... Fixes regression introduced in https://github.com/seaweedfs/seaweedfs/pull/4313 Related to #4307	2023-03-16 20:50:38 -07:00
LHHDZ	d695119073	The filer does not set defaultReplication to the defaultReplication o… (#4315) ... The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master) Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-03-15 22:10:39 -07:00
Stewart Miles	57ab1f8516	Use exponential backoff to query leader. (#4313) ... `topology.Leader()` was using a backoff that typically resulted in at least a 5s delay when initially starting a master and raft server. This changes the backoff algorithm to use exponential backoff starting with 100ms and waiting up to 20s for leader selection. Related to #4307	2023-03-15 17:49:46 -07:00
Stewart Miles	dd71f54c6b	Fix -raftHashicorp and -raftBootstrap flag propagation. (#4309) ... `weed server` was not correctly propagating `-master.raftHashicorp` and `-master.raftBootstrap` flags when starting the master server. Related to #4307	2023-03-15 13:03:20 -07:00
Patrick Schmidt	71b33faef0	Skip parent directory creation in mount (#4310) ... A POSIX filesystem does not implicitly create parent directories when they do not exist. Directories must be explicitly created and permissions be set. This also fixes a bug where asynchronous operations would create a file in the filer before the parent directory was created. If the file was a symlink or another special type of file the directory would inherit that type and become unusable in the mounted FS.	2023-03-15 12:37:25 -07:00
chrislu	f5854d13df	fix test	2023-03-14 22:05:16 -07:00
chrislu	0454bb2d88	fix test ... when size is zero, the "LastModified" is not written	2023-03-14 22:02:49 -07:00
chrislu	2268d38263	fix naming ... fix https://github.com/seaweedfs/seaweedfs/issues/4305	2023-03-14 08:38:02 -07:00
chrislu	98bc93cf31	fix tests	2023-03-13 10:00:27 -07:00
Konstantin Lebedev	e17429223e	shell script unclean variables (#4298)	2023-03-13 07:16:31 -07:00
Thomas Anderson	bd70683871	fix(weed/topology/node.go): typo in error message (#4292)	2023-03-08 23:58:03 -08:00
Rohit Chormale	59706c89fb	#4270 set http status code to 409 if dir already exists (#4287)	2023-03-08 07:45:08 -08:00
chrislu	7111c08289	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-03-08 01:48:30 -08:00
chrislu	c589e9837c	fix tests	2023-03-08 01:48:28 -08:00
Kevin Liu	244385bf0d	Fix binding metrics to ipv6 (#4286) ... * Fix binding metrics to ipv6 * Update weed/stats/metrics.go --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-03-08 01:04:02 -08:00
zemul	bd6f2716d3	fix_read_tomb_needle_meta (#4263) ... Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-03-02 08:26:55 -08:00
zemul	0bf56298d5	fix chunk.ModifiedTsNs (#4264) ... * fix * fix mtime s > ns --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-03-02 08:24:36 -08:00
chrislu	8241c9a829	adjust ui pagination size ... fix https://github.com/seaweedfs/seaweedfs/issues/4258	2023-02-28 08:31:26 -08:00
lfhy	1976ca9160	add -disk to filer command (#4247) ... * add -disk to filer command * add diskType to filer.grpc * use filer.disk when filerWebDavOptions.disk is empty * add filer.disk to weed server command. --------- Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>	2023-02-25 09:48:59 -08:00
chrislu	214b7cd286	volume.fix.replication: adjust the retry checking times	2023-02-22 10:47:52 -08:00
LHHDZ	db5515eada	Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists (#4240) ... Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-02-21 23:08:52 -08:00
Zachary Walters	ef2f741823	Updated the deprecated ioutil dependency (#4239)	2023-02-21 19:47:33 -08:00
chrislu	3227e4175e	3.43	2023-02-20 16:56:05 -08:00
chrislu	9bb0a9e306	clean comments	2023-02-17 01:31:44 -08:00
chrislu	545d5d0cc3	fix for io.ReaderAt used in filer.remote.sync ... fix https://github.com/seaweedfs/seaweedfs/issues/4194	2023-02-17 01:30:55 -08:00
wusong	79af6b41d4	[mount] Fix read file-data in entry.content (#4207)	2023-02-15 07:18:56 -08:00
chrislu	5869945f16	avoid infinite loop ... fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904	2023-02-10 13:05:04 -08:00
chrislu	e037c71ec3	adjust text	2023-02-10 13:04:29 -08:00
chrislu	67b8c2853a	add line return	2023-02-10 12:53:43 -08:00
chrislu	302155a6fa	fix compilation	2023-02-10 12:32:59 -08:00
chrislu	5520b81551	add some error messages for args ... related to https://github.com/seaweedfs/seaweedfs/issues/4115	2023-02-10 12:30:53 -08:00
chrislu	dc4ed2cd9b	do not move cloud tier volumes ... fix https://github.com/seaweedfs/seaweedfs/issues/4195	2023-02-09 22:17:54 -08:00
chrislu	31bb91583f	fix bug when vid not found ... fix https://github.com/seaweedfs/seaweedfs/issues/4193	2023-02-09 17:30:44 -08:00
chrislu	79d68a81fd	text change	2023-02-07 15:00:41 -08:00
chrislu	999a126a3f	typo	2023-02-07 14:59:47 -08:00
chrislu	1c2364936a	detect invalid bucket name ... fix https://github.com/seaweedfs/seaweedfs/issues/4143	2023-02-07 14:57:29 -08:00
chrislu	5de93fe442	refactoring	2023-02-07 14:45:20 -08:00
Eric Yang	5083429704	ADHOC: s3 bucket name verification (#4189) ... * ADHOC: s3 bucket name verification * add test	2023-02-07 14:37:29 -08:00
Neo	d5f77706a8	volume:fix return if Replicate write error (#4188)	2023-02-06 22:34:56 -08:00
chrislu	8821d6b161	3.42	2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery	9bd422d2c9	adding support for B2 region (#4177) ... Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>	2023-02-05 21:24:21 -08:00
chrislu	84e9934bf9	fix filer.remote.sync on a S3 cloud mount ... fix https://github.com/seaweedfs/seaweedfs/issues/4175	2023-02-01 20:44:00 -08:00
chrislu	0932437a1c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-02-01 02:38:09 -08:00
chrislu	c47f75eb5b	fix null content type caused exception in hdfs client ... fix https://github.com/seaweedfs/seaweedfs/issues/4170	2023-02-01 02:38:05 -08:00
wusong	fe573f26a8	memchunk save content twice (#4174)	2023-01-31 19:36:59 -08:00
Neo	b9b613a78e	filter system log dir does not make subscribe event (#4172)	2023-01-31 18:56:11 -08:00
wusong	665917fdf1	[mount]: delete redundant code (#4173)	2023-01-31 18:54:46 -08:00
a	0d085ec175	make directory lister trigger if mime type is httpd/unix-directory (#4169)	2023-01-31 06:54:40 -08:00
chrislu	4d71af87f3	3.41	2023-01-29 22:56:01 -08:00
chrislu	ef70415c70	skipping manifest chunk volume id change	2023-01-29 22:55:06 -08:00
chrislu	6e8dbf8868	add fs.meta.changeVolumeId	2023-01-29 22:51:58 -08:00
wusong	de081c0d64	[mount] fix metacache update (#4161)	2023-01-29 07:55:27 -08:00
Guo Lei	b63d1e9c6b	fix index (#4158)	2023-01-28 06:55:27 -08:00
chrislu	81fdf3651b	grpc connection to filer add sw-client-id header	2023-01-20 01:48:12 -08:00
chrislu	bfe5d910c6	use one readerCache for the whole file	2023-01-16 22:43:02 -08:00
chrislu	4614e85efa	adjust help message	2023-01-15 21:28:36 -08:00
chrislu	c24678153c	use memory chunks only for sequential writes	2023-01-15 21:10:08 -08:00
chrislu	2885ba0e50	3.40	2023-01-15 20:12:05 -08:00
chrislu	12f8251d32	avoid index out of slice range	2023-01-14 13:31:22 -08:00
chrislu	313fd17ae4	allow random writes for mem chunk	2023-01-13 20:32:18 -08:00
Chris Lu	1cd2e64aac	merge chunks during upload (#4130) ... * merge chunks during upload * fix test	2023-01-12 23:07:36 -08:00
chrislu	2452f3b2f7	sort chunks by offset	2023-01-12 20:29:07 -08:00
chrislu	361d97941b	mysql: Missed escaping column name ... fix https://github.com/seaweedfs/seaweedfs/issues/4129	2023-01-11 08:46:56 -08:00
chrislu	efcffe4f77	no need to set chunks during flush	2023-01-10 10:01:52 -08:00
chrislu	85f1b70207	reduce group size from 256MB to 64MB	2023-01-10 01:35:44 -08:00
chrislu	86657ed3a8	avoid repeatedly adding manifest chunks	2023-01-10 01:35:10 -08:00
chrislu	28fe578944	avoid possible nil entry	2023-01-10 01:07:34 -08:00
chrislu	47fa5ef979	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-01-10 00:52:50 -08:00
chrislu	9e0d526163	fix compilation	2023-01-10 00:52:05 -08:00
chrislu	7f49c59c14	cluster.ps add filer meta sync progress	2023-01-10 00:51:25 -08:00
chrislu	340e7c3a2e	chunk group remove manifestChunks and reset sections in SetChunks()	2023-01-10 00:49:31 -08:00
chrislu	e650c8397d	ensure section.chunks is always garbage free	2023-01-10 00:48:01 -08:00
chrislu	75bdd4a0d1	refactor	2023-01-10 00:46:46 -08:00
Guo Lei	83e29faa55	use time.NewTicker instead of time.Tick. (#4119)	2023-01-09 06:47:08 -08:00
chrislu	51d4a4b28d	3.39	2023-01-08 18:20:22 -08:00
chrislu	0b52f08226	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-01-08 18:03:25 -08:00
chrislu	469c959e6a	add missing options for webdav	2023-01-08 18:03:22 -08:00
monchickey	3e2c9ea73d	Add image cropping. (#4117)	2023-01-06 09:28:07 -08:00
chrislu	296fdc296c	mount: faster add chunks	2023-01-06 01:03:29 -08:00
chrislu	2abf817580	fix for stream reader ... fix https://github.com/seaweedfs/seaweedfs/issues/4112	2023-01-05 11:19:21 -08:00
chrislu	e367444586	add notes for full version	2023-01-04 09:52:25 -08:00
chrislu	5423790b2c	add back previous chunk upload selection algo	2023-01-03 22:50:39 -08:00
chrislu	7bdae5172e	batch delete EC needles ... fix https://github.com/seaweedfs/seaweedfs/issues/4107	2023-01-03 22:05:26 -08:00
zemul	6b4c033431	add mount log (#4101) ... * filer.backup use replication.source.filer * add mount log * Revert "filer.backup use replication.source.filer" This reverts commit 07bf6f956c. * fix Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-01-03 00:00:45 -08:00
chrislu	48f2edc065	3.38	2023-01-02 23:23:02 -08:00
Chris Lu	d4566d4aaa	more solid weed mount (#4089) ... * compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit 985137ec47. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as *semaphore.Weighted not found impactful * optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to *list.List * refactor chunkViews to *list.List * add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to *IntervalList[*ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation	2023-01-02 23:20:45 -08:00
CommanderRoot	c7c9d22f37	filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094)	2023-01-01 05:07:53 -08:00
CommanderRoot	c2280e94cf	filer.store.mysql: Replace deprecated upsert syntax (#4096)	2023-01-01 05:06:57 -08:00
CommanderRoot	265a56630b	filer.store.mysql: Escape table columns in SQL query (#4095)	2023-01-01 05:06:41 -08:00
Guo Lei	ed5f3f073b	add more help message, in case of misunderstanding (#4092)	2022-12-29 07:05:05 -08:00