seaweedfs

mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-16 03:29:01 +08:00

Author	SHA1	Message	Date
chrislu	9e4f985698	publish, benchmark	2023-09-04 21:43:30 -07:00
Lars Lehtonen	e6385e5e8b	weed/util: fix dropped errors (#4803 )	2023-09-01 13:10:45 -07:00
chrislu	cb470d44df	can pub and sub	2023-09-01 00:36:51 -07:00
chrislu	1eb2da46d5	connect and publish	2023-08-28 09:02:12 -07:00
chrislu	504ae8383a	protoc version	2023-08-28 09:01:25 -07:00
chrislu	dbcba75271	rename to lookup	2023-08-27 18:59:04 -07:00
chrislu	c9caf33119	move functions	2023-08-27 18:33:46 -07:00
chrislu	4d6c18d86f	pub sub initial tests	2023-08-27 17:50:59 -07:00
chrislu	4eb8e8624d	rename	2023-08-27 13:14:39 -07:00
chrislu	1990456670	sub	2023-08-27 13:13:14 -07:00
chrislu	905911853d	adjust proto	2023-08-26 13:39:21 -07:00
Nico D'Cotta	796b7508f3	Implement SRV lookups for filer (#4767 )	2023-08-24 07:08:56 -07:00
chrislu	733db2bc88	turn on streaming assign file id	2023-08-23 00:34:21 -07:00
chrislu	99f037b958	streaming assign file ids	2023-08-23 00:31:33 -07:00
chrislu	94b7e2a37c	add stream assign server side implementation	2023-08-22 09:59:04 -07:00
chrislu	ccedad5196	refactor files	2023-08-22 09:54:06 -07:00
chrislu	010c5e91e3	add stream assign proto	2023-08-22 09:53:54 -07:00
chrislu	a1149f060b	upgrade jwt	2023-08-22 08:26:31 -07:00
chrislu	c49caff5a1	change from deprecated jwt.StandardClaims to new jwt.RegisteredClaims	2023-08-21 23:42:39 -07:00
chrislu	c45665eb97	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-08-21 09:47:36 -07:00
SmsS4	f61490966f	Add time to first byte metric for s3 (#4768 ) * Add time to first byte metric for s3 * Change second to millisecond	2023-08-21 00:42:39 -07:00
chrislu	01d70c21f3	Squashed commit of the following: commit 32f4b1a13057d56b6de487cdb80ff7c205af01a6 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:52:19 2023 -0700 fix compilation commit e77ad33b7ca0423138fbae26a4433b60923a9588 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 22:46:44 2023 -0700 pub commit f431f30cc7ca277ca299e3cd118c05537fb9f5c3 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 13:27:39 2023 -0700 fix generic type commit 4e9dcb18293fd1e3e306e2dceb995dfd67a35e1d Merge: 30f942580 `16e3f2d52` Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 12:47:14 2023 -0700 Merge branch 'master' into pubsub commit 30f942580ad1bb32ae94aade2e3a21ec3ab63e21 Author: chrislu <chris.lu@gmail.com> Date: Sun Aug 20 11:10:58 2023 -0700 wip commit f8b00980bc2f3879bb43decffd9a08d842f196f2 Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:35 2023 -0700 add design document commit 08d2bebe42a26ebc39f1542f54d99e73620727dd Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:14:06 2023 -0700 minor commit bcfa7982b262a40fcdce6fc6613fad2ce07c13da Author: chrislu <chris.lu@gmail.com> Date: Tue Jul 25 09:13:49 2023 -0700 rename	2023-08-20 22:53:05 -07:00
chrislu	16e3f2d528	fix log buffer test	2023-08-20 12:46:15 -07:00
chrislu	3e9c32a3f0	refactoring to use atomic bool fix compilation	2023-08-20 12:13:06 -07:00
SmsS4	1e72579ac7	Fix s3.clean.uploads (#4766 )	2023-08-19 06:30:18 -07:00
chrislu	6792db21dc	ErrInvalidPart when completed parts has extra parts fix https://github.com/seaweedfs/seaweedfs/discussions/4746	2023-08-17 00:51:42 -07:00
chrislu	f5c4b642bb	re-ordering	2023-08-16 23:54:23 -07:00
chrislu	3852307e94	renaming	2023-08-16 23:47:43 -07:00
chrislu	6c7fa567d4	add separate cache directory for write buffers	2023-08-16 23:39:21 -07:00
SmsS4	ea1b259c12	Update weed fix description (#4745 )	2023-08-13 03:50:53 -07:00
Nikita Mochalov	4b1ba7f5b2	Save disk space metrics immediately (#4740 )	2023-08-10 06:52:32 -07:00
Nikita Mochalov	e6a49dc533	Fix resource leaks (#4737 ) * Fix division by zero * Fix file handle leak * Fix file handle leak * Fix file handle leak * Fix goroutine leak	2023-08-09 15:30:36 -07:00
chrislu	3365468d0d	added an error message	2023-08-08 20:35:21 -07:00
wusong	3ec977cd6c	minor (#4723 )	2023-08-01 02:34:59 -07:00
chrislu	c09100f928	fix building mongo index	2023-07-24 11:09:51 -07:00
chrislu	7eafa3420b	3.55	2023-07-23 23:11:37 -07:00
SmoothDenis	51bcc219ea	s3api should return 500 code from filer (#4699 )	2023-07-22 07:22:38 -07:00
wusong	33b9192536	Fix large file chunks deleted by mistake (#4678 )	2023-07-17 07:52:14 -07:00
chrislu	358b3a4894	3.54	2023-07-16 22:47:57 -07:00
chrislu	6f588b5b18	fix refactoring mistake fix https://github.com/seaweedfs/seaweedfs/issues/4639	2023-07-16 11:22:48 -07:00
mervynzhang	df400e6c71	Concurrency works better (#4663 ) Co-authored-by: mervyn.zhang <mervyn.zhang@sap.com>	2023-07-12 23:04:54 -07:00
wusong	0cb9ddd8ec	Fix data loss: add lock for metacache (#4664 ) Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-07-11 22:23:32 -07:00
Henco Appel	89542db2d0	Fix S3 API multipart upload on Windows (#4666 ) The file paths were being having '/' changed to '\' by filepath.Dir() resulting in a file being created with '\' separators, but when trying to read the files, the same wasn't happening. Co-authored-by: Hendrik Appel <happel@europe.altair.com>	2023-07-11 09:05:14 -07:00
Henco Appel	680c2e76f1	Fix the segfault from not setting s3 https port (#4667 )	2023-07-11 08:03:20 -07:00
Konstantin Lebedev	2c076ebd96	avoid logging nil error (#4659 ) Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-07-10 23:38:52 -07:00
LHHDZ	3730772e05	refresh max volume count after adjust (#4662 ) Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-07-10 23:37:33 -07:00
chrislu	c278bac263	avoid nil needle map fix https://github.com/seaweedfs/seaweedfs/issues/4640	2023-07-07 22:16:58 -07:00
chrislu	0606b59a3e	mount: create mount root on filer fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/127	2023-07-07 09:30:32 -07:00
chrislu	14546f0ad8	minor	2023-07-06 09:12:28 -07:00
chrislu	ab30f489c3	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-07-06 00:33:02 -07:00
chrislu	31b2751aff	clone volume locations in case they are changed fix https://github.com/seaweedfs/seaweedfs/issues/4642	2023-07-06 00:32:58 -07:00
wusong	61553beba2	Fix DataBackend nil pointer (#4641 )	2023-07-04 23:22:10 -07:00
chrislu	a315490f7d	proxy to master uses http address fix https://github.com/seaweedfs/seaweedfs/issues/4607	2023-07-04 11:45:21 -07:00
chrislu	2aa59ab37c	fix copying level db files during commi fix https://github.com/seaweedfs/seaweedfs/issues/4635	2023-07-04 11:28:12 -07:00
wusong	c44a6552d2	Fix buff use (#4628 )	2023-07-03 08:33:34 -07:00
Amir H. Yeganemehr	f0d3c874b6	Unix Socket listener for S3 server (#4621 ) Add localSocket option to s3 server	2023-06-26 16:22:45 -07:00
chrislu	2c4c2f0994	3.53	2023-06-25 22:51:14 -07:00
chrislu	f193942adc	lock on append metadata operation	2023-06-25 22:07:23 -07:00
chrislu	61c42f9991	adjust lock APIs	2023-06-25 20:30:20 -07:00
chrislu	aafb874ad2	lock until cancelled	2023-06-25 19:31:25 -07:00
chrislu	c4f44d56f7	add distributed lock client	2023-06-25 17:38:34 -07:00
chrislu	f8aa5ea844	adjust filer.proto	2023-06-25 17:38:19 -07:00
chrislu	13a8593ab9	adjust errors	2023-06-25 17:37:54 -07:00
chrislu	06471dac9d	init lock ring	2023-06-25 15:28:16 -07:00
chrislu	5149b3d07b	filer can proxy to peer filer holding the lock	2023-06-25 15:28:01 -07:00
chrislu	868f7875d7	refactor	2023-06-25 14:30:58 -07:00
chrislu	26eff062c8	refactor	2023-06-25 14:17:58 -07:00
chrislu	464a71a373	add distributed lock manager	2023-06-25 14:14:40 -07:00
chrislu	ee4f7cd636	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-06-25 00:58:24 -07:00
chrislu	3fd659df2a	add distributed lock manager	2023-06-25 00:58:21 -07:00
Damiano Albani	0d9ba8c612	Support templating name of files stored via Rclone backend (#4606 )	2023-06-24 07:08:24 -07:00
chrislu	a82c44972b	refactor	2023-06-20 00:12:59 -07:00
chrislu	14a19170e2	refactor	2023-06-19 23:51:04 -07:00
chrislu	84ced2abf5	add LockRing	2023-06-19 23:48:33 -07:00
chrislu	fe1716ccdf	remove deprecated tests	2023-06-19 18:43:51 -07:00
chrislu	1ce725728d	minor	2023-06-19 18:43:41 -07:00
chrislu	11549706ef	refactor	2023-06-19 18:21:07 -07:00
chrislu	8ec1bc2c99	remove unused cluster node leader	2023-06-19 18:19:13 -07:00
Bai Jie	3b88ab42aa	remove duplicate `fileCount` query (#4588 )	2023-06-18 00:14:14 -07:00
Bai Jie	44b9d72ef0	doIsEmpty() return error if v.DataBackend is nil (#4587 )	2023-06-18 00:13:40 -07:00
chrislu	94fbf02ea2	generate jwt token when looking up needle id fix https://github.com/seaweedfs/seaweedfs/issues/4577	2023-06-17 00:11:23 -07:00
柏杰	0b0fb9b9e4	avoid data race read volume.IsEmpty (#4574 ) * avoid data race read volume.IsEmpty - avoid phantom read isEmpty for onlyEmpty - use `v.DataBackend.GetStat()` in v.dataFileAccessLock scope * add Destroy(onlyEmpty: true) test * add Destroy(onlyEmpty: false) test * remove unused `IsEmpty()` * change literal `8` to `SuperBlockSize`	2023-06-14 14:39:58 -07:00
Konstantin Lebedev	1e22d5caf2	fix get file stats for IsEmpty (#4576 )	2023-06-14 01:43:30 -07:00
Konstantin Lebedev	4dd890d4a2	optional https port for s3 (#4482 ) Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-13 23:58:49 -07:00
chrislu	3fbf4f6189	Revert "Initial patch for redis version 7 support (#4572 )" This reverts commit `a25bca0692`.	2023-06-12 22:27:38 -07:00
Konstantin Lebedev	4527ead295	fix from comment delete volume is empty (#4573 ) * fix from coments https://github.com/seaweedfs/seaweedfs/pull/4561 * fix tests --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-12 22:22:46 -07:00
Aaron Gipson	a25bca0692	Initial patch for redis version 7 support (#4572 )	2023-06-12 13:30:03 -07:00
Konstantin Lebedev	25535e9c36	Delete volume is empty (#4561 ) * use onlyEmpty for deleteVolume https://github.com/seaweedfs/seaweedfs/issues/4559 * fix IsEmpty * fix test --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-12 10:42:44 -07:00
柏杰	68feffa4e6	avoid overwriting non-nil uploadReaderToChunks.uploadErr with nil value (#4560 )	2023-06-10 23:15:27 -07:00
Konstantin Lebedev	5ee04d20fa	Healthz check for deadlocks (#4558 )	2023-06-09 09:42:48 -07:00
wusong	d6af3a8eb6	fix 404 not found (#4536 ) Signed-off-by: wang wusong <wangwusong@virtaitech.com> Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-06 09:14:49 -07:00
Konstantin Lebedev	5caff2c0f0	fix: deletes newly created volumes (#4551 ) https://github.com/seaweedfs/seaweedfs/issues/4531 Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-06 09:08:27 -07:00
wusong	9bdbf9c880	revert #4491 (#4550 ) Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-06 00:17:51 -07:00
chrislu	0bffb3770f	support TTL when creating folders related to https://github.com/seaweedfs/seaweedfs/discussions/4549	2023-06-05 23:29:25 -07:00
wusong	26f15d0079	Fix no more writable volumes by delay judgment (#4548 ) * fix nomore writables volumes while disk free space is sufficient by time delay * reset --------- Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-05 10:17:21 -07:00
chrislu	fb4b61036c	3.52	2023-06-04 22:34:26 -07:00
Konstantin Lebedev	a0931be0c0	S3 TLS credentials Refreshing (#4506 ) * S3 TLS credentials Refreshing * fix: logging --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-06-04 14:27:56 -07:00
wusong	5aec6da8a3	fix deadlock for filer upload (#4527 ) Signed-off-by: wang wusong <wangwusong@virtaitech.com> Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-06-03 13:38:27 -07:00
chrislu	e23f3d6eca	set df.File to nil after it is closed possibly fix https://github.com/seaweedfs/seaweedfs/issues/4530	2023-06-02 17:16:45 -07:00
chrislu	ca7cc61319	reduce the window size between unregistering a volume server and creating volumes on that server fix https://github.com/seaweedfs/seaweedfs/issues/4467	2023-05-30 08:20:00 -07:00
chrislu	68c3cf767c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-05-29 16:45:28 -07:00
chrislu	d1d1757520	adjust error message	2023-05-29 16:45:25 -07:00
wusong	e7f4418855	add return for error handle (#4502 )	2023-05-24 07:45:06 -07:00
chrislu	4310e1fac4	3.51	2023-05-21 23:43:39 -07:00
wusong	8fffe3e822	fix no more writables volumes while disk free space is sufficient (#4491 ) Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-05-21 22:18:50 -07:00
mervynzhang	1ebb549f77	support swift (#4480 )	2023-05-19 06:39:25 -07:00
Mesar Hameed	a90b777ff4	Feat: etcd filer store keys should have customizable prefix (#4484 ) An etcd cluster is not necessarily only dedicated to seaweedfs. This security enhancement adds a customizable key_prefix option to the etcd filer store. This will allow an etcd cluster administrator to limit the seaweedfs etcd user to only read/write a subset of keys under the key_prefix, instead of all keys on the etcd cluster.	2023-05-18 23:08:56 -07:00
SmsS4	17e91d2917	Use filerGroup for s3 buckets collection prefix (#4465 ) * Use filerGroup for s3 buckets collection prefix * Fix templates * Remove flags * Remove s3CollectionPrefix	2023-05-16 09:39:43 -07:00
chrislu	b7f011f777	fix test	2023-05-15 17:46:56 -07:00
Mesar Hameed	65484e80ef	Feat: support username/password authentication for etcd filer store s… (#4477 ) Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262 Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>	2023-05-15 17:45:14 -07:00
chrislu	9204ee2d2d	3.50	2023-05-11 08:03:11 -07:00
chrislu	d999f1f0e2	update raft version fix #4460	2023-05-09 22:54:23 -07:00
chrislu	25b237b83c	reset wait time on a successful retry	2023-05-07 23:39:33 -07:00
chrislu	59f55c1a66	3.49	2023-05-07 14:00:19 -07:00
chrislu	b2a3860290	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-05-07 13:33:47 -07:00
chrislu	4511edc871	update raft	2023-05-07 13:33:44 -07:00
themarkchen	7592d013fe	fix shell volume.balance bug (#4447 )	2023-05-04 07:35:37 -07:00
chrislu	42766bccc8	3.48	2023-04-30 22:20:18 -07:00
zemul	e9fda774f4	[Filer] post add param:saveInside (#4434 ) * fix:mount deadlock * feat: filer http upload to metadata * feat: /etc save inside --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-29 08:31:05 -07:00
wusong	4867aa03ab	Fix: http rename move dir to subdir (#4432 ) Co-authored-by: wang wusong <wangwusong@virtaitech.com>	2023-04-26 20:54:36 -07:00
Konstantin Lebedev	d75a7b7f62	allow deleting only older empty dir without recursion (#4430 )	2023-04-25 08:31:14 -07:00
zemul	f352616b7d	Volume range read use sync.pool (#4422 )	2023-04-24 07:00:48 -07:00
chrislu	18686b7375	3.47	2023-04-23 22:45:41 -07:00
zemul	68f1cc34a5	[mount]Fix copy file range (#4407 ) * fix:mount deadlock * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-23 22:34:38 -07:00
chrislu	e1b92bc701	reset local file content if remote file is updated fix https://github.com/seaweedfs/seaweedfs/issues/4289	2023-04-22 22:32:21 -07:00
chrislu	ff7b6d779e	avoid overwriting variables fix https://github.com/seaweedfs/seaweedfs/issues/4365	2023-04-22 11:49:59 -07:00
wusong	2e240704ab	Writables inconsistency (#4417 ) fix: inconsistent read and write permissions between master and volume server Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-21 00:14:41 -07:00
wusong	19245dde50	mount: add retry for read only case (#4416 ) * mount: add retry for read only case Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * add new util retry function for mount Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> * change error list param Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> --------- Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-21 00:09:31 -07:00
SmsS4	4131874fa8	use env for flags wtih dot in name (#4415 )	2023-04-20 09:33:27 -07:00
Damiano Albani	9f55c7c90e	Rclone storage backend (#4402 ) * Add Rclone storage backend * Support templating the name of files stored via Rclone * Enable Rclone accounting * Remove redundant type conversion * Provide progress information for Rclone download/upload operations * Log error when Rclone can't instantiate filesystem * Remove filename templating functionality for Rclone storage To (maybe) be later reintroduced as a generic functionality for all storage backends. * Remove S3 specific check * Move Rclone config initialisation to init() method	2023-04-18 14:12:12 -07:00
wusong	a408b46d95	compilation fail (#4414 ) Signed-off-by: Wusong Wang <wangwusong@virtaitech.com> Co-authored-by: Wusong Wang <wangwusong@virtaitech.com>	2023-04-18 00:36:55 -07:00
chrislu	0a22eea55d	collect ec shard from multiple locations fix https://github.com/seaweedfs/seaweedfs/issues/4365	2023-04-17 22:56:21 -07:00
chrislu	ae9388723f	adjust error message	2023-04-17 22:01:39 -07:00
chrislu	8ecdf958ab	3.46	2023-04-16 21:57:43 -07:00
Chengyu Liu	0f653d2153	update fuse client log which should output the fh of FileHandle. (#4404 )	2023-04-15 22:15:16 -07:00
zemul	0122e022ea	Mount concurrent read (#4400 ) * fix:mount deadlock * feat: concurrent read * fix * Remove useless code * fix --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-04-13 22:32:45 -07:00
Konstantin Lebedev	5614ad0000	fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter (#4399 ) * fix s3test test_bucket_listv2_delimiter_prefix_ends_with_delimiter * fix list with delimiter and start token --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-12 16:53:49 -07:00
Konstantin Lebedev	88b2bbadd1	try fix s3test test_bucket_listv2_delimiter_prefix (#4396 )	2023-04-12 08:47:20 -07:00
Konstantin Lebedev	095cdb0c7f	fix test prefix is a directory (#4393 ) Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-12 00:08:13 -07:00
Konstantin Lebedev	44ad07276c	s3 fix get list of dir object key with slash suffix (#4391 ) * s3 fix get list of dir object key with slash suffix https://github.com/seaweedfs/seaweedfs/issues/3086 * list only entry dir eq prefix --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co>	2023-04-11 23:52:35 -07:00
Konstantin Lebedev	130bc3e668	s3 fix get fake dir object key (#4390 )	2023-04-11 07:36:22 -07:00
Matt	da1d3b5a33	adjusted regex to be from 1 to 255 for the value (#4377 ) * compatibility patch for csi driver * added namespace to all component parameters * added namespace to all component parameters * dereference in range * added namespace to values.yml defaults * added namespace to s3 component * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * push on all tags * push on all tags * push on all tags * push on all tags * push on all tags * changed helm directory structure * update charts location * fixed dereference * updated permissions * updated permissions * match current action schema * added helm chart liniting CI * modified chart list changed * fixed nodejs warning * standardized a few defaults * added chart dirs * lowerd period seconds for volume startup to test chart lint changes * lowerd period seconds for volume startup to test chart lint changes * test * changed back * adjustment * debug ls statment * removed change detection * remvoed change detection * always lint the charts * added missing ) * fixed typo * added spaces in front of all comments * pdated values.yaml * pdated values.yaml * don't validate maintainers for now * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * - adds helm chart lint on changes - adds test helm chart install on k8s * updated helm chart readme.md * added artifact hub * added artifact hub * added ttl validation * adjusted regex to be from 1 to 255 for the value * added better error message * fixed regex	2023-04-10 12:24:38 -07:00
Matt	2d213d87e1	File Path Configuration TTL Validation (#4376 ) * compatibility patch for csi driver * added namespace to all component parameters * added namespace to all component parameters * dereference in range * added namespace to values.yml defaults * added namespace to s3 component * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * added helm chart to github pages * push on all tags * push on all tags * push on all tags * push on all tags * push on all tags * changed helm directory structure * update charts location * fixed dereference * updated permissions * updated permissions * match current action schema * added helm chart liniting CI * modified chart list changed * fixed nodejs warning * standardized a few defaults * added chart dirs * lowerd period seconds for volume startup to test chart lint changes * lowerd period seconds for volume startup to test chart lint changes * test * changed back * adjustment * debug ls statment * removed change detection * remvoed change detection * always lint the charts * added missing ) * fixed typo * added spaces in front of all comments * pdated values.yaml * pdated values.yaml * don't validate maintainers for now * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * update helm_ci.yml * - adds helm chart lint on changes - adds test helm chart install on k8s * updated helm chart readme.md * added artifact hub * added artifact hub * added ttl validation	2023-04-04 22:07:17 -07:00
chrislu	efef6e94bf	use UPSERT for postgres style databases	2023-04-04 11:52:36 -07:00
chrislu	422bfaed69	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-04-02 22:53:02 -07:00
chrislu	b9ab65dba5	3.45	2023-04-02 22:52:58 -07:00
wusong	2b82d0f871	style: clerical error (#4361 )	2023-03-31 07:36:21 -07:00
LHHDZ	8e80ce9476	fix key corrupt when fs.configure copy path trie (#4353 ) Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-03-29 11:29:44 -07:00
chrislu	3cc7b90414	print wait in progress	2023-03-26 12:22:48 -07:00
chrislu	f1f14e28bc	adjust name	2023-03-26 12:17:23 -07:00
chrislu	d54798c3ae	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-03-25 22:03:00 -07:00
chrislu	6bfb91b539	update entry Attributes.Mtime = time.Now().Unix() fix https://github.com/seaweedfs/seaweedfs/issues/4337	2023-03-25 22:02:57 -07:00
renweijun	bbd461b2c6	weed mount default EnableACL，Support chmod chown (#4335 )	2023-03-25 04:47:53 -07:00
chrislu	5db9fcccd4	refactoring	2023-03-21 23:01:49 -07:00
Konstantin Lebedev	de4545c28b	add validate config for raft (#4332 )	2023-03-21 06:36:13 -07:00
chrislu	adb90bd252	avoid lower casing the command fix https://github.com/seaweedfs/seaweedfs/pull/4321	2023-03-19 21:20:46 -07:00
chrislu	5b43c4bb98	3.44	2023-03-19 20:27:55 -07:00
chrislu	50dc2fe96b	cleaning variables	2023-03-19 18:48:40 -07:00
Konstantin Lebedev	29d1312625	Fs.verify.concurrency (#4293 ) * init * refactor * refactor print msg * param just concurrency * fix race * next fix race * reset variables before each run * clean --------- Co-authored-by: Konstantin Lebedev <9497591+kmlebedev@users.noreply.github.co> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-03-19 18:38:57 -07:00
chrislu	784daea1fa	fix volume not found if marked as read only fix https://github.com/seaweedfs/seaweedfs/issues/4088	2023-03-19 18:30:13 -07:00
Stewart Miles	264be0d2d4	Retry until a leader is selected. (#4318 ) Fixes regression introduced in https://github.com/seaweedfs/seaweedfs/pull/4313 Related to #4307	2023-03-16 20:50:38 -07:00
LHHDZ	d695119073	The filer does not set defaultReplication to the defaultReplication o… (#4315 ) The filer does not set defaultReplication to the defaultReplication of the master (it is not necessary, because if the filer is not set, the configuration of the master will be used when calling Assign), otherwise, when the defaultReplication of the master is modified and all master nodes are restarted, the defaultReplication will not take effect (because the filer Keep the previous defaultReplication setting from the master) Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-03-15 22:10:39 -07:00
Stewart Miles	57ab1f8516	Use exponential backoff to query leader. (#4313 ) `topology.Leader()` was using a backoff that typically resulted in at least a 5s delay when initially starting a master and raft server. This changes the backoff algorithm to use exponential backoff starting with 100ms and waiting up to 20s for leader selection. Related to #4307	2023-03-15 17:49:46 -07:00
Stewart Miles	dd71f54c6b	Fix -raftHashicorp and -raftBootstrap flag propagation. (#4309 ) `weed server` was not correctly propagating `-master.raftHashicorp` and `-master.raftBootstrap` flags when starting the master server. Related to #4307	2023-03-15 13:03:20 -07:00
Patrick Schmidt	71b33faef0	Skip parent directory creation in mount (#4310 ) A POSIX filesystem does not implicitly create parent directories when they do not exist. Directories must be explicitly created and permissions be set. This also fixes a bug where asynchronous operations would create a file in the filer before the parent directory was created. If the file was a symlink or another special type of file the directory would inherit that type and become unusable in the mounted FS.	2023-03-15 12:37:25 -07:00
chrislu	f5854d13df	fix test	2023-03-14 22:05:16 -07:00
chrislu	0454bb2d88	fix test when size is zero, the "LastModified" is not written	2023-03-14 22:02:49 -07:00
chrislu	2268d38263	fix naming fix https://github.com/seaweedfs/seaweedfs/issues/4305	2023-03-14 08:38:02 -07:00
chrislu	98bc93cf31	fix tests	2023-03-13 10:00:27 -07:00
Konstantin Lebedev	e17429223e	shell script unclean variables (#4298 )	2023-03-13 07:16:31 -07:00
Thomas Anderson	bd70683871	fix(weed/topology/node.go): typo in error message (#4292 )	2023-03-08 23:58:03 -08:00
Rohit Chormale	59706c89fb	#4270 set http status code to 409 if dir already exists (#4287 )	2023-03-08 07:45:08 -08:00
chrislu	7111c08289	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-03-08 01:48:30 -08:00
chrislu	c589e9837c	fix tests	2023-03-08 01:48:28 -08:00
Kevin Liu	244385bf0d	Fix binding metrics to ipv6 (#4286 ) * Fix binding metrics to ipv6 * Update weed/stats/metrics.go --------- Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2023-03-08 01:04:02 -08:00
zemul	bd6f2716d3	fix_read_tomb_needle_meta (#4263 ) Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-03-02 08:26:55 -08:00
zemul	0bf56298d5	fix chunk.ModifiedTsNs (#4264 ) * fix * fix mtime s > ns --------- Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-03-02 08:24:36 -08:00
chrislu	8241c9a829	adjust ui pagination size fix https://github.com/seaweedfs/seaweedfs/issues/4258	2023-02-28 08:31:26 -08:00
lfhy	1976ca9160	add -disk to filer command (#4247 ) * add -disk to filer command * add diskType to filer.grpc * use filer.disk when filerWebDavOptions.disk is empty * add filer.disk to weed server command. --------- Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>	2023-02-25 09:48:59 -08:00
chrislu	214b7cd286	volume.fix.replication: adjust the retry checking times	2023-02-22 10:47:52 -08:00
LHHDZ	db5515eada	Fix s3api_object_list_handlers returning contents less than the specified limit when more data actually exists (#4240 ) Fix when the stored data is actually enough but s3api_object_list_handlers returns less than the specified limit Signed-off-by: changlin.shi <changlin.shi@ly.com>	2023-02-21 23:08:52 -08:00
Zachary Walters	ef2f741823	Updated the deprecated ioutil dependency (#4239 )	2023-02-21 19:47:33 -08:00
chrislu	3227e4175e	3.43	2023-02-20 16:56:05 -08:00
chrislu	9bb0a9e306	clean comments	2023-02-17 01:31:44 -08:00
chrislu	545d5d0cc3	fix for io.ReaderAt used in filer.remote.sync fix https://github.com/seaweedfs/seaweedfs/issues/4194	2023-02-17 01:30:55 -08:00
wusong	79af6b41d4	[mount] Fix read file-data in entry.content (#4207 )	2023-02-15 07:18:56 -08:00
chrislu	5869945f16	avoid infinite loop fix https://github.com/seaweedfs/seaweedfs/issues/4195#issuecomment-1426100904	2023-02-10 13:05:04 -08:00
chrislu	e037c71ec3	adjust text	2023-02-10 13:04:29 -08:00
chrislu	67b8c2853a	add line return	2023-02-10 12:53:43 -08:00
chrislu	302155a6fa	fix compilation	2023-02-10 12:32:59 -08:00
chrislu	5520b81551	add some error messages for args related to https://github.com/seaweedfs/seaweedfs/issues/4115	2023-02-10 12:30:53 -08:00
chrislu	dc4ed2cd9b	do not move cloud tier volumes fix https://github.com/seaweedfs/seaweedfs/issues/4195	2023-02-09 22:17:54 -08:00
chrislu	31bb91583f	fix bug when vid not found fix https://github.com/seaweedfs/seaweedfs/issues/4193	2023-02-09 17:30:44 -08:00
chrislu	79d68a81fd	text change	2023-02-07 15:00:41 -08:00
chrislu	999a126a3f	typo	2023-02-07 14:59:47 -08:00
chrislu	1c2364936a	detect invalid bucket name fix https://github.com/seaweedfs/seaweedfs/issues/4143	2023-02-07 14:57:29 -08:00
chrislu	5de93fe442	refactoring	2023-02-07 14:45:20 -08:00
Eric Yang	5083429704	ADHOC: s3 bucket name verification (#4189 ) * ADHOC: s3 bucket name verification * add test	2023-02-07 14:37:29 -08:00
Neo	d5f77706a8	volume:fix return if Replicate write error (#4188 )	2023-02-06 22:34:56 -08:00
chrislu	8821d6b161	3.42	2023-02-05 21:28:02 -08:00
Muhammad Hallaj bin Subery	9bd422d2c9	adding support for B2 region (#4177 ) Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>	2023-02-05 21:24:21 -08:00
chrislu	84e9934bf9	fix filer.remote.sync on a S3 cloud mount fix https://github.com/seaweedfs/seaweedfs/issues/4175	2023-02-01 20:44:00 -08:00
chrislu	0932437a1c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-02-01 02:38:09 -08:00
chrislu	c47f75eb5b	fix null content type caused exception in hdfs client fix https://github.com/seaweedfs/seaweedfs/issues/4170	2023-02-01 02:38:05 -08:00
wusong	fe573f26a8	memchunk save content twice (#4174 )	2023-01-31 19:36:59 -08:00
Neo	b9b613a78e	filter system log dir does not make subscribe event (#4172 )	2023-01-31 18:56:11 -08:00
wusong	665917fdf1	[mount]: delete redundant code (#4173 )	2023-01-31 18:54:46 -08:00
a	0d085ec175	make directory lister trigger if mime type is httpd/unix-directory (#4169 )	2023-01-31 06:54:40 -08:00
chrislu	4d71af87f3	3.41	2023-01-29 22:56:01 -08:00
chrislu	ef70415c70	skipping manifest chunk volume id change	2023-01-29 22:55:06 -08:00
chrislu	6e8dbf8868	add fs.meta.changeVolumeId	2023-01-29 22:51:58 -08:00
wusong	de081c0d64	[mount] fix metacache update (#4161 )	2023-01-29 07:55:27 -08:00
Guo Lei	b63d1e9c6b	fix index (#4158 )	2023-01-28 06:55:27 -08:00
chrislu	81fdf3651b	grpc connection to filer add sw-client-id header	2023-01-20 01:48:12 -08:00
chrislu	bfe5d910c6	use one readerCache for the whole file	2023-01-16 22:43:02 -08:00
chrislu	4614e85efa	adjust help message	2023-01-15 21:28:36 -08:00
chrislu	c24678153c	use memory chunks only for sequential writes	2023-01-15 21:10:08 -08:00
chrislu	2885ba0e50	3.40	2023-01-15 20:12:05 -08:00
chrislu	12f8251d32	avoid index out of slice range	2023-01-14 13:31:22 -08:00
chrislu	313fd17ae4	allow random writes for mem chunk	2023-01-13 20:32:18 -08:00
Chris Lu	1cd2e64aac	merge chunks during upload (#4130 ) * merge chunks during upload * fix test	2023-01-12 23:07:36 -08:00
chrislu	2452f3b2f7	sort chunks by offset	2023-01-12 20:29:07 -08:00
chrislu	361d97941b	mysql: Missed escaping column name fix https://github.com/seaweedfs/seaweedfs/issues/4129	2023-01-11 08:46:56 -08:00
chrislu	efcffe4f77	no need to set chunks during flush	2023-01-10 10:01:52 -08:00
chrislu	85f1b70207	reduce group size from 256MB to 64MB	2023-01-10 01:35:44 -08:00
chrislu	86657ed3a8	avoid repeatedly adding manifest chunks	2023-01-10 01:35:10 -08:00
chrislu	28fe578944	avoid possible nil entry	2023-01-10 01:07:34 -08:00
chrislu	47fa5ef979	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-01-10 00:52:50 -08:00
chrislu	9e0d526163	fix compilation	2023-01-10 00:52:05 -08:00
chrislu	7f49c59c14	cluster.ps add filer meta sync progress	2023-01-10 00:51:25 -08:00
chrislu	340e7c3a2e	chunk group remove manifestChunks and reset sections in SetChunks()	2023-01-10 00:49:31 -08:00
chrislu	e650c8397d	ensure section.chunks is always garbage free	2023-01-10 00:48:01 -08:00
chrislu	75bdd4a0d1	refactor	2023-01-10 00:46:46 -08:00
Guo Lei	83e29faa55	use time.NewTicker instead of time.Tick. (#4119 )	2023-01-09 06:47:08 -08:00
chrislu	51d4a4b28d	3.39	2023-01-08 18:20:22 -08:00
chrislu	0b52f08226	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2023-01-08 18:03:25 -08:00
chrislu	469c959e6a	add missing options for webdav	2023-01-08 18:03:22 -08:00
monchickey	3e2c9ea73d	Add image cropping. (#4117 )	2023-01-06 09:28:07 -08:00
chrislu	296fdc296c	mount: faster add chunks	2023-01-06 01:03:29 -08:00
chrislu	2abf817580	fix for stream reader fix https://github.com/seaweedfs/seaweedfs/issues/4112	2023-01-05 11:19:21 -08:00
chrislu	e367444586	add notes for full version	2023-01-04 09:52:25 -08:00
chrislu	5423790b2c	add back previous chunk upload selection algo	2023-01-03 22:50:39 -08:00
chrislu	7bdae5172e	batch delete EC needles fix https://github.com/seaweedfs/seaweedfs/issues/4107	2023-01-03 22:05:26 -08:00
zemul	6b4c033431	add mount log (#4101 ) * filer.backup use replication.source.filer * add mount log * Revert "filer.backup use replication.source.filer" This reverts commit `07bf6f956c`. * fix Co-authored-by: zemul <zhouzemiao@ihuman.com>	2023-01-03 00:00:45 -08:00
chrislu	48f2edc065	3.38	2023-01-02 23:23:02 -08:00
Chris Lu	d4566d4aaa	more solid weed mount (#4089 ) * compare chunks by timestamp * fix slab clearing error * fix test compilation * move oldest chunk to sealed, instead of by fullness * lock on fh.entryViewCache * remove verbose logs * revert slat clearing * less logs * less logs * track write and read by timestamp * remove useless logic * add entry lock on file handle release * use mem chunk only, swap file chunk has problems * comment out code that maybe used later * add debug mode to compare data read and write * more efficient readResolvedChunks with linked list * small optimization * fix test compilation * minor fix on writer * add SeparateGarbageChunks * group chunks into sections * turn off debug mode * fix tests * fix tests * tmp enable swap file chunk * Revert "tmp enable swap file chunk" This reverts commit `985137ec47`. * simple refactoring * simple refactoring * do not re-use swap file chunk. Sealed chunks should not be re-used. * comment out debugging facilities * either mem chunk or swap file chunk is fine now * remove orderedMutex as semaphore.Weighted not found impactful optimize size calculation for changing large files * optimize performance to avoid going through the long list of chunks * still problems with swap file chunk * rename * tiny optimization * swap file chunk save only successfully read data * fix * enable both mem and swap file chunk * resolve chunks with range * rename * fix chunk interval list * also change file handle chunk group when adding chunks * pick in-active chunk with time-decayed counter * fix compilation * avoid nil with empty fh.entry * refactoring * rename * rename * refactor visible intervals to list.List refactor chunkViews to list.List add IntervalList for generic interval list * change visible interval to use IntervalList in generics * cahnge chunkViews to IntervalList[ChunkView] * use NewFileChunkSection to create * rename variables * refactor * fix renaming leftover * renaming * renaming * add insert interval * interval list adds lock * incrementally add chunks to readers Fixes: 1. set start and stop offset for the value object 2. clone the value object 3. use pointer instead of copy-by-value when passing to interval.Value 4. use insert interval since adding chunk could be out of order * fix tests compilation * fix tests compilation	2023-01-02 23:20:45 -08:00
CommanderRoot	c7c9d22f37	filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094 )	2023-01-01 05:07:53 -08:00
CommanderRoot	c2280e94cf	filer.store.mysql: Replace deprecated upsert syntax (#4096 )	2023-01-01 05:06:57 -08:00
CommanderRoot	265a56630b	filer.store.mysql: Escape table columns in SQL query (#4095 )	2023-01-01 05:06:41 -08:00
Guo Lei	ed5f3f073b	add more help message, in case of misunderstanding (#4092 )	2022-12-29 07:05:05 -08:00
Guo Lei	d8cfa1552b	support enable/disable vacuum (#4087 ) * stop vacuum * suspend/resume vacuum * remove unused code * rename * rename param	2022-12-28 01:36:44 -08:00
Guo Lei	5f993a5f20	Fix ide warning (#4082 ) fix warning message from IDE "redundant type from array, slice, or map composite literal"	2022-12-26 00:03:47 -08:00
Konstantin Lebedev	569bc7b54d	fs.verify needles status of file (#4060 ) * fs.verify needles status of file * fix parse args * refactor logging * set modifyTimeAgoAtSec	2022-12-21 15:54:17 -08:00
chrislu	77b7c9f7cc	lock and unlock chunks for read	2022-12-21 14:15:35 -08:00
chrislu	d1797deccb	remove duplicated logic of removing from writableChunks	2022-12-21 14:15:11 -08:00
chrislu	6ede19e825	add a simple file replication progress bar	2022-12-20 19:47:21 -08:00
chrislu	f7beba8515	unlock before submitting the uploading jobs	2022-12-20 12:52:30 -08:00
chrislu	28d479e5c0	mount: adjust locking for upload pipeline	2022-12-19 15:07:22 -08:00
chrislu	f9fb3cd1fc	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-12-19 11:32:00 -08:00
chrislu	6c7fe40305	filer sink retries reading file chunks, skipping missing chunks if the file chunk is not available during replication time, the file is skipped	2022-12-19 11:31:58 -08:00
chrislu	3fc3e7083c	metadata skip reading un-available logs this means the volumes for metadata histories have been lost.	2022-12-19 11:30:50 -08:00
lfhy	038bab3726	add -filer.path to webdav command (#4061 )	2022-12-17 20:07:02 -08:00
chrislu	8a40fa8993	more detailed logs	2022-12-17 13:18:35 -08:00
LHHDZ	f84c13dabd	fix response not allowed (#4057 )	2022-12-13 05:48:50 -08:00
LHHDZ	bb289d7d69	sort tags by key (#4047 ) Signed-off-by: changlin.shi <changlin.shi@ly.com> Signed-off-by: changlin.shi <changlin.shi@ly.com>	2022-12-11 22:49:57 -08:00
chrislu	438146249f	3.37	2022-12-11 21:08:06 -08:00
chrislu	ac9dea0ad9	rotate log files	2022-12-11 21:05:58 -08:00
chrislu	fc6b9e6e0c	volume server logs add url info debug https://github.com/seaweedfs/seaweedfs/issues/3964	2022-12-11 13:15:38 -08:00
chrislu	38c63c429b	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-12-10 21:49:09 -08:00
chrislu	e1ca6308cb	add chunk etag when downloading from remote storage fix https://github.com/seaweedfs/seaweedfs/issues/3987	2022-12-10 21:49:07 -08:00
wusong	549354e324	Fix hardlink counting (#4042 ) Signed-off-by: wusong <wangwusong@virtaitech.com> Signed-off-by: wusong <wangwusong@virtaitech.com> Co-authored-by: wusong <wangwusong@virtaitech.com>	2022-12-08 10:50:57 -08:00
Jiffs Maverick	5f59d43c88	[filer] Fix uri escape during renaming (#4039 )	2022-12-07 07:47:40 -08:00
chrislu	659f28b855	simple refactoring	2022-12-05 23:19:38 -08:00
chrislu	ca49ef8ed8	duplicated logic	2022-12-05 23:19:22 -08:00
Chris Lu	dac9c28d05	Revert "refactor: moved to locked entry" (#4035 ) * Revert "refactor: moved to locked entry" This reverts commit `94bc9afd9d`. * only add LockedEntry, no changes to entryLock * fix compilation	2022-12-05 12:32:27 -08:00
chrislu	fad7e1f7cb	3.36	2022-12-04 23:34:29 -08:00
chrislu	94bc9afd9d	refactor: moved to locked entry	2022-12-04 23:33:05 -08:00
aronneagu	2b783738d6	Update Makefile - use binary in current directory (#4027 )	2022-12-01 14:09:14 -08:00
aronneagu	77699855a7	Return ETag from remote when file doesn't exist on Filer (#4025 )	2022-11-30 07:43:30 -08:00
Konstantin Lebedev	f59c3acd5f	volume.list show volumes only from the specified dc/rack/dn (#4024 )	2022-11-30 06:30:55 -08:00
Konstantin Lebedev	2b910d1cf8	avoid recursive deleting newly created empty directories (#4016 )	2022-11-25 08:45:47 -08:00
Jiffs Maverick	4b0430e71d	[metrics] Add the ability to control bind ip (#4012 )	2022-11-24 10:22:59 -08:00
famosss	15284e2bee	fix: Lost Content-Encoding metadata (#4010 )	2022-11-23 22:52:34 -08:00
James Hartig	81624de27b	Include name/mime in ReadAllNeedles (#4005 )	2022-11-23 15:59:38 -08:00
Eric Yang	26d9b662b1	ADHOC: fix fs.meta.save typo (#3993 ) Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-11-20 21:05:00 -08:00
James Hartig	4c85da7844	Include meta in ReadAllNeedles (#3991 ) This is useful for doing backups on the data so we can accurately store the last modified time, the compression state, and verify the crc. Previously we were doing VolumeNeedleStatus and then an HTTP request which needlessly read from the dat file twice.	2022-11-20 20:19:41 -08:00
chrislu	70a4c98b00	refactor filer_pb.Entry and filer.Entry to use GetChunks() for later locking on reading chunks	2022-11-15 06:33:36 -08:00
chrislu	b1d7963e57	3.35	2022-11-14 00:21:00 -08:00
Guo Lei	5b905fb2b7	Lazy loading (#3958 ) * types packages is imported more than onece * lazy-loading * fix bugs * fix bugs * fix unit tests * fix test error * rename function * unload ldb after initial startup * Don't load ldb when starting volume server if ldbtimeout is set. * remove uncessary unloadldb * Update weed/command/server.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> * Update weed/command/volume.go Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com> Co-authored-by: guol-fnst <goul-fnst@fujitsu.com> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-11-14 00:19:27 -08:00
chrislu	d90aa31d5f	filer store: skip disabled location specific filer store initialization fix https://github.com/seaweedfs/seaweedfs/issues/3971	2022-11-13 23:30:23 -08:00
chrislu	7ac40ae3d4	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-11-13 23:01:58 -08:00
chrislu	aae82dca6f	file name escape uri fix https://github.com/seaweedfs/seaweedfs/issues/3969	2022-11-13 23:01:56 -08:00
LHHDZ	39f0246deb	fix 'NoSuchUpload' when upload part (#3968 )	2022-11-11 04:53:53 -08:00
famosss	df0662038c	feat:add a uuid after uploadid (#3963 )	2022-11-10 07:17:29 -08:00
aronneagu	180853a2c9	Replace dashes with underscores in x-amz-meta headers (#3965 )	2022-11-10 07:09:53 -08:00
chrislu	47db75a695	3.34	2022-11-06 23:06:22 -08:00
Konstantin Lebedev	5431c445cd	fix filer.remote.sync to azure with ContentType (#3949 ) * fix filer.remote.sync to azure with ContentType * fix pass X-Amz-Meta to X-Ms-Meta	2022-11-04 09:10:33 -07:00
zemul	cec55feeb0	[filer.backup] add retention_duration (#3941 )	2022-11-04 00:34:40 -07:00
chrislu	4193dafce1	azure metadata: skip metadata prefixed with "X-" fix https://github.com/seaweedfs/seaweedfs/issues/3875	2022-11-02 21:42:02 -07:00
pavelzhurov	abe4a61659	Bug fix: empty key in DeleteMultipleObjects request caused bucket delete (#3939 )	2022-11-02 05:51:13 -07:00
Konstantin Lebedev	0999f9b7ff	[volume.fsck] collect ids without cut off time for finding missing data from volumes (#3934 ) collect all file ids from the file without cut off time for finding missing data from volumes	2022-10-31 11:38:12 -07:00
Konstantin Lebedev	a322ba042e	[volume.fsck] param volumeId is comma separated the volume id (#3933 ) volume.fsck param volumeId is comma separated the volume id Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-10-31 11:36:26 -07:00
Konstantin Lebedev	4a48332248	refactor error contains already deleted (#3932 )	2022-10-31 11:34:45 -07:00
Konstantin Lebedev	c0deaa4948	[volume.fsck] check needles status from volume server (#3926 ) check needles status from volume server	2022-10-31 11:33:04 -07:00
chrislu	0f360862bf	3.33	2022-10-30 20:40:18 -07:00
Konstantin Lebedev	409c9328de	[master] avoid vacuum if not enough replica copies (#3924 ) avoid vacuum if not enough replica copies	2022-10-30 20:34:19 -07:00
Konstantin Lebedev	bf8a9d2db1	[volume.chek.disk] sync of deletions the fix (#3923 ) * sync of deletions the fix * avoid return if only partiallyDeletedNeedles * refactor sync deletions	2022-10-30 20:32:46 -07:00
chrislu	8b9957d461	add back "/" prefix if it is missing in object fix https://github.com/seaweedfs/seaweedfs/issues/3737	2022-10-29 17:54:30 -07:00
chrislu	0d9f2f9e7a	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-28 12:53:23 -07:00
chrislu	ea2637734a	refactor filer proto chunk variable from mtime to modified_ts_ns	2022-10-28 12:53:19 -07:00
Konstantin Lebedev	764d9cb105	[voluche.chek.disk] needles older than the cutoff time are not missing yet (#3922 ) needles older than the cutoff time are not missing yet https://github.com/seaweedfs/seaweedfs/issues/3919	2022-10-28 12:12:20 -07:00
Konstantin Lebedev	f9f499b8d5	[fix] add param for ignore error (#3918 )	2022-10-28 06:59:39 -07:00
chrislu	1e0d64c048	master: delete partially created replicated volumes during volume growth fix https://github.com/seaweedfs/seaweedfs/discussions/3792#discussioncomment-3973120	2022-10-26 12:32:24 -07:00
Konstantin Lebedev	c6f2899375	avoid fix missing of deleted needles (#3909 )	2022-10-25 21:35:54 -07:00
Eric Yang	51d462f204	ADHOC: volume fsck using append at ns (#3906 ) * ADHOC: volume fsck using append at ns * nit * nit Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-10-24 22:09:38 -07:00
famosss	34132b2c9f	fix: isTruncated returns problematic (#3907 )	2022-10-24 21:15:37 -07:00
famosss	874fd197b5	feat: simplify a bit (#3905 )	2022-10-24 17:58:26 -07:00
famosss	25e012d30b	fix: set user metadata key to lowercase (#3894 ) * fix: set user metadata key to lowercase * feat: simplify a bit	2022-10-24 17:29:52 -07:00
chrislu	091f999e8c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-24 16:50:41 -07:00
chrislu	377870f4a9	keep system log data	2022-10-24 16:50:39 -07:00
Guo Lei	97edb40275	Fix errinfo (#3893 ) * types packages is imported more than onece * Fix error response when format of --expires is wrong. It MUST be in RFC 1123 date format.	2022-10-24 09:25:48 -07:00
chrislu	d5364218b2	adjust help message	2022-10-24 09:25:08 -07:00
chrislu	d880fc2bb3	fix merge	2022-10-23 18:26:22 -07:00
Konstantin Lebedev	6253058d9d	ensure monotonic n.AppendAtNs in each place (#3880 ) https://github.com/seaweedfs/seaweedfs/issues/3852 Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-10-23 18:24:52 -07:00
Konstantin Lebedev	452202f35b	ReadNeedleMeta recovery panic occurred (#3883 ) ReadNeedleMeta panic occurred https://github.com/seaweedfs/seaweedfs/issues/3882	2022-10-23 14:00:28 -07:00
chrislu	e3ea544d40	skip reading needle if size is invalid related to https://github.com/seaweedfs/seaweedfs/issues/3882	2022-10-23 14:00:15 -07:00
chrislu	5294b6dd4d	remove volume server tcp client	2022-10-23 13:15:03 -07:00
chrislu	f2d9049e6a	fix size variable	2022-10-23 13:05:37 -07:00
chrislu	184fbb6c50	volume server: remote tier volumes only soft delete in local index fix https://github.com/seaweedfs/seaweedfs/issues/3889	2022-10-23 13:04:38 -07:00
liubaojiang	25471d579a	add md5 header when UploadData to replication in ReplicatedWrite (#3881 )	2022-10-19 01:34:14 -07:00
chrislu	758d70bdc0	3.32	2022-10-16 20:43:37 -07:00
chrislu	de286fe662	shell: volume.move handles volume moved to cloud tier fix https://github.com/seaweedfs/seaweedfs/issues/3803	2022-10-16 17:52:22 -07:00
chrislu	52e0a88a15	shell: volume.tier.upload clean up replicated copies fix https://github.com/seaweedfs/seaweedfs/issues/3804	2022-10-16 17:02:56 -07:00
Konstantin Lebedev	e20f0dbd2d	avoid data race of TraverseBfs (#3856 ) * avoid data race of TraverseBfs * close is enough avoid panic I1014 12:29:59.207120 volume_loading.go:131 loading sorted db /tmp/sw/test2_19.sdx error: unexpected file /tmp/sw/test2_19.idx size: 255 I1014 12:29:59.207125 volume_loading.go:119 open to write file /tmp/sw/test4_26.idx panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x5260a4c] goroutine 287 [running]: github.com/seaweedfs/seaweedfs/weed/storage.(SortedFileNeedleMap).Close(0x0) /Users/tochka/GolandProjects/seaweedfs/weed/storage/needle_map_sorted_file.go:97 +0x2c github.com/seaweedfs/seaweedfs/weed/storage.(Volume).load.func1() /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:32 +0x8e github.com/seaweedfs/seaweedfs/weed/storage.(Volume).load(0xc001b36280, 0x1, 0x1, 0x0, 0x69228c0?) /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume_loading.go:205 +0x256c github.com/seaweedfs/seaweedfs/weed/storage.NewVolume({0x7ffeefbff6e0, 0x7}, {0x7ffeefbff6e0, 0x7}, {0xc0009a9284, 0x5}, 0x13, 0x0, 0x0, 0x0, ...) /Users/tochka/GolandProjects/seaweedfs/weed/storage/volume.go:62 +0x30f github.com/seaweedfs/seaweedfs/weed/storage.(DiskLocation).loadExistingVolume(0xc0006f40c0, {0x846c8d0, 0xc0009ce600}, 0x0?, 0x1) /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:161 +0x4da github.com/seaweedfs/seaweedfs/weed/storage.(DiskLocation).concurrentLoadingVolumes.func2() /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:201 +0xf9 created by github.com/seaweedfs/seaweedfs/weed/storage.(DiskLocation).concurrentLoadingVolumes /Users/tochka/GolandProjects/seaweedfs/weed/storage/disk_location.go:198 +0x150	2022-10-16 11:30:41 -07:00
Konstantin Lebedev	ee38ab8581	avoid data race of LogBuffer isStopping (#3859 )	2022-10-16 11:28:56 -07:00
Konstantin Lebedev	7836f7574e	[volume.fsck] hotfix apply purging and add option verifyNeedle #3860 (#3861 ) * fix apply purging and add verifyNeedle * common readSourceNeedleBlob * use consts	2022-10-15 20:38:46 -07:00
Konstantin Lebedev	f476cf3403	fix volume_server.proto conflict (#3858 ) fix conflict https://github.com/seaweedfs/seaweedfs/pull/3841	2022-10-15 20:01:07 -07:00
chrislu	5c2eb602c9	Revert "mount: remove mount directory from generating cache directory id" This reverts commit `109dc7fdf6`.	2022-10-14 22:24:14 -07:00
Konstantin Lebedev	2f72103c83	avoid load volume file with BytesOffset mismatch (#3841 ) * avoid load volume file with BytesOffset mismatch https://github.com/seaweedfs/seaweedfs/issues/2966 * set BytesOffset if has not VolumeInfoFile * typos fail => failed * exit if bytesOffset mismatch	2022-10-14 00:18:09 -07:00
Konstantin Lebedev	f19c9e3d9d	Volume fsck by volume (#3851 ) * refactor * refactor args verbose and writer * refactor readFilerFileIdFile * fix filter by collectMtime * skip system log collection	2022-10-13 23:30:30 -07:00
chrislu	9c8678ded9	ensure monotonic n.AppendAtNs fix https://github.com/seaweedfs/seaweedfs/issues/3852	2022-10-13 23:15:00 -07:00
chrislu	e05637c42c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-13 23:10:49 -07:00
chrislu	dcd0743a35	remove unused ReadNeedleBlobRequest.needle_id fix https://github.com/seaweedfs/seaweedfs/issues/3853	2022-10-13 23:10:46 -07:00
LHHDZ	a5b867af69	don't skip empty value at `AssembleEntryWithAcp` (#3855 ) * add acl helper functionalities Signed-off-by: changlin.shi <changlin.shi@ly.com> * add tests Signed-off-by: changlin.shi <changlin.shi@ly.com> * remove 0 when create map Signed-off-by: changlin.shi <changlin.shi@ly.com> * delete when empty at `AssembleEntryWithAcp` `PutBucketAcl/PutObjectAcl` allow request with empty grants, `AssembleEntryWithAcp` shouldn't skip empty value Signed-off-by: changlin.shi <changlin.shi@ly.com> Signed-off-by: changlin.shi <changlin.shi@ly.com>	2022-10-13 22:50:44 -07:00
Konstantin Lebedev	1f7e52c63e	vacuum metrics and force sync dst files (#3832 )	2022-10-13 00:51:20 -07:00
chrislu	f5d4952d73	filer: redis store reduce from 2 redis operations to 1 for updates.	2022-10-12 23:50:09 -07:00
Guo Lei	f95c25e113	types packages is imported more than onece (#3838 )	2022-10-12 22:59:07 -07:00
chrislu	9a339a9cfb	default hasSlowRead to true	2022-10-12 21:15:10 -07:00
famosss	cacc3e883b	volume server:set the default value of "hasSlowRead" to true (#3710 ) * simplify a bit * feat: volume: add "readBufSize" option to customize read optimization * refactor : redbufSIze -> readBufferSize * simplify a bit * simplify a bit * volume server:set the default value of "hasSlowRead" to true	2022-10-12 21:13:26 -07:00
LHHDZ	bf5e45b66a	add acl helper functionalities (#3831 )	2022-10-12 20:15:16 -07:00
Konstantin Lebedev	401315f337	master fix interruption through ctrl+c (#3834 )	2022-10-12 07:18:40 -07:00
zemul	a05725aea6	filer: get directory metadata (#3833 )	2022-10-12 07:14:49 -07:00
chrislu	e55076c46f	cloud tier: add retry when copying data file fix https://github.com/seaweedfs/seaweedfs/issues/3828	2022-10-12 00:38:32 -07:00
chrislu	cea73ac008	serialize sqlite operations fix https://github.com/seaweedfs/seaweedfs/issues/3827	2022-10-12 00:03:27 -07:00
chrislu	dff85e9c71	fix error handling	2022-10-12 00:03:14 -07:00
chrislu	8aec786a6d	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-11 21:58:19 -07:00
chrislu	0d817bc347	fix invalid memory address or nil pointer dereference on filer.sync fix https://github.com/seaweedfs/seaweedfs/issues/3826	2022-10-11 21:58:17 -07:00
Guo Lei	84c401e693	Optimiz leveldb metric (#3830 ) * optimiz updating mapmetric for leveldb * import loading leveldb * add comments	2022-10-11 21:13:25 -07:00
LHHDZ	d21e2f523d	split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unn… (#3824 ) split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call Signed-off-by: changlin.shi <changlin.shi@ly.com> Signed-off-by: changlin.shi <changlin.shi@ly.com>	2022-10-11 20:14:14 -07:00
Konstantin Lebedev	5b28c3f728	revert disabling FSync for non Mac (#3814 )	2022-10-10 07:28:02 -07:00
LHHDZ	3550692afc	change s3_account.go package to avoid cycle dependency (#3813 )	2022-10-10 06:44:29 -07:00
chrislu	0711870f43	3.31	2022-10-10 00:15:30 -07:00
chrislu	109dc7fdf6	mount: remove mount directory from generating cache directory id fix https://github.com/seaweedfs/seaweedfs/discussions/3811	2022-10-10 00:12:10 -07:00
chrislu	fc54afdba9	tests add parsing ec shard info	2022-10-09 22:47:55 -07:00
chrislu	0623bf582e	include ec shard for capacityByFreeVolumeCount	2022-10-09 22:31:30 -07:00
chrislu	98dc1e5c15	move volume: find target volume server by exiting/max ratio	2022-10-09 22:30:59 -07:00
chrislu	049f040c3c	refactor	2022-10-09 22:28:39 -07:00
chrislu	124b97f6d7	parsing ec volumes	2022-10-09 19:21:23 -07:00
chrislu	1e27d36576	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-09 18:58:14 -07:00
chrislu	f9383aa726	refactor to change capacity data type	2022-10-09 18:58:10 -07:00
binbinshi	b7de4a967e	fix: compact_map get error mismatching cokie (#3748 ) * fix: compact_map get error * fix: CompactSection delete lock and move test to compact_map Co-authored-by: shibinbin <shibinbin@megvii.com>	2022-10-09 16:56:40 -07:00
BakerBunker	b9d8a837af	Fix `mount -t weed` with extra options (#3807 ) fuse: Ignore "-o" command	2022-10-08 11:25:19 -07:00
Konstantin Lebedev	5db25a8f2a	metric shows who is currently blocking the cluster or not (#3799 ) * master_admin_lock Shows whether cluster is locked now or not https://github.com/seaweedfs/seaweedfs/issues/3452 * fix metric MasterAdminLock	2022-10-07 13:26:29 -07:00
Viktor Kuzmin	f8d3ff466d	Allow parallel volume loading from different dirs during startup. (#3802 )	2022-10-07 11:27:15 -07:00
Konstantin Lebedev	a522507f95	configure raft metrics (#3798 )	2022-10-07 04:20:34 -07:00
Konstantin Lebedev	b9933d5589	master server graceful stop (#3797 )	2022-10-06 09:30:30 -07:00
Konstantin Lebedev	36daa7709d	show raft leader via shell (#3796 )	2022-10-06 07:10:41 -07:00
chrislu	ea271600ec	fix parameters	2022-10-04 12:36:05 -07:00
chrislu	0452ae6a6c	filer.sync: limit concurrency when fetching file chunks fix https://github.com/seaweedfs/seaweedfs/issues/3787	2022-10-04 11:35:07 -07:00
Konstantin Lebedev	ec46a34f33	avoid increasing the coordinates of metric labels (#3785 )	2022-10-04 01:20:01 -07:00
chrislu	29632d5a34	3.30	2022-10-02 18:13:44 -07:00
LHHDZ	e9584d9661	add ownership rest apis (#3765 )	2022-10-01 19:18:00 -07:00
Konstantin Lebedev	6fa3d0cc46	fix delete key panic of remote sync dir (#3770 )	2022-10-01 02:33:47 -07:00
chrislu	c3487d3da7	better fix for option format	2022-09-30 08:49:38 -07:00
chrislu	ba0db28dbb	mount: fix extra option format	2022-09-30 08:44:41 -07:00
chrislu	8e81619d02	mount: accept all extra mount options fix https://github.com/seaweedfs/seaweedfs/issues/3767	2022-09-30 08:40:37 -07:00
Eric Yang	56c94cc08e	ADHOC: filter deleted files from idx file binary search (#3763 ) * ADHOC: filter deleted files from idx file binary search * remove unwanted check Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-09-29 12:48:36 -07:00
LHHDZ	3de1e19780	s3: sync bucket info from filer (#3759 )	2022-09-29 12:29:01 -07:00
LHHDZ	e00a12b099	associate `Account` and `Identity` by accountId (#3754 )	2022-09-28 13:25:59 -07:00
LHHDZ	aacdcc4cad	s3: add account (#3753 ) associate `Account` and `Identity` by accountId	2022-09-28 12:45:18 -07:00
Konstantin Lebedev	301b678147	[volume] Add new volumes to HUP(reload) signal (#3755 ) Add new volumes to HUP(reload) signal	2022-09-28 12:44:13 -07:00
chrislu	47fd5d3fa1	skip debug message	2022-09-27 13:12:31 -07:00
Konstantin Lebedev	7ae51d1ec6	fix set master options for single server (#3708 ) Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-09-27 12:06:35 -07:00
jerebear12	190afcc3fd	Allow CORS on Filer (#3750 ) Adjusted filer response headers	2022-09-27 11:47:52 -07:00
Konstantin Lebedev	faa6167b6b	fs.meta.load load any dirs with prefix "important" (#3747 ) * fs.meta.load load any dirs with prefix "important" * replace dirPattern to dirPrefix * help dirPrefix	2022-09-27 09:58:46 -07:00
chrislu	b6d7556dda	skip truncation on error fix https://github.com/seaweedfs/seaweedfs/issues/3746	2022-09-27 09:48:23 -07:00
chrislu	1ffb1e696e	3.29	2022-09-25 19:20:45 -07:00
chrislu	640c53cda8	add future list	2022-09-25 17:43:42 -07:00
chrislu	cc570a4477	assert results are sorted	2022-09-25 13:49:28 -07:00
chrislu	5c8f1467a1	ordered execution async wait	2022-09-25 13:45:55 -07:00
chrislu	973b8ff067	rename	2022-09-25 13:34:55 -07:00
chrislu	e31fdbc89b	rename file	2022-09-25 12:59:38 -07:00
chrislu	31922b2bf2	s3 to watch specific directories	2022-09-20 09:30:05 -07:00
chrislu	b90d92fca9	grpc: watch metadata changes by directory	2022-09-20 09:25:18 -07:00
chrislu	b463ca1a2f	filer replication: compare content changes directly Fix https://github.com/seaweedfs/seaweedfs/issues/3714 The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks. For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.	2022-09-20 08:35:10 -07:00
chrislu	b2a171632b	mount: fix 0 file size in linux	2022-09-18 21:42:13 -07:00
chrislu	23f3aea4cf	adjust logs	2022-09-18 18:14:24 -07:00
chrislu	901e8157de	3.28	2022-09-18 17:46:04 -07:00
chrislu	9d230521a4	report write error	2022-09-18 16:49:48 -07:00
chrislu	10d5b4b32b	volume server: rename readBufferSize to readBufferSizeMB	2022-09-17 10:56:28 -07:00
chrislu	3fc261d27c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-16 23:54:39 -07:00
chrislu	06748f2771	remove unused code	2022-09-16 23:54:37 -07:00
Konstantin Lebedev	5d87ad72d8	mute log filer: no entry is found in filer store (#3707 )	2022-09-16 09:05:38 -07:00
Ryan Russell	824f7ad9e1	refactor(shell): readability improvements (#3704 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-16 02:43:49 -07:00
Ryan Russell	277976bd76	refactor(storage): readability improvements (#3703 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-16 02:43:17 -07:00
famosss	d949a238b8	volume: add "readBufSize" option to customize read optimization (#3702 ) * simplify a bit * feat: volume: add "readBufSize" option to customize read optimization * refactor : redbufSIze -> readBufferSize * simplify a bit * simplify a bit	2022-09-16 00:30:40 -07:00
chrislu	cf90f76a35	mark "hasSlowRead" as experimental	2022-09-15 23:33:46 -07:00
Ryan Russell	6f7ef8711a	docs(s3api): readability improvements (#3696 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:13:21 -07:00
Ryan Russell	ad3a3c8782	refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `de… (#3695 ) refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:48 -07:00
Ryan Russell	c4fc007630	refactor(raft_hashicorp): `timeSpeep` -> `timeSleep` (#3694 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:30 -07:00
Ryan Russell	c202d4c1b0	refactor(master_grpc_server_cluster): `seletedSet` -> `selectedSet` (#3693 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:13 -07:00
chrislu	f9e3e7d1c4	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-15 03:11:34 -07:00
chrislu	896a85d6e4	volume: add "hasSlowRead" option to customize read optimization	2022-09-15 03:11:32 -07:00
Ryan Russell	e0064ae097	docs: update `fileIdsToDelete` var (#3692 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:09:06 -07:00
Ryan Russell	d65bdeef08	docs(meta_cache): readability fixes (#3691 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:04:57 -07:00
Ryan Russell	92658b05a8	refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConne… (#3690 ) refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:04:29 -07:00
Ryan Russell	a635973847	docs(topology): readability improvements (#3688 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:03:10 -07:00
Ryan Russell	0fc242b084	docs: `panicing` -> `panicking` (#3687 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:38:04 -07:00
Ryan Russell	f6d391c9da	docs(command): readability fixes (#3686 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:33:04 -07:00
Ryan Russell	150ce9f3f6	docs(wdclient): readability fix (#3685 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:29:26 -07:00
ImFantuan	b64411bda8	fix 0 size (#3683 )	2022-09-15 01:27:02 -07:00
famosss	bb880d6253	IsTruncated returned by the ListMultipartUploads method is set to false by default (#3682 ) * simplify a bit * fix: IsTruncated returned by the ListMultipartUploads method is set to false by default	2022-09-15 00:27:15 -07:00
chrislu	21c0587900	go fmt	2022-09-14 23:06:44 -07:00
chrislu	c8645fd232	master: implement grpc VolumeMarkWritable fix https://github.com/seaweedfs/seaweedfs/issues/3657	2022-09-14 23:05:30 -07:00
Ryan Russell	c30f6abb11	refactor(command_ec_rebuild): `rebultErr` -> `rebuildErr` (#3679 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 13:10:27 -07:00
Ryan Russell	12914af4d8	Character readability (#3678 ) * refactor(pb): `quote_charactoer` -> `quote_character` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(volume_server): `quoteCharactoer` -> `quoteCharacter` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 13:09:53 -07:00
Ryan Russell	3429cbe3fc	refactor(webdav_server): `modifiledTime` -> `modifiedTime` (#3676 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:20:27 -07:00
Ryan Russell	d5239ab983	refactor(server): `suppported` -> `supported` (#3677 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:16:39 -07:00
Ryan Russell	bd2dc6d641	refactor(shell): `Decending` -> `Descending` (#3675 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:06:48 -07:00
Ryan Russell	dfbd8efd26	refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcInde… (#3674 ) refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:02:33 -07:00
Ryan Russell	f2dde2b1da	refactor(ItemList): `Incluseive` -> `Inclusive` (#3673 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:00:16 -07:00
Ryan Russell	8efe1db01a	refactor(various): `Listner` -> `Listener` readability improvements (#3672 ) * refactor(net_timeout): `listner` -> `listener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(filer): `localPublicListner` -> `localPublicListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(command): `masterLocalListner` -> `masterLocalListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(net_timeout): `ipListner` -> `ipListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:59:55 -07:00
Ryan Russell	a8d7615eec	refactor(filer_client): `EachEntryFunciton` -> `EachEntryFunction` (#3671 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:45 -07:00
Ryan Russell	c4363c4b5e	refactor(command_fs_rm): `entiries` -> `entries` (#3670 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:27 -07:00
Ryan Russell	a36d7b77be	refactor(cipher_test): `plantext` -> `plaintext` (#3669 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:09 -07:00
Ryan Russell	72d8a9f9a8	refactor(exclusive_locker): `Interval` readability batch of updates (#3668 ) * refactor(filechunk_manifest): `localProcesed` -> `localProcessed` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `SafeRenewInteval` -> `SafeRenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `InitLockInteval` -> `InitLockInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `RenewInteval` -> `RenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 10:29:55 -07:00
Ryan Russell	e22335ba78	refactor(queue_unbounded): `inbountLen` -> `inboundLen` (#3666 )	2022-09-14 10:12:45 -07:00
Ryan Russell	d54eb9966f	refactor: `Directory` readability (#3665 )	2022-09-14 10:11:31 -07:00
Ryan Russell	d734fff322	docs: `replicte` -> `replicate` (#3664 )	2022-09-14 10:01:18 -07:00
Ryan Russell	dfaa602661	refactor(notification_kafka): `parition` -> `partition` (#3663 )	2022-09-14 09:15:21 -07:00
Ryan Russell	19652c1b83	refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662 )	2022-09-14 09:14:44 -07:00
Ryan Russell	b6a1b84a00	docs: `orignial` -> `original` (#3661 )	2022-09-14 09:13:59 -07:00
LHHDZ	6d2000e8da	fix 'localFilerSocket' nil pointer (#3654 )	2022-09-14 06:33:00 -07:00
Ryan Russell	c9f573eed8	docs: `indicates` readability comment fix (#3659 )	2022-09-14 06:31:07 -07:00
Ryan Russell	2c92a9ff74	refactor: `DefaultConcurrencyLimit` var rename (#3658 )	2022-09-14 06:30:32 -07:00
Ryan Russell	60fa26ef45	refactor: `concurrentFiles` and `concurrentChunks` var name fix (#3655 )	2022-09-14 04:53:04 -07:00
chrislu	58d18b68d8	rename variable	2022-09-14 00:06:54 -07:00
Konstantin Lebedev	f8ef25099c	Removing chunks on failed to write to replicas (#3591 ) * Removing chunks on failed to write to replicas https://github.com/seaweedfs/seaweedfs/issues/3578 * put with in the util.Retry * just purge on any errors	2022-09-13 22:33:35 -07:00
chrislu	0baf3d38c2	adjust logs	2022-09-13 10:33:28 -07:00
chrislu	ce55922884	fixing	2022-09-12 09:43:54 -07:00
Konstantin Lebedev	721c6197f9	skip deltaBeat if dn is zero (#3630 ) * skip deltaBeat https://github.com/seaweedfs/seaweedfs/issues/3629 * fix GrpcPort * skip url :0 * skip empty DataCenter or Rack * skip empty heartbeat Ip * dell msg add DataCenter * comment todo * fix	2022-09-11 22:31:53 -07:00
chrislu	b834027c5a	refactor	2022-09-11 21:53:15 -07:00
chrislu	d8ca7d34fe	3.27	2022-09-11 19:47:53 -07:00
chrislu	22064c3425	mount: ensure ordered file handle lock and unlock	2022-09-11 19:44:34 -07:00
chrislu	b9112747b5	volume server: synchronously report volume readonly status to master fix https://github.com/seaweedfs/seaweedfs/issues/3628	2022-09-11 19:29:10 -07:00
chrislu	6690236754	rename variables	2022-09-11 19:18:42 -07:00
chrislu	4af8c427eb	refactor	2022-09-10 16:56:03 -07:00
chrislu	4957d8eec6	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-10 15:36:02 -07:00
chrislu	b1501f61dd	Revert "a new test case" This reverts commit `eb7cf3de81`.	2022-09-10 15:35:42 -07:00
Eric Yang	ddd6bee970	ADHOC: Volume fsck use a time cutoff param (#3626 ) * ADHOC: cut off volumn fsck * more * fix typo * add test * modify name * fix comment * fix comments * nit * fix typo * Update weed/shell/command_volume_fsck.go Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-09-10 15:29:17 -07:00
chrislu	2c6b68b40e	simplify a bit	2022-09-10 14:18:28 -07:00
Konstantin Lebedev	b64674018a	[sync] override amz storage class, None to delete (#3639 ) * override amz storage class, None to delete https://github.com/seaweedfs/seaweedfs/issues/3636 * use empty string to delete * without nil check	2022-09-10 14:15:42 -07:00
chrislu	3cb914f7e1	avoid dead lock	2022-09-10 11:26:19 -07:00
chrislu	ebef579ebb	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-10 11:22:18 -07:00
chrislu	576c113c59	replace PR https://github.com/seaweedfs/seaweedfs/pull/3621 replace https://github.com/seaweedfs/seaweedfs/pull/3621	2022-09-10 11:22:16 -07:00
Brian	4e3e2b1b82	Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640 )	2022-09-10 08:05:28 -07:00
chrislu	205ecb5d03	skip directories under ".uploads" directory related to https://github.com/seaweedfs/seaweedfs/issues/3636 skipping all directories under ".uploads" directory.	2022-09-09 22:57:38 -07:00
chrislu	fc4208d128	volume.balance: default to balance ALL_COLLECTIONS	2022-09-09 16:06:05 -07:00
gfx	48db56ddad	arangodb s3 bucket name compatibility (#3588 ) * Update arangodb_store.go * update readme, properly escape queries, add name patching * use underscore * use underscore * better comment * fix readme Co-authored-by: a <a@a.a>	2022-09-09 09:43:42 -07:00
VictorSFF	10d545060f	mount: fix new inode allocation (#3633 )	2022-09-09 06:32:19 -07:00
famosss	41af5a6ee5	simplify a bit (#3631 )	2022-09-08 21:01:21 -07:00
chrislu	64e75a286e	simplify a bit	2022-09-08 19:07:57 -07:00
chrislu	d14d029c73	reduce v.dataFileAccessLock lock scope	2022-09-08 19:05:31 -07:00
chrislu	406a80da4b	adjust variable names	2022-09-08 18:54:16 -07:00
chrislu	2bfc8970d2	refactor: move ReadNeedleDataInto into volume_read.go	2022-09-08 18:54:02 -07:00
chrislu	9b084d4c88	purge tcp implementation	2022-09-08 18:03:43 -07:00
chrislu	eb7cf3de81	a new test case	2022-09-08 14:04:42 -07:00
chrislu	4260804613	volume.balance: avoid moving out volume with max=1	2022-09-08 14:04:21 -07:00
chrislu	d653c5f811	unused	2022-09-08 12:27:53 -07:00
chrislu	03e22ddb43	rename	2022-09-08 12:06:03 -07:00
Konstantin Lebedev	31d2f77ceb	refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625 )	2022-09-07 23:23:33 -07:00
chrislu	7de112943b	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-07 20:14:11 -07:00
chrislu	7e52b1e195	return err for processRangeRequest()	2022-09-07 20:14:09 -07:00
famosss	449582343f	fix:Sometimes a nil pointer exception is thrown (#3618 )	2022-09-07 18:57:13 -07:00
Konstantin Lebedev	695e3a909c	avoid data race on currentDiskUsage.maxVolumeCount (#3592 ) https://github.com/seaweedfs/seaweedfs/issues/3590	2022-09-07 09:51:15 -07:00
Konstantin Lebedev	cca45b02a2	avoid data race on calc freeVolumeSlotCount (#3594 ) https://github.com/seaweedfs/seaweedfs/issues/3593	2022-09-07 09:50:17 -07:00
famosss	9678fc2106	fix: volume heartbeat processing error (#3616 )	2022-09-07 09:48:51 -07:00
famosss	5ff33eb558	fix: volume Deadlock when exception is thrown (#3613 ) * fix: volume Deadlock when exception is thrown * adjust log level	2022-09-06 23:56:29 -07:00
Eric Yang	b324a6536c	ADHOC: add read needle meta grpc (#3581 ) * ADHOC: add read needle meta grpc * add test * nit Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-09-06 23:51:27 -07:00
chrislu	c07ab9c060	3.26	2022-09-06 08:26:20 -07:00
Konstantin Lebedev	f4f6855c61	avoid SIGSEGV on localFilerSocket (#3609 ) https://github.com/seaweedfs/seaweedfs/issues/3607	2022-09-06 08:20:23 -07:00
Konstantin Lebedev	3c3682fcce	more log detail for upload err and deleting (#3577 )	2022-09-06 08:19:13 -07:00
Konstantin Lebedev	01a25755c5	[filer] masterClient updates volume adjust log level (#3610 )	2022-09-06 07:26:59 -07:00
LHHDZ	bc629665de	fix bug due to data racing on VidMap (#3606 )	2022-09-05 20:00:16 -07:00
Konstantin Lebedev	9b2b7d4f5a	avoid data race on GoCDKPubSub.topic (#3596 )	2022-09-05 07:57:44 -07:00
Konstantin Lebedev	916673ae24	do not print diskInfos with doVolumeCheckDisk in error log (#3598 )	2022-09-05 07:49:26 -07:00
Konstantin Lebedev	5307cadb32	[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597 )	2022-09-05 07:47:43 -07:00
chrislu	5b38f22e6e	3.25	2022-09-04 22:45:55 -07:00
famosss	131d389fc4	adjust log level (#3589 )	2022-09-04 22:37:32 -07:00
chrislu	f4abefc109	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-04 22:23:31 -07:00
chrislu	39340f7e42	cloud tier: s3 consume all read response body fix https://github.com/seaweedfs/seaweedfs/issues/3584	2022-09-04 22:23:28 -07:00
chrislu	7c6324b114	adjust log level	2022-09-04 22:21:24 -07:00
gfx	bf844d8e46	Fix crash in arangodb filer when attempting to access a deleted bucket (#3587 ) * Update helpers.go * Update arangodb_store_bucket.go	2022-09-04 20:32:10 -07:00
gfx	3621987727	Correctly pass arangodb driver Transaction ID into the context (#3586 ) Update arangodb_store.go	2022-09-04 19:29:52 -07:00
chrislu	8c5759a06d	minor optimization	2022-09-04 18:50:45 -07:00
chrislu	70777d9b9f	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-04 18:26:51 -07:00
chrislu	cb6cf331ca	filer.backup and filer.sync: include headers during backup and sync fix https://github.com/seaweedfs/seaweedfs/issues/3532	2022-09-04 18:26:36 -07:00
famosss	7f56a40553	Turn down the 404 log level (#3585 )	2022-09-04 18:17:01 -07:00
Brian	a28b668647	Added ability to change replication settings upon volume.tier.move (#3583 )	2022-09-04 16:47:21 -07:00
bernardx	228b133afa	new 'concurrency' parameter for filer.sync (#3579 ) Co-authored-by: XIAOYQ <xiaoyq@eudic.net>	2022-09-02 23:03:23 -07:00
Konstantin Lebedev	853880bd83	[filer] DATA RACE on signal_handling fixed (#3575 )	2022-09-02 06:20:19 -07:00
chrislu	a112cbe44c	filer: rocksdb store report not found correctly fix https://github.com/seaweedfs/seaweedfs/discussions/3555	2022-09-01 15:42:59 -07:00
chrislu	d1bb23645e	Revert "avoid data race on grace.hooks (#3572 )" This reverts commit `c37d6fc01a`.	2022-09-01 13:08:34 -07:00
Konstantin Lebedev	c37d6fc01a	avoid data race on grace.hooks (#3572 ) https://github.com/seaweedfs/seaweedfs/issues/3564	2022-09-01 10:34:06 -07:00
Konstantin Lebedev	8c3040db81	avoid DATA RACE on S3Options.localFilerSocket (#3571 ) * avoid DATA RACE on S3Options.localFilerSocket https://github.com/seaweedfs/seaweedfs/issues/3552 * copy localSocket	2022-09-01 10:33:23 -07:00
Konstantin Lebedev	90d55cd179	avoid data race read fileChunksSize (#3570 ) https://github.com/seaweedfs/seaweedfs/issues/3557	2022-09-01 10:31:57 -07:00
Konstantin Lebedev	82c2ce74db	more error details in the log (#3568 ) * more error details in the log https://github.com/seaweedfs/seaweedfs/issues/3567 * format message	2022-09-01 09:04:15 -07:00
chrislu	10efdc7aab	align memory	2022-09-01 09:02:44 -07:00
Konstantin Lebedev	803ca3c958	avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566 ) https://github.com/seaweedfs/seaweedfs/issues/3565	2022-09-01 09:01:47 -07:00
chrislu	bcf35876d1	add more logs	2022-08-31 23:16:05 -07:00
dependabot[bot]	97d69d5336	Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541 ) * Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * update code * more code fix Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>	2022-08-31 10:16:49 -07:00
Patrick Schmidt	a73e177ecf	Add an End-to-End workflow for FUSE mount (#3562 ) * Add an e2e workflow to test FUSE mount * Fix deadlocks during concurrent r/w	2022-08-31 09:27:53 -07:00
chrislu	4a4ef3cc3c	upload_content: upload close response.Body	2022-08-31 00:25:28 -07:00
chrislu	3d8b8ffced	refactor	2022-08-31 00:24:29 -07:00
chrislu	ef78631a7c	just a bit safer	2022-08-31 00:10:17 -07:00
chrislu	9c944377dc	minor	2022-08-31 00:09:32 -07:00
chrislu	a54f30b6c6	s3: close response body with copying object	2022-08-31 00:09:23 -07:00
chrislu	7eb121d5ae	print process before do the work	2022-08-30 13:37:05 -07:00
Konstantin Lebedev	77098de69a	migrate from boltdb to bbolt (#3554 )	2022-08-30 10:12:02 -07:00
chrislu	de5263364b	master client: use GetMaster() for latest master address	2022-08-30 09:52:06 -07:00
chrislu	e583dbdd2a	more detailed error	2022-08-30 09:43:17 -07:00
chrislu	d81db3c703	s3: fix configuring IAM for the same user hi, how can I add bucket permission to a user now? Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name. Now I am getting error: duplicate accessKey[ХХХХ], already configured in user[YYYY] s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1 s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1	2022-08-30 09:37:52 -07:00
Konstantin Lebedev	ade94b0d0a	avoid race conditions access to SuperBlock.Version (#3539 ) * avoid race conditions access to SuperBlock.Version https://github.com/seaweedfs/seaweedfs/issues/3515 * superBlockAccessLock replace to sync.Mutex	2022-08-30 00:08:00 -07:00
chrislu	ae6292f9f0	rename variable	2022-08-30 00:07:15 -07:00
Konstantin Lebedev	e16dda88e4	avoid race conditions access to MasterClient.currentMaster (#3538 ) https://github.com/seaweedfs/seaweedfs/issues/3510	2022-08-30 00:05:38 -07:00
Konstantin Lebedev	105702ebe0	avoid data race access to uploadReaderToChunks.uploadErr (#3550 ) avoid data race access to uploadErr https://github.com/seaweedfs/seaweedfs/issues/3549	2022-08-30 00:03:01 -07:00
Konstantin Lebedev	903a06a160	avoid data race access to startDelay (#3548 )	2022-08-30 00:02:09 -07:00
chrislu	570b488572	ensure memory alignment on different CPUs	2022-08-29 13:24:12 -07:00
Konstantin Lebedev	4966a3abc7	avoid race conditions access to growRequestCount (#3537 ) https://github.com/seaweedfs/seaweedfs/issues/3511	2022-08-29 13:23:02 -07:00
Patrick Schmidt	00041f6e08	Fix corrupted mount reads after changes on fh (#3536 )	2022-08-29 08:27:13 -07:00
chrislu	b7a887fea1	3.24	2022-08-28 21:53:13 -07:00
Rain Li	089446f5b1	Fix reload page before all file uploaded bug (#3534 )	2022-08-28 07:51:47 -07:00
chrislu	cc0c8c5f81	simplify	2022-08-27 00:21:57 -07:00
chrislu	3741779144	minor	2022-08-27 00:20:58 -07:00
chrislu	580a7fa685	adjust for data race fix https://github.com/seaweedfs/seaweedfs/issues/3512	2022-08-27 00:19:13 -07:00
chrislu	87b70a6809	clean up	2022-08-27 00:09:04 -07:00
chrislu	27c7611ebf	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 23:47:39 -07:00
chrislu	9920d65bc0	gateway to remote object store: adjust upload concurrency	2022-08-26 23:47:37 -07:00
chrislu	c839ce1b19	s3 sink use s3 upload manager fix https://github.com/seaweedfs/seaweedfs/issues/3531	2022-08-26 23:47:12 -07:00
famosss	dc4037925d	fix: Build DeletedVids before reset dn's children (#3530 )	2022-08-26 22:52:08 -07:00
chrislu	3f3a1341d8	make CodeQL happy	2022-08-26 17:09:11 -07:00
chrislu	301b49b63f	atomic operation	2022-08-26 17:04:11 -07:00
chrislu	57a46f46a0	nano level precision	2022-08-26 16:55:15 -07:00
chrislu	8dae81c5ed	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 16:49:51 -07:00
chrislu	e0f4366f4c	more detailed error related to https://github.com/seaweedfs/seaweedfs/issues/3528	2022-08-26 16:49:48 -07:00
Patrick Schmidt	5df105b1f9	Fix a few data races when reading files in mount (#3527 )	2022-08-26 16:41:37 -07:00
chrislu	f5156cf3a8	rename varaible	2022-08-26 10:23:42 -07:00
Konstantin Lebedev	e90ab4ac60	avoid race conditions for OnPeerUpdate (#3525 ) https://github.com/seaweedfs/seaweedfs/issues/3524	2022-08-26 10:18:49 -07:00
Konstantin Lebedev	4f7a1f67cd	avoid race conditions for diskLocation.MaxVolumeCount (#3526 )	2022-08-26 08:41:42 -07:00
chrislu	6f483a4d36	atomic add fix https://github.com/seaweedfs/seaweedfs/issues/3514	2022-08-25 22:20:34 -07:00
chrislu	7394f7feee	minor: adjust ordering	2022-08-25 00:20:23 -07:00
chrislu	dbf0de4ce1	minor clean up	2022-08-25 00:19:08 -07:00
Patrick Schmidt	2930263dfd	Fix race conditions during in-flight size checks (#3505 )	2022-08-24 20:03:34 -07:00
Patrick Schmidt	7b424a54dc	Add raft server access mutex to avoid races (#3503 )	2022-08-24 09:49:05 -07:00
Konstantin Lebedev	f7aeb06544	s3: report metadata if the directory is explicitly created (#3498 ) * replace mkdir to mkFile * ContentLength must be zero * revert mkDir * Seaweedfs-Is-Directory-Key return metadata	2022-08-24 00:15:44 -07:00
Guo Lei	c57c79a0ab	optimiz commitig compact (#3388 ) * optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index	2022-08-23 23:53:35 -07:00
chrislu	10414fd81c	ping timeout at 15 seconds this 72 minute timeout setting seems unreasonably long 15 seconds is around the time when a new raft leader should be elected.	2022-08-23 23:28:16 -07:00
askeipx	2e78a522ab	remove old raft servers if they don't answer to pings for too long (#3398 ) * remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix	2022-08-23 23:18:21 -07:00
Guo Lei	762dc219a6	fix deletecount bug (#3499 )	2022-08-23 08:53:24 -07:00
chrislu	21a933c932	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-23 01:52:32 -07:00
chrislu	f0b4a7659a	fix test	2022-08-23 01:52:29 -07:00
Patrick Schmidt	cda8cc22bc	Implement lseek syscall in FUSE (#3491 ) See the man page of lseek: https://man7.org/linux/man-pages/man2/lseek.2.html	2022-08-23 01:47:18 -07:00
chrislu	bf88006037	format	2022-08-23 01:20:45 -07:00
chrislu	9fce75607d	s3: report http.StatusOK if the directory is explicitly created fix https://github.com/seaweedfs/seaweedfs/issues/3457	2022-08-23 01:16:46 -07:00
famosss	911475526c	fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497 )	2022-08-22 23:54:51 -07:00
famosss	7eb15b1969	fix:Handle preflight cors requests (#3496 )	2022-08-22 23:39:34 -07:00
chrislu	676e27c589	shell: stop long running jobs if lock is lost	2022-08-22 14:12:23 -07:00
chrislu	601ba5fb68	master: disable locking if not leader	2022-08-22 14:11:48 -07:00
chrislu	57e7582c36	refactoring	2022-08-22 14:11:13 -07:00
chrislu	7ea2c24c5d	metadata processing use retry instead of retryForEver	2022-08-22 11:25:40 -07:00
Patrick Schmidt	f875031f06	Reuse readDataByFileHandle in Read call (#3482 )	2022-08-22 08:24:06 -07:00
famosss	bdba3da2e4	Handle preflight cors requests (#3481 )	2022-08-22 08:21:38 -07:00
chrislu	f4b52d4c52	fix format	2022-08-22 00:20:21 -07:00
chrislu	b4131a3a72	filer.remote.sync: parallelize uploading to remove object storage	2022-08-22 00:01:48 -07:00
chrislu	f6b92b8561	refactor	2022-08-22 00:00:51 -07:00
chrislu	3c8e95eeb3	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-21 23:26:09 -07:00
chrislu	8bac903d29	mount: add locking to prevent nil pointer fix https://github.com/seaweedfs/seaweedfs/issues/3479	2022-08-21 23:26:06 -07:00
qzh	74b53729e1	feat(weed.move): add a speed limit parameter of moving files (#3478 ) * feat(weed.move): add a speed limit parameter of moving files * fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-21 23:08:31 -07:00
chrislu	7807f6641a	minor	2022-08-21 22:54:56 -07:00
chrislu	c4e862e908	3.23	2022-08-21 19:13:26 -07:00
Guo Lei	81e7e6ceb3	the statistical methord for fileCount should stay same (#3477 ) the statistical methord for fileCount between should stay same They all equal to the entry count in .idx file. relate to commit(`c7892bc`)	2022-08-21 19:11:18 -07:00
chrislu	63fbf281c7	remove same file copying rage limitation	2022-08-21 17:16:21 -07:00
Patrick Schmidt	3a75d7f7aa	Implement copy_file_range syscall in FUSE (#3475 ) See the man page of copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html	2022-08-21 17:11:43 -07:00
chrislu	fef9c6a520	filer: do not always save files in "/etc" folder to filer store fix https://github.com/seaweedfs/seaweedfs/issues/3476	2022-08-21 17:02:16 -07:00
chrislu	2762154130	fix compilation	2022-08-21 16:09:23 -07:00
chrislu	fb5808e0c3	EC: with multiple volume locations, the ec rebuilding may fail	2022-08-21 15:04:50 -07:00
chrislu	5790d01c6f	EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations related to https://github.com/seaweedfs/seaweedfs/issues/3459	2022-08-21 14:52:37 -07:00
chrislu	8b3429858d	align memory for atomic read/write	2022-08-21 12:20:27 -07:00
Patrick Schmidt	2ef6ab998c	Avoid race conditions with current filer address (#3474 ) When multiple filer requests are in-flight and the current filer disappears and a new one is selected by the first goroutine, then there can be a lot of race conditions while retrieving the current filer. Therefore, load/save the current filer index atomically.	2022-08-21 12:18:13 -07:00
Patrick Schmidt	f49a9297c2	Fix hanging reads in chunk cacher (#3473 ) Sometimes when an unexpected error occurs the cacher would set an error and return. However, it would not broadcast the condition signal in that case, therefore leaving the goroutine that runs readChunkAt stuck forever. I figured that the condition is unnecessary because readChunkAt is acquiring a lock that is still held by the cacher goroutine anyway. Callees of startCaching have to wait for a WaitGroup which makes sure that readChunkAt can't acquire the lock before startCaching. This way readChunkAt can execute normally and check for the error.	2022-08-21 11:54:02 -07:00
chrislu	388f82f322	minor	2022-08-21 11:49:29 -07:00
chrislu	77e4b1376e	refactoring	2022-08-21 11:35:54 -07:00
Patrick Schmidt	3f758820c1	Fix FUSE server buffer leaks in file gaps (#3472 ) * Fix FUSE server buffer leaks in file gaps This change zeros read buffers when encountering file gaps during file/chunk reads in FUSE mounts. It prevents leaking internal buffers of the FUSE server which could otherwise reveal metadata, directory listings, file contents and other data related to FUSE API calls. The issue was that buffers are reused, but when a file gap was found the buffer was not zeroed accordingly and the existing data of the buffer was kept and returned. * Move zero logic into its own method	2022-08-21 11:33:58 -07:00
chrislu	c7892bc7c4	volume: file counter should be all files address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228	2022-08-20 23:35:31 -07:00
chrislu	f7e0a65e75	retry for all errors	2022-08-20 23:34:45 -07:00
chrislu	649555b223	adjust log	2022-08-20 23:34:07 -07:00
chrislu	973f6dd162	refactoring	2022-08-20 22:38:15 -07:00
chrislu	65ff7198fe	adjust type	2022-08-20 22:37:24 -07:00
chrislu	28b862f45f	use util.Retry to retry	2022-08-20 22:03:27 -07:00
chrislu	3bf8e772f8	webdav: retryable data chunk upload	2022-08-20 19:18:12 -07:00
chrislu	f8fa430257	filer: retryable data chunk upload	2022-08-20 19:15:44 -07:00
chrislu	4081d50607	filer sink: retryable data chunk uploading	2022-08-20 19:09:15 -07:00
chrislu	409f39390d	mount: retryable data chunk uploading	2022-08-20 19:04:36 -07:00
chrislu	6c8822f269	filer.copy: retryable file part upload	2022-08-20 18:59:57 -07:00
chrislu	a3553da7f7	add fileId to UploadWithRetry return result	2022-08-20 18:54:59 -07:00
chrislu	d49d0a9fc2	filer.copy: retryable upload	2022-08-20 18:50:57 -07:00
chrislu	689b4ecdcc	remove unused collection and replication from upload result	2022-08-20 18:14:57 -07:00
chrislu	e3f40d538d	cleaner code	2022-08-20 17:51:30 -07:00
chrislu	9f50b310a0	remove unused code	2022-08-20 17:51:12 -07:00
chrislu	11f99836c3	filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0) fix https://github.com/seaweedfs/seaweedfs/issues/3468	2022-08-19 23:00:56 -07:00
chrislu	fdd8c5d5e0	Filer if-modified-since check doesn't fire on "exact" #3467 fix https://github.com/seaweedfs/seaweedfs/issues/3467	2022-08-19 22:21:51 -07:00
chrislu	aac45f3e89	filer: retryable when error is not found	2022-08-19 03:36:15 -07:00
chrislu	58dd880322	minor	2022-08-18 23:51:08 -07:00
Eric Yang	b6f99c4d83	fix naming typo (#3465 ) Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-08-18 23:25:32 -07:00
chrislu	5702727db5	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-18 02:39:27 -07:00
chrislu	4d72a1ce6e	s3: remove deleted collection during metrics reporting fix https://github.com/seaweedfs/seaweedfs/issues/3453	2022-08-18 02:39:07 -07:00
Guo Lei	3172c33fa6	fix redundant type from array (#3462 )	2022-08-18 02:25:25 -07:00
chrislu	42c6e52513	s3: fix regression on HEAD directory operation	2022-08-18 02:13:58 -07:00
chrislu	4573c99ae3	fix tests	2022-08-18 00:15:46 -07:00
chrislu	2b580a7566	also migrate jsonpb	2022-08-17 12:42:03 -07:00
chrislu	eaeb141b09	move proto package	2022-08-17 12:05:07 -07:00
chrislu	e259052a70	volume: fix metrics for volume request time	2022-08-17 04:29:31 -07:00
qzh	7fcfaf7bc9	fix(filer.sync): offset may be set to 0 (#3451 ) * fix(filer.sync): initializing the offset is related to the path * fix(filer.sync): the offset maybe to be set to 0. Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 23:43:52 -07:00
qzh	400f0c3e5d	fix(filer.sync): initializing the offset is related to the path (#3450 ) Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 21:56:47 -07:00
chrislu	fa4d0093e1	3.22	2022-08-15 16:48:23 -07:00
chrislu	93261f5199	Revert "Refactor for Sync method (#3426 )" This reverts commit `670cb759f8`. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]	2022-08-15 15:35:31 -07:00
chrislu	9c588d4010	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-15 10:03:56 -07:00
chrislu	96caf21d09	less verbose log	2022-08-15 10:03:52 -07:00
Andrey Triumfov	31faa6d43d	Remove duplicate slashes in object path to prevent 500 errors (#3442 )	2022-08-15 08:19:28 -07:00
chrislu	cb476a53ff	remove logs	2022-08-15 01:05:35 -07:00

... 11 12 13 14 15 ...

6822 Commits