seaweedfs

mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-23 17:07:57 +08:00

Author	SHA1	Message	Date
Guo Lei	84c401e693	Optimiz leveldb metric (#3830 ) * optimiz updating mapmetric for leveldb * import loading leveldb * add comments	2022-10-11 21:13:25 -07:00
LHHDZ	d21e2f523d	split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unn… (#3824 ) split `ExtAcpKey` to `ExtAmzOwnerKey` and `ExtAmzAclKey` to avoid unnecessary `json.Unmarshal()` call Signed-off-by: changlin.shi <changlin.shi@ly.com> Signed-off-by: changlin.shi <changlin.shi@ly.com>	2022-10-11 20:14:14 -07:00
Konstantin Lebedev	5b28c3f728	revert disabling FSync for non Mac (#3814 )	2022-10-10 07:28:02 -07:00
LHHDZ	3550692afc	change s3_account.go package to avoid cycle dependency (#3813 )	2022-10-10 06:44:29 -07:00
chrislu	0711870f43	3.31	2022-10-10 00:15:30 -07:00
chrislu	109dc7fdf6	mount: remove mount directory from generating cache directory id fix https://github.com/seaweedfs/seaweedfs/discussions/3811	2022-10-10 00:12:10 -07:00
chrislu	fc54afdba9	tests add parsing ec shard info	2022-10-09 22:47:55 -07:00
chrislu	0623bf582e	include ec shard for capacityByFreeVolumeCount	2022-10-09 22:31:30 -07:00
chrislu	98dc1e5c15	move volume: find target volume server by exiting/max ratio	2022-10-09 22:30:59 -07:00
chrislu	049f040c3c	refactor	2022-10-09 22:28:39 -07:00
chrislu	124b97f6d7	parsing ec volumes	2022-10-09 19:21:23 -07:00
chrislu	1e27d36576	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-10-09 18:58:14 -07:00
chrislu	f9383aa726	refactor to change capacity data type	2022-10-09 18:58:10 -07:00
binbinshi	b7de4a967e	fix: compact_map get error mismatching cokie (#3748 ) * fix: compact_map get error * fix: CompactSection delete lock and move test to compact_map Co-authored-by: shibinbin <shibinbin@megvii.com>	2022-10-09 16:56:40 -07:00
BakerBunker	b9d8a837af	Fix `mount -t weed` with extra options (#3807 ) fuse: Ignore "-o" command	2022-10-08 11:25:19 -07:00
Konstantin Lebedev	5db25a8f2a	metric shows who is currently blocking the cluster or not (#3799 ) * master_admin_lock Shows whether cluster is locked now or not https://github.com/seaweedfs/seaweedfs/issues/3452 * fix metric MasterAdminLock	2022-10-07 13:26:29 -07:00
Viktor Kuzmin	f8d3ff466d	Allow parallel volume loading from different dirs during startup. (#3802 )	2022-10-07 11:27:15 -07:00
Konstantin Lebedev	a522507f95	configure raft metrics (#3798 )	2022-10-07 04:20:34 -07:00
Konstantin Lebedev	b9933d5589	master server graceful stop (#3797 )	2022-10-06 09:30:30 -07:00
Konstantin Lebedev	36daa7709d	show raft leader via shell (#3796 )	2022-10-06 07:10:41 -07:00
chrislu	ea271600ec	fix parameters	2022-10-04 12:36:05 -07:00
chrislu	0452ae6a6c	filer.sync: limit concurrency when fetching file chunks fix https://github.com/seaweedfs/seaweedfs/issues/3787	2022-10-04 11:35:07 -07:00
Konstantin Lebedev	ec46a34f33	avoid increasing the coordinates of metric labels (#3785 )	2022-10-04 01:20:01 -07:00
chrislu	29632d5a34	3.30	2022-10-02 18:13:44 -07:00
LHHDZ	e9584d9661	add ownership rest apis (#3765 )	2022-10-01 19:18:00 -07:00
Konstantin Lebedev	6fa3d0cc46	fix delete key panic of remote sync dir (#3770 )	2022-10-01 02:33:47 -07:00
chrislu	c3487d3da7	better fix for option format	2022-09-30 08:49:38 -07:00
chrislu	ba0db28dbb	mount: fix extra option format	2022-09-30 08:44:41 -07:00
chrislu	8e81619d02	mount: accept all extra mount options fix https://github.com/seaweedfs/seaweedfs/issues/3767	2022-09-30 08:40:37 -07:00
Eric Yang	56c94cc08e	ADHOC: filter deleted files from idx file binary search (#3763 ) * ADHOC: filter deleted files from idx file binary search * remove unwanted check Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-09-29 12:48:36 -07:00
LHHDZ	3de1e19780	s3: sync bucket info from filer (#3759 )	2022-09-29 12:29:01 -07:00
LHHDZ	e00a12b099	associate `Account` and `Identity` by accountId (#3754 )	2022-09-28 13:25:59 -07:00
LHHDZ	aacdcc4cad	s3: add account (#3753 ) associate `Account` and `Identity` by accountId	2022-09-28 12:45:18 -07:00
Konstantin Lebedev	301b678147	[volume] Add new volumes to HUP(reload) signal (#3755 ) Add new volumes to HUP(reload) signal	2022-09-28 12:44:13 -07:00
chrislu	47fd5d3fa1	skip debug message	2022-09-27 13:12:31 -07:00
Konstantin Lebedev	7ae51d1ec6	fix set master options for single server (#3708 ) Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-09-27 12:06:35 -07:00
jerebear12	190afcc3fd	Allow CORS on Filer (#3750 ) Adjusted filer response headers	2022-09-27 11:47:52 -07:00
Konstantin Lebedev	faa6167b6b	fs.meta.load load any dirs with prefix "important" (#3747 ) * fs.meta.load load any dirs with prefix "important" * replace dirPattern to dirPrefix * help dirPrefix	2022-09-27 09:58:46 -07:00
chrislu	b6d7556dda	skip truncation on error fix https://github.com/seaweedfs/seaweedfs/issues/3746	2022-09-27 09:48:23 -07:00
chrislu	1ffb1e696e	3.29	2022-09-25 19:20:45 -07:00
chrislu	640c53cda8	add future list	2022-09-25 17:43:42 -07:00
chrislu	cc570a4477	assert results are sorted	2022-09-25 13:49:28 -07:00
chrislu	5c8f1467a1	ordered execution async wait	2022-09-25 13:45:55 -07:00
chrislu	973b8ff067	rename	2022-09-25 13:34:55 -07:00
chrislu	e31fdbc89b	rename file	2022-09-25 12:59:38 -07:00
chrislu	31922b2bf2	s3 to watch specific directories	2022-09-20 09:30:05 -07:00
chrislu	b90d92fca9	grpc: watch metadata changes by directory	2022-09-20 09:25:18 -07:00
chrislu	b463ca1a2f	filer replication: compare content changes directly Fix https://github.com/seaweedfs/seaweedfs/issues/3714 The destination chunks may be empty. For example, the file is updated and the volume is vacuumed. In this case, the sync would miss the old chunks. This is fine. However, the entry would have correct metadata but missing chunks. For this case, the simple metadata comparison would be wrongly skipping data changes, and the file will stay empty unless file content md5 is changed.	2022-09-20 08:35:10 -07:00
chrislu	b2a171632b	mount: fix 0 file size in linux	2022-09-18 21:42:13 -07:00
chrislu	23f3aea4cf	adjust logs	2022-09-18 18:14:24 -07:00
chrislu	901e8157de	3.28	2022-09-18 17:46:04 -07:00
chrislu	9d230521a4	report write error	2022-09-18 16:49:48 -07:00
chrislu	10d5b4b32b	volume server: rename readBufferSize to readBufferSizeMB	2022-09-17 10:56:28 -07:00
chrislu	3fc261d27c	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-16 23:54:39 -07:00
chrislu	06748f2771	remove unused code	2022-09-16 23:54:37 -07:00
Konstantin Lebedev	5d87ad72d8	mute log filer: no entry is found in filer store (#3707 )	2022-09-16 09:05:38 -07:00
Ryan Russell	824f7ad9e1	refactor(shell): readability improvements (#3704 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-16 02:43:49 -07:00
Ryan Russell	277976bd76	refactor(storage): readability improvements (#3703 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-16 02:43:17 -07:00
famosss	d949a238b8	volume: add "readBufSize" option to customize read optimization (#3702 ) * simplify a bit * feat: volume: add "readBufSize" option to customize read optimization * refactor : redbufSIze -> readBufferSize * simplify a bit * simplify a bit	2022-09-16 00:30:40 -07:00
chrislu	cf90f76a35	mark "hasSlowRead" as experimental	2022-09-15 23:33:46 -07:00
Ryan Russell	6f7ef8711a	docs(s3api): readability improvements (#3696 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:13:21 -07:00
Ryan Russell	ad3a3c8782	refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `de… (#3695 ) refactor(s3api_object_handlers): `deleteMultipleObjectsLimmit` -> `deleteMultipleObjectsLimit` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:48 -07:00
Ryan Russell	c4fc007630	refactor(raft_hashicorp): `timeSpeep` -> `timeSleep` (#3694 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:30 -07:00
Ryan Russell	c202d4c1b0	refactor(master_grpc_server_cluster): `seletedSet` -> `selectedSet` (#3693 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 03:12:13 -07:00
chrislu	f9e3e7d1c4	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-15 03:11:34 -07:00
chrislu	896a85d6e4	volume: add "hasSlowRead" option to customize read optimization	2022-09-15 03:11:32 -07:00
Ryan Russell	e0064ae097	docs: update `fileIdsToDelete` var (#3692 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:09:06 -07:00
Ryan Russell	d65bdeef08	docs(meta_cache): readability fixes (#3691 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:04:57 -07:00
Ryan Russell	92658b05a8	refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConne… (#3690 ) refactor(volume_grpc_client_to_master): `grpcConection` -> `grpcConnection` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:04:29 -07:00
Ryan Russell	a635973847	docs(topology): readability improvements (#3688 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 02:03:10 -07:00
Ryan Russell	0fc242b084	docs: `panicing` -> `panicking` (#3687 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:38:04 -07:00
Ryan Russell	f6d391c9da	docs(command): readability fixes (#3686 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:33:04 -07:00
Ryan Russell	150ce9f3f6	docs(wdclient): readability fix (#3685 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-15 01:29:26 -07:00
ImFantuan	b64411bda8	fix 0 size (#3683 )	2022-09-15 01:27:02 -07:00
famosss	bb880d6253	IsTruncated returned by the ListMultipartUploads method is set to false by default (#3682 ) * simplify a bit * fix: IsTruncated returned by the ListMultipartUploads method is set to false by default	2022-09-15 00:27:15 -07:00
chrislu	21c0587900	go fmt	2022-09-14 23:06:44 -07:00
chrislu	c8645fd232	master: implement grpc VolumeMarkWritable fix https://github.com/seaweedfs/seaweedfs/issues/3657	2022-09-14 23:05:30 -07:00
Ryan Russell	c30f6abb11	refactor(command_ec_rebuild): `rebultErr` -> `rebuildErr` (#3679 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 13:10:27 -07:00
Ryan Russell	12914af4d8	Character readability (#3678 ) * refactor(pb): `quote_charactoer` -> `quote_character` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(volume_server): `QuoteCharactoer` -> `QuoteCharacter` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(volume_server): `quoteCharactoer` -> `quoteCharacter` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 13:09:53 -07:00
Ryan Russell	3429cbe3fc	refactor(webdav_server): `modifiledTime` -> `modifiedTime` (#3676 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:20:27 -07:00
Ryan Russell	d5239ab983	refactor(server): `suppported` -> `supported` (#3677 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:16:39 -07:00
Ryan Russell	bd2dc6d641	refactor(shell): `Decending` -> `Descending` (#3675 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:06:48 -07:00
Ryan Russell	dfbd8efd26	refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcInde… (#3674 ) refactor(command_ec_decode): `exisitngEcIndexBits` -> `existingEcIndexBits` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:02:33 -07:00
Ryan Russell	f2dde2b1da	refactor(ItemList): `Incluseive` -> `Inclusive` (#3673 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 12:00:16 -07:00
Ryan Russell	8efe1db01a	refactor(various): `Listner` -> `Listener` readability improvements (#3672 ) * refactor(net_timeout): `listner` -> `listener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(filer): `localPublicListner` -> `localPublicListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(command): `masterLocalListner` -> `masterLocalListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor(net_timeout): `ipListner` -> `ipListener` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:59:55 -07:00
Ryan Russell	a8d7615eec	refactor(filer_client): `EachEntryFunciton` -> `EachEntryFunction` (#3671 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:45 -07:00
Ryan Russell	c4363c4b5e	refactor(command_fs_rm): `entiries` -> `entries` (#3670 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:27 -07:00
Ryan Russell	a36d7b77be	refactor(cipher_test): `plantext` -> `plaintext` (#3669 ) Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 11:24:09 -07:00
Ryan Russell	72d8a9f9a8	refactor(exclusive_locker): `Interval` readability batch of updates (#3668 ) * refactor(filechunk_manifest): `localProcesed` -> `localProcessed` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `saveChunkedFileIntevalToStorage` -> `saveChunkedFileIntervalToStorage` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `SafeRenewInteval` -> `SafeRenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `InitLockInteval` -> `InitLockInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> * refactor: `RenewInteval` -> `RenewInterval` Signed-off-by: Ryan Russell <git@ryanrussell.org> Signed-off-by: Ryan Russell <git@ryanrussell.org>	2022-09-14 10:29:55 -07:00
Ryan Russell	e22335ba78	refactor(queue_unbounded): `inbountLen` -> `inboundLen` (#3666 )	2022-09-14 10:12:45 -07:00
Ryan Russell	d54eb9966f	refactor: `Directory` readability (#3665 )	2022-09-14 10:11:31 -07:00
Ryan Russell	d734fff322	docs: `replicte` -> `replicate` (#3664 )	2022-09-14 10:01:18 -07:00
Ryan Russell	dfaa602661	refactor(notification_kafka): `parition` -> `partition` (#3663 )	2022-09-14 09:15:21 -07:00
Ryan Russell	19652c1b83	refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662 )	2022-09-14 09:14:44 -07:00
Ryan Russell	b6a1b84a00	docs: `orignial` -> `original` (#3661 )	2022-09-14 09:13:59 -07:00
LHHDZ	6d2000e8da	fix 'localFilerSocket' nil pointer (#3654 )	2022-09-14 06:33:00 -07:00
Ryan Russell	c9f573eed8	docs: `indicates` readability comment fix (#3659 )	2022-09-14 06:31:07 -07:00
Ryan Russell	2c92a9ff74	refactor: `DefaultConcurrencyLimit` var rename (#3658 )	2022-09-14 06:30:32 -07:00
Ryan Russell	60fa26ef45	refactor: `concurrentFiles` and `concurrentChunks` var name fix (#3655 )	2022-09-14 04:53:04 -07:00
chrislu	58d18b68d8	rename variable	2022-09-14 00:06:54 -07:00
Konstantin Lebedev	f8ef25099c	Removing chunks on failed to write to replicas (#3591 ) * Removing chunks on failed to write to replicas https://github.com/seaweedfs/seaweedfs/issues/3578 * put with in the util.Retry * just purge on any errors	2022-09-13 22:33:35 -07:00
chrislu	0baf3d38c2	adjust logs	2022-09-13 10:33:28 -07:00
chrislu	ce55922884	fixing	2022-09-12 09:43:54 -07:00
Konstantin Lebedev	721c6197f9	skip deltaBeat if dn is zero (#3630 ) * skip deltaBeat https://github.com/seaweedfs/seaweedfs/issues/3629 * fix GrpcPort * skip url :0 * skip empty DataCenter or Rack * skip empty heartbeat Ip * dell msg add DataCenter * comment todo * fix	2022-09-11 22:31:53 -07:00
chrislu	b834027c5a	refactor	2022-09-11 21:53:15 -07:00
chrislu	d8ca7d34fe	3.27	2022-09-11 19:47:53 -07:00
chrislu	22064c3425	mount: ensure ordered file handle lock and unlock	2022-09-11 19:44:34 -07:00
chrislu	b9112747b5	volume server: synchronously report volume readonly status to master fix https://github.com/seaweedfs/seaweedfs/issues/3628	2022-09-11 19:29:10 -07:00
chrislu	6690236754	rename variables	2022-09-11 19:18:42 -07:00
chrislu	4af8c427eb	refactor	2022-09-10 16:56:03 -07:00
chrislu	4957d8eec6	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-10 15:36:02 -07:00
chrislu	b1501f61dd	Revert "a new test case" This reverts commit `eb7cf3de81`.	2022-09-10 15:35:42 -07:00
Eric Yang	ddd6bee970	ADHOC: Volume fsck use a time cutoff param (#3626 ) * ADHOC: cut off volumn fsck * more * fix typo * add test * modify name * fix comment * fix comments * nit * fix typo * Update weed/shell/command_volume_fsck.go Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local> Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>	2022-09-10 15:29:17 -07:00
chrislu	2c6b68b40e	simplify a bit	2022-09-10 14:18:28 -07:00
Konstantin Lebedev	b64674018a	[sync] override amz storage class, None to delete (#3639 ) * override amz storage class, None to delete https://github.com/seaweedfs/seaweedfs/issues/3636 * use empty string to delete * without nil check	2022-09-10 14:15:42 -07:00
chrislu	3cb914f7e1	avoid dead lock	2022-09-10 11:26:19 -07:00
chrislu	ebef579ebb	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-10 11:22:18 -07:00
chrislu	576c113c59	replace PR https://github.com/seaweedfs/seaweedfs/pull/3621 replace https://github.com/seaweedfs/seaweedfs/pull/3621	2022-09-10 11:22:16 -07:00
Brian	4e3e2b1b82	Add option in volume.fix.replication to only fix under-replication and not delete volumes (#3640 )	2022-09-10 08:05:28 -07:00
chrislu	205ecb5d03	skip directories under ".uploads" directory related to https://github.com/seaweedfs/seaweedfs/issues/3636 skipping all directories under ".uploads" directory.	2022-09-09 22:57:38 -07:00
chrislu	fc4208d128	volume.balance: default to balance ALL_COLLECTIONS	2022-09-09 16:06:05 -07:00
gfx	48db56ddad	arangodb s3 bucket name compatibility (#3588 ) * Update arangodb_store.go * update readme, properly escape queries, add name patching * use underscore * use underscore * better comment * fix readme Co-authored-by: a <a@a.a>	2022-09-09 09:43:42 -07:00
VictorSFF	10d545060f	mount: fix new inode allocation (#3633 )	2022-09-09 06:32:19 -07:00
famosss	41af5a6ee5	simplify a bit (#3631 )	2022-09-08 21:01:21 -07:00
chrislu	64e75a286e	simplify a bit	2022-09-08 19:07:57 -07:00
chrislu	d14d029c73	reduce v.dataFileAccessLock lock scope	2022-09-08 19:05:31 -07:00
chrislu	406a80da4b	adjust variable names	2022-09-08 18:54:16 -07:00
chrislu	2bfc8970d2	refactor: move ReadNeedleDataInto into volume_read.go	2022-09-08 18:54:02 -07:00
chrislu	9b084d4c88	purge tcp implementation	2022-09-08 18:03:43 -07:00
chrislu	eb7cf3de81	a new test case	2022-09-08 14:04:42 -07:00
chrislu	4260804613	volume.balance: avoid moving out volume with max=1	2022-09-08 14:04:21 -07:00
chrislu	d653c5f811	unused	2022-09-08 12:27:53 -07:00
chrislu	03e22ddb43	rename	2022-09-08 12:06:03 -07:00
Konstantin Lebedev	31d2f77ceb	refactor https://github.com/seaweedfs/seaweedfs/pull/3616 (#3625 )	2022-09-07 23:23:33 -07:00
chrislu	7de112943b	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-07 20:14:11 -07:00
chrislu	7e52b1e195	return err for processRangeRequest()	2022-09-07 20:14:09 -07:00
famosss	449582343f	fix:Sometimes a nil pointer exception is thrown (#3618 )	2022-09-07 18:57:13 -07:00
Konstantin Lebedev	695e3a909c	avoid data race on currentDiskUsage.maxVolumeCount (#3592 ) https://github.com/seaweedfs/seaweedfs/issues/3590	2022-09-07 09:51:15 -07:00
Konstantin Lebedev	cca45b02a2	avoid data race on calc freeVolumeSlotCount (#3594 ) https://github.com/seaweedfs/seaweedfs/issues/3593	2022-09-07 09:50:17 -07:00
famosss	9678fc2106	fix: volume heartbeat processing error (#3616 )	2022-09-07 09:48:51 -07:00
famosss	5ff33eb558	fix: volume Deadlock when exception is thrown (#3613 ) * fix: volume Deadlock when exception is thrown * adjust log level	2022-09-06 23:56:29 -07:00
Eric Yang	b324a6536c	ADHOC: add read needle meta grpc (#3581 ) * ADHOC: add read needle meta grpc * add test * nit Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-09-06 23:51:27 -07:00
chrislu	c07ab9c060	3.26	2022-09-06 08:26:20 -07:00
Konstantin Lebedev	f4f6855c61	avoid SIGSEGV on localFilerSocket (#3609 ) https://github.com/seaweedfs/seaweedfs/issues/3607	2022-09-06 08:20:23 -07:00
Konstantin Lebedev	3c3682fcce	more log detail for upload err and deleting (#3577 )	2022-09-06 08:19:13 -07:00
Konstantin Lebedev	01a25755c5	[filer] masterClient updates volume adjust log level (#3610 )	2022-09-06 07:26:59 -07:00
LHHDZ	bc629665de	fix bug due to data racing on VidMap (#3606 )	2022-09-05 20:00:16 -07:00
Konstantin Lebedev	9b2b7d4f5a	avoid data race on GoCDKPubSub.topic (#3596 )	2022-09-05 07:57:44 -07:00
Konstantin Lebedev	916673ae24	do not print diskInfos with doVolumeCheckDisk in error log (#3598 )	2022-09-05 07:49:26 -07:00
Konstantin Lebedev	5307cadb32	[s3] Replace code 500 to 499 for upload with ErrUnexpectedEOF (#3597 )	2022-09-05 07:47:43 -07:00
chrislu	5b38f22e6e	3.25	2022-09-04 22:45:55 -07:00
famosss	131d389fc4	adjust log level (#3589 )	2022-09-04 22:37:32 -07:00
chrislu	f4abefc109	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-04 22:23:31 -07:00
chrislu	39340f7e42	cloud tier: s3 consume all read response body fix https://github.com/seaweedfs/seaweedfs/issues/3584	2022-09-04 22:23:28 -07:00
chrislu	7c6324b114	adjust log level	2022-09-04 22:21:24 -07:00
gfx	bf844d8e46	Fix crash in arangodb filer when attempting to access a deleted bucket (#3587 ) * Update helpers.go * Update arangodb_store_bucket.go	2022-09-04 20:32:10 -07:00
gfx	3621987727	Correctly pass arangodb driver Transaction ID into the context (#3586 ) Update arangodb_store.go	2022-09-04 19:29:52 -07:00
chrislu	8c5759a06d	minor optimization	2022-09-04 18:50:45 -07:00
chrislu	70777d9b9f	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-09-04 18:26:51 -07:00
chrislu	cb6cf331ca	filer.backup and filer.sync: include headers during backup and sync fix https://github.com/seaweedfs/seaweedfs/issues/3532	2022-09-04 18:26:36 -07:00
famosss	7f56a40553	Turn down the 404 log level (#3585 )	2022-09-04 18:17:01 -07:00
Brian	a28b668647	Added ability to change replication settings upon volume.tier.move (#3583 )	2022-09-04 16:47:21 -07:00
bernardx	228b133afa	new 'concurrency' parameter for filer.sync (#3579 ) Co-authored-by: XIAOYQ <xiaoyq@eudic.net>	2022-09-02 23:03:23 -07:00
Konstantin Lebedev	853880bd83	[filer] DATA RACE on signal_handling fixed (#3575 )	2022-09-02 06:20:19 -07:00
chrislu	a112cbe44c	filer: rocksdb store report not found correctly fix https://github.com/seaweedfs/seaweedfs/discussions/3555	2022-09-01 15:42:59 -07:00
chrislu	d1bb23645e	Revert "avoid data race on grace.hooks (#3572 )" This reverts commit `c37d6fc01a`.	2022-09-01 13:08:34 -07:00
Konstantin Lebedev	c37d6fc01a	avoid data race on grace.hooks (#3572 ) https://github.com/seaweedfs/seaweedfs/issues/3564	2022-09-01 10:34:06 -07:00
Konstantin Lebedev	8c3040db81	avoid DATA RACE on S3Options.localFilerSocket (#3571 ) * avoid DATA RACE on S3Options.localFilerSocket https://github.com/seaweedfs/seaweedfs/issues/3552 * copy localSocket	2022-09-01 10:33:23 -07:00
Konstantin Lebedev	90d55cd179	avoid data race read fileChunksSize (#3570 ) https://github.com/seaweedfs/seaweedfs/issues/3557	2022-09-01 10:31:57 -07:00
Konstantin Lebedev	82c2ce74db	more error details in the log (#3568 ) * more error details in the log https://github.com/seaweedfs/seaweedfs/issues/3567 * format message	2022-09-01 09:04:15 -07:00
chrislu	10efdc7aab	align memory	2022-09-01 09:02:44 -07:00
Konstantin Lebedev	803ca3c958	avoid data race on doSubscribeToOneFiler/ma.filer.UniqueFilerEpoch (#3566 ) https://github.com/seaweedfs/seaweedfs/issues/3565	2022-09-01 09:01:47 -07:00
chrislu	bcf35876d1	add more logs	2022-08-31 23:16:05 -07:00
dependabot[bot]	97d69d5336	Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 (#3541 ) * Bump gocloud.dev/pubsub/rabbitpubsub from 0.25.0 to 0.26.0 Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.25.0 to 0.26.0. - [Release notes](https://github.com/google/go-cloud/releases) - [Commits](https://github.com/google/go-cloud/compare/v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: gocloud.dev/pubsub/rabbitpubsub dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * update code * more code fix Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: chrislu <chris.lu@gmail.com>	2022-08-31 10:16:49 -07:00
Patrick Schmidt	a73e177ecf	Add an End-to-End workflow for FUSE mount (#3562 ) * Add an e2e workflow to test FUSE mount * Fix deadlocks during concurrent r/w	2022-08-31 09:27:53 -07:00
chrislu	4a4ef3cc3c	upload_content: upload close response.Body	2022-08-31 00:25:28 -07:00
chrislu	3d8b8ffced	refactor	2022-08-31 00:24:29 -07:00
chrislu	ef78631a7c	just a bit safer	2022-08-31 00:10:17 -07:00
chrislu	9c944377dc	minor	2022-08-31 00:09:32 -07:00
chrislu	a54f30b6c6	s3: close response body with copying object	2022-08-31 00:09:23 -07:00
chrislu	7eb121d5ae	print process before do the work	2022-08-30 13:37:05 -07:00
Konstantin Lebedev	77098de69a	migrate from boltdb to bbolt (#3554 )	2022-08-30 10:12:02 -07:00
chrislu	de5263364b	master client: use GetMaster() for latest master address	2022-08-30 09:52:06 -07:00
chrislu	e583dbdd2a	more detailed error	2022-08-30 09:43:17 -07:00
chrislu	d81db3c703	s3: fix configuring IAM for the same user hi, how can I add bucket permission to a user now? Previously, if I needed to add permission to an existing credential, I simply repeated the s3.configure command with a different bucket name. Now I am getting error: duplicate accessKey[ХХХХ], already configured in user[YYYY] s3.configure -access_key key -actions Read,Write,List -buckets bucket1 -secret_key secr -user user1 s3.configure -access_key key -actions Read,Write,List -buckets bucket2 -secret_key secr -user user1	2022-08-30 09:37:52 -07:00
Konstantin Lebedev	ade94b0d0a	avoid race conditions access to SuperBlock.Version (#3539 ) * avoid race conditions access to SuperBlock.Version https://github.com/seaweedfs/seaweedfs/issues/3515 * superBlockAccessLock replace to sync.Mutex	2022-08-30 00:08:00 -07:00
chrislu	ae6292f9f0	rename variable	2022-08-30 00:07:15 -07:00
Konstantin Lebedev	e16dda88e4	avoid race conditions access to MasterClient.currentMaster (#3538 ) https://github.com/seaweedfs/seaweedfs/issues/3510	2022-08-30 00:05:38 -07:00
Konstantin Lebedev	105702ebe0	avoid data race access to uploadReaderToChunks.uploadErr (#3550 ) avoid data race access to uploadErr https://github.com/seaweedfs/seaweedfs/issues/3549	2022-08-30 00:03:01 -07:00
Konstantin Lebedev	903a06a160	avoid data race access to startDelay (#3548 )	2022-08-30 00:02:09 -07:00
chrislu	570b488572	ensure memory alignment on different CPUs	2022-08-29 13:24:12 -07:00
Konstantin Lebedev	4966a3abc7	avoid race conditions access to growRequestCount (#3537 ) https://github.com/seaweedfs/seaweedfs/issues/3511	2022-08-29 13:23:02 -07:00
Patrick Schmidt	00041f6e08	Fix corrupted mount reads after changes on fh (#3536 )	2022-08-29 08:27:13 -07:00
chrislu	b7a887fea1	3.24	2022-08-28 21:53:13 -07:00
Rain Li	089446f5b1	Fix reload page before all file uploaded bug (#3534 )	2022-08-28 07:51:47 -07:00
chrislu	cc0c8c5f81	simplify	2022-08-27 00:21:57 -07:00
chrislu	3741779144	minor	2022-08-27 00:20:58 -07:00
chrislu	580a7fa685	adjust for data race fix https://github.com/seaweedfs/seaweedfs/issues/3512	2022-08-27 00:19:13 -07:00
chrislu	87b70a6809	clean up	2022-08-27 00:09:04 -07:00
chrislu	27c7611ebf	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 23:47:39 -07:00
chrislu	9920d65bc0	gateway to remote object store: adjust upload concurrency	2022-08-26 23:47:37 -07:00
chrislu	c839ce1b19	s3 sink use s3 upload manager fix https://github.com/seaweedfs/seaweedfs/issues/3531	2022-08-26 23:47:12 -07:00
famosss	dc4037925d	fix: Build DeletedVids before reset dn's children (#3530 )	2022-08-26 22:52:08 -07:00
chrislu	3f3a1341d8	make CodeQL happy	2022-08-26 17:09:11 -07:00
chrislu	301b49b63f	atomic operation	2022-08-26 17:04:11 -07:00
chrislu	57a46f46a0	nano level precision	2022-08-26 16:55:15 -07:00
chrislu	8dae81c5ed	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 16:49:51 -07:00
chrislu	e0f4366f4c	more detailed error related to https://github.com/seaweedfs/seaweedfs/issues/3528	2022-08-26 16:49:48 -07:00
Patrick Schmidt	5df105b1f9	Fix a few data races when reading files in mount (#3527 )	2022-08-26 16:41:37 -07:00
chrislu	f5156cf3a8	rename varaible	2022-08-26 10:23:42 -07:00
Konstantin Lebedev	e90ab4ac60	avoid race conditions for OnPeerUpdate (#3525 ) https://github.com/seaweedfs/seaweedfs/issues/3524	2022-08-26 10:18:49 -07:00
Konstantin Lebedev	4f7a1f67cd	avoid race conditions for diskLocation.MaxVolumeCount (#3526 )	2022-08-26 08:41:42 -07:00
chrislu	6f483a4d36	atomic add fix https://github.com/seaweedfs/seaweedfs/issues/3514	2022-08-25 22:20:34 -07:00
chrislu	7394f7feee	minor: adjust ordering	2022-08-25 00:20:23 -07:00
chrislu	dbf0de4ce1	minor clean up	2022-08-25 00:19:08 -07:00
Patrick Schmidt	2930263dfd	Fix race conditions during in-flight size checks (#3505 )	2022-08-24 20:03:34 -07:00
Patrick Schmidt	7b424a54dc	Add raft server access mutex to avoid races (#3503 )	2022-08-24 09:49:05 -07:00
Konstantin Lebedev	f7aeb06544	s3: report metadata if the directory is explicitly created (#3498 ) * replace mkdir to mkFile * ContentLength must be zero * revert mkDir * Seaweedfs-Is-Directory-Key return metadata	2022-08-24 00:15:44 -07:00
Guo Lei	c57c79a0ab	optimiz commitig compact (#3388 ) * optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index	2022-08-23 23:53:35 -07:00
chrislu	10414fd81c	ping timeout at 15 seconds this 72 minute timeout setting seems unreasonably long 15 seconds is around the time when a new raft leader should be elected.	2022-08-23 23:28:16 -07:00
askeipx	2e78a522ab	remove old raft servers if they don't answer to pings for too long (#3398 ) * remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix	2022-08-23 23:18:21 -07:00
Guo Lei	762dc219a6	fix deletecount bug (#3499 )	2022-08-23 08:53:24 -07:00
chrislu	21a933c932	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-23 01:52:32 -07:00
chrislu	f0b4a7659a	fix test	2022-08-23 01:52:29 -07:00
Patrick Schmidt	cda8cc22bc	Implement lseek syscall in FUSE (#3491 ) See the man page of lseek: https://man7.org/linux/man-pages/man2/lseek.2.html	2022-08-23 01:47:18 -07:00
chrislu	bf88006037	format	2022-08-23 01:20:45 -07:00
chrislu	9fce75607d	s3: report http.StatusOK if the directory is explicitly created fix https://github.com/seaweedfs/seaweedfs/issues/3457	2022-08-23 01:16:46 -07:00
famosss	911475526c	fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497 )	2022-08-22 23:54:51 -07:00
famosss	7eb15b1969	fix:Handle preflight cors requests (#3496 )	2022-08-22 23:39:34 -07:00
chrislu	676e27c589	shell: stop long running jobs if lock is lost	2022-08-22 14:12:23 -07:00
chrislu	601ba5fb68	master: disable locking if not leader	2022-08-22 14:11:48 -07:00
chrislu	57e7582c36	refactoring	2022-08-22 14:11:13 -07:00
chrislu	7ea2c24c5d	metadata processing use retry instead of retryForEver	2022-08-22 11:25:40 -07:00
Patrick Schmidt	f875031f06	Reuse readDataByFileHandle in Read call (#3482 )	2022-08-22 08:24:06 -07:00
famosss	bdba3da2e4	Handle preflight cors requests (#3481 )	2022-08-22 08:21:38 -07:00
chrislu	f4b52d4c52	fix format	2022-08-22 00:20:21 -07:00
chrislu	b4131a3a72	filer.remote.sync: parallelize uploading to remove object storage	2022-08-22 00:01:48 -07:00
chrislu	f6b92b8561	refactor	2022-08-22 00:00:51 -07:00
chrislu	3c8e95eeb3	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-21 23:26:09 -07:00
chrislu	8bac903d29	mount: add locking to prevent nil pointer fix https://github.com/seaweedfs/seaweedfs/issues/3479	2022-08-21 23:26:06 -07:00
qzh	74b53729e1	feat(weed.move): add a speed limit parameter of moving files (#3478 ) * feat(weed.move): add a speed limit parameter of moving files * fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-21 23:08:31 -07:00
chrislu	7807f6641a	minor	2022-08-21 22:54:56 -07:00
chrislu	c4e862e908	3.23	2022-08-21 19:13:26 -07:00
Guo Lei	81e7e6ceb3	the statistical methord for fileCount should stay same (#3477 ) the statistical methord for fileCount between should stay same They all equal to the entry count in .idx file. relate to commit(`c7892bc`)	2022-08-21 19:11:18 -07:00
chrislu	63fbf281c7	remove same file copying rage limitation	2022-08-21 17:16:21 -07:00
Patrick Schmidt	3a75d7f7aa	Implement copy_file_range syscall in FUSE (#3475 ) See the man page of copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html	2022-08-21 17:11:43 -07:00
chrislu	fef9c6a520	filer: do not always save files in "/etc" folder to filer store fix https://github.com/seaweedfs/seaweedfs/issues/3476	2022-08-21 17:02:16 -07:00
chrislu	2762154130	fix compilation	2022-08-21 16:09:23 -07:00
chrislu	fb5808e0c3	EC: with multiple volume locations, the ec rebuilding may fail	2022-08-21 15:04:50 -07:00
chrislu	5790d01c6f	EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations related to https://github.com/seaweedfs/seaweedfs/issues/3459	2022-08-21 14:52:37 -07:00
chrislu	8b3429858d	align memory for atomic read/write	2022-08-21 12:20:27 -07:00
Patrick Schmidt	2ef6ab998c	Avoid race conditions with current filer address (#3474 ) When multiple filer requests are in-flight and the current filer disappears and a new one is selected by the first goroutine, then there can be a lot of race conditions while retrieving the current filer. Therefore, load/save the current filer index atomically.	2022-08-21 12:18:13 -07:00
Patrick Schmidt	f49a9297c2	Fix hanging reads in chunk cacher (#3473 ) Sometimes when an unexpected error occurs the cacher would set an error and return. However, it would not broadcast the condition signal in that case, therefore leaving the goroutine that runs readChunkAt stuck forever. I figured that the condition is unnecessary because readChunkAt is acquiring a lock that is still held by the cacher goroutine anyway. Callees of startCaching have to wait for a WaitGroup which makes sure that readChunkAt can't acquire the lock before startCaching. This way readChunkAt can execute normally and check for the error.	2022-08-21 11:54:02 -07:00
chrislu	388f82f322	minor	2022-08-21 11:49:29 -07:00
chrislu	77e4b1376e	refactoring	2022-08-21 11:35:54 -07:00
Patrick Schmidt	3f758820c1	Fix FUSE server buffer leaks in file gaps (#3472 ) * Fix FUSE server buffer leaks in file gaps This change zeros read buffers when encountering file gaps during file/chunk reads in FUSE mounts. It prevents leaking internal buffers of the FUSE server which could otherwise reveal metadata, directory listings, file contents and other data related to FUSE API calls. The issue was that buffers are reused, but when a file gap was found the buffer was not zeroed accordingly and the existing data of the buffer was kept and returned. * Move zero logic into its own method	2022-08-21 11:33:58 -07:00
chrislu	c7892bc7c4	volume: file counter should be all files address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228	2022-08-20 23:35:31 -07:00
chrislu	f7e0a65e75	retry for all errors	2022-08-20 23:34:45 -07:00
chrislu	649555b223	adjust log	2022-08-20 23:34:07 -07:00
chrislu	973f6dd162	refactoring	2022-08-20 22:38:15 -07:00
chrislu	65ff7198fe	adjust type	2022-08-20 22:37:24 -07:00
chrislu	28b862f45f	use util.Retry to retry	2022-08-20 22:03:27 -07:00
chrislu	3bf8e772f8	webdav: retryable data chunk upload	2022-08-20 19:18:12 -07:00
chrislu	f8fa430257	filer: retryable data chunk upload	2022-08-20 19:15:44 -07:00
chrislu	4081d50607	filer sink: retryable data chunk uploading	2022-08-20 19:09:15 -07:00
chrislu	409f39390d	mount: retryable data chunk uploading	2022-08-20 19:04:36 -07:00
chrislu	6c8822f269	filer.copy: retryable file part upload	2022-08-20 18:59:57 -07:00
chrislu	a3553da7f7	add fileId to UploadWithRetry return result	2022-08-20 18:54:59 -07:00
chrislu	d49d0a9fc2	filer.copy: retryable upload	2022-08-20 18:50:57 -07:00
chrislu	689b4ecdcc	remove unused collection and replication from upload result	2022-08-20 18:14:57 -07:00
chrislu	e3f40d538d	cleaner code	2022-08-20 17:51:30 -07:00
chrislu	9f50b310a0	remove unused code	2022-08-20 17:51:12 -07:00
chrislu	11f99836c3	filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0) fix https://github.com/seaweedfs/seaweedfs/issues/3468	2022-08-19 23:00:56 -07:00
chrislu	fdd8c5d5e0	Filer if-modified-since check doesn't fire on "exact" #3467 fix https://github.com/seaweedfs/seaweedfs/issues/3467	2022-08-19 22:21:51 -07:00
chrislu	aac45f3e89	filer: retryable when error is not found	2022-08-19 03:36:15 -07:00
chrislu	58dd880322	minor	2022-08-18 23:51:08 -07:00
Eric Yang	b6f99c4d83	fix naming typo (#3465 ) Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-08-18 23:25:32 -07:00
chrislu	5702727db5	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-18 02:39:27 -07:00
chrislu	4d72a1ce6e	s3: remove deleted collection during metrics reporting fix https://github.com/seaweedfs/seaweedfs/issues/3453	2022-08-18 02:39:07 -07:00
Guo Lei	3172c33fa6	fix redundant type from array (#3462 )	2022-08-18 02:25:25 -07:00
chrislu	42c6e52513	s3: fix regression on HEAD directory operation	2022-08-18 02:13:58 -07:00
chrislu	4573c99ae3	fix tests	2022-08-18 00:15:46 -07:00
chrislu	2b580a7566	also migrate jsonpb	2022-08-17 12:42:03 -07:00
chrislu	eaeb141b09	move proto package	2022-08-17 12:05:07 -07:00
chrislu	e259052a70	volume: fix metrics for volume request time	2022-08-17 04:29:31 -07:00
qzh	7fcfaf7bc9	fix(filer.sync): offset may be set to 0 (#3451 ) * fix(filer.sync): initializing the offset is related to the path * fix(filer.sync): the offset maybe to be set to 0. Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 23:43:52 -07:00
qzh	400f0c3e5d	fix(filer.sync): initializing the offset is related to the path (#3450 ) Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 21:56:47 -07:00
chrislu	fa4d0093e1	3.22	2022-08-15 16:48:23 -07:00
chrislu	93261f5199	Revert "Refactor for Sync method (#3426 )" This reverts commit `670cb759f8`. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]	2022-08-15 15:35:31 -07:00
chrislu	9c588d4010	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-15 10:03:56 -07:00
chrislu	96caf21d09	less verbose log	2022-08-15 10:03:52 -07:00
Andrey Triumfov	31faa6d43d	Remove duplicate slashes in object path to prevent 500 errors (#3442 )	2022-08-15 08:19:28 -07:00
chrislu	cb476a53ff	remove logs	2022-08-15 01:05:35 -07:00
chrislu	7c029b2183	3.21	2022-08-15 00:32:15 -07:00
Chris Lu	e88392b50f	Fix s3 pagination (#3436 ) * Revert previous changes * s3: use cursor to track tree traversal fix https://github.com/seaweedfs/seaweedfs/issues/3166 * special cases for empty prefix and empty directory * use constants * address empty folder * undo local changes * fix IsTruncated * adjust counting directories * fix cases when prefix is a directory * s3: handle directory object works for aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"	2022-08-15 00:30:19 -07:00
chrislu	7457c746f0	s3: fix aws s3api head-object	2022-08-14 23:52:35 -07:00
chrislu	789d9dbfe1	webdav: fix nil fix https://github.com/seaweedfs/seaweedfs/issues/3440	2022-08-14 23:33:30 -07:00
Konstantin Lebedev	6199d9b923	rand start garbageThreshold and reset metric ReplicaPlacementMismatch (#3396 ) * rand start garbageThreshold reset metrics MasterReplicaPlacementMismatch * vacuum through sleep	2022-08-12 13:11:09 -07:00
Konstantin Lebedev	4d4cd0948d	avoid infinite loop WaitUntilConnected() (#3431 ) https://github.com/seaweedfs/seaweedfs/issues/3421	2022-08-11 15:03:26 -07:00
Eric Yang	986daecbfe	add upload response (#3428 ) * add upload response * fixing comments Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-08-10 22:43:09 -07:00
chrislu	ae93c966d9	ensure memory is aligned fix https://github.com/seaweedfs/seaweedfs/issues/3427	2022-08-10 22:27:13 -07:00
Rain Li	670cb759f8	Refactor for Sync method (#3426 )	2022-08-10 08:21:57 -07:00
chrislu	662ec97602	mount: adjust on du stats reporting fix https://github.com/seaweedfs/seaweedfs/issues/3424	2022-08-09 20:21:01 -07:00
Abirdcfly	b0633716b7	delete minor unreachable code (#3423 ) Signed-off-by: Abirdcfly <fp544037857@gmail.com>	2022-08-09 01:24:43 -07:00
chrislu	0854171d22	3.20	2022-08-07 14:42:28 -07:00
chrislu	d8e69df295	adjust comments	2022-08-07 11:35:33 -07:00
chrislu	0aeec04c31	quicker to adapt to pattern change	2022-08-07 10:14:01 -07:00
chrislu	928d29af9e	fix wrong logic about reader isRandomMode()	2022-08-07 10:13:04 -07:00
chrislu	d3d52b1818	shell: fs.meta.load add quieter mode fix https://github.com/seaweedfs/seaweedfs/issues/3414	2022-08-07 01:42:15 -07:00
chrislu	358ccb5e0e	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-07 01:34:36 -07:00
chrislu	67814a5c79	refactor and fix strings.Split	2022-08-07 01:34:32 -07:00
John W Higgins	3afda0c89c	Allow postgresql to use standard environment variables for connection (#3413 )	2022-08-07 00:58:53 -07:00
chrislu	1a4bf0dcb5	filer.sync: parallelize the filer.sync	2022-08-07 00:56:15 -07:00
chrislu	02b38f894b	debug	2022-08-07 00:55:49 -07:00
chrislu	0e9478488d	filer.sync: fix when excluded paths is empty	2022-08-07 00:55:34 -07:00
chrislu	8a880a139d	mount: fix truncate operation fix https://github.com/seaweedfs/seaweedfs/issues/2609	2022-08-06 13:58:45 -07:00
LHHDZ	84ec68e11a	Add download speed limit support (#3408 )	2022-08-05 01:16:42 -07:00
chrislu	b278bb24d3	mount: adjust df stats reporting when close to the limit fix https://github.com/seaweedfs/seaweedfs/issues/3407	2022-08-05 00:31:41 -07:00
Konstantin Lebedev	4d08393b7c	filer prefer volume server in same data center (#3405 ) * initial prefer same data center https://github.com/seaweedfs/seaweedfs/issues/3404 * GetDataCenter * prefer same data center for ReplicationSource * GetDataCenterId * remove glog	2022-08-04 17:35:00 -07:00
Konstantin Lebedev	22181dd018	refactor FilerRequest metrics (#3402 ) * refactor FilerRequest metrics * avoid double count proxy * defer to	2022-08-04 01:44:54 -07:00
chrislu	bd13a7968f	remove unused symlink resolving	2022-08-04 01:35:18 -07:00
chrislu	88945d9954	mount: do not follow soft link with xattr	2022-08-04 01:21:15 -07:00
chrislu	90db4b5a44	minor	2022-08-03 00:10:47 -07:00
Chris Lu	fc8035d672	Merge pull request #3394 from kmlebedev/metricsReplicatedWrite Detailed metrics VolumeServerRequestHistogram for writing to disk and replication	2022-08-01 13:31:01 -07:00
Konstantin Lebedev	3ffa4ba91d	clean	2022-08-02 01:30:38 +05:00
Konstantin Lebedev	2a5629dd28	align	2022-08-02 01:29:44 +05:00
Konstantin Lebedev	df9cc31a0a	refactor	2022-08-02 01:18:26 +05:00
Konstantin Lebedev	0a5d3c3cf6	rename to localWrite and replicatedWrite	2022-08-01 23:09:09 +05:00
chrislu	bd5c9904be	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-01 09:40:05 -07:00
chrislu	e26d1cf4ef	fix test	2022-08-01 09:40:01 -07:00
Konstantin Lebedev	fc65122766	rename to LoadAvg_1M	2022-08-01 21:32:21 +05:00
Konstantin Lebedev	ab50f21ad9	Detailed metrics VolumeServerRequestHistogram for writing to disk and replication	2022-08-01 21:16:57 +05:00
Konstantin Lebedev	2f85e31f5c	Merge branch 'master' into gentle_vacuum	2022-08-01 20:51:58 +05:00
Konstantin Lebedev	5209ebbeef	remove percent	2022-08-01 20:40:38 +05:00
Chris Lu	b59bc607bf	Merge pull request #3338 from kmlebedev/issues/3083 rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping	2022-08-01 08:23:10 -07:00
Konstantin Lebedev	78cbd8002f	revert Sleep	2022-08-01 20:21:23 +05:00
Konstantin Lebedev	cd5c7ad052	move to github.com/seaweedfs/seaweedfs	2022-08-01 16:36:32 +05:00
Konstantin Lebedev	634bc97840	fix conflicts	2022-08-01 14:47:00 +05:00
Konstantin Lebedev	3c75479e2b	Merge branch 'master' into gentle_vacuum # Conflicts: # weed/pb/messaging_pb/messaging.pb.go # weed/pb/messaging_pb/messaging_grpc.pb.go # weed/pb/s3_pb/s3.pb.go # weed/pb/volume_server_pb/volume_server.pb.go # weed/server/volume_grpc_vacuum.go	2022-08-01 14:45:22 +05:00
Konstantin Lebedev	c0d92f61a1	comment	2022-08-01 14:40:42 +05:00
Konstantin Lebedev	1d29f67c02	revert disk stats	2022-08-01 14:29:41 +05:00
Konstantin Lebedev	a98f6d66a3	rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping	2022-08-01 12:51:41 +05:00
chrislu	036566629a	filer.sync: fix synchronization logic in active-active mode fix https://github.com/seaweedfs/seaweedfs/issues/3328	2022-08-01 00:06:18 -07:00
chrislu	303bd067b5	Revert "rename: delete source entry metadata only, skipping hard links" This reverts commit `03466f955e`. fix https://github.com/seaweedfs/seaweedfs/issues/3386	2022-07-31 22:51:41 -07:00
chrislu	03c6f978f4	3.19	2022-07-31 17:16:51 -07:00
chrislu	afef014832	scaffold	2022-07-31 13:23:44 -07:00
Chris Lu	0bb3ba17ae	Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy Add healthy indicator for raft status	2022-07-30 11:55:30 -07:00
Patrick Schmidt	1a4a36d510	Add healthy indicator for raft status	2022-07-30 19:34:26 +02:00
chrislu	cb096d1f53	minor	2022-07-30 09:52:52 -07:00
chrislu	0423d50106	wip note	2022-07-30 09:36:27 -07:00
chrislu	0f50fcb96a	moving	2022-07-29 23:39:11 -07:00
chrislu	13b9a52f80	fix deprecated functions	2022-07-29 01:34:39 -07:00
chrislu	be0ec7bc2f	fix test	2022-07-29 01:15:26 -07:00
chrislu	1f161f53dc	add back rack	2022-07-29 01:15:19 -07:00
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	2022-07-29 00:17:28 -07:00
Konstantin Lebedev	2f0dda384d	vacuum show LA	2022-07-29 11:59:33 +05:00
chrislu	de3afd4199	fix rebasing	2022-07-28 23:25:46 -07:00
chrislu	06cd491abc	add message batch	2022-07-28 23:24:38 -07:00
chrislu	7576c244c4	add flatbuffer serde for message	2022-07-28 23:24:38 -07:00
chrislu	a3fae7cd16	add an empty placeholder for mq.topic.list	2022-07-28 23:24:38 -07:00
chrislu	aa4a22ad47	segment serde	2022-07-28 23:24:38 -07:00
chrislu	74f60f246f	dynamically connect to a filer	2022-07-28 23:24:38 -07:00
chrislu	9f479aab98	allocate brokers to serve segments	2022-07-28 23:24:38 -07:00
chrislu	bb01b68fa0	refactor	2022-07-28 23:24:38 -07:00
chrislu	94b8c42b2c	clean up	2022-07-28 23:24:38 -07:00
chrislu	8060fdcac5	remove old code	2022-07-28 23:24:38 -07:00
chrislu	f25e273e32	display data center and rack in cluster.ps	2022-07-28 23:22:52 -07:00
chrislu	68065128b8	add dc and rack	2022-07-28 23:22:51 -07:00
chrislu	8d31e73ffd	mq broker adds data center and rack	2022-07-28 23:22:07 -07:00
chrislu	87ccef72db	cluster.ps can list brokers	2022-07-28 23:22:07 -07:00
chrislu	ac66227798	renaming	2022-07-28 23:22:07 -07:00
chrislu	21b6b07dd8	renaming	2022-07-28 23:22:06 -07:00
chrislu	8ca7d1ef17	rename to SeaweedMQ	2022-07-28 23:22:06 -07:00
Chris Lu	3e7e922d16	Merge pull request #3378 from vanhtuan0409/change-put-policy PutUserAction should completely replace identity actions	2022-07-28 19:43:25 -07:00
chrislu	89e276de64	fix loop	2022-07-28 18:06:49 -07:00
chrislu	a458bee9f2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-28 16:32:02 -07:00
chrislu	e0eda52c54	mount: ensure symlink parent directory is tracked fix https://github.com/chrislusf/seaweedfs/issues/3373	2022-07-28 16:32:00 -07:00
Tuan Vuong	1d9a888be9	PutUserAction should completely replace identity actions	2022-07-28 17:02:56 +07:00
zzq09494	01b7aa79fa	fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted	2022-07-28 12:04:37 +08:00
chrislu	ca836568ac	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-27 12:12:43 -07:00
chrislu	3828b8ce87	"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"	2022-07-27 12:12:40 -07:00
Chris Lu	b7e4e26778	Merge pull request #3371 from askeipx/issues/3341 delete disk_size metrics when collection deleted	2022-07-27 11:25:42 -07:00
Konstantin Lebedev	7e09a548a6	exclude directories to sync on filer	2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev	ac5ce31278	leave notion to refactor after prometheus upgrade	2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev	47c72e6f35	remove all (currently existing) collection volume metrics	2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev	709c83716c	delete disk_size metrics when collection deleted	2022-07-27 15:02:41 +05:00
chrislu	f401b996eb	mount: fix symlink size reporting	2022-07-26 23:12:25 -07:00
Konstantin Lebedev	c88ea31f62	fix RUnlock of unlocked RWMutex	2022-07-26 12:57:07 +05:00
Konstantin Lebedev	da9d3e8f6c	refactor	2022-07-26 11:56:45 +05:00
Konstantin Lebedev	046c3d5ad4	fix logic else brake	2022-07-26 11:47:11 +05:00
chrislu	475185fb72	3.18	2022-07-25 22:41:38 -07:00
chrislu	ec0edb1ac4	filer: fix wrong logic during read	2022-07-25 22:40:00 -07:00
guosj	cc7a4b0a6e	correct comment	2022-07-25 11:46:41 +08:00
chrislu	33cff61992	fix glog test	2022-07-24 19:13:54 -07:00
chrislu	dbe4849ffd	3.17	2022-07-24 18:46:50 -07:00
chrislu	b98c0b4bf4	disable symlink translation for now	2022-07-24 18:46:32 -07:00
chrislu	2dc5d4adac	disable symlink translation for now	2022-07-24 18:24:34 -07:00
chrislu	3d2bbf96d2	adjust glog format	2022-07-24 17:29:08 -07:00
chrislu	2807fe08fc	adjust glog format	2022-07-24 16:44:58 -07:00
chrislu	2b64916d3e	set status to fuse.OK	2022-07-24 15:30:55 -07:00
chrislu	7386cf6a06	mount: resolve symlink	2022-07-24 15:11:24 -07:00
chrislu	40089b6753	maintain node lookup counter	2022-07-24 13:43:11 -07:00
chrislu	d24db396cc	fix removing path from inode2path	2022-07-23 18:20:29 -07:00
chrislu	41eeb4deef	do not add new inode during link	2022-07-23 13:37:07 -07:00
chrislu	02c2d81cde	fix removePathFromInode2Path	2022-07-23 13:21:42 -07:00
chrislu	4d5dc55766	mount: better support hard links	2022-07-23 12:57:00 -07:00
chrislu	ebda0689ab	use linux default max name length	2022-07-23 10:51:06 -07:00
chrislu	64f3d6fb6e	metadata subscription uses client epoch	2022-07-23 10:50:28 -07:00
chrislu	2c8818351f	mount: comment out stuck fuse inode notify during renaming	2022-07-22 18:17:34 -07:00
chrislu	a5c94acd4f	mount: check possible nil entry case	2022-07-22 15:33:37 -07:00
Chris Lu	0716092b39	Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap Solve the problem that `LookupFileId` lookup urls is empty due to lea…	2022-07-22 02:57:50 -07:00
LHHDZ	994a2dec78	fix `vid_map_test` tests	2022-07-22 17:54:57 +08:00
LHHDZ	58c02d6429	Solve the problem that `LookupFileId` lookup urls is empty due to leader switching The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.	2022-07-22 17:22:38 +08:00
ningfd	58f2dd6740	change topology mapping definition, improve spelling	2022-07-22 17:18:56 +08:00
ningfd	6f882eb354	optimiz: master ui will render data in order	2022-07-22 14:37:49 +08:00
chrislu	7a6c559ab4	fix Change replication via volume.configure.replication by collection fix https://github.com/chrislusf/seaweedfs/issues/3346	2022-07-21 22:01:05 -07:00
chrislu	fc8241fb5e	leveldb3: add instant dropping bucket	2022-07-21 18:48:51 -07:00
chrislu	c93f7ffa44	explicit bucket aware declaration	2022-07-21 18:23:53 -07:00
Konstantin Lebedev	3c42814b58	avoid deadlock	2022-07-21 17:15:10 +05:00
Konstantin Lebedev	93ca87b7cb	use safe onPeerUpdateDoneCns	2022-07-21 15:51:14 +05:00
chrislu	6147b61b0a	mount: fix concurrent map read and map write fix https://github.com/chrislusf/seaweedfs/issues/3344	2022-07-21 01:38:26 -07:00
Chris Lu	9ec0d1caaa	Merge pull request #3337 from guol-fnst/loading_volume	2022-07-20 09:06:06 -07:00
Konstantin Lebedev	7875470e74	onPeerUpdateGoroutineCount use int32	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	c5189c343b	remove ticker update the topology before each file	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	e2d991d8d0	ticker.Stop	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	de4fcc0e2c	sync update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	fa88dff7ce	update otherNodes	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	72dca31cfa	fix update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	884ffbafee	clouse background update	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	39eaf426f8	fix TestVolumeServerEvacuate	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	b5e5f6f55a	update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	867269cdcf	help rack	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	6f764e1014	volume server evacuate from rack	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	ba0e3ce5fa	volume server evacuate to target server	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	b6471ecd75	err msg with duplicated local subscription detected move to log level 1 https://github.com/chrislusf/seaweedfs/issues/3320	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	6c390851e7	fix design	2022-07-20 18:08:12 +05:00
Tuan Vuong	312e13416b	filter duplicated action	2022-07-20 16:43:50 +07:00
guol-fnst	ac694f0c8f	rename parameter and reuse functions rename milestone to watermark	2022-07-20 17:00:40 +08:00
Chris Lu	4820b594a7	Merge pull request #3298 from kmlebedev/evacuateTargetServer volume server evacuate to target server	2022-07-19 13:56:49 -07:00
Konstantin Lebedev	d3f7c09c03	remove ticker update the topology before each file	2022-07-20 00:54:23 +05:00
Konstantin Lebedev	11e393dbe7	err msg with duplicated local subscription detected move to log level 1 https://github.com/chrislusf/seaweedfs/issues/3320	2022-07-20 00:45:13 +05:00
Konstantin Lebedev	f6a966b4fc	add waiting log message	2022-07-20 00:31:57 +05:00
Konstantin Lebedev	6cfbfb0849	check for ping before deleting raft server https://github.com/chrislusf/seaweedfs/issues/3083	2022-07-20 00:04:12 +05:00
chrislu	2ae3f812f8	minor	2022-07-19 11:43:31 -07:00
Chris Lu	9667588af0	Merge pull request #3318 from kmlebedev/issues/3310 Use fallback if urls are not found	2022-07-19 11:38:43 -07:00
Chris Lu	f0d560060f	Merge pull request #3336 from kmlebedev/raftSrvRemove raftServerRemoval typo and logs	2022-07-19 11:32:17 -07:00
guol-fnst	91285bb51d	remove uncessary code	2022-07-19 20:00:44 +08:00
guol-fnst	b9256e0b34	optimiz	2022-07-19 16:58:53 +08:00
Konstantin Lebedev	f419d5643a	fix typo add remove logs	2022-07-19 11:50:52 +05:00
guol-fnst	bec9f79659	use snapshot	2022-07-19 11:14:41 +08:00

... 7 8 9 10 11 ...

6265 Commits