7222 Commits

Author	SHA1	Message	Date
Chris Lu	9e2d6e897e	Merge pull request #3198 from guol-fnst/fix_statuscode ... AbortMultipartUploadHandler should return 204 instead of 200	2022-06-19 22:57:13 -07:00
LHHDZ	8db9f13bc6	Merge branch 'master' into circuit_breaker	2022-06-20 13:42:18 +08:00
guol-fnst	1d77deccd0	AbortMultipartUploadHandler should return 204 instead of 200 ... https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html	2022-06-20 13:26:22 +08:00
石昌林	9e036df356	remove go.uber.org/atomic	2022-06-20 12:35:29 +08:00
geekboood	fdacd94af5	fix: invalid chunk data when failed to read manifests	2022-06-19 16:54:04 +08:00
石昌林	3dd60529c5	some code optimizations	2022-06-17 19:07:39 +08:00
石昌林	37df209195	add some unit tests and some code optimizes	2022-06-17 17:11:18 +08:00
Chris Lu	d28f8bb88f	Merge pull request #3192 from guol-fnst/weed_update ... remove "weed" from UsageLine, or weed will not show usage of weed update	2022-06-16 22:58:29 -07:00
guol-fnst	c7c20881f5	remove "weed" from UsageLine, or weed will not show usage of weed update	2022-06-17 12:47:04 +08:00
chrislu	625fd16a2e	reduce upfront memory usage for low density volume	2022-06-16 15:39:29 -07:00
Chris Lu	5f8d6b5f13	Merge pull request #3184 from guol-fnst/weed_update ... Weed update	2022-06-16 10:02:45 -07:00
guol-fnst	cd078471c4	fix path bug on windows ... fix usage	2022-06-16 17:24:56 +08:00
chrislu	aaa9938353	Revert "adjust conditions" ... This reverts commit e024586ff1.	2022-06-16 01:26:36 -07:00
guol-fnst	995e221e97	remove -output, add -dir,-name for more flexibility	2022-06-16 16:07:32 +08:00
chrislu	e024586ff1	adjust conditions	2022-06-15 23:19:39 -07:00
Chris Lu	82f3bcc65e	Merge pull request #3081 from paochiang/volume_upload_limit_fix ... exclude replication from the concurrentUploadLimitMB	2022-06-15 23:16:44 -07:00
liubaojiang	df0ce31a2e	add condition when inFlightUploadDataLimitCond signal	2022-06-16 14:07:11 +08:00
zzq09494	62f74f5e3c	add bucket label to s3 prometheus metrics	2022-06-16 13:21:25 +08:00
guol-fnst	4bc8792643	support specific version of weed	2022-06-16 11:40:23 +08:00
guol-fnst	1dbb925d29	remove unused function ... remove log when weed is up to date add check whether weed is full version	2022-06-16 11:40:20 +08:00
liubaojiang	3076ac101e	move vs.concurrentUploadLimit != 0 out of the lock	2022-06-16 09:58:44 +08:00
Chris Lu	097a12dfb4	Merge pull request #3187 from ningfdx/remote	2022-06-15 06:34:32 -07:00
Chris Lu	55b9f281d1	Merge pull request #3185 from kmlebedev/logging_err_process_range_req	2022-06-15 06:33:33 -07:00
石昌林	78b3728169	add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations ... configure s3 circuit breaker by 'command_s3_circuitbreaker.go': usage eg: # Configure the number of simultaneous global (current s3api node) requests s3.circuit.breaker -global -type count -actions Write -values 1000 -apply # Configure the number of simultaneous requests for bucket x read and write s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply # Configure the total bytes of simultaneous requests for bucket write s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply # Disable circuit breaker config of bucket 'x' s3.circuit.breaker -buckets x -enable false -apply # Delete circuit breaker config of bucket 'x' s3.circuit.breaker -buckets x -delete -apply	2022-06-15 21:07:55 +08:00
ningfd	338705f375	fix(wdclient): GetLocations return	2022-06-15 19:20:13 +08:00
Konstantin Lebedev	31a929f7f2	logging processRangeRequest errors	2022-06-15 15:36:43 +05:00
Chris Lu	4eeeb5f50d	Merge pull request #3180 from blacktear23/issue-3149	2022-06-15 00:55:20 -07:00
yulai.li	358c0c0b37	Fix: provide filer.ui.deleteDir options for server command	2022-06-15 15:53:31 +08:00
Guo Lei	3df84cabf6	Merge branch 'chrislusf:master' into weed_update	2022-06-15 15:28:40 +08:00
guol-fnst	695e8c7555	fix minor bug	2022-06-15 15:26:04 +08:00
zhihao.qu	4d0d1848c6	fix(filer.sync): modify clientName format : from -> to	2022-06-15 13:33:20 +08:00
yulai.li	b90d213a87	Change ui.deleteDir default to true	2022-06-15 11:37:37 +08:00
zhihao.qu	42d04c581b	feat(filer.sync): add metricsServer in filer.sync. ... Metrics include: (1) the offset of the filer.sync (2) the last send timestamp of the filer subscription	2022-06-15 11:33:18 +08:00
chrislu	d4ef06cdcf	3.11	2022-06-14 12:14:52 -07:00
chrislu	b347b2fb54	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-14 11:16:51 -07:00
chrislu	36d2756978	volume: avoid writing too much data for range requests in large files ... related to https://github.com/chrislusf/seaweedfs/issues/3178	2022-06-14 11:16:47 -07:00
yulai.li	28cda854eb	Add filer command line parameter to let Filer UI show/hide directory delete button	2022-06-15 00:30:49 +08:00
zhihao.qu	14d82c3dea	feat(filer.sync): add offset to path.	2022-06-14 19:46:02 +08:00
guol-fnst	78fc9cdfa0	add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate"	2022-06-14 19:10:54 +08:00
guol-fnst	322dbcf372	add update feature	2022-06-14 19:01:57 +08:00
Chris Lu	b22ca85fbb	Merge pull request #3169 from shichanglin5/fix_auth ... fix: When there is no access permission configured before startup, th…	2022-06-12 23:40:42 -07:00
shichanglin5	f01dd27752	fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup	2022-06-13 13:23:26 +08:00
chrislu	e437ec5a68	3.10	2022-06-12 19:30:27 -07:00
chrislu	d12f431d98	collect volume server status	2022-06-12 11:56:23 -07:00
chrislu	444ac21050	go fmt	2022-06-11 09:51:11 -07:00
chrislu	a74e926063	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-11 09:51:02 -07:00
chrislu	61b8c9c361	remote object store gateway: disable tagging for backblaze	2022-06-11 09:50:59 -07:00
Chris Lu	0a8b3948db	Merge pull request #3165 from blacktear23/reload-bug-fix	2022-06-10 07:47:43 -07:00
yulai.li	83f8cdd5cb	Bug Fix: force reload filer page when upload/delete/rename/create directory	2022-06-10 22:39:21 +08:00
guol-fnst	eabf4d4e64	fix typo	2022-06-10 22:03:11 +08:00
Chris Lu	36757d28a8	Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch ... stats master_replica_placement_mismatch	2022-06-10 06:44:16 -07:00
chrislu	9f8b72a54d	Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" ... This reverts commit 37da689319, reversing changes made to 00d53c34c4.	2022-06-10 06:38:17 -07:00
Konstantin Lebedev	fb57aa431c	stats master_replica_placement_mismatch	2022-06-10 15:30:40 +05:00
shichanglin5	a63559ee6c	fix: fix command prompt information	2022-06-10 12:46:10 +08:00
Chris Lu	37da689319	Merge pull request #3159 from shichanglin5/_duplicateUUID ... perf: Optimized volume handling duplicateUUID logic to avoid quitting…	2022-06-09 18:01:02 -07:00
shichanglin5	f5b0c04b14	perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal ... Under normal circumstances, there will be no problems, but when the master is debugged in the local environment, the volume client cannot communicate with the master normally, so the sendHeartBeat logic is restarted, and a new connection is created to report the heartbeat. If the master has not cleared the uuid of the volume at this time, then The master will respond to volume duplicateUUIDS, and the volume service will exit, but in fact the uuid of the volume is not duplicated	2022-06-09 20:41:16 +08:00
zhihao.qu	cd5cca36a4	feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges	2022-06-09 10:53:19 +08:00
Chris Lu	4a5135961f	Merge pull request #3154 from kmlebedev/s3_test_invalid_range	2022-06-08 01:45:40 -07:00
Konstantin Lebedev	49f058da85	fix s3 tests: ... ranged_request_invalid_range ranged_request_empty_object	2022-06-08 13:25:13 +05:00
chrislu	8a1166c021	auto remove previous leftover socket	2022-06-08 00:32:09 -07:00
chrislu	633a4aed18	filer: fix customizable local socket file name ... fix https://github.com/chrislusf/seaweedfs/issues/3147	2022-06-08 00:15:28 -07:00
chrislu	acbc0286e2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-07 22:33:38 -07:00
chrislu	ebd6097e6f	fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 ... related to https://github.com/chrislusf/seaweedfs/pull/3074	2022-06-07 22:33:35 -07:00
Chris Lu	f43ec9f363	Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud	2022-06-07 08:56:55 -07:00
Konstantin Lebedev	c07820178f	fix s3 tests ... bucket_list_delimiter_prefix bucket_list_delimiter_prefix_underscore bucket_list_delimiter_prefix_ends_with_delimiter	2022-06-07 14:43:10 +05:00
chrislu	78be3b0f6c	fix compilation	2022-06-06 10:09:21 -07:00
chrislu	4fd5f96598	filer: remove replication, collection, disk_type info from entry metadata ... these metadata can change and are not used	2022-06-06 00:39:35 -07:00
chrislu	cbf46de5f4	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-06 00:15:40 -07:00
chrislu	8402af0882	filer: detect ttl based on fs configure setting	2022-06-06 00:15:37 -07:00
ningfd	f32142f6f5	add disableXAttr in mount option	2022-06-06 14:09:01 +08:00
chrislu	4a046e4de7	3.09	2022-06-05 19:00:24 -07:00
chrislu	11f1e7996b	fix test	2022-06-05 18:50:41 -07:00
chrislu	d65bb2c6df	mount: file handle locks entry better ... related to https://github.com/chrislusf/seaweedfs/issues/2952	2022-06-05 18:15:06 -07:00
chrislu	746092a60b	add a comment	2022-06-05 16:44:07 -07:00
chrislu	056c480eb0	volume: checksum remove the hashing step in Value()	2022-06-05 15:24:02 -07:00
chrislu	ecef844dfc	stream read large files	2022-06-05 11:54:04 -07:00
chrislu	48cca4e54f	remove unused logic	2022-06-04 19:11:06 -07:00
chrislu	685643d506	needle read into a writer	2022-06-04 19:10:52 -07:00
chrislu	663bc5dc23	add needle reading in chunks	2022-06-04 18:15:39 -07:00
chrislu	18b29f709c	refactoring, split file	2022-06-03 23:31:14 -07:00
chrislu	39c3cb04af	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-03 08:45:32 -07:00
chrislu	f97acdd489	volume.fix.replication fix retry logic ... fix https://github.com/chrislusf/seaweedfs/issues/3136	2022-06-03 08:45:29 -07:00
creeew	02ae102731	fix filer.sync missing source srv uploaded files to target when target down	2022-06-02 01:28:47 +08:00
Konstantin Lebedev	b86628f85d	fix show empty dir ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-06-01 20:10:52 +05:00
chrislu	6793bc853c	help message when in simulation mode	2022-05-31 14:48:46 -07:00
chrislu	89948a373b	fix error reporting for "Need to a hdd disk type" ... related to https://github.com/chrislusf/seaweedfs/issues/3128	2022-05-31 12:43:55 -07:00
chrislu	8a49240d64	3.08	2022-05-31 11:57:41 -07:00
chrislu	376b005ad3	fix wrong logic	2022-05-31 11:55:58 -07:00
chrislu	fbd99d53c1	3.07	2022-05-31 00:36:24 -07:00
chrislu	27732ecfa4	move s3 related constants from package http to s3_constants	2022-05-30 22:57:41 -07:00
chrislu	f4a6da6cb2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-30 22:47:29 -07:00
chrislu	596c3860ca	use final destination to resolve fs configuration ... related to https://github.com/chrislusf/seaweedfs/issues/3075	2022-05-30 22:47:26 -07:00
Chris Lu	ca01ce0524	Merge pull request #3122 from chrislusf/filer-sync-with-peers ... Filer bootstrap from peers	2022-05-30 21:38:31 -07:00
chrislu	6adc42147f	fresh filer store bootstrap from the oldest peer	2022-05-30 21:27:48 -07:00
chrislu	490e0850bf	filer.meta.tail add one example of using untilTimeAgo	2022-05-30 20:24:59 -07:00
liubaojiang	f0ee3e6f21	reduce the scope of inFlightUploadDataLimitCond lock	2022-05-31 09:49:07 +08:00
chrislu	c59068d0f3	refactor	2022-05-30 16:28:36 -07:00
chrislu	682382648e	collect cluster node start time	2022-05-30 16:23:52 -07:00
chrislu	968ca95b49	filer.meta.tail: support untilTimeAgo for a range	2022-05-30 16:16:23 -07:00
chrislu	f214dfb1f5	stop when in memory log is done	2022-05-30 15:25:21 -07:00
chrislu	aece35a64f	stop when on disk log is done	2022-05-30 15:20:51 -07:00
chrislu	a2b101a737	subscribe metadata between a range	2022-05-30 15:04:19 -07:00
chrislu	8902fa6ff6	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-30 13:28:32 -07:00
Chris Lu	730c9cf601	Merge pull request #3114 from kmlebedev/nextcloud ... Create folders in s3 via nextcloud	2022-05-30 13:10:10 -07:00
chrislu	5b9347c938	typo	2022-05-30 12:37:41 -07:00
Konstantin Lebedev	af562e1a20	rm log info	2022-05-30 23:26:26 +05:00
Konstantin Lebedev	62e5e3822c	list self dir ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-05-30 18:09:08 +05:00
Konstantin Lebedev	26f3646961	show fake dirs ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-05-30 15:03:27 +05:00
chrislu	b8f3db0d46	s3: keep auth enabled in case identities are set to empty ... fix https://github.com/chrislusf/seaweedfs/issues/3084	2022-05-29 19:00:03 -07:00
chrislu	81d6159290	volume: report error if a volume has nil data backend ... fix https://github.com/chrislusf/seaweedfs/issues/3105	2022-05-29 16:59:30 -07:00
chrislu	ca16fbf0ef	shell: cluster.ps display filer group	2022-05-29 16:37:14 -07:00
chrislu	9219664505	reset sync offset if peer filer resets ... fix https://github.com/chrislusf/seaweedfs/issues/3089	2022-05-29 13:11:24 -07:00
Konstantin Lebedev	54356211c2	list empty folder	2022-05-27 13:17:18 +05:00
Kaiwalya Joshi	4598581741	fix: Minor spelling fixes on FilerStorePathTranslator.	2022-05-25 01:11:02 -07:00
chrislu	5b8b022985	remove unused parameter	2022-05-24 18:52:04 -07:00
Robert Coelho	0e6e72d462	mount: ReadDir return EIO on EnsureVisited err	2022-05-24 13:02:59 -04:00
Robert Coelho	1fabbe8a25	mount: cleanup ReadDir rewrite's branches to not assume offsets	2022-05-24 12:48:32 -04:00
Robert Coelho	cb422d96f7	mount: rewrite ReadDir to respect input.Offset to fix partial results	2022-05-24 12:16:58 -04:00
Robert Coelho	05602167f4	mount: do not mark meta_cache when there is an err	2022-05-24 12:13:15 -04:00
chrislu	7e25a2d416	reduce busy waiting when reading metadata logs	2022-05-24 00:23:53 -07:00
chrislu	2f846777bb	3.06	2022-05-23 01:21:48 -07:00
chrislu	a5cdcc7e63	fix reading files larger than 20GB due to manifest resolving bug	2022-05-23 01:16:10 -07:00
chrislu	59cc3c0bfd	fix reading files larger than 20GB due to manifest resolving bug ... this is introduced since Mar 08 2022	2022-05-23 01:14:56 -07:00
chrislu	acc318e12b	3.05	2022-05-22 21:43:59 -07:00
liubaojiang	076e48a676	add inflight upload data wait timeout	2022-05-21 10:38:08 +08:00
liubaojiang	71b2e6223e	add inFlightUploadDataLimitCond signal	2022-05-20 15:27:06 +08:00
liubaojiang	1a41691b4c	exclude replication from the concurrentUploadLimitMB	2022-05-20 14:40:05 +08:00
Chris Lu	cb64983c6b	Merge pull request #3079 from ztinpn/hotfix_fsync	2022-05-19 10:33:42 -07:00
tianzhang	66747ee9c9	hotfix_fsync fix fsync	2022-05-19 23:17:17 +08:00
Chris Lu	8efe55f981	Merge pull request #3059 from guol-fnst/avoid_dup_vol ... avoid loading duplicated volume directory	2022-05-17 23:48:57 -07:00
guol-fnst	8e2426a879	revert changes of pb file ... orignal protoc-gen-go-grpc verion is 1.1.0	2022-05-18 11:52:14 +08:00
guol-fnst	b12944f9c6	fix naming convention ... notify volume server of duplicate directoris improve searching efficiency	2022-05-17 15:41:49 +08:00
guol-fnst	076595fbdd	just exit in case of duplicated volume directories were loaded	2022-05-17 15:41:49 +08:00
guol-fnst	8fab39e775	rename UUID file ... fix typo move locationUUID into DiskLocation	2022-05-17 15:41:47 +08:00
Chris Lu	72e7dcde51	Merge pull request #3060 from natmaka/master	2022-05-16 07:42:58 -07:00
Nat Makarevitch	b72f0634d7	minor (typos...), done while reading around	2022-05-16 22:11:33 +08:00
guol-fnst	de6aa9cce8	avoid duplicated volume directory	2022-05-16 19:33:51 +08:00
Konstantin Lebedev	44f53ceda6	fix collectionIsMismatch charset	2022-05-16 13:23:23 +05:00
Konstantin Lebedev	10d435f2c2	fix skip loop	2022-05-16 13:16:27 +05:00
Konstantin Lebedev	279053572c	avoid delete volume replica if collection mismatch	2022-05-16 13:07:05 +05:00
chrislu	9ff0d99002	3.04	2022-05-15 21:32:21 -07:00
chrislu	866fed1eab	avoid grpc name conflict ... fix https://github.com/chrislusf/seaweedfs/issues/3055	2022-05-15 21:31:29 -07:00
chrislu	cd2d10118c	3.03	2022-05-15 18:17:07 -07:00
chrislu	271b5aed96	shell: volume.fsck add a note for -reallyDeleteFromVolume option	2022-05-15 11:07:04 -07:00
chrislu	4076d74813	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-15 00:43:39 -07:00
chrislu	29198720f2	s3: add grpc server to accept configuration changes	2022-05-15 00:43:37 -07:00
Chris Lu	ea4918394c	Merge pull request #3047 from shichanglin5/master ... fix the problem of metadata and tagging loss when files are copied	2022-05-13 22:52:48 -07:00
shichanglin5	688d55488c	test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go	2022-05-14 10:40:29 +08:00
chrislu	735038b2c1	backup do not need to use preallocation ... fix https://github.com/chrislusf/seaweedfs/issues/3044	2022-05-13 13:46:52 -07:00
shichanglin5	1166dead00	fix the problem of metadata and tagging loss when files are copied ... by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)	2022-05-13 19:46:20 +08:00
Konstantin Lebedev	d12a423aa0	refactor ... add some tests https://github.com/chrislusf/seaweedfs/pull/2996	2022-05-12 12:23:02 +05:00
Konstantin Lebedev	f523aed3c9	fixes from comments ... https://github.com/chrislusf/seaweedfs/pull/2996	2022-05-11 21:50:09 +05:00
chrislu	bff1ccc1de	fix compilation	2022-05-11 00:52:15 -07:00
chrislu	76da688176	adjust log level	2022-05-11 00:45:19 -07:00
chrislu	3e20336bde	fix atomic add int64 ... fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831	2022-05-11 00:44:35 -07:00
chrislu	a85720f0f6	remove deprecated function	2022-05-10 00:02:23 -07:00
chrislu	25559eeb92	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-10 00:01:46 -07:00
chrislu	791c29ae93	remove deprecated function ... WithQueryCachePolicyKeepInCache is deprecated and already default behavior	2022-05-10 00:01:43 -07:00
qiang.zeng	06cb337b82	fix typo	2022-05-09 14:21:16 +08:00
chrislu	a4ca3ed1f0	3.02	2022-05-08 22:28:32 -07:00
chrislu	8bc16f1149	rename	2022-05-06 23:38:22 -07:00
chrislu	139e039c44	filer.sync: pass attributes for mount ... fix https://github.com/chrislusf/seaweedfs/issues/3012	2022-05-06 03:54:12 -07:00
chrislu	10207a48a4	how to setup local test env	2022-05-06 02:25:56 -07:00
chrislu	db33b03735	add comments	2022-05-06 02:19:47 -07:00
chrislu	e350ba8896	larger test range	2022-05-06 02:17:28 -07:00
chrislu	71e0bcc5d5	a bit optimization	2022-05-06 02:11:58 -07:00
chrislu	b2a43dd044	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-06 01:51:28 -07:00
chrislu	d5fe5d12b7	filer: etcd fix pagination ... fix https://github.com/chrislusf/seaweedfs/issues/3016	2022-05-06 01:51:19 -07:00
Chris Lu	a69bf53405	Merge pull request #2996 from kmlebedev/ydb ... ydb filer store	2022-05-05 15:34:37 -07:00
Konstantin Lebedev	e6d2cb5911	ydb add empty doc file	2022-05-05 11:12:14 +05:00
chrislu	ef6c6c450e	avoid fatal error if port is already in use	2022-05-04 14:55:14 -07:00
chrislu	f7366a9668	skip unix socket mode for windows ... fix https://github.com/chrislusf/seaweedfs/issues/3013	2022-05-04 10:14:34 -07:00
Konstantin Lebedev	f58adaab25	build tag ydb since the growth of the binary is 5 mb	2022-05-03 23:20:09 +05:00
Konstantin Lebedev	f127b326bf	add options to scaffold	2022-05-03 22:54:31 +05:00
Konstantin Lebedev	7b544576af	refactor	2022-05-03 20:16:00 +05:00
chrislu	f17cd0d5cd	return false if not found ... fix https://github.com/chrislusf/seaweedfs/issues/3011	2022-05-03 07:18:34 -07:00
chrislu	b201edb9df	fix wrong assignment	2022-05-03 07:12:33 -07:00
Konstantin Lebedev	04d5dff6bb	add support native ttl	2022-05-03 17:52:23 +05:00
Konstantin Lebedev	cb3c7a3cdb	enable query cache policy instead of prepare	2022-05-03 16:03:10 +05:00
Konstantin Lebedev	8342f651f3	fix scanNamed	2022-05-03 15:18:28 +05:00
chrislu	9271866d1e	fix segmentation violation ... fix https://github.com/chrislusf/seaweedfs/issues/3000	2022-05-02 23:56:11 -07:00
Konstantin Lebedev	2d27afb515	Merge branch 'new_master' into ydb	2022-05-03 10:47:39 +05:00
chrislu	76148ce0f7	use storage_class for backend configuration ... fix https://github.com/chrislusf/seaweedfs/issues/3008	2022-05-02 12:20:01 -07:00
Konstantin Lebedev	d8925b4e83	Merge branch 'new_master' into ydb ... # Conflicts: # go.mod # go.sum	2022-05-03 00:13:57 +05:00
Konstantin Lebedev	7640e650e5	fix queries	2022-05-03 00:11:37 +05:00
Konstantin Lebedev	319d300d48	fix createDB	2022-05-02 22:23:07 +05:00
Konstantin Lebedev	0dc44dda63	ydb do Tx or DB	2022-05-02 15:33:29 +05:00
Konstantin Lebedev	a3e48831f3	ydb BucketAware interface	2022-05-02 12:42:20 +05:00
chrislu	73961e24d8	3.01	2022-05-01 23:33:45 -07:00
chrislu	00c1dfec4f	go fmt	2022-05-01 23:16:29 -07:00
chrislu	ce1e8e58ae	fix test	2022-05-01 23:15:58 -07:00
chrislu	2c0fdf3693	fix tests	2022-05-01 23:14:04 -07:00
chrislu	94635e9b5c	filer: add filer group	2022-05-01 21:59:16 -07:00
Konstantin Lebedev	50c4f62ed4	ydb-go-sdk move to v3	2022-05-02 02:07:47 +05:00
Konstantin Lebedev	21033ff4c3	refactor use const CountEntryChunksForGzip	2022-05-01 22:28:55 +05:00
Konstantin Lebedev	ec0ed41e37	ydb kv interface	2022-05-01 21:20:37 +05:00
chrislu	4bd6bea429	fix test	2022-05-01 09:02:01 -07:00
Konstantin Lebedev	6a052f6ff2	ydb Sql interface	2022-05-01 20:33:03 +05:00
Konstantin Lebedev	275c3bb19c	ydb initial ... https://github.com/chrislusf/seaweedfs/issues/2942	2022-05-01 17:28:58 +05:00
chrislu	5d081d5c3c	skip if already copied to remote ... fix https://github.com/chrislusf/seaweedfs/issues/2993	2022-04-30 20:03:04 -07:00
chrislu	1aae7a3f1b	volume.tier.upload progress starts negative #2992 ... fix https://github.com/chrislusf/seaweedfs/issues/2992	2022-04-30 18:10:01 -07:00
chrislu	ea8a7a5584	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-30 17:36:45 -07:00
chrislu	192983b464	s3 backend support customizing storage class	2022-04-30 17:36:40 -07:00
Konstantin Lebedev	03aec9ce7e	avoid panic if hashicorp raft not initialized	2022-04-30 13:56:35 +05:00
guosj	1a5de36257	fix return value	2022-04-29 11:19:36 +08:00
Chris Lu	4e97b6146a	Merge pull request #2982 from kmlebedev/issues/2981 ... avoid empty listMultipartUploads response	2022-04-27 17:21:44 -07:00
chrislu	b2a6111090	skip ipv6 all interfaces and localhost ... fix https://github.com/chrislusf/seaweedfs/issues/2983	2022-04-27 17:18:09 -07:00
Konstantin Lebedev	306cf70c4a	avoid empty listMultipartUploads response	2022-04-27 19:27:44 +05:00
chrislu	70e5a1b632	volume close should wait for committing compaction	2022-04-26 23:34:05 -07:00
Chris Lu	ffe028f8d0	Merge pull request #2974 from kmlebedev/wait_volume_closed_compression ... waite volume being closed during compression idx	2022-04-26 23:29:22 -07:00
chrislu	37ab8909b0	use two flags: v.isCompacting and v.isCommitCompacting	2022-04-26 23:28:34 -07:00
Chris Lu	b557faf175	Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid ... volume.check.disk add param volumeId	2022-04-26 22:09:06 -07:00
Chris Lu	eb49bd5638	Merge pull request #2970 from kmlebedev/vlm_fsck_forcePurging ... delete missing data from volumes in one replica	2022-04-26 22:08:34 -07:00
chrislu	94f824e1ce	volume: sync to disk before copying volume files ... address https://github.com/chrislusf/seaweedfs/issues/2976	2022-04-26 13:03:43 -07:00
Konstantin Lebedev	7315d1d039	waite volume being closed during compression idx	2022-04-26 13:40:42 +05:00
Konstantin Lebedev	d4343ab7da	forcePurging desc	2022-04-25 23:11:56 +05:00
Konstantin Lebedev	ae56b2c00f	change forcePurging to a pointer	2022-04-25 23:10:01 +05:00
Konstantin Lebedev	6d2fda27d2	delete missing data from volumes in one replica	2022-04-25 22:59:46 +05:00
Konstantin Lebedev	67476d830a	volume.check.disk add param volumeId	2022-04-25 18:27:14 +05:00
chrislu	2aef5b8b32	3.00	2022-04-24 21:46:03 -07:00
Chris Lu	184e982898	Merge pull request #2957 from guo-sj/handle_implicit_username ... Handle implicit username	2022-04-24 09:07:48 -07:00
guosj	8fd7b24b80	remove redundant logs & add unit test	2022-04-24 16:09:10 +08:00
guol-fnst	8b2552f944	fix http response error code	2022-04-24 11:07:27 +08:00
guosj	09befe871f	Merge branch 'master' into handle_implicit_username	2022-04-23 09:48:25 +08:00
guosj	893a0eea57	add more checks and comments	2022-04-22 22:47:11 +08:00
chrislu	ad01c63b84	conditionally skip hdfs related code	2022-04-21 01:43:01 -07:00
chrislu	4aa39ef33f	conditionally compile sqlite	2022-04-21 01:22:38 -07:00
chrislu	3885374edf	conditionally build elastic, gocdk to reduce binary size	2022-04-21 01:10:46 -07:00
chrislu	6994e88192	fix typo	2022-04-21 00:02:18 -07:00
chrislu	cb1e7aa27d	fix build	2022-04-20 16:21:56 -07:00
a	1d6a9e66b6	Merge branch 'master' into a	2022-04-20 14:01:42 -07:00
chrislu	bdc4d67de8	Revert "Merge pull request #2944 from guo-sj/handle_implicit_username" ... This reverts commit bc96e73b8d, reversing changes made to 88a669dd19.	2022-04-20 09:40:41 -07:00
chrislu	3aa4dc6ad5	Revert "fix index out of range" ... This reverts commit 4792e07228.	2022-04-20 09:40:24 -07:00
chrislu	4792e07228	fix index out of range	2022-04-20 09:32:17 -07:00
chrislu	f3fdfb07b8	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-19 22:58:59 -07:00
chrislu	8e9ad7db5a	fix bug deleting volume or unmount volume if a volume server has multiple directories	2022-04-19 22:58:57 -07:00
Chris Lu	bc96e73b8d	Merge pull request #2944 from guo-sj/handle_implicit_username ... handle implicit username	2022-04-19 21:42:21 -07:00
guosj	a23629b2fe	handle implicit username	2022-04-20 10:12:33 +08:00
Chris Lu	88a669dd19	Merge pull request #2858 from kmlebedev/fsck_replicas ... fsck replicas	2022-04-19 00:17:43 -07:00
Chris Lu	f76c318a1f	Merge pull request #2940 from kmlebedev/volume.list.grep ... volume.list show only readonly, collectionPattern and volumeId	2022-04-19 00:17:32 -07:00
Konstantin Lebedev	2364fab927	volume.list show only readonly, collectionPattern and volumeId	2022-04-19 11:44:41 +05:00
guosj	94ad557d97	add missing return	2022-04-19 14:16:32 +08:00
Konstantin Lebedev	36c5a59ed8	add help	2022-04-18 19:36:14 +05:00
Konstantin Lebedev	1e35b4929f	shell vacuum volume by collection and volume id	2022-04-18 18:40:58 +05:00
Chris Lu	97c6e0efb1	Merge pull request #2931 from kmlebedev/fix_nil_dataBackend ... avoid invalid memory address or nil pointer dereference	2022-04-18 00:13:26 -07:00
Konstantin Lebedev	9438738693	avoid invalid memory address or nil pointer dereference	2022-04-18 12:10:22 +05:00
chrislu	a69f0b62c2	start filer local socket only on non windows ... fix https://github.com/chrislusf/seaweedfs/issues/2930	2022-04-18 00:08:31 -07:00
chrislu	9ae66f55fc	2.99	2022-04-17 23:01:42 -07:00
chrislu	4524ceae6b	less verbose logs ... fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085	2022-04-17 22:55:26 -07:00
leyou240	89eb87c1d1	Merge branch 'master' into slices.SortFunc	2022-04-18 10:39:29 +08:00
justin	3551ca2fcf	enhancement: replace sort.Slice with slices.SortFunc to reduce reflection	2022-04-18 10:35:43 +08:00
chrislu	b597baf488	mount: set file write time correct to work well with rsync ... fix https://github.com/chrislusf/seaweedfs/issues/2867	2022-04-16 22:56:45 -07:00
chrislu	460d56d283	shell: cluster.check prints out clock delta and network latency	2022-04-16 13:24:17 -07:00
chrislu	b4be56bb3b	add timing info during ping operation	2022-04-16 12:45:49 -07:00
chrislu	1384529eb7	Fix filer.backup deletes files in backup folder in incremental mode ... fix https://github.com/chrislusf/seaweedfs/issues/2919	2022-04-14 13:35:01 -07:00
a	846858fb43	merge master	2022-04-12 14:42:29 -07:00
chrislu	1f03fcccb1	fix nil in cluster_check shell command ... fix https://github.com/chrislusf/seaweedfs/issues/2905	2022-04-12 08:47:27 -07:00
Chris Lu	dd13764a5c	Merge pull request #2900 from kmlebedev/fix_cleanupUploads ... avoid breaking loop in cleanupUploads if error is empty	2022-04-12 02:03:31 -07:00
Konstantin Lebedev	76b1c5ce67	cleanupUploads use jwt token	2022-04-12 13:46:11 +05:00
Chris Lu	72db181d68	Merge pull request #2901 from guol-fnst/fix_multiupload ... check object name and uploadID when processing multiupload	2022-04-11 23:38:12 -07:00
guol-fnst	44d810d163	rename functions and remove uncessary check	2022-04-12 13:40:53 +08:00
guol-fnst	180aa88a92	check uploadid using object name hash string	2022-04-12 11:04:38 +08:00
guol-fnst	2232cfb5b7	Check object name and uploadID when processing multipart uploading	2022-04-11 19:53:44 +08:00
Konstantin Lebedev	2dfbe210e0	avoid breaking loop in cleanupUploads if error is empty	2022-04-11 11:16:20 +05:00
Chris Lu	a87f57e47c	Merge pull request #2868 from kmlebedev/hashicorp_raft ... hashicorp raft	2022-04-10 23:00:05 -07:00
Konstantin Lebedev	931cb9e581	use "cluster.raft.{ps,add,remove}"	2022-04-11 10:50:01 +05:00
Konstantin Lebedev	d2fe0fae33	use the full ServerAddress as the identifier	2022-04-11 10:44:44 +05:00
chrislu	c6ec5269f4	2.98	2022-04-10 18:56:40 -07:00
chrislu	261ba0208e	lock-protected setting entry	2022-04-09 22:52:59 -07:00
Chris Lu	9f4ecd50ac	Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style ... Change filer UI icon from picture to icon font.	2022-04-08 00:50:32 -07:00
yulai.li	ceb058ddd9	Change filer UI icon from picture to icon font.	2022-04-08 15:07:56 +08:00
chrislu	7f26676f57	adjust td width	2022-04-07 22:34:32 -07:00
yulai.li	dff63e9024	Make operation buttons show when table row hovered	2022-04-08 13:10:12 +08:00
chrislu	4dba102137	use icon instead of button text	2022-04-07 11:15:01 -07:00
Chris Lu	bbc95dac47	Merge pull request #2885 from binbinshi/master ... fix: master lose some volumes	2022-04-07 10:16:19 -07:00
Konstantin Lebedev	104ea7b029	master ui add raft stats and Max Volume Id	2022-04-07 20:52:01 +05:00
a	41d396edc4	Merge branch 'master' into a	2022-04-07 08:22:59 -07:00
Konstantin Lebedev	a2fdb3e277	fix master ui	2022-04-07 19:37:40 +05:00
Konstantin Lebedev	35bc67f030	fix conflicts	2022-04-07 18:52:13 +05:00
Konstantin Lebedev	f5246b748d	Merge branch 'new_master' into hashicorp_raft ... # Conflicts: # weed/pb/master_pb/master.pb.go	2022-04-07 18:50:27 +05:00
yulai.li	11a9f993a8	Make filer-ui bootstrap style	2022-04-07 20:17:00 +08:00
Konstantin Lebedev	7ff248d5cd	refactor OnPeerUpdate	2022-04-07 16:23:22 +05:00
Konstantin Lebedev	85d80fd36d	fix removing old raft server	2022-04-07 15:31:37 +05:00
chrislu	abe3cc6df2	filer UI touch up	2022-04-07 01:25:55 -07:00
Chris Lu	869d23b099	Merge pull request #2884 from blacktear23/filer-ui-new-features ... Filer UI new features	2022-04-07 00:58:56 -07:00
chrislu	86ed27f602	mount: remove leftover socket file	2022-04-07 00:33:13 -07:00
yulai.li	2347c21cdd	Tune filer UI add rename feature	2022-04-07 15:19:41 +08:00
yulai.li	2454020a92	Add upload progress support	2022-04-07 15:19:37 +08:00
yulai.li	4f87ee7755	Add create directory and delete web UI features for filer	2022-04-07 15:19:33 +08:00
shibinbin	c20e1edd99	fix: master lose some volumes	2022-04-07 15:18:28 +08:00
eddy-gfx	ec53eec94f	Merge branch 'master' into a	2022-04-06 18:45:55 -05:00
chrislu	3ab2c0e5c0	avoid possible nil attributes	2022-04-06 10:11:11 -07:00
chrislu	9d44b89c21	Revert "avoid possible nil attributes" ... This reverts commit 0a6703c7f7.	2022-04-06 10:01:45 -07:00
chrislu	0a6703c7f7	avoid possible nil attributes	2022-04-06 09:58:48 -07:00
Konstantin Lebedev	14a2cc83bf	raft update peers via OnPeerUpdate	2022-04-06 21:17:04 +05:00
Konstantin Lebedev	357aa818fe	add raft shell cmds	2022-04-06 15:23:53 +05:00
chrislu	3d229fe45c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-06 00:28:51 -07:00
chrislu	79b8e6a8c3	add a place holder for later merge consecutive chunks	2022-04-06 00:28:48 -07:00
Konstantin Lebedev	c1aeded2dd	force raft bootstrap ... avoid err bootstrap only works on new clusters	2022-04-06 12:18:43 +05:00
Chris Lu	49ecb8d11c	Merge pull request #2879 from leyou240/master ... remove Redundant type conversion and use strings.TrimSuffix to enhance readability	2022-04-06 00:02:19 -07:00
justin	d02f13c2d1	remove Redundant type conversion and use strings.TrimSuffix to enhance readability	2022-04-06 14:58:09 +08:00
root	f15a737a0a	Add "Location:" in response when creating bucket ... according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html"	2022-04-06 13:56:27 +08:00
chrislu	bc888226fc	erasure coding: tracking encoded/decoded volumes ... If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.	2022-04-05 19:03:02 -07:00
chrislu	8b3d76b24d	prevent concurrent access during shutdown	2022-04-05 10:53:36 -07:00
chrislu	bb4beebce3	prevent nil	2022-04-05 10:49:17 -07:00
Konstantin Lebedev	0e796a5582	rm set NoSnapshotRestoreOnStart	2022-04-05 18:28:42 +05:00
Konstantin Lebedev	68f11b9687	sleep bootstraping	2022-04-05 15:42:19 +05:00
Konstantin Lebedev	1ae7f509f5	LocalID from raw string	2022-04-05 13:50:39 +05:00
Konstantin Lebedev	89d32a0670	fix glog Info	2022-04-05 13:43:34 +05:00
Konstantin Lebedev	17c6e8e39f	Merge branch 'new_master' into hashicorp_raft ... # Conflicts: # go.mod # go.sum	2022-04-05 13:29:46 +05:00
Konstantin Lebedev	b7cdde14ae	auto bootstraping and update peers	2022-04-05 13:17:53 +05:00
chrislu	4aae87f405	check missing hdd disk type	2022-04-04 14:48:00 -07:00
Konstantin Lebedev	622297f1a7	add stats raft handler	2022-04-04 19:16:06 +05:00
Konstantin Lebedev	1a1e5778c3	fix cluster status	2022-04-04 18:52:08 +05:00
Konstantin Lebedev	14dd971890	hashicorp raft with state machine	2022-04-04 17:51:51 +05:00
Konstantin Lebedev	c514710b7b	initial add hashicorp raft	2022-04-04 13:50:56 +05:00
chrislu	d310711de0	avoid possible deadlock: move metadata log flush channel out of lock scope	2022-04-03 21:53:30 -07:00
chrislu	77a7d7253f	2.97	2022-04-03 19:08:01 -07:00
chrislu	e8d7bb42e2	avoid concurrent access to map object ... fix https://github.com/chrislusf/seaweedfs/issues/2866	2022-04-03 01:08:25 -07:00
chrislu	6a2bcd03aa	configure mount quota	2022-04-02 21:34:26 -07:00
chrislu	958f880b70	mount: add grpc method to adjust quota	2022-04-02 15:14:37 -07:00
chrislu	a8eefda452	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-01 20:25:37 -07:00
chrislu	105578a2f2	skip pinging self for master and volume server	2022-04-01 20:25:35 -07:00
root	b7e34016d4	fix error response in case of wrong expire time format	2022-04-02 11:01:33 +08:00
chrislu	4ecba915f3	add check between peers	2022-04-01 17:40:25 -07:00
chrislu	daca2d22a5	use original server address string as map key	2022-04-01 17:34:42 -07:00
chrislu	4b5c0e3fa9	check cluster connectivities	2022-04-01 17:27:49 -07:00
chrislu	6d55716fc3	use constants	2022-04-01 16:55:26 -07:00
chrislu	bbbbbd70a4	master supports grpc ping	2022-04-01 16:50:58 -07:00
chrislu	743ad690b6	filer supports grpc ping	2022-04-01 16:44:58 -07:00
chrislu	2305508b65	refactor: separate into two files	2022-04-01 16:40:49 -07:00
chrislu	800cbc004c	volume server adds ping function	2022-04-01 16:37:06 -07:00
a	7e92517571	change user and pass to username and password	2022-04-01 14:09:25 -05:00
Konstantin Lebedev	7f1383a41e	findExtraChunksInVolumeServers in consideration of replication	2022-04-01 14:45:41 +05:00
Konstantin Lebedev	3817e05dd0	fix collect filer files	2022-04-01 10:17:09 +05:00
Konstantin Lebedev	3cedb21bb7	skip new entities	2022-03-31 21:36:10 +05:00
Konstantin Lebedev	1d9e30d8c0	fsck replicas	2022-03-31 19:10:06 +05:00
Konstantin Lebedev	4f5b018a02	collecting only bucket volumes	2022-03-31 13:35:58 +05:00
Konstantin Lebedev	691ed50957	use constant for ErrBucketNotEmpty	2022-03-31 08:44:48 +05:00
Konstantin Lebedev	b6a2b43efe	allowDeleteBucketNotEmpty	2022-03-30 22:46:13 +05:00
Konstantin Lebedev	a8b15c676c	s3 test bucket delete nonempty	2022-03-30 21:06:26 +05:00
a	549e341c6f	resolv	2022-03-30 06:31:19 -05:00
Chris Lu	ce8ef60c59	Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit ... fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit	2022-03-30 01:27:13 -07:00
Konstantin Lebedev	6317f6a8a9	fix Description	2022-03-30 13:04:15 +05:00
Chris Lu	8732cc24c8	Merge pull request #2849 from guo-sj/update_user ... Add AWS IAM update user API and its test case	2022-03-29 22:36:26 -07:00
Konstantin Lebedev	993283bb1d	revert sort.Search	2022-03-29 19:53:12 +05:00
Konstantin Lebedev	5b90a39954	fix s3 tests: ... multipart_upload_incorrect_etag multipart_resend_first_finishes_last	2022-03-29 19:21:09 +05:00
Konstantin Lebedev	9231971c88	fix test_s3.test_multi_object_delete_key_limit and ... test_multi_objectv2_delete_key_limit	2022-03-29 16:55:09 +05:00
guosj	ed07e76f0e	add test function	2022-03-29 17:21:59 +08:00
a	bc603e534f	diff	2022-03-28 16:50:28 +00:00
Konstantin Lebedev	84b7b83517	fix permission mkdir snapshot ... avoid open file operation not permitted	2022-03-28 18:41:52 +05:00
Chris Lu	93615b2a49	Merge pull request #2828 from guo-sj/fix-tagging-error ... Fix delete all "Seaweed-" prefixed tagging error	2022-03-28 00:35:22 -07:00
guosj	dd2b9d93cc	update code according to PR's comment	2022-03-28 15:23:31 +08:00
guosj	02076f01a6	fix tagging error	2022-03-28 14:50:48 +08:00
justin	d51a724101	fix: encode small chunk return error maybe have some bug.	2022-03-28 13:11:24 +08:00
chrislu	0490ee87ef	2.96	2022-03-27 16:11:17 -07:00
chrislu	86d4c52d6e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-26 13:33:20 -07:00
chrislu	21e0898631	refactor: change masters from a slice to a map	2022-03-26 13:33:17 -07:00
chrislu	4ba7127ab1	refactor	2022-03-26 13:13:19 -07:00
Chris Lu	3153c836a7	Merge pull request #2744 from kmlebedev/issues_2545 ... avoid connect to the old filler address	2022-03-26 13:02:49 -07:00
chrislu	cae998eda1	Revert "Revert "remove duplicated metadata subscription in filer"" ... This reverts commit 34b743c481.	2022-03-26 13:00:16 -07:00
chrislu	34b743c481	Revert "remove duplicated metadata subscription in filer" ... This reverts commit 34742be029. Related to https://github.com/chrislusf/seaweedfs/issues/2545	2022-03-26 12:33:45 -07:00
chrislu	fba1cfc2d6	simplify a bit	2022-03-26 10:24:05 -07:00
chrislu	a3411dd9da	refactor	2022-03-26 10:21:26 -07:00
Chris Lu	1b27f96669	Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers ... Fix remove deleted peers	2022-03-25 11:36:15 -07:00
Konstantin Lebedev	ddd3945c26	fix remove deleted peers of raft server ... https://github.com/chrislusf/seaweedfs/issues/2804	2022-03-25 15:09:38 +05:00
Konstantin Lebedev	c1450bf9fe	always clear previous log to avoid server is promotable ... https://github.com/chrislusf/seaweedfs/issues/2804	2022-03-25 13:40:19 +05:00
Chris Lu	89d84e275b	Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request ... Need to exit waiting if request is was canceled	2022-03-24 12:21:44 -07:00
Konstantin Lebedev	0b790d2bbf	exclude the replication from the concurrentUploadLimitMB	2022-03-24 13:54:42 +05:00
chrislu	9b04f17555	remove dead code	2022-03-23 23:18:32 -07:00
chrislu	0b5faef612	fix	2022-03-23 23:16:54 -07:00
chrislu	0562fceb99	volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. ... fix https://github.com/chrislusf/seaweedfs/issues/2755	2022-03-23 22:53:58 -07:00
Konstantin Lebedev	849a2c1f68	fix cmdFiler	2022-03-23 14:30:46 +05:00
Konstantin Lebedev	105bcfdbd4	set iam ip from filer	2022-03-23 13:45:51 +05:00
chrislu	e48764be75	s3: multipart upload verifies uploaded parts	2022-03-23 01:05:14 -07:00
chrislu	18ae8943ef	better clean up upload directory	2022-03-23 01:03:51 -07:00
chrislu	a0bad1c997	remove any go mod changes ... This reverts commit 6c7f7d6887.	2022-03-21 23:04:00 -07:00
chrislu	6c7f7d6887	Revert "Merge pull request #2782 from SadmiB/upstream" ... This reverts commit a644b7236a, reversing changes made to 349257f822.	2022-03-21 23:00:50 -07:00
Chris Lu	0d692a6254	Merge branch 'master' into upstream	2022-03-21 11:02:04 -07:00
SadmiB	d12540c9f2	Add contabo api client	2022-03-21 17:16:49 +01:00
chrislu	4e2388e1b5	mount: set file size if it is only on remote gateway	2022-03-21 02:09:54 -07:00
chrislu	8f0410af2c	2.95	2022-03-21 01:47:03 -07:00
chrislu	ae558fa073	log reasons volumes became unwritable	2022-03-21 00:41:44 -07:00
chrislu	57c6eddd22	avoid possible deadlock if volume layout is used in some logs	2022-03-21 00:04:01 -07:00
chrislu	581c93d970	change to config_dir, to be consistent with "weed scaffold -config" ... fix https://github.com/chrislusf/seaweedfs/issues/2753	2022-03-19 00:24:52 -07:00
chrislu	b75c6dcff5	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-19 00:22:52 -07:00
chrislu	3da2b83b38	Added a "-conf_dir" option to customize *.toml configuration file directory. ... fix https://github.com/chrislusf/seaweedfs/issues/2753	2022-03-19 00:22:47 -07:00
elee	beb406bbbb	fix ls, onBucketDelete still not triggering	2022-03-18 22:21:57 -05:00
elee	411c0df3fe	switch to multi collection, change readme	2022-03-18 21:51:16 -05:00
zzq09494	9f16df736e	fix-filer: calculation error of the method skipCheckParentDirEntry	2022-03-19 09:15:42 +08:00
elee	1cea6c73d3	update readme	2022-03-18 00:34:19 -05:00
elee	25be96832a	update readme with index info	2022-03-18 00:33:08 -05:00
elee	06f23aa675	put in delete folder children query	2022-03-18 00:29:48 -05:00
elee	0701feeb17	added more to readme	2022-03-18 00:26:11 -05:00
elee	171c27ddf1	zz	2022-03-17 22:35:30 -05:00
elee	2f0cdcdceb	update readme	2022-03-17 21:26:16 -05:00
elee	423ce57cde	prefix search, bucket implemented	2022-03-17 21:12:25 -05:00
chrislu	2824940ecf	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-17 16:54:30 -07:00
chrislu	f247cab5cd	skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1	2022-03-17 16:54:29 -07:00
Berck Nash	7ee38fa3a4	The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems: ... (1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified. Instead, it bound to localhost which broke the default configuration. This change implements the documented behavior instead. (2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP. I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?	2022-03-17 15:30:23 -06:00
elee	bf745bdccb	revise comment	2022-03-17 14:09:34 -05:00
elee	a0df993cef	cleanup, add a few comments	2022-03-17 14:08:36 -05:00
elee	94883e2fad	ok then	2022-03-17 05:23:12 -05:00
elee	f3ab6769e9	make start from indexing work	2022-03-17 05:09:13 -05:00
elee	31571dd96e	fix file listing	2022-03-17 05:07:40 -05:00
elee	b25d03340b	Merge branch 'a' of github.com:gfxlabs/seaweedfs into a	2022-03-17 04:51:59 -05:00
elee	921535001a	arangodb adapter	2022-03-17 04:49:26 -05:00
chrislu	6886167d5f	minor optimization	2022-03-17 00:03:28 -07:00
chrislu	7542fd7f4d	mount: optimize a bit when writing files	2022-03-17 00:02:38 -07:00
chrislu	4042fdf3bb	rename to skipCheckParentDir ... related to https://github.com/chrislusf/seaweedfs/pull/2761 It's better to default to false.	2022-03-16 23:55:31 -07:00
zzq09494	40b0033fa7	go fmt	2022-03-17 14:19:48 +08:00
zzq09494	ee5417a5b3	fix: ut test error	2022-03-17 11:16:46 +08:00
zzq09494	81cce4b4c3	filer: support uploading file without needEnsureParentDir	2022-03-17 10:53:47 +08:00
zzq09494	a6a8892255	Revert "filer: support uploading file without needEnsureParentDir" ... This reverts commit a93c4947ba.	2022-03-17 10:27:17 +08:00
zzq09494	a93c4947ba	filer: support uploading file without needEnsureParentDir	2022-03-17 10:18:23 +08:00
Berck Nash	9b14f0c81a	Add mTLS support for both master and volume http server.	2022-03-16 09:52:17 -06:00
chrislu	ed818c90a9	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-15 22:28:21 -07:00
chrislu	3639cad69c	master, filer, s3: also listen to "localhost" in addition to specific ip address ... related to https://github.com/chrislusf/seaweedfs/issues/1937	2022-03-15 22:28:18 -07:00
Konstantin Lebedev	f43c6daeda	Need to exit waiting if request is was canceled	2022-03-15 19:55:22 +05:00
Konstantin Lebedev	c2e7d663bc	reduced the timeout for creating a connection from 30 to 10 seconds	2022-03-15 13:32:05 +05:00
chrislu	fbc9f0eb64	minor	2022-03-14 03:19:16 -07:00
chrislu	2eda3a686f	2.94	2022-03-14 00:55:01 -07:00
chrislu	bd5c5586b5	generate inode via path and time	2022-03-14 00:03:29 -07:00
chrislu	5cba8e51c5	refactor	2022-03-13 18:34:57 -07:00
chrislu	f2f68f675e	write to disk during random writes, limiting total disk spaces used	2022-03-13 18:17:35 -07:00
chrislu	2f4cd20f90	tests skip reader pattern monitoring	2022-03-13 18:15:53 -07:00
chrislu	53513475bf	mount: add back random read support ... avoid too much memory used also	2022-03-13 01:38:52 -08:00
chrislu	f70c1e449b	add useful doc link	2022-03-13 00:14:50 -08:00
chrislu	b20ddc57a7	mount: return open status	2022-03-12 22:38:14 -08:00
chrislu	b40d252761	mount: chmod for root	2022-03-12 12:10:56 -08:00
chrislu	3a6eb8ca5f	default bind to one ip address ... fix https://github.com/chrislusf/seaweedfs/issues/1937	2022-03-11 14:02:39 -08:00
Konstantin Lebedev	f53cff045f	checks disk file exist	2022-03-10 18:58:56 +05:00
Konstantin Lebedev	834210a9dc	avoid connect to the old filler address ... https://github.com/chrislusf/seaweedfs/issues/2545	2022-03-10 15:24:45 +05:00
chrislu	197ade6aeb	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-09 22:26:55 -08:00
chrislu	011a41b561	add back writes to swap file when too many in memory chunks are used.	2022-03-09 22:26:51 -08:00
zzq09494	2ea18fca48	fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command'	2022-03-10 13:37:34 +08:00
banjiaojuhao	f28dbbe5c5	[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.	2022-03-10 11:40:39 +08:00
chrislu	e2b07737da	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-08 23:02:33 -08:00
chrislu	dc204dd137	fix nil entry ... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d340b4] goroutine 130523 [running]: github.com/chrislusf/seaweedfs/weed/filer.FileSize(...) /code/seaweedfs/weed/filer/filechunks.go:26 github.com/chrislusf/seaweedfs/weed/mount.(*WFS).Lookup(0xc000866d80, 0x1, 0xc002897f40, {0xc004b00980, 0x39}, 0x1ec19e0) /code/seaweedfs/weed/mount/weedfs_dir_lookup.go:59 +0x654 github.com/hanwen/go-fuse/v2/fuse.doLookup(0xc00033c000, 0xc00033c000) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/opcode.go:333 +0x6b github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000ab2420, 0xc00033c000) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:483 +0x1f3 github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000ab2420, 0x0) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:456 +0x110 created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:323 +0x534	2022-03-08 23:02:30 -08:00
banjiaojuhao	f7f2a597dd	minor	2022-03-08 16:22:55 +08:00
banjiaojuhao	d61bea9038	[bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized).	2022-03-08 16:22:55 +08:00
chrislu	3aeee3d748	ensure releasing file handle	2022-03-07 14:01:24 -08:00
chrislu	8136384473	remove debug message	2022-03-07 11:22:26 -08:00
banjiaojuhao	b9ff7723dd	[bugfix] filer: nil pointer dereference	2022-03-07 23:26:25 +08:00
chrislu	da3d330616	s3 and filer transport using unix domain socket instead of tcp	2022-03-07 02:00:14 -08:00
chrislu	0cb17b45b1	refactoring	2022-03-07 01:59:01 -08:00
chrislu	6d3db4445b	buffer for all range requests	2022-03-07 01:56:47 -08:00
chrislu	f3bcbeb60a	a little optimization	2022-03-07 00:24:59 -08:00
chrislu	f7ee60996c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-07 00:08:09 -08:00
chrislu	6f3ec989a7	fix manifest batch size	2022-03-07 00:07:53 -08:00
Chris Lu	1e7fcef581	Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest ... filer: support get metadata with resolved manifest chunk	2022-03-07 00:05:46 -08:00
chrislu	bb0b784544	minor	2022-03-07 00:04:59 -08:00
banjiaojuhao	bfcc9ca808	filer: support metadata with resolved manifest chunk	2022-03-07 15:47:51 +08:00
banjiaojuhao	71f3046841	filer: add back isAppend function	2022-03-07 15:41:07 +08:00
chrislu	0ba4e4cd23	2.93	2022-03-06 18:54:12 -08:00
chrislu	46a28b8819	mount: adjust disk space based on quota	2022-03-06 17:22:49 -08:00
chrislu	ede6ce44c6	fix test	2022-03-06 17:09:55 -08:00
chrislu	21ef152423	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-06 17:04:24 -08:00
chrislu	f3442e36e6	mount: quota adjust error type to be syscall.ENOSPC	2022-03-06 17:04:21 -08:00
Chris Lu	89cd0c27a8	Merge pull request #2728 from kmlebedev/set_default_leveldb2	2022-03-06 07:26:30 -08:00
Konstantin Lebedev	cf444ebd07	Set default leveldb2 enabled ... avoid Filer store is enabled for both leveldb2 and mysql	2022-03-06 18:27:25 +05:00
banjiaojuhao	a07c93f4dd	filer: reset bytesBuffer before use	2022-03-06 21:07:36 +08:00
chrislu	c7e8ac18f0	mount: quota for one mounted collection ... related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48	2022-03-06 02:44:40 -08:00
chrislu	b7c992f410	add flag to enforce quota	2022-03-05 22:10:43 -08:00
chrislu	f1713c96ae	avoid possible runtime error: index out of range [0] with length 0	2022-03-05 21:14:31 -08:00
chrislu	da76af187f	mount: avoid possible index out of bounds error	2022-03-04 22:36:01 -08:00
chrislu	6e49e75a5b	use logical number of files and sizes for statistics and quota	2022-03-04 18:47:44 -08:00
chrislu	f51e20028a	mount: avoid comma in mount options ... fix https://github.com/chrislusf/seaweedfs/issues/2719	2022-03-03 03:42:29 -08:00
chrislu	dc0f48682c	unused	2022-03-03 03:41:35 -08:00
chrislu	28b8974a3a	mount: fix directory pagination when using midnight commander	2022-03-03 02:59:31 -08:00
chrislu	a96d4254e9	filer, s3, volume server: a bit memory optimization	2022-03-02 20:15:28 -08:00
chrislu	6fbbc78574	stream reading a whole chunk	2022-03-02 13:50:46 -08:00
chrislu	784583afc6	avoid pool memory allocation if too large	2022-03-02 13:50:28 -08:00
chrislu	ba14307319	2.92	2022-02-28 15:22:19 -08:00
chrislu	fcf3714443	mount: add back support for filer.path	2022-02-28 12:16:53 -08:00
chrislu	80c017907b	filer.backup: fix backing up encrypted chunks ... I have done filer.backup test: replication.toml: [sink.local] enabled = true directory = "/srv/test" ___ system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec system@dat1:/srv/test$ ls -l total 16 drwxr-xr-x 2 system system 4096 Feb 28 12:39 a -rw-r--r-- 1 system system 48 Feb 28 12:39 fu.txt -rw-r--r-- 1 system system 32 Feb 28 12:39 _index.html -rw-r--r-- 1 system system 68 Feb 28 12:39 index.php system@dat1:/srv/test$ cat fu.txt ? ?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~?? system@dat1:/srv/test$ On the active mount on the target server it's: system@app1:/srv/app$ ls -l total 2 drwxrwxr-x 1 system system 0 Feb 28 12:04 a -rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt -rw-r--r-- 1 system system 4 Feb 28 12:04 _index.html -rw-r--r-- 1 system system 40 Feb 28 12:04 index.php system@app1:/srv/app$ cat fu.txt This is static boy! Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData It seems like it's still encrypted?	2022-02-28 10:07:06 -08:00
chrislu	aad62ee148	mount: mark directory uncached if forgotten	2022-02-28 02:08:24 -08:00
chrislu	554e239097	release file handle	2022-02-28 01:23:14 -08:00
chrislu	3639fedd01	mount: fix fsync opened and renamed files	2022-02-28 00:34:17 -08:00
chrislu	63a9d8f01d	ensure inodes are not duplicating unless hardlinked	2022-02-27 23:13:49 -08:00
chrislu	de77d00c81	correctly clean up for a file	2022-02-27 23:12:28 -08:00
chrislu	18543c6e8b	minor	2022-02-27 23:11:09 -08:00
chrislu	09cd00f356	2.91	2022-02-27 04:03:39 -08:00
chrislu	941ced60a4	download 2 chunks if at the beginning of a file	2022-02-27 03:57:24 -08:00
chrislu	f9d9eed0c9	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-27 03:41:35 -08:00
chrislu	d602d68fd1	remove dead code	2022-02-27 03:41:32 -08:00
Chris Lu	6c07af6014	Merge pull request #2715 from gfx-labs/acl-pr	2022-02-27 03:27:32 -08:00
chrislu	c3792c8352	remove dead code	2022-02-27 03:03:19 -08:00
chrislu	aa9eef81e6	retire mount v1	2022-02-27 02:57:27 -08:00
eddy-gfx	fb940dd807	Merge branch 'chrislusf:master' into acl-pr	2022-02-27 04:52:08 -06:00
elee	881a0fe806	ensure compatibility	2022-02-27 04:50:59 -06:00
elee	954ad98e0d	set canned acl on replication create	2022-02-27 04:49:31 -06:00
chrislu	6e1ab97988	use debug option to see operations	2022-02-27 02:02:30 -08:00
chrislu	2112d99140	mount2: add back readonly mode	2022-02-27 01:13:32 -08:00
chrislu	9ef5bb20f6	mount2: invalidate fuse cache for replaced inode	2022-02-27 00:00:23 -08:00
chrislu	4ddcbaab57	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-26 23:20:47 -08:00
chrislu	551d00d51a	prefetch other chunks when stream reading	2022-02-26 23:20:45 -08:00
Chris Lu	e1d3dd5e18	Merge pull request #2712 from guo-sj/correct_comments	2022-02-26 09:16:12 -08:00
guosj	82cad5e330	correct comments	2022-02-26 22:44:26 +08:00
chrislu	7b1a713d2a	remove dead code	2022-02-26 03:23:15 -08:00
chrislu	3a58b7bac6	a little safer	2022-02-26 03:23:06 -08:00
chrislu	708e14fcfa	avoid possible too big memory allocation	2022-02-26 03:22:41 -08:00
chrislu	3345a50d9b	prefetch 2 chunks	2022-02-26 03:06:17 -08:00
chrislu	86ce69f709	remove logs	2022-02-26 03:00:20 -08:00
chrislu	b2a148cb4c	use file size as max range	2022-02-26 03:00:08 -08:00
chrislu	2ab0ad24a3	use memory pool	2022-02-26 02:59:19 -08:00
chrislu	28b395bef4	better control for reader caching	2022-02-26 02:16:47 -08:00
chrislu	3ad5fa6f6f	chunk cache adds function ReadChunkAt	2022-02-25 21:55:04 -08:00
chrislu	fc7a4957ea	fix mount2 options	2022-02-25 21:22:44 -08:00
chrislu	72c0233938	less logs	2022-02-25 15:34:24 -08:00
chrislu	101e6d80b0	mount2: listXattr return ok if xattr is empty	2022-02-25 14:38:56 -08:00
chrislu	03466f955e	rename: delete source entry metadata only, skipping hard links	2022-02-25 02:57:54 -08:00
chrislu	8080fe4cc1	logs	2022-02-25 02:56:23 -08:00
chrislu	e423548673	rename: pass along entry metadata	2022-02-25 02:53:37 -08:00
chrislu	e8110bb54c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-25 01:17:28 -08:00
chrislu	202a29d014	refactoring	2022-02-25 01:17:26 -08:00
chrislu	91d6785cf3	define metadata action types	2022-02-25 00:54:16 -08:00
chrislu	be3fc77391	mount2: use consistent inode	2022-02-25 00:53:27 -08:00
Chris Lu	9873bae115	Merge pull request #2706 from guo-sj/fix_log_info	2022-02-24 22:48:19 -08:00
guosj	121b31f750	fix incorrect log information	2022-02-25 13:41:20 +08:00
chrislu	ceaf993a27	mount2: add rdev	2022-02-24 14:51:25 -08:00
chrislu	95717d1006	simpler output	2022-02-24 13:50:08 -08:00
chrislu	b05962b90e	rename: handle hard links	2022-02-24 02:59:00 -08:00
chrislu	e31ec04f4f	mount2: POSIX deleted opened file nlink should be 0	2022-02-24 01:59:37 -08:00
chrislu	419e355e9e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-24 01:31:07 -08:00
chrislu	91f0481f4e	mount2: SetAttr set mode correctly	2022-02-24 01:31:04 -08:00
Chris Lu	97a4b66df7	Merge pull request #2704 from guo-sj/fix_bugs_in_return_value ... fix return value in storage/volume_vacuum.go:444	2022-02-24 00:49:29 -08:00
chrislu	abe7214c1f	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-24 00:37:59 -08:00
chrislu	b93d57da31	mount2: dir read opened file	2022-02-24 00:37:57 -08:00
guosj	3e7aa1caf5	fix return value in storage/volume_vacuum.go:444	2022-02-24 15:54:36 +08:00
Chris Lu	6834df77a0	Merge pull request #2701 from guo-sj/fix_bugs_in_return_value	2022-02-23 19:23:37 -08:00
guosj	26f3ab8d4b	fix a return bug in func (c *commandVacuum) Do	2022-02-24 09:04:38 +08:00
chrislu	320637dc7a	use "mv.from" for moving files	2022-02-23 15:34:42 -08:00
chrislu	c29bc9a367	fix error handling	2022-02-23 15:34:25 -08:00
Chris Lu	da58c748bc	Merge pull request #2698 from guo-sj/fix_bugs_in_return_value	2022-02-23 00:41:30 -08:00
Chris Lu	bd092d8318	Merge pull request #2697 from guo-sj/fix_bugs_in_return_value	2022-02-23 00:30:52 -08:00
guosj	d68c27f82d	fix another return value bug	2022-02-23 16:21:25 +08:00
guosj	8f9aa0cddd	fix bugs in return value	2022-02-23 16:17:48 +08:00
Tuan Vuong	d2ec62656d	initialize master address in iam options	2022-02-23 12:01:54 +07:00
banjiaojuhao	6ab09e9071	filer_http: support uploading file with offset	2022-02-22 00:15:00 +08:00
chrislu	497ebbbd45	2.90	2022-02-20 22:00:13 -08:00
chrislu	56da3494cb	leveldb3: adjust memory allocation for each bucket	2022-02-20 13:40:13 -08:00
banjiaojuhao	e6126cef62	filer_web: support moving entry	2022-02-20 23:56:23 +08:00
chrislu	7bc67399e4	listing for filer stores without prefixed query: break if no more progress	2022-02-19 10:12:51 -08:00
chrislu	248c0c8087	minor	2022-02-19 09:54:09 -08:00
chrislu	4ee0a6f47b	filer store: reduce one possible listing operation	2022-02-19 00:43:42 -08:00
chrislu	b3594278c9	optimize a bit	2022-02-18 23:36:10 -08:00
chrislu	61811dc2f1	comments	2022-02-18 22:14:40 -08:00
chrislu	daa27b2119	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-18 20:55:10 -08:00
chrislu	9014d00fd0	Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist" ... This reverts commit 6cf2e7d493.	2022-02-18 20:54:54 -08:00
banjiaojuhao	4c30934cd9	filer: support get file entry	2022-02-18 22:52:26 +08:00
chrislu	63062ed7f0	mount2: fix unlink	2022-02-18 01:10:53 -08:00
chrislu	93e12d5f30	fix build	2022-02-18 00:48:00 -08:00
chrislu	d62370d4e0	add todo	2022-02-18 00:47:15 -08:00
chrislu	e8ce30fdc5	mount2: adjust file mode	2022-02-18 00:47:02 -08:00
chrislu	f9d33f70b0	return fuse.Status when looking up by inode	2022-02-18 00:45:43 -08:00
chrislu	b9cf4f12fc	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-16 21:33:51 -08:00
chrislu	49b84b6e2a	list entries while reading from remote	2022-02-16 21:32:15 -08:00
Chris Lu	38dcaaa76e	Merge pull request #2683 from guo-sj/fix_fsconfigure_bug ... fix minor bug in commandFsConfigure.Do	2022-02-16 18:16:47 -08:00
guosj	ca121ecd66	fix minor bug in commandFsConfigure.Do	2022-02-17 09:57:52 +08:00
chrislu	65a19e3abc	fix listing with correct inode	2022-02-16 17:01:39 -08:00
chrislu	6ac066d1dc	count lookup or not	2022-02-16 16:49:03 -08:00
chrislu	a129bda7d9	sync data first before stopping	2022-02-16 09:11:34 -08:00
chrislu	118d0e01a8	less logs	2022-02-16 08:45:07 -08:00
chrislu	a6bc67c34c	less logs	2022-02-16 08:38:51 -08:00
chrislu	2facd65998	fix second listing	2022-02-16 08:16:27 -08:00
chrislu	1013fc90d8	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-16 07:57:11 -08:00
chrislu	22739c653a	clean up	2022-02-16 07:57:08 -08:00
Chris Lu	3cb19cf6db	Merge pull request #2680 from kmlebedev/volume_healthz	2022-02-16 04:43:44 -08:00
Konstantin Lebedev	9ea09cc41c	healthz check to avoid drain pod with last replicas	2022-02-16 14:18:36 +05:00
chrislu	65bfeafb5a	add back mkdir	2022-02-16 01:09:21 -08:00
chrislu	37e8fce841	clean up cache on exit	2022-02-16 00:39:21 -08:00
chrislu	1560ec7e26	remove unused code	2022-02-16 00:37:24 -08:00
chrislu	3cbce878f2	mount2: fix directory pagination	2022-02-15 22:42:10 -08:00
chrislu	df51e0c042	mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying	2022-02-15 16:18:36 -08:00
chrislu	98cce6a150	mongodb: ensure closing cursor	2022-02-15 16:15:44 -08:00
Chris Lu	aa7fc299b8	Merge pull request #2676 from banjiaojuhao/add_filer_store-redis_lua ... FilerStore: add redis_lua	2022-02-15 12:10:34 -08:00
banjiaojuhao	fc3b75f2f8	weed/operation/delete_content.go: nil pointer dereference	2022-02-15 23:07:03 +08:00
banjiaojuhao	b5ec346700	FilerStore: add redis_lua	2022-02-15 20:54:57 +08:00
chrislu	17ac5244c3	mount2: avoid double listing directories	2022-02-15 01:44:17 -08:00
guosj	db7766323e	complete project code, remain test code	2022-02-15 17:35:33 +08:00
chrislu	222798d926	mount2: fix for read dir plus on linux	2022-02-15 00:29:17 -08:00
chrislu	4e72863ba5	mount2 add debug mode	2022-02-15 00:26:30 -08:00
Chris Lu	64afbc5235	Merge pull request #2640 from Radtoo/fsck_with_delete ... Added basic deletion capabilities to fsck.	2022-02-14 23:00:01 -08:00
chrislu	4e181db21a	mount: default disable cache ... * Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43 * Improve read write benchmarks * Improve AI training performance. Most of the files are just read once.	2022-02-14 20:42:33 -08:00
chrislu	ebc22625b4	less noisy on mac	2022-02-14 16:38:01 -08:00
chrislu	377bf31445	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-14 13:48:51 -08:00
chrislu	e8420aaed7	fix building for windows freebsd	2022-02-14 13:48:48 -08:00
Chris Lu	d3ee621fce	Merge pull request #2661 from garenchan/ck-dev1	2022-02-14 10:08:37 -08:00
Konstantin Lebedev	526094d2da	StopTimeout 30 sec	2022-02-14 21:42:27 +05:00
Konstantin Lebedev	275e9a4e86	reduce to default http server KillTimeout and StopTimeout	2022-02-14 21:38:24 +05:00
garenchan	bd032eabe7	[UPDATE] Make heartbeat interval and election timeout of masters configurable.	2022-02-14 21:09:07 +08:00
chrislu	ff666104c4	fix GOOS	2022-02-14 03:14:05 -08:00
chrislu	05724a68d4	skip other OS	2022-02-14 02:59:51 -08:00
chrislu	4244ef8b72	fix building on linux	2022-02-14 02:59:27 -08:00
chrislu	b9c2bff931	clean up	2022-02-14 02:14:26 -08:00
chrislu	a990cd29cd	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-14 01:52:25 -08:00
chrislu	6a40fd1c65	2.89	2022-02-14 01:52:16 -08:00
Chris Lu	305418fac5	Merge pull request #2667 from kmlebedev/syncReplicatedWrite ... atomically write to replicas	2022-02-14 01:48:21 -08:00
Chris Lu	aa13168b4d	Merge pull request #2668 from chrislusf/mount2 ... Mount2	2022-02-14 01:41:09 -08:00
chrislu	fe57a2e770	file set attribute	2022-02-14 01:36:10 -08:00
Konstantin Lebedev	0ed76a0556	clearly	2022-02-14 14:10:06 +05:00
chrislu	dbeeda8123	listen for metadata updates	2022-02-14 01:09:31 -08:00
Konstantin Lebedev	36013f63ed	https://github.com/chrislusf/seaweedfs/issues/2648	2022-02-14 13:59:12 +05:00
chrislu	7286e525ad	support write	2022-02-13 23:27:11 -08:00
chrislu	2b955c1713	support read	2022-02-13 22:50:44 -08:00
chrislu	f3c1e00521	rename	2022-02-13 19:16:56 -08:00
chrislu	bb9919b07a	add open release, refactor	2022-02-13 19:14:34 -08:00
Eng Zer Jun	b92df1654c	test: use T.TempDir to create temporary test directory ... The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2022-02-14 10:38:13 +08:00
chrislu	072f923a9b	add doc	2022-02-13 16:56:35 -08:00
chrislu	3d0e9e5197	supports renaming	2022-02-13 16:34:57 -08:00
chrislu	417b59b893	doc	2022-02-13 14:37:47 -08:00
chrislu	24290fed9d	add comments	2022-02-13 06:02:21 -08:00
chrislu	6200b6abb1	avoid fatal message in some edge cases	2022-02-13 05:59:10 -08:00
chrislu	6a921e15f3	forget() factor in nlookup	2022-02-13 05:49:29 -08:00
chrislu	f8af0f93d9	support link	2022-02-13 05:29:43 -08:00
chrislu	0381338d40	fix import	2022-02-13 04:23:06 -08:00
chrislu	a1ef0e48a9	doc	2022-02-13 04:22:02 -08:00
chrislu	be40ff6048	add symlink	2022-02-13 03:50:16 -08:00
chrislu	3d93570979	supports forget	2022-02-13 03:31:47 -08:00
chrislu	6a42cb6b0b	supports mknod, unlink	2022-02-13 03:09:24 -08:00
chrislu	813b868b9a	add rmdir	2022-02-13 01:43:11 -08:00
chrislu	e85ca10a1a	add mkdir	2022-02-13 01:34:19 -08:00
chrislu	21046c6a28	split files	2022-02-13 01:05:30 -08:00
chrislu	a4c9223b9d	support xattr	2022-02-13 00:58:46 -08:00
chrislu	c81833a192	add directory setAttr	2022-02-12 23:08:56 -08:00
chrislu	5c48c23235	remove println	2022-02-12 22:45:07 -08:00
chrislu	7cfbf1e85f	fix ok status	2022-02-12 22:41:45 -08:00
chrislu	4c75fd5f9c	sync format	2022-02-12 22:41:29 -08:00
chrislu	b0a5193e32	working	2022-02-12 22:21:30 -08:00
chrislu	661a34e23d	Merge branch 'master' into mount2	2022-02-12 17:58:32 -08:00
chrislu	5e814afe88	blocks count	2022-02-12 17:58:12 -08:00
Radtoo	724ce04b1c	We picked the second flag's name.	2022-02-13 00:53:35 +01:00
chrislu	5a0a709016	it runs, but directory listing output is not showing up	2022-02-12 05:27:16 -08:00
chrislu	866981d8ac	rename	2022-02-12 02:49:15 -08:00
chrislu	72faae91e1	implement read directory and read directory plus	2022-02-12 02:48:44 -08:00
chrislu	a10c28ba82	simplify	2022-02-12 01:59:36 -08:00
chrislu	f4d88862c4	can attr root directory	2022-02-12 01:54:16 -08:00
chrislu	180445f5a8	change to use fuse file system	2022-02-11 21:35:09 -08:00
chrislu	45a0fda9bd	need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go	2022-02-11 03:12:52 -08:00
chrislu	f87da798a4	to be re-written following fuse virtual file system	2022-02-11 03:09:30 -08:00
chrislu	9a913457dd	supports stats	2022-02-10 23:23:47 -08:00
chrislu	b6143de52a	mount with name	2022-02-10 22:43:55 -08:00
chrislu	7a0c35674c	clean up previously mounted folder	2022-02-10 20:46:53 -08:00
chrislu	c3f9d9fa2e	initial setup	2022-02-10 20:32:13 -08:00
Radtoo	fbe7ed7927	Revert the the flag to "reallyDeleteFromVolume".	2022-02-10 20:01:48 +01:00
chrislu	21aaa4c1f1	ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) ... fix https://github.com/chrislusf/seaweedfs/issues/2642	2022-02-08 01:51:13 -08:00
chrislu	13f6ec1c4e	test checking ec distribution	2022-02-08 01:50:05 -08:00
chrislu	f34c2ff7c5	use embed txt	2022-02-08 01:14:27 -08:00
chrislu	c8c7c10c3f	volume.tier.move: avoid double counting ... related to https://github.com/chrislusf/seaweedfs/issues/2637	2022-02-08 00:57:35 -08:00
chrislu	f18803424a	volume.balance: add delay during tight loop ... fix https://github.com/chrislusf/seaweedfs/issues/2637	2022-02-08 00:53:55 -08:00
chrislu	9860405974	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-07 23:08:57 -08:00
chrislu	85c1615b43	filer read empty file may cause OOM in some cases ... fix https://github.com/chrislusf/seaweedfs/issues/2641	2022-02-07 23:08:54 -08:00
Chris Lu	a2ac540ecc	Merge pull request #2645 from guol-fnst/fix_Precedence ... fix preconditions	2022-02-07 18:32:16 -08:00
root	7f0c793083	fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions	2022-02-08 10:13:19 +08:00
chrislu	b1cff07ab0	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-07 11:38:01 -08:00
chrislu	81f86c381d	s3: avoid nil response ... fix https://github.com/chrislusf/seaweedfs/issues/2636	2022-02-07 11:37:59 -08:00
Chris Lu	f46763d74f	Merge pull request #2644 from kmlebedev/vacum_fsync ... updated needle with fsync	2022-02-07 11:33:58 -08:00
Konstantin Lebedev	ef541972f8	updated needle with fsync	2022-02-08 00:10:53 +05:00
Chris Lu	1b7fb3ce84	Merge pull request #2643 from kmlebedev/graceful_volume_stop	2022-02-07 09:58:26 -08:00
Konstantin Lebedev	fb97e234c9	skips compact if store is stopping	2022-02-07 20:16:15 +05:00
chrislu	9405eaefdb	filer.sync: fix replicating partially updated file ... Run two servers with volumes and fillers: server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002 server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008 Run Active-Passive filler.sync: filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive Upload file to 11007 port: curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/" If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again: curl "http://localhost:11007/testFacebook/9.xml" EQUALS curl "http://localhost:11001/testFacebook/9.xml" However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file Снимок экрана 2022-02-07 в 14 21 11 This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten. Deleted chunks are not actually deleted on the server to which the file is replicated.	2022-02-07 03:46:28 -08:00
Radtoo	fbb14e0ea8	Adding separate toggle to purge absent vols ... While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected. Also renamed forcePurge to just purge.	2022-02-06 23:46:52 +01:00
Radtoo	6b17f45da2	Also delete paths for which a volume is entirely absent.	2022-02-06 23:22:04 +01:00
Radtoo	fa0cfdfb7f	Added basic deletion capabilities to fsck.	2022-02-06 17:05:58 +01:00
chrislu	433fde4b18	move error to a separate file ... This file contains metric names for all errors The naming convention is ErrorSomeThing = "error.some.thing"	2022-02-04 22:57:51 -08:00
Chris Lu	a23fcb9a7c	Merge pull request #2634 from kmlebedev/errorMetrics ... error metrics for filer and store	2022-02-04 22:35:13 -08:00
Chris Lu	247bbabda5	Merge pull request #2632 from lapshin-vitaly/s3api_errors ... add s3api error for copy in file, not directory	2022-02-04 22:34:34 -08:00
chrislu	ced3b89395	add util package	2022-02-04 21:34:58 -08:00
chrislu	affe3c2c12	change to util.WriteFile	2022-02-04 21:32:27 -08:00
chrislu	76e297d64f	sync call to write file, avoid vif loading error ... fix https://github.com/chrislusf/seaweedfs/issues/2633	2022-02-04 11:14:04 -08:00
zerospiel	f3364fec99	weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting ... Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject). Added generic comment rule of traversing methods	2022-02-04 15:14:48 +03:00
Konstantin Lebedev	9978f54acf	fix metric names	2022-02-04 16:45:16 +05:00
Konstantin Lebedev	3f4e17aa24	error metrics for filer and store	2022-02-04 14:07:14 +05:00
Lapshinn Vitaly	6bdc274d4d	add s3api error for copy in file, not directory	2022-02-04 03:28:37 +03:00
chrislu	6cf2e7d493	s3: listObjectParts return ErrNoSuchUpload if does not exist ... ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc { "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.	2022-02-03 12:34:16 -08:00
zerospiel	b54a65ba5a	weed/s3api: added new bucket handlers for more compatibility with AWS S3 ... Protocol Otherwise any requests to the underlying handlers results in calls to ListObjects (v1) that may intensively load gateway and volume servers. Added the following handlers with default responses: - GetBucketLocation - GetBucketRequestPayment Added the following handlers with NotFound and NotImplemented responses: - PutBucketAcl - GetBucketPolicy - PutBucketPolicy - DeleteBucketPolicy - GetBucketCors - PutBucketCors - DeleteBucketCors	2022-02-03 17:17:05 +03:00
chrislu	6bee1e9714	[volume.check.disk] fix wrong logic to compare ... fix https://github.com/chrislusf/seaweedfs/issues/2626 minuend - subtrahend	2022-02-01 07:48:28 -08:00
chrislu	7270067289	2.88	2022-01-30 20:25:26 -08:00
chrislu	84c9bc4389	edge case: old entry was not replicated to remote storage	2022-01-30 20:23:24 -08:00
chrislu	b8490fe427	adjust volume count even when not applying the changes	2022-01-28 19:11:46 -08:00
Chris Lu	b3f0f170b4	Merge pull request #2618 from divanikus/master	2022-01-28 03:34:52 -08:00
chrislu	b9b684194f	remove max connection age ... following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779	2022-01-27 02:28:22 -08:00
divanikus	67e3fe996a	async volumeTierMove	2022-01-26 18:22:31 +03:00
chrislu	62d815d1ca	use limited in memory buffer instead of swap file	2022-01-26 00:36:19 -08:00
Chris Lu	9596fce562	Merge pull request #2613 from kmlebedev/masterPromMetrics ... Master prom metrics	2022-01-25 08:13:14 -08:00
Konstantin Lebedev	5c9259fa3c	fix metrics master name	2022-01-25 14:42:47 +05:00
chrislu	7328962009	revert "POSIX: should not delete if a directory is not empty" ... revert 0c75f15062 Reported: Hi, about commit: 0c75f15062 POSIX: should not delete if a directory is not empty It should still delete with a command like rm -rf "${path}"/ because it is a forced delete, but now it gets fail to delete non-empty folder: [...] Can you enable the delete if it is forced?	2022-01-24 14:21:50 -08:00
Konstantin Lebedev	c9952759c4	metrics master is leader	2022-01-24 20:13:07 +05:00
Konstantin Lebedev	28efe31524	new master metrics	2022-01-24 19:09:43 +05:00
chrislu	3bba2124ef	use a sliding window of in-memory writable chunks	2022-01-23 23:02:05 -08:00
chrislu	520591e6ea	reset swap file chunk after uploading	2022-01-23 18:30:53 -08:00
chrislu	e185d90d24	2.87	2022-01-23 16:18:55 -08:00
chrislu	4a311c7f5e	dedup local metadata subscribers ... fix https://github.com/chrislusf/seaweedfs/discussions/2542	2022-01-23 16:14:22 -08:00
chrislu	bb6854b972	adjust log	2022-01-22 12:35:09 -08:00
chrislu	2bdd737971	rename file	2022-01-22 08:28:35 -08:00
chrislu	643bbbeb49	rename	2022-01-22 08:27:40 -08:00
chrislu	18ed06b420	use memory when under 16 chunks	2022-01-22 08:11:01 -08:00
chrislu	8e80f3cd65	move upload pipeline locking to a different file	2022-01-22 08:09:55 -08:00
chrislu	dde34fa99d	rename	2022-01-22 08:08:01 -08:00
chrislu	a2aa542370	rename	2022-01-22 08:07:11 -08:00
chrislu	c376ccc5a5	swap file based random write large file upload	2022-01-22 08:06:53 -08:00
chrislu	b9ae16fbc5	fix memory allocation	2022-01-22 08:05:04 -08:00
chrislu	1ee828b768	refactor ... do not expose internal offset	2022-01-22 06:34:29 -08:00
chrislu	02d0c12cdd	rename	2022-01-22 06:00:10 -08:00
chrislu	5dea5c0449	refactor	2022-01-22 05:59:07 -08:00
chrislu	8aa6bf0bb9	refactoring	2022-01-22 05:40:10 -08:00
chrislu	d97bd54e63	just refactoring	2022-01-22 04:18:54 -08:00
chrislu	482014f9da	rename file	2022-01-22 03:50:18 -08:00
chrislu	3b4a9addaf	rename	2022-01-22 01:46:10 -08:00
chrislu	4acfc098e9	re-order	2022-01-22 01:43:14 -08:00
chrislu	e71dcfb3a6	add logging for memory allocation	2022-01-22 01:35:12 -08:00
chrislu	9d0f58c329	skip printing fs configuration	2022-01-21 13:29:47 -08:00
chrislu	ce2049cdb6	refactoring, move genFn before saveFn	2022-01-21 12:08:58 -08:00
chrislu	e47f63d159	enforce bucket quota	2022-01-21 02:34:42 -08:00
chrislu	6e57d8d0de	s3: check bucket usage and adjust read only according to quota	2022-01-21 02:15:27 -08:00
chrislu	606667f205	able to configure the quota for a bucket	2022-01-21 01:42:20 -08:00
chrislu	b1063162b6	display bucket quota	2022-01-21 00:55:04 -08:00
chrislu	f103491912	s3: list bucket size from weed shell	2022-01-21 00:26:49 -08:00
chrislu	6c7135d77e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-21 00:17:06 -08:00
chrislu	ea57654e34	refactoring	2022-01-21 00:16:50 -08:00
Chris Lu	7c66f3b5fb	Merge pull request #2602 from kmlebedev/master_metrics ... master metricsHttpPort	2022-01-20 09:26:25 -08:00
chrislu	b3e526ba95	url should be always using forward slash	2022-01-19 22:16:26 -08:00
Konstantin Lebedev	77c98b657e	master metricsHttpPort	2022-01-19 21:43:22 +05:00
chrislu	77362700e1	S3: fail fast when "X-Amz-Copy-Source" is a folder ... fix #2593	2022-01-18 12:04:40 -08:00
chrislu	05c3c3f56b	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-17 23:38:13 -08:00
chrislu	9b77f0054e	2.86	2022-01-17 23:38:03 -08:00
Chris Lu	ec254d8a89	Merge pull request #2597 from guol-fnst/gocql_to ... add gocql timeout setting	2022-01-17 23:35:13 -08:00
chrislu	9274557552	keep dirty pages based on temp file	2022-01-17 23:23:49 -08:00
guol-fnst	da9540e666	add gocql timeout setting	2022-01-18 15:21:13 +08:00
chrislu	c87b8f4c30	S3: fail fast when "X-Amz-Copy-Source" is a folder ... fix https://github.com/chrislusf/seaweedfs/issues/2593	2022-01-17 23:09:37 -08:00
chrislu	b2acfd75e9	ensure entry view cache is invalidated	2022-01-17 23:02:30 -08:00
chrislu	f4ad63528a	wait for reading threads to complete before dropping sealed chunks	2022-01-17 22:24:44 -08:00
chrislu	0a3f95ca01	more logs	2022-01-17 20:41:00 -08:00
chrislu	b068bc291d	testing with always resetting entry view cache	2022-01-17 20:07:01 -08:00
chrislu	047446d5ca	remove extra async execution	2022-01-17 15:50:11 -08:00
chrislu	7bf7af971b	more logs	2022-01-17 14:15:10 -08:00
chrislu	fc22071a2f	more logs	2022-01-17 14:02:37 -08:00
chrislu	381f4e73a0	delete actual reference first	2022-01-17 13:56:47 -08:00
chrislu	0ba88596e8	invalidate filehandle entry view cache	2022-01-17 13:53:30 -08:00
chrislu	1734017ba1	add test	2022-01-17 13:40:41 -08:00
chrislu	da7f13e73e	Revert "testing skip memory management" ... This reverts commit 6c908352cb.	2022-01-17 03:21:31 -08:00
chrislu	6c908352cb	testing skip memory management	2022-01-17 03:19:24 -08:00
chrislu	77d9993f38	remove unused variables	2022-01-17 03:19:11 -08:00
chrislu	f710d5ffca	a little speed up	2022-01-17 03:19:00 -08:00
chrislu	fc0628c038	working	2022-01-17 01:53:56 -08:00
chrislu	1bd6d289d4	better locking on file handle	2022-01-15 05:45:29 -08:00
chrislu	2bfeb5d1c8	add filer to iam option	2022-01-15 03:37:52 -08:00
chrislu	b17c426e99	weed server: optionally start IAM service ... related to https://github.com/chrislusf/seaweedfs/issues/2560	2022-01-13 22:49:49 -08:00
chrislu	3c8b74318e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-13 13:03:07 -08:00
chrislu	8907e6a40a	add more help messages	2022-01-13 13:03:04 -08:00
banjiaojuhao	45e9c83421	padding zero for sparse file	2022-01-13 22:21:22 +08:00
chrislu	fe5b9e39cc	POSIX: check permission when removing items	2022-01-13 02:07:39 -08:00
chrislu	1453263b63	remove dead code	2022-01-13 02:02:04 -08:00
chrislu	e69c374956	minor	2022-01-13 02:01:53 -08:00
chrislu	9b954dc0d4	adjust make file	2022-01-13 01:33:13 -08:00
chrislu	f2847f1266	POSIX: check deletion permission	2022-01-12 23:58:11 -08:00
chrislu	0c75f15062	POSIX: should not delete if a directory is not empty	2022-01-12 23:57:54 -08:00
chrislu	de27058d0b	POSIX: differentiate device and char device	2022-01-12 21:45:38 -08:00
chrislu	d400a11832	POSIX: adjust source file ctime ... SeaweedFS uses mtime as ctime	2022-01-12 21:45:18 -08:00
chrislu	b44f05a2d0	POSIX: change timestamp on each attribute change	2022-01-12 19:31:25 -08:00
chrislu	15c01d8b7f	add some notes	2022-01-12 15:04:48 -08:00
chrislu	107a4884a8	shell: tighter memory allocation	2022-01-12 14:59:29 -08:00
chrislu	fec8428fd8	POSIX: different inode for same named different file types	2022-01-12 11:51:13 -08:00
chrislu	e82ad60122	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-12 11:07:49 -08:00
chrislu	caf0a3486b	POSIX: adjust ctime for file truncate	2022-01-12 11:07:39 -08:00
Konstantin Lebedev	edb753ab4d	https://github.com/chrislusf/seaweedfs/issues/2583	2022-01-12 16:04:59 +05:00
chrislu	adfd54e7c4	fix compilation	2022-01-12 01:24:24 -08:00
chrislu	6cc92817dc	add logs for request mode	2022-01-12 01:13:19 -08:00
chrislu	826a7b307e	master: remove hard coded filer settings in master.toml ... fix https://github.com/chrislusf/seaweedfs/issues/2529	2022-01-12 01:11:25 -08:00
chrislu	cd1ad88f30	POSIX: check name is too long ENAMETOOLONG	2022-01-12 00:16:00 -08:00
chrislu	2dcb8cb93b	POSIX: ensure file and directory inodes are different ... this is just an in memory representation. POSIX wants different inode numbers for the same named file or directory.	2022-01-11 23:44:48 -08:00
chrislu	5bb37d5905	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-11 23:29:12 -08:00
chrislu	10ecf80ca1	add a debug capability to list all metadata keys	2022-01-11 23:25:04 -08:00
Kyle Sanderson	9e012001be	filer.copy: don't crash when volume creation fails ... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7482 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7480 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225	2022-01-11 22:22:39 -08:00
chrislu	1a7d5b5b5e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-11 12:24:56 -08:00
chrislu	41daecfdca	Update mount_std.go	2022-01-11 12:23:12 -08:00
chrislu	2d0ccc4d34	add logs	2022-01-11 12:23:01 -08:00
Chris Lu	abe5da7d2c	Merge pull request #2575 from Radtoo/fix_paths2 ... Fix paths2	2022-01-11 12:04:30 -08:00
chrislu	b8fbf19e9a	mount: rename follow POSIX	2022-01-11 03:23:03 -08:00
chrislu	6a12520a96	fix logging	2022-01-10 01:00:11 -08:00
chrislu	cbc055dc2b	mount: file fsync ... fix https://github.com/chrislusf/seaweedfs/issues/2561	2022-01-10 00:52:16 -08:00
chrislu	19555385f7	2.85	2022-01-09 19:30:23 -08:00
Radtoo	389002f195	Using positional arguments rather than option flag to enable better shell usage	2022-01-08 16:52:12 +01:00
Radtoo	fba1efb77a	Now works with a single file too ... Parsing removed from doFixOneVolume Needle init removed from runFix	2022-01-08 16:31:53 +01:00
chrislu	110d5a5233	support fixing a collection of volumes, or volumes under one directory	2022-01-07 14:52:16 -08:00
chrislu	60dc450091	skip fixing read only volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2562	2022-01-06 09:52:28 -08:00
chrislu	3df8f96117	avoid changing inode	2022-01-06 01:36:11 -08:00
chrislu	67b0645808	mount: need to change entry name after renaming	2022-01-05 21:27:41 -08:00
chrislu	4de060daa6	mount: skip special character in the filenames ... fix https://github.com/chrislusf/seaweedfs/issues/2559	2022-01-05 03:57:24 -08:00
chrislu	e76105e2ab	fix auth permission checking	2022-01-03 21:05:20 -08:00
chrislu	a7887166cf	wildcard prefix to restrict access to directories in s3 bucket ... https://github.com/chrislusf/seaweedfs/discussions/2551	2022-01-03 15:39:36 -08:00
chrislu	5799a20f71	2.84	2022-01-02 17:05:19 -08:00
Chris Lu	42c849e0df	Merge branch 'master' into metadata_follow_with_client_id	2022-01-02 01:07:30 -08:00
Chris Lu	9b94177380	Merge pull request #2543 from skurfuerst/seaweedfs-158 ... FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client	2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst	c35660175d	BUGFIX: ensure Authorization header is only added once	2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst	1cd3b6b4e1	BUGFIX: security.toml contained wrong keys	2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst	10404c4275	FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client ... - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158	2021-12-30 14:45:27 +01:00
chrislu	34742be029	remove duplicated metadata subscription in filer ... https://github.com/chrislusf/seaweedfs/issues/2545	2021-12-30 01:51:52 -08:00
chrislu	5c87fcc6d2	add client id for all metadata listening clients	2021-12-30 00:23:57 -08:00
chrislu	fb434318e3	dynamically adjust connection timeout ... better fix for https://github.com/chrislusf/seaweedfs/issues/2541	2021-12-29 22:44:39 -08:00
chrislu	5788bf2270	s3: increase timeout limit ... https://github.com/chrislusf/seaweedfs/issues/2541	2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst	fcc09cef6f	Refactor: pass in claim type into security.DecodeJwt	2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst	d156d410ef	rename security.GenJwt to security.GenJwtForVolumeServer	2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst	eda4c43a08	fix typo in error message	2021-12-29 12:38:14 +01:00
chrislu	498661e3bb	mount: remove limits on number of parallel requests	2021-12-28 17:41:01 -08:00
chrislu	9a00c17555	reader: avoid wrong pattern detection due to lock waiting	2021-12-28 16:30:33 -08:00
chrislu	f7a6f6b4c0	if this is enabled, there are some "bus error" with git clone	2021-12-28 13:44:52 -08:00
chrislu	0da2dfd640	fuse: change to direct io mode ... before and after: chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 8192+0 records in 8192+0 records out 1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec) dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.86s system 84% cpu 4.561 total chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 8192+0 records in 8192+0 records out 1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec) dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.22s system 83% cpu 3.857 total	2021-12-28 12:22:56 -08:00
chrislu	3fd4da34a4	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-28 01:10:38 -08:00
chrislu	2422556456	monitor write pattern: avoid timing due to locking	2021-12-28 01:10:35 -08:00
chenkai	47c30e3add	filer list entries use context to break job	2021-12-28 15:03:41 +08:00
chrislu	80db8b13d8	bug: cleanup function was called twice	2021-12-27 20:53:02 -08:00
chrislu	67b723f74e	Filer Server API support fsync ... fix https://github.com/chrislusf/seaweedfs/issues/2528	2021-12-26 17:28:47 -08:00
chrislu	9f9ef1340c	use streaming mode for long poll grpc calls ... streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.	2021-12-26 00:15:03 -08:00
chrislu	c935b9669e	2.83	2021-12-25 01:01:34 -08:00
chrislu	eb4ad2546f	use proper chunk size limit option	2021-12-24 22:52:18 -08:00
chrislu	41bbf320bb	use 2MB chunk size. cache size is the wrong option	2021-12-24 22:50:19 -08:00
chrislu	982ea85d81	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-24 22:40:07 -08:00
chrislu	083d8e9ece	add stream writer ... this should improve streaming write performance, which is common in many cases, e.g., copying large files. This is additional to improved random read write operations: 3e69d19380...19084d8791	2021-12-24 22:38:22 -08:00
Sebastian Kurfürst	6db49100d6	BUGFIX: add access.ui setting to scaffolded security.toml ... ... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)	2021-12-24 13:59:04 +01:00
chrislu	255a1c7dcd	refactor type names	2021-12-23 18:23:18 -08:00
chrislu	f77ca41769	refactor	2021-12-23 17:48:34 -08:00
chrislu	1d36884845	rename files	2021-12-23 17:47:58 -08:00
chrislu	2d1a1f5e03	rename variables and functions	2021-12-23 17:35:57 -08:00
chrislu	7bf48ee135	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-23 17:23:26 -08:00
chrislu	6de331b014	clean up	2021-12-23 17:23:21 -08:00
chrislu	032df784ed	chunked file works now	2021-12-23 17:17:32 -08:00
banjiaojuhao	083bf3a137	filer server: add "datacenter, rack and datanode" for path specific configuration	2021-12-23 23:25:05 +08:00
chrislu	c2aad1c7ff	detect non streaming mode on first write request	2021-12-22 17:20:44 -08:00
chrislu	b541e39a2c	fix tests	2021-12-22 16:17:30 -08:00
chrislu	2bc6fa90ff	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-22 16:05:43 -08:00
chrislu	0ec7bc6710	detect non streaming mode on the first read	2021-12-22 16:05:38 -08:00
chrislu	4c1368d621	fix test	2021-12-22 16:05:08 -08:00
Chris Lu	4e73705533	Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node ... filer server: allow upload file to specific dataNode	2021-12-22 12:49:15 -08:00
banjiaojuhao	08336be92e	filer server: allow upload file to specific dataNode	2021-12-22 21:57:26 +08:00
chrislu	7b78fc72b0	add page chunk interval list	2021-12-22 02:53:33 -08:00
chrislu	b7cd52636b	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-21 22:24:43 -08:00
chrislu	3981d65b68	remove println	2021-12-21 22:24:38 -08:00
chrislu	9a73319b45	mount: different write strategy for streaming write and random write	2021-12-21 17:28:55 -08:00
Chris Lu	b0665a15f4	Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode	2021-12-21 08:56:51 -08:00
banjiaojuhao	dda6b90d25	assign fileId according to DataNode with empty DataCenter and Rack	2021-12-21 17:28:33 +08:00
chrislu	4b8dcff448	reverting default admin scripts ... fix https://github.com/chrislusf/seaweedfs/issues/2525 this new default value was introduced in 2.80 this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.	2021-12-20 13:34:57 -08:00
chrislu	b21a67bbe6	add writer pattern object for later use	2021-12-20 11:53:48 -08:00
chrislu	4fd29dad86	remove writeOnly flag	2021-12-20 01:11:43 -08:00
chrislu	bc96682760	refactor, change file locations	2021-12-20 01:02:23 -08:00
chrislu	866c2657f0	avoid FUSE cache only for the first 512 bytes	2021-12-19 23:13:36 -08:00
chrislu	0cb9036f66	mount: only cache the first chunk on stream read	2021-12-19 23:06:03 -08:00
chrislu	a152f17937	mount: improve read performance on random reads	2021-12-19 22:43:14 -08:00
chrislu	85c526c583	s3: bind to a specific IP ... fix https://github.com/chrislusf/seaweedfs/issues/2516	2021-12-17 11:34:37 -08:00
chrislu	5eacff9d4f	log message adds server name ... address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733	2021-12-16 10:46:26 -08:00
chrislu	50ddd8c8e2	remove debug messages ... fix https://github.com/chrislusf/seaweedfs/issues/2514	2021-12-16 00:58:15 -08:00
chrislu	7210558c7b	s3: pass through s3 presigned headers ... fix https://github.com/chrislusf/seaweedfs/discussions/2502	2021-12-15 13:18:53 -08:00
chrislu	bf4d7affc0	gateway to remote object store: skip replicating multipart upload part files ... fix https://github.com/chrislusf/seaweedfs/issues/2509	2021-12-14 19:48:31 -08:00
chrislu	316f326464	add more help message	2021-12-13 13:14:36 -08:00
Konstantin Lebedev	969f513265	disable audit log for s3 statusHandler	2021-12-13 16:08:19 +05:00
Chris Lu	ab0c6a84bf	Merge pull request #2503 from kmlebedev/audit_log_nonblocking ... Audit log force async	2021-12-13 00:47:16 -08:00
Konstantin Lebedev	34779e8f38	force enable asynchronous I/O sending events to Fluentd	2021-12-13 13:39:39 +05:00
chrislu	c3b73ec23b	2.82	2021-12-12 23:25:24 -08:00
chrislu	7bf14452f7	s3: handle Response-Content-Disposition	2021-12-12 23:05:40 -08:00
chrislu	94e5c0a454	skip purging from ec volumes	2021-12-12 22:55:27 -08:00
chrislu	a2d3f89c7b	add lock messages	2021-12-10 13:24:38 -08:00
chrislu	991a3dca0d	rename file	2021-12-10 13:11:00 -08:00
kmlebedev	4f98553ba9	audit log SignatureVersion	2021-12-10 19:40:32 +05:00
chrislu	2d6fcdf83a	add delay if need to resume from disk	2021-12-09 13:13:05 -08:00
liubaojiang	c7abc34dbb	rename operation returns the correct old parent path	2021-12-10 00:02:57 +08:00
Konstantin Lebedev	98251fe16a	non blocking audit log	2021-12-09 19:47:16 +05:00
chrislu	40ee9b4d7b	sleep between the gap of each execution ... related to https://github.com/chrislusf/seaweedfs/issues/2476	2021-12-07 11:44:08 -08:00
Chris Lu	2ba08afed1	Merge pull request #2498 from kmlebedev/s3_audit_log	2021-12-07 09:35:48 -08:00
Konstantin Lebedev	10678cde81	audit log config	2021-12-07 18:20:52 +05:00
Konstantin Lebedev	4ec8715f20	audit log	2021-12-07 12:15:48 +05:00
Konstantin Lebedev	3ac48cd540	audit log	2021-12-07 12:13:21 +05:00
chrislu	7f0a97c7b6	shell: volume.list supports different verbosity level	2021-12-05 21:54:40 -08:00
chrislu	5ea9715721	2.81 ... also sync java client version to SeaweedFS version	2021-12-05 18:05:24 -08:00
chrislu	e6c026db65	volume.fix.replication: fix misplaced volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2416	2021-12-05 16:56:25 -08:00
chrislu	53e2dee177	[volume.check.disk] was using the wrong source and target locations ... fix https://github.com/chrislusf/seaweedfs/issues/2268	2021-12-05 14:32:04 -08:00
chrislu	c146c76d10	avoid creating the same bucket with a different randomized name ... related to https://github.com/chrislusf/seaweedfs/issues/2492	2021-12-05 13:06:41 -08:00
chrislu	42d97a3442	adjust randomized bucket name	2021-12-05 12:36:58 -08:00
chrislu	488afa5002	volume: load volume can optionally be skipped, if ec volume exists ... fix https://github.com/chrislusf/seaweedfs/issues/2489	2021-12-05 02:28:52 -08:00
chrislu	59e58c4b23	volume: fix loading old volume format ... fix https://github.com/chrislusf/seaweedfs/issues/2487	2021-12-05 01:06:01 -08:00
chrislu	b70cb3e0b2	upgrade protoc to 3.17.3 ... $brew install protobuf $ protoc --version libprotoc 3.17.3 $ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26 $ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1	2021-12-05 00:42:25 -08:00
chrislu	5c3b783310	s3: copy object API needs to escape special characters ... fix https://github.com/chrislusf/seaweedfs/issues/2482	2021-12-04 23:24:53 -08:00
Bl1tz23	83bffca7e6	fix MongodbStore.ListDirectoryEntries panic on find failure	2021-12-03 13:58:37 +03:00
Tanmoy Majumdar	ea09fb477a	return ' shouldRetry=true' so that filer can retry the failed chunk	2021-12-03 11:54:20 +06:00
Chris Lu	689f5513a9	redis3 supports sentinel	2021-11-29 01:09:51 -08:00
Chris Lu	7227cfddf5	2.80	2021-11-29 00:57:08 -08:00
Chris Lu	3d7390302d	add s3.clean.uploads -timeAgo=24h	2021-11-29 00:49:49 -08:00
Chris Lu	2f72c24498	skip the rest logic	2021-11-29 00:32:21 -08:00
Chris Lu	88ff8fc27b	ensure uploaded chunks are deleted on error	2021-11-29 00:28:26 -08:00
Chris Lu	bea26549f2	avoid empty extended attribute value	2021-11-28 23:50:52 -08:00
Chris Lu	ce2af0082e	revert	2021-11-28 23:35:22 -08:00
Chris Lu	1c9f3c7ac0	read deleted chunks when replcating data	2021-11-28 23:34:34 -08:00
Chris Lu	96c66ca2aa	read deleted chunks when replicating data	2021-11-28 23:33:03 -08:00
Chris Lu	f3a334965d	Update command_remote_uncache.go ... minor	2021-11-28 23:10:32 -08:00
Chris Lu	9ccfc1cfee	Update command_volume_fsck.go ... minor	2021-11-28 22:39:24 -08:00
Chris Lu	71a94267c9	Update command_volume_fsck.go ... remove unused code	2021-11-28 22:37:11 -08:00
Chris Lu	ad16221a35	adjust error log	2021-11-28 22:06:17 -08:00
Chris Lu	cf1586a34d	add logs for writing to remote file	2021-11-27 22:09:23 -08:00
Chris Lu	3a19eea97c	allocate memory by slabs	2021-11-27 12:13:00 -08:00
limd	8805c04128	fix redis2 sentinel config example	2021-11-25 19:20:02 +08:00
limd	ec03f22cc3	Merge remote-tracking branch 'origin/master'	2021-11-25 16:07:14 +08:00
limd	220797bd71	support redis sentinel	2021-11-25 15:57:03 +08:00
Chris Lu	6c27845be0	add retries when writing to remote s3 ... fix https://github.com/chrislusf/seaweedfs/issues/2465	2021-11-22 21:48:04 -08:00
Chris Lu	f3c789d662	2.79	2021-11-21 18:40:24 -08:00
Konstantin Lebedev	40abae7caa	Avoid forbiddening for HeadBucketHandler if owners are specified ... https://github.com/chrislusf/seaweedfs/issues/2434	2021-11-17 14:42:24 +05:00
Chris Lu	3d87aa767d	fix same dc and other dc	2021-11-16 09:14:01 -08:00
Chris Lu	a0ef6e3611	prevent nil response ... fix https://github.com/chrislusf/seaweedfs/issues/2452	2021-11-15 08:46:01 -08:00
Chris Lu	100c654ec3	2.78	2021-11-14 23:29:59 -08:00
Chris Lu	7bf891c00a	randomize same-dc servers and other-dc servers	2021-11-12 11:30:11 -08:00
Chris Lu	1f75f1f9dc	filer: fix mysql2 SQL template	2021-11-11 22:28:28 -08:00
Chris Lu	3abbaccb70	filer: fix mysql command to upsert	2021-11-11 22:27:13 -08:00
user	9668b15f38	Filtering by volume id is prioritized.	2021-11-11 18:18:56 +09:00
user	563a74a9eb	Volume filter by collection pattern added.	2021-11-11 18:01:47 +09:00
user	dbb8003ce3	Volume filter function added.	2021-11-11 17:45:17 +09:00
user	c387fe957b	Unused parameter removed.	2021-11-11 16:08:41 +09:00
Konstantin Lebedev	42f4aa824b	force overwrite s3-is-admin ... https://github.com/chrislusf/seaweedfs/issues/2433	2021-11-11 11:34:01 +05:00
Chris Lu	c77c0de418	make install	2021-11-09 12:34:57 -08:00
Chris Lu	a7eee8b14b	Merge pull request #2427 from Bl1tz23/master ... Fix: potencial iam identities data race	2021-11-09 08:53:48 -08:00
Chris Lu	1d4a61af5e	add brokers	2021-11-09 08:50:55 -08:00
Bl1tz23	c683409e92	s3: add RWMutex to iam, use RLock for concurrent reading	2021-11-09 18:11:06 +03:00
Bl1tz23	5018b22f36	s3: fix potencial iam identities data race	2021-11-09 12:19:50 +03:00
Chris Lu	59d1435d65	s3: avoid possible attacks by version	2021-11-08 17:47:56 -08:00
Chris Lu	5c99b8acc3	rename variables	2021-11-08 17:47:56 -08:00
Chris Lu	00fdab1284	adjust client name	2021-11-08 17:47:56 -08:00
Chris Lu	4729a57cc0	use constants	2021-11-08 17:47:56 -08:00
Chris Lu	d9dd72ea56	rename pacakge	2021-11-08 17:47:56 -08:00
Chris Lu	5cf332357b	2.77	2021-11-07 13:52:45 -08:00
Chris Lu	244841f080	adjust logs	2021-11-07 13:27:57 -08:00
Chris Lu	6e999f1176	s3: fix ListObject if more than 10000 objects	2021-11-07 12:39:36 -08:00
Chris Lu	a8b0f8864d	fix help message	2021-11-07 12:38:35 -08:00
Chris Lu	55973e8572	log unknown access key	2021-11-07 12:37:46 -08:00
Chris Lu	c4e22b5a9a	filer: deprecate "-peers" option	2021-11-06 14:36:45 -07:00
Chris Lu	751a7073e3	Merge branch 'flexible_filer_addition'	2021-11-06 14:26:26 -07:00
Chris Lu	e0fc2898e9	auto updated filer peer list	2021-11-06 14:23:35 -07:00
Chris Lu	04663c3611	remote.mount: print out metadata sync errors	2021-11-06 11:29:50 -07:00
Chris Lu	330d1fde7f	send peers info to filers	2021-11-06 11:29:50 -07:00
Chris Lu	84bb8e7365	send peers info to filers	2021-11-06 04:07:38 -07:00
Chris Lu	4b9c42996a	refactor grpc API	2021-11-05 18:11:40 -07:00
Chris Lu	5ea86ef1da	Revert "master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()" ... This reverts commit af71ae11aa.	2021-11-05 17:52:15 -07:00
Chris Lu	314c32514b	add command to list filers	2021-11-05 17:39:41 -07:00
Chris Lu	7348d2ae63	randomize traffic if replicated	2021-11-05 00:03:13 -07:00
Chris Lu	00ae965d8d	randomize a bit for ec shards distribution	2021-11-04 09:23:40 -07:00
Chris Lu	77f90ae288	add leader election in master	2021-11-04 00:54:38 -07:00
Chris Lu	35c37562bc	check auth only when enabled	2021-11-03 12:11:36 -07:00
Chris Lu	3c245c69d3	handle possible race condition	2021-11-03 01:36:39 -07:00
Chris Lu	af71ae11aa	master: rename grpc function KeepConnected() to SubscribeVolumeLocationUpdates()	2021-11-03 01:09:48 -07:00
Chris Lu	ab97b17e62	better printout	2021-11-02 23:45:47 -07:00
Chris Lu	e66865a8c5	adjust master client log level	2021-11-02 23:45:28 -07:00
Chris Lu	0c8dea9de8	go fmt	2021-11-02 23:39:16 -07:00
Chris Lu	5160eb08f7	shell: optionally read filer address from master	2021-11-02 23:38:45 -07:00
Chris Lu	18bfbf62fc	add one unit test	2021-11-02 23:36:43 -07:00
Chris Lu	e1ab8b01d0	s3: adjust permission for HEAD bucket operation ... fix https://github.com/chrislusf/seaweedfs/issues/2417#issuecomment-958391856	2021-11-02 16:28:55 -07:00
Chris Lu	b25661c6df	s3: skip permission checking for creating bucket if the bucket already exists ... fix https://github.com/chrislusf/seaweedfs/issues/2417 Rclone was trying to create the bucket even though the bucket already exists.	2021-11-02 13:13:36 -07:00
Chris Lu	1b90d60765	filer: paginate large directories to delete file chunks	2021-11-02 01:04:50 -07:00
Chris Lu	794375ca0a	adjust help message since both fullPercent and quietFor are needed.	2021-11-01 17:22:47 -07:00
Chris Lu	fc9e246592	2.76	2021-10-31 18:08:28 -07:00
Chris Lu	9cf756b188	S3: support CORS ... related to https://github.com/chrislusf/seaweedfs/issues/1271	2021-10-31 18:06:43 -07:00
Chris Lu	f17fa400d5	refactoring	2021-10-31 18:05:34 -07:00
Chris Lu	006b4ec0f8	refactoring	2021-10-31 18:02:08 -07:00
Chris Lu	edbf6d297b	filer.meta.tail: add example to send metadata to elastic search	2021-10-31 18:01:33 -07:00
Chris Lu	24858507cc	rename API to avoid confusion	2021-10-30 19:27:25 -07:00
Chris Lu	f5824f369c	avoid reporting error when it is a user request error	2021-10-29 14:14:29 -07:00
Chris Lu	c857cc7286	cloud tier: remove tagging since not all s3 vendors support this	2021-10-29 12:39:19 -07:00
Chris Lu	d04cdcf40d	s3 header add user agent	2021-10-29 12:28:24 -07:00
Konstantin Lebedev	dc3ae5e8a6	DeleteBucketLifecycleConfiguration return 204	2021-10-28 19:34:37 +05:00
Konstantin Lebedev	2afb5a13af	fix DeleteBucketLifecycleConfiguration	2021-10-28 18:30:33 +05:00
Chris Lu	2e76834e4d	filer store: redis2 fix wrong pagination	2021-10-27 23:46:25 -07:00
Chris Lu	54b6e0f3fd	adjust logs	2021-10-27 23:46:07 -07:00
Chris Lu	900e888695	filer: avoid possible prefixed pagination loop on unsupported filer stores	2021-10-27 23:45:48 -07:00
Chris Lu	d774fa6c9a	rename variable	2021-10-25 14:39:20 -07:00
Chris Lu	2539ba0b62	fix compilation	2021-10-25 14:38:48 -07:00
Chris Lu	5f2d7c1589	erasure coding: skip erasure coding if less than recommended 4 nodes	2021-10-25 14:38:11 -07:00
Chris Lu	c9d3fb4a30	2.75	2021-10-24 18:15:59 -07:00
Chris Lu	5435027ff0	volume copy: stream out copying progress and avoid grpc request timeout ... fix https://github.com/chrislusf/seaweedfs/issues/2386	2021-10-24 02:52:56 -07:00
Chris Lu	3be3c17f59	volume vacuum: avoid timeout with streaming progress report ... fix https://github.com/chrislusf/seaweedfs/issues/2396	2021-10-24 01:55:34 -07:00
Konstantin Lebedev	235329a92a	fix Tagging add xmlns	2021-10-20 19:40:22 +05:00
Konstantin Lebedev	dc7e525cb9	fix Tagging test	2021-10-20 19:12:00 +05:00
Konstantin Lebedev	0b834600d5	fix PutObjectTaggingHandler Unmarshal	2021-10-20 17:58:06 +05:00
Konstantin Lebedev	c78220a7f2	fix object tagging https://github.com/chrislusf/seaweedfs/issues/2389	2021-10-20 16:01:06 +05:00
Chris Lu	182f43ae5f	2.74	2021-10-18 14:23:54 -07:00
Chris Lu	309f46e3af	S3: fix upload limit if the size is more than 8GB	2021-10-18 14:14:18 -07:00
Chris Lu	cd4fa7561b	2.73	2021-10-18 10:47:48 -07:00
Chris Lu	a6bb509cb7	adjust mime detection	2021-10-18 10:47:39 -07:00
Konstantin Lebedev	dd2cc1acaf	fix https://github.com/chrislusf/seaweedfs/issues/2387	2021-10-18 16:27:57 +05:00
Chris Lu	97c963bac9	2.72	2021-10-17 17:40:27 -07:00
Chris Lu	3833dac3f7	continue to read from memory if there is no flush	2021-10-17 13:53:04 -07:00
Chris Lu	29fa1b9fdc	add logs	2021-10-17 13:50:34 -07:00
Chris Lu	cca62fdb30	mount: streaming renaming folders	2021-10-17 04:22:42 -07:00
Chris Lu	004e56c1a6	save updated lastTsNs	2021-10-17 02:01:20 -07:00
Chris Lu	71a8c80f6f	s3: continuously listen to file updates ... fix https://github.com/chrislusf/seaweedfs/issues/2342	2021-10-17 01:04:19 -07:00
Chris Lu	93bb7869b8	Revert "mount: fix renaming a deep directory with unvisited directories" ... This reverts commit 0ccdb937bb.	2021-10-16 23:52:30 -07:00
Chris Lu	8e2c9713a3	turn on new faster algorithm to translate into visible chunks	2021-10-16 23:35:20 -07:00
Chris Lu	e7d1f183cc	ensure file id is populated	2021-10-16 23:34:27 -07:00
Chris Lu	fc1694b321	add tests	2021-10-16 23:34:12 -07:00
Chris Lu	0ccdb937bb	mount: fix renaming a deep directory with unvisited directories	2021-10-16 23:33:45 -07:00
Chris Lu	b9a2efd69b	temporarily reverting	2021-10-16 16:29:10 -07:00
Chris Lu	8a124ef9fc	Revert "remove deprecated code" ... This reverts commit de7688c539.	2021-10-16 16:09:33 -07:00
Chris Lu	93624635b4	Revert "go fmt" ... This reverts commit 6c704eb2ba.	2021-10-16 16:09:30 -07:00
Chris Lu	6c704eb2ba	go fmt	2021-10-16 16:06:18 -07:00
Chris Lu	de7688c539	remove deprecated code	2021-10-16 16:06:12 -07:00
Chris Lu	7336990639	faster file read for large files	2021-10-16 16:03:16 -07:00
Chris Lu	8965a53c4d	add warning error	2021-10-16 15:57:30 -07:00
Chris Lu	8e3075d051	ensure file entry chunks have populated the file id	2021-10-16 01:05:48 -07:00
Chris Lu	822ee2475b	remove unnecessary file	2021-10-16 01:05:06 -07:00
Konstantin Lebedev	8de7915950	s3 mime detect	2021-10-14 15:03:11 +05:00
Chris Lu	5fd4b05c5e	Merge pull request #2381 from Juneezee/deprecate-ioutil ... refactor: move from io/ioutil to io and os package	2021-10-13 22:38:58 -07:00
Chris Lu	46a09c6074	adjust test	2021-10-13 22:38:47 -07:00
Eng Zer Jun	a23bcbb7ec	refactor: move from io/ioutil to io and os package ... The io/ioutil package has been deprecated as of Go 1.16, see https://golang.org/doc/go1.16#ioutil. This commit replaces the existing io/ioutil functions with their new definitions in io and os packages. Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2021-10-14 12:27:58 +08:00
Chris Lu	4cbd390fbe	test: add fail message	2021-10-13 20:42:20 -07:00
Chris Lu	b693a8d6b9	Merge pull request #2380 from kmlebedev/bucket_policy ... Bucket policy	2021-10-13 15:29:53 -07:00
Konstantin Lebedev	9d6ffa0ea1	GetBucketLifecycleConfigurationHandler	2021-10-14 01:35:33 +05:00
Konstantin Lebedev	458145425e	S3 MultipartUpload pass contentType to meta	2021-10-12 17:14:54 +05:00
Chris Lu	de1ec050e8	fier/s3: save "Content-Disposition" to extended properties ... related to https://github.com/chrislusf/seaweedfs/issues/2371	2021-10-12 00:04:28 -07:00
Chris Lu	34f764007f	escape file name only when necessary	2021-10-11 23:33:45 -07:00
Chris Lu	8e9273db99	s3: use "response-content-disposition" to overwrite default content-disposition ... fix one part of https://github.com/chrislusf/seaweedfs/issues/2371 see https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html	2021-10-11 23:29:52 -07:00
Chris Lu	f0b928ff5e	go fmt	2021-10-11 23:23:46 -07:00
Chris Lu	3afa451cdc	volume: find a non-empty offset when binary searching by timestamp	2021-10-11 22:00:41 -07:00
Chris Lu	b530f12327	volume: find a non-empty offset when binary searching by timestamp ... fix https://github.com/chrislusf/seaweedfs/issues/2364	2021-10-11 16:11:50 -07:00
Konstantin Lebedev	be4b3ed509	AclHandlers	2021-10-11 15:03:56 +05:00
Chris Lu	e6ef7b2387	return error early ... fix https://github.com/chrislusf/seaweedfs/issues/2370	2021-10-11 01:24:30 -07:00
Chris Lu	3d586be552	2.71	2021-10-10 22:40:44 -07:00
Chris Lu	f4676824a7	fix test code compilation	2021-10-10 22:14:13 -07:00
Chris Lu	8205166009	disable testing with redis server ... --- FAIL: TestNameList (0.00s) panic: exec: "redis-server": executable file not found in $PATH [recovered] panic: exec: "redis-server": executable file not found in $PATH goroutine 37 [running]: testing.tRunner.func1.2({0xde2f80, 0xc0003da160}) /opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1209 +0x24e testing.tRunner.func1() /opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1212 +0x218 panic({0xde2f80, 0xc0003da160}) /opt/hostedtoolcache/go/1.17.1/x64/src/runtime/panic.go:1038 +0x215 github.com/chrislusf/seaweedfs/weed/filer/redis3.TestNameList(0x407c59) /home/runner/work/seaweedfs/seaweedfs/weed/filer/redis3/kv_directory_children_test.go:45 +0x376 testing.tRunner(0xc0003e2680, 0xf57f48) /opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1259 +0x102 created by testing.(*T).Run /opt/hostedtoolcache/go/1.17.1/x64/src/testing/testing.go:1306 +0x35a	2021-10-10 21:57:55 -07:00
Chris Lu	3e2acf677c	removing tikv to resolve "go mod tidy" problem ... tikv is causing "go mod tidy" problem. Need to resolve this before adding tikv back. go mod tidy go: finding module for package github.com/coreos/etcd/clientv3/balancer/picker go: finding module for package cloud.google.com/go/kms/apiv1 go: finding module for package github.com/coreos/etcd/clientv3/balancer/resolver/endpoint go: finding module for package google.golang.org/grpc/naming go: finding module for package github.com/coreos/etcd/clientv3/credentials go: finding module for package github.com/coreos/etcd/clientv3/balancer go: finding module for package github.com/d4l3k/messagediff go: found github.com/coreos/etcd/clientv3/balancer in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/balancer/picker in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/balancer/resolver/endpoint in github.com/coreos/etcd v3.3.26+incompatible go: found github.com/coreos/etcd/clientv3/credentials in github.com/coreos/etcd v3.3.26+incompatible go: found cloud.google.com/go/kms/apiv1 in cloud.google.com/go/kms v1.0.0 go: found github.com/d4l3k/messagediff in github.com/d4l3k/messagediff v1.2.1 go: finding module for package google.golang.org/grpc/naming github.com/chrislusf/seaweedfs/weed/filer/tikv imports github.com/tikv/client-go/v2/tikv imports go.etcd.io/etcd/clientv3 tested by go.etcd.io/etcd/clientv3.test imports github.com/coreos/etcd/integration imports github.com/coreos/etcd/proxy/grpcproxy imports google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming	2021-10-10 19:27:02 -07:00
Chris Lu	bf218cd59d	removing etcd sequencer ... causing go mod tidy problem. If anyone wants this, please help to resolve this first. github.com/chrislusf/seaweedfs/weed/sequence imports go.etcd.io/etcd/client tested by go.etcd.io/etcd/client.test imports github.com/coreos/etcd/integration imports github.com/coreos/etcd/proxy/grpcproxy imports google.golang.org/grpc/naming: module google.golang.org/grpc@latest found (v1.41.0), but does not contain package google.golang.org/grpc/naming	2021-10-10 19:10:46 -07:00
Chris Lu	17e08a16f4	fix redis3 deletion	2021-10-09 04:54:48 -07:00
Chris Lu	ce1efeb0eb	adds more error message	2021-10-09 04:54:14 -07:00
Chris Lu	e4830bd93d	go fmt	2021-10-07 21:13:31 -07:00
Chris Lu	0a856241fe	avoid int bigger than math.MaxInt32 ... fix https://github.com/chrislusf/seaweedfs/issues/2363	2021-10-07 21:12:57 -07:00
Chris Lu	d688e10ed1	do not try to compress if not sure about the file content	2021-10-07 13:29:00 -07:00
Chris Lu	332f5ad3a8	revert temporary changes that disabled compression ... fix https://github.com/chrislusf/seaweedfs/issues/2362	2021-10-07 13:24:16 -07:00
Chris Lu	f3d8232e14	reduce one redis lookup on hot path	2021-10-06 22:01:19 -07:00
Chris Lu	2336a397dc	use pipeline to save some time	2021-10-06 20:40:18 -07:00
Chris Lu	371fead8a5	redis3 using redis native sorted set	2021-10-06 18:18:24 -07:00
Chris Lu	8668d49c9d	test with real redis ... cpu: Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz BenchmarkNameList-12 1789 760599 ns/op BenchmarkRedis-12 17539 64122 ns/op PASS	2021-10-06 01:25:37 -07:00
Chris Lu	6b31f3c97a	add benchmark test	2021-10-06 00:37:57 -07:00
Chris Lu	893f0587b1	redis3 adds distributed locking	2021-10-06 00:03:54 -07:00
Chris Lu	f0d1e7bd05	skip ec volumes when loading normal volumes	2021-10-05 02:31:44 -07:00
Chris Lu	332d49432d	reduce concurrent volume grow requests	2021-10-05 01:58:30 -07:00
Chris Lu	96119eab00	refactor	2021-10-05 00:40:04 -07:00
Chris Lu	8a66306064	calculate disk usage in case of race condition ... related to https://github.com/chrislusf/seaweedfs/issues/2357	2021-10-04 23:32:07 -07:00
Chris Lu	4ed2994555	use tsMemory to determine whether read from disk or memory ... remove lastFlushTime	2021-10-04 16:02:56 -07:00
Chris Lu	6a030547a2	server: remove peer check if not starting master ... more fix https://github.com/chrislusf/seaweedfs/issues/2352	2021-10-04 03:27:10 -07:00
Chris Lu	62c2732fd1	add back logic to check master peers ... fix https://github.com/chrislusf/seaweedfs/issues/2352	2021-10-04 02:51:26 -07:00
Chris Lu	d4bb16e20e	Merge pull request #2354 from chrislusf/bptree ... Add Redis3	2021-10-04 02:43:15 -07:00
Chris Lu	947add39e6	clean up *SkipListElementReference loaded from Redis	2021-10-04 02:31:38 -07:00
Chris Lu	513fed323a	SkipListElementReference can be an empty object	2021-10-04 02:30:44 -07:00
Chris Lu	280ab7f95c	add test	2021-10-04 02:30:24 -07:00
Chris Lu	04662126bb	add redis3	2021-10-04 01:04:27 -07:00
Chris Lu	2b9aab3442	use 1000 per batch	2021-10-04 01:03:40 -07:00
Chris Lu	366f522a2d	add redis3	2021-10-04 01:01:31 -07:00
Chris Lu	ba7fbac07f	rename	2021-10-03 19:23:34 -07:00
Chris Lu	e6196cdc50	add name list	2021-10-03 17:54:25 -07:00
Chris Lu	a481c4a45e	return previous element if visited	2021-10-03 13:50:52 -07:00
Chris Lu	22d8684e88	refactor out listStore	2021-10-03 02:19:21 -07:00
Chris Lu	d343b0db57	update value	2021-10-03 01:15:14 -07:00
Chris Lu	4f50f8c2ca	insert key and value	2021-10-03 01:07:35 -07:00
Chris Lu	69b84bb771	TestFindGreaterOrEqual	2021-10-02 14:15:49 -07:00
Chris Lu	57e2fd3f9b	remove bptree	2021-10-02 14:03:54 -07:00
Chris Lu	4c1741fdbb	working skiplist	2021-10-02 14:02:56 -07:00
Chris Lu	b297849147	typo	2021-10-01 23:24:54 -07:00
Chris Lu	af207bbaf0	retry both assign volume and uploading data ... fix https://github.com/chrislusf/seaweedfs/issues/2351	2021-10-01 23:23:39 -07:00
Chris Lu	b6694279d7	Merge branch 'master' into bptree	2021-10-01 16:55:44 -07:00
Chris Lu	1e3fdf366f	go fmt	2021-10-01 12:10:24 -07:00
Chris Lu	e862b2529a	refactor	2021-10-01 12:10:11 -07:00
Chris Lu	88e006bc6d	Merge pull request #2349 from kmlebedev/fix_repl_volumes_per_step ... Topology update for every Nth id volume	2021-10-01 12:05:51 -07:00
Konstantin Lebedev	5e64b22b45	check that the topology has been updated	2021-10-01 18:51:22 +05:00
Chris Lu	f58ea6a2ee	add source name to error message	2021-10-01 02:19:30 -07:00
chrislusf	3ffbaaa071	use github.com/linxGnu/grocksdb	2021-10-01 03:16:03 +00:00
Konstantin Lebedev	2cecde89c3	rename opt volumesPerStep	2021-10-01 00:17:54 +05:00
Konstantin Lebedev	fc51ffce2b	https://github.com/chrislusf/seaweedfs/issues/1846	2021-09-30 20:24:24 +05:00
Chris Lu	a067deaabc	avoid possible modified location list ... fix issue 1 of https://github.com/chrislusf/seaweedfs/issues/2345	2021-09-28 16:54:18 -07:00
Chris Lu	040443e2d1	fix possible error case	2021-09-27 23:59:45 -07:00
Chris Lu	2e9372dcf7	volume stream read skips deleted content	2021-09-27 03:07:44 -07:00
Chris Lu	225b019fe0	stream read multiple volumes in a volume server	2021-09-27 02:51:31 -07:00
Chris Lu	1904448d4e	adjust starting offset	2021-09-27 02:01:29 -07:00
Chris Lu	5956a8b05a	adjust comment	2021-09-27 01:58:03 -07:00
Chris Lu	aa64f2ac4c	send needle data instead of raw needle body	2021-09-27 01:53:41 -07:00
Chris Lu	c4d7ee6c5c	volume server: read all files in a volume	2021-09-27 01:45:32 -07:00
Chris Lu	4a1d4d7462	s3: default to allow empty folder for better performance	2021-09-26 22:34:14 -07:00
Chris Lu	cee4d20bc1	2.70	2021-09-26 17:37:46 -07:00
Chris Lu	603ea2db73	avoid looping forever if there are no more metadata updates	2021-09-26 11:55:27 -07:00
Chris Lu	9887610b54	log tsNs should be processing time	2021-09-26 11:54:13 -07:00
Chris Lu	2baed2e1e9	avoid possible metadata subscription data loss ... Previous implementation append filer logs into one file. So one file is not always sorted, which can lead to miss reading some entries, especially when different filers have different write throughput.	2021-09-25 01:18:44 -07:00
Chris Lu	a814f3f0a8	adjust metadata tail output	2021-09-25 01:04:51 -07:00
Chris Lu	52fe86df45	use default 10000 for grpc port	2021-09-20 14:05:59 -07:00
Chris Lu	b3d88180ca	Merge branch 'master' into bptree	2021-09-19 23:56:59 -07:00
Chris Lu	fa7c65bd4b	2.69	2021-09-19 21:44:06 -07:00
Chris Lu	84fdda85e7	go fmt	2021-09-19 12:06:15 -07:00
Chris Lu	ad5099e570	refactor	2021-09-19 12:02:23 -07:00
Chris Lu	5abdc0be77	s3: avoid overwriting object with ACL/LegalHold/Retension/LockConfiguration requests	2021-09-19 03:24:47 -07:00
Chris Lu	ede7a65a50	update log	2021-09-19 01:33:50 -07:00
Chris Lu	c1255d30ad	add cache error log	2021-09-19 01:33:26 -07:00
Chris Lu	59dd271734	more glog	2021-09-19 00:29:51 -07:00
Chris Lu	4cbba2b1c6	add more glog to s3	2021-09-19 00:28:22 -07:00
Chris Lu	71175461ef	add glog for s3 handlers	2021-09-19 00:18:59 -07:00
Chris Lu	e066e2642c	add NodeStore	2021-09-18 15:32:17 -07:00
Chris Lu	198fa58e3c	Merge branch 'master' into bptree	2021-09-18 14:06:16 -07:00
Chris Lu	49d971e602	filer: redis store needs to clean up batch deleted sub folders	2021-09-18 14:05:16 -07:00
Chris Lu	8f2e4be074	wip	2021-09-18 14:04:30 -07:00
Chris Lu	b751debd31	split node based on the last inserted key	2021-09-18 01:29:47 -07:00
Chris Lu	2226c3c8b6	Merge branch 'master' into bptree	2021-09-17 10:35:21 -07:00
Chris Lu	94a01fcfcb	filer.remote.gateway: add options to include or exclude new bucket names to mirror	2021-09-15 23:04:16 -07:00
Chris Lu	b5f4910412	adjust help messages	2021-09-15 22:53:10 -07:00
Chris Lu	f0907eb83c	add command filre.remote.gateway	2021-09-15 22:48:04 -07:00
Chris Lu	63da4bbb54	separate filer.remote.gateway command to avoid confusion	2021-09-15 22:47:17 -07:00
Chris Lu	2789d10342	go fmt	2021-09-14 10:37:06 -07:00
Chris Lu	119d5908dd	shell: do not need to lock to see volume -h	2021-09-13 22:13:34 -07:00
Chris Lu	7504be58f9	Avoid xattr printed out as HTTP headers ... fix https://github.com/chrislusf/seaweedfs/issues/2336	2021-09-13 04:00:57 -07:00
Chris Lu	96514f0f00	fix tests	2021-09-13 02:19:48 -07:00
Chris Lu	20ac710ceb	2.68	2021-09-13 02:16:09 -07:00
Chris Lu	6cd1ce8b74	erasure coding: add cleanup step if anything goes wrong	2021-09-13 01:55:49 -07:00
Chris Lu	f74b29416a	better etag matching	2021-09-13 00:31:46 -07:00
Chris Lu	e9760f261e	minor	2021-09-13 00:31:06 -07:00
Chris Lu	e5fc35ed0c	change server address from string to a type	2021-09-12 22:47:52 -07:00
Chris Lu	2c9d4c8f43	custom grpc port: filer	2021-09-12 02:28:37 -07:00
Chris Lu	e690a2be16	custom grpc port: volume server	2021-09-12 02:25:15 -07:00
Chris Lu	232ad2fe65	custom grpc port: master follower	2021-09-12 02:25:00 -07:00
Chris Lu	bd1f800efe	custom grpc port: master	2021-09-12 02:19:10 -07:00
Chris Lu	7591336a22	log format	2021-09-11 14:27:57 -07:00
Chris Lu	8c6ff55226	add volume not found error type, to reduce error log	2021-09-11 14:26:41 -07:00
Chris Lu	7e600bff5f	adjust log level	2021-09-11 12:59:15 -07:00
Chris Lu	a243d7e047	EC: clean up if failed in the middle	2021-09-11 02:07:14 -07:00
Chris Lu	f7f01ac209	return if error	2021-09-11 02:06:43 -07:00
Chris Lu	59cb6146a3	refactor	2021-09-11 02:06:30 -07:00
Chris Lu	5496d68f6a	increase counter only if not early terminated	2021-09-11 02:05:55 -07:00
Chris Lu	2b8ab1534a	mysql table DDL: minor, not necessary, but a little more strict ... the directory is already hashed into dirhash. So this change is not strictly required.	2021-09-08 23:55:18 -07:00
joshuafc	5654d0d60d	CompactionTableSizeMultiplier of leveldb use default value. #2325 ... To improve performance of leveldb find key in condition of large directory(millions of files) which use uuid as filename.	2021-09-09 10:42:34 +08:00
Chris Lu	0b4269b6a8	remember commands even if failed	2021-09-08 15:55:19 -07:00
Chris Lu	0207f5fe9b	replicated remote.cache	2021-09-08 15:54:55 -07:00
Chris Lu	00edcf71fc	fix compilation	2021-09-08 07:27:03 -07:00
Chris Lu	574485ec69	better IP v6 support	2021-09-07 19:29:42 -07:00
Chris Lu	0128239c0f	handle ipv6 addresses	2021-09-07 16:43:54 -07:00
zhoub	6a7ed1bd0e	add bloom filter to leveldb_store to improve fuse performance.	2021-09-07 21:09:10 +08:00
Chris Lu	9fdf02bcda	remove detecting ipv6 ... Got this error on my local: transport: Error while dialing dial tcp [fe80::1]:19333: connect: no route to host related to https://github.com/chrislusf/seaweedfs/pull/2310	2021-09-07 02:31:34 -07:00
Chris Lu	889a39a24f	ipv6	2021-09-07 02:30:25 -07:00
Chris Lu	8c6d706328	2.67	2021-09-07 00:08:03 -07:00
Chris Lu	2b1feb732c	remote.cache supports replication	2021-09-06 18:30:44 -07:00
Chris Lu	3adc3da291	refactor	2021-09-06 16:35:55 -07:00
Chris Lu	6923af7280	refactoring	2021-09-06 16:20:49 -07:00
Chris Lu	64f6532fbe	update help message	2021-09-06 16:16:22 -07:00
Chris Lu	308d4b2dae	filer.remote.sync: createBucketWithRandomSuffix defaults to true	2021-09-06 15:14:26 -07:00
Chris Lu	8e4b43a017	minor	2021-09-06 15:13:38 -07:00
Chris Lu	c218ef20c7	filer.remote.sync: automatically detect the primary remote storage	2021-09-06 15:10:55 -07:00
Chris Lu	1702ce5395	remove unused code	2021-09-06 12:01:44 -07:00
Chris Lu	0cda61d539	return error if not found	2021-09-05 23:43:44 -07:00
Chris Lu	f62df7f695	fix wrong srcPath checking	2021-09-05 23:37:40 -07:00
Chris Lu	e93d4935e3	add other replica locations when assigning volumes	2021-09-05 23:32:25 -07:00
Chris Lu	7a13816e94	refactor	2021-09-05 23:17:15 -07:00
Chris Lu	6022db6d6a	2.66	2021-09-05 16:21:14 -07:00
Chris Lu	006c01a519	fix format	2021-09-05 16:18:50 -07:00
Chris Lu	65af3cf4df	master: disconnect only the phantom volume server ... fix https://github.com/chrislusf/seaweedfs/issues/2311	2021-09-05 15:20:03 -07:00
Chris Lu	57a95887d2	remote.cache remote.uncache supports all mounted directories	2021-09-05 14:47:06 -07:00
Chris Lu	28f45f8fa6	re-org	2021-09-05 14:38:31 -07:00
Chris Lu	c735608685	obfuscate secret keys on display	2021-09-05 14:23:49 -07:00
Chris Lu	32e94de86a	default auto trimming suffix to true	2021-09-05 13:27:35 -07:00
Chris Lu	60573fd3e2	option to map remote bucket to trimmed bucket name	2021-09-05 11:55:52 -07:00
Chris Lu	2348e8d8da	Merge pull request #2310 from nivekuil/ipv6 ... Detect ipv6 addresses	2021-09-05 10:56:44 -07:00
nivekuil	0fe9d2997b	Detect ipv6 addresses	2021-09-05 06:21:40 -07:00
Chris Lu	bdefdee4e6	filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts	2021-09-04 22:46:28 -07:00
Chris Lu	d57d4c5f8f	shell: add remote.mount.buckets	2021-09-04 21:37:25 -07:00
Chris Lu	98c68d7a79	filer.remote.sync fix upload logic	2021-09-04 18:46:28 -07:00
Chris Lu	d983aa4c7d	correct filtering	2021-09-04 13:58:14 -07:00
Chris Lu	c5ee03d6af	format	2021-09-04 13:57:55 -07:00
Chris Lu	cc278a5928	Update Makefile	2021-09-04 13:46:49 -07:00
Chris Lu	16a342eb8a	re-ordering	2021-09-04 13:46:44 -07:00
Chris Lu	796b2eb929	remember sync time	2021-09-04 13:46:22 -07:00
Chris Lu	53b9b521c9	adjust error message	2021-09-04 13:46:06 -07:00
Chris Lu	49952ba905	fix cache/uncache filters	2021-09-04 13:45:39 -07:00
Chris Lu	99d4594ee8	update remote storage mount mapping	2021-09-04 05:22:34 -07:00
Chris Lu	df29281536	rename	2021-09-04 05:11:06 -07:00
Chris Lu	c3db389e42	refactor	2021-09-04 05:09:38 -07:00
Chris Lu	e9ce099453	rename	2021-09-04 05:03:51 -07:00
Chris Lu	8707ef00a6	refactor	2021-09-04 05:02:20 -07:00
Chris Lu	13cc2ef723	when creating bucket, use remote storage client for default storage	2021-09-04 04:54:55 -07:00
Chris Lu	63c84584bb	refactor	2021-09-04 04:53:36 -07:00
Chris Lu	4b28c5f6c3	filer.remote.sync: split into buckets mode and single directory mode	2021-09-04 04:35:46 -07:00
Chris Lu	da49d25950	auto list of storage types	2021-09-04 00:18:21 -07:00
Chris Lu	38f73c8324	add gcs project id	2021-09-03 23:38:33 -07:00
Chris Lu	8ec357b3d3	go mod	2021-09-03 23:25:33 -07:00
Chris Lu	bbc77f7af4	fix compilation	2021-09-03 22:56:59 -07:00
Chris Lu	0652805236	cloud drive: add createBucket() deleteBucket()	2021-09-03 22:30:55 -07:00
Chris Lu	03a31587ce	go fmt	2021-09-03 20:42:28 -07:00
Chris Lu	83cd0fc739	cloud drive: add list buckets	2021-09-03 20:42:02 -07:00
Chris Lu	fbfc90fd1e	adjust formatting remote location	2021-09-03 18:52:37 -07:00
Chris Lu	491f7636f8	s3: CopyObject return http Status 400 Bad Request for non-existing source ... fix https://github.com/chrislusf/seaweedfs/issues/2306	2021-09-03 17:38:39 -07:00
Chris Lu	bca4a9de78	simplify	2021-09-02 23:09:24 -07:00
Chris Lu	958125bd02	conforming to http user agent common practice	2021-09-02 22:55:35 -07:00
Chris Lu	11a496404b	reset wait time	2021-09-02 19:55:01 -07:00
Chris Lu	7f2a8246a9	minor	2021-09-02 12:32:59 -07:00
Chris Lu	1b942dfa3c	add error message	2021-09-02 12:32:46 -07:00
Chris Lu	9a73b0e3c9	refactor	2021-09-02 07:07:16 -07:00
Chris Lu	e281f0fa82	refactor	2021-09-02 06:53:21 -07:00
Chris Lu	24fda6f47f	remove QR code for mobile upload	2021-09-01 16:59:37 -07:00
Chris Lu	7ce97b59d8	go fmt	2021-09-01 02:45:42 -07:00
Chris Lu	d1a4e19a3f	volume: copy file also copies modification time ... to ensure ttl can work well	2021-09-01 02:42:57 -07:00
Chris Lu	3bd48c4f29	filer.remote.sync: exit when directory is unmounted ... this will not propagate the deletions back to the cloud	2021-09-01 01:29:22 -07:00
Chris Lu	3faaa6e360	ensure cached client with updated storage conf	2021-09-01 01:27:45 -07:00
Chris Lu	8e125339d5	line wrap	2021-08-31 23:30:28 -07:00
Chris Lu	68ea99190f	add additionalPathPrefixes	2021-08-31 23:26:26 -07:00
Chris Lu	43fd11278e	support follow additional path prefixes	2021-08-31 23:23:08 -07:00
Chris Lu	cca588429b	shell: fs.meta.save skip saving system logs	2021-08-31 23:05:56 -07:00
Chris Lu	0ee208c1a4	adjust example	2021-08-31 02:04:32 -07:00
Chris Lu	97b7948cd9	adjust error message	2021-08-31 01:07:34 -07:00
Chris Lu	c9205458a1	cloud drive: create mount directory if not exists	2021-08-31 01:07:19 -07:00
Chris Lu	1dfcdc53b7	cloud drive: add storj, filebase	2021-08-30 17:43:14 -07:00
Chris Lu	701cc133f3	cloud drive: s3 add option for v4 signature	2021-08-30 17:28:33 -07:00
Chris Lu	f117f7bf54	adjust comment	2021-08-30 15:56:14 -07:00
Chris Lu	72d4ff6b28	Merge pull request #2280 from blacktear23/filerstore-tikv ... Use TiKV as Filerstore	2021-08-30 02:21:14 -07:00
Chris Lu	f811fd0903	fix build	2021-08-30 02:03:08 -07:00
yulai.li	b17b81529e	Add build tags for TiKV filerstore	2021-08-30 15:59:25 +08:00
Rain Li	e5f9ff983d	Merge branch 'master' into filerstore-tikv	2021-08-30 15:52:54 +08:00
Chris Lu	44d0ac95f6	add cluster id if already set	2021-08-29 22:19:46 -07:00
Chris Lu	18bf556d93	fix comments	2021-08-29 22:19:25 -07:00
Chris Lu	21f37a0a75	remove unused code	2021-08-29 22:19:11 -07:00
Chris Lu	5adfdd7982	change to a new bloom filter library	2021-08-29 21:26:38 -07:00
Chris Lu	05f32376eb	add cluster id in filer configuration response	2021-08-29 21:02:10 -07:00
Chris Lu	eacaa44dc2	refactor	2021-08-29 19:13:48 -07:00
Chris Lu	6deee4c0b9	import hdfs	2021-08-29 18:49:43 -07:00
Chris Lu	a31f2907f0	cloud drive: filer.remote.sync supports remove folder	2021-08-29 18:46:28 -07:00
Chris Lu	001a472057	cloud mount: remote storage support hdfs	2021-08-29 18:41:29 -07:00
Chris Lu	05d2774117	refactoring	2021-08-28 22:49:38 -07:00
Chris Lu	975fbc9ee3	add wasabi example	2021-08-28 22:49:38 -07:00
Chris Lu	bec3f63298	2.65	2021-08-28 05:27:33 -07:00
Chris Lu	223c1b3571	adjust invalidation logic	2021-08-28 05:21:01 -07:00
nivekuil	8c523d50ad	Invalidate mount meta cache more aggressively	2021-08-28 04:26:37 -07:00
Chris Lu	65a29ad42a	add warning on unmount a folder	2021-08-28 02:25:48 -07:00
Chris Lu	a22f37b01c	remove tencent region	2021-08-28 02:23:03 -07:00
Chris Lu	a9b41900d8	handle cases when mounted directory does not exist	2021-08-28 01:56:56 -07:00
Chris Lu	0d8936f6cd	add aliyun example	2021-08-28 01:09:48 -07:00
Chris Lu	dc481c081c	remove unused function	2021-08-27 22:18:54 -07:00
Chris Lu	9242f3aaeb	fix build	2021-08-26 17:33:57 -07:00
Chris Lu	49a8dfb976	adjust default concurrent level	2021-08-26 17:05:56 -07:00
Chris Lu	6a0bb7106b	cloud drive: parallelize remote storage downloading	2021-08-26 16:16:26 -07:00
Chris Lu	05a648bb96	refactor: separating out remote.proto	2021-08-26 15:18:34 -07:00
yulai.li	c1dc5ab4ac	Add deleterange_concurrency to filer configuration file	2021-08-26 18:25:08 +08:00
yulai.li	318757ef8c	Change DeleteFolderChildren to DeleteRange api	2021-08-26 17:49:56 +08:00
yulai.li	de8ef28460	Update filer.toml scaffold to support tikv store	2021-08-26 16:26:27 +08:00
yulai.li	546efeba8f	Fix build bug	2021-08-26 16:20:35 +08:00
yulai.li	2088f28424	init post	2021-08-26 15:20:18 +08:00
Chris Lu	c08ac536ed	cloud drive: add support for Wasabi ... * disable md5, sha256 checking to avoid reading one chunk twice * single threaded upload to avoid chunk swapping (to be enhanced later)	2021-08-25 17:34:29 -07:00
Chris Lu	9bcf94b2b1	ensure multi-threaded correctness	2021-08-25 17:28:50 -07:00
Chris Lu	a7a914f120	cloud drive: add support for Baidu BOS	2021-08-24 23:46:33 -07:00
Chris Lu	2d805beed4	add help mesages	2021-08-24 23:28:35 -07:00
Chris Lu	c6c97bd83d	add default env variables	2021-08-24 23:25:36 -07:00
Chris Lu	a19c728034	cloud drive: add support for Tencent COS	2021-08-24 23:19:45 -07:00
Chris Lu	19a81d25af	cloud drive: add support for Aliyun OSS	2021-08-24 23:14:24 -07:00
Chris Lu	47d775cf68	cloud drive: add support for BackBlaze	2021-08-24 22:30:06 -07:00
Chris Lu	6bab20d862	cat entry content	2021-08-24 02:38:32 -07:00
Chris Lu	e9ebe24f2e	cloud drive: add support for Azure	2021-08-24 01:18:30 -07:00
Chris Lu	7c39a18ba5	update azure library	2021-08-24 00:32:35 -07:00
Chris Lu	a78d0227cd	adjust package name	2021-08-23 23:19:31 -07:00
Chris Lu	12631a3f5b	cloud drive: gcs simplify a little bit	2021-08-23 14:43:01 -07:00
Chris Lu	00c4e06caa	cloud drive: s3 configurable force path style	2021-08-23 03:30:41 -07:00
Chris Lu	f0cc130849	do not force path style for better compatibility	2021-08-23 03:09:41 -07:00
Chris Lu	2836a58d87	cloud drive: S3 supports storage class	2021-08-23 02:18:59 -07:00
Chris Lu	95e2b83ca5	fix format	2021-08-23 00:49:59 -07:00
Chris Lu	ff7dc3b44c	2.64	2021-08-23 00:39:15 -07:00
Chris Lu	ca35a77e85	adjust help message	2021-08-23 00:37:55 -07:00
Chris Lu	258063de26	cloud drive: add google cloud storage	2021-08-23 00:29:27 -07:00
Chris Lu	df1d6133a8	bptree does not work well for auto-increasing keys	2021-08-22 18:19:26 -07:00
Chris Lu	3b2c39f1be	mount: set name when mount path equals mount folder ... fix https://github.com/chrislusf/seaweedfs/issues/2275#issuecomment-903255876	2021-08-22 04:59:09 -07:00
Chris Lu	51c8f2518f	change key type to ItemKey	2021-08-21 15:54:42 -07:00
Chris Lu	b3e49d2758	change value type to ItemValue	2021-08-21 15:52:17 -07:00
Chris Lu	38c8470d1d	add back non_dedup	2021-08-21 15:13:13 -07:00
Chris Lu	849f185a20	add memory kv store	2021-08-21 15:00:44 -07:00
Chris Lu	5f6cc9a814	make proto node	2021-08-21 13:36:52 -07:00
Chris Lu	2158d4fe4d	adjust help message	2021-08-21 02:17:10 -07:00
Chris Lu	172da83449	bpnode use get prev and next	2021-08-20 18:50:16 -07:00
Chris Lu	01661ec6a7	move to getter setter file	2021-08-20 18:37:34 -07:00
Chris Lu	0c360eb6b2	add getter and setter for root of tree and map	2021-08-20 18:34:50 -07:00
Chris Lu	1b1ab331f6	Merge pull request #2274 from qieqieplus/gzip-pool	2021-08-20 04:22:41 -07:00
Chris Lu	88d68cad87	remove dedup	2021-08-20 04:14:52 -07:00
qieqieplus	7720533f84	reduce gzip allocation	2021-08-20 18:38:18 +08:00
Chris Lu	2d237da74a	remove size since each put/get will have to update the root node	2021-08-20 01:19:11 -07:00
Chris Lu	ec72547c8d	started by copying from https://sourcegraph.com/github.com/timtadh/data-structures@master/-/tree/tree/bptree	2021-08-20 01:12:52 -07:00
Chris Lu	05fc7db755	filer.copy: large files pack file ids into chunks	2021-08-18 23:46:54 -07:00
mauroparente	c63713774b	Fix image resize ... https://github.com/chrislusf/seaweedfs/issues/2262	2021-08-18 23:15:35 +02:00
Chris Lu	570d2eccec	skip body if not allowed by http status	2021-08-18 00:56:35 -07:00
Chris Lu	ccb3df41f8	elide secret key	2021-08-17 11:27:08 -07:00
Konstantin Lebedev	865b06b7fe	The IEEE assembler optimizations has been submitted and will be part of the Go 1.6 standard library. ... https://github.com/klauspost/crc32	2021-08-17 13:06:48 +05:00
Chris Lu	e2aa3cf63b	fix go test	2021-08-15 23:20:46 -07:00
Chris Lu	1f35d32be0	2.63	2021-08-15 23:14:59 -07:00
Chris Lu	40dc283b2d	fix locating data chunks	2021-08-15 23:07:58 -07:00
Chris Lu	72eb6d5b9d	ensure no writes to remote storage if content is not changed	2021-08-15 20:23:41 -07:00
Chris Lu	5a7c40510f	format output	2021-08-15 20:07:13 -07:00
Chris Lu	bb94930196	add some delays if error	2021-08-15 20:06:47 -07:00
Chris Lu	5d5a21ba2d	adjust log format	2021-08-15 19:46:45 -07:00
Chris Lu	c45c12aedc	metadata updates	2021-08-15 19:27:30 -07:00
Chris Lu	507411bdf0	fix https://github.com/chrislusf/seaweedfs/issues/2260 ... fix https://github.com/chrislusf/seaweedfs/issues/2260	2021-08-15 19:16:00 -07:00
Chris Lu	c3ffd457ef	fix compilation error	2021-08-15 12:40:22 -07:00
Chris Lu	49b5e47bd1	retry forever with filer.remote.sync, and some refactoring	2021-08-15 12:38:26 -07:00
Chris Lu	fda2fc47b1	add RetryForever	2021-08-15 12:37:35 -07:00
Chris Lu	a539d64896	refactor	2021-08-15 12:09:54 -07:00
Chris Lu	8f7d2d317f	readerAt need to use the right offset ... fix https://github.com/chrislusf/seaweedfs/issues/2259	2021-08-15 11:55:58 -07:00
Chris Lu	0b2a92d371	skip if http.StatusNoContent	2021-08-15 11:27:49 -07:00
Chris Lu	ec989b0377	remove println	2021-08-15 02:10:27 -07:00
Chris Lu	9462f5129a	shell: add "remote.meta.sync"	2021-08-15 01:53:46 -07:00
Chris Lu	3ada61c875	rename	2021-08-14 21:50:35 -07:00
Chris Lu	c34747c79d	rename, fix wrong logic.	2021-08-14 21:46:34 -07:00
Chris Lu	cb53802752	adjust help message	2021-08-14 15:55:53 -07:00
Chris Lu	889b143fa7	adjust modification detection logic	2021-08-14 15:44:47 -07:00
Chris Lu	f365af81c2	parallelize remote content fetching	2021-08-14 15:41:37 -07:00
Chris Lu	53e66980b2	add comments	2021-08-14 15:16:10 -07:00
Chris Lu	9921801e0c	Revert "use default or path-specific setting for cache replication level" ... This reverts commit ba6923b223.	2021-08-14 15:14:26 -07:00
Chris Lu	ba6923b223	use default or path-specific setting for cache replication level	2021-08-14 15:14:01 -07:00
Chris Lu	708debca14	remote.cache and uncache: more flexible options to select files to cache or uncache	2021-08-14 15:11:55 -07:00
Chris Lu	08258a819d	fix mistake	2021-08-14 05:10:30 -07:00
Chris Lu	4909bd9684	gRpc connection error on filer when no volume left #2243 ... fix https://github.com/chrislusf/seaweedfs/issues/2243 grpc do not cache connections only when connection problem happens. Normal error results should not close the shared grpc connection.	2021-08-14 05:06:44 -07:00
Chris Lu	8126ab4b5d	rename	2021-08-14 05:03:45 -07:00
Chris Lu	0db2517994	go fmt	2021-08-14 02:55:44 -07:00
Chris Lu	7937db52e1	Filer locationPrefix configure does not exec replication #2257 ... fix https://github.com/chrislusf/seaweedfs/issues/2257	2021-08-14 02:54:13 -07:00
Chris Lu	0c66b173a4	fix	2021-08-13 11:31:43 -07:00
Chris Lu	e02a317d3d	adjust retry logic in case some data is partially written	2021-08-13 11:30:38 -07:00
Chris Lu	2d519c6cb6	adjust the retry logic	2021-08-13 11:13:30 -07:00
Chris Lu	b961fcd338	filer: stream read from volume server, reduce memory usage	2021-08-13 11:00:11 -07:00
Chris Lu	f4decf02df	volume copying: clean up stale volume data files ... fix https://github.com/chrislusf/seaweedfs/issues/2250	2021-08-13 03:24:21 -07:00
Chris Lu	0f7d4556d8	shell: volume.tier.move makes up changes if volume move failed	2021-08-13 03:09:28 -07:00
Chris Lu	333cdce485	add verbose message	2021-08-13 02:57:14 -07:00
Chris Lu	78e8ddf910	Only when tailing volume, the zero-ed cookie should skip checking. ... This only happens when checkCookie == false and fsync == false.	2021-08-13 02:09:35 -07:00
Chris Lu	a8617c1a39	tail volume: fix zero cookie problem from batch deletion	2021-08-13 01:54:35 -07:00
Chris Lu	5a0f92423e	use grpc and jwt	2021-08-12 21:40:33 -07:00
Chris Lu	6238644c35	remove gateway command	2021-08-12 20:52:04 -07:00
Chris Lu	d1d1fc772c	move some volume lookup operations to grpc ... jwt related lookup will come in next commit	2021-08-12 20:33:00 -07:00
Chris Lu	5571f4f70a	master: add master.follower to handle read file id lookup requests	2021-08-12 18:10:59 -07:00
Chris Lu	5469019852	adjust data type	2021-08-12 17:54:34 -07:00
Chris Lu	bfac55e6c0	avoid integer overflow ... fix https://github.com/chrislusf/seaweedfs/issues/2254	2021-08-11 22:22:49 -07:00
Chris Lu	5516fa8e80	Merge pull request #2251 from kmlebedev/rertyUpdateIamIdentity ... Retry save and update IAM identity	2021-08-11 09:21:03 -07:00
Konstantin Lebedev	f0afd35eec	Retry save and update IAM identity ... https://github.com/chrislusf/seaweedfs/issues/2242	2021-08-11 19:29:04 +05:00
byunghwa.yun	775dfbae85	Synchronize number of open files	2021-08-11 23:14:56 +09:00
Konstantin Lebedev	ec09966fd3	Retry save and update IAM identity ... https://github.com/chrislusf/seaweedfs/issues/2242	2021-08-11 17:53:48 +05:00
Chris Lu	f2cd753bf9	fix avoid lock error ... fix https://github.com/chrislusf/seaweedfs/issues/2247	2021-08-10 14:34:13 -07:00
Chris Lu	01336d71eb	minor	2021-08-10 13:04:33 -07:00
Chris Lu	1154e23e2d	add logs for volume creation	2021-08-10 13:04:25 -07:00
Chris Lu	057ef429ac	format	2021-08-10 12:33:29 -07:00
Chris Lu	b63b042afc	dedup keeps the largest replica	2021-08-10 12:30:41 -07:00
Chris Lu	0526db12e2	do not treat read only volumes differently	2021-08-10 11:37:12 -07:00
Chris Lu	e50a5b8e28	minor: print disk type	2021-08-10 11:10:09 -07:00
Chris Lu	db6275a0c8	print out balance ratio	2021-08-10 10:43:42 -07:00
Chris Lu	9d85569c55	ensure using local quorum consistency	2021-08-10 05:10:57 -07:00
Chris Lu	85832d02c0	wait for goroutines	2021-08-10 04:13:12 -07:00
Bl1tz23	1c94b3d013	merge master, resolve conflicts	2021-08-10 13:45:24 +03:00
Bl1tz23	e6e57db530	Add liveness\readiness probe for s3 api handler on /status path	2021-08-10 13:42:46 +03:00
Chris Lu	8ff6c9a0c6	output format	2021-08-10 03:25:18 -07:00
Chris Lu	48f448ee09	parallelize tier move	2021-08-10 03:08:29 -07:00
Chris Lu	69a6da7969	avoid fail on tail error	2021-08-10 02:50:28 -07:00
Chris Lu	18228f3044	fix help message	2021-08-10 02:48:41 -07:00
Chris Lu	69655ba8e5	mount: cache on reading remote storage	2021-08-09 22:11:57 -07:00
Chris Lu	a7012d9729	fix	2021-08-09 16:03:03 -07:00
Chris Lu	02f728cb54	fix bug with remote.uncache	2021-08-09 15:24:21 -07:00
Chris Lu	8d3e275735	remote: filer cache remote content on read	2021-08-09 15:16:45 -07:00
Chris Lu	9096f6f4f7	cache: set upper limit of chunk size	2021-08-09 15:08:53 -07:00
Chris Lu	402315f117	go fmt	2021-08-09 14:37:34 -07:00
Chris Lu	a6be2520c9	fix	2021-08-09 14:37:25 -07:00
Chris Lu	713c035a6e	shell: remote.cache remote.uncache	2021-08-09 14:35:18 -07:00
Chris Lu	8cfd487608	2.62	2021-08-08 23:33:12 -07:00
Chris Lu	734c980040	volume: support concurrent download data size limit	2021-08-08 23:25:16 -07:00
Chris Lu	96ce85f5ae	rename	2021-08-08 22:33:31 -07:00
Chris Lu	c5f38c365d	go fmt	2021-08-08 22:30:36 -07:00
Chris Lu	df85f7a1eb	adjust help message	2021-08-08 22:30:12 -07:00
Chris Lu	c0b12da4ef	shell: add filer.remote.unmount	2021-08-08 22:26:37 -07:00
Chris Lu	882a93dacd	fix tests	2021-08-08 17:56:26 -07:00
Chris Lu	7412ccdf88	write back remote entry to local entry after uploading to remote	2021-08-08 17:55:03 -07:00
Chris Lu	dcf614a8c3	skip if the remote entry update is because of internal managerial operations	2021-08-08 15:58:10 -07:00
Chris Lu	eed26af266	Merge branch 'master' into add_remote_storage	2021-08-08 15:48:04 -07:00
Chris Lu	4370a4db63	use int64 for volume count in case of negative overflow	2021-08-08 15:19:39 -07:00
Chris Lu	26c222f596	shell: volume.tier.move avoid moving all volumes to one destination	2021-08-08 15:12:39 -07:00
Chris Lu	13e45e1605	filer.remote.sync can work now	2021-08-08 01:21:42 -07:00
Chris Lu	8f5170c138	remove imports	2021-08-07 16:20:17 -07:00
Chris Lu	46b0cb8c86	Merge branch 'master' into add_remote_storage	2021-08-07 15:41:27 -07:00
Chris Lu	de730b079d	ChunkStreamReader implenents io.ReaderAt	2021-08-07 15:41:07 -07:00
Chris Lu	59732a0529	refactoring	2021-08-07 15:35:27 -07:00
Chris Lu	ecb234f75a	refactor	2021-08-07 14:46:23 -07:00
Chris Lu	270770d7d7	refactor	2021-08-07 14:18:53 -07:00
Chris Lu	679f800caa	Merge branch 'master' into add_remote_storage	2021-08-06 20:27:12 -07:00
Chris Lu	0c0f77e2ae	skip not found error on deletion	2021-08-06 19:35:47 -07:00
Chris Lu	8828f485c0	print volume deletion error	2021-08-06 19:30:22 -07:00
Chris Lu	d2b23f3d38	Merge pull request #2238 from combineads/change_default_mode ... Change default permissions	2021-08-05 21:28:06 -07:00
Chris Lu	e23ca3845b	Merge pull request #2237 from combineads/add_mtime ... Add crtime and mtime	2021-08-05 21:27:13 -07:00
Chris Lu	286e5dd375	Merge branch 'master' into add_remote_storage	2021-08-05 21:07:04 -07:00
Chris Lu	1e22166939	adjust error message	2021-08-05 21:06:55 -07:00
byunghwa.yun	f3dc909b21	Change default permissions	2021-08-06 12:24:35 +09:00
byunghwa.yun	e84fad9acf	Add crtime and mtime	2021-08-06 12:13:37 +09:00
Chris Lu	1a5d29520c	add tests	2021-08-05 14:49:24 -07:00
Chris Lu	6b743dbbf9	refactor client subscribe metadata	2021-08-04 16:25:46 -07:00
Chris Lu	b9ecf1e3a8	refacotring	2021-08-04 14:56:13 -07:00
Chris Lu	d84c311699	refactoring	2021-08-04 12:30:18 -07:00
Chris Lu	f6a9ad8001	fix tests	2021-08-04 00:31:06 -07:00
Chris Lu	42969c9c62	Merge branch 'master' into add_remote_storage	2021-08-03 21:20:05 -07:00
Chris Lu	8cf0c515bf	shell: volume.fix.replication retries even when there is no error #2235 ... fix https://github.com/chrislusf/seaweedfs/issues/2235	2021-08-03 09:32:55 -07:00
Chris Lu	d2ddf1dbdb	typo	2021-08-03 09:32:31 -07:00
Chris Lu	3afbf04007	add TLS grpc support for filer meta clients	2021-08-03 01:30:35 -07:00
Chris Lu	fe60c6ef9a	minor changes	2021-08-02 11:49:40 -07:00
byunghwa.yun	bdc7730fdb	Add autocomplete	2021-08-03 02:56:52 +09:00
Chris Lu	2ae9705442	adjust text	2021-08-01 22:55:19 -07:00
Chris Lu	3739717092	Revert "adds a test" ... This reverts commit f690643b47.	2021-08-01 22:54:45 -07:00
Chris Lu	f690643b47	adds a test	2021-08-01 22:53:50 -07:00
Chris Lu	6de786185d	volume.balance: balance read only volumes first	2021-08-01 22:32:50 -07:00
Chris Lu	3bb640b786	add -force option following #2228	2021-08-01 20:03:05 -07:00
byunghwa.yun	9e839cb5cc	Add force option in volume.deleteEmpty command	2021-08-02 11:37:20 +09:00
Chris Lu	56ee1d5ef1	2.61	2021-08-01 15:50:19 -07:00
Chris Lu	81b255df8b	Merge branch 'master' into add_remote_storage	2021-08-01 15:44:21 -07:00
Chris Lu	58bc3ecf47	add default quietFor value	2021-08-01 15:36:06 -07:00
Chris Lu	2ca1839d77	shell: add volume.deleteEmpty command	2021-08-01 15:33:45 -07:00
Chris Lu	89933c46d2	s3: skip hidden directories in /buckets folder	2021-08-01 12:28:08 -07:00
Chris Lu	1c7e404abe	remove buckets folder option ... the related code still works for old deployments	2021-08-01 12:23:16 -07:00
Chris Lu	cb1dbd3135	refactor	2021-08-01 11:53:46 -07:00
Chris Lu	1161c70dab	Merge branch 'master' into add_remote_storage	2021-08-01 01:10:13 -07:00
Chris Lu	9cc84a910f	volume: deletion can skip volume size checking ... fix https://github.com/chrislusf/seaweedfs/issues/2225	2021-08-01 00:32:51 -07:00
Chris Lu	767edd3c08	rename	2021-07-31 23:52:09 -07:00
Chris Lu	9df7d16791	read <- remote_storage	2021-07-31 22:39:38 -07:00
Chris Lu	f5a69a0e44	Merge branch 'master' into add_remote_storage	2021-07-31 09:39:48 -07:00
Chris Lu	6ba65c3382	customizable debug port	2021-07-31 09:18:41 -07:00
byunghwa.yun	5d4438a72f	Fix typo	2021-07-31 22:23:52 +09:00
Chris Lu	1ff8285d82	debug from any server	2021-07-31 02:13:21 -07:00
Chris Lu	0d6f45cb46	add debug option to generate full goroutine stack dump	2021-07-31 02:00:01 -07:00
Chris Lu	a3290faf17	shell command to calculate size / number of volumes in a collection ... fix https://github.com/chrislusf/seaweedfs/issues/2224	2021-07-30 15:49:46 -07:00
Chris Lu	c6f992b2a3	remove dead code	2021-07-30 15:18:01 -07:00
Chris Lu	899963ac20	remote storage location changed to struct	2021-07-29 02:08:55 -07:00
Chris Lu	c090d6bb25	add ReadRemote(), add read remote setup when filer starts	2021-07-28 22:43:12 -07:00
Chris Lu	4deac06da0	log fix	2021-07-28 22:25:04 -07:00
divanikus	5c6270a93a	shell: ability to use wildcards for collections, all collections if ommited	2021-07-27 23:53:01 +03:00
Chris Lu	035b0bae29	refactor	2021-07-27 03:32:24 -07:00
Chris Lu	1752eeb538	remote.mount saves the mapping	2021-07-27 03:26:35 -07:00
Chris Lu	4b94b03d90	directory to remote storage mapping	2021-07-27 01:16:28 -07:00
Chris Lu	99b599aa8a	remote.mount	2021-07-26 22:53:44 -07:00
Chris Lu	5dede5d38d	2.60	2021-07-25 22:09:09 -07:00
Chris Lu	35f70c51b0	refactor	2021-07-25 03:37:37 -07:00
Chris Lu	72eec84167	shell: add fs.mkdir	2021-07-25 02:37:39 -07:00
byunghwa.yun	2b28a818f1	Fix mysql sql for batch delete	2021-07-25 11:06:14 +09:00
Chris Lu	10464f47d0	Merge pull request #2217 from combineads/add_webp_format	2021-07-24 09:42:41 -07:00
byunghwa.yun	7374b5b473	Add webp extension for resizing	2021-07-24 14:32:31 +09:00
byunghwa.yun	2595f269d1	Add the webp image type	2021-07-24 14:26:40 +09:00
Chris Lu	ac28611817	snowflake sequencer need an unique id ... fix https://github.com/chrislusf/seaweedfs/issues/2213	2021-07-23 20:54:03 -07:00
Chris Lu	5c14da0f1e	filer.copy: fail early if assign request failed ... fix https://github.com/chrislusf/seaweedfs/issues/2216	2021-07-23 20:01:43 -07:00
Chris Lu	4cc2165061	weed -h subcommand list is not sorted, makes discovery difficult ... fix https://github.com/chrislusf/seaweedfs/issues/2214	2021-07-23 18:44:53 -07:00
Chris Lu	10fc478557	scaffold -config= should output to stdout ... fix https://github.com/chrislusf/seaweedfs/issues/2212	2021-07-23 18:41:25 -07:00
Chris Lu	092932af8f	shell: faster bucket delete ... avoid waiting for large object metadata	2021-07-23 11:25:46 -07:00
Chris Lu	bdb632fa62	Merge pull request #2211 from qieqieplus/filer-bloom-filter ... add bloom filter for filer leveldb/rocksdb	2021-07-22 23:35:54 -07:00
qieqieplus	2b46df13f8	add bloom filter for filer leveldb/rocksdb	2021-07-23 14:05:59 -06:00
Chris Lu	60f5c0a2f5	fix security alert on github.com/dgrijalva/jwt-go ... resolve https://github.com/chrislusf/seaweedfs/security/dependabot/go.mod/github.com%2Fdgrijalva%2Fjwt-go/open	2021-07-22 10:22:09 -07:00
Chris Lu	182288f860	filer: fix mysql, postgres batch delete error	2021-07-22 08:23:20 -07:00
Chris Lu	84d91f143f	mount: hide /etc folder also from mount	2021-07-22 00:40:16 -07:00
Chris Lu	7359193e97	go fmt	2021-07-21 14:38:12 -07:00
Chris Lu	cc32436d63	Merge pull request #2209 from combineads/fuse_option ... Fix error handling	2021-07-21 12:37:20 -07:00
Chris Lu	67537ff05b	Merge pull request #2208 from combineads/add_rm_options	2021-07-21 11:08:43 -07:00
byunghwa.yun	e0c7708b03	Fix error handling	2021-07-21 22:48:04 +09:00
byunghwa.yun	41d559f54e	Add fs rm options	2021-07-21 21:10:36 +09:00
Chris Lu	70effac0d3	configure and store remote configurations	2021-07-21 02:24:34 -07:00
Chris Lu	3138805b33	Merge pull request #2200 from combineads/add_fs_rm ... weed shell: fs.rm remove file or directory	2021-07-21 00:35:54 -07:00
Chris Lu	a125c8fbe2	Merge pull request #2204 from nivekuil/tokenpolicy ... cassandra: use LocalQuorum for all queries	2021-07-20 15:37:31 -07:00
nivekuil	b9a67d46c5	cassandra: use LocalQuorum for all queries ... This changes this filer store from eventual to strong consistency at the cost of read performance.	2021-07-20 14:47:39 -07:00
Chris Lu	54c8bc8673	Merge branch 'remote_overlay'	2021-07-19 23:18:35 -07:00
Chris Lu	7ab389e7ec	optimization: improve random range query for large files	2021-07-19 23:07:22 -07:00
Chris Lu	93c37cfded	Update filechunk_manifest.go	2021-07-19 18:41:41 -07:00
byunghwa.yun	b23b307e08	Add fs rm	2021-07-20 08:49:28 +09:00
Chris Lu	b938df97a2	remove unused parameter	2021-07-19 02:59:12 -07:00
Chris Lu	450222dd64	add remote to filer.Entry and filer_pb entry, add RemoteConf	2021-07-19 02:47:27 -07:00
Chris Lu	8dc5def435	proto: add remote	2021-07-19 01:12:31 -07:00
Chris Lu	99155e4880	add -retry option for volume.fix.replication ... fix https://github.com/chrislusf/seaweedfs/issues/2191	2021-07-16 12:13:46 -07:00
Chris Lu	2f209675ab	Added -retry option for volumeServer.evacuate ... related to https://github.com/chrislusf/seaweedfs/issues/2191	2021-07-16 12:08:21 -07:00
Chris Lu	fb7a1be1c4	refactor	2021-07-16 11:47:37 -07:00
nivekuil	2faf96f002	cassandra: Use LocalOne instead of One consistency	2021-07-16 04:29:46 -07:00
Chris Lu	4be5b4ff5c	[volume.check.disk] skip errors and check next volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2129	2021-07-16 02:03:32 -07:00
Chris Lu	e95166d739	S	2021-07-15 17:43:38 -07:00
Chris Lu	a45bbc0b75	2.59	2021-07-15 15:52:22 -07:00
Chris Lu	f0042f62dd	readable logs	2021-07-15 12:17:48 -07:00
Chris Lu	18c40686d9	s3: multipart upload miss data if file is chunked in 4MB ... fix https://github.com/chrislusf/seaweedfs/issues/2195	2021-07-15 11:56:28 -07:00
Chris Lu	5a838dbe53	fix compilation	2021-07-13 11:40:21 -07:00
Chris Lu	6103649ffb	shell: volume.check.disk adds retries in case the volumes are just moved ... related to https://github.com/chrislusf/seaweedfs/issues/2194	2021-07-13 11:19:56 -07:00
Chris Lu	49c66e88a0	volume: change all writes to fsync during graceful stopping ... fix https://github.com/chrislusf/seaweedfs/issues/2193	2021-07-13 01:29:57 -07:00
Chris Lu	01adc567aa	shell: volume.fsck deletes a volume is the volume has only orphaned data. ... fix https://github.com/chrislusf/seaweedfs/issues/2190	2021-07-12 11:22:00 -07:00
Chris Lu	297b41266b	2.58	2021-07-12 01:33:47 -07:00
Chris Lu	b194f91f47	add version to filer configuration response	2021-07-12 01:23:20 -07:00
bingoohuang	9e48bff8ee	fix typo in fs.configure help message text	2021-07-12 13:51:21 +08:00
Chris Lu	d013d6d968	shell: volume.fsck "reallyDeleteFromVolume" should send padded file ids to delete ... fix https://github.com/chrislusf/seaweedfs/issues/2188	2021-07-10 23:16:06 -07:00
Chris Lu	3d624d1e16	rename	2021-07-09 03:19:21 -07:00
Chris Lu	ecce300964	s3 config read via grpc	2021-07-09 02:48:03 -07:00
Chris Lu	da7bd62822	/etc files are stored inside metadata store	2021-07-09 02:33:14 -07:00
Chris Lu	a6d73e0a66	Merge pull request #2185 from bingoohuang/master ... show RemoteVolumes/EcVolumes only if it is not empty	2021-07-06 00:28:07 -07:00
bingoohuang	ed57a55eae	show RemoteVolumes/EcVolumes only if it is not empty	2021-07-06 15:20:18 +08:00
Chris Lu	44b50b2fdf	another fix related to 64 bit alignment ... fix https://github.com/chrislusf/seaweedfs/issues/2177	2021-07-05 16:30:43 -07:00
Chris Lu	141388367e	Merge pull request #2183 from bingoohuang/master ... extract embed html of master/volume/filer ui to separate files	2021-07-05 03:22:26 -07:00
bingoohuang	44a2538f67	extract embed html of master/volume/filer ui to separate files	2021-07-05 18:09:44 +08:00
Chris Lu	e2da647fa3	Merge pull request #2182 from qieqieplus/fix-event-notification ... Subscribe meta data: sync empty notification with timestamp	2021-07-05 02:00:53 -07:00
qieqieplus	233103f6b2	sync empty notification with timestamp	2021-07-05 16:01:16 +08:00
bingoohuang	5dbbe19c8b	extract embed toml example to separate files	2021-07-05 11:16:49 +08:00
Chris Lu	5bcc77b46c	volume: default readMode to proxy	2021-07-03 15:55:56 -07:00
Chris Lu	f5fa0b08fd	2.57	2021-07-03 15:10:57 -07:00
Chris Lu	a024254ad7	logging	2021-07-03 14:51:01 -07:00
Chris Lu	d39b2689a5	S3 authorization: StreamingSigned enforces access control ... fix https://github.com/chrislusf/seaweedfs/issues/2180	2021-07-03 14:50:53 -07:00
Chris Lu	fa0dab6029	mount: rename also recursively move file handles ... related to https://github.com/chrislusf/seaweedfs/issues/2169	2021-07-03 02:59:35 -07:00
Chris Lu	8fe75692ee	volume: address "unaligned 64-bit atomic operation" ... fix https://github.com/chrislusf/seaweedfs/issues/2177	2021-07-02 13:57:43 -07:00
Chris Lu	2cb8b31ea7	Merge pull request #2179 from nivekuil/tokenpolicy ... Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default	2021-07-02 13:51:53 -07:00
nivekuil	8425705643	Cassandra: Use TokenAwareHostPolicy by default with fallback ... See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing	2021-07-02 13:50:01 -07:00
Chris Lu	d8bda0b229	locate the weed binary	2021-07-02 13:33:17 -07:00
danielflira	586e066897	fix weed fuse parameters parsing	2021-07-02 16:52:52 -03:00
lyg	1a4db87e19	fix s3 metadata error with multipart upload	2021-07-02 11:00:42 +08:00
Chris Lu	c6d4c16079	S3: add metadata with multipart upload ... fix https://github.com/chrislusf/seaweedfs/issues/2173	2021-07-01 19:12:11 -07:00
Chris Lu	2420c60fc4	log reading adds delay between retries	2021-07-01 14:01:25 -07:00
Chris Lu	067eb15e70	remove debug messages	2021-07-01 01:24:07 -07:00
Chris Lu	b624090398	go fmt	2021-07-01 01:21:14 -07:00
Chris Lu	215b169562	mount: recursively rename locally	2021-07-01 01:19:31 -07:00
Chris Lu	c795183887	return node itself as directory handler	2021-07-01 01:19:31 -07:00
Chris Lu	30dbe98dff	Update Makefile	2021-07-01 01:19:31 -07:00
Chris Lu	07f20155fd	small optimization	2021-07-01 01:19:31 -07:00
danielflira	a66112c9d2	configurable fusermount path	2021-07-01 00:07:54 -03:00
Kevin Liu	253b4ed898	remote -> proxy in server to match volume flags ... noticed this was missed in https://github.com/chrislusf/seaweedfs/pull/2168	2021-06-30 13:05:45 -07:00
zhangsong	20d33ae025	add proxy mode to read non-local volumes	2021-06-30 18:33:18 +08:00
zhangsong	7566782c2e	add proxy mode to read non-local volumes	2021-06-30 17:28:37 +08:00
陈杨文	6f683e6572	also failed on mips64 ... alpine support mips64, will try to build on next version	2021-06-30 04:36:00 +08:00
Chris Lu	41db292332	skip s390 ppc64le ... due to https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/22703	2021-06-29 10:06:38 -07:00
Chris Lu	3668d10664	range query for mp4 video play ... fix https://github.com/chrislusf/seaweedfs/issues/2156	2021-06-29 02:13:29 -07:00
Chris Lu	24e11d1e90	look back when adding to sorted values ... look back when adding to sorted values, before adding it to overflow	2021-06-28 22:46:49 -07:00
Chris Lu	fc8dd58aea	volume: large_volume version has bug when using in memory index ... fix https://github.com/chrislusf/seaweedfs/issues/2162	2021-06-28 15:48:07 -07:00
Chris Lu	a2979aa051	2.56	2021-06-27 23:33:45 -07:00
Chris Lu	08377fecb8	fixes	2021-06-27 23:32:57 -07:00
Chris Lu	17477b37d5	sleep before re-reading the messages ... If there are no more metadata changes and the client disconnects, it would go into a busy loop without this fix.	2021-06-27 06:31:04 -07:00
Chris Lu	c764596f96	filer: slow metadata topic read may lose some change events ... fix https://github.com/chrislusf/seaweedfs/issues/2117	2021-06-27 05:54:16 -07:00
Chris Lu	cc7714fdbe	logging changes to debug	2021-06-27 05:54:16 -07:00
thephoenixofthevoid	8d70ba2eaa	Fix: Loosing environment variables at StartProcess	2021-06-26 15:20:27 +03:00
Chris Lu	1cac2f2278	shell: volume.fsck checks missing chunks in filer ... fix https://github.com/chrislusf/seaweedfs/issues/2154	2021-06-24 23:56:24 -07:00
Chris Lu	9dd09bbb33	refactor	2021-06-24 17:22:53 -07:00
Chris Lu	c2e0a75c1f	adjust logs	2021-06-24 12:46:00 -07:00
Chris Lu	78b1fb921c	adjust log level	2021-06-23 20:59:54 -07:00
Chris Lu	05af54ad10	2.55 ... trigger migrated travis build	2021-06-23 00:41:04 -07:00
Chris Lu	d4b0dcf442	Merge pull request #2147 from jonaz/process_metrics ... Add process metrics of weed itself	2021-06-22 04:13:17 -07:00
Jonas Falck	829b195084	Add process metrics of weed itself	2021-06-22 13:09:42 +02:00
Chris Lu	d474ce6fe3	master: avoid repeated leader redirection ... fix https://github.com/chrislusf/seaweedfs/issues/2146	2021-06-21 22:56:07 -07:00
danielflira	f1d207a0fe	start weed with mount in background	2021-06-20 02:48:46 -03:00
Chris Lu	b3eb4fecc7	2.54	2021-06-19 03:48:15 -07:00
Chris Lu	f24bb9e688	mount: fix for deletion stopped working since 2.53 ... fix https://github.com/chrislusf/seaweedfs/issues/2138 due to 4d0cbd2700	2021-06-19 03:46:39 -07:00
Chris Lu	88d52adfdd	remove unused fields	2021-06-18 15:35:22 -07:00
Chris Lu	56eb522b13	fix stats when a collection is deleted	2021-06-15 21:11:31 -07:00
Chris Lu	055374a50b	FUSE: skip flushing if file is deleted ... related to https://github.com/chrislusf/seaweedfs/issues/2110	2021-06-15 12:45:23 -07:00
Konstantin Lebedev	6aa1a56ec8	avoid crashes Galera Cluster ... https://github.com/chrislusf/seaweedfs/issues/2125	2021-06-15 18:12:39 +05:00
Chris Lu	ebe971da2e	Merge pull request #2127 from nagy/check-history-error ... Check for history file errors as well	2021-06-15 03:29:21 -07:00
Chris Lu	7a81caa31e	2.53	2021-06-13 17:12:34 -07:00
Chris Lu	4d0cbd2700	skip cookie checking if from grpc api	2021-06-13 16:16:11 -07:00
Chris Lu	7225cb4ac5	add block and mutex profiling	2021-06-13 16:15:54 -07:00
Chris Lu	28a4a1f8d6	fix for mysql2 postgres2 on fast dropping buckets	2021-06-13 07:31:56 -07:00
Chris Lu	1e76fc994a	filer.copy: zero fileSize for directories	2021-06-13 07:09:06 -07:00
Chris Lu	ed6aa13520	minor	2021-06-12 18:36:25 -07:00
Chris Lu	ee6c67682c	minor	2021-06-12 02:52:41 -07:00
Chris Lu	9357911a95	remove all bucket metadata ... fix https://github.com/chrislusf/seaweedfs/issues/2118	2021-06-10 23:37:54 -07:00
Chris Lu	b71c3cfba4	avoid possible empty object	2021-06-10 22:17:53 -07:00
Chris Lu	310e31424e	adjust the error output ... fix https://github.com/chrislusf/seaweedfs/issues/2123	2021-06-10 21:55:13 -07:00
Chris Lu	8b382a8209	refactor	2021-06-10 21:50:21 -07:00
Chris Lu	33b87244ef	refactoring	2021-06-10 16:54:36 -07:00
Daniel Nagy	7ce3bee0c0	Check for history file errors as well	2021-06-08 11:14:49 +02:00
Daniel Nagy	0e3adde47f	Return correct exitcode when wheed upload fails	2021-06-07 23:38:14 +02:00
Chris Lu	76f24af79a	2.52	2021-06-07 12:13:23 -07:00
Chris Lu	5e6dfbc25f	locks for data racing	2021-06-07 12:04:50 -07:00
Chris Lu	452c6ef183	limits concurrent uploads for one file	2021-06-06 23:05:17 -07:00
Chris Lu	8295e2feb6	skip md5 checking for now because of race condition	2021-06-06 22:16:32 -07:00
Chris Lu	c8dea3dd89	2.51	2021-06-06 21:54:00 -07:00
Chris Lu	19caeb7b02	fix writing the small file	2021-06-06 20:57:03 -07:00
Chris Lu	bb45dea15a	filer: parallel data upload	2021-06-06 20:23:36 -07:00
Chris Lu	e00443a940	mount: adjust starting order ... avoid possible nil wfs.Server	2021-06-06 20:22:42 -07:00
Chris Lu	44f1ba6894	refactor	2021-06-06 18:43:04 -07:00
Chris Lu	6c82326575	use bytes.Buffer to reduce memory allocation and gc	2021-06-06 13:42:36 -07:00
Chris Lu	9cba5cca0b	optionally disable concurrent upload limit	2021-06-06 13:13:33 -07:00
Chris Lu	21ad9a4ac2	filer: mongodb avoids E11000 duplicate key error collection: seaweedfs.filemeta index: directory_1_name_1 dup key	2021-06-06 13:12:01 -07:00
Chris Lu	e6ba2f9c37	verbose but cleaner logic to invalidate fuse cache	2021-06-05 13:09:37 -07:00
Chris Lu	fadc1febdb	FUSE: invalidate FUSE cached entries ... fix https://github.com/chrislusf/seaweedfs/issues/2108	2021-06-05 02:23:07 -07:00
Chris Lu	ab606dec2a	filer: add path-specific option to enforce readonly	2021-06-04 01:03:41 -07:00
Chris Lu	0a5388744c	Merge pull request #2105 from Woellchen/current_and_parent_dot_directories ... Return artificial . and .. directories	2021-06-02 12:54:44 -07:00
Patrick Schmidt	77100754e6	Return artificial . and .. directories	2021-06-02 21:28:02 +02:00
Chris Lu	62142ff1d2	Merge pull request #2104 from kmlebedev/sync_volume_on_close ... avoid data loss after restarting a container with a volum server	2021-06-02 11:47:57 -07:00
Chris Lu	cc34475012	remove file handle locking for setattr	2021-06-02 11:44:12 -07:00
Konstantin Lebedev	99ef280c7c	avoid data loss after restarting a container with a volum server	2021-06-02 17:07:19 +05:00
Chris Lu	3db1642392	mount: skip persisting metadata if file is open	2021-06-02 03:40:08 -07:00
Chris Lu	556cc3a4ca	mount: avoid exception if disk cache is not initialized ... related to https://github.com/chrislusf/seaweedfs/issues/2102	2021-05-31 16:42:55 -07:00
Chris Lu	615cb24ba6	shell: use tab print style for completion	2021-05-31 03:29:29 -07:00
Chris Lu	c5de97ddbe	keep alive for streaming connections ... fix https://github.com/chrislusf/seaweedfs/issues/2096 related to https://github.com/chrislusf/seaweedfs/issues/1277	2021-05-31 01:26:24 -07:00
Chris Lu	2b60e2abb1	only disallow streaming signed when no auth enabled ... fix https://github.com/chrislusf/seaweedfs/issues/2101	2021-05-31 01:03:04 -07:00
Chris Lu	4233ad3f07	2.50	2021-05-30 20:40:30 -07:00
Chris Lu	1456616a77	recreate grpc connections if too many errors ... address https://github.com/chrislusf/seaweedfs/issues/2098	2021-05-30 00:07:43 -07:00
Chris Lu	fb8036385a	s3: save metadata during put-object ... fix https://github.com/chrislusf/seaweedfs/issues/2092	2021-05-29 14:14:30 -07:00
Chris Lu	2d7b4e5bb6	filer.backup: escape colon from path on windows ... fix https://github.com/chrislusf/seaweedfs/issues/2084	2021-05-29 06:45:27 -07:00
Chris Lu	ae185b997f	Merge pull request #2099 from danielflira/mount-helper ... fix parameter multiple values	2021-05-28 21:57:15 -07:00
danielflira	849f36c1ac	fix parameter multiple values	2021-05-29 00:37:25 -03:00
Eugeniy Kozlov	6e8bd16819	freespace params fix	2021-05-28 14:19:24 +03:00
Chris Lu	4d55132c7d	scaffold add comments	2021-05-27 14:40:25 -07:00
Chris Lu	45bffc92a8	filer.backup: fix cloud sinks when updating entry	2021-05-26 14:53:11 -07:00
Chris Lu	921e0d5008	remove verbose log	2021-05-26 14:43:34 -07:00
Chris Lu	1a70cb9b63	local sink: write and update files ... fix issues with https://github.com/chrislusf/seaweedfs/issues/2084	2021-05-26 14:42:21 -07:00
Chris Lu	e699d16c85	Merge pull request #2085 from danielflira/mount-helper ... create fuse subcommand to use weed with mount	2021-05-26 12:19:40 -07:00
danielflira	84488ebb33	replace filer.remote and parts[1]	2021-05-26 12:07:36 -03:00
Chris Lu	ac9bf71544	use jquery 3.6.0	2021-05-26 00:21:11 -07:00
danielflira	877c192af2	create fuse subcommand to use weed with mount	2021-05-25 23:32:35 -03:00
Chris Lu	67dd094b35	sqlite: does not support non-linux/darwin/windows	2021-05-25 17:36:15 -07:00
Chris Lu	dce1f02c9e	filer.backup: backup to local directory optionally is incremental ... fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084	2021-05-25 17:19:20 -07:00
Konstantin Lebedev	b612d5aebd	s3 test get w/ If-Match: bogus ETag	2021-05-24 16:59:44 +05:00
Konstantin Lebedev	69c768870b	- object write cache control ... - object write expires	2021-05-24 15:43:55 +05:00
Konstantin Lebedev	84dce32a57	Merge branch 'master' into head_check_all_chunks	2021-05-24 12:28:19 +05:00
Konstantin Lebedev	0e404cd84b	Chunk download stats	2021-05-24 12:14:50 +05:00
Konstantin Lebedev	bb42633522	rm chunk Download stats	2021-05-24 12:08:12 +05:00
Chris Lu	064269bb57	filer: Support sqlite as filer meta store	2021-05-23 23:58:28 -07:00
Chris Lu	b70aeb7585	bucket list only directories	2021-05-23 10:36:22 -07:00
Chris Lu	42fb03a66e	2.49	2021-05-23 00:51:47 -07:00
Chris Lu	3d3fa43542	filer: re-create grpc connections if having transport error ... fix https://github.com/chrislusf/seaweedfs/issues/2070	2021-05-22 17:46:53 -07:00
Chris Lu	2d0a0733ec	filer: bootstrap from peer filer ... fix https://github.com/chrislusf/seaweedfs/issues/1861	2021-05-22 03:17:02 -07:00
Chris Lu	e5a2bf1287	s3: deprecating filer.options.buckets_fsync	2021-05-22 00:24:23 -07:00
Chris Lu	431684798b	s3: add errors if requests are signed by no authentication is setup ... fix https://github.com/chrislusf/seaweedfs/issues/2075	2021-05-21 14:08:47 -07:00
Chris Lu	2f136a04a1	organize a bit better for temp file directory	2021-05-21 13:02:18 -07:00
Konstantin Lebedev	3325b850cc	rm func CheckAllChunkViews	2021-05-21 16:05:59 +05:00
Konstantin Lebedev	26a4f34a57	del checks ... url err logging stats chunk fetch	2021-05-21 15:59:12 +05:00
Chris Lu	e8b7d4ad64	simplify a bit	2021-05-21 02:10:44 -07:00
Chris Lu	0b00edfc3b	waits in case master lost connection	2021-05-21 02:09:22 -07:00
Chris Lu	f37a4cbd0f	randomize initial filer	2021-05-21 01:41:34 -07:00
Chris Lu	5d77840cff	adjust help message	2021-05-21 01:38:57 -07:00
Chris Lu	dc1309f084	FUSE mount: support multiple filers ... fix https://github.com/chrislusf/seaweedfs/issues/2015 fix https://github.com/chrislusf/seaweedfs/issues/1531	2021-05-21 01:28:00 -07:00
Konstantin Lebedev	42f631f549	fix https://github.com/chrislusf/seaweedfs/issues/2046	2021-05-20 13:19:35 +05:00
Chris Lu	30c67e3652	minor	2021-05-19 23:59:39 -07:00
Konstantin Lebedev	03d1199d5f	Revert "revert PR #1903 avoid http error: superfluous response.WriteHeader" ... This reverts commit ac71117e	2021-05-20 11:45:21 +05:00
Chris Lu	87a32bfef4	avoid possible nil when node is disconnected from its parent ... fix https://github.com/chrislusf/seaweedfs/issues/2073	2021-05-19 10:02:01 -07:00
Chris Lu	1aa7e99a89	skip file not found error when deleting	2021-05-15 09:37:39 -07:00
Chris Lu	45a762223a	2.48	2021-05-14 10:26:42 -07:00
Chris Lu	789465d99e	set renamed item to new directory ... related to https://github.com/chrislusf/seaweedfs/issues/2064	2021-05-13 22:52:19 -07:00
Chris Lu	83c037e093	fix logs	2021-05-13 22:41:19 -07:00
Chris Lu	5a29d284fa	ensure proper directory name ... a fix related to https://github.com/chrislusf/seaweedfs/issues/2064	2021-05-13 10:18:35 -07:00
Chris Lu	3ff307e842	reduce logs	2021-05-13 00:32:37 -07:00
Chris Lu	a48ebd7c73	mount: read file when file is still being written ... a possible fix for https://github.com/chrislusf/seaweedfs/issues/2065	2021-05-13 00:32:23 -07:00
Chris Lu	26a55bbb5c	Adjust error message when bucket name conflicts with existing collections ... fix https://github.com/chrislusf/seaweedfs/issues/2069	2021-05-12 22:30:39 -07:00
Chris Lu	ceb620a30a	directory rename: change directory name after renaming ... fix https://github.com/chrislusf/seaweedfs/issues/2068	2021-05-12 22:04:47 -07:00
Chris Lu	b430d1b6ee	filer.copy: "check.size" before copying files ... fix https://github.com/chrislusf/seaweedfs/issues/2067	2021-05-12 21:45:39 -07:00
Chris Lu	0f7b43af99	2.47	2021-05-11 10:12:15 -07:00
Chris Lu	d2d36a3f9d	master: avoid creating too many volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2062	2021-05-11 10:05:31 -07:00
Chris Lu	4596e64710	2.46	2021-05-10 21:58:37 -07:00
Chris Lu	e55aa41690	Merge branch 'upload_via_temp_file'	2021-05-10 21:48:18 -07:00
Chris Lu	1737af480a	adjust logs	2021-05-10 21:47:51 -07:00
Chris Lu	d06ecc2649	working properly	2021-05-10 21:47:07 -07:00
Chris Lu	fe2edd4b50	2.45	2021-05-10 13:53:03 -07:00
Chris Lu	55d547afec	switch back to in memory uploading for now	2021-05-10 13:50:10 -07:00
Chris Lu	9a6aa00e9d	avoid nil locations ... fix https://github.com/chrislusf/seaweedfs/issues/2059	2021-05-10 02:39:52 -07:00
Chris Lu	da0a4e775b	Revert "Revert "Merge pull request #2027 from bingoohuang/master"" ... This reverts commit d74cdf0115.	2021-05-10 00:13:55 -07:00
Chris Lu	aa6949ef25	2.44	2021-05-09 23:28:48 -07:00
Chris Lu	ba77833343	EOF handling	2021-05-09 23:04:24 -07:00
Chris Lu	f4cf30b497	mount: write to tmp file before uploading	2021-05-09 22:56:10 -07:00
Chris Lu	280e7cf289	mount: in case the set attribute is called before persisting the file	2021-05-09 22:55:30 -07:00
Chris Lu	74052064b6	add one missing optimization	2021-05-09 17:25:42 -07:00
Chris Lu	59ace54925	refactor	2021-05-09 17:22:30 -07:00
Chris Lu	50be19d23e	refactor	2021-05-09 15:33:01 -07:00
Chris Lu	55e060cf61	refactor	2021-05-09 15:28:54 -07:00
Chris Lu	735e65be17	refactor	2021-05-09 15:22:38 -07:00
Chris Lu	93e84a12f2	refactor	2021-05-09 15:15:18 -07:00
Chris Lu	3942e3b2ef	a better fix	2021-05-09 01:42:19 -07:00
Chris Lu	957e1a1bc1	fuse: important: if filer -filer.path is not root, directory listing will fail	2021-05-09 00:32:21 -07:00
Chris Lu	7ca75347ec	minor	2021-05-07 21:56:45 -07:00
Chris Lu	8f8738867f	add retry to assign volume ... fix https://github.com/chrislusf/seaweedfs/issues/2056	2021-05-07 07:29:26 -07:00
Chris Lu	007401f3a0	remove duplicated code	2021-05-07 07:14:24 -07:00
Chris Lu	aaad4b578b	Merge pull request #2049 from qieqieplus/async-assign ... ahead of time volume assignment	2021-05-06 10:00:25 -07:00
qieqieplus	c4d32f6937	ahead of time volume assignment	2021-05-06 18:55:44 +08:00
Chris Lu	55a8f57381	go fmt	2021-05-06 03:37:51 -07:00
Chris Lu	5753749c90	remove verbose logs	2021-05-06 03:34:34 -07:00
Chris Lu	38f411219a	mount: skip local chunk cache if opened write only	2021-05-06 03:31:40 -07:00
Chris Lu	c899bdf063	a little optimization	2021-05-06 03:03:00 -07:00
Chris Lu	3eb336e0b0	report error only for the first multipart upload ... the glog.Errorf would always print for s3 multipart uploads	2021-05-06 01:57:54 -07:00
Chris Lu	76c48ffe27	optional parallel copy ec shards ... fix https://github.com/chrislusf/seaweedfs/issues/2048	2021-05-06 01:53:35 -07:00
Chris Lu	ac71117ee6	revert PR #1903 avoid http error: superfluous response.WriteHeader	2021-05-05 15:11:39 -07:00
Chris Lu	24efa31e49	Merge pull request #2045 from qieqieplus/fix-vacuum-commit	2021-05-05 07:41:38 -07:00
qieqieplus	ac26080bd2	fix concurrent vacuum & delete panic	2021-05-05 17:54:50 +08:00
Chris Lu	e24ba2aadc	filer: delete specific tags ... fix https://github.com/chrislusf/seaweedfs/issues/2041	2021-05-02 21:53:43 -07:00
Chris Lu	e87e6ef33c	s3: return 404 if bucket does not exist ... fix https://github.com/chrislusf/seaweedfs/issues/2039	2021-05-02 21:30:37 -07:00
Chris Lu	a4bb37a5fe	add debug info	2021-05-01 16:10:26 -07:00
Chris Lu	2e56407c6b	fix visited checking	2021-05-01 16:09:29 -07:00
Chris Lu	c48ef78670	2.43	2021-05-01 00:39:04 -07:00
Chris Lu	3a86d4dbfd	mount: fix directory invalidation ... fix https://github.com/chrislusf/seaweedfs/issues/2038	2021-04-30 22:51:06 -07:00
Chris Lu	d74cdf0115	Revert "Merge pull request #2027 from bingoohuang/master" ... Need to revert because docker image build failed. The docker apk package only has go 1.15.	2021-04-30 03:36:15 -07:00
Chris Lu	84312e6799	2.42	2021-04-30 03:14:07 -07:00
Chris Lu	fa0d973113	Merge pull request #2037 from utsl42/reader_at ... make reader_at handle random reads more efficiently for FUSE	2021-04-28 17:31:46 -07:00
Nathan Hawkins	042de9359c	make reader_at handle random reads more efficiently for FUSE	2021-04-28 19:13:37 -04:00
Chris Lu	a26a37dfa3	fix compilation ... fix related to #2032	2021-04-28 13:36:53 -07:00
Chris Lu	ef94ff6837	Merge pull request #2035 from kmlebedev/fix_chunks_etag ... fix aws style Etag for chunks	2021-04-28 10:37:18 -07:00
Chris Lu	8ae3ea4b1d	Merge pull request #2034 from kmlebedev/iam_listkeys_filter ... iam ListAccessKeys filtred by username	2021-04-28 10:34:40 -07:00
Chris Lu	9dca75aea8	Merge pull request #2032 from tobiasmuehl/patch-2 ... Detect rar archives by mime type	2021-04-28 10:31:19 -07:00
Konstantin Lebedev	c2269123d3	fix aws style Etag for chunks	2021-04-28 22:28:05 +05:00
Konstantin Lebedev	39f636c682	iam ListAccessKeys filtred by username	2021-04-28 17:15:22 +05:00
Tobias Mühl	a8864e2abd	Detect rar archives by mime type ... RAR archives might not have .rar extension, see [Wikipedia](https://en.wikipedia.org/wiki/RAR_(file_format))	2021-04-28 15:54:19 +07:00
Tobias Mühl	12a7e87007	Do not compress brotli archives	2021-04-28 15:51:49 +07:00
Konstantin Lebedev	a48785c7df	auth use bucket wild cards	2021-04-27 21:45:40 +05:00
bingoohuang	7a9d27fce8	promote to go:embed instead of github.com/rakyll/statik	2021-04-27 17:22:24 +08:00
bingoohuang	cf552417a7	minFreeSpace refactored	2021-04-27 10:37:24 +08:00
bingoohuang	31f1cdeac2	minFreeSpace argument allows size like 10GiB	2021-04-26 18:48:34 +08:00
Chris Lu	86185262bb	2.41	2021-04-24 16:54:36 -07:00
Chris Lu	79f2e780c1	ensure name pattern checking is case sensitive	2021-04-24 11:51:23 -07:00
Chris Lu	ddc8643ee0	filer: directory listing adds namePatternExclude ... fix https://github.com/chrislusf/seaweedfs/issues/2023	2021-04-24 11:49:03 -07:00
Chris Lu	f0ad172e80	shell: show which server holds the lock ... fix https://github.com/chrislusf/seaweedfs/issues/1983	2021-04-22 23:56:35 -07:00
Chris Lu	89eb9f6e70	clean up .uploads directory in mysql and postgres tables ... fix https://github.com/chrislusf/seaweedfs/issues/1957 When no uploads are running, you can run this SQL to clean up. delete from <bucket_name> where directory like '/.uploads/%'	2021-04-22 23:23:23 -07:00
Chris Lu	46ef1811a1	correct help message	2021-04-22 22:26:38 -07:00
Chris Lu	11c120c040	master UI adds volume size limit	2021-04-22 14:22:48 -07:00
Chris Lu	6e5df901e4	adjust package names	2021-04-22 14:22:48 -07:00
Chris Lu	ca998328c2	do not add new volumes when below minFreeSpacePercent ... fix https://github.com/chrislusf/seaweedfs/issues/2017	2021-04-21 23:11:11 -07:00
Chris Lu	cd7bf1a72a	filer.copy copy empty folders ... fix https://github.com/chrislusf/seaweedfs/issues/2016	2021-04-21 11:17:43 -07:00
Chris Lu	a8114da02d	avoid thundering herd effect ... transient errors may cause thundering herd effect to all trying to recover from remove ec shards	2021-04-21 10:17:12 -07:00
Chris Lu	ae74d8f02a	fix error message ... related to https://github.com/chrislusf/seaweedfs/issues/2012	2021-04-21 01:40:16 -07:00
Chris Lu	11c405fc85	ensure file handles are released	2021-04-20 19:56:55 -07:00
liuxiaobo	c31c5e829c	fix path-specific filer store comment error	2021-04-20 10:08:58 +08:00
Chris Lu	83cf94ad2d	delay new file creation unless file is opened exclusively	2021-04-19 10:58:25 -07:00
Chris Lu	e983f91b03	2.40	2021-04-18 13:58:01 -07:00
Chris Lu	d1c813c470	let the fuse library manage directory id ... otherwise, on mac, during large directory deletion, if some ReaDirAll happens, the lib seems confused about the directories, and some child directories are not deleted.	2021-04-18 13:07:28 -07:00
Chris Lu	372872ebbf	set root node inode number	2021-04-18 13:07:28 -07:00
Chris Lu	e332da4837	set inode value	2021-04-18 13:07:28 -07:00
Chris Lu	2acf6be24e	resend the http request if connection is stale	2021-04-18 13:07:28 -07:00
Chris Lu	6cbd786db9	correctly runs git clone	2021-04-18 13:07:28 -07:00
Chris Lu	d41e6826d3	adjust logging	2021-04-18 13:06:38 -07:00
Chris Lu	d9a2a7f1c4	WIP ... no memory issue if some directory is removed, it may have this error $ rm -Rf ~/tmp/m2/s1 rm: fts_read: Device not configured	2021-04-18 13:06:38 -07:00
Chris Lu	54410ca955	cleaner way to set readonly	2021-04-18 10:02:02 -07:00
Konstantin Lebedev	198688c717	revert volume etag	2021-04-16 23:22:31 +05:00
Chris Lu	c83ab91e2e	remove unused variable	2021-04-16 10:34:02 -07:00
Chris Lu	3074e9b428	ensure consistent inode value	2021-04-15 22:42:24 -07:00
Chris Lu	b971317a16	avoid possible corrupted file names	2021-04-15 11:41:34 -07:00
Chris Lu	283d703d50	adjust text	2021-04-15 11:29:58 -07:00
Chris Lu	609e228578	avoid forward slash in file names	2021-04-15 10:53:04 -07:00
Chris Lu	ba92f2e714	add node.selectedVolumes ... fix https://github.com/chrislusf/seaweedfs/issues/1990	2021-04-15 03:19:28 -07:00
Chris Lu	16c0304416	ensure to delete on filer also	2021-04-15 02:29:04 -07:00
Chris Lu	217e0f9066	mount: remove folder recursively	2021-04-15 01:51:10 -07:00
Chris Lu	3e669e6d7b	mostly refactoring, add some error handling	2021-04-14 23:33:37 -07:00
Chris Lu	07f712c83f	fix typo	2021-04-14 23:21:38 -07:00
Chris Lu	36c79de3f4	fuse mount: dir ReadDirAll avoid extra conversion to filer_pb.Entry	2021-04-14 23:21:24 -07:00
Chris Lu	e41766feb6	fuse mount: dir lookup avoids extra conversion to filer_pb.Entry object	2021-04-14 22:38:34 -07:00
Chris Lu	1adc8f86ea	lighten up File object ... file.entry only exists when file.isOpen	2021-04-14 20:49:15 -07:00
Chris Lu	6bc09b18c4	truncate is a bit faster to reuse the storage	2021-04-14 20:26:56 -07:00
Chris Lu	c04b7e106f	mount: remove entry from Dir object	2021-04-14 20:26:13 -07:00
Chris Lu	fc0cbf565f	add option to obfuscate the file names	2021-04-14 15:37:24 -07:00
Chris Lu	e75633c64f	volume.check.disk: break loop for read only volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2002	2021-04-14 12:40:13 -07:00
Chris Lu	9d50867d08	volume.tier.move: avoid data loss when destination volume server already has the volume ... fix https://github.com/chrislusf/seaweedfs/issues/2001	2021-04-14 10:26:26 -07:00
Chris Lu	ff4c1d5965	adjust logging ... fix https://github.com/chrislusf/seaweedfs/issues/1999	2021-04-14 10:04:26 -07:00
qieqieplus	270645f8d7	fix #1996	2021-04-14 18:29:28 +08:00
Chris Lu	90677e1097	ensure to call line.Close() ... fix https://github.com/chrislusf/seaweedfs/issues/1995 similar to https://github.com/peterh/liner/issues/104	2021-04-14 01:33:21 -07:00
Chris Lu	3f3268cd1b	go fmt	2021-04-14 00:30:16 -07:00
Chris Lu	ca0f07a188	move file reader, entryViewCache to file handle ... reduce file object size	2021-04-14 00:29:58 -07:00
Chris Lu	5985a7d38d	add log level possible values ... fix https://github.com/chrislusf/seaweedfs/issues/1989	2021-04-12 21:15:51 -07:00
Chris Lu	f5de42fae3	Merge pull request #1975 from kmlebedev/iam_handlers ... IAM handlers	2021-04-12 12:07:45 -07:00
Chris Lu	1e033d45b8	simpler logic ... related to https://github.com/chrislusf/seaweedfs/pull/1981	2021-04-12 12:04:53 -07:00
Chris Lu	52200a903b	Merge pull request #1981 from ueni-ltd/if-modified-since ... Fix If-Modified-Since behavior	2021-04-12 12:02:54 -07:00
Chris Lu	519b0e1e49	filer: upload to a directory without "/" suffix ... fix https://github.com/chrislusf/seaweedfs/issues/1988	2021-04-12 11:56:56 -07:00
Konstantin Lebedev	8e02e138ea	Merge branch 'upstreamMaster' into iam_handlers	2021-04-12 11:22:51 +05:00
Chris Lu	742ab1ec81	2.39	2021-04-11 19:47:11 -07:00
Chris Lu	0df5b53ad8	adjust help message	2021-04-11 00:26:28 -07:00
Chris Lu	f62c153274	go fmt	2021-04-10 23:48:18 -07:00
Chris Lu	af313dff58	add gateway for easier POST and DELETE blobs	2021-04-10 23:47:47 -07:00
Konstantin Lebedev	5021bea698	GetUserPolicy	2021-04-10 23:57:45 +05:00
Chris Lu	0b82edc0d2	filer: avoid stuck uploader ... fix https://github.com/chrislusf/seaweedfs/issues/1980 reverting the file upload batch executor	2021-04-09 13:05:15 -07:00
Chris Lu	93f4146ffa	properly release the view cache	2021-04-09 12:36:39 -07:00
Merlin Gaillard	f952f979d1	filer: return 304 when If-Modified-Since == Last-Modified	2021-04-09 15:04:17 +02:00
Merlin Gaillard	4d4acc715e	s3api: handle 304 response code from filer	2021-04-09 12:13:19 +02:00
Chris Lu	6deb647a8f	mount: fix possible memory leak ... if many files are read repeatedly, their metadata are accumulated in memory. This fix cleared the metadata after the file is read.	2021-04-08 19:47:31 -07:00
Konstantin Lebedev	ba175f81b5	add auth aws signV4	2021-04-08 17:40:47 +05:00
Konstantin Lebedev	995ae91007	add DeleteUserPolicy	2021-04-08 11:16:36 +05:00
Philippe Pepiot	42a761ee20	Fix typo in weed filer long help	2021-04-07 22:47:23 +02:00
Chris Lu	3be061994f	skip connection reset error ... fix https://github.com/chrislusf/seaweedfs/issues/1971 this is because the connections are pooled but the volume server has reset the connection	2021-04-07 00:54:13 -07:00
Chris Lu	c5b08bac1b	remove mac specific mount options	2021-04-07 00:54:13 -07:00
Konstantin Lebedev	f5f8eec8e2	fix get filerGrpcAddress	2021-04-06 13:53:56 +05:00
Konstantin Lebedev	011e6e90ee	Merge branch 'upstreamMaster' into iamapipr	2021-04-06 13:50:33 +05:00
Konstantin Lebedev	ed79baa30f	add tests	2021-04-06 13:43:08 +05:00
Chris Lu	2327c0756b	fix to avoid loop	2021-04-05 23:24:26 -07:00
Chris Lu	a37eca78cd	2.38	2021-04-05 19:41:54 -07:00
Chris Lu	c131764c34	ensure error is sent back in channel ... fix https://github.com/chrislusf/seaweedfs/issues/1966 avoid shared readErr variable	2021-04-05 19:40:12 -07:00
Chris Lu	0f64f5b9c8	mount: add readOnly option ... fix https://github.com/chrislusf/seaweedfs/issues/1961	2021-04-04 21:40:58 -07:00
Chris Lu	6eee200c13	2.37	2021-04-04 18:45:48 -07:00
Chris Lu	fbb82a5c9c	skip limiting if limit is zero	2021-04-04 18:38:33 -07:00
Chris Lu	8251d1140e	refactor	2021-04-04 18:38:33 -07:00
Chris Lu	5dca98fe71	fix typo	2021-04-03 11:37:13 -07:00
Chris Lu	5e64f65632	ensure tables are created ... fix https://github.com/chrislusf/seaweedfs/issues/1957	2021-04-02 12:02:26 -07:00
Chris Lu	319743d330	go fmt	2021-04-02 02:22:59 -07:00
Chris Lu	8e404a1433	go fmt	2021-04-02 02:22:26 -07:00
Chris Lu	bdf2ddddfd	revert to same implementation as before ... This reverts commit 7e8edc3c4a.	2021-04-02 02:21:38 -07:00
Chris Lu	67e019d54b	add missing changes	2021-04-02 01:11:44 -07:00
Chris Lu	7e8edc3c4a	refactoring	2021-04-02 01:10:24 -07:00
Chris Lu	cc0df36a9e	smaller file chunks are faster	2021-04-01 02:21:59 -07:00
Chris Lu	cefe66f159	dedicated upload processor ... avoid thundering effect of overloading volume servers	2021-04-01 02:21:40 -07:00
Chris Lu	1f984d2645	refactor buffer pool	2021-04-01 02:20:00 -07:00
Chris Lu	b5880334fc	refactor	2021-03-30 21:07:34 -07:00
Chris Lu	e79e2ddeed	remove unused MoveEvents	2021-03-30 20:42:44 -07:00
Chris Lu	e06807a55b	adjust logs	2021-03-30 20:36:06 -07:00
Chris Lu	ac875976c0	filer, volume: add concurrent upload size limit to avoid OOM ... add some back pressure when writes are slow	2021-03-30 02:10:53 -07:00
Chris Lu	a1e18a1384	server add memory profiling	2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision	9385e2224d	fix postgres2 upsertQuery syntax also	2021-03-30 10:51:52 +03:00
Chris Lu	a5662c9e5e	adjust ordering	2021-03-30 00:26:57 -07:00
Chris Lu	bec74c3e6a	Merge pull request #1948 from LazyDBA247-Anyvision/master ... add enableUpsert=true	2021-03-30 00:25:01 -07:00
LazyDBA247-Anyvision	9f1cab179c	use a compatible pg syntax for upsert	2021-03-30 10:21:27 +03:00
Chris Lu	a95929e53c	reduce default concurrentWriters to 32	2021-03-30 00:17:52 -07:00
LazyDBA247-Anyvision	7f44d953b5	fix GetBool	2021-03-30 01:36:02 +03:00
LazyDBA247-Anyvision	4c51e6a660	add enableUpsert=true ... and rename config to upsertQuery	2021-03-30 00:32:03 +03:00
Chris Lu	b61e9f6a1a	adjust ordering	2021-03-29 00:49:50 -07:00
Chris Lu	1ebdbad222	Merge pull request #1946 from LazyDBA247-Anyvision/master ... Adding Customisation - insertQuery for postgres/mysql	2021-03-29 00:42:48 -07:00
Konstantin Lebedev	8a95f9c10c	iam GetUser	2021-03-29 12:01:44 +05:00
LazyDBA247-Anyvision	96c62bd34d	scaffold - insertQuery ... Example for upsert query for mysql & postgres variants	2021-03-29 10:01:12 +03:00
LazyDBA247-Anyvision	4a02389eb0	Adding custom insertQuery support for postgres/2 mysql/2	2021-03-29 09:58:13 +03:00
Chris Lu	6b7aa9633f	2.36	2021-03-28 19:09:06 -07:00
Chris Lu	bd7471d877	refactor	2021-03-25 12:05:59 -07:00
李海	06be5dc6c3	log snowflake sequencer nodeid's hex when start, it'll be in part of new assigned fid	2021-03-25 18:59:40 +08:00
李海	69b2dab9c6	add a snowflake sequencer as more robust fid generator, but less compressable than small auto-inc id	2021-03-25 18:49:26 +08:00
Konstantin Lebedev	40938d6a47	SaveInsideFiler S3 Configuration	2021-03-25 12:34:11 +05:00
Konstantin Lebedev	9f26f2815c	SaveAs S3 Configuration	2021-03-25 12:34:11 +05:00
Konstantin Lebedev	82b0463fac	handler PutUserPolicy ... https://docs.aws.amazon.com/IAM/latest/APIReference/API_PutUserPolicy.html	2021-03-25 12:34:11 +05:00
Konstantin Lebedev	d7719d0542	base handlers	2021-03-25 12:34:11 +05:00
Konstantin Lebedev	03c7953254	init Iam Api Server	2021-03-25 12:34:11 +05:00
Chris Lu	82dfe06066	Merge pull request #1933 from kmlebedev/missingKeysThreshold ... add missingKeysThreshold for fsck	2021-03-24 10:56:25 -07:00
Konstantin Lebedev	020a5d40c3	avoid counter is null	2021-03-24 22:51:22 +05:00
Konstantin Lebedev	df6cf0a2fa	nonRepairThreshold	2021-03-24 22:07:13 +05:00
Konstantin Lebedev	9ce82a9cd2	GOMAXPROCS use defaults to the value of runtime.NumCPU ... https://github.com/chrislusf/seaweedfs/issues/1942	2021-03-24 18:45:58 +05:00
Konstantin Lebedev	60972f1c97	rename option repairThreshold	2021-03-24 13:24:49 +05:00
Chris Lu	a801332b0d	filer: return 409 if file conflicts with a directory ... fix https://github.com/chrislusf/seaweedfs/issues/1938	2021-03-24 01:03:11 -07:00
Chris Lu	c42b95c596	more help message ... fix https://github.com/chrislusf/seaweedfs/issues/1937	2021-03-23 17:27:57 -07:00
Konstantin Lebedev	40a9e88c07	rename var	2021-03-23 21:42:35 +05:00
Konstantin Lebedev	d21a5bf5d0	add missingKeysThreshold for fsck	2021-03-23 15:04:07 +05:00
Chris Lu	85cff10787	mysql: ensure name column is case sensitive ... fix https://github.com/chrislusf/seaweedfs/issues/1931	2021-03-23 00:46:50 -07:00
Chris Lu	4abb511db3	make a local copy of the in memory cached data	2021-03-22 22:33:07 -07:00
Chris Lu	3cbc40fa48	avoid creating multiple reader	2021-03-22 22:32:47 -07:00
Chris Lu	2f7c7afdec	refactor: remove unused parameter	2021-03-22 22:13:19 -07:00
Chris Lu	d48dd0c738	use []byte directly instead of bytes.Buffer	2021-03-22 22:12:57 -07:00
Chris Lu	8eb14967ef	shell: volume.check.disk adds optional slow mode ... fix https://github.com/chrislusf/seaweedfs/issues/1930	2021-03-22 21:01:43 -07:00
Chris Lu	288369cfc7	mount: release resources when Forget() is called ... address https://github.com/chrislusf/seaweedfs/issues/1929	2021-03-22 16:22:19 -07:00
Chris Lu	5d931eff27	avoid possible nil ... fix https://github.com/chrislusf/seaweedfs/issues/1928 The nil was because of `dn.Parent().UnlinkChildNode(dn.Id())` in topo.UnRegisterDataNode() function, when the dn leaves the cluster.	2021-03-22 13:24:07 -07:00
Chris Lu	f315eb2bb8	fix build error	2021-03-22 00:12:53 -07:00
Chris Lu	1dd5bc134c	2.35	2021-03-22 00:05:09 -07:00
Chris Lu	b465095db1	shell: add volume.check.disk to fix inconsistency for replicated volumes ... fix https://github.com/chrislusf/seaweedfs/issues/1923	2021-03-22 00:03:16 -07:00
Chris Lu	df461402cc	ensure entry attribute is not nill ... fix https://github.com/chrislusf/seaweedfs/issues/1926	2021-03-21 22:27:30 -07:00
Chris Lu	102a951377	refactor, split into 2 files	2021-03-21 13:05:53 -07:00
Chris Lu	f6e2566ee5	better logs for cookie mismatch	2021-03-20 12:02:57 -07:00
Chris Lu	27e24162f2	minor	2021-03-20 06:37:03 -07:00
Chris Lu	41cc6e661b	fs.mv ensures there are 2 arguments	2021-03-20 06:34:13 -07:00
Chris Lu	b2c1c209a5	refactor	2021-03-20 06:34:13 -07:00
Chris Lu	e9b1853d63	go fmt	2021-03-19 01:31:56 -07:00
Chris Lu	b1a86cf808	s3: copy object to itself ... fix https://github.com/chrislusf/seaweedfs/issues/1922	2021-03-19 01:31:49 -07:00
Konstantin Lebedev	bf94149920	add line	2021-03-17 23:41:34 +05:00
Konstantin Lebedev	c5705e7a35	dir + slash key does not exist ... https://github.com/chrislusf/seaweedfs/issues/1917	2021-03-17 23:40:42 +05:00
Konstantin Lebedev	eb54993a4e	Merge branch 'upstreamMaster' into check_chunkviews_mr ... # Conflicts: # weed/filer/filechunk_manifest.go # weed/filer/stream.go # weed/replication/repl_util/replication_util.go # weed/util/fasthttp_util.go	2021-03-16 15:29:49 +05:00
Chris Lu	9672f9e1b2	2.34	2021-03-16 03:01:15 -07:00
Chris Lu	69694a17be	reverting 7d57664c2d	2021-03-16 02:59:26 -07:00
Chris Lu	c00dd5e62e	report error if entry not found	2021-03-16 02:55:49 -07:00
Konstantin Lebedev	06da02739d	CheckAllChunkViews() for HEAD requests only	2021-03-16 14:15:17 +05:00
Chris Lu	91a3ac9731	2.33	2021-03-16 00:36:06 -07:00
Chris Lu	4b1ed227d1	revert fasthttp changes ... related to https://github.com/chrislusf/seaweedfs/issues/1907	2021-03-16 00:33:14 -07:00
Chris Lu	10164d0386	Merge pull request #1905 from wuh-fnst/s3ListCounterErr ... s3: count correction for the number of files in the dir	2021-03-15 20:54:58 -07:00
Chris Lu	22a07a12d2	[mount] [regression] filer.path option results in empty mount in 2.32 ... fix https://github.com/chrislusf/seaweedfs/issues/1906	2021-03-15 20:46:25 -07:00
wuh-fnst	050a7f9599	s3: count correction for the number of files in the dir	2021-03-16 09:22:03 +08:00
Konstantin Lebedev	3a3699867b	Status PartialContent for Content-Range response	2021-03-15 23:30:22 +05:00
Konstantin Lebedev	90510e3137	If WriteHeader is not called explicitly, the first call to Write ... will trigger an implicit WriteHeader(http.StatusOK). WriteHeader: Only one header may be written. Go does not currently!	2021-03-15 21:22:59 +05:00
Konstantin Lebedev	46b9f5cff4	add debug logging	2021-03-15 19:33:33 +05:00
Konstantin Lebedev	7194a5e7bf	avoid http error: superfluous response.WriteHeader ... https://github.com/chrislusf/seaweedfs/issues/1838	2021-03-15 18:52:59 +05:00
Chris Lu	9f00f95bfb	2.32	2021-03-14 21:32:00 -07:00
Chris Lu	a4cfffc264	shell: fix moving volume, volume server evacuate ... fix https://github.com/chrislusf/seaweedfs/issues/1534	2021-03-14 21:29:55 -07:00
Chris Lu	ca100568f7	fix parsing	2021-03-14 20:50:14 -07:00
Chris Lu	352ba23f83	revert previous change ... revert 29e62aba00	2021-03-14 20:49:56 -07:00
Chris Lu	29e62aba00	possible fix for volume balance ... address https://github.com/chrislusf/seaweedfs/issues/1534	2021-03-14 20:23:19 -07:00
Chris Lu	35d939a1d2	go fmt	2021-03-14 13:21:02 -07:00
Chris Lu	2d4c2db81d	filer: leveldb, rocksdb auto create store directory ... fix https://github.com/chrislusf/seaweedfs/issues/1901	2021-03-14 13:20:14 -07:00
Chris Lu	6d3a96eb56	filer: mysql2, postgres2 trigger actions on bucket creation and deletion ... fix https://github.com/chrislusf/seaweedfs/issues/1877	2021-03-13 22:07:39 -08:00
Chris Lu	cb423312a4	prevent nil volume message	2021-03-13 11:05:29 -08:00
Chris Lu	972327f966	prevent nil volume nm	2021-03-13 11:04:51 -08:00
Chris Lu	2a68ddb963	default to empty host to bind to tcp4 and tcp6	2021-03-12 15:32:51 -08:00
Chris Lu	5fbcaaf37a	fullpath() test is not needed	2021-03-12 09:45:25 -08:00
Chris Lu	7d57664c2d	mount: internals switch to filer.Entry instead of protobuf	2021-03-12 00:36:38 -08:00
wuh-fnst	4c1d945e46	make List correctly judge whether it is the last file	2021-03-12 11:22:26 +08:00
Chris Lu	cca66c7fbe	print out the url	2021-03-11 18:39:45 -08:00
Chris Lu	7f887c78d2	fix	2021-03-11 18:18:16 -08:00
Chris Lu	fcc52d1e4e	shell: add s3.clean.uploads to clean up stale multipart uploads ... fix https://github.com/chrislusf/seaweedfs/issues/1855	2021-03-11 18:05:24 -08:00
Chris Lu	40dc5ac904	mount: add a bit retry when connecting to filer during startup	2021-03-11 16:36:43 -08:00
Chris Lu	aba47fd9e8	reduce repeated calls to dir.FullPath()	2021-03-11 14:08:20 -08:00
Chris Lu	d084334ffd	Merge pull request #1883 from kmlebedev/passDelNonEmptyFolder ... s3 delete-objects pass OK for fail to delete non-empty folder	2021-03-11 11:01:44 -08:00
Chris Lu	0e35836cc1	Merge pull request #1890 from kmlebedev/ZeroDataSize ... error if read chunk zero data size	2021-03-11 10:51:48 -08:00
Konstantin Lebedev	58cdcc6d6e	error if urls not found	2021-03-11 23:34:36 +05:00
Chris Lu	79280e1513	escape special characters ... fix https://github.com/chrislusf/seaweedfs/issues/1884	2021-03-11 09:49:40 -08:00
Konstantin Lebedev	4db21012c1	error if read chunk zero data size	2021-03-11 22:38:59 +05:00
wuheng	828fbf3fb7	s3: "isLast" returns true when the file does not exist	2021-03-11 15:20:50 +08:00
Chris Lu	5d446673a6	fix error printing	2021-03-10 17:13:04 -08:00
Chris Lu	ca546f47e1	s3: escape object key if containing special characters ... fix https://github.com/chrislusf/seaweedfs/issues/1884	2021-03-10 13:19:28 -08:00
Chris Lu	105fcd7bfc	Merge pull request #1872 from kmlebedev/grpc_auth ... TLS allowed CommonNames	2021-03-10 11:17:49 -08:00
Konstantin Lebedev	bb3af2d70b	no directoriesWithDeletion	2021-03-10 19:52:41 +05:00
Konstantin Lebedev	15b018da34	s3 delete-objects pass OK for fail to delete non-empty folder	2021-03-10 19:41:35 +05:00
Konstantin Lebedev	348e21a08c	add comments	2021-03-10 14:42:39 +05:00
Konstantin Lebedev	831953c55c	allowed wildcard domain	2021-03-10 14:02:13 +05:00
Konstantin Lebedev	4bf93d6e63	comma-separated	2021-03-10 12:43:13 +05:00
Konstantin Lebedev	0e02f7e258	comma-separated SSL certificate common names	2021-03-10 12:42:44 +05:00
Chris Lu	737bde5ab7	refactoring, remove duplicated code	2021-03-09 23:23:01 -08:00
Chris Lu	5ba4b479f8	properly lock file.entry object ... fix https://github.com/chrislusf/seaweedfs/issues/1882	2021-03-09 23:08:38 -08:00
Chris Lu	5be83dd4a9	add comments	2021-03-09 14:13:48 -08:00
Chris Lu	9d8ca513d9	refactor	2021-03-09 13:21:26 -08:00
Chris Lu	f5b5e4da2d	2.31	2021-03-09 12:52:16 -08:00
Chris Lu	828f6e9f4d	volume: auto add missing vif files ... fix https://github.com/chrislusf/seaweedfs/issues/1878	2021-03-09 12:09:32 -08:00
Chris Lu	387c6f4218	Revert "delete the folder object in multi_object_delete" ... This reverts commit 2e89c8c9ae.	2021-03-09 10:07:27 -08:00
qieqieplus	5b16820924	fix: collection stats won't update if all volumes expired at same iteration	2021-03-09 18:49:45 +08:00
Konstantin Lebedev	190fada1ef	TLS allowed commonNames	2021-03-08 21:39:44 +05:00
Konstantin Lebedev	c6d3735605	permitCommonNames ... https://github.com/chrislusf/seaweedfs/issues/1841 https://jbrandhorst.com/post/grpc-auth/	2021-03-08 13:16:17 +05:00
Chris Lu	d888e9a90f	2.30	2021-03-07 15:01:19 -08:00
Chris Lu	60686a0bb7	make tcp optional	2021-03-07 14:45:36 -08:00
Chris Lu	3b2a9c98ef	fix nil	2021-03-07 11:45:35 -08:00
Chris Lu	df72dc206d	stats collect chunk upload retry count	2021-03-07 11:26:15 -08:00
Chris Lu	726edab054	avoid nil when closing an index ... fix https://github.com/chrislusf/seaweedfs/issues/1870	2021-03-07 11:03:09 -08:00
Chris Lu	bdfed16d42	avoid nil exception ... fix https://github.com/chrislusf/seaweedfs/issues/1869	2021-03-07 01:49:06 -08:00
Chris Lu	1bd880dcdb	adds tcp writes benchmark	2021-03-06 14:26:27 -08:00
Chris Lu	9d402ebe9f	refactoring	2021-03-06 14:26:27 -08:00
Chris Lu	e9eddfb8a2	remove wrong writes	2021-03-06 14:26:27 -08:00
Chris Lu	8363be8548	trim out trailing return character	2021-03-06 14:26:27 -08:00
Chris Lu	38fc200e56	CRCWriter consistent with CRC	2021-03-06 14:26:27 -08:00
Chris Lu	1444e9d275	migrated multi host connection pool from godropbox package ... removing unneeded dependencies, which involved etcd versions.	2021-03-06 14:26:27 -08:00
Patrick Schmidt	7413d59750	Fix EC shard count logic ... This fixes the calculation of the amount of EC shards a node holds. Previously a global counter was increased, but also used inside the loop to apply disk usage deltas. This led to wrong absolute numbers. The fix is to apply only deltas of single EC shards per iteration.	2021-03-05 12:50:58 +01:00
Chris Lu	400de380f4	volume server: support tcp direct put/get/delete	2021-03-05 02:29:38 -08:00
Chris Lu	2e89c8c9ae	delete the folder object in multi_object_delete ... fix https://github.com/chrislusf/seaweedfs/issues/1857	2021-03-04 14:03:40 -08:00
Chris Lu	c5cb346a08	fix compilation error	2021-03-04 12:36:20 -08:00
Chris Lu	f5cc96f289	skip checking master peers if not starting master	2021-03-04 12:31:26 -08:00
Chris Lu	40ff30b83f	adjust help message	2021-03-03 02:12:46 -08:00
Chris Lu	c0842fe99f	add filer.meta.backup command	2021-03-03 02:02:29 -08:00
Chris Lu	0c5c51eb98	adjust help message	2021-03-02 23:07:39 -08:00
Chris Lu	bd727b7b53	go fmt	2021-03-02 20:59:56 -08:00
Chris Lu	e982b06bcd	support IPv6	2021-03-02 20:59:39 -08:00
Chris Lu	bcf32591b7	reduce possibility of nil entry	2021-03-02 13:33:56 -08:00
Chris Lu	be9c7c21ec	reduce possibility of nil file entry	2021-03-02 11:17:17 -08:00
Chris Lu	02e146f3e1	return err	2021-03-02 08:54:18 -08:00
Chris Lu	5511722420	s3: list permission is needed to list my buckets ... fix https://github.com/chrislusf/seaweedfs/issues/1837	2021-03-01 12:41:55 -08:00
Chris Lu	c46d16b2f3	skip network error when remote volumes start	2021-03-01 01:20:06 -08:00
Chris Lu	6e43e8ce17	add logs for remote file loading	2021-03-01 00:48:59 -08:00
Chris Lu	015d16f43f	add vif file versions in case loading superblock fails	2021-03-01 00:48:30 -08:00
Chris Lu	d680676d45	skip already loaded backends	2021-03-01 00:47:03 -08:00
Chris Lu	540441fd38	go fmt	2021-02-28 20:34:14 -08:00
Chris Lu	c9722dceb2	adjust log level	2021-02-28 20:26:55 -08:00
Chris Lu	1b1c018165	adjust text	2021-02-28 19:02:43 -08:00
Chris Lu	2cca07b44c	cloud tier to non-AWS s3 gateways	2021-02-28 18:59:09 -08:00
Chris Lu	f29ae0db76	2.29	2021-02-28 18:08:26 -08:00
bingoohuang	5cdff56731	set default env prefix to WEED_	2021-03-01 09:35:23 +08:00
Chris Lu	e52c94640e	filer.backup: added to replace filer.replicate	2021-02-28 16:22:27 -08:00
Chris Lu	014a31d11a	minor	2021-02-28 16:21:09 -08:00
Chris Lu	f2fcb77808	local file sink: create backup files with permission 0755	2021-02-28 16:20:47 -08:00
Chris Lu	984fdd6192	always use non bucket prefixing url	2021-02-28 16:20:13 -08:00
Chris Lu	9abb041763	filer source: support filerProxy mode	2021-02-28 16:19:47 -08:00
Chris Lu	678c54d705	data sink: add incremental mode	2021-02-28 16:19:03 -08:00
Chris Lu	4ff2c5c4c9	rename file	2021-02-28 16:14:21 -08:00
Chris Lu	3b76a51f5f	add help message	2021-02-27 12:15:49 -08:00
Chris Lu	70434df105	use "options"	2021-02-27 12:12:53 -08:00
Chris Lu	d3c31c69a7	avoid confusion: conf and config are too similar	2021-02-27 12:08:09 -08:00
Chris Lu	543c8daa6d	Merge pull request #1844 from kmlebedev/processRangeRequest	2021-02-26 09:18:22 -08:00
Patrick Schmidt	5f7b024891	Show the real disk usage in stats calls ... Currently the file size of only one volume location is taken into account in the stats. This commit multiplies the disk usages by the amount of nodes holding a replica of the volume. This will yield the expected amount of disk usage and matches the total size calculations from before.	2021-02-26 13:58:40 +01:00
Konstantin Lebedev	ee21c0042e	log error write entry	2021-02-26 14:18:01 +05:00
Chris Lu	7ba75e3d5a	filer: do not return no content for empty files ... fix https://github.com/chrislusf/seaweedfs/issues/1831 fix https://github.com/chrislusf/seaweedfs/issues/1830	2021-02-23 12:25:27 -08:00
Chris Lu	9edd964627	volume.tier.move: avoid repeated move for replicated volumes ... fix https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-784139348	2021-02-23 03:49:14 -08:00
Chris Lu	37f104f88f	2.28	2021-02-22 22:54:34 -08:00
Chris Lu	90cdf9dcac	avoid conflict with "weed scaffold -config=xxx"	2021-02-22 16:57:27 -08:00
Chris Lu	72b0d9d8c4	avoid unnecessary user home checking	2021-02-22 16:50:56 -08:00
Chris Lu	2270737344	volume: avoid fixed vacuum timeout for large volumes ... 1GB for 3 minutes, about 5.7MB/s	2021-02-22 12:52:37 -08:00
Chris Lu	44bdfb2d15	filer: avoid encryption and compression at the same time ... fix https://github.com/chrislusf/seaweedfs/issues/1828	2021-02-22 12:22:49 -08:00
Chris Lu	62191b08ea	disk type support custom tags	2021-02-22 02:03:12 -08:00
Chris Lu	30b30b8fe0	volume.tier.move: passing non-empty disk type	2021-02-22 01:59:03 -08:00
Chris Lu	5da63e045e	avoid moving to another server with the same volume id	2021-02-22 01:44:18 -08:00
Chris Lu	6a4546d2c0	shell: add volume.tier.move	2021-02-22 01:30:07 -08:00
Chris Lu	1c233ad986	refactoring	2021-02-22 00:28:42 -08:00
Chris Lu	151c281f36	2.27	2021-02-21 19:29:27 -08:00
Chris Lu	258e93bc86	Merge pull request #1823 from bingoohuang/master ... feature: support command line arguments in a configuration file	2021-02-20 22:35:18 -08:00
Chris Lu	03c643aa6b	fix test ... fileSize need to be divided by 8	2021-02-20 21:19:21 -08:00
Chris Lu	7635f6b9fa	disk file avoid file.Stat()	2021-02-20 20:06:06 -08:00
bingoo	1af6c96cd0	fix log help	2021-02-21 10:46:01 +08:00
Chris Lu	a2383b3b12	Revert "Revert "avoid file.Stat()"" ... This reverts commit 7ef4c24f18.	2021-02-20 12:42:09 -08:00
Chris Lu	7ef4c24f18	Revert "avoid file.Stat()" ... This reverts commit 98c93ca465.	2021-02-20 12:39:33 -08:00
Chris Lu	1ad3200094	skip seek() when index file writes	2021-02-20 12:39:25 -08:00
Chris Lu	98c93ca465	avoid file.Stat() ... avoid one Syscall, but did not help on performance though	2021-02-20 09:44:17 -08:00
bingoohuang	ee082ae402	revert comment temporarily	2021-02-20 16:59:38 +08:00
bingoohuang	ee7cdf3668	revert ErrXyz to ErrorXyz temporarily.	2021-02-20 16:58:08 +08:00
bingoohuang	50df484d86	support command line arguments in a configuration file so we can weed master -config=master.conf, weed volume -config=volume.conf and etc.	2021-02-20 16:52:57 +08:00
bingoohuang	f69356f589	Merge pull request #3 from chrislusf/master	2021-02-20 16:45:02 +08:00
Chris Lu	565f7a6e72	Update data_node.go	2021-02-19 14:22:36 -08:00
Chris Lu	a37473ae60	add back volume ids ... address https://github.com/chrislusf/seaweedfs/issues/1792#issuecomment-782339576	2021-02-19 14:22:12 -08:00
Chris Lu	a1210d1e8d	Revert "volume: avoid file.stat(), file.seek() if possible during writes" ... This reverts commit c78409a598.	2021-02-19 03:56:27 -08:00
Chris Lu	b961cd6208	add WIP message	2021-02-19 03:39:19 -08:00
Chris Lu	f465d63b5d	grpc should fail when heart beating to master ... fix https://github.com/chrislusf/seaweedfs/issues/1820	2021-02-19 03:37:29 -08:00
Chris Lu	c576ad04ac	fix volume server display for volumes	2021-02-19 01:38:56 -08:00
bingoohuang	7ffe736d20	renaming to ErrXyz	2021-02-19 15:59:55 +08:00
Chris Lu	c78409a598	volume: avoid file.stat(), file.seek() if possible during writes	2021-02-18 23:22:54 -08:00
Chris Lu	29575dadc5	remove extra files	2021-02-18 19:14:58 -08:00
Chris Lu	73958e357d	add descriptive error if no free volumes	2021-02-18 19:10:20 -08:00
Chris Lu	776f497469	filer: fs.configure should try to read from entry.content also ... related to https://github.com/chrislusf/seaweedfs/issues/1792	2021-02-18 17:07:02 -08:00
Chris Lu	e1992c83de	webdav adds disktype	2021-02-18 15:45:44 -08:00
Chris Lu	c2ad6f1047	webdav add replication setting ... fix https://github.com/chrislusf/seaweedfs/issues/1817	2021-02-18 12:15:09 -08:00
Chris Lu	816ef0ed30	add more help message ... fix https://github.com/chrislusf/seaweedfs/issues/1816	2021-02-18 10:34:49 -08:00
bingoohuang	352ac2f271	Merge remote-tracking branch 'origin/master'	2021-02-18 14:05:51 +08:00
bingoohuang	eab6e31d34	use backticks instead of double quotes to avoid escaped additionally in regex	2021-02-18 14:05:28 +08:00
Chris Lu	3575d41009	go fmt	2021-02-17 20:57:08 -08:00
Chris Lu	6daa932f5c	refactoring to get master function, instead of passing master values directly ... this will enable retrying later	2021-02-17 20:55:55 -08:00
Chris Lu	dd9f3a0104	add sleep between upload retries	2021-02-17 20:55:11 -08:00
Chris Lu	aa17311063	s3: set filename when uploading file chunks	2021-02-17 20:54:53 -08:00
Chris Lu	3f8b0da677	filer: do not print password on error ... fix https://github.com/chrislusf/seaweedfs/issues/1809	2021-02-17 02:13:52 -08:00
Chris Lu	cd866664a8	skip JWT if fileId is empty ... related to https://github.com/chrislusf/seaweedfs/issues/1808	2021-02-16 15:39:12 -08:00
Chris Lu	632c94d438	Merge branch 'extend_to_disk_type'	2021-02-16 11:06:13 -08:00
Chris Lu	0611233f16	adjust printing	2021-02-16 10:55:30 -08:00
Chris Lu	68775d29e3	fix tests	2021-02-16 10:51:03 -08:00
Chris Lu	7403cd43c5	only use "" for hdd to avoid two values for the same thing	2021-02-16 10:50:42 -08:00
Chris Lu	b314d78e97	fix print	2021-02-16 10:48:28 -08:00
Chris Lu	53ca7e66ef	avoid dead lock	2021-02-16 10:48:16 -08:00
Chris Lu	38efc6f572	simplify	2021-02-16 10:47:32 -08:00
Chris Lu	ad70107eb4	ec balance among nodes with HDD	2021-02-16 06:22:50 -08:00
Chris Lu	3097b9a9b7	fix existence checking	2021-02-16 05:59:43 -08:00
Chris Lu	ebf320ec22	avoid divided by zero	2021-02-16 05:59:24 -08:00
Chris Lu	a0c6db361c	avoid nil	2021-02-16 05:33:38 -08:00
Chris Lu	36f95e50a9	avoid possible nil disk info	2021-02-16 05:13:48 -08:00
Chris Lu	43101ccea0	move to the empty nodes first	2021-02-16 04:27:16 -08:00
Chris Lu	38bbef7ec1	avoid nil	2021-02-16 04:16:46 -08:00
Chris Lu	cb9cc29518	volume.list display; fix updating maxVolumeCount for disk	2021-02-16 03:55:24 -08:00
Chris Lu	3fe628f04e	use hdd instead of empty string	2021-02-16 03:03:00 -08:00