924 Commits

Author	SHA1	Message	Date
Chris Lu	366f522a2d	add redis3	2021-10-04 01:01:31 -07:00
Chris Lu	b297849147	typo	2021-10-01 23:24:54 -07:00
Chris Lu	af207bbaf0	retry both assign volume and uploading data ... fix https://github.com/chrislusf/seaweedfs/issues/2351	2021-10-01 23:23:39 -07:00
Chris Lu	4a1d4d7462	s3: default to allow empty folder for better performance	2021-09-26 22:34:14 -07:00
Chris Lu	a814f3f0a8	adjust metadata tail output	2021-09-25 01:04:51 -07:00
Chris Lu	52fe86df45	use default 10000 for grpc port	2021-09-20 14:05:59 -07:00
Chris Lu	84fdda85e7	go fmt	2021-09-19 12:06:15 -07:00
Chris Lu	94a01fcfcb	filer.remote.gateway: add options to include or exclude new bucket names to mirror	2021-09-15 23:04:16 -07:00
Chris Lu	b5f4910412	adjust help messages	2021-09-15 22:53:10 -07:00
Chris Lu	f0907eb83c	add command filre.remote.gateway	2021-09-15 22:48:04 -07:00
Chris Lu	63da4bbb54	separate filer.remote.gateway command to avoid confusion	2021-09-15 22:47:17 -07:00
Chris Lu	2789d10342	go fmt	2021-09-14 10:37:06 -07:00
Chris Lu	e5fc35ed0c	change server address from string to a type	2021-09-12 22:47:52 -07:00
Chris Lu	2c9d4c8f43	custom grpc port: filer	2021-09-12 02:28:37 -07:00
Chris Lu	e690a2be16	custom grpc port: volume server	2021-09-12 02:25:15 -07:00
Chris Lu	232ad2fe65	custom grpc port: master follower	2021-09-12 02:25:00 -07:00
Chris Lu	bd1f800efe	custom grpc port: master	2021-09-12 02:19:10 -07:00
Chris Lu	2b8ab1534a	mysql table DDL: minor, not necessary, but a little more strict ... the directory is already hashed into dirhash. So this change is not strictly required.	2021-09-08 23:55:18 -07:00
Chris Lu	574485ec69	better IP v6 support	2021-09-07 19:29:42 -07:00
Chris Lu	0128239c0f	handle ipv6 addresses	2021-09-07 16:43:54 -07:00
Chris Lu	6923af7280	refactoring	2021-09-06 16:20:49 -07:00
Chris Lu	64f6532fbe	update help message	2021-09-06 16:16:22 -07:00
Chris Lu	308d4b2dae	filer.remote.sync: createBucketWithRandomSuffix defaults to true	2021-09-06 15:14:26 -07:00
Chris Lu	c218ef20c7	filer.remote.sync: automatically detect the primary remote storage	2021-09-06 15:10:55 -07:00
Chris Lu	1702ce5395	remove unused code	2021-09-06 12:01:44 -07:00
Chris Lu	60573fd3e2	option to map remote bucket to trimmed bucket name	2021-09-05 11:55:52 -07:00
Chris Lu	bdefdee4e6	filer.remote.sync: add option to add randomized suffix to buckets to avoid conflicts	2021-09-04 22:46:28 -07:00
Chris Lu	98c68d7a79	filer.remote.sync fix upload logic	2021-09-04 18:46:28 -07:00
Chris Lu	16a342eb8a	re-ordering	2021-09-04 13:46:44 -07:00
Chris Lu	796b2eb929	remember sync time	2021-09-04 13:46:22 -07:00
Chris Lu	99d4594ee8	update remote storage mount mapping	2021-09-04 05:22:34 -07:00
Chris Lu	13cc2ef723	when creating bucket, use remote storage client for default storage	2021-09-04 04:54:55 -07:00
Chris Lu	4b28c5f6c3	filer.remote.sync: split into buckets mode and single directory mode	2021-09-04 04:35:46 -07:00
Chris Lu	03a31587ce	go fmt	2021-09-03 20:42:28 -07:00
Chris Lu	7f2a8246a9	minor	2021-09-02 12:32:59 -07:00
Chris Lu	9a73b0e3c9	refactor	2021-09-02 07:07:16 -07:00
Chris Lu	e281f0fa82	refactor	2021-09-02 06:53:21 -07:00
Chris Lu	7ce97b59d8	go fmt	2021-09-01 02:45:42 -07:00
Chris Lu	3bd48c4f29	filer.remote.sync: exit when directory is unmounted ... this will not propagate the deletions back to the cloud	2021-09-01 01:29:22 -07:00
Chris Lu	8e125339d5	line wrap	2021-08-31 23:30:28 -07:00
Chris Lu	68ea99190f	add additionalPathPrefixes	2021-08-31 23:26:26 -07:00
Rain Li	e5f9ff983d	Merge branch 'master' into filerstore-tikv	2021-08-30 15:52:54 +08:00
Chris Lu	6deee4c0b9	import hdfs	2021-08-29 18:49:43 -07:00
Chris Lu	a31f2907f0	cloud drive: filer.remote.sync supports remove folder	2021-08-29 18:46:28 -07:00
Chris Lu	05d2774117	refactoring	2021-08-28 22:49:38 -07:00
Chris Lu	a9b41900d8	handle cases when mounted directory does not exist	2021-08-28 01:56:56 -07:00
Chris Lu	05a648bb96	refactor: separating out remote.proto	2021-08-26 15:18:34 -07:00
yulai.li	c1dc5ab4ac	Add deleterange_concurrency to filer configuration file	2021-08-26 18:25:08 +08:00
yulai.li	de8ef28460	Update filer.toml scaffold to support tikv store	2021-08-26 16:26:27 +08:00
yulai.li	546efeba8f	Fix build bug	2021-08-26 16:20:35 +08:00
Chris Lu	6bab20d862	cat entry content	2021-08-24 02:38:32 -07:00
Chris Lu	e9ebe24f2e	cloud drive: add support for Azure	2021-08-24 01:18:30 -07:00
Chris Lu	258063de26	cloud drive: add google cloud storage	2021-08-23 00:29:27 -07:00
Chris Lu	05fc7db755	filer.copy: large files pack file ids into chunks	2021-08-18 23:46:54 -07:00
Chris Lu	72eb6d5b9d	ensure no writes to remote storage if content is not changed	2021-08-15 20:23:41 -07:00
Chris Lu	5a7c40510f	format output	2021-08-15 20:07:13 -07:00
Chris Lu	5d5a21ba2d	adjust log format	2021-08-15 19:46:45 -07:00
Chris Lu	c45c12aedc	metadata updates	2021-08-15 19:27:30 -07:00
Chris Lu	49b5e47bd1	retry forever with filer.remote.sync, and some refactoring	2021-08-15 12:38:26 -07:00
Chris Lu	c34747c79d	rename, fix wrong logic.	2021-08-14 21:46:34 -07:00
Chris Lu	0db2517994	go fmt	2021-08-14 02:55:44 -07:00
Chris Lu	7937db52e1	Filer locationPrefix configure does not exec replication #2257 ... fix https://github.com/chrislusf/seaweedfs/issues/2257	2021-08-14 02:54:13 -07:00
Chris Lu	5a0f92423e	use grpc and jwt	2021-08-12 21:40:33 -07:00
Chris Lu	6238644c35	remove gateway command	2021-08-12 20:52:04 -07:00
Chris Lu	d1d1fc772c	move some volume lookup operations to grpc ... jwt related lookup will come in next commit	2021-08-12 20:33:00 -07:00
Chris Lu	5571f4f70a	master: add master.follower to handle read file id lookup requests	2021-08-12 18:10:59 -07:00
Chris Lu	5469019852	adjust data type	2021-08-12 17:54:34 -07:00
Chris Lu	713c035a6e	shell: remote.cache remote.uncache	2021-08-09 14:35:18 -07:00
Chris Lu	734c980040	volume: support concurrent download data size limit	2021-08-08 23:25:16 -07:00
Chris Lu	c5f38c365d	go fmt	2021-08-08 22:30:36 -07:00
Chris Lu	df85f7a1eb	adjust help message	2021-08-08 22:30:12 -07:00
Chris Lu	7412ccdf88	write back remote entry to local entry after uploading to remote	2021-08-08 17:55:03 -07:00
Chris Lu	dcf614a8c3	skip if the remote entry update is because of internal managerial operations	2021-08-08 15:58:10 -07:00
Chris Lu	13e45e1605	filer.remote.sync can work now	2021-08-08 01:21:42 -07:00
Chris Lu	6b743dbbf9	refactor client subscribe metadata	2021-08-04 16:25:46 -07:00
Chris Lu	d84c311699	refactoring	2021-08-04 12:30:18 -07:00
Chris Lu	d2ddf1dbdb	typo	2021-08-03 09:32:31 -07:00
Chris Lu	3afbf04007	add TLS grpc support for filer meta clients	2021-08-03 01:30:35 -07:00
Chris Lu	fe60c6ef9a	minor changes	2021-08-02 11:49:40 -07:00
byunghwa.yun	bdc7730fdb	Add autocomplete	2021-08-03 02:56:52 +09:00
Chris Lu	3bb640b786	add -force option following #2228	2021-08-01 20:03:05 -07:00
Chris Lu	58bc3ecf47	add default quietFor value	2021-08-01 15:36:06 -07:00
Chris Lu	2ca1839d77	shell: add volume.deleteEmpty command	2021-08-01 15:33:45 -07:00
Chris Lu	1c7e404abe	remove buckets folder option ... the related code still works for old deployments	2021-08-01 12:23:16 -07:00
Chris Lu	6ba65c3382	customizable debug port	2021-07-31 09:18:41 -07:00
Chris Lu	1ff8285d82	debug from any server	2021-07-31 02:13:21 -07:00
Chris Lu	0d6f45cb46	add debug option to generate full goroutine stack dump	2021-07-31 02:00:01 -07:00
Chris Lu	ac28611817	snowflake sequencer need an unique id ... fix https://github.com/chrislusf/seaweedfs/issues/2213	2021-07-23 20:54:03 -07:00
Chris Lu	5c14da0f1e	filer.copy: fail early if assign request failed ... fix https://github.com/chrislusf/seaweedfs/issues/2216	2021-07-23 20:01:43 -07:00
Chris Lu	4cc2165061	weed -h subcommand list is not sorted, makes discovery difficult ... fix https://github.com/chrislusf/seaweedfs/issues/2214	2021-07-23 18:44:53 -07:00
Chris Lu	10fc478557	scaffold -config= should output to stdout ... fix https://github.com/chrislusf/seaweedfs/issues/2212	2021-07-23 18:41:25 -07:00
byunghwa.yun	e0c7708b03	Fix error handling	2021-07-21 22:48:04 +09:00
Chris Lu	49c66e88a0	volume: change all writes to fsync during graceful stopping ... fix https://github.com/chrislusf/seaweedfs/issues/2193	2021-07-13 01:29:57 -07:00
bingoohuang	5dbbe19c8b	extract embed toml example to separate files	2021-07-05 11:16:49 +08:00
Chris Lu	5bcc77b46c	volume: default readMode to proxy	2021-07-03 15:55:56 -07:00
Chris Lu	2cb8b31ea7	Merge pull request #2179 from nivekuil/tokenpolicy ... Cassandra: Use TokenAwareHostPolicy with fallback to localDC by default	2021-07-02 13:51:53 -07:00
nivekuil	8425705643	Cassandra: Use TokenAwareHostPolicy by default with fallback ... See https://pkg.go.dev/github.com/gocql/gocql#hdr-Data_center_awareness_and_query_routing	2021-07-02 13:50:01 -07:00
Chris Lu	d8bda0b229	locate the weed binary	2021-07-02 13:33:17 -07:00
danielflira	586e066897	fix weed fuse parameters parsing	2021-07-02 16:52:52 -03:00
Chris Lu	b624090398	go fmt	2021-07-01 01:21:14 -07:00
danielflira	a66112c9d2	configurable fusermount path	2021-07-01 00:07:54 -03:00
Kevin Liu	253b4ed898	remote -> proxy in server to match volume flags ... noticed this was missed in https://github.com/chrislusf/seaweedfs/pull/2168	2021-06-30 13:05:45 -07:00
zhangsong	20d33ae025	add proxy mode to read non-local volumes	2021-06-30 18:33:18 +08:00
zhangsong	7566782c2e	add proxy mode to read non-local volumes	2021-06-30 17:28:37 +08:00
thephoenixofthevoid	8d70ba2eaa	Fix: Loosing environment variables at StartProcess	2021-06-26 15:20:27 +03:00
Chris Lu	c2e0a75c1f	adjust logs	2021-06-24 12:46:00 -07:00
danielflira	f1d207a0fe	start weed with mount in background	2021-06-20 02:48:46 -03:00
Chris Lu	1e76fc994a	filer.copy: zero fileSize for directories	2021-06-13 07:09:06 -07:00
Daniel Nagy	0e3adde47f	Return correct exitcode when wheed upload fails	2021-06-07 23:38:14 +02:00
Chris Lu	e00443a940	mount: adjust starting order ... avoid possible nil wfs.Server	2021-06-06 20:22:42 -07:00
Patrick Schmidt	77100754e6	Return artificial . and .. directories	2021-06-02 21:28:02 +02:00
Chris Lu	2d7b4e5bb6	filer.backup: escape colon from path on windows ... fix https://github.com/chrislusf/seaweedfs/issues/2084	2021-05-29 06:45:27 -07:00
Chris Lu	ae185b997f	Merge pull request #2099 from danielflira/mount-helper ... fix parameter multiple values	2021-05-28 21:57:15 -07:00
danielflira	849f36c1ac	fix parameter multiple values	2021-05-29 00:37:25 -03:00
Eugeniy Kozlov	6e8bd16819	freespace params fix	2021-05-28 14:19:24 +03:00
Chris Lu	4d55132c7d	scaffold add comments	2021-05-27 14:40:25 -07:00
danielflira	84488ebb33	replace filer.remote and parts[1]	2021-05-26 12:07:36 -03:00
danielflira	877c192af2	create fuse subcommand to use weed with mount	2021-05-25 23:32:35 -03:00
Chris Lu	dce1f02c9e	filer.backup: backup to local directory optionally is incremental ... fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084	2021-05-25 17:19:20 -07:00
Chris Lu	064269bb57	filer: Support sqlite as filer meta store	2021-05-23 23:58:28 -07:00
Chris Lu	5d77840cff	adjust help message	2021-05-21 01:38:57 -07:00
Chris Lu	dc1309f084	FUSE mount: support multiple filers ... fix https://github.com/chrislusf/seaweedfs/issues/2015 fix https://github.com/chrislusf/seaweedfs/issues/1531	2021-05-21 01:28:00 -07:00
Chris Lu	b430d1b6ee	filer.copy: "check.size" before copying files ... fix https://github.com/chrislusf/seaweedfs/issues/2067	2021-05-12 21:45:39 -07:00
Chris Lu	8f8738867f	add retry to assign volume ... fix https://github.com/chrislusf/seaweedfs/issues/2056	2021-05-07 07:29:26 -07:00
Chris Lu	007401f3a0	remove duplicated code	2021-05-07 07:14:24 -07:00
Chris Lu	55a8f57381	go fmt	2021-05-06 03:37:51 -07:00
Chris Lu	ac71117ee6	revert PR #1903 avoid http error: superfluous response.WriteHeader	2021-05-05 15:11:39 -07:00
bingoohuang	cf552417a7	minFreeSpace refactored	2021-04-27 10:37:24 +08:00
bingoohuang	31f1cdeac2	minFreeSpace argument allows size like 10GiB	2021-04-26 18:48:34 +08:00
Chris Lu	cd7bf1a72a	filer.copy copy empty folders ... fix https://github.com/chrislusf/seaweedfs/issues/2016	2021-04-21 11:17:43 -07:00
liuxiaobo	c31c5e829c	fix path-specific filer store comment error	2021-04-20 10:08:58 +08:00
Chris Lu	54410ca955	cleaner way to set readonly	2021-04-18 10:02:02 -07:00
Chris Lu	c83ab91e2e	remove unused variable	2021-04-16 10:34:02 -07:00
Chris Lu	ff4c1d5965	adjust logging ... fix https://github.com/chrislusf/seaweedfs/issues/1999	2021-04-14 10:04:26 -07:00
Chris Lu	0df5b53ad8	adjust help message	2021-04-11 00:26:28 -07:00
Chris Lu	f62c153274	go fmt	2021-04-10 23:48:18 -07:00
Chris Lu	af313dff58	add gateway for easier POST and DELETE blobs	2021-04-10 23:47:47 -07:00
Philippe Pepiot	42a761ee20	Fix typo in weed filer long help	2021-04-07 22:47:23 +02:00
Chris Lu	c5b08bac1b	remove mac specific mount options	2021-04-07 00:54:13 -07:00
Konstantin Lebedev	f5f8eec8e2	fix get filerGrpcAddress	2021-04-06 13:53:56 +05:00
Konstantin Lebedev	011e6e90ee	Merge branch 'upstreamMaster' into iamapipr	2021-04-06 13:50:33 +05:00
Chris Lu	0f64f5b9c8	mount: add readOnly option ... fix https://github.com/chrislusf/seaweedfs/issues/1961	2021-04-04 21:40:58 -07:00
Chris Lu	8e404a1433	go fmt	2021-04-02 02:22:26 -07:00
Chris Lu	cc0df36a9e	smaller file chunks are faster	2021-04-01 02:21:59 -07:00
Chris Lu	1f984d2645	refactor buffer pool	2021-04-01 02:20:00 -07:00
Chris Lu	ac875976c0	filer, volume: add concurrent upload size limit to avoid OOM ... add some back pressure when writes are slow	2021-03-30 02:10:53 -07:00
Chris Lu	a1e18a1384	server add memory profiling	2021-03-30 02:10:53 -07:00
LazyDBA247-Anyvision	9385e2224d	fix postgres2 upsertQuery syntax also	2021-03-30 10:51:52 +03:00
Chris Lu	a5662c9e5e	adjust ordering	2021-03-30 00:26:57 -07:00
Chris Lu	bec74c3e6a	Merge pull request #1948 from LazyDBA247-Anyvision/master ... add enableUpsert=true	2021-03-30 00:25:01 -07:00