533 Commits

Author	SHA1	Message	Date
Chris Lu	faec9076a4	adjust parameter names	2019-10-21 22:57:01 -07:00
Chris Lu	d16c450682	fix unreachable code	2019-10-21 22:36:41 -07:00
joeslay	d53aee179b	Merge pull request #10 from chrislusf/master ... merge seaweed master	2019-10-14 16:03:40 +01:00
Chris Lu	939e4b57a8	go fmt	2019-10-09 00:03:18 -07:00
Chris Lu	f8d4b7d1c0	support basic json filtering and selection	2019-10-06 22:35:08 -07:00
Chris Lu	cf47f657af	scaffold for volume server query feature	2019-10-06 22:35:08 -07:00
joeslay	9d03aa6a38	Merge pull request #8 from chrislusf/master ... merge seaweed master	2019-10-01 17:55:31 +01:00
Lei Liu	3cc084269c	master api: return http 404 when volumeId not exist ... Signed-off-by: Lei Liu <lei01.liu@horizon.ai>	2019-09-29 14:23:39 +08:00
j.laycock	6fc6322c90	Change joeslay paths to chrislusf paths	2019-09-12 14:18:21 +01:00
joeslay	d8c34b032f	Merge pull request #6 from chrislusf/master ... merge seaweed master	2019-09-12 11:44:09 +01:00
Chris Lu	ae53f63680	filer: recursive deletion optionally ignoring any errors ... fix https://github.com/chrislusf/seaweedfs/issues/1062	2019-09-11 20:26:20 -07:00
Tom Maxwell	4a878c0006	Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve	2019-09-04 15:27:14 +01:00
joeslay	6ee65356e3	Merge pull request #3 from chrislusf/master ... merge seaweed master	2019-09-04 14:58:04 +01:00
j.laycock	cbd0a98fa1	Add InMemory to Volume Grow structure	2019-09-03 18:19:02 +01:00
Chris Lu	ee8bcb0c12	remove println	2019-09-03 09:22:56 -07:00
j.laycock	1f01eb78e8	Rename mem_map to mMap, remove some in_memory variables being passed around, added MemoryMapped member to volume struct	2019-09-03 17:00:59 +01:00
Tom Maxwell	d637d86d22	Changes to try and pass the URL parameters through - in memory flag not working still	2019-09-03 15:41:28 +01:00
joeslay	e91139348e	Merge pull request #2 from chrislusf/master ... merge seaweed master	2019-09-02 11:39:40 +01:00
j.laycock	595a1beff0	Swap imports to use joeslay	2019-09-02 11:28:40 +01:00
Chris Lu	60c9215a00	HCFS: avoid verbose file not found exception stack trace	2019-09-01 21:40:26 -07:00
Chris Lu	0dc43ecdca	skip writing content if not modified ... related to https://github.com/chrislusf/seaweedfs/issues/1036	2019-08-09 14:38:54 -07:00
Feng Ye	fc0498798b	Add etcd filestore	2019-08-01 10:16:45 +08:00
Chris Lu	ca2b81a944	fix test error	2019-07-31 02:09:13 -07:00
Chris Lu	19360ab10a	master: redirect clients to the new leader	2019-07-31 01:54:42 -07:00
Chris Lu	8afd8d35b3	master: followers can also lookup and redirect ... improve scalability	2019-07-28 03:58:13 -07:00
Chris Lu	898d943b25	refactoring	2019-07-17 23:43:48 -07:00
Chris Lu	0264a7f50a	set the max file key when ever possible	2019-07-17 23:23:01 -07:00
Chris Lu	aff911c00d	skip all azuresb	2019-07-17 01:24:20 -07:00
Chris Lu	0ea98dc6a2	disable azure due to api changes	2019-07-17 00:39:36 -07:00
Chris Lu	9e47c4749a	shell: add fs.mv ... fix https://github.com/chrislusf/seaweedfs/issues/954	2019-07-11 09:29:04 -07:00
Chris Lu	1d1b355f9e	filer: set file name when uploading to a directory ... fix https://github.com/chrislusf/seaweedfs/issues/1005	2019-07-11 04:20:03 -07:00
Chris Lu	476a7c269c	detect a filer path is a file or a directory ... fix https://github.com/chrislusf/seaweedfs/issues/1004	2019-07-10 12:11:18 -07:00
Chris Lu	15ecf4e3bf	filer: default to leveldb2	2019-06-30 00:44:57 -07:00
Chris Lu	6883f9e322	mark volume readonly before ec encoding	2019-06-26 23:02:22 -07:00
Chris Lu	359563e95a	refactoring	2019-06-25 09:49:27 -07:00
Chris Lu	c777102da9	go fmt	2019-06-23 15:30:16 -07:00
Chris Lu	d5560f2705	simplify metrics settings	2019-06-23 15:29:49 -07:00
Chris Lu	f16375621f	big refactoring	2019-06-23 03:08:27 -07:00
Chris Lu	6c01fb6d2d	go fmt	2019-06-23 01:57:51 -07:00
Chris Lu	8da5d5b094	filer.copy: use filer settings, avoid unnecessary command line options ... fix https://github.com/chrislusf/seaweedfs/issues/968	2019-06-23 01:57:35 -07:00
Chris Lu	d1cd8f8c5b	add metrics, refactoring	2019-06-22 22:53:52 -07:00
Chris Lu	3fa1f150d9	refactoring	2019-06-22 20:04:56 -07:00
Chris Lu	f88a8bda7b	ec deletion works	2019-06-21 01:14:10 -07:00
Chris Lu	613a2e8060	delete garbage only when successful	2019-06-20 23:46:00 -07:00
Chris Lu	c74dc2b306	go fmt	2019-06-20 00:55:52 -07:00
Chris Lu	e63317fb08	ec deletion code complete, not tested yet	2019-06-20 00:55:30 -07:00
Chris Lu	4cea8aefd0	add grpc VolumeEcBlobDelete	2019-06-20 00:17:11 -07:00
Chris Lu	856da7aae2	ec volume support deletes	2019-06-19 22:57:14 -07:00
Chris Lu	d8ed73926d	volume servers get metrics address and interval from the master	2019-06-17 14:51:47 -07:00
Chris Lu	ff6a6dd11e	refactoring	2019-06-15 21:46:55 -07:00
Chris Lu	8b43679ae3	refactoring	2019-06-15 12:21:44 -07:00
Chris Lu	5336008dcd	refactoring	2019-06-14 13:06:01 -07:00
Chris Lu	d67189921f	go fmt	2019-06-14 00:55:12 -07:00
Chris Lu	5f6c9825f8	volume server: adds basic metrics	2019-06-14 00:54:56 -07:00
Chris Lu	a11525fe4e	filer: adds basic metrics pushing to Prometheus gateway	2019-06-13 02:01:54 -07:00
Chris Lu	57092d69ac	go fmt	2019-06-10 21:33:32 -07:00
Chris Lu	e66bddd84c	customizable sleep intervals	2019-06-06 00:39:08 -07:00
Chris Lu	50aa769554	jwt for read access control	2019-06-06 00:29:02 -07:00
Chris Lu	d344e0a035	fix ec related bugs	2019-06-05 23:20:26 -07:00
Chris Lu	450f4733ce	report empty volume or ec shards	2019-06-05 13:32:33 -07:00
Chris Lu	7f6fb9ffd0	every 17 minutes instead of seconds ... TODO: make this configurable	2019-06-05 13:32:08 -07:00
Chris Lu	ede876cfdb	periodic scripts exeuction from leader master	2019-06-05 01:30:24 -07:00
Chris Lu	ca8a2bb534	go fmt	2019-06-04 22:04:10 -07:00
Chris Lu	2215e81be7	ui add ec shard statuses	2019-06-04 21:52:37 -07:00
Chris Lu	b05456fe07	able to purge extra ec shard copies	2019-06-03 20:25:02 -07:00
Chris Lu	11cffb3168	fix ec.rebuild bugs	2019-06-03 11:50:54 -07:00
Chris Lu	d85b41b904	fix ec.encode not finding the local ec shards	2019-06-03 10:38:21 -07:00
Chris Lu	7e80b2b882	fix multiple bugs	2019-06-03 02:26:31 -07:00
Chris Lu	55be09996d	fix volume balance bug	2019-06-03 00:13:31 -07:00
Chris Lu	133b772fb5	destroy ec volume if it is empty	2019-06-01 01:51:28 -07:00
Chris Lu	ba18314aab	ec shard delete also check ec volumes, in addition to volumes	2019-06-01 01:41:22 -07:00
Chris Lu	f53024d79d	delete ec collection	2019-05-30 10:40:22 -07:00
Chris Lu	1d111d6ce8	collection list normal and ec volumes	2019-05-30 09:27:23 -07:00
Chris Lu	e941d0c2f4	list ec collections also	2019-05-30 09:17:58 -07:00
Chris Lu	3f9ecee40f	working with reading remote intervals	2019-05-28 21:29:07 -07:00
Chris Lu	2858a954b3	read ec volume shards locations from master	2019-05-28 00:13:13 -07:00
Chris Lu	713596e781	caching ec shard locations	2019-05-27 22:54:58 -07:00
Chris Lu	217cde0a3b	refactoring	2019-05-27 21:40:51 -07:00
Chris Lu	03b9291e5d	volum server passes grpc option and master location to store	2019-05-27 21:22:23 -07:00
Chris Lu	ac1ba3b667	remove comments	2019-05-27 21:21:52 -07:00
Chris Lu	b4b407e403	add grpc ec shard read	2019-05-27 11:59:03 -07:00
Chris Lu	a4f3d82c57	convert needle id to ec intervals to read from	2019-05-27 01:29:46 -07:00
Chris Lu	92f9c7b1aa	adjust ec shard status on disconnect	2019-05-26 01:05:08 -07:00
Chris Lu	b58e25e588	broadcast vid->location map to master clients for ec chards	2019-05-26 00:49:15 -07:00
Chris Lu	af67d99ca4	incrementally update master ec shards state	2019-05-26 00:21:17 -07:00
Chris Lu	db94a41f9e	mount/unmount ec shards	2019-05-25 23:23:19 -07:00
Chris Lu	41e8ae61f8	generate, balance, delete copied shards, delete old volume	2019-05-25 14:02:06 -07:00
Chris Lu	6f4b09b6a4	pb shard info uses ShardBits instead one message for one shard	2019-05-24 13:28:44 -07:00
Chris Lu	24b8ad9a14	push the ec shard info to master	2019-05-23 23:47:49 -07:00
Chris Lu	4659d80035	prepare to register ec shard info in master	2019-05-23 00:42:28 -07:00
Chris Lu	17ac1290c0	volume: load ec shards during heartbeats to master	2019-05-21 22:41:20 -07:00
Chris Lu	fbbc74abb4	adds VolumeEcGenerateSlices, VolumeEcCopy	2019-05-20 00:53:17 -07:00
Chris Lu	698dea779c	refactoring	2019-05-19 21:37:49 -07:00
Chris Lu	017d0957c3	avoid switching master node if same ip and host name ... fix https://github.com/chrislusf/seaweedfs/issues/955	2019-05-11 18:42:46 -07:00
Chris Lu	8ffdd5dcdb	writel throttling moving volumes	2019-05-06 13:58:42 -07:00
Chris Lu	3ff92037cc	shell: add volume.balance command	2019-05-05 21:17:23 -07:00
Chris Lu	25941e0500	master: add jwt expires_after_seconds	2019-05-04 08:42:25 -07:00
Chris Lu	bd8af92b54	remove unused code	2019-05-04 08:20:22 -07:00
Chris Lu	5681208f2d	refactor: remove logger	2019-05-03 22:21:06 -07:00
Chris Lu	b335f81a4f	volume: add option to limit compaction speed	2019-05-03 17:22:39 -07:00
Chris Lu	f0f981e7c8	webdav: fix directory renaming	2019-05-03 14:12:51 -07:00
Chris Lu	bfbecd7253	webdav: fix directory creation	2019-05-03 13:13:11 -07:00
Chris Lu	9d3c945b00	Merge pull request #950 from chrislusf/webdav ... add WebDav	2019-05-03 01:25:10 -07:00
Chris Lu	7064b2ea48	webdav: can write now	2019-05-03 00:55:52 -07:00
Chris Lu	b30c14b631	webdav: can read now	2019-05-03 00:24:35 -07:00
Chris Lu	47f14775d7	scaffolding, but seems hard to satisfy mac, windows, and different os versions ... https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/	2019-05-02 14:22:10 -07:00
Wine93	32f93fb09e	style: go fmt for all	2019-04-30 03:22:19 +00:00
Chris Lu	108d0fb08d	adjust log level	2019-04-26 09:32:37 -07:00
Chris Lu	00b6f653fa	shell: add fs.cat	2019-04-21 15:43:43 -07:00
Chris Lu	440111a349	volume: support http status 304 for the same file id	2019-04-21 13:33:23 -07:00
Chris Lu	0302b9496c	volume: also check cookie for possible duplicates	2019-04-21 13:31:45 -07:00
Chris Lu	132921ad41	adjust log or error	2019-04-21 10:14:17 -07:00
Chris Lu	2ffe98443b	refactoring	2019-04-20 12:05:28 -07:00
Chris Lu	6fc1f53018	shell: add command volume.move	2019-04-20 11:35:23 -07:00
Chris Lu	3b3651dea3	volume: atomic copying file, adds version and stopOffset	2019-04-19 12:29:49 -07:00
Chris Lu	ac2727853f	fix needle map entry size	2019-04-19 00:39:34 -07:00
Chris Lu	e5506152c0	refactoring	2019-04-18 21:43:36 -07:00
Chris Lu	fa176fe80f	volume tailing chunks large files	2019-04-18 19:22:13 -07:00
Chris Lu	3dce1016cb	add volume tailer	2019-04-18 11:05:02 -07:00
Chris Lu	b142f9f1d5	go fmt	2019-04-18 00:19:18 -07:00
Chris Lu	b09e8dbb37	add VolumeStreamFollow, but not used yet	2019-04-18 00:18:29 -07:00
Chris Lu	13ad5c1966	refactoring	2019-04-17 22:04:49 -07:00
Chris Lu	3b9446606a	adjust log level	2019-04-16 23:31:44 -07:00
Chris Lu	338e6d60a5	refactor: prepare for snapshotting	2019-04-16 09:55:37 -07:00
Chris Lu	79c2cca9c1	better error message	2019-04-16 01:15:30 -07:00
Chris Lu	6e116b3d67	volume: reset leader if error happens	2019-04-15 09:09:46 -07:00
Chris Lu	e85048bcdc	http exhaust and close response body	2019-04-14 23:28:24 -07:00
Chris Lu	3e8a3a8fec	fix race detector found problems	2019-04-14 23:00:37 -07:00
Chris Lu	0e33272f56	small refactoring	2019-04-10 23:39:53 -07:00
Chris Lu	cb2061f86a	Merge pull request #925 from stlpmo-jn/checkUnhealthReplication ... the implement of repair the error replications of the volume	2019-04-10 23:37:12 -07:00
Chris Lu	2f76681d62	refactor: adjust grpc API name	2019-04-10 21:41:17 -07:00
stlpmo-jn	f2031884f0	fix bug : CI build failed	2019-04-11 09:53:31 +08:00
stlpmo-jn	c1a0403da2	repair the error replications of the volume	2019-04-10 19:41:55 +08:00
Chris Lu	000ee725fc	refactor Offset into a struct of bytes	2019-04-08 19:40:56 -07:00
Chris Lu	766396d249	weed master: atomic volume counting ... possible fix for https://github.com/chrislusf/seaweedfs/issues/913	2019-04-04 19:27:00 -07:00
Chris Lu	189c890715	weed replicate: replicate atomic rename to other systems	2019-03-31 08:10:47 -07:00
Chris Lu	78ac2bef3c	go fmt	2019-03-30 23:09:16 -07:00
Chris Lu	97406333a5	support atomic renaming for mysql/postgres filer store	2019-03-30 23:08:29 -07:00
Chris Lu	c7e7b6229f	detect mime type	2019-03-27 14:25:18 -07:00
Chris Lu	65757ae2fd	weed filer: set content-type consistent with filer store	2019-03-27 11:41:11 -07:00
Chris Lu	7a14cdc90c	refactoring, go fmt	2019-03-25 23:18:40 -07:00
Chris Lu	19728fe3f6	remove deprecated code	2019-03-25 23:12:14 -07:00
Chris Lu	df95ce0b6c	weed backup: efficient delta backup ... fix https://github.com/chrislusf/seaweedfs/issues/399	2019-03-25 23:01:53 -07:00
Chris Lu	c913583e2d	Merge branch 'master' into add_volume_follow	2019-03-25 21:19:34 -07:00
Chris Lu	a32797518b	weed master: redirect to leader for default admin UI ... fix https://github.com/chrislusf/seaweedfs/issues/898	2019-03-25 09:20:05 -07:00
Chris Lu	70815e9124	WIP	2019-03-25 09:16:12 -07:00
Chris Lu	95e0520182	weed volume: add grpc operation to relicate a volume to local	2019-03-23 11:33:34 -07:00
Chris Lu	a3490b600c	weed filer, weed master: add option to disable http	2019-03-21 16:00:46 -07:00
Chris Lu	531add52c2	weed master: skip proxied file read in http read only mode	2019-03-21 09:49:04 -07:00
Jonathan Amsterdam	e966033098	Merge branch 'master' of https://github.com/chrislusf/seaweedfs into gocdk	2019-03-20 16:02:38 -04:00