355 Commits

Author	SHA1	Message	Date
Chris Lu	f4abd01adf	filer: cache small file to filer store	2020-11-30 04:34:04 -08:00
Chris Lu	965413c21b	shell: add volume.vacuum command	2020-11-28 23:18:02 -08:00
Chris Lu	dc304342b2	fs.configure: configurable volume growth	2020-11-17 01:00:02 -08:00
Chris Lu	5f19e81dab	filer confi: support hierachical configuration	2020-11-16 16:50:12 -08:00
Chris Lu	ee2fa14dbe	filer conf: delete location specific configuration	2020-11-15 20:15:47 -08:00
Chris Lu	0ea5c087ce	go fmt	2020-11-15 16:59:28 -08:00
Chris Lu	95c0de285d	refactoring	2020-11-15 16:58:48 -08:00
Chris Lu	0a406f652e	load filer conf and match by prefix	2020-11-15 00:26:05 -08:00
Chris Lu	68043cfcac	add reference implementation to detect create/update/delete/rename events	2020-11-14 21:21:58 -08:00
Chris Lu	a4dfa00030	adjust filer conf	2020-11-12 00:39:59 -08:00
Chris Lu	2251a8e472	minor	2020-11-12 00:38:58 -08:00
Chris Lu	5a16f17e47	remove unused message type	2020-11-12 00:38:23 -08:00
Chris Lu	c32b6f4d9d	prepare for path based configuration	2020-11-11 23:46:45 -08:00
Konstantin Lebedev	1eec5c8d5d	gen pb	2020-11-12 04:10:06 +05:00
Konstantin Lebedev	dc26012a3b	initial	2020-11-11 15:03:47 +05:00
Konstantin Lebedev	918546bbdb	add WaitForReady to grpc DialOption	2020-10-29 08:29:04 +05:00
Chris Lu	f375b93aef	renaming	2020-10-25 15:32:43 -07:00
Chris Lu	6c92fa349d	filer: support default rack ... fix https://github.com/chrislusf/seaweedfs/issues/1546	2020-10-20 17:41:39 -07:00
Chris Lu	ace0ea3d28	s3: avoid duplicated bucket	2020-10-15 10:52:20 -07:00
Chris Lu	723ae11db4	refactoring in order to adjust volume server url later	2020-10-11 20:15:10 -07:00
Chris Lu	da4edf3651	master: check peers for existing leader before starting a leader election ... fix https://github.com/chrislusf/seaweedfs/issues/1509	2020-10-07 01:25:39 -07:00
Chris Lu	55cb68c09c	filer: replication follows master config if not specified	2020-09-30 09:15:55 -07:00
Chris Lu	1012df7bb5	switch hardlink id from int64 to bytes	2020-09-24 11:11:42 -07:00
Chris Lu	5e239afdfc	hardlink works now	2020-09-24 03:06:48 -07:00
Chris Lu	cb427d48fa	filer report metrics configuration	2020-09-17 06:46:51 -07:00
Chris Lu	c9202c4b3d	add storage backend to GetMasterConfigurationResponse	2020-09-16 01:12:22 -07:00
Chris Lu	200fe5c83e	go fmt	2020-09-13 21:26:30 -07:00
Chris Lu	5d6753fb98	shell: add volumeServer.leave command	2020-09-13 21:25:51 -07:00
Chris Lu	daf0a449f7	properly cancel context for streaming grpc	2020-09-09 12:07:15 -07:00
Chris Lu	387ab6796f	filer: cross cluster synchronization	2020-09-09 11:21:23 -07:00
Chris Lu	bba90ff3c8	read filer signature	2020-09-05 22:52:15 -07:00
Chris Lu	a41588279a	change log level 5 to 4	2020-08-30 20:12:04 -07:00
Chris Lu	ca658a97c5	add signatures to messages to avoid double processing	2020-08-28 23:48:48 -07:00
James Hartig	3ccfa4c6ad	Added VolumeMarkWritable and VolumeStatus grpc methods ... This is necessary for copy to mark as read-only and then restore the original state afterwards.	2020-08-19 11:42:56 -04:00
Chris Lu	208849702d	logs	2020-08-18 12:52:54 -07:00
Chris Lu	003d48da21	adjust logs	2020-08-15 19:55:28 -07:00
Chris Lu	eb493283dd	deletion error report	2020-08-14 00:23:01 -07:00
Chris Lu	edfa73782f	adjust log level	2020-08-14 00:22:21 -07:00
James Hartig	229f11c660	Added VolumeNeedleStatus volume server grpc method ... This is needed for the diffing tool to get the cookie for a needle	2020-07-22 15:02:21 -04:00
Chris Lu	2286d27730	rename	2020-07-15 14:19:15 -07:00
Chris Lu	0db4204c81	add is_file_chunks to filer.proto	2020-07-14 22:42:52 -07:00
Chris Lu	70d8a3a1d3	add SubscribeLocalMetadata without checking persisted meta logs	2020-07-05 15:50:07 -07:00
Chris Lu	353bea8ddb	refactoring	2020-07-03 22:25:35 -07:00
Chris Lu	37d5b3ba12	replication: pass isFromOtherCluster also to EventNotification ... EventNotification is consistent with message queue and metadata logs.	2020-07-01 08:06:20 -07:00
Chris Lu	31e23e9783	filer: support active<=>active filer replication	2020-06-30 22:53:57 -07:00
Chris Lu	de5ca9b258	remove fixJpgOrientation	2020-06-20 12:50:40 -07:00
Chris Lu	97239ce6f1	rename filechunk is_gzipped to is_compressed	2020-06-20 08:15:49 -07:00
Chris Lu	ca3516ac6d	adjust protoc	2020-06-20 08:00:25 -07:00
Chris Lu	d4235afe4d	support multiple locks	2020-05-25 18:39:53 -07:00
Chris Lu	1ea9bc66d9	avoid leaking grpc.NewClientStream	2020-05-17 08:56:33 -07:00
Chris Lu	dfccc3c263	able to read chan and write chan	2020-05-08 02:47:22 -07:00
Chris Lu	1e3e4b3072	add broker connects to filer	2020-05-05 02:05:28 -07:00
Chris Lu	9e72e9e4b8	able to subscribe any topic from any point of time	2020-04-29 17:40:08 -07:00
Chris Lu	bdc337a719	make lock/unlock optional	2020-04-23 03:50:05 -07:00
Chris Lu	ff0a7c1d18	clean up	2020-04-23 03:10:47 -07:00
Chris Lu	30ee4f3291	add exclusive lock library on shell	2020-04-23 02:31:04 -07:00
Chris Lu	77873b832b	add master side code for cluster wise exclusive lock	2020-04-23 01:55:44 -07:00
Chris Lu	aebe39a803	avoid repeated grpc connection creation ... fix https://github.com/chrislusf/seaweedfs/issues/1277	2020-04-22 18:26:24 -07:00
Chris Lu	9f9826e95a	refactor	2020-04-21 21:28:47 -07:00
Chris Lu	e24b25de78	async meta caching: can stream updates now	2020-04-21 21:16:13 -07:00
Chris Lu	5c348087dc	messaging: able to pub sub multiple partitions	2020-04-21 00:59:55 -07:00
Chris Lu	978da73cdb	revert this grpc related change ... maybe related to https://github.com/chrislusf/seaweedfs/issues/1277	2020-04-20 02:53:22 -07:00
Chris Lu	ce3cb25cfb	working for in memory single log buffer	2020-04-19 23:37:50 -07:00
Chris Lu	71ffb98475	broker: add profiling	2020-04-19 03:03:40 -07:00
Chris Lu	076c8bd3bc	filer master start up with default ip address instead of just localhost	2020-04-18 15:17:27 -07:00
Chris Lu	5af49c7aa5	adjust API	2020-04-18 11:45:57 -07:00
Chris Lu	3f3dba5a68	broker: append message logs	2020-04-17 02:29:38 -07:00
Chris Lu	bda82f61bc	filer: able to append to a file	2020-04-17 02:28:09 -07:00
Chris Lu	508f3490a0	update messaging proto	2020-04-16 03:29:57 -07:00
Chris Lu	ce4b369be2	scaffolding messaging	2020-04-16 02:21:23 -07:00
Chris Lu	7764e0465c	refactoring	2020-04-12 21:00:55 -07:00
Chris Lu	8e23dc078b	refactoring	2020-04-12 20:48:21 -07:00
Chris Lu	6f948e4887	remove configurable topics folder location	2020-04-12 13:07:59 -07:00
Chris Lu	e909b55633	refactoring	2020-04-11 12:39:46 -07:00
Chris Lu	ec2eb8bc48	add If-None-Match and If-Modified-Since ... fix https://github.com/chrislusf/seaweedfs/issues/1269	2020-04-08 08:12:00 -07:00
Chris Lu	5a5908407d	filer: support larger file size ... fix https://github.com/chrislusf/seaweedfs/issues/1257	2020-04-05 21:48:45 -07:00
Chris Lu	2d43f85577	watch entries with common path prefix	2020-04-05 12:51:21 -07:00
Chris Lu	dcc6b6a3ed	go fmt	2020-04-05 00:52:31 -07:00
Chris Lu	bf270d9e8c	filer: able to tail meta data changes	2020-04-05 00:51:16 -07:00
Chris Lu	50a5018b7f	writing meta logs is working	2020-03-30 01:19:33 -07:00
Chris Lu	a75d50bbb8	FUSE: add configurable in memory chunk cache size	2020-03-28 14:07:16 -07:00
Chris Lu	826bc0b7e3	FUSE: add chunk cache for recently accessed file chunks	2020-03-28 13:43:31 -07:00
Chris Lu	712720864d	error checking for remove	2020-03-26 02:21:30 -07:00
Chris Lu	dd5b582d05	go fmt	2020-03-26 00:09:01 -07:00
Chris Lu	6d87a0c5f5	batch delete orphan data by file keys	2020-03-25 02:21:15 -07:00
Chris Lu	7f1e3c843d	refactoring	2020-03-23 01:14:21 -07:00
Chris Lu	c07bcd5065	refactoring	2020-03-23 00:30:02 -07:00
Chris Lu	654a69ff52	refactoring	2020-03-23 00:06:24 -07:00
Chris Lu	c0f0fdb3ba	refactoring	2020-03-23 00:01:34 -07:00
Chris Lu	c4bea45099	S3 API: fix DeleteMultipleObjectsHandler ... fix https://github.com/chrislusf/seaweedfs/issues/1241	2020-03-20 14:17:31 -07:00
Chris Lu	bec6ec7db6	go fmt	2020-03-17 10:01:55 -07:00
Chris Lu	c3cb6fa1d7	volume: compaction can cause readonly volumes ... address https://github.com/chrislusf/seaweedfs/issues/1233	2020-03-17 09:43:57 -07:00
Chris Lu	2e3f6ad3a9	filer: remember content is gzipped or not	2020-03-08 21:39:33 -07:00
Chris Lu	afb20de14c	breaks dependency loop	2020-03-07 17:01:39 -08:00
Chris Lu	8645283a7b	fuse mount: avoid lookup nil entry ... fix https://github.com/chrislusf/seaweedfs/issues/1221	2020-03-07 16:51:46 -08:00
Chris Lu	1ae83c2938	go fmt	2020-03-07 06:12:57 -08:00
Chris Lu	13e215ee5c	filer: option to encrypt data on volume server	2020-03-06 00:49:47 -08:00
Chris Lu	f90c43635d	refactoring	2020-03-04 00:39:47 -08:00
Chris Lu	410bce3925	go fmt	2020-03-01 22:39:08 -08:00
Chris Lu	6a8484b4ae	master able to list all master clients by type	2020-03-01 22:13:47 -08:00
Chris Lu	0ca68a2a6d	WIP	2020-03-01 22:13:15 -08:00
Chris Lu	555413d9fc	weed queue starts	2020-02-27 00:07:13 -08:00
Chris Lu	892e726eb9	avoid reusing context object ... fix https://github.com/chrislusf/seaweedfs/issues/1182	2020-02-25 21:50:12 -08:00
Chris Lu	bd3254b53f	adjust logging	2020-02-25 17:24:08 -08:00
Chris Lu	0841bedb15	move filer assign volume grpc errror to response	2020-02-25 17:15:09 -08:00
Chris Lu	bc38b72a20	s3: implemented DeleteMultipleObjects	2020-02-25 14:38:36 -08:00
Chris Lu	6ab7368ef2	filer: dynamically create bucket under /buckets folder	2020-02-24 22:28:45 -08:00
Chris Lu	d8dec2323b	s3: move buckets folder configuration to filer	2020-02-24 14:34:14 -08:00
Chris Lu	e96f85ed8a	make pb files	2020-02-23 18:06:34 -08:00
Chris Lu	346a1cf0b9	Merge pull request #1207 from LazyDBA247-Anyvision/update-stats-and-status-volume-server ... Update stats and status volume server	2020-02-23 18:05:25 -08:00
Chris Lu	bbb163c5ad	add for https://github.com/chrislusf/seaweedfs/pull/1207	2020-02-23 18:04:31 -08:00
LazyDBA247-Anyvision	ec8619176a	volume server protobuf: add precentFree & percentUsage to the DiskStatus	2020-02-23 23:24:24 +02:00
Chris Lu	f55f49970f	move volume server stats to grpc	2020-02-21 21:45:03 -08:00
Chris Lu	0a57db7026	Merge pull request #1200 from chrislusf/add_s3_acl ... Add s3 ACL	2020-02-17 12:53:30 -08:00
Chris Lu	4cdde5f569	configuration stores the identity list	2020-02-17 12:31:59 -08:00
Chris Lu	9b6296e77a	volume: add grpc file read operation ... This is added more for performance benchmarking.	2020-02-14 00:37:32 -08:00
Chris Lu	9ed364f053	support acl	2020-02-09 14:30:02 -08:00
Chris Lu	40ae533fa3	shell: add volume.configure.replication to change replication for a volume ... fix https://github.com/chrislusf/seaweedfs/issues/1192	2020-02-02 15:37:23 -08:00
Chris Lu	c48fc8b4de	grpc send error via response instead of grpc error	2020-01-25 09:17:19 -08:00
Chris Lu	d4e75a0d18	filer: option to create file only if it is new, O_EXCL	2020-01-22 11:42:40 -08:00
Chris Lu	03ab0b1b9a	correct .vif file creation	2019-12-28 21:40:12 -08:00
Chris Lu	0cc8347a3a	go fmt	2019-12-28 21:13:10 -08:00
Chris Lu	37b64a50b4	ec: generate and copy .vif file	2019-12-28 12:44:59 -08:00
Chris Lu	c06f7eb48a	load volume info from .vif file, use superblock as a backup	2019-12-28 12:28:58 -08:00
Chris Lu	2000284435	rename volume tier info to volume info	2019-12-28 11:21:49 -08:00
Chris Lu	b3c83f9404	fix spelling	2019-12-28 11:16:10 -08:00
Chris Lu	48d28d3eb2	tier: support remote file attributes and remember the file extension	2019-12-25 21:37:24 -08:00
Chris Lu	d960b3474a	tier storage: support downloading the remote dat files	2019-12-25 09:53:13 -08:00
Chris Lu	09ca936c78	shell: add ec.decode command	2019-12-23 12:48:20 -08:00
Chris Lu	0fa1269bc7	filer: streaming file listing	2019-12-13 00:22:37 -08:00
Chris Lu	e426bd541e	grow volumes on volume servers with slots freed by cloud storage	2019-12-03 21:36:42 -08:00
Chris Lu	1becbce657	display remote volumes on volume server ui page	2019-12-02 23:23:54 -08:00
Chris Lu	70648d35ad	go fmt	2019-12-02 20:49:58 -08:00
Chris Lu	126fb492d2	fix test	2019-12-02 15:52:33 -08:00
Chris Lu	ec8de250e2	tiered storage: can copy to s3, read from s3 ... master not aware tiered volume yet, file assigning is not working yet	2019-12-02 15:08:31 -08:00
Chris Lu	228231f3d7	minor	2019-12-02 15:08:31 -08:00
Chris Lu	0da7b894cc	pass backend config from master to volume servers	2019-11-29 01:05:09 -08:00
Chris Lu	586798ecc0	go fmt	2019-11-27 12:34:57 -08:00
Chris Lu	0f9ba84274	s3	2019-11-27 03:09:45 -08:00
Chris Lu	471902cd51	change parameter name	2019-11-10 12:05:45 -08:00
zhangsong	61fa485700	add volume number param in assign operation	2019-11-10 20:11:03 +08:00
Chris Lu	faec9076a4	adjust parameter names	2019-10-21 22:57:01 -07:00
joeslay	d53aee179b	Merge pull request #10 from chrislusf/master ... merge seaweed master	2019-10-14 16:03:40 +01:00
Chris Lu	f8d4b7d1c0	support basic json filtering and selection	2019-10-06 22:35:08 -07:00
Chris Lu	cf47f657af	scaffold for volume server query feature	2019-10-06 22:35:08 -07:00
j.laycock	6fc6322c90	Change joeslay paths to chrislusf paths	2019-09-12 14:18:21 +01:00
joeslay	d8c34b032f	Merge pull request #6 from chrislusf/master ... merge seaweed master	2019-09-12 11:44:09 +01:00
Chris Lu	ae53f63680	filer: recursive deletion optionally ignoring any errors ... fix https://github.com/chrislusf/seaweedfs/issues/1062	2019-09-11 20:26:20 -07:00
j.laycock	84f2dc3b53	Remove test code	2019-09-05 11:14:02 +01:00
Tom Maxwell	4a878c0006	Changed the InMemory bool to a uint32 so that it can be used to alter how much space to reserve	2019-09-04 15:27:14 +01:00