2202 Commits

Author	SHA1	Message	Date
Chris Lu	0952eb2854	Merge pull request #1302 from song-zhang/master ... write request keep compatible with before	2020-05-06 15:37:34 -07:00
Chris Lu	5568395edd	Revert "Revert "Merge pull request #1299 from song-zhang/master"" ... This reverts commit afb6a1dbb4.	2020-05-06 15:37:17 -07:00
zhangsong	3703c7e176	write request keep compatible with before	2020-05-06 21:35:32 +08:00
Chris Lu	1e3e4b3072	add broker connects to filer	2020-05-05 02:05:28 -07:00
Chris Lu	afb6a1dbb4	Revert "Merge pull request #1299 from song-zhang/master" ... This reverts commit 9016fa19ba, reversing changes made to 47234760f4.	2020-05-04 20:34:26 -07:00
Chris Lu	9016fa19ba	Merge pull request #1299 from song-zhang/master ... use async write to persistent file to disk - part1	2020-05-04 15:51:03 -07:00
zhangsong	f9e8702bb4	use async write to persistent file to disk - part1	2020-05-04 17:39:44 +08:00
Chris Lu	47234760f4	log adjust format	2020-05-02 01:33:49 -07:00
Chris Lu	97a504b0bb	fix possible directory deletion error	2020-05-02 01:11:53 -07:00
Chris Lu	78422c2f69	Merge pull request #1298 from song-zhang/master ... add fsync option for benchmark	2020-04-30 20:16:42 -07:00
Chris Lu	871efa4fc1	refactoring ... some previous chunk etag was using md5, which should be wrong.	2020-04-30 17:20:47 -07:00
zhangsong	d819865fe7	add fsync option for benchmark	2020-04-30 23:31:08 +08:00
Chris Lu	ea809a45ab	add cipher key	2020-04-30 03:05:43 -07:00
Chris Lu	f9b6178b8f	log messages	2020-04-30 03:05:34 -07:00
Chris Lu	8c73410a51	subscribe from a timestamp	2020-04-30 02:19:51 -07:00
Chris Lu	4e16a90454	for debugging	2020-04-30 02:19:32 -07:00
Chris Lu	a25a8d8822	s3: print out time in UTC format ... fix https://github.com/chrislusf/seaweedfs/issues/1297	2020-04-29 21:35:24 -07:00
Chris Lu	e93588ec78	FUSE mount: atomic local cache updates	2020-04-29 18:20:54 -07:00
Chris Lu	9e72e9e4b8	able to subscribe any topic from any point of time	2020-04-29 17:40:08 -07:00
Chris Lu	ed3cf811f5	refactoring	2020-04-29 13:26:02 -07:00
Chris Lu	7c10602b49	read in case cross chunks	2020-04-29 02:42:58 -07:00
Chris Lu	f9da859720	better error message	2020-04-29 02:42:06 -07:00
Chris Lu	d57559d3f7	chunk rememober encryption or not	2020-04-29 02:41:47 -07:00
Chris Lu	5a0986dca0	reset on getting the buffer	2020-04-29 02:41:01 -07:00
Chris Lu	8d42a1263b	meta subscription: update the last read time	2020-04-28 17:30:04 -07:00
Chris Lu	31583b5d6c	master and volume server: avoid race condition ... The volume server may disconnect and reconnect to the same master. The master's unregistration may happen after the reconnection. Thus the volume server will disappear.	2020-04-28 17:29:10 -07:00
Chris Lu	98edec7c3e	remove printouts	2020-04-28 16:45:48 -07:00
Chris Lu	8ed490164e	refactoring	2020-04-28 02:05:44 -07:00
Chris Lu	66effaed9e	fix compilation	2020-04-28 02:05:32 -07:00
Chris Lu	eab4c9219c	randomize benchmark content ... fix https://github.com/chrislusf/seaweedfs/issues/1294	2020-04-28 01:50:56 -07:00
Chris Lu	fb81f12686	filer: simplify image resize	2020-04-28 00:05:57 -07:00
Chris Lu	5c57297bd1	metadata log: read from any timestamp	2020-04-28 00:05:57 -07:00
wuyuxiang	6850d28d6b	refacotr: move signal handling and pprof to grace package	2020-04-28 14:10:23 +08:00
Chris Lu	2a7957b4ca	FUSE: subcribe meta events based on timestamp	2020-04-27 16:52:20 -07:00
Chris Lu	6190fd665d	printout error	2020-04-27 12:41:31 -07:00
Chris Lu	725e47dcc3	filer: pass along user name and group name for HDFS	2020-04-27 05:48:48 -07:00
Chris Lu	508ec01ab0	tweaking	2020-04-26 16:11:06 -07:00
Chris Lu	f4fcde05b6	1.77	2020-04-26 05:54:47 -07:00
Chris Lu	b52b8ec685	Hadoop: fix entry not found for HCFS ... also fix cipher related changes.	2020-04-26 05:21:54 -07:00
Chris Lu	0c2248f83a	add lock/unlock to example admin scripts	2020-04-23 18:41:48 -07:00
Chris Lu	913aa0f14e	mount: default to enable asyncMetaDataCaching	2020-04-23 14:14:28 -07:00
Chris Lu	662b5d0cf7	support semicolon seperated command lines	2020-04-23 14:01:46 -07:00
Chris Lu	73564e6a01	master: add cluster wide lock/unlock operation in weed shell ... fix https://github.com/chrislusf/seaweedfs/issues/1286	2020-04-23 13:37:31 -07:00
Chris Lu	bdc337a719	make lock/unlock optional	2020-04-23 03:50:05 -07:00
Chris Lu	57ec7f2ba9	fix builds	2020-04-23 03:43:45 -07:00
Chris Lu	eab9701823	allow lock with an existing lock	2020-04-23 03:35:52 -07:00
Chris Lu	842e3301ee	make second lock instant	2020-04-23 03:32:18 -07:00
Chris Lu	69f336e59f	shell: add lock/unlock command	2020-04-23 03:11:07 -07:00
Chris Lu	ff0a7c1d18	clean up	2020-04-23 03:10:47 -07:00
Chris Lu	30ee4f3291	add exclusive lock library on shell	2020-04-23 02:31:04 -07:00
Chris Lu	77873b832b	add master side code for cluster wise exclusive lock	2020-04-23 01:55:44 -07:00
Chris Lu	369aa8a10a	1.76	2020-04-22 23:31:59 -07:00
Chris Lu	69eda558ef	reduce the connections ... fix https://github.com/chrislusf/seaweedfs/issues/1277 this avoids sending error to wdclients, and clients retry the connections	2020-04-22 20:43:34 -07:00
Chris Lu	aebe39a803	avoid repeated grpc connection creation ... fix https://github.com/chrislusf/seaweedfs/issues/1277	2020-04-22 18:26:24 -07:00
Chris Lu	5d0e1d8d74	also writes to local meta cache ... before waiting for subscribed meta events	2020-04-22 15:40:47 -07:00
Chris Lu	72c5dba4d4	1.75	2020-04-21 22:03:07 -07:00
Chris Lu	a207285af7	cache metadata on startup	2020-04-21 22:00:34 -07:00
Chris Lu	9f9826e95a	refactor	2020-04-21 21:28:47 -07:00
Chris Lu	e24b25de78	async meta caching: can stream updates now	2020-04-21 21:16:13 -07:00
Chris Lu	4f02f7121d	read from meta cache ... meta cache is not initialized	2020-04-21 18:50:30 -07:00
Chris Lu	b8e4238ad2	add placeholder for cached meta data reading	2020-04-21 18:02:08 -07:00
Chris Lu	27128c7875	adjust logging	2020-04-21 16:13:13 -07:00
Chris Lu	bafa95045b	volume: deletion checks all disk locations ... fix https://github.com/chrislusf/seaweedfs/issues/1283	2020-04-21 14:49:58 -07:00
Chris Lu	57df14f76f	report error if copying failed	2020-04-21 14:28:46 -07:00
Chris Lu	cf54d70311	filer: add -ip.bind option	2020-04-21 14:21:06 -07:00
Chris Lu	5c348087dc	messaging: able to pub sub multiple partitions	2020-04-21 00:59:55 -07:00
Chris Lu	cb3985be70	go fmt	2020-04-20 17:48:06 -07:00
Chris Lu	5eb83dfbd3	add memory buffer size limit	2020-04-20 17:43:50 -07:00
Chris Lu	258fba8a0f	continue for reading from sealed memory buffer	2020-04-20 17:28:18 -07:00
Chris Lu	4bf959edf0	message broker: read also from sealed memory buffer	2020-04-20 17:26:38 -07:00
Chris Lu	bd43c62fbd	lower the test criteria	2020-04-20 03:08:10 -07:00
Chris Lu	c3f9fdf48f	adjust test	2020-04-20 02:54:21 -07:00
Chris Lu	5e8bb86995	close the connection with better accouting	2020-04-20 02:53:50 -07:00
Chris Lu	978da73cdb	revert this grpc related change ... maybe related to https://github.com/chrislusf/seaweedfs/issues/1277	2020-04-20 02:53:22 -07:00
Chris Lu	ebfab42a50	refactoring	2020-04-20 00:08:47 -07:00
Chris Lu	2955b96ef1	refactor	2020-04-19 23:54:32 -07:00
Chris Lu	df25ed3c5b	logs	2020-04-19 23:37:50 -07:00
Chris Lu	ce3cb25cfb	working for in memory single log buffer	2020-04-19 23:37:50 -07:00
Chris Lu	f373232227	Merge pull request #1280 from bukton123/filer_mongodb ... Add new filer support mongodb	2020-04-19 20:35:19 -07:00
bukton	da37455254	filer mongo add func create index unique directory, name	2020-04-20 08:13:57 +07:00
Chris Lu	4642ed565e	fix tests	2020-04-19 10:42:58 -07:00
bukton	3defa718a1	add default config filer mongodb to scaffold.go	2020-04-19 17:52:38 +07:00
bukton	d6d245dc89	filer mongodb add option poll size and fixed order by list meta	2020-04-19 17:51:32 +07:00
Chris Lu	71ffb98475	broker: add profiling	2020-04-19 03:03:40 -07:00
Chris Lu	d278b4c228	basic pub and sub works!	2020-04-19 00:18:32 -07:00
bukton	082f079ead	filer mongo change func update to insert entry	2020-04-19 11:08:10 +07:00
Chris Lu	788578d4ea	add subscriber api	2020-04-18 16:05:29 -07:00
Chris Lu	767f14dfcd	master, volume default to detected host	2020-04-18 15:20:20 -07:00
Chris Lu	076c8bd3bc	filer master start up with default ip address instead of just localhost	2020-04-18 15:17:27 -07:00
Chris Lu	5d346d44bd	Revert "move go client out" ... This reverts commit ef1274edc9.	2020-04-18 12:44:55 -07:00
Chris Lu	ef1274edc9	move go client out	2020-04-18 12:11:50 -07:00
Chris Lu	5af49c7aa5	adjust API	2020-04-18 11:45:57 -07:00
bukton	8c58b0d017	filer mongodb delete folder children	2020-04-19 00:31:49 +07:00
bukton	290c6b7f01	Merge remote-tracking branch 'origin/master' into filer_mongodb ... # Conflicts: # go.mod # go.sum # weed/server/filer_server.go	2020-04-19 00:21:45 +07:00
bukton	6234ea441b	filer mongodb delete	2020-04-19 00:20:00 +07:00
bukton	24a8192298	filer mongo list	2020-04-19 00:16:50 +07:00
bukton	3c70163798	filer mongodb insert entry and find entry	2020-04-18 23:48:38 +07:00
bukton	cd9cccec55	add import mongodb to filer server	2020-04-18 22:30:13 +07:00
bukton	fd4576651b	add new package mongodb and install lib mongodb	2020-04-18 22:26:57 +07:00
Chris Lu	11f5a6d913	messaging can compile now	2020-04-18 01:12:01 -07:00
Chris Lu	5bea77010f	refactor	2020-04-17 22:39:21 -07:00
Chris Lu	2e2537a9ea	rename files	2020-04-17 22:35:44 -07:00
Chris Lu	e0f5996560	fix "call of Unmarshal passes non-pointer as second argument"	2020-04-17 10:04:41 -07:00
Chris Lu	826f96b6f3	fix unreachable code	2020-04-17 10:04:28 -07:00
Chris Lu	ffa69b24ae	fix travis	2020-04-17 10:01:11 -07:00
Chris Lu	3f3dba5a68	broker: append message logs	2020-04-17 02:29:38 -07:00
Chris Lu	2a45897237	broker: read cipher value from filer	2020-04-17 02:29:00 -07:00
Chris Lu	bda82f61bc	filer: able to append to a file	2020-04-17 02:28:09 -07:00
Chris Lu	722b14337e	add cipher option to meta data updates	2020-04-17 01:26:27 -07:00
Chris Lu	7d3672c60a	simplify	2020-04-17 00:00:48 -07:00
Chris Lu	508f3490a0	update messaging proto	2020-04-16 03:29:57 -07:00
Chris Lu	f5a748d33c	refactoring	2020-04-16 02:55:09 -07:00
Chris Lu	ce4b369be2	scaffolding messaging	2020-04-16 02:21:23 -07:00
Reed	aada5b9aa7	making volume stats size info readable.	2020-04-16 10:08:08 +08:00
Chris Lu	9cacaf5eca	better detect file mime type	2020-04-14 11:32:31 -07:00
Chris Lu	3f1d79512f	only detect when not sure about the types	2020-04-14 11:02:05 -07:00
Chris Lu	d2d1597850	volume: detect gzip effectiveness only when content type is empty ... revert part of 2286eda575	2020-04-14 10:52:53 -07:00
Chris Lu	23715af402	Merge pull request #1273 from wyxloading/config_race ... Fix data race about config	2020-04-13 22:24:51 -07:00
Chris Lu	2b5c4fbbf3	tiered caching ... 1/4 for small less than 1MB files. 1/4 for 1~4MB files, 1/2 for bigger than 4MB files	2020-04-13 22:19:27 -07:00
Chris Lu	f282ed444b	refactoring	2020-04-13 21:58:10 -07:00
wuyuxiang	b556d3d035	Fix data race about config	2020-04-14 10:22:40 +08:00
Chris Lu	d8f5985e5e	add logs	2020-04-13 12:58:45 -07:00
Chris Lu	5c9e91393f	1.74	2020-04-13 10:25:48 -07:00
Chris Lu	7764e0465c	refactoring	2020-04-12 21:00:55 -07:00
Chris Lu	8e23dc078b	refactoring	2020-04-12 20:48:21 -07:00
Chris Lu	b062393f3f	refactoring	2020-04-12 17:51:31 -07:00
Chris Lu	5361f999ed	shell: only one shell is allowed to connect to the cluster ... fix https://github.com/chrislusf/seaweedfs/issues/1266	2020-04-12 17:48:39 -07:00
Chris Lu	eb16bb0287	go mod cleanup	2020-04-12 16:44:58 -07:00
Chris Lu	d30483d642	re-enable system logs	2020-04-12 14:03:07 -07:00
Chris Lu	6f948e4887	remove configurable topics folder location	2020-04-12 13:07:59 -07:00
Chris Lu	94e35cdb35	mount: fix ... fix bug found by git bisect, but I do not understand why it can cause error!	2020-04-12 03:34:36 -07:00
Chris Lu	1c2e920fa2	fix compilation error	2020-04-12 03:11:19 -07:00
Chris Lu	b5a713be68	filer store: add redis2 redis_cluster2 as default redis store	2020-04-12 02:50:41 -07:00
Chris Lu	e613695af7	a little optimization	2020-04-12 01:13:57 -07:00
Chris Lu	2a1f396df5	avoid duplicated setting chunks into cache	2020-04-12 01:06:50 -07:00
Chris Lu	b9b7da905e	handle nil chunk cache	2020-04-12 01:00:12 -07:00
Chris Lu	211d87cf4c	mount: option to disable caching	2020-04-12 00:52:54 -07:00
Chris Lu	1c65656fb4	s3: add option to fsync buckets	2020-04-11 23:37:10 -07:00
Chris Lu	e4af63a721	volume server: accept fsync=true in write requests	2020-04-11 21:39:16 -07:00
Chris Lu	5987810e5e	set default chunk size to 16	2020-04-11 21:22:52 -07:00
Chris Lu	df97da25f9	mount: add on disk caching	2020-04-11 21:12:41 -07:00
Chris Lu	c8ca234773	refactoring	2020-04-11 14:27:25 -07:00
Chris Lu	df9d538044	rename function	2020-04-11 14:19:44 -07:00
Chris Lu	d7f3acb2c0	refactor	2020-04-11 12:45:24 -07:00
Chris Lu	ed54682598	refactoring	2020-04-11 12:43:17 -07:00
Chris Lu	e909b55633	refactoring	2020-04-11 12:39:46 -07:00
Chris Lu	b7f0ba3800	refactoring	2020-04-11 12:37:41 -07:00
Chris Lu	417125457e	add additional buffers for logs	2020-04-11 02:19:48 -07:00
Chris Lu	4b27afe176	fix sqs message reading	2020-04-10 15:27:47 -07:00
Chris Lu	dce2702e1b	fix the instruction	2020-04-10 14:50:10 -07:00
Chris Lu	13ab1fdaf0	mount: use file default permission 0666	2020-04-10 02:23:23 -07:00
Chris Lu	1101a42e5c	mv : create new folder, move children, and delete old folder	2020-04-10 01:37:03 -07:00
Chris Lu	bcf37346ef	add timestamp inside lock	2020-04-10 01:35:59 -07:00
Chris Lu	9fa065f600	typo in logs	2020-04-09 23:43:09 -07:00
Chris Lu	8db2120bee	add logs	2020-04-09 23:42:59 -07:00
Chris Lu	43c62a8a95	1.73	2020-04-09 21:44:30 -07:00
Chris Lu	eb39df2704	avoid dead loop	2020-04-09 21:43:26 -07:00
Chris Lu	56ec8c4513	fix to avoid nil file an dirty pages	2020-04-09 21:43:05 -07:00
Chris Lu	59f40e2027	volume: best effort to detect ip address ... fix https://github.com/chrislusf/seaweedfs/issues/1264	2020-04-09 00:26:24 -07:00
Chris Lu	f6a7e79dc3	weed shell: simplify CLI option for filer	2020-04-08 23:57:15 -07:00
Chris Lu	2329d9e0c1	add volume.fix.replication to default scaffold	2020-04-08 23:36:22 -07:00
Chris Lu	bb78ab9915	wait for master to be aware of the failed volumes ... fix https://github.com/chrislusf/seaweedfs/issues/1268	2020-04-08 23:12:37 -07:00
Chris Lu	8764bdb9df	1.72	2020-04-08 22:35:14 -07:00
Chris Lu	8a73276602	refactor a bit	2020-04-08 22:33:36 -07:00
Chris Lu	006f78b70b	make a copy of request data ... to avoid concurrency issues	2020-04-08 22:31:19 -07:00
Chris Lu	00b993a234	add util for md5	2020-04-08 22:30:38 -07:00
Chris Lu	ed98223b08	still log, but not persisting the changes	2020-04-08 20:32:57 -07:00
Chris Lu	6630541399	ensure correct file size when opening existing files	2020-04-08 12:50:59 -07:00
Chris Lu	b524a40375	add locking to fs cache	2020-04-08 12:50:34 -07:00
Chris Lu	bd56172b82	simplify file handle management	2020-04-08 12:50:20 -07:00
Chris Lu	dc08e4098f	add etag only for PUT or large chunked uploads	2020-04-08 09:13:26 -07:00
Chris Lu	ec2eb8bc48	add If-None-Match and If-Modified-Since ... fix https://github.com/chrislusf/seaweedfs/issues/1269	2020-04-08 08:12:00 -07:00
Chris Lu	4c498e73de	filer replication: add s3 endpoint	2020-04-07 17:49:00 -07:00
Chris Lu	745f5d2a25	fix setting default value	2020-04-07 01:58:48 -07:00
Chris Lu	54debdc6f7	filer: fix configuration settings	2020-04-07 01:30:53 -07:00
Chris Lu	90d576affe	disable meta data change event logging for now.	2020-04-07 01:05:24 -07:00
Chris Lu	b51d8f03bb	1.71	2020-04-05 22:04:16 -07:00
Chris Lu	5a5908407d	filer: support larger file size ... fix https://github.com/chrislusf/seaweedfs/issues/1257	2020-04-05 21:48:45 -07:00
Chris Lu	19edd9c091	comment out debug messages	2020-04-05 20:15:17 -07:00
Chris Lu	78ea937606	fix	2020-04-05 17:34:26 -07:00
Chris Lu	53626734d4	iterate through the log buffer	2020-04-05 16:51:30 -07:00
Chris Lu	f71b855db5	adjust caching and notification for recursive directory deletion	2020-04-05 15:03:25 -07:00
Chris Lu	f14b6a09a8	remove duplicated update events	2020-04-05 13:22:15 -07:00
Chris Lu	91da7057b1	refactoring	2020-04-05 13:11:43 -07:00
Chris Lu	2d43f85577	watch entries with common path prefix	2020-04-05 12:51:21 -07:00
Chris Lu	7bc3c93512	add util.PathJoin	2020-04-05 12:40:46 -07:00
Chris Lu	af1f64d244	change from 'weed tail' to 'weed watch'	2020-04-05 12:39:20 -07:00
Chris Lu	b10679fcf0	adjust log file names	2020-04-05 12:38:46 -07:00
Chris Lu	2cb1700ecb	set to 4MB meta log buffer size	2020-04-05 12:38:29 -07:00
Chris Lu	e78ee087e9	fix tests	2020-04-05 01:17:35 -07:00
Chris Lu	dcc6b6a3ed	go fmt	2020-04-05 00:52:31 -07:00
Chris Lu	bf270d9e8c	filer: able to tail meta data changes	2020-04-05 00:51:16 -07:00
Chris Lu	f7b5a1d697	add disk stat to logs	2020-04-04 13:17:23 -07:00
Chris Lu	6e776aaafc	loop through all masters	2020-04-03 00:47:48 -07:00
Chris Lu	4a5e381feb	fail fast if filer configured wrong masters ... fix https://github.com/chrislusf/seaweedfs/issues/1257	2020-04-03 00:47:33 -07:00
Chris Lu	aaf5e658cc	adjust parameter name	2020-04-03 00:41:05 -07:00
Chris Lu	6c96e51414	re-ordering	2020-04-03 00:40:54 -07:00
Chris Lu	d2f8e50459	make error a little bit more clearer ... fix https://github.com/chrislusf/seaweedfs/issues/1259	2020-04-03 00:24:09 -07:00
Chris Lu	5f881d2fd5	mount: fail early rather than after mounted the directory ... fix https://github.com/chrislusf/seaweedfs/issues/1258	2020-04-03 00:21:03 -07:00
Chris Lu	fee3319cbf	handle gzip for image resizing	2020-04-02 23:44:24 -07:00