1150 Commits

Author	SHA1	Message	Date
Chris Lu	8123a488cd	filer: optimize leveldb parameters	2019-05-17 18:14:25 -07:00
Chris Lu	76cf57a0e1	volume: optimzation for leveldb	2019-05-17 17:34:03 -07:00
Chris Lu	0fe286a6ca	filer: optimization for leveldb, add leveldb2 ... large filer db may see high CPU and disk usage due to background compaction	2019-05-17 17:33:49 -07:00
Chris Lu	7e0c3bb732	filer: adjust recommended mysql meta data type to LONGBLOB ... fix https://github.com/chrislusf/seaweedfs/issues/966	2019-05-17 09:11:08 -07:00
Chris Lu	bf9d490538	fix for tests	2019-05-17 02:28:20 -07:00
Chris Lu	82b0759493	filer: migrating filer store from persisting shorter structured file id instead of a string	2019-05-17 02:03:23 -07:00
Chris Lu	8156958ee9	move function to make travis happy	2019-05-15 10:02:44 -07:00
Chris Lu	0a36f628c6	testing RS coding	2019-05-15 01:02:00 -07:00
Chris Lu	ab22407763	filer ip only use bind ip	2019-05-14 23:43:40 -07:00
Chris Lu	017d0957c3	avoid switching master node if same ip and host name ... fix https://github.com/chrislusf/seaweedfs/issues/955	2019-05-11 18:42:46 -07:00
Chris Lu	4dfcd2169e	weedfuse: adapt to fstab command line pattern	2019-05-11 01:17:55 -07:00
Chris Lu	9a4fb14ea0	weedfuse: daemonize	2019-05-10 17:08:15 -07:00
Chris Lu	1ca1ec906a	add weedfuse	2019-05-10 15:03:31 -07:00
Chris Lu	69b7dd398d	adjust help message	2019-05-07 14:02:01 -07:00
Chris Lu	d01807ba05	shell: volume.balance add option for all_collections, each_collection	2019-05-06 14:12:19 -07:00
Chris Lu	8ffdd5dcdb	writel throttling moving volumes	2019-05-06 13:58:42 -07:00
Chris Lu	cf58fc0e63	refactor: extract out the write throttler	2019-05-06 13:56:08 -07:00
Chris Lu	4e42e7b5e7	shell: volume.balance add dataCenter filter	2019-05-06 13:30:14 -07:00
Chris Lu	4c392ca762	use ip.bind to listen for grpc calls ... help to avoid https://github.com/chrislusf/seaweedfs/issues/952	2019-05-06 08:10:18 -07:00
Chris Lu	8656102127	shell: volume.balance runs for each collection, and for ALL at the end	2019-05-05 22:28:14 -07:00
Chris Lu	f08bbb72de	refactoring	2019-05-05 22:21:28 -07:00
Chris Lu	0df6346611	add error checking	2019-05-05 21:58:46 -07:00
Chris Lu	3ff92037cc	shell: add volume.balance command	2019-05-05 21:17:23 -07:00
Chris Lu	3f2a30bd05	shell: volume.list output sorted voume list	2019-05-05 20:23:50 -07:00
Chris Lu	46a675ecc2	align memory to avoid nil with atomic operations	2019-05-04 21:33:05 -07:00
Chris Lu	177f3add81	adjust memory layout to avoid possible nil pointer	2019-05-04 08:47:11 -07:00
Chris Lu	25941e0500	master: add jwt expires_after_seconds	2019-05-04 08:42:25 -07:00
Chris Lu	bd8af92b54	remove unused code	2019-05-04 08:20:22 -07:00
Chris Lu	4b27f892e6	remove 128 bit needle id build option ... fix https://github.com/chrislusf/seaweedfs/issues/949	2019-05-03 22:39:00 -07:00
Chris Lu	3fbe22b203	better error handling for nil raftServer ... addressing https://github.com/chrislusf/seaweedfs/issues/717	2019-05-03 22:33:08 -07:00
Chris Lu	5681208f2d	refactor: remove logger	2019-05-03 22:21:06 -07:00
Chris Lu	8c982db25d	1.33	2019-05-03 17:27:50 -07:00
Chris Lu	b335f81a4f	volume: add option to limit compaction speed	2019-05-03 17:22:39 -07:00
Chris Lu	f0f981e7c8	webdav: fix directory renaming	2019-05-03 14:12:51 -07:00
Chris Lu	bfbecd7253	webdav: fix directory creation	2019-05-03 13:13:11 -07:00
Chris Lu	9d3c945b00	Merge pull request #950 from chrislusf/webdav ... add WebDav	2019-05-03 01:25:10 -07:00
Chris Lu	7064b2ea48	webdav: can write now	2019-05-03 00:55:52 -07:00
Chris Lu	b30c14b631	webdav: can read now	2019-05-03 00:24:35 -07:00
Chris Lu	47f14775d7	scaffolding, but seems hard to satisfy mac, windows, and different os versions ... https://www.reddit.com/r/golang/comments/3wrxo3/webdav_server_that_provide_filesystem_using/	2019-05-02 14:22:10 -07:00
Wine93	32f93fb09e	style: go fmt for all	2019-04-30 03:22:19 +00:00
Chris Lu	82e06a780b	mount: allow nonempty fuse mount to work with Samba ... possibly fix https://github.com/chrislusf/seaweedfs/issues/936	2019-04-29 10:42:09 -07:00
Chris Lu	108d0fb08d	adjust log level	2019-04-26 09:32:37 -07:00
Chris Lu	575926cdec	minor	2019-04-26 09:32:37 -07:00
Chris Lu	630cf1e33e	fix	2019-04-24 12:31:52 -07:00
Chris Lu	ea4592161c	fix	2019-04-24 12:31:14 -07:00
Chris Lu	c5a931b22c	logging	2019-04-24 12:30:36 -07:00
Chris Lu	9beea63960	memory alignment ... fix https://github.com/chrislusf/seaweedfs/issues/939	2019-04-24 09:23:19 -07:00
Chris Lu	9774ede3cf	adjust usage description	2019-04-24 00:25:20 -07:00
Chris Lu	b04d7e3ac0	weed server: also optionally start S3 gateway	2019-04-24 00:18:01 -07:00
Chris Lu	dabc9c9a89	filer: listen on specified ip address	2019-04-24 00:01:39 -07:00
Chris Lu	21fe5bde7c	filer: remove customizable grpc port, fixed to filer port + 10000	2019-04-23 23:58:52 -07:00
Chris Lu	067b935763	report unknown commands	2019-04-23 22:29:36 -07:00
Chris Lu	53e2e73de2	fix metrics ... fix https://github.com/chrislusf/seaweedfs/issues/935	2019-04-22 12:32:10 -07:00
Chris Lu	0c96cfdb2e	minor	2019-04-21 15:50:51 -07:00
Chris Lu	00b6f653fa	shell: add fs.cat	2019-04-21 15:43:43 -07:00
Chris Lu	59d532a8c3	1.31	2019-04-21 14:09:50 -07:00
Chris Lu	5c411f3e5f	minor	2019-04-21 13:33:32 -07:00
Chris Lu	440111a349	volume: support http status 304 for the same file id	2019-04-21 13:33:23 -07:00
Chris Lu	ebe94be1a1	maintain layout correctness when changing volumes	2019-04-21 13:32:36 -07:00
Chris Lu	0302b9496c	volume: also check cookie for possible duplicates	2019-04-21 13:31:45 -07:00
Chris Lu	132921ad41	adjust log or error	2019-04-21 10:14:17 -07:00
Chris Lu	316bd27f75	add remove volumes with version info	2019-04-20 23:53:37 -07:00
Chris Lu	64a9a0e104	shell: fix volume.copy	2019-04-20 20:48:07 -07:00
Chris Lu	5f3d0e33a1	shell: add commands volume.copy volume.delete volume.mount volume.unmount	2019-04-20 20:31:35 -07:00
Chris Lu	d06d3c57d4	add notes	2019-04-20 20:12:07 -07:00
Chris Lu	2ffe98443b	refactoring	2019-04-20 12:05:28 -07:00
Chris Lu	6fc1f53018	shell: add command volume.move	2019-04-20 11:35:23 -07:00
stlpmo-jn	2200ea9cb9	add function ParseFileIdFromString	2019-04-20 18:39:06 +08:00
Chris Lu	3b3651dea3	volume: atomic copying file, adds version and stopOffset	2019-04-19 12:29:49 -07:00
Chris Lu	730a032137	filer mysql: adjust scaffolding instructions	2019-04-19 09:55:46 -07:00
Chris Lu	0be2d51c96	read volume lastAppendAtNs when loading a volume	2019-04-19 01:56:38 -07:00
Chris Lu	ac2727853f	fix needle map entry size	2019-04-19 00:39:34 -07:00
Chris Lu	e5506152c0	refactoring	2019-04-18 21:43:36 -07:00
Chris Lu	33c92b819a	refactoring	2019-04-18 21:17:43 -07:00
Chris Lu	a2d34d4802	filer.copy: add adjustable compression level	2019-04-18 20:21:28 -07:00
Chris Lu	fa176fe80f	volume tailing chunks large files	2019-04-18 19:22:13 -07:00
Chris Lu	3dce1016cb	add volume tailer	2019-04-18 11:05:02 -07:00
Chris Lu	b142f9f1d5	go fmt	2019-04-18 00:19:18 -07:00
Chris Lu	b09e8dbb37	add VolumeStreamFollow, but not used yet	2019-04-18 00:18:29 -07:00
Chris Lu	13ad5c1966	refactoring	2019-04-17 22:04:49 -07:00
Chris Lu	3b9446606a	adjust log level	2019-04-16 23:31:44 -07:00
Chris Lu	07091c44cd	lock btree writes ... fix https://github.com/chrislusf/seaweedfs/issues/930	2019-04-16 19:48:07 -07:00
Chris Lu	4393b99332	add notes	2019-04-16 13:57:45 -07:00
Chris Lu	338e6d60a5	refactor: prepare for snapshotting	2019-04-16 09:55:37 -07:00
Chris Lu	af49aea0c6	weed shell: fs.meta.save adjusts meta data file name	2019-04-16 09:40:27 -07:00
Chris Lu	8ea1ee6dfa	weed shell: add fs.meta.notify, removing filer.export	2019-04-16 01:58:28 -07:00
Chris Lu	014906ec79	weed shell: add command fs.meta.load to restore meta data	2019-04-16 01:37:11 -07:00
Chris Lu	79c2cca9c1	better error message	2019-04-16 01:15:30 -07:00
Chris Lu	d35023c713	weed shell: add command fs.meta.save	2019-04-16 01:06:32 -07:00
Chris Lu	b3b42bc947	replicate need to include new entry path	2019-04-16 00:44:31 -07:00
Chris Lu	967e108b9a	increase max idle conn per hosts	2019-04-15 22:00:50 -07:00
Chris Lu	f567777d64	volume: fix needle checking for deleted needles	2019-04-15 21:58:43 -07:00
Chris Lu	bd76ad0ff1	Revert "Revert "volume: close request body if any parsing error"" ... This reverts commit 359b7c1650.	2019-04-15 21:43:28 -07:00
Chris Lu	359b7c1650	Revert "volume: close request body if any parsing error" ... This reverts commit 5996656141.	2019-04-15 21:35:12 -07:00
Chris Lu	5996656141	volume: close request body if any parsing error	2019-04-15 21:11:45 -07:00
Chris Lu	6e116b3d67	volume: reset leader if error happens	2019-04-15 09:09:46 -07:00
Chris Lu	e85048bcdc	http exhaust and close response body	2019-04-14 23:28:24 -07:00
Chris Lu	3e8a3a8fec	fix race detector found problems	2019-04-14 23:00:37 -07:00
Chris Lu	2e03f3adef	filer: rename -port.public to -port.readonly	2019-04-11 20:42:55 -07:00
Chris Lu	59a9e15733	1.30	2019-04-11 19:40:36 -07:00
Chris Lu	e11e127d91	close grpc connections during errors	2019-04-11 09:27:08 -07:00
Wine93	361912224d	typo: remove blank	2019-04-11 09:18:53 +00:00
Chris Lu	0e33272f56	small refactoring	2019-04-10 23:39:53 -07:00
Chris Lu	cb2061f86a	Merge pull request #925 from stlpmo-jn/checkUnhealthReplication ... the implement of repair the error replications of the volume	2019-04-10 23:37:12 -07:00
Chris Lu	c0d6cc0067	weed mount: mount to the same folder name	2019-04-10 23:25:41 -07:00
stlpmo-jn	c3fa50d3b3	remove the health chekcer, because it's the same as command_volume_fix_replication	2019-04-11 13:40:31 +08:00
Chris Lu	2f76681d62	refactor: adjust grpc API name	2019-04-10 21:41:17 -07:00
stlpmo-jn	f2031884f0	fix bug : CI build failed	2019-04-11 09:53:31 +08:00
Chris Lu	4df10e50b3	update glide.yaml	2019-04-10 14:27:28 -07:00
stlpmo-jn	c1a0403da2	repair the error replications of the volume	2019-04-10 19:41:55 +08:00
Chris Lu	3555628ad5	weed volume: fix bug, remove boltdb, btree	2019-04-09 10:08:59 -07:00
Chris Lu	8a5ce16e96	weed volume: remove boltdb, btree options, add options for leveldb medium, large	2019-04-09 09:42:06 -07:00
Chris Lu	9924fa3b1a	weed volume: disable block cache on leveldb ... saving 8MB for each volume, but at the cost of slower lookup	2019-04-09 00:24:32 -07:00
Chris Lu	35aba35c07	Revert "weed volume: disable block cache on leveldb" ... This reverts commit a55e708d36.	2019-04-09 00:22:40 -07:00
Chris Lu	a55e708d36	weed volume: disable block cache on leveldb ... saving 8MB for each volume, but at the cost of slower (offset,size) lookup	2019-04-09 00:18:32 -07:00
Chris Lu	6d4c44a85c	1.29	2019-04-08 22:41:59 -07:00
Chris Lu	88ba08cb9e	add large disk support	2019-04-08 22:32:42 -07:00
Chris Lu	784c5bb73a	add build option support 5-byte offset	2019-04-08 22:01:29 -07:00
Chris Lu	7a4b234ea1	divide offset into higher and lower sections	2019-04-08 21:44:06 -07:00
Chris Lu	000ee725fc	refactor Offset into a struct of bytes	2019-04-08 19:40:56 -07:00
Chris Lu	a41ba79119	Merge pull request #914 from jba/gocdk-receive ... replication: add GoCDK PubSub support	2019-04-08 10:31:02 -07:00
Chris Lu	d14b614407	weed filer.copy: use existing file owner and gropu id	2019-04-07 11:31:50 -07:00
Chris Lu	a32abda1a3	adjust compression to optimize for speed	2019-04-07 11:01:28 -07:00
Chris Lu	c731d9be6d	weed filer.copy: add option to profile the execution	2019-04-07 09:13:24 -07:00
Chris Lu	174bf1e8b2	more efficient client side gzip compression	2019-04-06 14:14:28 -07:00
Chris Lu	868913aa04	printout statistics	2019-04-06 11:12:35 -07:00
Chris Lu	de7626bd22	weed shell: volume.list add summary statistics	2019-04-06 09:25:29 -07:00
Chris Lu	94302935b2	weed volume: only store compressed data if compression is effective	2019-04-06 00:10:52 -07:00
Chris Lu	b2d92a2992	weed shell: fs.tree fix directory tree listing	2019-04-05 23:42:36 -07:00
Chris Lu	24c020104a	weed filer.copy: parallelize the copying to increase throughput ... improvement https://github.com/chrislusf/seaweedfs/issues/915	2019-04-05 23:35:30 -07:00
Chris Lu	c789b496d8	use cached grpc client	2019-04-05 20:31:58 -07:00
Chris Lu	5808caa2f5	use cached grpc client	2019-04-05 20:26:52 -07:00
Chris Lu	cd6d35aa52	weed shell: fs.tree improvements	2019-04-05 11:40:54 -07:00
Jonathan Amsterdam	72920efc20	added entry to scaffold	2019-04-05 13:43:38 -04:00
Jonathan Amsterdam	a6a5d80401	fix typo	2019-04-05 08:13:32 -04:00
Chris Lu	300b39b5af	weed filer.copy: use cached filer grpc connection ... another attemp for https://github.com/chrislusf/seaweedfs/issues/915	2019-04-05 01:09:06 -07:00
Chris Lu	921231268b	weed filer.copy: skip files that can not be opened ... such as socket	2019-04-05 00:04:00 -07:00
Chris Lu	4f714cef22	default weed filer.copy and weed upload chunk size to 32MB	2019-04-04 23:14:32 -07:00
Chris Lu	c94823f52c	set default http idle connection per host ... possible fix https://github.com/chrislusf/seaweedfs/issues/915	2019-04-04 20:28:40 -07:00
Chris Lu	0bd7ced7c2	fix test	2019-04-04 20:11:59 -07:00
Chris Lu	9cc73f4a9a	fix test	2019-04-04 19:34:15 -07:00
Chris Lu	af37b374cb	weed shell: fs.cd change current directory	2019-04-04 19:27:51 -07:00
Chris Lu	766396d249	weed master: atomic volume counting ... possible fix for https://github.com/chrislusf/seaweedfs/issues/913	2019-04-04 19:27:00 -07:00
Jonathan Amsterdam	c0c9a8bad5	replication: add GoCDK PubSub support	2019-04-04 17:22:45 -04:00
Chris Lu	715a38da1e	weed shell: add fs.cd, fs.pwd to change to a directory and print current directory	2019-04-03 00:20:00 -07:00
Chris Lu	20dcb44077	fix tests	2019-04-01 23:59:31 -07:00
Chris Lu	beaa2bd71a	weed filer: properly delete cached directory entry ... fix https://github.com/chrislusf/seaweedfs/issues/910	2019-04-01 17:03:04 -07:00
Chris Lu	78b9db34d5	weed s3: generate time format acceptable by aws cli s3 ... fix https://github.com/chrislusf/seaweedfs/issues/911	2019-04-01 16:13:29 -07:00
Chris Lu	2a52e70a41	weed filer: increase default mysql/postgres dir or name length to 65535	2019-04-01 12:37:54 -07:00
Chris Lu	b4c276263f	weed filer: mysql/postgres use ReadCommitted transaction isolation level for atomic rename	2019-04-01 11:03:04 -07:00