6924 Commits

Author	SHA1	Message	Date
Konstantin Lebedev	4966a3abc7	avoid race conditions access to growRequestCount (#3537) ... https://github.com/seaweedfs/seaweedfs/issues/3511	2022-08-29 13:23:02 -07:00
Patrick Schmidt	00041f6e08	Fix corrupted mount reads after changes on fh (#3536)	2022-08-29 08:27:13 -07:00
chrislu	b7a887fea1	3.24	2022-08-28 21:53:13 -07:00
Rain Li	089446f5b1	Fix reload page before all file uploaded bug (#3534)	2022-08-28 07:51:47 -07:00
chrislu	cc0c8c5f81	simplify	2022-08-27 00:21:57 -07:00
chrislu	3741779144	minor	2022-08-27 00:20:58 -07:00
chrislu	580a7fa685	adjust for data race ... fix https://github.com/seaweedfs/seaweedfs/issues/3512	2022-08-27 00:19:13 -07:00
chrislu	87b70a6809	clean up	2022-08-27 00:09:04 -07:00
chrislu	27c7611ebf	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 23:47:39 -07:00
chrislu	9920d65bc0	gateway to remote object store: adjust upload concurrency	2022-08-26 23:47:37 -07:00
chrislu	c839ce1b19	s3 sink use s3 upload manager ... fix https://github.com/seaweedfs/seaweedfs/issues/3531	2022-08-26 23:47:12 -07:00
famosss	dc4037925d	fix: Build DeletedVids before reset dn's children (#3530)	2022-08-26 22:52:08 -07:00
chrislu	3f3a1341d8	make CodeQL happy	2022-08-26 17:09:11 -07:00
chrislu	301b49b63f	atomic operation	2022-08-26 17:04:11 -07:00
chrislu	57a46f46a0	nano level precision	2022-08-26 16:55:15 -07:00
chrislu	8dae81c5ed	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-26 16:49:51 -07:00
chrislu	e0f4366f4c	more detailed error ... related to https://github.com/seaweedfs/seaweedfs/issues/3528	2022-08-26 16:49:48 -07:00
Patrick Schmidt	5df105b1f9	Fix a few data races when reading files in mount (#3527)	2022-08-26 16:41:37 -07:00
chrislu	f5156cf3a8	rename varaible	2022-08-26 10:23:42 -07:00
Konstantin Lebedev	e90ab4ac60	avoid race conditions for OnPeerUpdate (#3525) ... https://github.com/seaweedfs/seaweedfs/issues/3524	2022-08-26 10:18:49 -07:00
Konstantin Lebedev	4f7a1f67cd	avoid race conditions for diskLocation.MaxVolumeCount (#3526)	2022-08-26 08:41:42 -07:00
chrislu	6f483a4d36	atomic add ... fix https://github.com/seaweedfs/seaweedfs/issues/3514	2022-08-25 22:20:34 -07:00
chrislu	7394f7feee	minor: adjust ordering	2022-08-25 00:20:23 -07:00
chrislu	dbf0de4ce1	minor clean up	2022-08-25 00:19:08 -07:00
Patrick Schmidt	2930263dfd	Fix race conditions during in-flight size checks (#3505)	2022-08-24 20:03:34 -07:00
Patrick Schmidt	7b424a54dc	Add raft server access mutex to avoid races (#3503)	2022-08-24 09:49:05 -07:00
Konstantin Lebedev	f7aeb06544	s3: report metadata if the directory is explicitly created (#3498) ... * replace mkdir to mkFile * ContentLength must be zero * revert mkDir * Seaweedfs-Is-Directory-Key return metadata	2022-08-24 00:15:44 -07:00
Guo Lei	c57c79a0ab	optimiz commitig compact (#3388) ... * optimiz vacuuming volume * fix bugx * rename parameters * fix conflict * change copyDataBasedOnIndexFile to an instance method * close needlemap * optimiz commiting Vacuum volume for leveldb index * fix bugs * fix leveldb loading bugs * refactor * fix leveldb loading bug * add leveldb recovery * add test case for levelDB * modify test case to cover all the new branches * use one tmpNm instead of two instances * refactor * refactor * move setWatermark to the end * add test for watermark and updating leveldb * fix error logic * refactor, add test * check nil before close needlemapeer add test case fix metric bug * add tests, fix bugs * adjust log level remove wrong test case refactor * avoid duplicate updating metric for leveldb index	2022-08-23 23:53:35 -07:00
chrislu	10414fd81c	ping timeout at 15 seconds ... this 72 minute timeout setting seems unreasonably long 15 seconds is around the time when a new raft leader should be elected.	2022-08-23 23:28:16 -07:00
askeipx	2e78a522ab	remove old raft servers if they don't answer to pings for too long (#3398) ... * remove old raft servers if they don't answer to pings for too long add ping durations as options rename ping fields fix some todos get masters through masterclient raft remove server from leader use raft servers to ping them CheckMastersAlive for hashicorp raft only * prepare blocking ping * pass waitForReady as param * pass waitForReady through all functions * waitForReady works * refactor * remove unneeded params * rollback unneeded changes * fix	2022-08-23 23:18:21 -07:00
Guo Lei	762dc219a6	fix deletecount bug (#3499)	2022-08-23 08:53:24 -07:00
chrislu	21a933c932	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-23 01:52:32 -07:00
chrislu	f0b4a7659a	fix test	2022-08-23 01:52:29 -07:00
Patrick Schmidt	cda8cc22bc	Implement lseek syscall in FUSE (#3491) ... See the man page of lseek: https://man7.org/linux/man-pages/man2/lseek.2.html	2022-08-23 01:47:18 -07:00
chrislu	bf88006037	format	2022-08-23 01:20:45 -07:00
chrislu	9fce75607d	s3: report http.StatusOK if the directory is explicitly created ... fix https://github.com/seaweedfs/seaweedfs/issues/3457	2022-08-23 01:16:46 -07:00
famosss	911475526c	fix: TestCommandEcBalanceSmall Unit test fails when CommandEnv is nil (#3497)	2022-08-22 23:54:51 -07:00
famosss	7eb15b1969	fix:Handle preflight cors requests (#3496)	2022-08-22 23:39:34 -07:00
chrislu	676e27c589	shell: stop long running jobs if lock is lost	2022-08-22 14:12:23 -07:00
chrislu	601ba5fb68	master: disable locking if not leader	2022-08-22 14:11:48 -07:00
chrislu	57e7582c36	refactoring	2022-08-22 14:11:13 -07:00
chrislu	7ea2c24c5d	metadata processing use retry instead of retryForEver	2022-08-22 11:25:40 -07:00
Patrick Schmidt	f875031f06	Reuse readDataByFileHandle in Read call (#3482)	2022-08-22 08:24:06 -07:00
famosss	bdba3da2e4	Handle preflight cors requests (#3481)	2022-08-22 08:21:38 -07:00
chrislu	f4b52d4c52	fix format	2022-08-22 00:20:21 -07:00
chrislu	b4131a3a72	filer.remote.sync: parallelize uploading to remove object storage	2022-08-22 00:01:48 -07:00
chrislu	f6b92b8561	refactor	2022-08-22 00:00:51 -07:00
chrislu	3c8e95eeb3	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-21 23:26:09 -07:00
chrislu	8bac903d29	mount: add locking to prevent nil pointer ... fix https://github.com/seaweedfs/seaweedfs/issues/3479	2022-08-21 23:26:06 -07:00
qzh	74b53729e1	feat(weed.move): add a speed limit parameter of moving files (#3478) ... * feat(weed.move): add a speed limit parameter of moving files * fix(weed.move): set the default value of ioBytePerSecond to vs.compactionBytePerSecond Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-21 23:08:31 -07:00
chrislu	7807f6641a	minor	2022-08-21 22:54:56 -07:00
chrislu	c4e862e908	3.23	2022-08-21 19:13:26 -07:00
Guo Lei	81e7e6ceb3	the statistical methord for fileCount should stay same (#3477) ... the statistical methord for fileCount between should stay same They all equal to the entry count in .idx file. relate to commit(c7892bc)	2022-08-21 19:11:18 -07:00
chrislu	63fbf281c7	remove same file copying rage limitation	2022-08-21 17:16:21 -07:00
Patrick Schmidt	3a75d7f7aa	Implement copy_file_range syscall in FUSE (#3475) ... See the man page of copy_file_range: https://man7.org/linux/man-pages/man2/copy_file_range.2.html	2022-08-21 17:11:43 -07:00
chrislu	fef9c6a520	filer: do not always save files in "/etc" folder to filer store ... fix https://github.com/seaweedfs/seaweedfs/issues/3476	2022-08-21 17:02:16 -07:00
chrislu	2762154130	fix compilation	2022-08-21 16:09:23 -07:00
chrislu	fb5808e0c3	EC: with multiple volume locations, the ec rebuilding may fail	2022-08-21 15:04:50 -07:00
chrislu	5790d01c6f	EC: after ec encoding, the source ec shards may fail to purge if the volume server has multiple disk locations ... related to https://github.com/seaweedfs/seaweedfs/issues/3459	2022-08-21 14:52:37 -07:00
chrislu	8b3429858d	align memory for atomic read/write	2022-08-21 12:20:27 -07:00
Patrick Schmidt	2ef6ab998c	Avoid race conditions with current filer address (#3474) ... When multiple filer requests are in-flight and the current filer disappears and a new one is selected by the first goroutine, then there can be a lot of race conditions while retrieving the current filer. Therefore, load/save the current filer index atomically.	2022-08-21 12:18:13 -07:00
Patrick Schmidt	f49a9297c2	Fix hanging reads in chunk cacher (#3473) ... Sometimes when an unexpected error occurs the cacher would set an error and return. However, it would not broadcast the condition signal in that case, therefore leaving the goroutine that runs readChunkAt stuck forever. I figured that the condition is unnecessary because readChunkAt is acquiring a lock that is still held by the cacher goroutine anyway. Callees of startCaching have to wait for a WaitGroup which makes sure that readChunkAt can't acquire the lock before startCaching. This way readChunkAt can execute normally and check for the error.	2022-08-21 11:54:02 -07:00
chrislu	388f82f322	minor	2022-08-21 11:49:29 -07:00
chrislu	77e4b1376e	refactoring	2022-08-21 11:35:54 -07:00
Patrick Schmidt	3f758820c1	Fix FUSE server buffer leaks in file gaps (#3472) ... * Fix FUSE server buffer leaks in file gaps This change zeros read buffers when encountering file gaps during file/chunk reads in FUSE mounts. It prevents leaking internal buffers of the FUSE server which could otherwise reveal metadata, directory listings, file contents and other data related to FUSE API calls. The issue was that buffers are reused, but when a file gap was found the buffer was not zeroed accordingly and the existing data of the buffer was kept and returned. * Move zero logic into its own method	2022-08-21 11:33:58 -07:00
chrislu	c7892bc7c4	volume: file counter should be all files ... address https://github.com/seaweedfs/seaweedfs/pull/3388#issuecomment-1220466228	2022-08-20 23:35:31 -07:00
chrislu	f7e0a65e75	retry for all errors	2022-08-20 23:34:45 -07:00
chrislu	649555b223	adjust log	2022-08-20 23:34:07 -07:00
chrislu	973f6dd162	refactoring	2022-08-20 22:38:15 -07:00
chrislu	65ff7198fe	adjust type	2022-08-20 22:37:24 -07:00
chrislu	28b862f45f	use util.Retry to retry	2022-08-20 22:03:27 -07:00
chrislu	3bf8e772f8	webdav: retryable data chunk upload	2022-08-20 19:18:12 -07:00
chrislu	f8fa430257	filer: retryable data chunk upload	2022-08-20 19:15:44 -07:00
chrislu	4081d50607	filer sink: retryable data chunk uploading	2022-08-20 19:09:15 -07:00
chrislu	409f39390d	mount: retryable data chunk uploading	2022-08-20 19:04:36 -07:00
chrislu	6c8822f269	filer.copy: retryable file part upload	2022-08-20 18:59:57 -07:00
chrislu	a3553da7f7	add fileId to UploadWithRetry return result	2022-08-20 18:54:59 -07:00
chrislu	d49d0a9fc2	filer.copy: retryable upload	2022-08-20 18:50:57 -07:00
chrislu	689b4ecdcc	remove unused collection and replication from upload result	2022-08-20 18:14:57 -07:00
chrislu	e3f40d538d	cleaner code	2022-08-20 17:51:30 -07:00
chrislu	9f50b310a0	remove unused code	2022-08-20 17:51:12 -07:00
chrislu	11f99836c3	filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0) ... fix https://github.com/seaweedfs/seaweedfs/issues/3468	2022-08-19 23:00:56 -07:00
chrislu	fdd8c5d5e0	Filer if-modified-since check doesn't fire on "exact" #3467 ... fix https://github.com/seaweedfs/seaweedfs/issues/3467	2022-08-19 22:21:51 -07:00
chrislu	aac45f3e89	filer: retryable when error is not found	2022-08-19 03:36:15 -07:00
chrislu	58dd880322	minor	2022-08-18 23:51:08 -07:00
Eric Yang	b6f99c4d83	fix naming typo (#3465) ... Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-08-18 23:25:32 -07:00
chrislu	5702727db5	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-18 02:39:27 -07:00
chrislu	4d72a1ce6e	s3: remove deleted collection during metrics reporting ... fix https://github.com/seaweedfs/seaweedfs/issues/3453	2022-08-18 02:39:07 -07:00
Guo Lei	3172c33fa6	fix redundant type from array (#3462)	2022-08-18 02:25:25 -07:00
chrislu	42c6e52513	s3: fix regression on HEAD directory operation	2022-08-18 02:13:58 -07:00
chrislu	4573c99ae3	fix tests	2022-08-18 00:15:46 -07:00
chrislu	2b580a7566	also migrate jsonpb	2022-08-17 12:42:03 -07:00
chrislu	eaeb141b09	move proto package	2022-08-17 12:05:07 -07:00
chrislu	e259052a70	volume: fix metrics for volume request time	2022-08-17 04:29:31 -07:00
qzh	7fcfaf7bc9	fix(filer.sync): offset may be set to 0 (#3451) ... * fix(filer.sync): initializing the offset is related to the path * fix(filer.sync): the offset maybe to be set to 0. Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 23:43:52 -07:00
qzh	400f0c3e5d	fix(filer.sync): initializing the offset is related to the path (#3450) ... Co-authored-by: zhihao.qu <zhihao.qu@ly.com>	2022-08-15 21:56:47 -07:00
chrislu	fa4d0093e1	3.22	2022-08-15 16:48:23 -07:00
chrislu	93261f5199	Revert "Refactor for Sync method (#3426)" ... This reverts commit 670cb759f8. with the pr weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 18.486s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.025s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.864s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.110s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.414s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.203s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files] weed/storage () - (master) > weed/storage () - (master) > without the pr weed/storage () - (master) > weed/storage () - (master) > go test -count=1 ./... ok github.com/seaweedfs/seaweedfs/weed/storage 1.617s ? github.com/seaweedfs/seaweedfs/weed/storage/backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/backend/memory_map 0.026s ? github.com/seaweedfs/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/erasure_coding 0.906s ? github.com/seaweedfs/seaweedfs/weed/storage/idx [no test files] ok github.com/seaweedfs/seaweedfs/weed/storage/needle 0.202s ok github.com/seaweedfs/seaweedfs/weed/storage/needle_map 24.533s ok github.com/seaweedfs/seaweedfs/weed/storage/super_block 0.280s ? github.com/seaweedfs/seaweedfs/weed/storage/types [no test files] ? github.com/seaweedfs/seaweedfs/weed/storage/volume_info [no test files]	2022-08-15 15:35:31 -07:00
chrislu	9c588d4010	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-15 10:03:56 -07:00
chrislu	96caf21d09	less verbose log	2022-08-15 10:03:52 -07:00
Andrey Triumfov	31faa6d43d	Remove duplicate slashes in object path to prevent 500 errors (#3442)	2022-08-15 08:19:28 -07:00
chrislu	cb476a53ff	remove logs	2022-08-15 01:05:35 -07:00
chrislu	7c029b2183	3.21	2022-08-15 00:32:15 -07:00
Chris Lu	e88392b50f	Fix s3 pagination (#3436) ... * Revert previous changes * s3: use cursor to track tree traversal fix https://github.com/seaweedfs/seaweedfs/issues/3166 * special cases for empty prefix and empty directory * use constants * address empty folder * undo local changes * fix IsTruncated * adjust counting directories * fix cases when prefix is a directory * s3: handle directory object works for aws --endpoint-url http://127.0.0.1:8333/ s3api list-objects-v2 --bucket test --prefix "fakedir"	2022-08-15 00:30:19 -07:00
chrislu	7457c746f0	s3: fix aws s3api head-object	2022-08-14 23:52:35 -07:00
chrislu	789d9dbfe1	webdav: fix nil ... fix https://github.com/seaweedfs/seaweedfs/issues/3440	2022-08-14 23:33:30 -07:00
Konstantin Lebedev	6199d9b923	rand start garbageThreshold and reset metric ReplicaPlacementMismatch (#3396) ... * rand start garbageThreshold reset metrics MasterReplicaPlacementMismatch * vacuum through sleep	2022-08-12 13:11:09 -07:00
Konstantin Lebedev	4d4cd0948d	avoid infinite loop WaitUntilConnected() (#3431) ... https://github.com/seaweedfs/seaweedfs/issues/3421	2022-08-11 15:03:26 -07:00
Eric Yang	986daecbfe	add upload response (#3428) ... * add upload response * fixing comments Co-authored-by: root <root@HQ-10MSTD3EY.roblox.local>	2022-08-10 22:43:09 -07:00
chrislu	ae93c966d9	ensure memory is aligned ... fix https://github.com/seaweedfs/seaweedfs/issues/3427	2022-08-10 22:27:13 -07:00
Rain Li	670cb759f8	Refactor for Sync method (#3426)	2022-08-10 08:21:57 -07:00
chrislu	662ec97602	mount: adjust on du stats reporting ... fix https://github.com/seaweedfs/seaweedfs/issues/3424	2022-08-09 20:21:01 -07:00
Abirdcfly	b0633716b7	delete minor unreachable code (#3423) ... Signed-off-by: Abirdcfly <fp544037857@gmail.com>	2022-08-09 01:24:43 -07:00
chrislu	0854171d22	3.20	2022-08-07 14:42:28 -07:00
chrislu	d8e69df295	adjust comments	2022-08-07 11:35:33 -07:00
chrislu	0aeec04c31	quicker to adapt to pattern change	2022-08-07 10:14:01 -07:00
chrislu	928d29af9e	fix wrong logic about reader isRandomMode()	2022-08-07 10:13:04 -07:00
chrislu	d3d52b1818	shell: fs.meta.load add quieter mode ... fix https://github.com/seaweedfs/seaweedfs/issues/3414	2022-08-07 01:42:15 -07:00
chrislu	358ccb5e0e	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-07 01:34:36 -07:00
chrislu	67814a5c79	refactor and fix strings.Split	2022-08-07 01:34:32 -07:00
John W Higgins	3afda0c89c	Allow postgresql to use standard environment variables for connection (#3413)	2022-08-07 00:58:53 -07:00
chrislu	1a4bf0dcb5	filer.sync: parallelize the filer.sync	2022-08-07 00:56:15 -07:00
chrislu	02b38f894b	debug	2022-08-07 00:55:49 -07:00
chrislu	0e9478488d	filer.sync: fix when excluded paths is empty	2022-08-07 00:55:34 -07:00
chrislu	8a880a139d	mount: fix truncate operation ... fix https://github.com/seaweedfs/seaweedfs/issues/2609	2022-08-06 13:58:45 -07:00
LHHDZ	84ec68e11a	Add download speed limit support (#3408)	2022-08-05 01:16:42 -07:00
chrislu	b278bb24d3	mount: adjust df stats reporting when close to the limit ... fix https://github.com/seaweedfs/seaweedfs/issues/3407	2022-08-05 00:31:41 -07:00
Konstantin Lebedev	4d08393b7c	filer prefer volume server in same data center (#3405) ... * initial prefer same data center https://github.com/seaweedfs/seaweedfs/issues/3404 * GetDataCenter * prefer same data center for ReplicationSource * GetDataCenterId * remove glog	2022-08-04 17:35:00 -07:00
Konstantin Lebedev	22181dd018	refactor FilerRequest metrics (#3402) ... * refactor FilerRequest metrics * avoid double count proxy * defer to	2022-08-04 01:44:54 -07:00
chrislu	bd13a7968f	remove unused symlink resolving	2022-08-04 01:35:18 -07:00
chrislu	88945d9954	mount: do not follow soft link with xattr	2022-08-04 01:21:15 -07:00
chrislu	90db4b5a44	minor	2022-08-03 00:10:47 -07:00
Chris Lu	fc8035d672	Merge pull request #3394 from kmlebedev/metricsReplicatedWrite ... Detailed metrics VolumeServerRequestHistogram for writing to disk and replication	2022-08-01 13:31:01 -07:00
Konstantin Lebedev	3ffa4ba91d	clean	2022-08-02 01:30:38 +05:00
Konstantin Lebedev	2a5629dd28	align	2022-08-02 01:29:44 +05:00
Konstantin Lebedev	df9cc31a0a	refactor	2022-08-02 01:18:26 +05:00
Konstantin Lebedev	0a5d3c3cf6	rename to localWrite and replicatedWrite	2022-08-01 23:09:09 +05:00
chrislu	bd5c9904be	Merge branch 'master' of https://github.com/seaweedfs/seaweedfs	2022-08-01 09:40:05 -07:00
chrislu	e26d1cf4ef	fix test	2022-08-01 09:40:01 -07:00
Konstantin Lebedev	fc65122766	rename to LoadAvg_1M	2022-08-01 21:32:21 +05:00
Konstantin Lebedev	ab50f21ad9	Detailed metrics VolumeServerRequestHistogram for writing to disk and replication	2022-08-01 21:16:57 +05:00
Konstantin Lebedev	2f85e31f5c	Merge branch 'master' into gentle_vacuum	2022-08-01 20:51:58 +05:00
Konstantin Lebedev	5209ebbeef	remove percent	2022-08-01 20:40:38 +05:00
Chris Lu	b59bc607bf	Merge pull request #3338 from kmlebedev/issues/3083 ... rollback over onPeerUpdate implementation of automatic clean-up of failed servers in favor of synchronous ping	2022-08-01 08:23:10 -07:00
Konstantin Lebedev	78cbd8002f	revert Sleep	2022-08-01 20:21:23 +05:00
Konstantin Lebedev	cd5c7ad052	move to github.com/seaweedfs/seaweedfs	2022-08-01 16:36:32 +05:00
Konstantin Lebedev	634bc97840	fix conflicts	2022-08-01 14:47:00 +05:00
Konstantin Lebedev	3c75479e2b	Merge branch 'master' into gentle_vacuum ... # Conflicts: # weed/pb/messaging_pb/messaging.pb.go # weed/pb/messaging_pb/messaging_grpc.pb.go # weed/pb/s3_pb/s3.pb.go # weed/pb/volume_server_pb/volume_server.pb.go # weed/server/volume_grpc_vacuum.go	2022-08-01 14:45:22 +05:00
Konstantin Lebedev	c0d92f61a1	comment	2022-08-01 14:40:42 +05:00
Konstantin Lebedev	1d29f67c02	revert disk stats	2022-08-01 14:29:41 +05:00
Konstantin Lebedev	a98f6d66a3	rollback over onPeerupdate implementation of automatic clean-up of failed servers in favor of synchronous ping	2022-08-01 12:51:41 +05:00
chrislu	036566629a	filer.sync: fix synchronization logic in active-active mode ... fix https://github.com/seaweedfs/seaweedfs/issues/3328	2022-08-01 00:06:18 -07:00
chrislu	303bd067b5	Revert "rename: delete source entry metadata only, skipping hard links" ... This reverts commit 03466f955e. fix https://github.com/seaweedfs/seaweedfs/issues/3386	2022-07-31 22:51:41 -07:00
chrislu	03c6f978f4	3.19	2022-07-31 17:16:51 -07:00
chrislu	afef014832	scaffold	2022-07-31 13:23:44 -07:00
Chris Lu	0bb3ba17ae	Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy ... Add healthy indicator for raft status	2022-07-30 11:55:30 -07:00
Patrick Schmidt	1a4a36d510	Add healthy indicator for raft status	2022-07-30 19:34:26 +02:00
chrislu	cb096d1f53	minor	2022-07-30 09:52:52 -07:00
chrislu	0423d50106	wip note	2022-07-30 09:36:27 -07:00
chrislu	0f50fcb96a	moving	2022-07-29 23:39:11 -07:00
chrislu	13b9a52f80	fix deprecated functions	2022-07-29 01:34:39 -07:00
chrislu	be0ec7bc2f	fix test	2022-07-29 01:15:26 -07:00
chrislu	1f161f53dc	add back rack	2022-07-29 01:15:19 -07:00
chrislu	26dbc6c905	move to https://github.com/seaweedfs/seaweedfs	2022-07-29 00:17:28 -07:00
Konstantin Lebedev	2f0dda384d	vacuum show LA	2022-07-29 11:59:33 +05:00
chrislu	de3afd4199	fix rebasing	2022-07-28 23:25:46 -07:00
chrislu	06cd491abc	add message batch	2022-07-28 23:24:38 -07:00
chrislu	7576c244c4	add flatbuffer serde for message	2022-07-28 23:24:38 -07:00
chrislu	a3fae7cd16	add an empty placeholder for mq.topic.list	2022-07-28 23:24:38 -07:00
chrislu	aa4a22ad47	segment serde	2022-07-28 23:24:38 -07:00
chrislu	74f60f246f	dynamically connect to a filer	2022-07-28 23:24:38 -07:00
chrislu	9f479aab98	allocate brokers to serve segments	2022-07-28 23:24:38 -07:00
chrislu	bb01b68fa0	refactor	2022-07-28 23:24:38 -07:00
chrislu	94b8c42b2c	clean up	2022-07-28 23:24:38 -07:00
chrislu	8060fdcac5	remove old code	2022-07-28 23:24:38 -07:00
chrislu	f25e273e32	display data center and rack in cluster.ps	2022-07-28 23:22:52 -07:00
chrislu	68065128b8	add dc and rack	2022-07-28 23:22:51 -07:00
chrislu	8d31e73ffd	mq broker adds data center and rack	2022-07-28 23:22:07 -07:00
chrislu	87ccef72db	cluster.ps can list brokers	2022-07-28 23:22:07 -07:00
chrislu	ac66227798	renaming	2022-07-28 23:22:07 -07:00
chrislu	21b6b07dd8	renaming	2022-07-28 23:22:06 -07:00
chrislu	8ca7d1ef17	rename to SeaweedMQ	2022-07-28 23:22:06 -07:00
Chris Lu	3e7e922d16	Merge pull request #3378 from vanhtuan0409/change-put-policy ... PutUserAction should completely replace identity actions	2022-07-28 19:43:25 -07:00
chrislu	89e276de64	fix loop	2022-07-28 18:06:49 -07:00
chrislu	a458bee9f2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-28 16:32:02 -07:00
chrislu	e0eda52c54	mount: ensure symlink parent directory is tracked ... fix https://github.com/chrislusf/seaweedfs/issues/3373	2022-07-28 16:32:00 -07:00
Tuan Vuong	1d9a888be9	PutUserAction should completely replace identity actions	2022-07-28 17:02:56 +07:00
zzq09494	01b7aa79fa	fix: Buckets are not created and deleted correctly on the filer with the same signature when they are created and deleted	2022-07-28 12:04:37 +08:00
chrislu	ca836568ac	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-27 12:12:43 -07:00
chrislu	3828b8ce87	"github.com/chrislusf/raft" => "github.com/seaweedfs/raft"	2022-07-27 12:12:40 -07:00
Chris Lu	b7e4e26778	Merge pull request #3371 from askeipx/issues/3341 ... delete disk_size metrics when collection deleted	2022-07-27 11:25:42 -07:00
Konstantin Lebedev	7e09a548a6	exclude directories to sync on filer	2022-07-27 19:22:57 +05:00
Evgeny Kuzhelev	ac5ce31278	leave notion to refactor after prometheus upgrade	2022-07-27 16:48:56 +05:00
Evgeny Kuzhelev	47c72e6f35	remove all (currently existing) collection volume metrics	2022-07-27 16:31:49 +05:00
Evgeny Kuzhelev	709c83716c	delete disk_size metrics when collection deleted	2022-07-27 15:02:41 +05:00
chrislu	f401b996eb	mount: fix symlink size reporting	2022-07-26 23:12:25 -07:00
Konstantin Lebedev	c88ea31f62	fix RUnlock of unlocked RWMutex	2022-07-26 12:57:07 +05:00
Konstantin Lebedev	da9d3e8f6c	refactor	2022-07-26 11:56:45 +05:00
Konstantin Lebedev	046c3d5ad4	fix logic else brake	2022-07-26 11:47:11 +05:00
chrislu	475185fb72	3.18	2022-07-25 22:41:38 -07:00
chrislu	ec0edb1ac4	filer: fix wrong logic during read	2022-07-25 22:40:00 -07:00
guosj	cc7a4b0a6e	correct comment	2022-07-25 11:46:41 +08:00
chrislu	33cff61992	fix glog test	2022-07-24 19:13:54 -07:00
chrislu	dbe4849ffd	3.17	2022-07-24 18:46:50 -07:00
chrislu	b98c0b4bf4	disable symlink translation for now	2022-07-24 18:46:32 -07:00
chrislu	2dc5d4adac	disable symlink translation for now	2022-07-24 18:24:34 -07:00
chrislu	3d2bbf96d2	adjust glog format	2022-07-24 17:29:08 -07:00
chrislu	2807fe08fc	adjust glog format	2022-07-24 16:44:58 -07:00
chrislu	2b64916d3e	set status to fuse.OK	2022-07-24 15:30:55 -07:00
chrislu	7386cf6a06	mount: resolve symlink	2022-07-24 15:11:24 -07:00
chrislu	40089b6753	maintain node lookup counter	2022-07-24 13:43:11 -07:00
chrislu	d24db396cc	fix removing path from inode2path	2022-07-23 18:20:29 -07:00
chrislu	41eeb4deef	do not add new inode during link	2022-07-23 13:37:07 -07:00
chrislu	02c2d81cde	fix removePathFromInode2Path	2022-07-23 13:21:42 -07:00
chrislu	4d5dc55766	mount: better support hard links	2022-07-23 12:57:00 -07:00
chrislu	ebda0689ab	use linux default max name length	2022-07-23 10:51:06 -07:00
chrislu	64f3d6fb6e	metadata subscription uses client epoch	2022-07-23 10:50:28 -07:00
chrislu	2c8818351f	mount: comment out stuck fuse inode notify during renaming	2022-07-22 18:17:34 -07:00
chrislu	a5c94acd4f	mount: check possible nil entry case	2022-07-22 15:33:37 -07:00
Chris Lu	0716092b39	Merge pull request #3350 from shichanglin5/optimize_masterclient_vidmap ... Solve the problem that `LookupFileId` lookup urls is empty due to lea…	2022-07-22 02:57:50 -07:00
LHHDZ	994a2dec78	fix vid_map_test tests	2022-07-22 17:54:57 +08:00
LHHDZ	58c02d6429	Solve the problem that LookupFileId lookup urls is empty due to leader switching ... The vidMap structure is modified to a linked list structure (the length is limited to 5). When the vidMap is reset, the current vidMap is added to the new vidMap as a cache node. When the query locations is empty, the cache node is searched to avoid problems when the master switches leaders.	2022-07-22 17:22:38 +08:00
ningfd	58f2dd6740	change topology mapping definition, improve spelling	2022-07-22 17:18:56 +08:00
ningfd	6f882eb354	optimiz: master ui will render data in order	2022-07-22 14:37:49 +08:00
chrislu	7a6c559ab4	fix Change replication via volume.configure.replication by collection ... fix https://github.com/chrislusf/seaweedfs/issues/3346	2022-07-21 22:01:05 -07:00
chrislu	fc8241fb5e	leveldb3: add instant dropping bucket	2022-07-21 18:48:51 -07:00
chrislu	c93f7ffa44	explicit bucket aware declaration	2022-07-21 18:23:53 -07:00
Konstantin Lebedev	3c42814b58	avoid deadlock	2022-07-21 17:15:10 +05:00
Konstantin Lebedev	93ca87b7cb	use safe onPeerUpdateDoneCns	2022-07-21 15:51:14 +05:00
chrislu	6147b61b0a	mount: fix concurrent map read and map write ... fix https://github.com/chrislusf/seaweedfs/issues/3344	2022-07-21 01:38:26 -07:00
Chris Lu	9ec0d1caaa	Merge pull request #3337 from guol-fnst/loading_volume	2022-07-20 09:06:06 -07:00
Konstantin Lebedev	7875470e74	onPeerUpdateGoroutineCount use int32	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	c5189c343b	remove ticker ... update the topology before each file	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	e2d991d8d0	ticker.Stop	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	de4fcc0e2c	sync update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	fa88dff7ce	update otherNodes	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	72dca31cfa	fix update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	884ffbafee	clouse background update	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	39eaf426f8	fix TestVolumeServerEvacuate	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	b5e5f6f55a	update topologyInfo	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	867269cdcf	help rack	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	6f764e1014	volume server evacuate from rack	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	ba0e3ce5fa	volume server evacuate to target server	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	b6471ecd75	err msg with duplicated local subscription detected move to log level 1 ... https://github.com/chrislusf/seaweedfs/issues/3320	2022-07-20 18:40:50 +05:00
Konstantin Lebedev	6c390851e7	fix design	2022-07-20 18:08:12 +05:00
Tuan Vuong	312e13416b	filter duplicated action	2022-07-20 16:43:50 +07:00
guol-fnst	ac694f0c8f	rename parameter and reuse functions ... rename milestone to watermark	2022-07-20 17:00:40 +08:00
Chris Lu	4820b594a7	Merge pull request #3298 from kmlebedev/evacuateTargetServer ... volume server evacuate to target server	2022-07-19 13:56:49 -07:00
Konstantin Lebedev	d3f7c09c03	remove ticker ... update the topology before each file	2022-07-20 00:54:23 +05:00
Konstantin Lebedev	11e393dbe7	err msg with duplicated local subscription detected move to log level 1 ... https://github.com/chrislusf/seaweedfs/issues/3320	2022-07-20 00:45:13 +05:00
Konstantin Lebedev	f6a966b4fc	add waiting log message	2022-07-20 00:31:57 +05:00
Konstantin Lebedev	6cfbfb0849	check for ping before deleting raft server ... https://github.com/chrislusf/seaweedfs/issues/3083	2022-07-20 00:04:12 +05:00
chrislu	2ae3f812f8	minor	2022-07-19 11:43:31 -07:00
Chris Lu	9667588af0	Merge pull request #3318 from kmlebedev/issues/3310 ... Use fallback if urls are not found	2022-07-19 11:38:43 -07:00
Chris Lu	f0d560060f	Merge pull request #3336 from kmlebedev/raftSrvRemove ... raftServerRemoval typo and logs	2022-07-19 11:32:17 -07:00
guol-fnst	91285bb51d	remove uncessary code	2022-07-19 20:00:44 +08:00
guol-fnst	b9256e0b34	optimiz	2022-07-19 16:58:53 +08:00
Konstantin Lebedev	f419d5643a	fix typo ... add remove logs	2022-07-19 11:50:52 +05:00
guol-fnst	bec9f79659	use snapshot	2022-07-19 11:14:41 +08:00
Konstantin Lebedev	d422e7769c	ticker.Stop	2022-07-18 16:38:19 +05:00
Konstantin Lebedev	73a0dea16b	sync update topologyInfo	2022-07-18 16:27:02 +05:00
chrislu	56ec89625a	3.16	2022-07-18 00:53:05 -07:00
Konstantin Lebedev	2b4112e462	update otherNodes	2022-07-18 11:32:28 +05:00
Konstantin Lebedev	3c2774ec3d	fix update topologyInfo	2022-07-18 01:46:31 +05:00
Konstantin Lebedev	7b1497ee63	Use BackoffSchedule for getLookupFileId	2022-07-15 16:05:35 +05:00
Konstantin Lebedev	01996bccf8	Use fallback if urls are not found	2022-07-15 15:29:15 +05:00
Konstantin Lebedev	4d5144e50d	clouse background update	2022-07-15 13:51:08 +05:00
chrislu	1db012485f	mount: fix bug during busy writes ... fix https://github.com/chrislusf/seaweedfs/issues/3315	2022-07-15 01:03:17 -07:00
chrislu	aca20cd9f4	more logging related to filer metadata subscription	2022-07-15 00:42:13 -07:00
Chris Lu	abeb7f65f8	Merge pull request #3314 from shichanglin5/fix_ETag ... File upload should succeed should return `ETag` instead of `Etag`	2022-07-14 15:35:41 -07:00
chrislu	fa61074513	add clientId logging	2022-07-14 12:27:34 -07:00
chrislu	fbd8f868a1	filer may have trouble to re-connect ... clientId is used twice: one for local metadata subscription, one for combined metadata subscription.	2022-07-14 12:15:31 -07:00
LHHDZ	c08f6d0ff7	File upload should succeed should return ETag instead of Etag ... at `/go-1.18.1/src/net/textproto/header.go:39`, header is canonicalized by `textproto.CanonicalMIMEHeaderKey`	2022-07-14 19:45:01 +08:00
Chris Lu	957c9a2eb2	Merge pull request #3309 from guol-fnst/loading_volume ... optimiz concurrency	2022-07-13 23:12:24 -07:00
guol-fnst	300b383cdf	use 10 or numCPU workers if env is not found	2022-07-14 14:06:46 +08:00
guol-fnst	308a48c0c2	optimiz concurrency ... user can customize number of workers via env "GOMAXPROCS"	2022-07-14 09:57:25 +08:00
Chris Lu	1a130125de	Merge pull request #3308 from guo-sj/check_duplicate_ak	2022-07-13 09:23:37 -07:00
guosj	354b7bdff0	replace errors.New(fmt.Sprintf(...)) with fmt.Errorf()	2022-07-13 22:49:03 +08:00
chrislu	289402a741	writer pattern: similar changes to reader pattern	2022-07-13 02:30:44 -07:00
guosj	ab1b9697e6	supplement check duplicate accesskey	2022-07-13 17:28:20 +08:00
chrislu	6e90f7bdd0	detect sequential mode with a counter	2022-07-13 02:20:03 -07:00
chrislu	578c8aea0e	typo	2022-07-13 02:18:31 -07:00
chrislu	1451b389a4	Avoid fatal error: sync: Unlock of unlocked RWMutex ... fix https://github.com/chrislusf/seaweedfs/issues/3306	2022-07-13 00:58:15 -07:00
石昌林	392da4e038	Make the prompt information clearer	2022-07-13 11:39:45 +08:00
石昌林	ad5e8f68ec	Check whether there is a duplicate accessKey when modifying iam	2022-07-13 11:36:11 +08:00
Chris Lu	fba5219dab	Merge pull request #3303 from guol-fnst/loading_volume ... speeding up loading volumes	2022-07-12 03:01:24 -07:00
Konstantin Lebedev	6622240df7	fix TestVolumeServerEvacuate	2022-07-12 14:56:34 +05:00
chrislu	860c207c94	typo	2022-07-12 02:56:19 -07:00
Konstantin Lebedev	8372721a62	update topologyInfo	2022-07-12 13:47:21 +05:00
guol-fnst	26313060a3	speeding up loading volumes	2022-07-12 16:19:33 +08:00
Konstantin Lebedev	ee95d23a22	help rack	2022-07-12 11:56:58 +05:00
Konstantin Lebedev	087fa1347f	volume server evacuate from rack	2022-07-12 11:33:08 +05:00
Chris Lu	de97ba94c3	Merge pull request #3302 from guo-sj/fix_put_user_policy	2022-07-11 21:15:37 -07:00
guosj	b3076909d1	refactor code	2022-07-12 11:25:36 +08:00
Konstantin Lebedev	4236c36599	volume server evacuate to target server	2022-07-11 16:58:15 +05:00
Chris Lu	7929a50327	Merge pull request #3296 from guo-sj/fix_put_user_policy ... refine PutUserPolicy	2022-07-11 01:49:25 -07:00
guosj	7dbf19d09a	refine PutUserPolicy	2022-07-11 16:38:41 +08:00
chrislu	37578929d4	fix compilation	2022-07-11 01:30:00 -07:00
chrislu	93ccc6e05f	3.15	2022-07-11 00:50:07 -07:00
chrislu	f87952a411	remove locks	2022-07-11 00:25:20 -07:00
guosj	5469efa336	close SkipCheckParentDirectory flag	2022-07-11 09:44:48 +08:00
chrislu	3d0defa663	fix compilation	2022-07-08 19:16:58 -07:00
chrislu	6f00c7f061	comments	2022-07-08 19:07:31 -07:00
chrislu	d685b94102	mount: rename also invalidate source inode	2022-07-08 19:07:03 -07:00
chrislu	48382676d2	fix filtering by volume id	2022-07-08 10:29:24 -07:00
liubaojiang	4e83e92156	add subscriber clientId if it is the first time	2022-07-08 20:27:25 +08:00
guosj	5b3192344d	fix iam CreatePolicy error	2022-07-08 17:42:21 +08:00
chrislu	e2ecf137a2	fix resetting fh.reader	2022-07-08 01:04:15 -07:00
chrislu	185332b91f	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-07-08 00:29:41 -07:00
chrislu	28add5a534	mount: fix racing conditions ... prevent wrong reading when the SingleChunkCacher is started, but not finished yet	2022-07-08 00:29:39 -07:00
duanhongyi	1ceab96aba	filer tikv support tls	2022-07-08 14:23:06 +08:00
chrislu	a85ed3fe8f	minor	2022-07-07 22:27:24 -07:00
chrislu	2ea18cdcc8	remove dead code	2022-07-07 22:26:03 -07:00
chrislu	b7de5c6c43	shell: remove unused filer parameter from shell.toml	2022-07-07 15:01:23 -07:00
chrislu	f2f0482dd3	mount: random read also try to use the local cache first	2022-07-07 11:50:28 -07:00
chrislu	05c53820b9	mount: file handler release memory	2022-07-07 11:49:40 -07:00
chrislu	69ef6459c3	S3 ListObjectsV2 fix pagination under bucket root ... fix https://github.com/chrislusf/seaweedfs/issues/3166	2022-07-06 02:28:34 -07:00
Konstantin Lebedev	785223e587	rabbitpubsub enable durable	2022-07-06 10:05:29 +05:00
chrislu	57975b93ff	adjust used size reporting	2022-07-04 13:47:42 -07:00
chrislu	a79f5bd0d5	adjust used size reporting	2022-07-04 13:46:32 -07:00
chrislu	3c79c77056	3.14	2022-07-04 13:15:49 -07:00
guol-fnst	5bc895673e	add getting bucket quota	2022-07-04 15:36:44 +08:00
guol-fnst	dd4664e610	fix usage	2022-07-04 11:31:39 +08:00
chrislu	141f662734	edge case checking when volume server does not have capacity to balance ... fix https://github.com/chrislusf/seaweedfs/issues/3257	2022-07-01 12:16:18 -07:00
chrislu	bee3a7c798	fix tests	2022-07-01 12:09:42 -07:00
Chris Lu	ddb0f6bc74	Merge pull request #3258 from kmlebedev/issues_3255 ... volumeServer.evacuate fix select empty node	2022-07-01 02:36:30 -07:00
Konstantin Lebedev	3d5f1cbc5e	fix pagination	2022-07-01 14:17:49 +05:00
chrislu	06a8b174b5	also remove Sync() for idx file	2022-06-30 13:50:53 -07:00
chrislu	52580743b9	stuck with file.Sync() ... starting to get weed/storage () - (master) > go test ./... I0630 12:19:54 65819 needle_map_metric_test.go:26] FileCount expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:27] DeletedSize expected 1648 actual 1648 I0630 12:19:54 65819 needle_map_metric_test.go:28] ContentSize expected 10000 actual 10000 I0630 12:19:54 65819 needle_map_metric_test.go:29] DeletedCount expected 1648 actual 1959 I0630 12:19:54 65819 needle_map_metric_test.go:30] MaxFileKey expected 10000 actual 10000 I0630 12:19:54 65819 volume_loading.go:136] loading index /var/folders/jv/0rlhdck17jzgt7b3hcnq63mc0000gp/T/TestCompaction909350722/001/1.idx to memory I0630 12:20:36 65819 volume_vacuum.go:98] Committing volume 1 vacuuming... panic: test timed out after 10m0s goroutine 61 [running]: testing.(*M).startAlarm.func1() /usr/local/go/src/testing/testing.go:2029 +0x8e created by time.goFunc /usr/local/go/src/time/sleep.go:176 +0x32 goroutine 1 [chan receive, 10 minutes]: testing.(*T).Run(0xc0000cd520, {0x1c09bde?, 0x5ab798ea5c102?}, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1487 +0x37a testing.runTests.func1(0xc0000cd520?) /usr/local/go/src/testing/testing.go:1839 +0x6e testing.tRunner(0xc0000cd520, 0xc00039fcd8) /usr/local/go/src/testing/testing.go:1439 +0x102 testing.runTests(0xc0000dadc0?, {0x25b4460, 0x6, 0x6}, {0x2855108?, 0x40?, 0x25c4b80?}) /usr/local/go/src/testing/testing.go:1837 +0x457 testing.(*M).Run(0xc0000dadc0) /usr/local/go/src/testing/testing.go:1719 +0x5d9 main.main() _testmain.go:57 +0x1aa goroutine 21 [chan receive]: github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).flushDaemon(0x0?) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:883 +0x6a created by github.com/chrislusf/seaweedfs/weed/glog.init.0 /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:410 +0x1bf goroutine 55 [syscall]: syscall.syscall(0x44ac0f96?, 0xc00007ad80?, 0xc00098b480?, 0x10588df?) /usr/local/go/src/runtime/sys_darwin.go:22 +0x4e syscall.fcntl(0x100000001?, 0xc00098b4b0?, 0x1069a7c?) /usr/local/go/src/syscall/zsyscall_darwin_amd64.go:319 +0x30 internal/poll.(*FD).Fsync.func1(...) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:18 internal/poll.ignoringEINTR(...) /usr/local/go/src/internal/poll/fd_posix.go:74 internal/poll.(*FD).Fsync(0xc00038f1e0?) /usr/local/go/src/internal/poll/fd_fsync_darwin.go:17 +0xfc os.(*File).Sync(0xc00012a030) /usr/local/go/src/os/file_posix.go:168 +0x4e github.com/chrislusf/seaweedfs/weed/storage/backend.(*DiskFile).Sync(...) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/backend/disk_file.go:84 github.com/chrislusf/seaweedfs/weed/storage.(*Volume).makeupDiff(0xc0000bb440, {0xc000633a40, 0x52}, {0xc000633aa0, 0x52}, {0xc000633b00, 0x52}, {0xc000633b60, 0x52}) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:295 +0x12fa github.com/chrislusf/seaweedfs/weed/storage.(*Volume).CommitCompact(0xc0000bb440) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum.go:119 +0x3e8 github.com/chrislusf/seaweedfs/weed/storage.TestCompaction(0xc00025a000) /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_vacuum_test.go:89 +0x305 testing.tRunner(0xc00025a000, 0x1c67bd8) /usr/local/go/src/testing/testing.go:1439 +0x102 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:1486 +0x35f goroutine 56 [chan receive, 10 minutes]: github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker.func1() /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:244 +0x41a created by github.com/chrislusf/seaweedfs/weed/storage.(*Volume).startWorker /Users/chrislu/go/src/github.com/chrislusf/seaweedfs/weed/storage/volume_write.go:234 +0x56 FAIL github.com/chrislusf/seaweedfs/weed/storage 600.194s ? github.com/chrislusf/seaweedfs/weed/storage/backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/backend/memory_map (cached) ? github.com/chrislusf/seaweedfs/weed/storage/backend/s3_backend [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/erasure_coding (cached) ? github.com/chrislusf/seaweedfs/weed/storage/idx [no test files] ok github.com/chrislusf/seaweedfs/weed/storage/needle (cached) ok github.com/chrislusf/seaweedfs/weed/storage/needle_map (cached) ok github.com/chrislusf/seaweedfs/weed/storage/super_block (cached) ? github.com/chrislusf/seaweedfs/weed/storage/types [no test files] ? github.com/chrislusf/seaweedfs/weed/storage/volume_info [no test files] FAIL	2022-06-30 12:32:55 -07:00
Konstantin Lebedev	5ed8165161	fix logic ... add option targetServer https://github.com/chrislusf/seaweedfs/issues/3255	2022-06-30 17:58:24 +05:00
石昌林	200a862701	fixed volume xx not found caused by missing VolumeLocation events ... When the requested master node is the leader, `VolumeLocation` or `ClusterNodeUpdate` may be returned here. If it is `VolumeLocation`, the update will be performed while resetting the vidMap, otherwise the event will be lost	2022-06-30 13:41:56 +08:00
Chris Lu	4ae9884a6e	Merge pull request #3254 from shichanglin5/fix_subscribe_meta ... [Critical] Avoid synchronizing from scratch due to err	2022-06-29 12:36:28 -07:00
Chris Lu	b98556cff5	Merge pull request #3253 from kmlebedev/constMultipartUploadsFolder ... use const multipart uploads folder	2022-06-29 12:09:03 -07:00
石昌林	2b5b37231b	Make sure that onChangeEvent is called successfully before updating lastTsNs	2022-06-29 23:57:07 +08:00
石昌林	e17e6ed542	Avoid synchronizing from scratch due to err	2022-06-29 23:31:04 +08:00
Konstantin Lebedev	bcbdc4cb37	use const multipart uploads folder ... avoid error bucket NotEmpty if multipart uploads folder exist	2022-06-29 16:21:16 +05:00
chrislu	a8abab2412	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-29 02:44:14 -07:00
chrislu	449ecc8890	remove max connection age ... related to https://github.com/chrislusf/seaweedfs/pull/3226	2022-06-29 02:44:12 -07:00
Chris Lu	c6e6e303db	Merge pull request #3200 from lapshin-vitaly/bufix/validate-tags-on-copy ... validate tags on copy object and add regex for validating tags	2022-06-29 01:37:14 -07:00
Lapshin Vitaliy	7c3d9d0535	remove println	2022-06-29 10:43:02 +03:00
Lapshin Vitaliy	61b2e3f9a2	fix return	2022-06-28 18:31:46 +03:00
Lapshin Vitaliy	606252472c	fix return	2022-06-28 18:24:03 +03:00
Lapshin Vitaliy	e969370913	fix parseTagging function and add test for validate tags	2022-06-28 14:48:55 +03:00
Chris Lu	a79bf9d92a	Merge pull request #3252 from guol-fnst/update_tbk ... remove hdfs, add tikv for full version update	2022-06-27 22:37:41 -07:00
guol-fnst	74c9f39d48	remove hdfs, add tikv for full version update	2022-06-28 12:35:07 +08:00
石昌林	e1b94eb6b9	fix error: concurrent map writes when add or remove cluster node	2022-06-28 10:34:59 +08:00
chrislu	4b1f48a399	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-27 12:40:50 -07:00
chrislu	4a65159250	fix reading time	2022-06-27 12:40:47 -07:00
xdadrm	3a42269da8	Fixes WebDAV 0-bytes files ... Fixes the issue where files created via WebDAV show as 0-bytes size when read via fuse.	2022-06-27 20:19:21 +02:00
Chris Lu	4be406a095	Merge pull request #3249 from kmlebedev/issues_2589 ... avoid set currentMaster svc.local k8s discovery service domains	2022-06-27 11:05:07 -07:00
Konstantin Lebedev	9bff097b01	currentMaster redirected	2022-06-27 21:55:16 +05:00
Konstantin Lebedev	6c20a3b622	avoid set currentMaster k8s svc.local discoveruy service domains ... https://github.com/chrislusf/seaweedfs/issues/2589	2022-06-27 21:47:05 +05:00
Lapshin Vitaliy	1a5981d583	fix test	2022-06-27 18:46:21 +03:00
garenchan	8aa19577f4	fix 3238: handle errors for GenerateDirUuid method	2022-06-27 22:04:50 +08:00
Lapshin Vitaliy	d7c3493d15	Merge branch 'master' into bufix/validate-tags-on-copy	2022-06-27 13:53:57 +03:00
chrislu	b9f7b6fb9a	adjust log message	2022-06-26 23:12:16 -07:00
Chris Lu	fdd880b111	Merge pull request #3237 from shichanglin5/remove_query_collection ... Remove the collection query param of s3api and let the collection be …	2022-06-26 23:02:11 -07:00
石昌林	15bfc60a88	Remove the collection query param of s3api and let the collection be determined by the filer	2022-06-27 13:47:53 +08:00
yulai.li	af23e63e3f	Improve filer command help, add supported filer store list	2022-06-27 12:09:16 +08:00
chrislu	40a9634c3d	3.13	2022-06-26 20:13:52 -07:00
Chris Lu	cc01f71752	Merge pull request #3233 from blacktear23/filerstore-tikv ... Add TiKV Filerstore back	2022-06-26 20:12:18 -07:00
yulai.li	824cbe32be	Make tikv filer enable/disable by build tags	2022-06-27 10:57:24 +08:00
chrislu	9dfea44aeb	fix unit	2022-06-26 12:21:38 -07:00
chrislu	3142e77eca	mount: stats report physical size instead of logical size	2022-06-26 12:14:34 -07:00
yulai.li	13159ad0d8	Merge branch 'master' into filerstore-tikv	2022-06-26 22:44:01 +08:00
yulai.li	46e0b629e5	Update tikv client version and add one PC support	2022-06-26 22:43:37 +08:00
chrislu	f77eda4b4f	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-25 09:22:52 -07:00
chrislu	3ce46c3566	fix entry full url lookup	2022-06-25 09:22:49 -07:00
Chris Lu	dc59ccd110	Merge pull request #3228 from shichanglin5/fix_volumeNotFound ... When the connection with the leader is disconnected, the vidMap shoul…	2022-06-24 23:06:13 -07:00
石昌林	f1ea906c13	Code comment optimization	2022-06-25 11:56:09 +08:00
石昌林	0c8e033e68	When the connection with the leader is disconnected, the vidMap should not be reset immediately, but should be reset after connecting to a new leader	2022-06-24 17:41:46 +08:00
chrislu	9c517d2b35	masterclient: fallback to directly querying master in case of missing volume id location	2022-06-24 02:08:57 -07:00
Chris Lu	280e33092c	Merge pull request #3226 from kmlebedev/advancedtls ... Advancedtls	2022-06-24 01:34:58 -07:00
chrislu	509a9047db	test compact map with snowflake sequencer	2022-06-23 21:45:51 -07:00
chrislu	217d7dc377	verify seq is always unique	2022-06-23 21:22:05 -07:00
Konstantin Lebedev	2e782dfdcb	rm defer comments	2022-06-24 01:50:13 +05:00
Konstantin Lebedev	7efaafb0ee	fix Authenticate	2022-06-24 01:42:04 +05:00
Konstantin Lebedev	b0aa51d7ef	enable require client cert	2022-06-24 00:29:23 +05:00
Konstantin Lebedev	ea7cdb8b0e	seperate option	2022-06-24 00:26:56 +05:00
Konstantin Lebedev	5b388ed6c1	initial advancedtls	2022-06-23 23:32:15 +05:00
chrislu	0c13fc873c	fix compilation	2022-06-23 00:45:11 -07:00
chrislu	96496d5286	master: broadcast new volume locations to clients to avoid possible racing condition ... fix https://github.com/chrislusf/seaweedfs/issues/3220	2022-06-23 00:41:33 -07:00
chrislu	52c44d646e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-22 00:34:27 -07:00
chrislu	530da0cc89	remove unused code	2022-06-22 00:34:25 -07:00
famosss	8a6e7a7df8	Merge branch 'chrislusf:master' into master	2022-06-22 09:58:56 +08:00
zzq09494	9df5ad5309	fix: vacuum create a lot of connections quickly	2022-06-22 09:57:22 +08:00
Chris Lu	d652ce1a36	Merge pull request #3216 from kmlebedev/mk_raft_basedir ... avoid no such raft date directory	2022-06-21 12:21:30 -07:00
chrislu	6b3beb709d	fix format	2022-06-21 12:21:02 -07:00
Konstantin Lebedev	c1c8dad677	avoid no such raft date directory ... https://github.com/chrislusf/seaweedfs/issues/3214	2022-06-21 13:47:51 +05:00
chrislu	4dc27e1ed5	3.12	2022-06-20 19:07:00 -07:00
chrislu	492da3dbce	master: put metadata under instance specific folder	2022-06-20 19:04:49 -07:00
chrislu	41dfe27102	Revert "remove max connection age" ... This reverts commit b9b684194f.	2022-06-20 14:49:39 -07:00
chrislu	2295766e2c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-20 14:16:18 -07:00
chrislu	1d0c53ea56	remote storage: stop supporting hdfs as a remote storage	2022-06-20 14:15:59 -07:00
Stephan	1eb7826909	Fix link to osxfuse github page	2022-06-20 22:36:07 +02:00
chrislu	bf5a5abfb1	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-20 13:18:11 -07:00
chrislu	69a2e503ab	non-recursive directory deletion ... fix a problem during git checkout a different version, where updated directories are deleted even though the directory is not empty.	2022-06-20 13:18:08 -07:00
chrislu	76f26c4b99	mount: delete target entry first during move	2022-06-20 13:15:02 -07:00
Chris Lu	460c7cf68e	Merge pull request #3203 from shichanglin5/volume_copy_preallocate	2022-06-20 07:41:45 -07:00
石昌林	81f7f08708	Determine whether to preallocate according to the master configuration before executing copy volume	2022-06-20 21:12:44 +08:00
石昌林	be5c901bea	Fix global counter key is incorrect & Add read lock when reading counter map to avoid memory problems	2022-06-20 19:16:53 +08:00
Lapshin Vitaliy	5f5fd0bc48	validate tags on copy object and add regex for validating tags	2022-06-20 11:32:58 +03:00
Chris Lu	9e2d6e897e	Merge pull request #3198 from guol-fnst/fix_statuscode ... AbortMultipartUploadHandler should return 204 instead of 200	2022-06-19 22:57:13 -07:00
LHHDZ	8db9f13bc6	Merge branch 'master' into circuit_breaker	2022-06-20 13:42:18 +08:00
guol-fnst	1d77deccd0	AbortMultipartUploadHandler should return 204 instead of 200 ... https://docs.aws.amazon.com/AmazonS3/latest/API/API_AbortMultipartUpload.html	2022-06-20 13:26:22 +08:00
石昌林	9e036df356	remove go.uber.org/atomic	2022-06-20 12:35:29 +08:00
geekboood	fdacd94af5	fix: invalid chunk data when failed to read manifests	2022-06-19 16:54:04 +08:00
石昌林	3dd60529c5	some code optimizations	2022-06-17 19:07:39 +08:00
石昌林	37df209195	add some unit tests and some code optimizes	2022-06-17 17:11:18 +08:00
Chris Lu	d28f8bb88f	Merge pull request #3192 from guol-fnst/weed_update ... remove "weed" from UsageLine, or weed will not show usage of weed update	2022-06-16 22:58:29 -07:00
guol-fnst	c7c20881f5	remove "weed" from UsageLine, or weed will not show usage of weed update	2022-06-17 12:47:04 +08:00
chrislu	625fd16a2e	reduce upfront memory usage for low density volume	2022-06-16 15:39:29 -07:00
Chris Lu	5f8d6b5f13	Merge pull request #3184 from guol-fnst/weed_update ... Weed update	2022-06-16 10:02:45 -07:00
guol-fnst	cd078471c4	fix path bug on windows ... fix usage	2022-06-16 17:24:56 +08:00
chrislu	aaa9938353	Revert "adjust conditions" ... This reverts commit e024586ff1.	2022-06-16 01:26:36 -07:00
guol-fnst	995e221e97	remove -output, add -dir,-name for more flexibility	2022-06-16 16:07:32 +08:00
chrislu	e024586ff1	adjust conditions	2022-06-15 23:19:39 -07:00
Chris Lu	82f3bcc65e	Merge pull request #3081 from paochiang/volume_upload_limit_fix ... exclude replication from the concurrentUploadLimitMB	2022-06-15 23:16:44 -07:00
liubaojiang	df0ce31a2e	add condition when inFlightUploadDataLimitCond signal	2022-06-16 14:07:11 +08:00
zzq09494	62f74f5e3c	add bucket label to s3 prometheus metrics	2022-06-16 13:21:25 +08:00
guol-fnst	4bc8792643	support specific version of weed	2022-06-16 11:40:23 +08:00
guol-fnst	1dbb925d29	remove unused function ... remove log when weed is up to date add check whether weed is full version	2022-06-16 11:40:20 +08:00
liubaojiang	3076ac101e	move vs.concurrentUploadLimit != 0 out of the lock	2022-06-16 09:58:44 +08:00
Chris Lu	097a12dfb4	Merge pull request #3187 from ningfdx/remote	2022-06-15 06:34:32 -07:00
Chris Lu	55b9f281d1	Merge pull request #3185 from kmlebedev/logging_err_process_range_req	2022-06-15 06:33:33 -07:00
石昌林	78b3728169	add s3 circuit breaker support for 'simultaneous request count' and 'simultaneous request bytes' limitations ... configure s3 circuit breaker by 'command_s3_circuitbreaker.go': usage eg: # Configure the number of simultaneous global (current s3api node) requests s3.circuit.breaker -global -type count -actions Write -values 1000 -apply # Configure the number of simultaneous requests for bucket x read and write s3.circuit.breaker -buckets -type count -actions Read,Write -values 1000 -apply # Configure the total bytes of simultaneous requests for bucket write s3.circuit.breaker -buckets -type bytes -actions Write -values 100MiB -apply # Disable circuit breaker config of bucket 'x' s3.circuit.breaker -buckets x -enable false -apply # Delete circuit breaker config of bucket 'x' s3.circuit.breaker -buckets x -delete -apply	2022-06-15 21:07:55 +08:00
ningfd	338705f375	fix(wdclient): GetLocations return	2022-06-15 19:20:13 +08:00
Konstantin Lebedev	31a929f7f2	logging processRangeRequest errors	2022-06-15 15:36:43 +05:00
Chris Lu	4eeeb5f50d	Merge pull request #3180 from blacktear23/issue-3149	2022-06-15 00:55:20 -07:00
yulai.li	358c0c0b37	Fix: provide filer.ui.deleteDir options for server command	2022-06-15 15:53:31 +08:00
Guo Lei	3df84cabf6	Merge branch 'chrislusf:master' into weed_update	2022-06-15 15:28:40 +08:00
guol-fnst	695e8c7555	fix minor bug	2022-06-15 15:26:04 +08:00
zhihao.qu	4d0d1848c6	fix(filer.sync): modify clientName format : from -> to	2022-06-15 13:33:20 +08:00
yulai.li	b90d213a87	Change ui.deleteDir default to true	2022-06-15 11:37:37 +08:00
zhihao.qu	42d04c581b	feat(filer.sync): add metricsServer in filer.sync. ... Metrics include: (1) the offset of the filer.sync (2) the last send timestamp of the filer subscription	2022-06-15 11:33:18 +08:00
chrislu	d4ef06cdcf	3.11	2022-06-14 12:14:52 -07:00
chrislu	b347b2fb54	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-14 11:16:51 -07:00
chrislu	36d2756978	volume: avoid writing too much data for range requests in large files ... related to https://github.com/chrislusf/seaweedfs/issues/3178	2022-06-14 11:16:47 -07:00
yulai.li	28cda854eb	Add filer command line parameter to let Filer UI show/hide directory delete button	2022-06-15 00:30:49 +08:00
zhihao.qu	14d82c3dea	feat(filer.sync): add offset to path.	2022-06-14 19:46:02 +08:00
guol-fnst	78fc9cdfa0	add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate"	2022-06-14 19:10:54 +08:00
guol-fnst	322dbcf372	add update feature	2022-06-14 19:01:57 +08:00
Chris Lu	b22ca85fbb	Merge pull request #3169 from shichanglin5/fix_auth ... fix: When there is no access permission configured before startup, th…	2022-06-12 23:40:42 -07:00
shichanglin5	f01dd27752	fix: When there is no access permission configured before startup, the authentication does not take effect after configuring the permission after startup	2022-06-13 13:23:26 +08:00
chrislu	e437ec5a68	3.10	2022-06-12 19:30:27 -07:00
chrislu	d12f431d98	collect volume server status	2022-06-12 11:56:23 -07:00
chrislu	444ac21050	go fmt	2022-06-11 09:51:11 -07:00
chrislu	a74e926063	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-11 09:51:02 -07:00
chrislu	61b8c9c361	remote object store gateway: disable tagging for backblaze	2022-06-11 09:50:59 -07:00
Chris Lu	0a8b3948db	Merge pull request #3165 from blacktear23/reload-bug-fix	2022-06-10 07:47:43 -07:00
yulai.li	83f8cdd5cb	Bug Fix: force reload filer page when upload/delete/rename/create directory	2022-06-10 22:39:21 +08:00
guol-fnst	eabf4d4e64	fix typo	2022-06-10 22:03:11 +08:00
Chris Lu	36757d28a8	Merge pull request #3163 from kmlebedev/metric_replica_placement_mismatch ... stats master_replica_placement_mismatch	2022-06-10 06:44:16 -07:00
chrislu	9f8b72a54d	Revert "Merge pull request #3159 from shichanglin5/_duplicateUUID" ... This reverts commit 37da689319, reversing changes made to 00d53c34c4.	2022-06-10 06:38:17 -07:00
Konstantin Lebedev	fb57aa431c	stats master_replica_placement_mismatch	2022-06-10 15:30:40 +05:00
shichanglin5	a63559ee6c	fix: fix command prompt information	2022-06-10 12:46:10 +08:00
Chris Lu	37da689319	Merge pull request #3159 from shichanglin5/_duplicateUUID ... perf: Optimized volume handling duplicateUUID logic to avoid quitting…	2022-06-09 18:01:02 -07:00
shichanglin5	f5b0c04b14	perf: Optimized volume handling duplicateUUID logic to avoid quitting when volume is actualy normal ... Under normal circumstances, there will be no problems, but when the master is debugged in the local environment, the volume client cannot communicate with the master normally, so the sendHeartBeat logic is restarted, and a new connection is created to report the heartbeat. If the master has not cleared the uuid of the volume at this time, then The master will respond to volume duplicateUUIDS, and the volume service will exit, but in fact the uuid of the volume is not duplicated	2022-06-09 20:41:16 +08:00
zhihao.qu	cd5cca36a4	feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges	2022-06-09 10:53:19 +08:00
Chris Lu	4a5135961f	Merge pull request #3154 from kmlebedev/s3_test_invalid_range	2022-06-08 01:45:40 -07:00
Konstantin Lebedev	49f058da85	fix s3 tests: ... ranged_request_invalid_range ranged_request_empty_object	2022-06-08 13:25:13 +05:00
chrislu	8a1166c021	auto remove previous leftover socket	2022-06-08 00:32:09 -07:00
chrislu	633a4aed18	filer: fix customizable local socket file name ... fix https://github.com/chrislusf/seaweedfs/issues/3147	2022-06-08 00:15:28 -07:00
chrislu	acbc0286e2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-07 22:33:38 -07:00
chrislu	ebd6097e6f	fix: sequence read rate is slow when use filer.ChunkReadAt.ReadAt #3074 ... related to https://github.com/chrislusf/seaweedfs/pull/3074	2022-06-07 22:33:35 -07:00
Chris Lu	f43ec9f363	Merge pull request #3130 from kmlebedev/fix_rm_parent_dir_via_nextcloud	2022-06-07 08:56:55 -07:00
Konstantin Lebedev	c07820178f	fix s3 tests ... bucket_list_delimiter_prefix bucket_list_delimiter_prefix_underscore bucket_list_delimiter_prefix_ends_with_delimiter	2022-06-07 14:43:10 +05:00
chrislu	78be3b0f6c	fix compilation	2022-06-06 10:09:21 -07:00
chrislu	4fd5f96598	filer: remove replication, collection, disk_type info from entry metadata ... these metadata can change and are not used	2022-06-06 00:39:35 -07:00
chrislu	cbf46de5f4	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-06 00:15:40 -07:00
chrislu	8402af0882	filer: detect ttl based on fs configure setting	2022-06-06 00:15:37 -07:00
ningfd	f32142f6f5	add disableXAttr in mount option	2022-06-06 14:09:01 +08:00
chrislu	4a046e4de7	3.09	2022-06-05 19:00:24 -07:00
chrislu	11f1e7996b	fix test	2022-06-05 18:50:41 -07:00
chrislu	d65bb2c6df	mount: file handle locks entry better ... related to https://github.com/chrislusf/seaweedfs/issues/2952	2022-06-05 18:15:06 -07:00
chrislu	746092a60b	add a comment	2022-06-05 16:44:07 -07:00
chrislu	056c480eb0	volume: checksum remove the hashing step in Value()	2022-06-05 15:24:02 -07:00
chrislu	ecef844dfc	stream read large files	2022-06-05 11:54:04 -07:00
chrislu	48cca4e54f	remove unused logic	2022-06-04 19:11:06 -07:00
chrislu	685643d506	needle read into a writer	2022-06-04 19:10:52 -07:00
chrislu	663bc5dc23	add needle reading in chunks	2022-06-04 18:15:39 -07:00
chrislu	18b29f709c	refactoring, split file	2022-06-03 23:31:14 -07:00
chrislu	39c3cb04af	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-06-03 08:45:32 -07:00
chrislu	f97acdd489	volume.fix.replication fix retry logic ... fix https://github.com/chrislusf/seaweedfs/issues/3136	2022-06-03 08:45:29 -07:00
creeew	02ae102731	fix filer.sync missing source srv uploaded files to target when target down	2022-06-02 01:28:47 +08:00
Konstantin Lebedev	b86628f85d	fix show empty dir ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-06-01 20:10:52 +05:00
chrislu	6793bc853c	help message when in simulation mode	2022-05-31 14:48:46 -07:00
chrislu	89948a373b	fix error reporting for "Need to a hdd disk type" ... related to https://github.com/chrislusf/seaweedfs/issues/3128	2022-05-31 12:43:55 -07:00
chrislu	8a49240d64	3.08	2022-05-31 11:57:41 -07:00
chrislu	376b005ad3	fix wrong logic	2022-05-31 11:55:58 -07:00
chrislu	fbd99d53c1	3.07	2022-05-31 00:36:24 -07:00
chrislu	27732ecfa4	move s3 related constants from package http to s3_constants	2022-05-30 22:57:41 -07:00
chrislu	f4a6da6cb2	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-30 22:47:29 -07:00
chrislu	596c3860ca	use final destination to resolve fs configuration ... related to https://github.com/chrislusf/seaweedfs/issues/3075	2022-05-30 22:47:26 -07:00
Chris Lu	ca01ce0524	Merge pull request #3122 from chrislusf/filer-sync-with-peers ... Filer bootstrap from peers	2022-05-30 21:38:31 -07:00
chrislu	6adc42147f	fresh filer store bootstrap from the oldest peer	2022-05-30 21:27:48 -07:00
chrislu	490e0850bf	filer.meta.tail add one example of using untilTimeAgo	2022-05-30 20:24:59 -07:00
liubaojiang	f0ee3e6f21	reduce the scope of inFlightUploadDataLimitCond lock	2022-05-31 09:49:07 +08:00
chrislu	c59068d0f3	refactor	2022-05-30 16:28:36 -07:00
chrislu	682382648e	collect cluster node start time	2022-05-30 16:23:52 -07:00
chrislu	968ca95b49	filer.meta.tail: support untilTimeAgo for a range	2022-05-30 16:16:23 -07:00
chrislu	f214dfb1f5	stop when in memory log is done	2022-05-30 15:25:21 -07:00
chrislu	aece35a64f	stop when on disk log is done	2022-05-30 15:20:51 -07:00
chrislu	a2b101a737	subscribe metadata between a range	2022-05-30 15:04:19 -07:00
chrislu	8902fa6ff6	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-30 13:28:32 -07:00
Chris Lu	730c9cf601	Merge pull request #3114 from kmlebedev/nextcloud ... Create folders in s3 via nextcloud	2022-05-30 13:10:10 -07:00
chrislu	5b9347c938	typo	2022-05-30 12:37:41 -07:00
Konstantin Lebedev	af562e1a20	rm log info	2022-05-30 23:26:26 +05:00
Konstantin Lebedev	62e5e3822c	list self dir ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-05-30 18:09:08 +05:00
Konstantin Lebedev	26f3646961	show fake dirs ... https://github.com/chrislusf/seaweedfs/issues/3086	2022-05-30 15:03:27 +05:00
chrislu	b8f3db0d46	s3: keep auth enabled in case identities are set to empty ... fix https://github.com/chrislusf/seaweedfs/issues/3084	2022-05-29 19:00:03 -07:00
chrislu	81d6159290	volume: report error if a volume has nil data backend ... fix https://github.com/chrislusf/seaweedfs/issues/3105	2022-05-29 16:59:30 -07:00
chrislu	ca16fbf0ef	shell: cluster.ps display filer group	2022-05-29 16:37:14 -07:00
chrislu	9219664505	reset sync offset if peer filer resets ... fix https://github.com/chrislusf/seaweedfs/issues/3089	2022-05-29 13:11:24 -07:00
Konstantin Lebedev	54356211c2	list empty folder	2022-05-27 13:17:18 +05:00
Kaiwalya Joshi	4598581741	fix: Minor spelling fixes on FilerStorePathTranslator.	2022-05-25 01:11:02 -07:00
chrislu	5b8b022985	remove unused parameter	2022-05-24 18:52:04 -07:00
Robert Coelho	0e6e72d462	mount: ReadDir return EIO on EnsureVisited err	2022-05-24 13:02:59 -04:00
Robert Coelho	1fabbe8a25	mount: cleanup ReadDir rewrite's branches to not assume offsets	2022-05-24 12:48:32 -04:00
Robert Coelho	cb422d96f7	mount: rewrite ReadDir to respect input.Offset to fix partial results	2022-05-24 12:16:58 -04:00
Robert Coelho	05602167f4	mount: do not mark meta_cache when there is an err	2022-05-24 12:13:15 -04:00
chrislu	7e25a2d416	reduce busy waiting when reading metadata logs	2022-05-24 00:23:53 -07:00
chrislu	2f846777bb	3.06	2022-05-23 01:21:48 -07:00
chrislu	a5cdcc7e63	fix reading files larger than 20GB due to manifest resolving bug	2022-05-23 01:16:10 -07:00
chrislu	59cc3c0bfd	fix reading files larger than 20GB due to manifest resolving bug ... this is introduced since Mar 08 2022	2022-05-23 01:14:56 -07:00
chrislu	acc318e12b	3.05	2022-05-22 21:43:59 -07:00
liubaojiang	076e48a676	add inflight upload data wait timeout	2022-05-21 10:38:08 +08:00
liubaojiang	71b2e6223e	add inFlightUploadDataLimitCond signal	2022-05-20 15:27:06 +08:00
liubaojiang	1a41691b4c	exclude replication from the concurrentUploadLimitMB	2022-05-20 14:40:05 +08:00
Chris Lu	cb64983c6b	Merge pull request #3079 from ztinpn/hotfix_fsync	2022-05-19 10:33:42 -07:00
tianzhang	66747ee9c9	hotfix_fsync fix fsync	2022-05-19 23:17:17 +08:00
Chris Lu	8efe55f981	Merge pull request #3059 from guol-fnst/avoid_dup_vol ... avoid loading duplicated volume directory	2022-05-17 23:48:57 -07:00
guol-fnst	8e2426a879	revert changes of pb file ... orignal protoc-gen-go-grpc verion is 1.1.0	2022-05-18 11:52:14 +08:00
guol-fnst	b12944f9c6	fix naming convention ... notify volume server of duplicate directoris improve searching efficiency	2022-05-17 15:41:49 +08:00
guol-fnst	076595fbdd	just exit in case of duplicated volume directories were loaded	2022-05-17 15:41:49 +08:00
guol-fnst	8fab39e775	rename UUID file ... fix typo move locationUUID into DiskLocation	2022-05-17 15:41:47 +08:00
Chris Lu	72e7dcde51	Merge pull request #3060 from natmaka/master	2022-05-16 07:42:58 -07:00
Nat Makarevitch	b72f0634d7	minor (typos...), done while reading around	2022-05-16 22:11:33 +08:00
guol-fnst	de6aa9cce8	avoid duplicated volume directory	2022-05-16 19:33:51 +08:00
Konstantin Lebedev	44f53ceda6	fix collectionIsMismatch charset	2022-05-16 13:23:23 +05:00
Konstantin Lebedev	10d435f2c2	fix skip loop	2022-05-16 13:16:27 +05:00
Konstantin Lebedev	279053572c	avoid delete volume replica if collection mismatch	2022-05-16 13:07:05 +05:00
chrislu	9ff0d99002	3.04	2022-05-15 21:32:21 -07:00
chrislu	866fed1eab	avoid grpc name conflict ... fix https://github.com/chrislusf/seaweedfs/issues/3055	2022-05-15 21:31:29 -07:00
chrislu	cd2d10118c	3.03	2022-05-15 18:17:07 -07:00
chrislu	271b5aed96	shell: volume.fsck add a note for -reallyDeleteFromVolume option	2022-05-15 11:07:04 -07:00
chrislu	4076d74813	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-15 00:43:39 -07:00
chrislu	29198720f2	s3: add grpc server to accept configuration changes	2022-05-15 00:43:37 -07:00
Chris Lu	ea4918394c	Merge pull request #3047 from shichanglin5/master ... fix the problem of metadata and tagging loss when files are copied	2022-05-13 22:52:48 -07:00
shichanglin5	688d55488c	test(s3api_object_copy_handlers_test.go): some unit tests have been added to the processMetadata & processMetadataBytes methods of s3api_object_copy_handlers.go	2022-05-14 10:40:29 +08:00
chrislu	735038b2c1	backup do not need to use preallocation ... fix https://github.com/chrislusf/seaweedfs/issues/3044	2022-05-13 13:46:52 -07:00
shichanglin5	1166dead00	fix the problem of metadata and tagging loss when files are copied ... by adding processing of metadata and tagging in s3 api CopyObject (judging whether to copy or overwrite according to the directive header)	2022-05-13 19:46:20 +08:00
Konstantin Lebedev	d12a423aa0	refactor ... add some tests https://github.com/chrislusf/seaweedfs/pull/2996	2022-05-12 12:23:02 +05:00
Konstantin Lebedev	f523aed3c9	fixes from comments ... https://github.com/chrislusf/seaweedfs/pull/2996	2022-05-11 21:50:09 +05:00
chrislu	bff1ccc1de	fix compilation	2022-05-11 00:52:15 -07:00
chrislu	76da688176	adjust log level	2022-05-11 00:45:19 -07:00
chrislu	3e20336bde	fix atomic add int64 ... fix https://github.com/chrislusf/seaweedfs/issues/3038#issuecomment-1123269831	2022-05-11 00:44:35 -07:00
chrislu	a85720f0f6	remove deprecated function	2022-05-10 00:02:23 -07:00
chrislu	25559eeb92	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-10 00:01:46 -07:00
chrislu	791c29ae93	remove deprecated function ... WithQueryCachePolicyKeepInCache is deprecated and already default behavior	2022-05-10 00:01:43 -07:00
qiang.zeng	06cb337b82	fix typo	2022-05-09 14:21:16 +08:00
chrislu	a4ca3ed1f0	3.02	2022-05-08 22:28:32 -07:00
chrislu	8bc16f1149	rename	2022-05-06 23:38:22 -07:00
chrislu	139e039c44	filer.sync: pass attributes for mount ... fix https://github.com/chrislusf/seaweedfs/issues/3012	2022-05-06 03:54:12 -07:00
chrislu	10207a48a4	how to setup local test env	2022-05-06 02:25:56 -07:00
chrislu	db33b03735	add comments	2022-05-06 02:19:47 -07:00
chrislu	e350ba8896	larger test range	2022-05-06 02:17:28 -07:00
chrislu	71e0bcc5d5	a bit optimization	2022-05-06 02:11:58 -07:00
chrislu	b2a43dd044	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-05-06 01:51:28 -07:00
chrislu	d5fe5d12b7	filer: etcd fix pagination ... fix https://github.com/chrislusf/seaweedfs/issues/3016	2022-05-06 01:51:19 -07:00
Chris Lu	a69bf53405	Merge pull request #2996 from kmlebedev/ydb ... ydb filer store	2022-05-05 15:34:37 -07:00
Konstantin Lebedev	e6d2cb5911	ydb add empty doc file	2022-05-05 11:12:14 +05:00
chrislu	ef6c6c450e	avoid fatal error if port is already in use	2022-05-04 14:55:14 -07:00
chrislu	f7366a9668	skip unix socket mode for windows ... fix https://github.com/chrislusf/seaweedfs/issues/3013	2022-05-04 10:14:34 -07:00
Konstantin Lebedev	f58adaab25	build tag ydb since the growth of the binary is 5 mb	2022-05-03 23:20:09 +05:00
Konstantin Lebedev	f127b326bf	add options to scaffold	2022-05-03 22:54:31 +05:00
Konstantin Lebedev	7b544576af	refactor	2022-05-03 20:16:00 +05:00
chrislu	f17cd0d5cd	return false if not found ... fix https://github.com/chrislusf/seaweedfs/issues/3011	2022-05-03 07:18:34 -07:00
chrislu	b201edb9df	fix wrong assignment	2022-05-03 07:12:33 -07:00
Konstantin Lebedev	04d5dff6bb	add support native ttl	2022-05-03 17:52:23 +05:00
Konstantin Lebedev	cb3c7a3cdb	enable query cache policy instead of prepare	2022-05-03 16:03:10 +05:00
Konstantin Lebedev	8342f651f3	fix scanNamed	2022-05-03 15:18:28 +05:00
chrislu	9271866d1e	fix segmentation violation ... fix https://github.com/chrislusf/seaweedfs/issues/3000	2022-05-02 23:56:11 -07:00
Konstantin Lebedev	2d27afb515	Merge branch 'new_master' into ydb	2022-05-03 10:47:39 +05:00
chrislu	76148ce0f7	use storage_class for backend configuration ... fix https://github.com/chrislusf/seaweedfs/issues/3008	2022-05-02 12:20:01 -07:00
Konstantin Lebedev	d8925b4e83	Merge branch 'new_master' into ydb ... # Conflicts: # go.mod # go.sum	2022-05-03 00:13:57 +05:00
Konstantin Lebedev	7640e650e5	fix queries	2022-05-03 00:11:37 +05:00
Konstantin Lebedev	319d300d48	fix createDB	2022-05-02 22:23:07 +05:00
Konstantin Lebedev	0dc44dda63	ydb do Tx or DB	2022-05-02 15:33:29 +05:00
Konstantin Lebedev	a3e48831f3	ydb BucketAware interface	2022-05-02 12:42:20 +05:00
chrislu	73961e24d8	3.01	2022-05-01 23:33:45 -07:00
chrislu	00c1dfec4f	go fmt	2022-05-01 23:16:29 -07:00
chrislu	ce1e8e58ae	fix test	2022-05-01 23:15:58 -07:00
chrislu	2c0fdf3693	fix tests	2022-05-01 23:14:04 -07:00
chrislu	94635e9b5c	filer: add filer group	2022-05-01 21:59:16 -07:00
Konstantin Lebedev	50c4f62ed4	ydb-go-sdk move to v3	2022-05-02 02:07:47 +05:00
Konstantin Lebedev	21033ff4c3	refactor use const CountEntryChunksForGzip	2022-05-01 22:28:55 +05:00
Konstantin Lebedev	ec0ed41e37	ydb kv interface	2022-05-01 21:20:37 +05:00
chrislu	4bd6bea429	fix test	2022-05-01 09:02:01 -07:00
Konstantin Lebedev	6a052f6ff2	ydb Sql interface	2022-05-01 20:33:03 +05:00
Konstantin Lebedev	275c3bb19c	ydb initial ... https://github.com/chrislusf/seaweedfs/issues/2942	2022-05-01 17:28:58 +05:00
chrislu	5d081d5c3c	skip if already copied to remote ... fix https://github.com/chrislusf/seaweedfs/issues/2993	2022-04-30 20:03:04 -07:00
chrislu	1aae7a3f1b	volume.tier.upload progress starts negative #2992 ... fix https://github.com/chrislusf/seaweedfs/issues/2992	2022-04-30 18:10:01 -07:00
chrislu	ea8a7a5584	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-30 17:36:45 -07:00
chrislu	192983b464	s3 backend support customizing storage class	2022-04-30 17:36:40 -07:00
Konstantin Lebedev	03aec9ce7e	avoid panic if hashicorp raft not initialized	2022-04-30 13:56:35 +05:00
guosj	1a5de36257	fix return value	2022-04-29 11:19:36 +08:00
Chris Lu	4e97b6146a	Merge pull request #2982 from kmlebedev/issues/2981 ... avoid empty listMultipartUploads response	2022-04-27 17:21:44 -07:00
chrislu	b2a6111090	skip ipv6 all interfaces and localhost ... fix https://github.com/chrislusf/seaweedfs/issues/2983	2022-04-27 17:18:09 -07:00
Konstantin Lebedev	306cf70c4a	avoid empty listMultipartUploads response	2022-04-27 19:27:44 +05:00
chrislu	70e5a1b632	volume close should wait for committing compaction	2022-04-26 23:34:05 -07:00
Chris Lu	ffe028f8d0	Merge pull request #2974 from kmlebedev/wait_volume_closed_compression ... waite volume being closed during compression idx	2022-04-26 23:29:22 -07:00
chrislu	37ab8909b0	use two flags: v.isCompacting and v.isCommitCompacting	2022-04-26 23:28:34 -07:00
Chris Lu	b557faf175	Merge pull request #2967 from kmlebedev/vl_chk_disk_vlid ... volume.check.disk add param volumeId	2022-04-26 22:09:06 -07:00
Chris Lu	eb49bd5638	Merge pull request #2970 from kmlebedev/vlm_fsck_forcePurging ... delete missing data from volumes in one replica	2022-04-26 22:08:34 -07:00
chrislu	94f824e1ce	volume: sync to disk before copying volume files ... address https://github.com/chrislusf/seaweedfs/issues/2976	2022-04-26 13:03:43 -07:00
Konstantin Lebedev	7315d1d039	waite volume being closed during compression idx	2022-04-26 13:40:42 +05:00
Konstantin Lebedev	d4343ab7da	forcePurging desc	2022-04-25 23:11:56 +05:00
Konstantin Lebedev	ae56b2c00f	change forcePurging to a pointer	2022-04-25 23:10:01 +05:00
Konstantin Lebedev	6d2fda27d2	delete missing data from volumes in one replica	2022-04-25 22:59:46 +05:00
Konstantin Lebedev	67476d830a	volume.check.disk add param volumeId	2022-04-25 18:27:14 +05:00
chrislu	2aef5b8b32	3.00	2022-04-24 21:46:03 -07:00
Chris Lu	184e982898	Merge pull request #2957 from guo-sj/handle_implicit_username ... Handle implicit username	2022-04-24 09:07:48 -07:00
guosj	8fd7b24b80	remove redundant logs & add unit test	2022-04-24 16:09:10 +08:00
guol-fnst	8b2552f944	fix http response error code	2022-04-24 11:07:27 +08:00
guosj	09befe871f	Merge branch 'master' into handle_implicit_username	2022-04-23 09:48:25 +08:00
guosj	893a0eea57	add more checks and comments	2022-04-22 22:47:11 +08:00
chrislu	ad01c63b84	conditionally skip hdfs related code	2022-04-21 01:43:01 -07:00
chrislu	4aa39ef33f	conditionally compile sqlite	2022-04-21 01:22:38 -07:00
chrislu	3885374edf	conditionally build elastic, gocdk to reduce binary size	2022-04-21 01:10:46 -07:00
chrislu	6994e88192	fix typo	2022-04-21 00:02:18 -07:00
chrislu	cb1e7aa27d	fix build	2022-04-20 16:21:56 -07:00
a	1d6a9e66b6	Merge branch 'master' into a	2022-04-20 14:01:42 -07:00
chrislu	bdc4d67de8	Revert "Merge pull request #2944 from guo-sj/handle_implicit_username" ... This reverts commit bc96e73b8d, reversing changes made to 88a669dd19.	2022-04-20 09:40:41 -07:00
chrislu	3aa4dc6ad5	Revert "fix index out of range" ... This reverts commit 4792e07228.	2022-04-20 09:40:24 -07:00
chrislu	4792e07228	fix index out of range	2022-04-20 09:32:17 -07:00
chrislu	f3fdfb07b8	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-19 22:58:59 -07:00
chrislu	8e9ad7db5a	fix bug deleting volume or unmount volume if a volume server has multiple directories	2022-04-19 22:58:57 -07:00
Chris Lu	bc96e73b8d	Merge pull request #2944 from guo-sj/handle_implicit_username ... handle implicit username	2022-04-19 21:42:21 -07:00
guosj	a23629b2fe	handle implicit username	2022-04-20 10:12:33 +08:00
Chris Lu	88a669dd19	Merge pull request #2858 from kmlebedev/fsck_replicas ... fsck replicas	2022-04-19 00:17:43 -07:00
Chris Lu	f76c318a1f	Merge pull request #2940 from kmlebedev/volume.list.grep ... volume.list show only readonly, collectionPattern and volumeId	2022-04-19 00:17:32 -07:00
Konstantin Lebedev	2364fab927	volume.list show only readonly, collectionPattern and volumeId	2022-04-19 11:44:41 +05:00
guosj	94ad557d97	add missing return	2022-04-19 14:16:32 +08:00
Konstantin Lebedev	36c5a59ed8	add help	2022-04-18 19:36:14 +05:00
Konstantin Lebedev	1e35b4929f	shell vacuum volume by collection and volume id	2022-04-18 18:40:58 +05:00
Chris Lu	97c6e0efb1	Merge pull request #2931 from kmlebedev/fix_nil_dataBackend ... avoid invalid memory address or nil pointer dereference	2022-04-18 00:13:26 -07:00
Konstantin Lebedev	9438738693	avoid invalid memory address or nil pointer dereference	2022-04-18 12:10:22 +05:00
chrislu	a69f0b62c2	start filer local socket only on non windows ... fix https://github.com/chrislusf/seaweedfs/issues/2930	2022-04-18 00:08:31 -07:00
chrislu	9ae66f55fc	2.99	2022-04-17 23:01:42 -07:00
chrislu	4524ceae6b	less verbose logs ... fix https://github.com/chrislusf/seaweedfs/discussions/2925#discussioncomment-2584085	2022-04-17 22:55:26 -07:00
leyou240	89eb87c1d1	Merge branch 'master' into slices.SortFunc	2022-04-18 10:39:29 +08:00
justin	3551ca2fcf	enhancement: replace sort.Slice with slices.SortFunc to reduce reflection	2022-04-18 10:35:43 +08:00
chrislu	b597baf488	mount: set file write time correct to work well with rsync ... fix https://github.com/chrislusf/seaweedfs/issues/2867	2022-04-16 22:56:45 -07:00
chrislu	460d56d283	shell: cluster.check prints out clock delta and network latency	2022-04-16 13:24:17 -07:00
chrislu	b4be56bb3b	add timing info during ping operation	2022-04-16 12:45:49 -07:00
chrislu	1384529eb7	Fix filer.backup deletes files in backup folder in incremental mode ... fix https://github.com/chrislusf/seaweedfs/issues/2919	2022-04-14 13:35:01 -07:00
a	846858fb43	merge master	2022-04-12 14:42:29 -07:00
chrislu	1f03fcccb1	fix nil in cluster_check shell command ... fix https://github.com/chrislusf/seaweedfs/issues/2905	2022-04-12 08:47:27 -07:00
Chris Lu	dd13764a5c	Merge pull request #2900 from kmlebedev/fix_cleanupUploads ... avoid breaking loop in cleanupUploads if error is empty	2022-04-12 02:03:31 -07:00
Konstantin Lebedev	76b1c5ce67	cleanupUploads use jwt token	2022-04-12 13:46:11 +05:00
Chris Lu	72db181d68	Merge pull request #2901 from guol-fnst/fix_multiupload ... check object name and uploadID when processing multiupload	2022-04-11 23:38:12 -07:00
guol-fnst	44d810d163	rename functions and remove uncessary check	2022-04-12 13:40:53 +08:00
guol-fnst	180aa88a92	check uploadid using object name hash string	2022-04-12 11:04:38 +08:00
guol-fnst	2232cfb5b7	Check object name and uploadID when processing multipart uploading	2022-04-11 19:53:44 +08:00
Konstantin Lebedev	2dfbe210e0	avoid breaking loop in cleanupUploads if error is empty	2022-04-11 11:16:20 +05:00
Chris Lu	a87f57e47c	Merge pull request #2868 from kmlebedev/hashicorp_raft ... hashicorp raft	2022-04-10 23:00:05 -07:00
Konstantin Lebedev	931cb9e581	use "cluster.raft.{ps,add,remove}"	2022-04-11 10:50:01 +05:00
Konstantin Lebedev	d2fe0fae33	use the full ServerAddress as the identifier	2022-04-11 10:44:44 +05:00
chrislu	c6ec5269f4	2.98	2022-04-10 18:56:40 -07:00
chrislu	261ba0208e	lock-protected setting entry	2022-04-09 22:52:59 -07:00
Chris Lu	9f4ecd50ac	Merge pull request #2889 from blacktear23/filer-ui-bootstrap-style ... Change filer UI icon from picture to icon font.	2022-04-08 00:50:32 -07:00
yulai.li	ceb058ddd9	Change filer UI icon from picture to icon font.	2022-04-08 15:07:56 +08:00
chrislu	7f26676f57	adjust td width	2022-04-07 22:34:32 -07:00
yulai.li	dff63e9024	Make operation buttons show when table row hovered	2022-04-08 13:10:12 +08:00
chrislu	4dba102137	use icon instead of button text	2022-04-07 11:15:01 -07:00
Chris Lu	bbc95dac47	Merge pull request #2885 from binbinshi/master ... fix: master lose some volumes	2022-04-07 10:16:19 -07:00
Konstantin Lebedev	104ea7b029	master ui add raft stats and Max Volume Id	2022-04-07 20:52:01 +05:00
a	41d396edc4	Merge branch 'master' into a	2022-04-07 08:22:59 -07:00
Konstantin Lebedev	a2fdb3e277	fix master ui	2022-04-07 19:37:40 +05:00
Konstantin Lebedev	35bc67f030	fix conflicts	2022-04-07 18:52:13 +05:00
Konstantin Lebedev	f5246b748d	Merge branch 'new_master' into hashicorp_raft ... # Conflicts: # weed/pb/master_pb/master.pb.go	2022-04-07 18:50:27 +05:00
yulai.li	11a9f993a8	Make filer-ui bootstrap style	2022-04-07 20:17:00 +08:00
Konstantin Lebedev	7ff248d5cd	refactor OnPeerUpdate	2022-04-07 16:23:22 +05:00
Konstantin Lebedev	85d80fd36d	fix removing old raft server	2022-04-07 15:31:37 +05:00
chrislu	abe3cc6df2	filer UI touch up	2022-04-07 01:25:55 -07:00
Chris Lu	869d23b099	Merge pull request #2884 from blacktear23/filer-ui-new-features ... Filer UI new features	2022-04-07 00:58:56 -07:00
chrislu	86ed27f602	mount: remove leftover socket file	2022-04-07 00:33:13 -07:00
yulai.li	2347c21cdd	Tune filer UI add rename feature	2022-04-07 15:19:41 +08:00
yulai.li	2454020a92	Add upload progress support	2022-04-07 15:19:37 +08:00
yulai.li	4f87ee7755	Add create directory and delete web UI features for filer	2022-04-07 15:19:33 +08:00
shibinbin	c20e1edd99	fix: master lose some volumes	2022-04-07 15:18:28 +08:00
eddy-gfx	ec53eec94f	Merge branch 'master' into a	2022-04-06 18:45:55 -05:00
chrislu	3ab2c0e5c0	avoid possible nil attributes	2022-04-06 10:11:11 -07:00
chrislu	9d44b89c21	Revert "avoid possible nil attributes" ... This reverts commit 0a6703c7f7.	2022-04-06 10:01:45 -07:00
chrislu	0a6703c7f7	avoid possible nil attributes	2022-04-06 09:58:48 -07:00
Konstantin Lebedev	14a2cc83bf	raft update peers via OnPeerUpdate	2022-04-06 21:17:04 +05:00
Konstantin Lebedev	357aa818fe	add raft shell cmds	2022-04-06 15:23:53 +05:00
chrislu	3d229fe45c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-06 00:28:51 -07:00
chrislu	79b8e6a8c3	add a place holder for later merge consecutive chunks	2022-04-06 00:28:48 -07:00
Konstantin Lebedev	c1aeded2dd	force raft bootstrap ... avoid err bootstrap only works on new clusters	2022-04-06 12:18:43 +05:00
Chris Lu	49ecb8d11c	Merge pull request #2879 from leyou240/master ... remove Redundant type conversion and use strings.TrimSuffix to enhance readability	2022-04-06 00:02:19 -07:00
justin	d02f13c2d1	remove Redundant type conversion and use strings.TrimSuffix to enhance readability	2022-04-06 14:58:09 +08:00
root	f15a737a0a	Add "Location:" in response when creating bucket ... according to "https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateBucket.html"	2022-04-06 13:56:27 +08:00
chrislu	bc888226fc	erasure coding: tracking encoded/decoded volumes ... If an EC shard is created but not spread to other servers, the masterclient would think this shard is not located here.	2022-04-05 19:03:02 -07:00
chrislu	8b3d76b24d	prevent concurrent access during shutdown	2022-04-05 10:53:36 -07:00
chrislu	bb4beebce3	prevent nil	2022-04-05 10:49:17 -07:00
Konstantin Lebedev	0e796a5582	rm set NoSnapshotRestoreOnStart	2022-04-05 18:28:42 +05:00
Konstantin Lebedev	68f11b9687	sleep bootstraping	2022-04-05 15:42:19 +05:00
Konstantin Lebedev	1ae7f509f5	LocalID from raw string	2022-04-05 13:50:39 +05:00
Konstantin Lebedev	89d32a0670	fix glog Info	2022-04-05 13:43:34 +05:00
Konstantin Lebedev	17c6e8e39f	Merge branch 'new_master' into hashicorp_raft ... # Conflicts: # go.mod # go.sum	2022-04-05 13:29:46 +05:00
Konstantin Lebedev	b7cdde14ae	auto bootstraping and update peers	2022-04-05 13:17:53 +05:00
chrislu	4aae87f405	check missing hdd disk type	2022-04-04 14:48:00 -07:00
Konstantin Lebedev	622297f1a7	add stats raft handler	2022-04-04 19:16:06 +05:00
Konstantin Lebedev	1a1e5778c3	fix cluster status	2022-04-04 18:52:08 +05:00
Konstantin Lebedev	14dd971890	hashicorp raft with state machine	2022-04-04 17:51:51 +05:00
Konstantin Lebedev	c514710b7b	initial add hashicorp raft	2022-04-04 13:50:56 +05:00
chrislu	d310711de0	avoid possible deadlock: move metadata log flush channel out of lock scope	2022-04-03 21:53:30 -07:00
chrislu	77a7d7253f	2.97	2022-04-03 19:08:01 -07:00
chrislu	e8d7bb42e2	avoid concurrent access to map object ... fix https://github.com/chrislusf/seaweedfs/issues/2866	2022-04-03 01:08:25 -07:00
chrislu	6a2bcd03aa	configure mount quota	2022-04-02 21:34:26 -07:00
chrislu	958f880b70	mount: add grpc method to adjust quota	2022-04-02 15:14:37 -07:00
chrislu	a8eefda452	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-04-01 20:25:37 -07:00
chrislu	105578a2f2	skip pinging self for master and volume server	2022-04-01 20:25:35 -07:00
root	b7e34016d4	fix error response in case of wrong expire time format	2022-04-02 11:01:33 +08:00
chrislu	4ecba915f3	add check between peers	2022-04-01 17:40:25 -07:00
chrislu	daca2d22a5	use original server address string as map key	2022-04-01 17:34:42 -07:00
chrislu	4b5c0e3fa9	check cluster connectivities	2022-04-01 17:27:49 -07:00
chrislu	6d55716fc3	use constants	2022-04-01 16:55:26 -07:00
chrislu	bbbbbd70a4	master supports grpc ping	2022-04-01 16:50:58 -07:00
chrislu	743ad690b6	filer supports grpc ping	2022-04-01 16:44:58 -07:00
chrislu	2305508b65	refactor: separate into two files	2022-04-01 16:40:49 -07:00
chrislu	800cbc004c	volume server adds ping function	2022-04-01 16:37:06 -07:00
a	7e92517571	change user and pass to username and password	2022-04-01 14:09:25 -05:00
Konstantin Lebedev	7f1383a41e	findExtraChunksInVolumeServers in consideration of replication	2022-04-01 14:45:41 +05:00
Konstantin Lebedev	3817e05dd0	fix collect filer files	2022-04-01 10:17:09 +05:00
Konstantin Lebedev	3cedb21bb7	skip new entities	2022-03-31 21:36:10 +05:00
Konstantin Lebedev	1d9e30d8c0	fsck replicas	2022-03-31 19:10:06 +05:00
Konstantin Lebedev	4f5b018a02	collecting only bucket volumes	2022-03-31 13:35:58 +05:00
Konstantin Lebedev	691ed50957	use constant for ErrBucketNotEmpty	2022-03-31 08:44:48 +05:00
Konstantin Lebedev	b6a2b43efe	allowDeleteBucketNotEmpty	2022-03-30 22:46:13 +05:00
Konstantin Lebedev	a8b15c676c	s3 test bucket delete nonempty	2022-03-30 21:06:26 +05:00
a	549e341c6f	resolv	2022-03-30 06:31:19 -05:00
Chris Lu	ce8ef60c59	Merge pull request #2840 from kmlebedev/fix_multi_object_delete_key_limit ... fix test_s3.test_multi_object_delete_key_limit and test_multi_objectv2_delete_key_limit	2022-03-30 01:27:13 -07:00
Konstantin Lebedev	6317f6a8a9	fix Description	2022-03-30 13:04:15 +05:00
Chris Lu	8732cc24c8	Merge pull request #2849 from guo-sj/update_user ... Add AWS IAM update user API and its test case	2022-03-29 22:36:26 -07:00
Konstantin Lebedev	993283bb1d	revert sort.Search	2022-03-29 19:53:12 +05:00
Konstantin Lebedev	5b90a39954	fix s3 tests: ... multipart_upload_incorrect_etag multipart_resend_first_finishes_last	2022-03-29 19:21:09 +05:00
Konstantin Lebedev	9231971c88	fix test_s3.test_multi_object_delete_key_limit and ... test_multi_objectv2_delete_key_limit	2022-03-29 16:55:09 +05:00
guosj	ed07e76f0e	add test function	2022-03-29 17:21:59 +08:00
a	bc603e534f	diff	2022-03-28 16:50:28 +00:00
Konstantin Lebedev	84b7b83517	fix permission mkdir snapshot ... avoid open file operation not permitted	2022-03-28 18:41:52 +05:00
Chris Lu	93615b2a49	Merge pull request #2828 from guo-sj/fix-tagging-error ... Fix delete all "Seaweed-" prefixed tagging error	2022-03-28 00:35:22 -07:00
guosj	dd2b9d93cc	update code according to PR's comment	2022-03-28 15:23:31 +08:00
guosj	02076f01a6	fix tagging error	2022-03-28 14:50:48 +08:00
justin	d51a724101	fix: encode small chunk return error maybe have some bug.	2022-03-28 13:11:24 +08:00
chrislu	0490ee87ef	2.96	2022-03-27 16:11:17 -07:00
chrislu	86d4c52d6e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-26 13:33:20 -07:00
chrislu	21e0898631	refactor: change masters from a slice to a map	2022-03-26 13:33:17 -07:00
chrislu	4ba7127ab1	refactor	2022-03-26 13:13:19 -07:00
Chris Lu	3153c836a7	Merge pull request #2744 from kmlebedev/issues_2545 ... avoid connect to the old filler address	2022-03-26 13:02:49 -07:00
chrislu	cae998eda1	Revert "Revert "remove duplicated metadata subscription in filer"" ... This reverts commit 34b743c481.	2022-03-26 13:00:16 -07:00
chrislu	34b743c481	Revert "remove duplicated metadata subscription in filer" ... This reverts commit 34742be029. Related to https://github.com/chrislusf/seaweedfs/issues/2545	2022-03-26 12:33:45 -07:00
chrislu	fba1cfc2d6	simplify a bit	2022-03-26 10:24:05 -07:00
chrislu	a3411dd9da	refactor	2022-03-26 10:21:26 -07:00
Chris Lu	1b27f96669	Merge pull request #2814 from kmlebedev/fix_remove_deleted_peers ... Fix remove deleted peers	2022-03-25 11:36:15 -07:00
Konstantin Lebedev	ddd3945c26	fix remove deleted peers of raft server ... https://github.com/chrislusf/seaweedfs/issues/2804	2022-03-25 15:09:38 +05:00
Konstantin Lebedev	c1450bf9fe	always clear previous log to avoid server is promotable ... https://github.com/chrislusf/seaweedfs/issues/2804	2022-03-25 13:40:19 +05:00
Chris Lu	89d84e275b	Merge pull request #2759 from kmlebedev/skip_wait_cancelled_request ... Need to exit waiting if request is was canceled	2022-03-24 12:21:44 -07:00
Konstantin Lebedev	0b790d2bbf	exclude the replication from the concurrentUploadLimitMB	2022-03-24 13:54:42 +05:00
chrislu	9b04f17555	remove dead code	2022-03-23 23:18:32 -07:00
chrislu	0b5faef612	fix	2022-03-23 23:16:54 -07:00
chrislu	0562fceb99	volume: fail fast if too many concurrent requests, to avoid dead lock due to replication. ... fix https://github.com/chrislusf/seaweedfs/issues/2755	2022-03-23 22:53:58 -07:00
Konstantin Lebedev	849a2c1f68	fix cmdFiler	2022-03-23 14:30:46 +05:00
Konstantin Lebedev	105bcfdbd4	set iam ip from filer	2022-03-23 13:45:51 +05:00
chrislu	e48764be75	s3: multipart upload verifies uploaded parts	2022-03-23 01:05:14 -07:00
chrislu	18ae8943ef	better clean up upload directory	2022-03-23 01:03:51 -07:00
chrislu	a0bad1c997	remove any go mod changes ... This reverts commit 6c7f7d6887.	2022-03-21 23:04:00 -07:00
chrislu	6c7f7d6887	Revert "Merge pull request #2782 from SadmiB/upstream" ... This reverts commit a644b7236a, reversing changes made to 349257f822.	2022-03-21 23:00:50 -07:00
Chris Lu	0d692a6254	Merge branch 'master' into upstream	2022-03-21 11:02:04 -07:00
SadmiB	d12540c9f2	Add contabo api client	2022-03-21 17:16:49 +01:00
chrislu	4e2388e1b5	mount: set file size if it is only on remote gateway	2022-03-21 02:09:54 -07:00
chrislu	8f0410af2c	2.95	2022-03-21 01:47:03 -07:00
chrislu	ae558fa073	log reasons volumes became unwritable	2022-03-21 00:41:44 -07:00
chrislu	57c6eddd22	avoid possible deadlock if volume layout is used in some logs	2022-03-21 00:04:01 -07:00
chrislu	581c93d970	change to config_dir, to be consistent with "weed scaffold -config" ... fix https://github.com/chrislusf/seaweedfs/issues/2753	2022-03-19 00:24:52 -07:00
chrislu	b75c6dcff5	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-19 00:22:52 -07:00
chrislu	3da2b83b38	Added a "-conf_dir" option to customize *.toml configuration file directory. ... fix https://github.com/chrislusf/seaweedfs/issues/2753	2022-03-19 00:22:47 -07:00
elee	beb406bbbb	fix ls, onBucketDelete still not triggering	2022-03-18 22:21:57 -05:00
elee	411c0df3fe	switch to multi collection, change readme	2022-03-18 21:51:16 -05:00
zzq09494	9f16df736e	fix-filer: calculation error of the method skipCheckParentDirEntry	2022-03-19 09:15:42 +08:00
elee	1cea6c73d3	update readme	2022-03-18 00:34:19 -05:00
elee	25be96832a	update readme with index info	2022-03-18 00:33:08 -05:00
elee	06f23aa675	put in delete folder children query	2022-03-18 00:29:48 -05:00
elee	0701feeb17	added more to readme	2022-03-18 00:26:11 -05:00
elee	171c27ddf1	zz	2022-03-17 22:35:30 -05:00
elee	2f0cdcdceb	update readme	2022-03-17 21:26:16 -05:00
elee	423ce57cde	prefix search, bucket implemented	2022-03-17 21:12:25 -05:00
chrislu	2824940ecf	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-17 16:54:30 -07:00
chrislu	f247cab5cd	skip localhost if bound to all interfaces already 0.0.0.0 or 127.0.0.1	2022-03-17 16:54:29 -07:00
Berck Nash	7ee38fa3a4	The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems: ... (1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified. Instead, it bound to localhost which broke the default configuration. This change implements the documented behavior instead. (2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP. I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?	2022-03-17 15:30:23 -06:00
elee	bf745bdccb	revise comment	2022-03-17 14:09:34 -05:00
elee	a0df993cef	cleanup, add a few comments	2022-03-17 14:08:36 -05:00
elee	94883e2fad	ok then	2022-03-17 05:23:12 -05:00
elee	f3ab6769e9	make start from indexing work	2022-03-17 05:09:13 -05:00
elee	31571dd96e	fix file listing	2022-03-17 05:07:40 -05:00
elee	b25d03340b	Merge branch 'a' of github.com:gfxlabs/seaweedfs into a	2022-03-17 04:51:59 -05:00
elee	921535001a	arangodb adapter	2022-03-17 04:49:26 -05:00
chrislu	6886167d5f	minor optimization	2022-03-17 00:03:28 -07:00
chrislu	7542fd7f4d	mount: optimize a bit when writing files	2022-03-17 00:02:38 -07:00
chrislu	4042fdf3bb	rename to skipCheckParentDir ... related to https://github.com/chrislusf/seaweedfs/pull/2761 It's better to default to false.	2022-03-16 23:55:31 -07:00
zzq09494	40b0033fa7	go fmt	2022-03-17 14:19:48 +08:00
zzq09494	ee5417a5b3	fix: ut test error	2022-03-17 11:16:46 +08:00
zzq09494	81cce4b4c3	filer: support uploading file without needEnsureParentDir	2022-03-17 10:53:47 +08:00
zzq09494	a6a8892255	Revert "filer: support uploading file without needEnsureParentDir" ... This reverts commit a93c4947ba.	2022-03-17 10:27:17 +08:00
zzq09494	a93c4947ba	filer: support uploading file without needEnsureParentDir	2022-03-17 10:18:23 +08:00
Berck Nash	9b14f0c81a	Add mTLS support for both master and volume http server.	2022-03-16 09:52:17 -06:00
chrislu	ed818c90a9	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-15 22:28:21 -07:00
chrislu	3639cad69c	master, filer, s3: also listen to "localhost" in addition to specific ip address ... related to https://github.com/chrislusf/seaweedfs/issues/1937	2022-03-15 22:28:18 -07:00
Konstantin Lebedev	f43c6daeda	Need to exit waiting if request is was canceled	2022-03-15 19:55:22 +05:00
Konstantin Lebedev	c2e7d663bc	reduced the timeout for creating a connection from 30 to 10 seconds	2022-03-15 13:32:05 +05:00
chrislu	fbc9f0eb64	minor	2022-03-14 03:19:16 -07:00
chrislu	2eda3a686f	2.94	2022-03-14 00:55:01 -07:00
chrislu	bd5c5586b5	generate inode via path and time	2022-03-14 00:03:29 -07:00
chrislu	5cba8e51c5	refactor	2022-03-13 18:34:57 -07:00
chrislu	f2f68f675e	write to disk during random writes, limiting total disk spaces used	2022-03-13 18:17:35 -07:00
chrislu	2f4cd20f90	tests skip reader pattern monitoring	2022-03-13 18:15:53 -07:00
chrislu	53513475bf	mount: add back random read support ... avoid too much memory used also	2022-03-13 01:38:52 -08:00
chrislu	f70c1e449b	add useful doc link	2022-03-13 00:14:50 -08:00
chrislu	b20ddc57a7	mount: return open status	2022-03-12 22:38:14 -08:00
chrislu	b40d252761	mount: chmod for root	2022-03-12 12:10:56 -08:00
chrislu	3a6eb8ca5f	default bind to one ip address ... fix https://github.com/chrislusf/seaweedfs/issues/1937	2022-03-11 14:02:39 -08:00
Konstantin Lebedev	f53cff045f	checks disk file exist	2022-03-10 18:58:56 +05:00
Konstantin Lebedev	834210a9dc	avoid connect to the old filler address ... https://github.com/chrislusf/seaweedfs/issues/2545	2022-03-10 15:24:45 +05:00
chrislu	197ade6aeb	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-09 22:26:55 -08:00
chrislu	011a41b561	add back writes to swap file when too many in memory chunks are used.	2022-03-09 22:26:51 -08:00
zzq09494	2ea18fca48	fix:support some databases which not support 'IF NOT EXISTS' keyword of 'create table command'	2022-03-10 13:37:34 +08:00
banjiaojuhao	f28dbbe5c5	[bugfix] filer: 1. Delete uploaded chunks when upload failed. 2. Report error when upload is interrupted by user.	2022-03-10 11:40:39 +08:00
chrislu	e2b07737da	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-08 23:02:33 -08:00
chrislu	dc204dd137	fix nil entry ... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x1d340b4] goroutine 130523 [running]: github.com/chrislusf/seaweedfs/weed/filer.FileSize(...) /code/seaweedfs/weed/filer/filechunks.go:26 github.com/chrislusf/seaweedfs/weed/mount.(*WFS).Lookup(0xc000866d80, 0x1, 0xc002897f40, {0xc004b00980, 0x39}, 0x1ec19e0) /code/seaweedfs/weed/mount/weedfs_dir_lookup.go:59 +0x654 github.com/hanwen/go-fuse/v2/fuse.doLookup(0xc00033c000, 0xc00033c000) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/opcode.go:333 +0x6b github.com/hanwen/go-fuse/v2/fuse.(*Server).handleRequest(0xc000ab2420, 0xc00033c000) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:483 +0x1f3 github.com/hanwen/go-fuse/v2/fuse.(*Server).loop(0xc000ab2420, 0x0) /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:456 +0x110 created by github.com/hanwen/go-fuse/v2/fuse.(*Server).readRequest /code/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.1.0/fuse/server.go:323 +0x534	2022-03-08 23:02:30 -08:00
banjiaojuhao	f7f2a597dd	minor	2022-03-08 16:22:55 +08:00
banjiaojuhao	d61bea9038	[bugfix] filer: In file modification, old chunks will be mis-deleted when they are merged(Manifestized).	2022-03-08 16:22:55 +08:00
chrislu	3aeee3d748	ensure releasing file handle	2022-03-07 14:01:24 -08:00
chrislu	8136384473	remove debug message	2022-03-07 11:22:26 -08:00
banjiaojuhao	b9ff7723dd	[bugfix] filer: nil pointer dereference	2022-03-07 23:26:25 +08:00
chrislu	da3d330616	s3 and filer transport using unix domain socket instead of tcp	2022-03-07 02:00:14 -08:00
chrislu	0cb17b45b1	refactoring	2022-03-07 01:59:01 -08:00
chrislu	6d3db4445b	buffer for all range requests	2022-03-07 01:56:47 -08:00
chrislu	f3bcbeb60a	a little optimization	2022-03-07 00:24:59 -08:00
chrislu	f7ee60996c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-07 00:08:09 -08:00
chrislu	6f3ec989a7	fix manifest batch size	2022-03-07 00:07:53 -08:00
Chris Lu	1e7fcef581	Merge pull request #2729 from banjiaojuhao/filer_metadata-resolve-manifest ... filer: support get metadata with resolved manifest chunk	2022-03-07 00:05:46 -08:00
chrislu	bb0b784544	minor	2022-03-07 00:04:59 -08:00
banjiaojuhao	bfcc9ca808	filer: support metadata with resolved manifest chunk	2022-03-07 15:47:51 +08:00
banjiaojuhao	71f3046841	filer: add back isAppend function	2022-03-07 15:41:07 +08:00
chrislu	0ba4e4cd23	2.93	2022-03-06 18:54:12 -08:00
chrislu	46a28b8819	mount: adjust disk space based on quota	2022-03-06 17:22:49 -08:00
chrislu	ede6ce44c6	fix test	2022-03-06 17:09:55 -08:00
chrislu	21ef152423	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-03-06 17:04:24 -08:00
chrislu	f3442e36e6	mount: quota adjust error type to be syscall.ENOSPC	2022-03-06 17:04:21 -08:00
Chris Lu	89cd0c27a8	Merge pull request #2728 from kmlebedev/set_default_leveldb2	2022-03-06 07:26:30 -08:00
Konstantin Lebedev	cf444ebd07	Set default leveldb2 enabled ... avoid Filer store is enabled for both leveldb2 and mysql	2022-03-06 18:27:25 +05:00
banjiaojuhao	a07c93f4dd	filer: reset bytesBuffer before use	2022-03-06 21:07:36 +08:00
chrislu	c7e8ac18f0	mount: quota for one mounted collection ... related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48	2022-03-06 02:44:40 -08:00
chrislu	b7c992f410	add flag to enforce quota	2022-03-05 22:10:43 -08:00
chrislu	f1713c96ae	avoid possible runtime error: index out of range [0] with length 0	2022-03-05 21:14:31 -08:00
chrislu	da76af187f	mount: avoid possible index out of bounds error	2022-03-04 22:36:01 -08:00
chrislu	6e49e75a5b	use logical number of files and sizes for statistics and quota	2022-03-04 18:47:44 -08:00
chrislu	f51e20028a	mount: avoid comma in mount options ... fix https://github.com/chrislusf/seaweedfs/issues/2719	2022-03-03 03:42:29 -08:00
chrislu	dc0f48682c	unused	2022-03-03 03:41:35 -08:00
chrislu	28b8974a3a	mount: fix directory pagination when using midnight commander	2022-03-03 02:59:31 -08:00
chrislu	a96d4254e9	filer, s3, volume server: a bit memory optimization	2022-03-02 20:15:28 -08:00
chrislu	6fbbc78574	stream reading a whole chunk	2022-03-02 13:50:46 -08:00
chrislu	784583afc6	avoid pool memory allocation if too large	2022-03-02 13:50:28 -08:00
chrislu	ba14307319	2.92	2022-02-28 15:22:19 -08:00
chrislu	fcf3714443	mount: add back support for filer.path	2022-02-28 12:16:53 -08:00
chrislu	80c017907b	filer.backup: fix backing up encrypted chunks ... I have done filer.backup test: replication.toml: [sink.local] enabled = true directory = "/srv/test" ___ system@dat1:/srv/test$ weed filer.backup -filer=app1:8888 -filerProxy I0228 12:39:28 19571 filer_replication.go:129] Configure sink to local I0228 12:39:28 19571 filer_backup.go:98] resuming from 2022-02-28 12:04:20.210984693 +0100 CET I0228 12:39:29 19571 filer_backup.go:113] backup app1:8888 progressed to 2022-02-28 12:04:20.211726749 +0100 CET 0.33/sec system@dat1:/srv/test$ ls -l total 16 drwxr-xr-x 2 system system 4096 Feb 28 12:39 a -rw-r--r-- 1 system system 48 Feb 28 12:39 fu.txt -rw-r--r-- 1 system system 32 Feb 28 12:39 _index.html -rw-r--r-- 1 system system 68 Feb 28 12:39 index.php system@dat1:/srv/test$ cat fu.txt ? ?=?^??`?f^};?{4?Z%?X0=??rV????|"?1??踪~?? system@dat1:/srv/test$ On the active mount on the target server it's: system@app1:/srv/app$ ls -l total 2 drwxrwxr-x 1 system system 0 Feb 28 12:04 a -rw-r--r-- 1 system system 20 Feb 28 12:04 fu.txt -rw-r--r-- 1 system system 4 Feb 28 12:04 _index.html -rw-r--r-- 1 system system 40 Feb 28 12:04 index.php system@app1:/srv/app$ cat fu.txt This is static boy! Filer was started with: weed filer master="app1:9333,app2:9333,app3:9333" -encryptVolumeData It seems like it's still encrypted?	2022-02-28 10:07:06 -08:00
chrislu	aad62ee148	mount: mark directory uncached if forgotten	2022-02-28 02:08:24 -08:00
chrislu	554e239097	release file handle	2022-02-28 01:23:14 -08:00
chrislu	3639fedd01	mount: fix fsync opened and renamed files	2022-02-28 00:34:17 -08:00
chrislu	63a9d8f01d	ensure inodes are not duplicating unless hardlinked	2022-02-27 23:13:49 -08:00
chrislu	de77d00c81	correctly clean up for a file	2022-02-27 23:12:28 -08:00
chrislu	18543c6e8b	minor	2022-02-27 23:11:09 -08:00
chrislu	09cd00f356	2.91	2022-02-27 04:03:39 -08:00
chrislu	941ced60a4	download 2 chunks if at the beginning of a file	2022-02-27 03:57:24 -08:00
chrislu	f9d9eed0c9	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-27 03:41:35 -08:00
chrislu	d602d68fd1	remove dead code	2022-02-27 03:41:32 -08:00
Chris Lu	6c07af6014	Merge pull request #2715 from gfx-labs/acl-pr	2022-02-27 03:27:32 -08:00
chrislu	c3792c8352	remove dead code	2022-02-27 03:03:19 -08:00
chrislu	aa9eef81e6	retire mount v1	2022-02-27 02:57:27 -08:00
eddy-gfx	fb940dd807	Merge branch 'chrislusf:master' into acl-pr	2022-02-27 04:52:08 -06:00
elee	881a0fe806	ensure compatibility	2022-02-27 04:50:59 -06:00
elee	954ad98e0d	set canned acl on replication create	2022-02-27 04:49:31 -06:00
chrislu	6e1ab97988	use debug option to see operations	2022-02-27 02:02:30 -08:00
chrislu	2112d99140	mount2: add back readonly mode	2022-02-27 01:13:32 -08:00
chrislu	9ef5bb20f6	mount2: invalidate fuse cache for replaced inode	2022-02-27 00:00:23 -08:00
chrislu	4ddcbaab57	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-26 23:20:47 -08:00
chrislu	551d00d51a	prefetch other chunks when stream reading	2022-02-26 23:20:45 -08:00
Chris Lu	e1d3dd5e18	Merge pull request #2712 from guo-sj/correct_comments	2022-02-26 09:16:12 -08:00
guosj	82cad5e330	correct comments	2022-02-26 22:44:26 +08:00
chrislu	7b1a713d2a	remove dead code	2022-02-26 03:23:15 -08:00
chrislu	3a58b7bac6	a little safer	2022-02-26 03:23:06 -08:00
chrislu	708e14fcfa	avoid possible too big memory allocation	2022-02-26 03:22:41 -08:00
chrislu	3345a50d9b	prefetch 2 chunks	2022-02-26 03:06:17 -08:00
chrislu	86ce69f709	remove logs	2022-02-26 03:00:20 -08:00
chrislu	b2a148cb4c	use file size as max range	2022-02-26 03:00:08 -08:00
chrislu	2ab0ad24a3	use memory pool	2022-02-26 02:59:19 -08:00
chrislu	28b395bef4	better control for reader caching	2022-02-26 02:16:47 -08:00
chrislu	3ad5fa6f6f	chunk cache adds function ReadChunkAt	2022-02-25 21:55:04 -08:00
chrislu	fc7a4957ea	fix mount2 options	2022-02-25 21:22:44 -08:00
chrislu	72c0233938	less logs	2022-02-25 15:34:24 -08:00
chrislu	101e6d80b0	mount2: listXattr return ok if xattr is empty	2022-02-25 14:38:56 -08:00
chrislu	03466f955e	rename: delete source entry metadata only, skipping hard links	2022-02-25 02:57:54 -08:00
chrislu	8080fe4cc1	logs	2022-02-25 02:56:23 -08:00
chrislu	e423548673	rename: pass along entry metadata	2022-02-25 02:53:37 -08:00
chrislu	e8110bb54c	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-25 01:17:28 -08:00
chrislu	202a29d014	refactoring	2022-02-25 01:17:26 -08:00
chrislu	91d6785cf3	define metadata action types	2022-02-25 00:54:16 -08:00
chrislu	be3fc77391	mount2: use consistent inode	2022-02-25 00:53:27 -08:00
Chris Lu	9873bae115	Merge pull request #2706 from guo-sj/fix_log_info	2022-02-24 22:48:19 -08:00
guosj	121b31f750	fix incorrect log information	2022-02-25 13:41:20 +08:00
chrislu	ceaf993a27	mount2: add rdev	2022-02-24 14:51:25 -08:00
chrislu	95717d1006	simpler output	2022-02-24 13:50:08 -08:00
chrislu	b05962b90e	rename: handle hard links	2022-02-24 02:59:00 -08:00
chrislu	e31ec04f4f	mount2: POSIX deleted opened file nlink should be 0	2022-02-24 01:59:37 -08:00
chrislu	419e355e9e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-24 01:31:07 -08:00
chrislu	91f0481f4e	mount2: SetAttr set mode correctly	2022-02-24 01:31:04 -08:00
Chris Lu	97a4b66df7	Merge pull request #2704 from guo-sj/fix_bugs_in_return_value ... fix return value in storage/volume_vacuum.go:444	2022-02-24 00:49:29 -08:00
chrislu	abe7214c1f	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-24 00:37:59 -08:00
chrislu	b93d57da31	mount2: dir read opened file	2022-02-24 00:37:57 -08:00
guosj	3e7aa1caf5	fix return value in storage/volume_vacuum.go:444	2022-02-24 15:54:36 +08:00
Chris Lu	6834df77a0	Merge pull request #2701 from guo-sj/fix_bugs_in_return_value	2022-02-23 19:23:37 -08:00
guosj	26f3ab8d4b	fix a return bug in func (c *commandVacuum) Do	2022-02-24 09:04:38 +08:00
chrislu	320637dc7a	use "mv.from" for moving files	2022-02-23 15:34:42 -08:00
chrislu	c29bc9a367	fix error handling	2022-02-23 15:34:25 -08:00
Chris Lu	da58c748bc	Merge pull request #2698 from guo-sj/fix_bugs_in_return_value	2022-02-23 00:41:30 -08:00
Chris Lu	bd092d8318	Merge pull request #2697 from guo-sj/fix_bugs_in_return_value	2022-02-23 00:30:52 -08:00
guosj	d68c27f82d	fix another return value bug	2022-02-23 16:21:25 +08:00
guosj	8f9aa0cddd	fix bugs in return value	2022-02-23 16:17:48 +08:00
Tuan Vuong	d2ec62656d	initialize master address in iam options	2022-02-23 12:01:54 +07:00
banjiaojuhao	6ab09e9071	filer_http: support uploading file with offset	2022-02-22 00:15:00 +08:00
chrislu	497ebbbd45	2.90	2022-02-20 22:00:13 -08:00
chrislu	56da3494cb	leveldb3: adjust memory allocation for each bucket	2022-02-20 13:40:13 -08:00
banjiaojuhao	e6126cef62	filer_web: support moving entry	2022-02-20 23:56:23 +08:00
chrislu	7bc67399e4	listing for filer stores without prefixed query: break if no more progress	2022-02-19 10:12:51 -08:00
chrislu	248c0c8087	minor	2022-02-19 09:54:09 -08:00
chrislu	4ee0a6f47b	filer store: reduce one possible listing operation	2022-02-19 00:43:42 -08:00
chrislu	b3594278c9	optimize a bit	2022-02-18 23:36:10 -08:00
chrislu	61811dc2f1	comments	2022-02-18 22:14:40 -08:00
chrislu	daa27b2119	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-18 20:55:10 -08:00
chrislu	9014d00fd0	Revert "s3: listObjectParts return ErrNoSuchUpload if does not exist" ... This reverts commit 6cf2e7d493.	2022-02-18 20:54:54 -08:00
banjiaojuhao	4c30934cd9	filer: support get file entry	2022-02-18 22:52:26 +08:00
chrislu	63062ed7f0	mount2: fix unlink	2022-02-18 01:10:53 -08:00
chrislu	93e12d5f30	fix build	2022-02-18 00:48:00 -08:00
chrislu	d62370d4e0	add todo	2022-02-18 00:47:15 -08:00
chrislu	e8ce30fdc5	mount2: adjust file mode	2022-02-18 00:47:02 -08:00
chrislu	f9d33f70b0	return fuse.Status when looking up by inode	2022-02-18 00:45:43 -08:00
chrislu	b9cf4f12fc	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-16 21:33:51 -08:00
chrislu	49b84b6e2a	list entries while reading from remote	2022-02-16 21:32:15 -08:00
Chris Lu	38dcaaa76e	Merge pull request #2683 from guo-sj/fix_fsconfigure_bug ... fix minor bug in commandFsConfigure.Do	2022-02-16 18:16:47 -08:00
guosj	ca121ecd66	fix minor bug in commandFsConfigure.Do	2022-02-17 09:57:52 +08:00
chrislu	65a19e3abc	fix listing with correct inode	2022-02-16 17:01:39 -08:00
chrislu	6ac066d1dc	count lookup or not	2022-02-16 16:49:03 -08:00
chrislu	a129bda7d9	sync data first before stopping	2022-02-16 09:11:34 -08:00
chrislu	118d0e01a8	less logs	2022-02-16 08:45:07 -08:00
chrislu	a6bc67c34c	less logs	2022-02-16 08:38:51 -08:00
chrislu	2facd65998	fix second listing	2022-02-16 08:16:27 -08:00
chrislu	1013fc90d8	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-16 07:57:11 -08:00
chrislu	22739c653a	clean up	2022-02-16 07:57:08 -08:00
Chris Lu	3cb19cf6db	Merge pull request #2680 from kmlebedev/volume_healthz	2022-02-16 04:43:44 -08:00
Konstantin Lebedev	9ea09cc41c	healthz check to avoid drain pod with last replicas	2022-02-16 14:18:36 +05:00
chrislu	65bfeafb5a	add back mkdir	2022-02-16 01:09:21 -08:00
chrislu	37e8fce841	clean up cache on exit	2022-02-16 00:39:21 -08:00
chrislu	1560ec7e26	remove unused code	2022-02-16 00:37:24 -08:00
chrislu	3cbce878f2	mount2: fix directory pagination	2022-02-15 22:42:10 -08:00
chrislu	df51e0c042	mongodb: remove ErrNoDocuments checking since it is only for creating single record, not for querying	2022-02-15 16:18:36 -08:00
chrislu	98cce6a150	mongodb: ensure closing cursor	2022-02-15 16:15:44 -08:00
Chris Lu	aa7fc299b8	Merge pull request #2676 from banjiaojuhao/add_filer_store-redis_lua ... FilerStore: add redis_lua	2022-02-15 12:10:34 -08:00
banjiaojuhao	fc3b75f2f8	weed/operation/delete_content.go: nil pointer dereference	2022-02-15 23:07:03 +08:00
banjiaojuhao	b5ec346700	FilerStore: add redis_lua	2022-02-15 20:54:57 +08:00
chrislu	17ac5244c3	mount2: avoid double listing directories	2022-02-15 01:44:17 -08:00
guosj	db7766323e	complete project code, remain test code	2022-02-15 17:35:33 +08:00
chrislu	222798d926	mount2: fix for read dir plus on linux	2022-02-15 00:29:17 -08:00
chrislu	4e72863ba5	mount2 add debug mode	2022-02-15 00:26:30 -08:00
Chris Lu	64afbc5235	Merge pull request #2640 from Radtoo/fsck_with_delete ... Added basic deletion capabilities to fsck.	2022-02-14 23:00:01 -08:00
chrislu	4e181db21a	mount: default disable cache ... * Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43 * Improve read write benchmarks * Improve AI training performance. Most of the files are just read once.	2022-02-14 20:42:33 -08:00
chrislu	ebc22625b4	less noisy on mac	2022-02-14 16:38:01 -08:00
chrislu	377bf31445	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-14 13:48:51 -08:00
chrislu	e8420aaed7	fix building for windows freebsd	2022-02-14 13:48:48 -08:00
Chris Lu	d3ee621fce	Merge pull request #2661 from garenchan/ck-dev1	2022-02-14 10:08:37 -08:00
Konstantin Lebedev	526094d2da	StopTimeout 30 sec	2022-02-14 21:42:27 +05:00
Konstantin Lebedev	275e9a4e86	reduce to default http server KillTimeout and StopTimeout	2022-02-14 21:38:24 +05:00
garenchan	bd032eabe7	[UPDATE] Make heartbeat interval and election timeout of masters configurable.	2022-02-14 21:09:07 +08:00
chrislu	ff666104c4	fix GOOS	2022-02-14 03:14:05 -08:00
chrislu	05724a68d4	skip other OS	2022-02-14 02:59:51 -08:00
chrislu	4244ef8b72	fix building on linux	2022-02-14 02:59:27 -08:00
chrislu	b9c2bff931	clean up	2022-02-14 02:14:26 -08:00
chrislu	a990cd29cd	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-14 01:52:25 -08:00
chrislu	6a40fd1c65	2.89	2022-02-14 01:52:16 -08:00
Chris Lu	305418fac5	Merge pull request #2667 from kmlebedev/syncReplicatedWrite ... atomically write to replicas	2022-02-14 01:48:21 -08:00
Chris Lu	aa13168b4d	Merge pull request #2668 from chrislusf/mount2 ... Mount2	2022-02-14 01:41:09 -08:00
chrislu	fe57a2e770	file set attribute	2022-02-14 01:36:10 -08:00
Konstantin Lebedev	0ed76a0556	clearly	2022-02-14 14:10:06 +05:00
chrislu	dbeeda8123	listen for metadata updates	2022-02-14 01:09:31 -08:00
Konstantin Lebedev	36013f63ed	https://github.com/chrislusf/seaweedfs/issues/2648	2022-02-14 13:59:12 +05:00
chrislu	7286e525ad	support write	2022-02-13 23:27:11 -08:00
chrislu	2b955c1713	support read	2022-02-13 22:50:44 -08:00
chrislu	f3c1e00521	rename	2022-02-13 19:16:56 -08:00
chrislu	bb9919b07a	add open release, refactor	2022-02-13 19:14:34 -08:00
Eng Zer Jun	b92df1654c	test: use T.TempDir to create temporary test directory ... The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete. Reference: https://pkg.go.dev/testing#T.TempDir Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>	2022-02-14 10:38:13 +08:00
chrislu	072f923a9b	add doc	2022-02-13 16:56:35 -08:00
chrislu	3d0e9e5197	supports renaming	2022-02-13 16:34:57 -08:00
chrislu	417b59b893	doc	2022-02-13 14:37:47 -08:00
chrislu	24290fed9d	add comments	2022-02-13 06:02:21 -08:00
chrislu	6200b6abb1	avoid fatal message in some edge cases	2022-02-13 05:59:10 -08:00
chrislu	6a921e15f3	forget() factor in nlookup	2022-02-13 05:49:29 -08:00
chrislu	f8af0f93d9	support link	2022-02-13 05:29:43 -08:00
chrislu	0381338d40	fix import	2022-02-13 04:23:06 -08:00
chrislu	a1ef0e48a9	doc	2022-02-13 04:22:02 -08:00
chrislu	be40ff6048	add symlink	2022-02-13 03:50:16 -08:00
chrislu	3d93570979	supports forget	2022-02-13 03:31:47 -08:00
chrislu	6a42cb6b0b	supports mknod, unlink	2022-02-13 03:09:24 -08:00
chrislu	813b868b9a	add rmdir	2022-02-13 01:43:11 -08:00
chrislu	e85ca10a1a	add mkdir	2022-02-13 01:34:19 -08:00
chrislu	21046c6a28	split files	2022-02-13 01:05:30 -08:00
chrislu	a4c9223b9d	support xattr	2022-02-13 00:58:46 -08:00
chrislu	c81833a192	add directory setAttr	2022-02-12 23:08:56 -08:00
chrislu	5c48c23235	remove println	2022-02-12 22:45:07 -08:00
chrislu	7cfbf1e85f	fix ok status	2022-02-12 22:41:45 -08:00
chrislu	4c75fd5f9c	sync format	2022-02-12 22:41:29 -08:00
chrislu	b0a5193e32	working	2022-02-12 22:21:30 -08:00
chrislu	661a34e23d	Merge branch 'master' into mount2	2022-02-12 17:58:32 -08:00
chrislu	5e814afe88	blocks count	2022-02-12 17:58:12 -08:00
Radtoo	724ce04b1c	We picked the second flag's name.	2022-02-13 00:53:35 +01:00
chrislu	5a0a709016	it runs, but directory listing output is not showing up	2022-02-12 05:27:16 -08:00
chrislu	866981d8ac	rename	2022-02-12 02:49:15 -08:00
chrislu	72faae91e1	implement read directory and read directory plus	2022-02-12 02:48:44 -08:00
chrislu	a10c28ba82	simplify	2022-02-12 01:59:36 -08:00
chrislu	f4d88862c4	can attr root directory	2022-02-12 01:54:16 -08:00
chrislu	180445f5a8	change to use fuse file system	2022-02-11 21:35:09 -08:00
chrislu	45a0fda9bd	need to follow https://github.com/hanwen/go-fuse/blob/master/fuse/api.go	2022-02-11 03:12:52 -08:00
chrislu	f87da798a4	to be re-written following fuse virtual file system	2022-02-11 03:09:30 -08:00
chrislu	9a913457dd	supports stats	2022-02-10 23:23:47 -08:00
chrislu	b6143de52a	mount with name	2022-02-10 22:43:55 -08:00
chrislu	7a0c35674c	clean up previously mounted folder	2022-02-10 20:46:53 -08:00
chrislu	c3f9d9fa2e	initial setup	2022-02-10 20:32:13 -08:00
Radtoo	fbe7ed7927	Revert the the flag to "reallyDeleteFromVolume".	2022-02-10 20:01:48 +01:00
chrislu	21aaa4c1f1	ec.encode: calculate free ec slots based on (maxVolumeCount-volumeCount) ... fix https://github.com/chrislusf/seaweedfs/issues/2642	2022-02-08 01:51:13 -08:00
chrislu	13f6ec1c4e	test checking ec distribution	2022-02-08 01:50:05 -08:00
chrislu	f34c2ff7c5	use embed txt	2022-02-08 01:14:27 -08:00
chrislu	c8c7c10c3f	volume.tier.move: avoid double counting ... related to https://github.com/chrislusf/seaweedfs/issues/2637	2022-02-08 00:57:35 -08:00
chrislu	f18803424a	volume.balance: add delay during tight loop ... fix https://github.com/chrislusf/seaweedfs/issues/2637	2022-02-08 00:53:55 -08:00
chrislu	9860405974	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-07 23:08:57 -08:00
chrislu	85c1615b43	filer read empty file may cause OOM in some cases ... fix https://github.com/chrislusf/seaweedfs/issues/2641	2022-02-07 23:08:54 -08:00
Chris Lu	a2ac540ecc	Merge pull request #2645 from guol-fnst/fix_Precedence ... fix preconditions	2022-02-07 18:32:16 -08:00
root	7f0c793083	fix preconditions according to https://tools.ietf.org/id/draft-ietf-httpbis-p4-conditional-26.html#preconditions	2022-02-08 10:13:19 +08:00
chrislu	b1cff07ab0	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-02-07 11:38:01 -08:00
chrislu	81f86c381d	s3: avoid nil response ... fix https://github.com/chrislusf/seaweedfs/issues/2636	2022-02-07 11:37:59 -08:00
Chris Lu	f46763d74f	Merge pull request #2644 from kmlebedev/vacum_fsync ... updated needle with fsync	2022-02-07 11:33:58 -08:00
Konstantin Lebedev	ef541972f8	updated needle with fsync	2022-02-08 00:10:53 +05:00
Chris Lu	1b7fb3ce84	Merge pull request #2643 from kmlebedev/graceful_volume_stop	2022-02-07 09:58:26 -08:00
Konstantin Lebedev	fb97e234c9	skips compact if store is stopping	2022-02-07 20:16:15 +05:00
chrislu	9405eaefdb	filer.sync: fix replicating partially updated file ... Run two servers with volumes and fillers: server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002 server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008 Run Active-Passive filler.sync: filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive Upload file to 11007 port: curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/" If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again: curl "http://localhost:11007/testFacebook/9.xml" EQUALS curl "http://localhost:11001/testFacebook/9.xml" However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file Снимок экрана 2022-02-07 в 14 21 11 This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten. Deleted chunks are not actually deleted on the server to which the file is replicated.	2022-02-07 03:46:28 -08:00
Radtoo	fbb14e0ea8	Adding separate toggle to purge absent vols ... While this toggle is basically required to clean out entries for deleted volumes, having a separate description + toggling this separately seems like a good idea so people get a chance to check if their volumes are all mounted/connected as expected. Also renamed forcePurge to just purge.	2022-02-06 23:46:52 +01:00
Radtoo	6b17f45da2	Also delete paths for which a volume is entirely absent.	2022-02-06 23:22:04 +01:00
Radtoo	fa0cfdfb7f	Added basic deletion capabilities to fsck.	2022-02-06 17:05:58 +01:00
chrislu	433fde4b18	move error to a separate file ... This file contains metric names for all errors The naming convention is ErrorSomeThing = "error.some.thing"	2022-02-04 22:57:51 -08:00
Chris Lu	a23fcb9a7c	Merge pull request #2634 from kmlebedev/errorMetrics ... error metrics for filer and store	2022-02-04 22:35:13 -08:00
Chris Lu	247bbabda5	Merge pull request #2632 from lapshin-vitaly/s3api_errors ... add s3api error for copy in file, not directory	2022-02-04 22:34:34 -08:00
chrislu	ced3b89395	add util package	2022-02-04 21:34:58 -08:00
chrislu	affe3c2c12	change to util.WriteFile	2022-02-04 21:32:27 -08:00
chrislu	76e297d64f	sync call to write file, avoid vif loading error ... fix https://github.com/chrislusf/seaweedfs/issues/2633	2022-02-04 11:14:04 -08:00
zerospiel	f3364fec99	weed/s3api: rearrange s3 methods handlers to ensure correct methods requesting ... Otherwise current calls for some methods (i.e. GetObjectAcl) ends up with wrong method selection (i.e. GetObject). Added generic comment rule of traversing methods	2022-02-04 15:14:48 +03:00
Konstantin Lebedev	9978f54acf	fix metric names	2022-02-04 16:45:16 +05:00
Konstantin Lebedev	3f4e17aa24	error metrics for filer and store	2022-02-04 14:07:14 +05:00
Lapshinn Vitaly	6bdc274d4d	add s3api error for copy in file, not directory	2022-02-04 03:28:37 +03:00
chrislu	6cf2e7d493	s3: listObjectParts return ErrNoSuchUpload if does not exist ... ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api abort-multipart-upload --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc ubuntu@prod-master-1:~$ aws --endpoint http://10.244.15.66:8333 s3api list-parts --bucket prod-cache --key multipart-test --upload-id 5347f936-6adc-43de-8e5c-1fd137c3b2bc { "Initiator": null, "Owner": null, "StorageClass": "STANDARD" } If we abort a multipart upload, it appears that records are left behind. We should get a 404 NoSuchKey error.	2022-02-03 12:34:16 -08:00
zerospiel	b54a65ba5a	weed/s3api: added new bucket handlers for more compatibility with AWS S3 ... Protocol Otherwise any requests to the underlying handlers results in calls to ListObjects (v1) that may intensively load gateway and volume servers. Added the following handlers with default responses: - GetBucketLocation - GetBucketRequestPayment Added the following handlers with NotFound and NotImplemented responses: - PutBucketAcl - GetBucketPolicy - PutBucketPolicy - DeleteBucketPolicy - GetBucketCors - PutBucketCors - DeleteBucketCors	2022-02-03 17:17:05 +03:00
chrislu	6bee1e9714	[volume.check.disk] fix wrong logic to compare ... fix https://github.com/chrislusf/seaweedfs/issues/2626 minuend - subtrahend	2022-02-01 07:48:28 -08:00
chrislu	7270067289	2.88	2022-01-30 20:25:26 -08:00
chrislu	84c9bc4389	edge case: old entry was not replicated to remote storage	2022-01-30 20:23:24 -08:00
chrislu	b8490fe427	adjust volume count even when not applying the changes	2022-01-28 19:11:46 -08:00
Chris Lu	b3f0f170b4	Merge pull request #2618 from divanikus/master	2022-01-28 03:34:52 -08:00
chrislu	b9b684194f	remove max connection age ... following https://github.com/grpc/grpc-go/issues/3170#issuecomment-552517779	2022-01-27 02:28:22 -08:00
divanikus	67e3fe996a	async volumeTierMove	2022-01-26 18:22:31 +03:00
chrislu	62d815d1ca	use limited in memory buffer instead of swap file	2022-01-26 00:36:19 -08:00
Chris Lu	9596fce562	Merge pull request #2613 from kmlebedev/masterPromMetrics ... Master prom metrics	2022-01-25 08:13:14 -08:00
Konstantin Lebedev	5c9259fa3c	fix metrics master name	2022-01-25 14:42:47 +05:00
chrislu	7328962009	revert "POSIX: should not delete if a directory is not empty" ... revert 0c75f15062 Reported: Hi, about commit: 0c75f15062 POSIX: should not delete if a directory is not empty It should still delete with a command like rm -rf "${path}"/ because it is a forced delete, but now it gets fail to delete non-empty folder: [...] Can you enable the delete if it is forced?	2022-01-24 14:21:50 -08:00
Konstantin Lebedev	c9952759c4	metrics master is leader	2022-01-24 20:13:07 +05:00
Konstantin Lebedev	28efe31524	new master metrics	2022-01-24 19:09:43 +05:00
chrislu	3bba2124ef	use a sliding window of in-memory writable chunks	2022-01-23 23:02:05 -08:00
chrislu	520591e6ea	reset swap file chunk after uploading	2022-01-23 18:30:53 -08:00
chrislu	e185d90d24	2.87	2022-01-23 16:18:55 -08:00
chrislu	4a311c7f5e	dedup local metadata subscribers ... fix https://github.com/chrislusf/seaweedfs/discussions/2542	2022-01-23 16:14:22 -08:00
chrislu	bb6854b972	adjust log	2022-01-22 12:35:09 -08:00
chrislu	2bdd737971	rename file	2022-01-22 08:28:35 -08:00
chrislu	643bbbeb49	rename	2022-01-22 08:27:40 -08:00
chrislu	18ed06b420	use memory when under 16 chunks	2022-01-22 08:11:01 -08:00
chrislu	8e80f3cd65	move upload pipeline locking to a different file	2022-01-22 08:09:55 -08:00
chrislu	dde34fa99d	rename	2022-01-22 08:08:01 -08:00
chrislu	a2aa542370	rename	2022-01-22 08:07:11 -08:00
chrislu	c376ccc5a5	swap file based random write large file upload	2022-01-22 08:06:53 -08:00
chrislu	b9ae16fbc5	fix memory allocation	2022-01-22 08:05:04 -08:00
chrislu	1ee828b768	refactor ... do not expose internal offset	2022-01-22 06:34:29 -08:00
chrislu	02d0c12cdd	rename	2022-01-22 06:00:10 -08:00
chrislu	5dea5c0449	refactor	2022-01-22 05:59:07 -08:00
chrislu	8aa6bf0bb9	refactoring	2022-01-22 05:40:10 -08:00
chrislu	d97bd54e63	just refactoring	2022-01-22 04:18:54 -08:00
chrislu	482014f9da	rename file	2022-01-22 03:50:18 -08:00
chrislu	3b4a9addaf	rename	2022-01-22 01:46:10 -08:00
chrislu	4acfc098e9	re-order	2022-01-22 01:43:14 -08:00
chrislu	e71dcfb3a6	add logging for memory allocation	2022-01-22 01:35:12 -08:00
chrislu	9d0f58c329	skip printing fs configuration	2022-01-21 13:29:47 -08:00
chrislu	ce2049cdb6	refactoring, move genFn before saveFn	2022-01-21 12:08:58 -08:00
chrislu	e47f63d159	enforce bucket quota	2022-01-21 02:34:42 -08:00
chrislu	6e57d8d0de	s3: check bucket usage and adjust read only according to quota	2022-01-21 02:15:27 -08:00
chrislu	606667f205	able to configure the quota for a bucket	2022-01-21 01:42:20 -08:00
chrislu	b1063162b6	display bucket quota	2022-01-21 00:55:04 -08:00
chrislu	f103491912	s3: list bucket size from weed shell	2022-01-21 00:26:49 -08:00
chrislu	6c7135d77e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-21 00:17:06 -08:00
chrislu	ea57654e34	refactoring	2022-01-21 00:16:50 -08:00
Chris Lu	7c66f3b5fb	Merge pull request #2602 from kmlebedev/master_metrics ... master metricsHttpPort	2022-01-20 09:26:25 -08:00
chrislu	b3e526ba95	url should be always using forward slash	2022-01-19 22:16:26 -08:00
Konstantin Lebedev	77c98b657e	master metricsHttpPort	2022-01-19 21:43:22 +05:00
chrislu	77362700e1	S3: fail fast when "X-Amz-Copy-Source" is a folder ... fix #2593	2022-01-18 12:04:40 -08:00
chrislu	05c3c3f56b	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-17 23:38:13 -08:00
chrislu	9b77f0054e	2.86	2022-01-17 23:38:03 -08:00
Chris Lu	ec254d8a89	Merge pull request #2597 from guol-fnst/gocql_to ... add gocql timeout setting	2022-01-17 23:35:13 -08:00
chrislu	9274557552	keep dirty pages based on temp file	2022-01-17 23:23:49 -08:00
guol-fnst	da9540e666	add gocql timeout setting	2022-01-18 15:21:13 +08:00
chrislu	c87b8f4c30	S3: fail fast when "X-Amz-Copy-Source" is a folder ... fix https://github.com/chrislusf/seaweedfs/issues/2593	2022-01-17 23:09:37 -08:00
chrislu	b2acfd75e9	ensure entry view cache is invalidated	2022-01-17 23:02:30 -08:00
chrislu	f4ad63528a	wait for reading threads to complete before dropping sealed chunks	2022-01-17 22:24:44 -08:00
chrislu	0a3f95ca01	more logs	2022-01-17 20:41:00 -08:00
chrislu	b068bc291d	testing with always resetting entry view cache	2022-01-17 20:07:01 -08:00
chrislu	047446d5ca	remove extra async execution	2022-01-17 15:50:11 -08:00
chrislu	7bf7af971b	more logs	2022-01-17 14:15:10 -08:00
chrislu	fc22071a2f	more logs	2022-01-17 14:02:37 -08:00
chrislu	381f4e73a0	delete actual reference first	2022-01-17 13:56:47 -08:00
chrislu	0ba88596e8	invalidate filehandle entry view cache	2022-01-17 13:53:30 -08:00
chrislu	1734017ba1	add test	2022-01-17 13:40:41 -08:00
chrislu	da7f13e73e	Revert "testing skip memory management" ... This reverts commit 6c908352cb.	2022-01-17 03:21:31 -08:00
chrislu	6c908352cb	testing skip memory management	2022-01-17 03:19:24 -08:00
chrislu	77d9993f38	remove unused variables	2022-01-17 03:19:11 -08:00
chrislu	f710d5ffca	a little speed up	2022-01-17 03:19:00 -08:00
chrislu	fc0628c038	working	2022-01-17 01:53:56 -08:00
chrislu	1bd6d289d4	better locking on file handle	2022-01-15 05:45:29 -08:00
chrislu	2bfeb5d1c8	add filer to iam option	2022-01-15 03:37:52 -08:00
chrislu	b17c426e99	weed server: optionally start IAM service ... related to https://github.com/chrislusf/seaweedfs/issues/2560	2022-01-13 22:49:49 -08:00
chrislu	3c8b74318e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-13 13:03:07 -08:00
chrislu	8907e6a40a	add more help messages	2022-01-13 13:03:04 -08:00
banjiaojuhao	45e9c83421	padding zero for sparse file	2022-01-13 22:21:22 +08:00
chrislu	fe5b9e39cc	POSIX: check permission when removing items	2022-01-13 02:07:39 -08:00
chrislu	1453263b63	remove dead code	2022-01-13 02:02:04 -08:00
chrislu	e69c374956	minor	2022-01-13 02:01:53 -08:00
chrislu	9b954dc0d4	adjust make file	2022-01-13 01:33:13 -08:00
chrislu	f2847f1266	POSIX: check deletion permission	2022-01-12 23:58:11 -08:00
chrislu	0c75f15062	POSIX: should not delete if a directory is not empty	2022-01-12 23:57:54 -08:00
chrislu	de27058d0b	POSIX: differentiate device and char device	2022-01-12 21:45:38 -08:00
chrislu	d400a11832	POSIX: adjust source file ctime ... SeaweedFS uses mtime as ctime	2022-01-12 21:45:18 -08:00
chrislu	b44f05a2d0	POSIX: change timestamp on each attribute change	2022-01-12 19:31:25 -08:00
chrislu	15c01d8b7f	add some notes	2022-01-12 15:04:48 -08:00
chrislu	107a4884a8	shell: tighter memory allocation	2022-01-12 14:59:29 -08:00
chrislu	fec8428fd8	POSIX: different inode for same named different file types	2022-01-12 11:51:13 -08:00
chrislu	e82ad60122	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-12 11:07:49 -08:00
chrislu	caf0a3486b	POSIX: adjust ctime for file truncate	2022-01-12 11:07:39 -08:00
Konstantin Lebedev	edb753ab4d	https://github.com/chrislusf/seaweedfs/issues/2583	2022-01-12 16:04:59 +05:00
chrislu	adfd54e7c4	fix compilation	2022-01-12 01:24:24 -08:00
chrislu	6cc92817dc	add logs for request mode	2022-01-12 01:13:19 -08:00
chrislu	826a7b307e	master: remove hard coded filer settings in master.toml ... fix https://github.com/chrislusf/seaweedfs/issues/2529	2022-01-12 01:11:25 -08:00
chrislu	cd1ad88f30	POSIX: check name is too long ENAMETOOLONG	2022-01-12 00:16:00 -08:00
chrislu	2dcb8cb93b	POSIX: ensure file and directory inodes are different ... this is just an in memory representation. POSIX wants different inode numbers for the same named file or directory.	2022-01-11 23:44:48 -08:00
chrislu	5bb37d5905	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-11 23:29:12 -08:00
chrislu	10ecf80ca1	add a debug capability to list all metadata keys	2022-01-11 23:25:04 -08:00
Kyle Sanderson	9e012001be	filer.copy: don't crash when volume creation fails ... panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7482 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225 panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247] goroutine 7480 [running]: github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0) /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7 created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225	2022-01-11 22:22:39 -08:00
chrislu	1a7d5b5b5e	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2022-01-11 12:24:56 -08:00
chrislu	41daecfdca	Update mount_std.go	2022-01-11 12:23:12 -08:00
chrislu	2d0ccc4d34	add logs	2022-01-11 12:23:01 -08:00
Chris Lu	abe5da7d2c	Merge pull request #2575 from Radtoo/fix_paths2 ... Fix paths2	2022-01-11 12:04:30 -08:00
chrislu	b8fbf19e9a	mount: rename follow POSIX	2022-01-11 03:23:03 -08:00
chrislu	6a12520a96	fix logging	2022-01-10 01:00:11 -08:00
chrislu	cbc055dc2b	mount: file fsync ... fix https://github.com/chrislusf/seaweedfs/issues/2561	2022-01-10 00:52:16 -08:00
chrislu	19555385f7	2.85	2022-01-09 19:30:23 -08:00
Radtoo	389002f195	Using positional arguments rather than option flag to enable better shell usage	2022-01-08 16:52:12 +01:00
Radtoo	fba1efb77a	Now works with a single file too ... Parsing removed from doFixOneVolume Needle init removed from runFix	2022-01-08 16:31:53 +01:00
chrislu	110d5a5233	support fixing a collection of volumes, or volumes under one directory	2022-01-07 14:52:16 -08:00
chrislu	60dc450091	skip fixing read only volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2562	2022-01-06 09:52:28 -08:00
chrislu	3df8f96117	avoid changing inode	2022-01-06 01:36:11 -08:00
chrislu	67b0645808	mount: need to change entry name after renaming	2022-01-05 21:27:41 -08:00
chrislu	4de060daa6	mount: skip special character in the filenames ... fix https://github.com/chrislusf/seaweedfs/issues/2559	2022-01-05 03:57:24 -08:00
chrislu	e76105e2ab	fix auth permission checking	2022-01-03 21:05:20 -08:00
chrislu	a7887166cf	wildcard prefix to restrict access to directories in s3 bucket ... https://github.com/chrislusf/seaweedfs/discussions/2551	2022-01-03 15:39:36 -08:00
chrislu	5799a20f71	2.84	2022-01-02 17:05:19 -08:00
Chris Lu	42c849e0df	Merge branch 'master' into metadata_follow_with_client_id	2022-01-02 01:07:30 -08:00
Chris Lu	9b94177380	Merge pull request #2543 from skurfuerst/seaweedfs-158 ... FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client	2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst	c35660175d	BUGFIX: ensure Authorization header is only added once	2021-12-31 22:06:18 +01:00
Sebastian Kurfuerst	1cd3b6b4e1	BUGFIX: security.toml contained wrong keys	2021-12-31 22:05:41 +01:00
Sebastian Kurfuerst	10404c4275	FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client ... - one JWT for reading and one for writing, analogous to how the JWT between Master and Volume Server works - I did not implement IP `whiteList` parameter on the filer Additionally, because http_util.DownloadFile now sets the JWT, the `download` command should now work when `jwt.signing.read` is configured. By looking at the code, I think this case did not work before. ## Docs to be adjusted after a release Page `Amazon-S3-API`: ``` # Authentication with Filer You can use mTLS for the gRPC connection between S3-API-Proxy and the filer, as explained in [Security-Configuration](Security-Configuration) - controlled by the `grpc.*` configuration in `security.toml`. Starting with version XX, it is also possible to authenticate the HTTP operations between the S3-API-Proxy and the Filer (especially uploading new files). This is configured by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. With both configurations (gRPC and JWT), it is possible to have Filer and S3 communicate in fully authenticated fashion; so Filer will reject any unauthenticated communication. ``` Page `Security Overview`: ``` The following items are not covered, yet: - master server http REST services Starting with version XX, the Filer HTTP REST services can be secured with a JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. ... Before version XX: "weed filer -disableHttp", disable http operations, only gRPC operations are allowed. This works with "weed mount" by FUSE. It does **not work** with the [S3 Gateway](Amazon S3 API), as this does HTTP calls to the Filer. Starting with version XX: secured by JWT, by setting `filer_jwt.signing.key` and `filer_jwt.signing.read.key` in `security.toml`. **This now works with the [S3 Gateway](Amazon S3 API).** ... # Securing Filer HTTP with JWT To enable JWT-based access control for the Filer, 1. generate `security.toml` file by `weed scaffold -config=security` 2. set `filer_jwt.signing.key` to a secret string - and optionally filer_jwt.signing.read.key` as well to a secret string 3. copy the same `security.toml` file to the filers and all S3 proxies. If `filer_jwt.signing.key` is configured: When sending upload/update/delete HTTP operations to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.key`. If `filer_jwt.signing.read.key` is configured: When sending GET or HEAD requests to a filer server, the request header `Authorization` should be the JWT string (`Authorization: Bearer [JwtToken]`). The operation is authorized after the filer validates the JWT with `filer_jwt.signing.read.key`. The S3 API Gateway reads the above JWT keys and sends authenticated HTTP requests to the filer. ``` Page `Security Configuration`: ``` (update scaffold file) ... [filer_jwt.signing] key = "blahblahblahblah" [filer_jwt.signing.read] key = "blahblahblahblah" ``` Resolves: #158	2021-12-30 14:45:27 +01:00
chrislu	34742be029	remove duplicated metadata subscription in filer ... https://github.com/chrislusf/seaweedfs/issues/2545	2021-12-30 01:51:52 -08:00
chrislu	5c87fcc6d2	add client id for all metadata listening clients	2021-12-30 00:23:57 -08:00
chrislu	fb434318e3	dynamically adjust connection timeout ... better fix for https://github.com/chrislusf/seaweedfs/issues/2541	2021-12-29 22:44:39 -08:00
chrislu	5788bf2270	s3: increase timeout limit ... https://github.com/chrislusf/seaweedfs/issues/2541	2021-12-29 22:21:02 -08:00
Sebastian Kurfuerst	fcc09cef6f	Refactor: pass in claim type into security.DecodeJwt	2021-12-29 12:40:41 +01:00
Sebastian Kurfuerst	d156d410ef	rename security.GenJwt to security.GenJwtForVolumeServer	2021-12-29 12:39:41 +01:00
Sebastian Kurfuerst	eda4c43a08	fix typo in error message	2021-12-29 12:38:14 +01:00
chrislu	498661e3bb	mount: remove limits on number of parallel requests	2021-12-28 17:41:01 -08:00
chrislu	9a00c17555	reader: avoid wrong pattern detection due to lock waiting	2021-12-28 16:30:33 -08:00
chrislu	f7a6f6b4c0	if this is enabled, there are some "bus error" with git clone	2021-12-28 13:44:52 -08:00
chrislu	0da2dfd640	fuse: change to direct io mode ... before and after: chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 8192+0 records in 8192+0 records out 1073741824 bytes transferred in 4.534068 secs (236816430 bytes/sec) dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.86s system 84% cpu 4.561 total chrislu$ time dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 8192+0 records in 8192+0 records out 1073741824 bytes transferred in 3.824072 secs (280784948 bytes/sec) dd if=/dev/random of=/Users/chrislu/tmp/mm/testfile bs=131072 count=8192 0.01s user 3.22s system 83% cpu 3.857 total	2021-12-28 12:22:56 -08:00
chrislu	3fd4da34a4	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-28 01:10:38 -08:00
chrislu	2422556456	monitor write pattern: avoid timing due to locking	2021-12-28 01:10:35 -08:00
chenkai	47c30e3add	filer list entries use context to break job	2021-12-28 15:03:41 +08:00
chrislu	80db8b13d8	bug: cleanup function was called twice	2021-12-27 20:53:02 -08:00
chrislu	67b723f74e	Filer Server API support fsync ... fix https://github.com/chrislusf/seaweedfs/issues/2528	2021-12-26 17:28:47 -08:00
chrislu	9f9ef1340c	use streaming mode for long poll grpc calls ... streaming mode would create separate grpc connections for each call. this is to ensure the long poll connections are properly closed.	2021-12-26 00:15:03 -08:00
chrislu	c935b9669e	2.83	2021-12-25 01:01:34 -08:00
chrislu	eb4ad2546f	use proper chunk size limit option	2021-12-24 22:52:18 -08:00
chrislu	41bbf320bb	use 2MB chunk size. cache size is the wrong option	2021-12-24 22:50:19 -08:00
chrislu	982ea85d81	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-24 22:40:07 -08:00
chrislu	083d8e9ece	add stream writer ... this should improve streaming write performance, which is common in many cases, e.g., copying large files. This is additional to improved random read write operations: 3e69d19380...19084d8791	2021-12-24 22:38:22 -08:00
Sebastian Kurfürst	6db49100d6	BUGFIX: add access.ui setting to scaffolded security.toml ... ... The property is read here: b70cb3e0b2/weed/server/volume_server.go (L69)	2021-12-24 13:59:04 +01:00
chrislu	255a1c7dcd	refactor type names	2021-12-23 18:23:18 -08:00
chrislu	f77ca41769	refactor	2021-12-23 17:48:34 -08:00
chrislu	1d36884845	rename files	2021-12-23 17:47:58 -08:00
chrislu	2d1a1f5e03	rename variables and functions	2021-12-23 17:35:57 -08:00
chrislu	7bf48ee135	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-23 17:23:26 -08:00
chrislu	6de331b014	clean up	2021-12-23 17:23:21 -08:00
chrislu	032df784ed	chunked file works now	2021-12-23 17:17:32 -08:00
banjiaojuhao	083bf3a137	filer server: add "datacenter, rack and datanode" for path specific configuration	2021-12-23 23:25:05 +08:00
chrislu	c2aad1c7ff	detect non streaming mode on first write request	2021-12-22 17:20:44 -08:00
chrislu	b541e39a2c	fix tests	2021-12-22 16:17:30 -08:00
chrislu	2bc6fa90ff	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-22 16:05:43 -08:00
chrislu	0ec7bc6710	detect non streaming mode on the first read	2021-12-22 16:05:38 -08:00
chrislu	4c1368d621	fix test	2021-12-22 16:05:08 -08:00
Chris Lu	4e73705533	Merge pull request #2530 from banjiaojuhao/filer-upload-file-to-node ... filer server: allow upload file to specific dataNode	2021-12-22 12:49:15 -08:00
banjiaojuhao	08336be92e	filer server: allow upload file to specific dataNode	2021-12-22 21:57:26 +08:00
chrislu	7b78fc72b0	add page chunk interval list	2021-12-22 02:53:33 -08:00
chrislu	b7cd52636b	Merge branch 'master' of https://github.com/chrislusf/seaweedfs	2021-12-21 22:24:43 -08:00
chrislu	3981d65b68	remove println	2021-12-21 22:24:38 -08:00
chrislu	9a73319b45	mount: different write strategy for streaming write and random write	2021-12-21 17:28:55 -08:00
Chris Lu	b0665a15f4	Merge pull request #2527 from banjiaojuhao/master-assign-by-datanode	2021-12-21 08:56:51 -08:00
banjiaojuhao	dda6b90d25	assign fileId according to DataNode with empty DataCenter and Rack	2021-12-21 17:28:33 +08:00
chrislu	4b8dcff448	reverting default admin scripts ... fix https://github.com/chrislusf/seaweedfs/issues/2525 this new default value was introduced in 2.80 this affects production environments, e.g., EC is not desired, volume balancing is not preferred, etc.	2021-12-20 13:34:57 -08:00
chrislu	b21a67bbe6	add writer pattern object for later use	2021-12-20 11:53:48 -08:00
chrislu	4fd29dad86	remove writeOnly flag	2021-12-20 01:11:43 -08:00
chrislu	bc96682760	refactor, change file locations	2021-12-20 01:02:23 -08:00
chrislu	866c2657f0	avoid FUSE cache only for the first 512 bytes	2021-12-19 23:13:36 -08:00
chrislu	0cb9036f66	mount: only cache the first chunk on stream read	2021-12-19 23:06:03 -08:00
chrislu	a152f17937	mount: improve read performance on random reads	2021-12-19 22:43:14 -08:00
chrislu	85c526c583	s3: bind to a specific IP ... fix https://github.com/chrislusf/seaweedfs/issues/2516	2021-12-17 11:34:37 -08:00
chrislu	5eacff9d4f	log message adds server name ... address https://github.com/chrislusf/seaweedfs/issues/2514#issuecomment-995925733	2021-12-16 10:46:26 -08:00
chrislu	50ddd8c8e2	remove debug messages ... fix https://github.com/chrislusf/seaweedfs/issues/2514	2021-12-16 00:58:15 -08:00
chrislu	7210558c7b	s3: pass through s3 presigned headers ... fix https://github.com/chrislusf/seaweedfs/discussions/2502	2021-12-15 13:18:53 -08:00
chrislu	bf4d7affc0	gateway to remote object store: skip replicating multipart upload part files ... fix https://github.com/chrislusf/seaweedfs/issues/2509	2021-12-14 19:48:31 -08:00
chrislu	316f326464	add more help message	2021-12-13 13:14:36 -08:00
Konstantin Lebedev	969f513265	disable audit log for s3 statusHandler	2021-12-13 16:08:19 +05:00
Chris Lu	ab0c6a84bf	Merge pull request #2503 from kmlebedev/audit_log_nonblocking ... Audit log force async	2021-12-13 00:47:16 -08:00
Konstantin Lebedev	34779e8f38	force enable asynchronous I/O sending events to Fluentd	2021-12-13 13:39:39 +05:00
chrislu	c3b73ec23b	2.82	2021-12-12 23:25:24 -08:00
chrislu	7bf14452f7	s3: handle Response-Content-Disposition	2021-12-12 23:05:40 -08:00
chrislu	94e5c0a454	skip purging from ec volumes	2021-12-12 22:55:27 -08:00
chrislu	a2d3f89c7b	add lock messages	2021-12-10 13:24:38 -08:00
chrislu	991a3dca0d	rename file	2021-12-10 13:11:00 -08:00
kmlebedev	4f98553ba9	audit log SignatureVersion	2021-12-10 19:40:32 +05:00
chrislu	2d6fcdf83a	add delay if need to resume from disk	2021-12-09 13:13:05 -08:00
liubaojiang	c7abc34dbb	rename operation returns the correct old parent path	2021-12-10 00:02:57 +08:00
Konstantin Lebedev	98251fe16a	non blocking audit log	2021-12-09 19:47:16 +05:00
chrislu	40ee9b4d7b	sleep between the gap of each execution ... related to https://github.com/chrislusf/seaweedfs/issues/2476	2021-12-07 11:44:08 -08:00
Chris Lu	2ba08afed1	Merge pull request #2498 from kmlebedev/s3_audit_log	2021-12-07 09:35:48 -08:00
Konstantin Lebedev	10678cde81	audit log config	2021-12-07 18:20:52 +05:00
Konstantin Lebedev	4ec8715f20	audit log	2021-12-07 12:15:48 +05:00
Konstantin Lebedev	3ac48cd540	audit log	2021-12-07 12:13:21 +05:00
chrislu	7f0a97c7b6	shell: volume.list supports different verbosity level	2021-12-05 21:54:40 -08:00
chrislu	5ea9715721	2.81 ... also sync java client version to SeaweedFS version	2021-12-05 18:05:24 -08:00
chrislu	e6c026db65	volume.fix.replication: fix misplaced volumes ... fix https://github.com/chrislusf/seaweedfs/issues/2416	2021-12-05 16:56:25 -08:00
chrislu	53e2dee177	[volume.check.disk] was using the wrong source and target locations ... fix https://github.com/chrislusf/seaweedfs/issues/2268	2021-12-05 14:32:04 -08:00
chrislu	c146c76d10	avoid creating the same bucket with a different randomized name ... related to https://github.com/chrislusf/seaweedfs/issues/2492	2021-12-05 13:06:41 -08:00
chrislu	42d97a3442	adjust randomized bucket name	2021-12-05 12:36:58 -08:00
chrislu	488afa5002	volume: load volume can optionally be skipped, if ec volume exists ... fix https://github.com/chrislusf/seaweedfs/issues/2489	2021-12-05 02:28:52 -08:00
chrislu	59e58c4b23	volume: fix loading old volume format ... fix https://github.com/chrislusf/seaweedfs/issues/2487	2021-12-05 01:06:01 -08:00
chrislu	b70cb3e0b2	upgrade protoc to 3.17.3 ... $brew install protobuf $ protoc --version libprotoc 3.17.3 $ go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.26 $ go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.1	2021-12-05 00:42:25 -08:00
chrislu	5c3b783310	s3: copy object API needs to escape special characters ... fix https://github.com/chrislusf/seaweedfs/issues/2482	2021-12-04 23:24:53 -08:00
Bl1tz23	83bffca7e6	fix MongodbStore.ListDirectoryEntries panic on find failure	2021-12-03 13:58:37 +03:00
Tanmoy Majumdar	ea09fb477a	return ' shouldRetry=true' so that filer can retry the failed chunk	2021-12-03 11:54:20 +06:00
Chris Lu	689f5513a9	redis3 supports sentinel	2021-11-29 01:09:51 -08:00
Chris Lu	7227cfddf5	2.80	2021-11-29 00:57:08 -08:00
Chris Lu	3d7390302d	add s3.clean.uploads -timeAgo=24h	2021-11-29 00:49:49 -08:00
Chris Lu	2f72c24498	skip the rest logic	2021-11-29 00:32:21 -08:00
Chris Lu	88ff8fc27b	ensure uploaded chunks are deleted on error	2021-11-29 00:28:26 -08:00
Chris Lu	bea26549f2	avoid empty extended attribute value	2021-11-28 23:50:52 -08:00
Chris Lu	ce2af0082e	revert	2021-11-28 23:35:22 -08:00
Chris Lu	1c9f3c7ac0	read deleted chunks when replcating data	2021-11-28 23:34:34 -08:00
Chris Lu	96c66ca2aa	read deleted chunks when replicating data	2021-11-28 23:33:03 -08:00
Chris Lu	f3a334965d	Update command_remote_uncache.go ... minor	2021-11-28 23:10:32 -08:00
Chris Lu	9ccfc1cfee	Update command_volume_fsck.go ... minor	2021-11-28 22:39:24 -08:00
Chris Lu	71a94267c9	Update command_volume_fsck.go ... remove unused code	2021-11-28 22:37:11 -08:00
Chris Lu	ad16221a35	adjust error log	2021-11-28 22:06:17 -08:00
Chris Lu	cf1586a34d	add logs for writing to remote file	2021-11-27 22:09:23 -08:00