mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-21 15:28:14 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,889 Commits 32 Branches 296 Tags 152 MiB
915f9f5054
Commit Graph

7021 Commits

Author SHA1 Message Date
chrislu
e14b6e02df load ec volume id locations 
fix https://github.com/seaweedfs/seaweedfs/issues/5713
 2024-08-12 11:54:25 -07:00
chrislu
480fcb1d03 fix build 2024-08-12 09:45:51 -07:00
chrislu
ff66155302 fix tests 2024-08-12 09:30:42 -07:00
chrislu
270e91b0be adds locking 2024-08-11 13:06:01 -07:00
chrislu
b6fd1ff4ce refactor 2024-08-10 12:39:16 -07:00
chrislu
7438648d1c Merge branch 'master' into mq 2024-08-10 10:01:57 -07:00
Kamran Sarwar
63c707f9c1
added access-control-expose-headers to put request (#5876) 2024-08-09 15:52:05 -07:00
chrislu
e825dd7496 fix opposite logic 
fix https://github.com/seaweedfs/seaweedfs/issues/5871
 2024-08-08 23:16:38 -07:00
chrislu
e069009f09 [remote.cache, remote.uncache] fix min/max age 
fix https://github.com/seaweedfs/seaweedfs/issues/5871
 2024-08-08 21:25:52 -07:00
chrislu
9ca30e52d5 fuse mount handles ttl entries 
fix https://github.com/seaweedfs/seaweedfs/issues/5527
 2024-08-08 08:18:54 -07:00
chrislu
2074e7db67 fix invalid toml 2024-08-08 00:03:42 -07:00
chrislu
e86b0bcaaa simplify 2024-08-07 10:46:29 -07:00
chrislu
57dc39c451 randomizing next file handle id 2024-08-07 10:42:00 -07:00
Konstantin Lebedev
3e6ca6e706
[webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
Konstantin Lebedev
ef4c555e90
[shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
qinguoyi
bf2366ba3a
Fix KeyMaterial nil bug (#5849) 2024-08-02 11:06:48 -07:00
qinguoyi
e88da0be1b
Fix KeyMaterial nil bug (#5848) 2024-08-02 09:47:11 -07:00
qinguoyi
606d0cba64
Fix extractToFile gz file nil bug (#5847) 2024-08-02 09:46:37 -07:00
qinguoyi
b74b506e52
add command backup destory volume error log (#5846) 2024-08-02 00:04:57 -07:00
chrislu
8378a5b70b rename 2024-08-01 23:54:42 -07:00
chrislu
d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
zuzuviewer
5ee50d26d8
fix CVE-2016-2183 (#5844) 
* fix CVE-2016-2183

* allow tls.min_version and tls.max_version not set
 2024-08-01 20:00:22 -07:00
chrislu
4fee496b49 conditional delete 2024-08-01 13:00:54 -07:00
wyang
31b89c1062
fsck: only check the appendNs of deleted needle (#5841) 
increase fsck speed

Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-31 01:12:57 -07:00
Konstantin Lebedev
b2ffcdaab2
[master] do sync grow request only if absolutely necessary (#5821) 
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
 2024-07-30 13:21:35 -07:00
Jiffs Maverick
c8506609d8
[filer] Added a version to the title of Filer UI (#5836) 
Added a version to the title of Filer UI
 2024-07-29 14:07:12 -07:00
Konstantin Lebedev
e1f9336688
[filer] fix panic filer on webdav.maxMB (#5822) 
fix panic filer on webdav.maxMB
 2024-07-27 13:40:22 -07:00
wyang
4b1f539ab8
fix allocate reduplicated volumeId to different volume (#5811) 
* fix allocate reduplicated volumeId to different volume

* only check barrier when read

---------

Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-26 21:48:36 -07:00
wyang
c1bffca246
fix evacuate volume to different disk types (#5530) 
Co-authored-by: wyang <wyang@wyangs-Air.lan>
 2024-07-26 21:36:56 -07:00
chrislu
e2a07d11d5 Revert "Check ShouldGrowVolumes before returning error in assign. (#5819)" 
This reverts commit 98d66338d0333cd955f7840c64ef95e3c4807a17.
 2024-07-26 11:21:50 -07:00
wyang
0581ce6096
fix delete chunk failed if volumeSever specified grpc.port (#5820) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-26 11:14:19 -07:00
chrislu
ec9455dab7 same change as #5819 
follow up with https://github.com/seaweedfs/seaweedfs/pull/5819
 2024-07-26 11:06:47 -07:00
Ruoxi
d15966ae8e
Check ShouldGrowVolumes before returning error in assign. (#5819) 2024-07-26 11:04:38 -07:00
Chris Lu
952afd810c
Fix dead lock (#5815) 
* reduce locks to avoid dead lock

Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0

goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
	/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
	/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
	/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
	/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
	/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
	/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)

SaveContent -> MemChunk.RLock ->
	ChunkedDirtyPages.saveChunkedFileIntervalToStorage
	pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
		fh.entryLock.Lock()

sync.(*RWMutex).Lock(0x0?)
	/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
	/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
	/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
	/github/workspace/weed/util/limited_executor.go:33 +0x97

On metadata update
fh.entryLock.Lock()
	fh.dirtyPages.Destroy()
		up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock

goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
	/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
	/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
	/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
	/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
	/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
	/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570

* use locked entry everywhere

* modifiable remote entry

* skip locking after getting lock from fhLockTable
 2024-07-24 23:46:40 -07:00
chrislu
ed7e721efe 3.71 2024-07-21 21:03:27 -07:00
chrislu
9265be43c0 avoid nil 2024-07-21 21:01:29 -07:00
chrislu
4020149690 3.70 2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1 add version to filer 2024-07-21 20:13:51 -07:00
chrislu
9dcc576499 Revert "add collection for buckets" 
This reverts commit 96af571219.
 2024-07-20 12:00:50 -07:00
chrislu
aed2a18223 get, set and add path conf 2024-07-20 11:27:11 -07:00
chrislu
96af571219 add collection for buckets 2024-07-17 23:14:36 -07:00
chrislu
a8fa78b892 refactoring 2024-07-17 23:14:36 -07:00
chrislu
7a75928e7d Azure sink: avoid overwriting existing files 2024-07-17 23:14:36 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766) 
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
 2024-07-16 23:14:09 -07:00
chrislu
c6dec11ea5 [filer.sync] skip overwriting existing fresh entry 2024-07-16 09:38:10 -07:00
chrislu
19d14b4c71 minor 2024-07-16 09:21:01 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
Konstantin Lebedev
67edf1d014
[master] Do Automatic Volume Grow in background (#5781) 
* Do Automatic Volume Grow in backgound

* pass lastGrowCount to master

* fix build

* fix count to uint64
 2024-07-16 08:03:40 -07:00
wusong
ce61a66b65
Fix mmap write fail (#5791) 
* [mount] fix GetAttr blocks count

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

* [mount] fix mmap read

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

---------

Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2024-07-16 07:59:55 -07:00
Dan
794a5aa67d
Add warning with hints to why a EC shard cant be moved (#5790) 2024-07-15 11:17:22 -07:00