Bruce
f9e141a412
persist readonly state to volume info ( #5977 )
2024-09-05 07:58:24 -07:00
Bruce
310d41998d
Skip volume data integrity check for volume which has remote file ( #5976 )
2024-09-04 21:25:04 -07:00
Konstantin Lebedev
16984ef07a
[s3] reducing the number of metrics for 403 ( #5961 )
...
reducing the number of metrics
2024-09-04 11:06:54 -07:00
Konstantin Lebedev
67a252ee8a
[master] refactor func ShouldGrowVolumes ( #5884 )
2024-09-04 08:16:44 -07:00
chrislu
eb02946c97
support write once read many
...
fix https://github.com/seaweedfs/seaweedfs/issues/5954
2024-09-04 02:25:07 -07:00
chrislu
18afdb15b6
Revert "weed mount, weed dav add option to force cache"
...
This reverts commit 7367b976b0
.
2024-09-04 01:38:29 -07:00
chrislu
7367b976b0
weed mount, weed dav add option to force cache
2024-09-04 01:19:14 -07:00
chrislu
66ac82bb8f
default cacheDirWrite to cacheDir
2024-09-04 00:05:58 -07:00
chrislu
8d5a6d7e74
fix for spreading ec shards
2024-09-03 22:30:11 -07:00
chrislu
8e4bffc66b
copy ec shards to disks already having ec volumes
...
fix https://github.com/seaweedfs/seaweedfs/issues/5615
2024-09-03 21:19:10 -07:00
wyang
361e7ebc74
fix volume.move io.EOF ( #5960 )
2024-09-02 06:17:41 -07:00
Eugeniy E. Mikhailov
282867a71e
Show volume server which failed to verify a chunk ( #5958 )
2024-09-01 21:13:22 -07:00
Eugeniy E. Mikhailov
220061573d
update metric for ErrorCRC if it happens during volume readNeedleDataInto ( #5957 )
2024-09-01 20:30:09 -07:00
chrislu
3c0854e986
unnecessary skipping
2024-08-30 16:38:01 -07:00
chrislu
654b8210f7
parameter name
2024-08-30 14:52:07 -07:00
chrislu
8679870008
fix typo
2024-08-30 14:51:55 -07:00
chrislu
22d5d2ea96
adjust warning log
2024-08-30 14:51:42 -07:00
chrislu
a4b25a642d
math/rand => math/rand/v2
2024-08-29 09:52:21 -07:00
chrislu
ded5e084ea
ensure none zero lastGrowCount
2024-08-27 09:03:11 -07:00
chrislu
0cf2c15828
rename
2024-08-27 09:02:48 -07:00
chrislu
bebc385607
randomly pick one replica to write
2024-08-27 09:02:39 -07:00
chrislu
3b7bb62e38
logs on error
2024-08-26 09:09:11 -07:00
Shaun Alexander
0d435f9da4
filer.remote.gateway not filer.remote.sync for filer.remote.gateway help string. ( #5935 )
2024-08-23 20:42:59 -07:00
Shaun Alexander
766290411b
filer.remote.gateway not filer.remote.sync for remote.mount.buckets help string. ( #5934 )
2024-08-23 20:41:33 -07:00
chrislu
4463296811
add parallel vacuuming
2024-08-21 22:53:54 -07:00
chrislu
5535b60e8c
fix replication range check
...
fix for https://github.com/seaweedfs/seaweedfs/wiki/Replication#replication-string
2024-08-21 16:58:43 -07:00
zemul
95bae91ca7
Fix get dir metadata ( #5923 )
...
* fix:mount deadlock
* feat: query dir metadata
* feat: query dir metadata
---------
Co-authored-by: zemul <zhouzemiao@ihuman.com>
2024-08-21 11:01:18 -07:00
chrislu
6ce27affa4
also use /healthz
for most consistent health check
2024-08-21 10:55:39 -07:00
chrislu
b3696024d1
add warning for not enough copies when skipping vacuuming volumes
...
fix https://github.com/seaweedfs/seaweedfs/issues/5906
2024-08-20 09:39:35 -07:00
Riccardo Bertossa
6fe8639504
add http endpoint to get the size of a collection ( #5910 )
2024-08-19 07:44:45 -07:00
chrislu
e50d85c0f3
3.72
2024-08-18 21:51:53 -07:00
Oleg Salionov
4cfc7d3b10
Dont try lazy decode content in proxyToFiler if no accept-encoding provided ( #5907 )
2024-08-18 21:51:17 -07:00
chrislu
e8462ba3ad
prevent compacting on the same volume
2024-08-18 12:08:43 -07:00
augustazz
db833abfa2
fix ec volume lookup data sync ( #5900 )
2024-08-16 06:08:33 -07:00
augustazz
0b00706454
EC volume supports expiration and displays expiration message when executing volume.list ( #5895 )
...
* ec volume expire
* volume.list show DestroyTime
* comments
* code optimization
---------
Co-authored-by: xuwenfeng <xuwenfeng1@zto.com>
2024-08-16 00:20:00 -07:00
chrislu
8f1f1730e9
minor
2024-08-15 19:12:11 -07:00
wusong
6f58ab7e8b
[master] fix master panic ( #5893 )
2024-08-15 06:20:51 -07:00
chrislu
fdf7193ae7
rename
2024-08-13 13:59:24 -07:00
chrislu
07f4998188
add dat file size into vif for EC
2024-08-13 13:56:00 -07:00
chrislu
3a2e21fee7
fix EC reading on nLargeBlockRows
...
fix https://github.com/seaweedfs/seaweedfs/issues/5465
2024-08-13 12:01:46 -07:00
chrislu
e14b6e02df
load ec volume id locations
...
fix https://github.com/seaweedfs/seaweedfs/issues/5713
2024-08-12 11:54:25 -07:00
chrislu
480fcb1d03
fix build
2024-08-12 09:45:51 -07:00
chrislu
ff66155302
fix tests
2024-08-12 09:30:42 -07:00
chrislu
270e91b0be
adds locking
2024-08-11 13:06:01 -07:00
chrislu
b6fd1ff4ce
refactor
2024-08-10 12:39:16 -07:00
chrislu
7438648d1c
Merge branch 'master' into mq
2024-08-10 10:01:57 -07:00
Kamran Sarwar
63c707f9c1
added access-control-expose-headers to put request ( #5876 )
2024-08-09 15:52:05 -07:00
chrislu
e825dd7496
fix opposite logic
...
fix https://github.com/seaweedfs/seaweedfs/issues/5871
2024-08-08 23:16:38 -07:00
chrislu
e069009f09
[remote.cache, remote.uncache] fix min/max age
...
fix https://github.com/seaweedfs/seaweedfs/issues/5871
2024-08-08 21:25:52 -07:00
chrislu
9ca30e52d5
fuse mount handles ttl entries
...
fix https://github.com/seaweedfs/seaweedfs/issues/5527
2024-08-08 08:18:54 -07:00
chrislu
2074e7db67
fix invalid toml
2024-08-08 00:03:42 -07:00
chrislu
e86b0bcaaa
simplify
2024-08-07 10:46:29 -07:00
chrislu
57dc39c451
randomizing next file handle id
2024-08-07 10:42:00 -07:00
Konstantin Lebedev
3e6ca6e706
[webdav] status code 500 if internal error from filer ( #5865 )
2024-08-06 07:39:52 -07:00
Konstantin Lebedev
ef4c555e90
[shell] volume.grow ( #5855 )
2024-08-05 09:22:17 -07:00
qinguoyi
bf2366ba3a
Fix KeyMaterial nil bug ( #5849 )
2024-08-02 11:06:48 -07:00
qinguoyi
e88da0be1b
Fix KeyMaterial nil bug ( #5848 )
2024-08-02 09:47:11 -07:00
qinguoyi
606d0cba64
Fix extractToFile gz file nil bug ( #5847 )
2024-08-02 09:46:37 -07:00
qinguoyi
b74b506e52
add command backup destory volume error log ( #5846 )
2024-08-02 00:04:57 -07:00
chrislu
8378a5b70b
rename
2024-08-01 23:54:42 -07:00
chrislu
d49ecde535
rename functions
2024-08-01 23:27:23 -07:00
zuzuviewer
5ee50d26d8
fix CVE-2016-2183 ( #5844 )
...
* fix CVE-2016-2183
* allow tls.min_version and tls.max_version not set
2024-08-01 20:00:22 -07:00
chrislu
4fee496b49
conditional delete
2024-08-01 13:00:54 -07:00
wyang
31b89c1062
fsck: only check the appendNs of deleted needle ( #5841 )
...
increase fsck speed
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-31 01:12:57 -07:00
Konstantin Lebedev
b2ffcdaab2
[master] do sync grow request only if absolutely necessary ( #5821 )
...
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819
* remove check VolumeGrowStrategy Threshold on PickForWrite
* fix fmt.Errorf
2024-07-30 13:21:35 -07:00
Jiffs Maverick
c8506609d8
[filer] Added a version to the title of Filer UI ( #5836 )
...
Added a version to the title of Filer UI
2024-07-29 14:07:12 -07:00
Konstantin Lebedev
e1f9336688
[filer] fix panic filer on webdav.maxMB ( #5822 )
...
fix panic filer on webdav.maxMB
2024-07-27 13:40:22 -07:00
wyang
4b1f539ab8
fix allocate reduplicated volumeId to different volume ( #5811 )
...
* fix allocate reduplicated volumeId to different volume
* only check barrier when read
---------
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-26 21:48:36 -07:00
wyang
c1bffca246
fix evacuate volume to different disk types ( #5530 )
...
Co-authored-by: wyang <wyang@wyangs-Air.lan>
2024-07-26 21:36:56 -07:00
chrislu
e2a07d11d5
Revert "Check ShouldGrowVolumes
before returning error in assign. ( #5819 )"
...
This reverts commit 98d66338d0333cd955f7840c64ef95e3c4807a17.
2024-07-26 11:21:50 -07:00
wyang
0581ce6096
fix delete chunk failed if volumeSever specified grpc.port ( #5820 )
...
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-26 11:14:19 -07:00
chrislu
ec9455dab7
same change as #5819
...
follow up with https://github.com/seaweedfs/seaweedfs/pull/5819
2024-07-26 11:06:47 -07:00
Ruoxi
d15966ae8e
Check ShouldGrowVolumes
before returning error in assign. ( #5819 )
2024-07-26 11:04:38 -07:00
Chris Lu
952afd810c
Fix dead lock ( #5815 )
...
* reduce locks to avoid dead lock
Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0
goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)
SaveContent -> MemChunk.RLock ->
ChunkedDirtyPages.saveChunkedFileIntervalToStorage
pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
fh.entryLock.Lock()
sync.(*RWMutex).Lock(0x0?)
/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
/github/workspace/weed/util/limited_executor.go:33 +0x97
On metadata update
fh.entryLock.Lock()
fh.dirtyPages.Destroy()
up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock
goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570
* use locked entry everywhere
* modifiable remote entry
* skip locking after getting lock from fhLockTable
2024-07-24 23:46:40 -07:00
chrislu
ed7e721efe
3.71
2024-07-21 21:03:27 -07:00
chrislu
9265be43c0
avoid nil
2024-07-21 21:01:29 -07:00
chrislu
4020149690
3.70
2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1
add version to filer
2024-07-21 20:13:51 -07:00
chrislu
9dcc576499
Revert "add collection for buckets"
...
This reverts commit 96af571219
.
2024-07-20 12:00:50 -07:00
chrislu
aed2a18223
get, set and add path conf
2024-07-20 11:27:11 -07:00
chrislu
96af571219
add collection for buckets
2024-07-17 23:14:36 -07:00
chrislu
a8fa78b892
refactoring
2024-07-17 23:14:36 -07:00
chrislu
7a75928e7d
Azure sink: avoid overwriting existing files
2024-07-17 23:14:36 -07:00
vadimartynov
86d92a42b4
Added tls for http clients ( #5766 )
...
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
c6dec11ea5
[filer.sync] skip overwriting existing fresh entry
2024-07-16 09:38:10 -07:00
chrislu
19d14b4c71
minor
2024-07-16 09:21:01 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce ( #5792 )
2024-07-16 09:15:55 -07:00
Konstantin Lebedev
67edf1d014
[master] Do Automatic Volume Grow in background ( #5781 )
...
* Do Automatic Volume Grow in backgound
* pass lastGrowCount to master
* fix build
* fix count to uint64
2024-07-16 08:03:40 -07:00
wusong
ce61a66b65
Fix mmap write fail ( #5791 )
...
* [mount] fix GetAttr blocks count
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
* [mount] fix mmap read
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
---------
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2024-07-16 07:59:55 -07:00
Dan
794a5aa67d
Add warning with hints to why a EC shard cant be moved ( #5790 )
2024-07-15 11:17:22 -07:00
Konstantin Lebedev
a53e406c99
[master] refactor HasGrowRequest to atomic bool ( #5782 )
...
refactor HasGrowRequest to atomit bool
2024-07-15 10:51:21 -07:00
Konstantin Lebedev
33964fa292
metrics stats of volume layout depends on the data center ( #5775 )
...
stats volume layout depends on the data center
2024-07-12 12:32:25 -07:00
Konstantin Lebedev
04f4b10884
fix: avoid timeout if datacenter does not exist in topology ( #5772 )
...
* fix: avoid timeout if datacenter does not exist in topology
* fix: error msg
* fix: rm dublicate check
* fix: compare
* revert minor change
2024-07-12 11:19:08 -07:00
chrislu
3a82f5ffad
ensure metadata follow a specific folder
...
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
vadimartynov
ec9e7493b3
-metricsIp cmd flag ( #5773 )
...
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;
* Fixed cmd flags
2024-07-12 10:56:26 -07:00
小羽
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time ( #5771 )
2024-07-12 10:49:15 -07:00
小羽
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat ( #5770 )
2024-07-12 08:09:51 -07:00
chrislu
4e7d8eb3f1
small refactoring
2024-07-11 14:50:11 -07:00
vadimartynov
de60f383de
Added tls for http handlers ( #5764 )
...
* Added https handler for filer
* Added example for security.toml
2024-07-11 07:53:18 -07:00
chrislu
2addcd1623
fix test
2024-07-11 07:53:02 -07:00