Commit Graph

10875 Commits

Author SHA1 Message Date
chrislu
7438648d1c Merge branch 'master' into mq 2024-08-10 10:01:57 -07:00
Kamran Sarwar
63c707f9c1
added access-control-expose-headers to put request (#5876) 2024-08-09 15:52:05 -07:00
chrislu
e825dd7496 fix opposite logic
fix https://github.com/seaweedfs/seaweedfs/issues/5871
2024-08-08 23:16:38 -07:00
chrislu
e069009f09 [remote.cache, remote.uncache] fix min/max age
fix https://github.com/seaweedfs/seaweedfs/issues/5871
2024-08-08 21:25:52 -07:00
chrislu
9ca30e52d5 fuse mount handles ttl entries
fix https://github.com/seaweedfs/seaweedfs/issues/5527
2024-08-08 08:18:54 -07:00
chrislu
2074e7db67 fix invalid toml 2024-08-08 00:03:42 -07:00
Andrei Kvapil
23809a1e84
Add volume-resize-hook (#5868) 2024-08-07 11:23:30 -07:00
chrislu
e86b0bcaaa simplify 2024-08-07 10:46:29 -07:00
chrislu
57dc39c451 randomizing next file handle id 2024-08-07 10:42:00 -07:00
Konstantin Lebedev
3e6ca6e706
[webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
dependabot[bot]
8a1052cc5e
chore(deps): bump docker/setup-buildx-action from 3.6.0 to 3.6.1 (#5856)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.6.0 to 3.6.1.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](3d68780484...988b5a0280)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:12:01 -07:00
Andrei Kvapil
1256b3e917
Add: COSI BucketClass and BucketAccessClass (#5863)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-05 11:11:34 -07:00
Andrei Kvapil
38befb133d
Fix: Resolve conflicts for deployment in diferent namespaces (#5862)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-05 11:11:07 -07:00
dependabot[bot]
af85180a44
chore(deps): bump go.etcd.io/etcd/client/v3 from 3.5.13 to 3.5.15 (#5861)
Bumps [go.etcd.io/etcd/client/v3](https://github.com/etcd-io/etcd) from 3.5.13 to 3.5.15.
- [Release notes](https://github.com/etcd-io/etcd/releases)
- [Commits](https://github.com/etcd-io/etcd/compare/v3.5.13...v3.5.15)

---
updated-dependencies:
- dependency-name: go.etcd.io/etcd/client/v3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:10:09 -07:00
dependabot[bot]
f67c9b0ad7
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.4.2 to 0.5.0 (#5860)
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ

Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.4.2 to 0.5.0.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.4.2...v0.5.0)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:09:58 -07:00
dependabot[bot]
809f42e830
chore(deps): bump github.com/aws/aws-sdk-go from 1.54.13 to 1.55.5 (#5859)
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.54.13 to 1.55.5.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.54.13...v1.55.5)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:09:42 -07:00
dependabot[bot]
2d2d05005c
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.18 to 1.17.27 (#5857)
chore(deps): bump github.com/aws/aws-sdk-go-v2/credentials

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.18 to 1.17.27.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.18...credentials/v1.17.27)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:09:27 -07:00
dependabot[bot]
84be9442d7
chore(deps): bump golang.org/x/sync from 0.7.0 to 0.8.0 (#5858)
Bumps [golang.org/x/sync](https://github.com/golang/sync) from 0.7.0 to 0.8.0.
- [Commits](https://github.com/golang/sync/compare/v0.7.0...v0.8.0)

---
updated-dependencies:
- dependency-name: golang.org/x/sync
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-08-05 11:09:08 -07:00
Konstantin Lebedev
ef4c555e90
[shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
Andrei Kvapil
bd1b8119fe
Fix secretName for seaweedfs-cosi-driver (#5850) 2024-08-02 11:07:38 -07:00
qinguoyi
bf2366ba3a
Fix KeyMaterial nil bug (#5849) 2024-08-02 11:06:48 -07:00
qinguoyi
e88da0be1b
Fix KeyMaterial nil bug (#5848) 2024-08-02 09:47:11 -07:00
qinguoyi
606d0cba64
Fix extractToFile gz file nil bug (#5847) 2024-08-02 09:46:37 -07:00
qinguoyi
b74b506e52
add command backup destory volume error log (#5846) 2024-08-02 00:04:57 -07:00
chrislu
8378a5b70b rename 2024-08-01 23:54:42 -07:00
chrislu
d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
zuzuviewer
5ee50d26d8
fix CVE-2016-2183 (#5844)
* fix CVE-2016-2183

* allow tls.min_version and tls.max_version not set
2024-08-01 20:00:22 -07:00
chrislu
4fee496b49 conditional delete 2024-08-01 13:00:54 -07:00
Andrei Kvapil
a7e00d9074
Update COSI driver v0.1.1 (#5845)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-08-01 09:47:41 -07:00
Andrei Kvapil
3b0130b070
Add seaweedfs-cosi-driver (#5843)
add: seaweedfs-cosi-driver

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-07-31 13:53:30 -07:00
wyang
31b89c1062
fsck: only check the appendNs of deleted needle (#5841)
increase fsck speed

Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-31 01:12:57 -07:00
Konstantin Lebedev
b2ffcdaab2
[master] do sync grow request only if absolutely necessary (#5821)
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
2024-07-30 13:21:35 -07:00
dependabot[bot]
69bcdf470b
chore(deps): bump gocloud.dev/pubsub/rabbitpubsub from 0.37.0 to 0.38.0 (#5829)
Bumps [gocloud.dev/pubsub/rabbitpubsub](https://github.com/google/go-cloud) from 0.37.0 to 0.38.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.37.0...v0.38.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/rabbitpubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-07-30 10:49:01 -07:00
dependabot[bot]
41ddbe329e
chore(deps): bump github.com/rclone/rclone from 1.66.0 to 1.67.0 (#5831)
Bumps [github.com/rclone/rclone](https://github.com/rclone/rclone) from 1.66.0 to 1.67.0.
- [Release notes](https://github.com/rclone/rclone/releases)
- [Changelog](https://github.com/rclone/rclone/blob/master/RELEASE.md)
- [Commits](https://github.com/rclone/rclone/compare/v1.66.0...v1.67.0)

---
updated-dependencies:
- dependency-name: github.com/rclone/rclone
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2024-07-30 10:22:11 -07:00
Jiffs Maverick
c8506609d8
[filer] Added a version to the title of Filer UI (#5836)
Added a version to the title of Filer UI
2024-07-29 14:07:12 -07:00
Andrei Kvapil
c9e526665b
fix case for keyAlgorithm in Helm chart (#5835)
fix keyAlgorithm in Helm chart

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-07-29 12:17:43 -07:00
dependabot[bot]
ddd2ec21d0
chore(deps): bump docker/setup-buildx-action from 3.5.0 to 3.6.0 (#5825)
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.5.0 to 3.6.0.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](aa33708b10...3d68780484)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-29 09:14:15 -07:00
Andrei Kvapil
545c3f83bf
Correct gopher on SVG logo (#5833) 2024-07-29 09:13:41 -07:00
dependabot[bot]
b9063404ba
chore(deps): bump github.com/redis/go-redis/v9 from 9.5.1 to 9.6.1 (#5832)
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis) from 9.5.1 to 9.6.1.
- [Release notes](https://github.com/redis/go-redis/releases)
- [Changelog](https://github.com/redis/go-redis/blob/master/CHANGELOG.md)
- [Commits](https://github.com/redis/go-redis/compare/v9.5.1...v9.6.1)

---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-29 09:12:30 -07:00
dependabot[bot]
cb03496c15
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3 from 3.68.1 to 3.76.0 (#5830)
chore(deps): bump github.com/ydb-platform/ydb-go-sdk/v3

Bumps [github.com/ydb-platform/ydb-go-sdk/v3](https://github.com/ydb-platform/ydb-go-sdk) from 3.68.1 to 3.76.0.
- [Release notes](https://github.com/ydb-platform/ydb-go-sdk/releases)
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk/compare/v3.68.1...v3.76.0)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk/v3
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-29 09:12:17 -07:00
dependabot[bot]
f833c2b2fa
chore(deps): bump gocloud.dev/pubsub/natspubsub from 0.37.0 to 0.38.0 (#5828)
Bumps [gocloud.dev/pubsub/natspubsub](https://github.com/google/go-cloud) from 0.37.0 to 0.38.0.
- [Release notes](https://github.com/google/go-cloud/releases)
- [Commits](https://github.com/google/go-cloud/compare/v0.37.0...v0.38.0)

---
updated-dependencies:
- dependency-name: gocloud.dev/pubsub/natspubsub
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-07-29 09:12:03 -07:00
Andrei Kvapil
acccae831f
Add SVG logo (#5827)
add SVG logo

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2024-07-29 08:55:59 -07:00
Konstantin Lebedev
e1f9336688
[filer] fix panic filer on webdav.maxMB (#5822)
fix panic filer on webdav.maxMB
2024-07-27 13:40:22 -07:00
wyang
4b1f539ab8
fix allocate reduplicated volumeId to different volume (#5811)
* fix allocate reduplicated volumeId to different volume

* only check barrier when read

---------

Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-26 21:48:36 -07:00
wyang
c1bffca246
fix evacuate volume to different disk types (#5530)
Co-authored-by: wyang <wyang@wyangs-Air.lan>
2024-07-26 21:36:56 -07:00
chrislu
e2a07d11d5 Revert "Check ShouldGrowVolumes before returning error in assign. (#5819)"
This reverts commit 98d66338d0333cd955f7840c64ef95e3c4807a17.
2024-07-26 11:21:50 -07:00
wyang
0581ce6096
fix delete chunk failed if volumeSever specified grpc.port (#5820)
Co-authored-by: Yang Wang <yangwang@weride.ai>
2024-07-26 11:14:19 -07:00
chrislu
ec9455dab7 same change as #5819
follow up with https://github.com/seaweedfs/seaweedfs/pull/5819
2024-07-26 11:06:47 -07:00
Ruoxi
d15966ae8e
Check ShouldGrowVolumes before returning error in assign. (#5819) 2024-07-26 11:04:38 -07:00
Chris Lu
952afd810c
Fix dead lock (#5815)
* reduce locks to avoid dead lock

Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0

goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
	/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
	/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
	/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
	/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
	/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
	/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)

SaveContent -> MemChunk.RLock ->
	ChunkedDirtyPages.saveChunkedFileIntervalToStorage
	pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
		fh.entryLock.Lock()

sync.(*RWMutex).Lock(0x0?)
	/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
	/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
	/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
	/github/workspace/weed/util/limited_executor.go:33 +0x97

On metadata update
fh.entryLock.Lock()
	fh.dirtyPages.Destroy()
		up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock

goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
	/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
	/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
	/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
	/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
	/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
	/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570

* use locked entry everywhere

* modifiable remote entry

* skip locking after getting lock from fhLockTable
2024-07-24 23:46:40 -07:00