mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-19 21:47:50 +08:00
Code Issues Packages Projects Releases Wiki Activity
11,187 Commits 32 Branches 296 Tags 152 MiB
9987a65e8a
Commit Graph

7215 Commits

Author SHA1 Message Date
Kamran Sarwar
63c707f9c1
added access-control-expose-headers to put request (#5876) 2024-08-09 15:52:05 -07:00
chrislu
e825dd7496 fix opposite logic 
fix https://github.com/seaweedfs/seaweedfs/issues/5871
 2024-08-08 23:16:38 -07:00
chrislu
e069009f09 [remote.cache, remote.uncache] fix min/max age 
fix https://github.com/seaweedfs/seaweedfs/issues/5871
 2024-08-08 21:25:52 -07:00
chrislu
9ca30e52d5 fuse mount handles ttl entries 
fix https://github.com/seaweedfs/seaweedfs/issues/5527
 2024-08-08 08:18:54 -07:00
chrislu
2074e7db67 fix invalid toml 2024-08-08 00:03:42 -07:00
chrislu
e86b0bcaaa simplify 2024-08-07 10:46:29 -07:00
chrislu
57dc39c451 randomizing next file handle id 2024-08-07 10:42:00 -07:00
Konstantin Lebedev
3e6ca6e706
[webdav] status code 500 if internal error from filer (#5865) 2024-08-06 07:39:52 -07:00
Konstantin Lebedev
ef4c555e90
[shell] volume.grow (#5855) 2024-08-05 09:22:17 -07:00
qinguoyi
bf2366ba3a
Fix KeyMaterial nil bug (#5849) 2024-08-02 11:06:48 -07:00
qinguoyi
e88da0be1b
Fix KeyMaterial nil bug (#5848) 2024-08-02 09:47:11 -07:00
qinguoyi
606d0cba64
Fix extractToFile gz file nil bug (#5847) 2024-08-02 09:46:37 -07:00
qinguoyi
b74b506e52
add command backup destory volume error log (#5846) 2024-08-02 00:04:57 -07:00
chrislu
8378a5b70b rename 2024-08-01 23:54:42 -07:00
chrislu
d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
zuzuviewer
5ee50d26d8
fix CVE-2016-2183 (#5844) 
* fix CVE-2016-2183

* allow tls.min_version and tls.max_version not set
 2024-08-01 20:00:22 -07:00
chrislu
4fee496b49 conditional delete 2024-08-01 13:00:54 -07:00
wyang
31b89c1062
fsck: only check the appendNs of deleted needle (#5841) 
increase fsck speed

Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-31 01:12:57 -07:00
Konstantin Lebedev
b2ffcdaab2
[master] do sync grow request only if absolutely necessary (#5821) 
* do sync grow request only if absolutely necessary
https://github.com/seaweedfs/seaweedfs/pull/5819

* remove check VolumeGrowStrategy Threshold on PickForWrite

* fix fmt.Errorf
 2024-07-30 13:21:35 -07:00
Jiffs Maverick
c8506609d8
[filer] Added a version to the title of Filer UI (#5836) 
Added a version to the title of Filer UI
 2024-07-29 14:07:12 -07:00
Konstantin Lebedev
e1f9336688
[filer] fix panic filer on webdav.maxMB (#5822) 
fix panic filer on webdav.maxMB
 2024-07-27 13:40:22 -07:00
wyang
4b1f539ab8
fix allocate reduplicated volumeId to different volume (#5811) 
* fix allocate reduplicated volumeId to different volume

* only check barrier when read

---------

Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-26 21:48:36 -07:00
wyang
c1bffca246
fix evacuate volume to different disk types (#5530) 
Co-authored-by: wyang <wyang@wyangs-Air.lan>
 2024-07-26 21:36:56 -07:00
chrislu
e2a07d11d5 Revert "Check ShouldGrowVolumes before returning error in assign. (#5819)" 
This reverts commit 98d66338d0333cd955f7840c64ef95e3c4807a17.
 2024-07-26 11:21:50 -07:00
wyang
0581ce6096
fix delete chunk failed if volumeSever specified grpc.port (#5820) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-26 11:14:19 -07:00
chrislu
ec9455dab7 same change as #5819 
follow up with https://github.com/seaweedfs/seaweedfs/pull/5819
 2024-07-26 11:06:47 -07:00
Ruoxi
d15966ae8e
Check ShouldGrowVolumes before returning error in assign. (#5819) 2024-07-26 11:04:38 -07:00
Chris Lu
952afd810c
Fix dead lock (#5815) 
* reduce locks to avoid dead lock

Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0

goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
	/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
	/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
	/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
	/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
	/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
	/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)

SaveContent -> MemChunk.RLock ->
	ChunkedDirtyPages.saveChunkedFileIntervalToStorage
	pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
		fh.entryLock.Lock()

sync.(*RWMutex).Lock(0x0?)
	/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
	/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
	/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
	/github/workspace/weed/util/limited_executor.go:33 +0x97

On metadata update
fh.entryLock.Lock()
	fh.dirtyPages.Destroy()
		up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock

goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
	/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
	/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
	/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
	/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
	/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
	/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
	/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
	/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
	/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570

* use locked entry everywhere

* modifiable remote entry

* skip locking after getting lock from fhLockTable
 2024-07-24 23:46:40 -07:00
chrislu
ed7e721efe 3.71 2024-07-21 21:03:27 -07:00
chrislu
9265be43c0 avoid nil 2024-07-21 21:01:29 -07:00
chrislu
4020149690 3.70 2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1 add version to filer 2024-07-21 20:13:51 -07:00
chrislu
9dcc576499 Revert "add collection for buckets" 
This reverts commit 96af571219.
 2024-07-20 12:00:50 -07:00
chrislu
aed2a18223 get, set and add path conf 2024-07-20 11:27:11 -07:00
chrislu
96af571219 add collection for buckets 2024-07-17 23:14:36 -07:00
chrislu
a8fa78b892 refactoring 2024-07-17 23:14:36 -07:00
chrislu
7a75928e7d Azure sink: avoid overwriting existing files 2024-07-17 23:14:36 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766) 
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
 2024-07-16 23:14:09 -07:00
chrislu
c6dec11ea5 [filer.sync] skip overwriting existing fresh entry 2024-07-16 09:38:10 -07:00
chrislu
19d14b4c71 minor 2024-07-16 09:21:01 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce (#5792) 2024-07-16 09:15:55 -07:00
Konstantin Lebedev
67edf1d014
[master] Do Automatic Volume Grow in background (#5781) 
* Do Automatic Volume Grow in backgound

* pass lastGrowCount to master

* fix build

* fix count to uint64
 2024-07-16 08:03:40 -07:00
wusong
ce61a66b65
Fix mmap write fail (#5791) 
* [mount] fix GetAttr blocks count

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

* [mount] fix mmap read

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

---------

Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2024-07-16 07:59:55 -07:00
Dan
794a5aa67d
Add warning with hints to why a EC shard cant be moved (#5790) 2024-07-15 11:17:22 -07:00
Konstantin Lebedev
a53e406c99
[master] refactor HasGrowRequest to atomic bool (#5782) 
refactor HasGrowRequest to atomit bool
 2024-07-15 10:51:21 -07:00
Konstantin Lebedev
33964fa292
metrics stats of volume layout depends on the data center (#5775) 
stats volume layout depends on the data center
 2024-07-12 12:32:25 -07:00
Konstantin Lebedev
04f4b10884
fix: avoid timeout if datacenter does not exist in topology (#5772) 
* fix: avoid timeout if datacenter does not exist in topology

* fix: error msg

* fix: rm dublicate check

* fix: compare

* revert minor change
 2024-07-12 11:19:08 -07:00
chrislu
3a82f5ffad ensure metadata follow a specific folder 
fix https://github.com/seaweedfs/seaweedfs/issues/5774
 2024-07-12 11:17:30 -07:00
vadimartynov
ec9e7493b3
-metricsIp cmd flag (#5773) 
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;

* Fixed cmd flags
 2024-07-12 10:56:26 -07:00
小羽
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time (#5771) 2024-07-12 10:49:15 -07:00
小羽
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat (#5770) 2024-07-12 08:09:51 -07:00
chrislu
4e7d8eb3f1 small refactoring 2024-07-11 14:50:11 -07:00
vadimartynov
de60f383de
Added tls for http handlers (#5764) 
* Added https handler for filer

* Added example for security.toml
 2024-07-11 07:53:18 -07:00
chrislu
2addcd1623 fix test 2024-07-11 07:53:02 -07:00
chrislu
590397229e fix heap init 2024-07-11 00:26:18 -07:00
chrislu
06e3dd306f clean up 2024-07-11 00:19:39 -07:00
chrislu
c59517e284 upgrade to github.com/redis/go-redis/v9 
fix https://github.com/seaweedfs/seaweedfs/issues/4532
 2024-07-11 00:15:19 -07:00
chrislu
33b11f92bc Squashed commit of the following: 
commit 465b47b8f72668dfc12eaa72befb232513c9e1db
Merge: f39272b54 9c440d472
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:56:51 2024 -0700

    Merge branch 'master' into merged-persisted-logs

commit f39272b5416bdd4d1714ce3e99aded0fe62e7d89
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:49:26 2024 -0700

    merge meta logs from multiple files
 2024-07-11 00:04:24 -07:00
Konstantin Lebedev
9c440d472c
[s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function (#5760) 
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function

* fix response error

* fix test_lifecycle_get

* Revert "fix test_lifecycle_get"

This reverts commit 8f991bdcf9.
 2024-07-10 08:47:43 -07:00
chrislu
182f6d00a9 always copy ecj files 2024-07-10 01:25:57 -07:00
Dan
bd54669d58
Detect underflow when calculating unused space (#5758) 
* Detect underflow when calculating unused space

* Detect underflow when calculating unused space
 2024-07-10 00:30:28 -07:00
chrislu
b8121e4b24 simplify 2024-07-08 11:10:41 -07:00
chrislu
f618121e63 Fix S3 deletion in deep folders, and names with empty spaces 
fix https://github.com/seaweedfs/seaweedfs/issues/5748
 2024-07-08 10:20:09 -07:00
chrislu
f3d708d3b6 add a test case 2024-07-07 23:44:36 -07:00
chrislu
d29fccc521 skip checking if it is the root directory 2024-07-07 23:44:36 -07:00
chrislu
114ac39917 add a simple test 2024-07-07 23:44:36 -07:00
Konstantin Lebedev
f77eee667d
add s3test for sql (#5718) 
* add s3test for sql

* fix test test_bucket_listv2_delimiter_basic for s3

* fix action s3tests

* regen s3 api xsd

* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty

* add docs

* without xmlns
 2024-07-04 11:00:41 -07:00
chrislu
7c06306857 fix breadcrumb 2024-07-03 17:40:55 -07:00
rhysm
7ee1f520a4
volume.tier.upload: Fix deleting replicated volumes (#5743) 
Co-authored-by: Rhys M <rhysm@protonmail.com>
 2024-07-03 15:51:25 -07:00
chrislu
8943a572d8 log error message 2024-07-03 09:50:47 -07:00
chrislu
62c29b2398 Revert "directory can paginate" 
This reverts commit 69003d0e90.
 2024-07-02 23:55:48 -07:00
chrislu
761a1555de fix ordering 2024-07-02 23:55:40 -07:00
chrislu
69003d0e90 directory can paginate 2024-07-02 21:36:03 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const (#5726) 2024-07-01 01:00:39 -07:00
chrislu
0fdf7eca48 avoid possible nil case during other error cases 2024-07-01 00:54:37 -07:00
chrislu
ac6fd36c06 using fixed bootstrap from peer filer 2024-06-30 23:16:54 -07:00
chrislu
fbe1316c92 3.69 2024-06-30 23:08:49 -07:00
chrislu
d8dfadb617 ParallelProcessDirectoryStructure 
need to use this for simpler parallel processing
 2024-06-28 15:02:46 -07:00
chrislu
1c0718f26d Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-06-28 14:57:23 -07:00
chrislu
c030cb3ce9 bootstrap filer from one peer 2024-06-28 14:57:20 -07:00
chrislu
00f87e5bb5 remove unused 2024-06-28 14:54:39 -07:00
Taehyung Lim
4d0bf6ddd4
fixed fail to initialize existing ec volume when volume server has separate index dictory (#5723) 2024-06-28 13:04:51 -07:00
chrislu
cb67137a03 optionally open the leveldb in readonly mode 2024-06-25 21:07:04 -07:00
chrislu
90182e97fe remove unused function 2024-06-25 09:18:11 -07:00
chrislu
9ac1023362 check bucket name in sql backend 
fix https://github.com/seaweedfs/seaweedfs/issues/5710
 2024-06-25 08:54:19 -07:00
chrislu
ded761ffc5 adjust visibility 2024-06-24 17:14:52 -07:00
chrislu
07f47776c3 minor 2024-06-23 11:47:26 -07:00
Nikita Borzykh
64756c2d79
fix: correct command flag for master.raftBootstrap option in server.go (#5701) 2024-06-22 19:28:37 -07:00
chrislu
82ef66cc19 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-06-22 08:30:18 -07:00
chrislu
622448b993 update advancedtls 2024-06-22 08:30:16 -07:00
Numblgw
73baf82f05
bugfix: unregister ec shards when volume server disconnected (#5697) 
bugfix unregister ec shards when volume server disconnected

Co-authored-by: liguowei <liguowei@xinye.com>
 2024-06-20 15:29:36 -07:00
Konstantin Lebedev
34165013c8
chore: hot fix panic fs.verify (#5696) 2024-06-20 15:24:31 -07:00
Konstantin Lebedev
7988ee0805
[fs.verify] skip failed files if entry not found on filerStore (#5693) 2024-06-19 06:50:55 -07:00
Konstantin Lebedev
ece018b1a9
[fs.verify] getmeta data from filer log (#5678) 
* initial

* fix: print entry path
 2024-06-18 22:56:47 -07:00
chrislu
a829f36d17 adjust error handling 
fix https://github.com/seaweedfs/seaweedfs/issues/5677
 2024-06-17 10:59:41 -07:00
chrislu
49893267e9 Merge branch 'master' into mq 2024-06-15 11:53:19 -07:00
chrislu
464611f614 optionally skip deleting file chunks 2024-06-15 11:39:48 -07:00
chrislu
64fd233a59 DeleteUncommittedChunks 2024-06-15 09:55:22 -07:00
chrislu
843e778875 refactor 2024-06-15 09:41:08 -07:00
chrislu
677cfb8ad1 refactor 2024-06-15 09:39:49 -07:00
chrislu
cb1ba0275a sort imports 2024-06-14 11:44:42 -07:00
chrislu
32dbb6a2ce Merge branch 'master' into mq 2024-06-14 11:42:05 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628) 
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
 2024-06-14 11:40:34 -07:00
chrislu
70b31e3a28 Merge branch 'master' into mq 2024-06-14 09:17:58 -07:00
chrislu
2a169dde9a minor 2024-06-14 09:17:46 -07:00
chrislu
8ffda89ed6 Merge branch 'master' into mq 2024-06-13 21:48:57 -07:00
chrislu
b03c831ad2 more logs 2024-06-13 21:48:44 -07:00
chrislu
458dda56ae Merge branch 'master' into mq 2024-06-12 00:32:04 -07:00
Taehyung Lim
4744889973
fix issue: sometimes volume.fsck report 'volume not found' (#5537) 
* fix issue: sometimes volume.fsck report 'volume not found' when a volume server has multiple disk types

* rename variable

* adjust counters

---------

Co-authored-by: chrislu <chris.lu@gmail.com>
 2024-06-11 22:22:57 -07:00
MeenakshiSachdeva
d2516d99cc
Meenakshi fix filer sync (#5672) 
* Fix Filer Sync Issue: 5455

* Fix Filer Sync Issue: 5455
 2024-06-10 11:16:29 -07:00
MeenakshiSachdeva
1caa012a76
Fix Filer Sync Issue: 5455 (#5663) 2024-06-10 09:19:38 -07:00
chrislu
0cd1c4be48 Merge branch 'master' into mq 2024-06-09 21:21:43 -07:00
chrislu
a9cd9b0542 3.68 2024-06-09 21:10:46 -07:00
chrislu
b5d36541d9 Merge branch 'master' into mq 2024-06-09 12:14:57 -07:00
chrislu
dbfbabac55 simplify 2024-06-09 12:14:10 -07:00
steve.wei
2150289442
fix: Ensure that the clientAddress is unique (#5655) 2024-06-07 09:13:03 -07:00
chrislu
25b2850e7d refactor out FilerClientAccessor 2024-06-06 19:44:19 -07:00
chrislu
4093115ca9 Merge branch 'master' into mq 2024-06-06 19:27:07 -07:00
Gaspare Iengo
dc6b750424
Fix panic (#5654) 2024-06-06 18:59:50 -07:00
Gaspare Iengo
fce8fc1e16
Fix Issue #5649 (#5652) 2024-06-06 12:49:33 -07:00
steve.wei
d8da4bbaa7
Set the capacity of clientChan to 10000 (#5647) 2024-06-05 05:41:46 -07:00
chrislu
9f02bf4ede Merge branch 'master' into mq 2024-06-04 21:12:30 -07:00
Lê Hoàng Nam
1f2dc54647
fix(s3api): make post api upload object default return 204 (#5648) 2024-06-04 06:18:47 -07:00
LHHDZ
36b5b713ba
fix deadlock caused by message chan blocked (#5639) 2024-06-03 07:42:40 -07:00
Dominic Pearson
d8bde9b96e
Solaris disk support (#5638) 2024-06-02 22:10:28 -07:00
NyaMisty
579ebbdf60
Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed (#5636) 2024-06-02 14:25:42 -07:00
NyaMisty
0c62d591e2
Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) (#5635) 2024-06-02 14:16:05 -07:00
NyaMisty
e1c671c028
Fix volume.tier.upload nil pointer panic (#5634) 2024-06-02 14:15:39 -07:00
NyaMisty
f2a37933b1
Fix sorted needle file's write mode for (*SortedFileNeedleMap).Delete() (#5633) 2024-06-02 14:12:25 -07:00
chrislu
41cc825ddb Merge branch 'master' into mq 2024-05-30 12:03:22 -07:00
chrislu
d1f0c40402 remove per-message debug logs 2024-05-30 09:49:08 -07:00
chrislu
df9d889489 fix size calculation 2024-05-30 09:41:04 -07:00
chrislu
ff14dfa0ee debug 2024-05-30 09:15:50 -07:00
chrislu
b485b9695c fix passing sub follower offset 2024-05-30 09:15:23 -07:00
chrislu
f7b818483a fix cases when buffer wraps around 2024-05-30 09:10:30 -07:00
chrislu
9a5cc065b0 fix 2024-05-30 08:29:06 -07:00
chrislu
23a72db1df stop partitionOffsetChan if closed 2024-05-30 00:27:44 -07:00
chrislu
af19256dbf skip control messages 2024-05-29 23:34:39 -07:00
chrislu
ad2b878130 fix inflight message tracker 2024-05-29 23:33:37 -07:00
chrislu
f8a22038c5 passing along publisher name 2024-05-29 23:31:48 -07:00
chrislu
f0b9e163a0 avoid nil stopChan 2024-05-29 16:18:02 -07:00
chrislu
a5874954dc less logs 2024-05-29 16:17:37 -07:00
chrislu
3ddfd15ec1 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-05-29 10:41:38 -07:00
chrislu
3e7a92061b pass along volume server grpc port 
fix https://github.com/seaweedfs/seaweedfs/issues/5617
 2024-05-29 10:41:33 -07:00
chrislu
edc87c16f9 balance subscribers 
need to ensure the offsets are persisted
 2024-05-27 17:30:16 -07:00
dependabot[bot]
722a14ef57
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.2.0 to 0.4.2 (#5621) 
* chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ

Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.2.0 to 0.4.2.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.2.0...v0.4.2)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix compilation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2024-05-27 12:36:00 -07:00
Vlad
72f7b4e9b1
Replaces volume.unmount in the help of the volumeServer.leave (#5619) 2024-05-27 01:13:06 -07:00
chrislu
6366898af2 add market 2024-05-26 14:35:22 -07:00
chrislu
96b326a304 use type ConsumerGroupInstanceId 2024-05-26 14:09:51 -07:00
chrislu
ba778539d9 Merge branch 'master' into mq 2024-05-24 17:15:27 -07:00
chrislu
364bb6c7b4 avoid ticker leak 2024-05-24 17:15:12 -07:00
chrislu
37d1ee562d refactor 2024-05-23 09:21:48 -07:00
chrislu
cdeaaf95b4 go fmt 2024-05-23 08:25:53 -07:00
chrislu
77a7c5c2a0 Merge branch 'master' into mq 2024-05-23 08:25:27 -07:00
chrislu
ba98f02d02 go fmt 2024-05-23 08:25:16 -07:00
chrislu
d40b350f54 coordinator receives unassignment ack 2024-05-23 08:23:35 -07:00
chrislu
73d008fafa Merge branch 'master' into mq 2024-05-22 20:39:31 -07:00
Chris Lu
cfcf383ee7
fix wrong prefix example 2024-05-21 21:45:29 -07:00
Vlad
a9b057389f
chore: add missing concurrency help in fs.meta.load command (#5609) 2024-05-21 21:42:48 -07:00
chrislu
554ae09f82 rename 2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa rename file 2024-05-21 09:57:59 -07:00
chrislu
67e6051585 rename Coordinator to SubCoordinator 2024-05-21 09:57:45 -07:00
chrislu
b5099263a4 rename file 2024-05-21 09:56:46 -07:00
chrislu
d5abffa42c rename Balancer to PubBalancer 2024-05-21 09:56:30 -07:00
chrislu
fa98ecf71e client side stop partition subscribing if unassigned 2024-05-21 08:42:04 -07:00
chrislu
6634b42981 fix compilation 2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3 refactor 
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
 2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6 avoid concurrent processing for the same key 2024-05-20 13:29:37 -07:00
chrislu
a7d30d0705 refactor 2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4 remove ProcessorConfiguration 2024-05-20 12:28:01 -07:00
chrislu
65dd5ac6fb fix 2024-05-20 12:27:45 -07:00
chrislu
3b7f24b1fc unused 2024-05-20 12:26:54 -07:00
chrislu
f13475ac2e sending keyed offset 2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d minor 2024-05-20 11:07:54 -07:00
chrislu
3622196881 go fmt 2024-05-20 11:05:18 -07:00
chrislu
5038577f7e Merge branch 'master' into mq 2024-05-20 11:04:53 -07:00
chrislu
aba212dedd Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2024-05-20 11:03:58 -07:00
chrislu
d218fe54fa go fmt 2024-05-20 11:03:56 -07:00
chrislu
7a4ace5dfb adjust logs 2024-05-20 09:34:07 -07:00
chrislu
d8ab22012a track offset 2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5 add InflightMessageTracker 2024-05-20 09:19:39 -07:00
chrislu
a2885512e1 fix 2024-05-19 15:06:33 -07:00
chrislu
8d02712c63 ConcurrentPartitionLimit 2024-05-19 14:55:03 -07:00
chrislu
8ce2cced47 less logs 2024-05-19 14:54:46 -07:00
chrislu
c6db3f31a1 start consuming ASAP 2024-05-19 14:52:38 -07:00
chrislu
c21b32dd20 logs 2024-05-19 11:02:17 -07:00
chrislu
b1871427c3 persist consumer group offset 
1. use one follower
2. read write consumer group offset
 2024-05-19 00:46:12 -07:00
sb
a3a00d9499
Feature/mongodb security (#5602) 2024-05-18 12:19:10 -07:00
sb
3fae87632f
add filer configuration options for username/password and tls (#5601) 2024-05-18 09:02:03 -07:00
chrislu
8d5bb7420d rename 2024-05-17 18:21:21 -07:00
chrislu
94742b8ace add subscribe follower 2024-05-17 18:17:23 -07:00
Riccardo Bertossa
f6e8a9bf9c
added s3 iam DeleteBucket permission management (#5599) 2024-05-17 04:54:09 -07:00
chrislu
279cb2b85e consumer acks received messages 2024-05-16 20:28:19 -07:00
chrislu
faffb2973c adjust errors 2024-05-16 11:02:48 -07:00
chrislu
759a8e50cf remove logs 2024-05-14 23:22:51 -07:00
chrislu
1f20178ded subscriber receives partitions and dispatch to processors 2024-05-14 23:22:43 -07:00
chrislu
972e9faaa2 move initial assignment to rebalance logic 2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e move read write topic config into filer client accessor 2024-05-14 08:50:17 -07:00
Martin Stiborský
54f3913bed
[s3] Fixed s3 replication by sending content-md as base64 (#5596) 2024-05-14 06:48:24 -07:00
chrislu
ac63f2b5a1 refactor 2024-05-13 22:03:57 -07:00
chrislu
844a1a0192 rename 2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d consumer instance passing MaxPartitionCount to coordinator 2024-05-13 18:30:10 -07:00
chrislu
2142842f82 refactor 2024-05-13 18:25:38 -07:00
Konstantin Lebedev
d389c5b27e
fix: recreate index include deleted files (#5579) 
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508

* fix: counting the number of files

* fix: log
 2024-05-12 11:31:34 -07:00
Dave St.Germain
731b3aadbe
Add support for OpenBSD (#5578) 
Co-authored-by: Dave St.Germain <dcs@adullmoment.com>
 2024-05-10 14:35:41 -07:00
chrislu
8cc5298a08 clean up unused variables 2024-05-10 13:10:45 -07:00
chrislu
d3032d1e80 3.67 2024-05-07 08:47:30 -07:00
chrislu
1917f70f78 also delete parent folder if empty 
fix https://github.com/seaweedfs/seaweedfs/issues/5567
 2024-05-07 08:46:19 -07:00
Patrick Schmidt
96c48bd0da
Fix deadlock in lock table locks (#5566) 2024-05-07 05:44:53 -07:00
chrislu
3682eb929f 3.66 2024-05-05 22:23:18 -07:00
chrislu
fcedf97e94 fix test 2024-05-05 22:19:59 -07:00
chrislu
664b704e30 add delay 2024-05-05 14:17:41 -07:00
chrislu
4e3701df77 reset last error 2024-05-05 14:16:54 -07:00
chrislu
658d302aa4 init candidate servers when setting a starting snapshot 2024-05-05 14:16:26 -07:00
chrislu
a5645d50a7 ensure head index is within range 2024-05-05 12:20:08 -07:00
chrislu
253ec73a17 log 2024-05-05 12:19:31 -07:00
chrislu
30f8600a24 working 2024-05-03 21:27:06 -07:00
chrislu
7a9b115cc2 rename 2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb subscriber can unmarshal the record 2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7 rename 2024-05-02 09:47:29 -07:00
chrislu
f2f4630029 rename field types 2024-05-02 08:59:22 -07:00
chrislu
b8af997e41 consistent with protobuf data types 2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea add example to publish a record 2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff rename 2024-05-02 08:35:06 -07:00
chrislu
5bc1b70a4c rename to WithField() 2024-05-02 08:32:15 -07:00
chrislu
a568d128fb Merge branch 'master' into mq-subscribe 2024-05-02 08:13:53 -07:00
chrislu
f80fb1c02e Merge branch 'master' into mq-subscribe 2024-04-29 21:57:31 -07:00
shenxingwuying
ee25ada732
reduce ambiguity about use memory_sequencer (#5555) 2024-04-29 21:51:00 -07:00
dependabot[bot]
36a1cf0361
chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 (#5552) 
* chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1

Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie) from 0.3.1 to 1.0.1.
- [Release notes](https://github.com/viant/ptrie/releases)
- [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/viant/ptrie
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix compilation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2024-04-29 18:42:31 -07:00
chrislu
5e8d788311 Merge branch 'master' into mq-subscribe 2024-04-29 06:44:03 -07:00
chrislu
e6bfd061c7 AllowEmptyFolder checks during object deletion 2024-04-29 06:25:06 -07:00
chrislu
d521466a37 split file 2024-04-29 06:23:42 -07:00
chrislu
a1b59948cc rename files 2024-04-29 05:33:56 -07:00
chrislu
b570e015ce Update write_parquet_test.go 2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01 record value API 2024-04-28 13:42:45 -07:00
chrislu
b166597c83 fix test 2024-04-28 13:03:01 -07:00
chrislu
189a7fc90e record type begin and end 2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9 improve api 2024-04-28 12:42:58 -07:00
chrislu
1a2332c257 rename api 2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546 Delete schema_builder_test.go 2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54 struct to schema correctly 2024-04-28 12:23:54 -07:00
chrislu
f37862bffe compile test 2024-04-28 12:23:33 -07:00
chrislu
7310d022ad recursive struct to schema 2024-04-28 11:14:13 -07:00
Konstantin Lebedev
6f9267ebdf
fix: skip file size checking for volume with TTL (#5541) 
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
 2024-04-28 00:21:57 -07:00
chrislu
a2a872ca03 rename 2024-04-27 23:44:11 -07:00
chrislu
60103171b6 rename 2024-04-27 23:30:15 -07:00
chrislu
111a4e1126 add field index 2024-04-27 23:30:06 -07:00
Konstantin Lebedev
3d3ee04eb9
[s3] Put bucket lifecycle configuration (#5510) 2024-04-27 07:39:22 -07:00
Kevin Bulteel
a3dab1fcd0
feat: upload files directly without multipart/form-data (#5539) 2024-04-26 16:03:40 -07:00
Nikita Borzykh
0d04264494
Move file deletion logging to v2 verbisity (#5538) 2024-04-26 06:41:06 -07:00