Ruoxi
d15966ae8e
Check ShouldGrowVolumes before returning error in assign. ( #5819 )
2024-07-26 11:04:38 -07:00
Chris Lu
952afd810c
Fix dead lock ( #5815 )
...
* reduce locks to avoid dead lock
Flush->FlushData->uplloadPipeline.FluahAll
uploaderCount>0
goroutine 1 [sync.Cond.Wait, 71 minutes]:
sync.runtime_notifyListWait(0xc0007ae4d0, 0x0)
/usr/local/go/src/runtime/sema.go:569 +0x159
sync.(*Cond).Wait(0xc001a59290?)
/usr/local/go/src/sync/cond.go:70 +0x85
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).waitForCurrentWritersToComplete(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline_lock.go:58 +0x32
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).FlushAll(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:151 +0x25
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).FlushData(0xc00087e840)
/github/workspace/weed/mount/dirty_pages_chunked.go:54 +0x29
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).FlushData(...)
/github/workspace/weed/mount/page_writer.go:50
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).doFlush(0xc0006ad600, 0xc00030d380, 0x0, 0x0)
/github/workspace/weed/mount/weedfs_file_sync.go:101 +0x169
github.com/seaweedfs/seaweedfs/weed/mount.(*WFS).Flush(0xc0006ad600, 0xc001a594a8?, 0xc0004c1ca0)
/github/workspace/weed/mount/weedfs_file_sync.go:59 +0x48
github.com/hanwen/go-fuse/v2/fuse.doFlush(0xc0000da870?, 0xc0004c1b08)
SaveContent -> MemChunk.RLock ->
ChunkedDirtyPages.saveChunkedFileIntervalToStorage
pages.fh.AddChunks([]*filer_pb.FileChunk{chunk})
fh.entryLock.Lock()
sync.(*RWMutex).Lock(0x0?)
/usr/local/go/src/sync/rwmutex.go:146 +0x31
github.com/seaweedfs/seaweedfs/weed/mount.(*FileHandle).AddChunks(0xc00030d380, {0xc00028bdc8, 0x1, 0x1})
/github/workspace/weed/mount/filehandle.go:93 +0x45
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).saveChunkedFileIntervalToStorage(0xc00087e840, {0x2be7ac0, 0xc00018d9e0}, 0x0, 0x121, 0x17e3c624565ace45, 0x1?)
/github/workspace/weed/mount/dirty_pages_chunked.go:80 +0x2d4
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).SaveContent(0xc0008d9130, 0xc0008093e0)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:115 +0x112
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).moveToSealed.func1()
/github/workspace/weed/mount/page_writer/upload_pipeline.go:187 +0x55
github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute.func1()
/github/workspace/weed/util/limited_executor.go:38 +0x62
created by github.com/seaweedfs/seaweedfs/weed/util.(*LimitedConcurrentExecutor).Execute in goroutine 1
/github/workspace/weed/util/limited_executor.go:33 +0x97
On metadata update
fh.entryLock.Lock()
fh.dirtyPages.Destroy()
up.chunksLock.Lock => each sealed chunk.FreeReference => MemChunk.Lock
goroutine 134 [sync.RWMutex.Lock, 71 minutes]:
sync.runtime_SemacquireRWMutex(0xc0007c3558?, 0xea?, 0x3fb0800?)
/usr/local/go/src/runtime/sema.go:87 +0x25
sync.(*RWMutex).Lock(0xc0007c35a8?)
/usr/local/go/src/sync/rwmutex.go:151 +0x6a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*MemChunk).FreeResource(0xc0008d9130)
/github/workspace/weed/mount/page_writer/page_chunk_mem.go:38 +0x2a
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*SealedChunk).FreeReference(0xc00071cdb0, {0xc0006ba1a0, 0x20})
/github/workspace/weed/mount/page_writer/upload_pipeline.go:38 +0xb7
github.com/seaweedfs/seaweedfs/weed/mount/page_writer.(*UploadPipeline).Shutdown(0xc0002ee4d0)
/github/workspace/weed/mount/page_writer/upload_pipeline.go:220 +0x185
github.com/seaweedfs/seaweedfs/weed/mount.(*ChunkedDirtyPages).Destroy(0xc0008cea40?)
/github/workspace/weed/mount/dirty_pages_chunked.go:87 +0x17
github.com/seaweedfs/seaweedfs/weed/mount.(*PageWriter).Destroy(...)
/github/workspace/weed/mount/page_writer.go:78
github.com/seaweedfs/seaweedfs/weed/mount.NewSeaweedFileSystem.func3({0xc00069a6c0, 0x30}, 0x6?)
/github/workspace/weed/mount/weedfs.go:119 +0x17a
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.NewMetaCache.func1({0xc00069a6c0?, 0xc00069a480?}, 0x4015b40?)
/github/workspace/weed/mount/meta_cache/meta_cache.go:37 +0x1c
github.com/seaweedfs/seaweedfs/weed/mount/meta_cache.SubscribeMetaEvents.func1(0xc000661810)
/github/workspace/weed/mount/meta_cache/meta_cache_subscribe.go:43 +0x570
* use locked entry everywhere
* modifiable remote entry
* skip locking after getting lock from fhLockTable
2024-07-24 23:46:40 -07:00
chrislu
ed7e721efe
3.71
2024-07-21 21:03:27 -07:00
chrislu
9265be43c0
avoid nil
2024-07-21 21:01:29 -07:00
chrislu
4020149690
3.70
2024-07-21 20:27:39 -07:00
chrislu
9dd008f8f1
add version to filer
2024-07-21 20:13:51 -07:00
chrislu
9dcc576499
Revert "add collection for buckets"
...
This reverts commit 96af571219
2024-07-20 12:00:50 -07:00
chrislu
aed2a18223
get, set and add path conf
2024-07-20 11:27:11 -07:00
chrislu
96af571219
add collection for buckets
2024-07-17 23:14:36 -07:00
chrislu
a8fa78b892
refactoring
2024-07-17 23:14:36 -07:00
chrislu
7a75928e7d
Azure sink: avoid overwriting existing files
2024-07-17 23:14:36 -07:00
vadimartynov
86d92a42b4
Added tls for http clients ( #5766 )
...
* Added global http client
* Added Do func for global http client
* Changed the code to use the global http client
* Fix http client in volume uploader
* Fixed pkg name
* Fixed http util funcs
* Fixed http client for bench_filer_upload
* Fixed http client for stress_filer_upload
* Fixed http client for filer_server_handlers_proxy
* Fixed http client for command_fs_merge_volumes
* Fixed http client for command_fs_merge_volumes and command_volume_fsck
* Fixed http client for s3api_server
* Added init global client for main funcs
* Rename global_client to client
* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold
* Reduce the visibility of some functions in the util/http/client pkg
* Added the loadSecurityConfig function
* Use util.LoadSecurityConfiguration() in NewHttpClient func
2024-07-16 23:14:09 -07:00
chrislu
c6dec11ea5
[filer.sync] skip overwriting existing fresh entry
2024-07-16 09:38:10 -07:00
chrislu
19d14b4c71
minor
2024-07-16 09:21:01 -07:00
vadimartynov
b796c21fa9
Added loadSecurityConfigOnce ( #5792 )
2024-07-16 09:15:55 -07:00
Konstantin Lebedev
67edf1d014
[master] Do Automatic Volume Grow in background ( #5781 )
...
* Do Automatic Volume Grow in backgound
* pass lastGrowCount to master
* fix build
* fix count to uint64
2024-07-16 08:03:40 -07:00
wusong
ce61a66b65
Fix mmap write fail ( #5791 )
...
* [mount] fix GetAttr blocks count
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
* [mount] fix mmap read
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
---------
Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
2024-07-16 07:59:55 -07:00
Dan
794a5aa67d
Add warning with hints to why a EC shard cant be moved ( #5790 )
2024-07-15 11:17:22 -07:00
Konstantin Lebedev
a53e406c99
[master] refactor HasGrowRequest to atomic bool ( #5782 )
...
refactor HasGrowRequest to atomit bool
2024-07-15 10:51:21 -07:00
Konstantin Lebedev
33964fa292
metrics stats of volume layout depends on the data center ( #5775 )
...
stats volume layout depends on the data center
2024-07-12 12:32:25 -07:00
Konstantin Lebedev
04f4b10884
fix: avoid timeout if datacenter does not exist in topology ( #5772 )
...
* fix: avoid timeout if datacenter does not exist in topology
* fix: error msg
* fix: rm dublicate check
* fix: compare
* revert minor change
2024-07-12 11:19:08 -07:00
chrislu
3a82f5ffad
ensure metadata follow a specific folder
...
fix https://github.com/seaweedfs/seaweedfs/issues/5774
2024-07-12 11:17:30 -07:00
vadimartynov
ec9e7493b3
-metricsIp cmd flag ( #5773 )
...
* Added/Updated:
- Added metrics ip options for all servers;
- Fixed a bug with the selection of the binIp or ip parameter for the metrics handler;
* Fixed cmd flags
2024-07-12 10:56:26 -07:00
小羽
cb5dae0c9c
Fix "no more writable volumes" error when volume grows and master leader changed at the same time ( #5771 )
2024-07-12 10:49:15 -07:00
小羽
e8537d7172
Different disk labels should not use the same DiskUsages instance while master received volume heatbeat ( #5770 )
2024-07-12 08:09:51 -07:00
chrislu
4e7d8eb3f1
small refactoring
2024-07-11 14:50:11 -07:00
vadimartynov
de60f383de
Added tls for http handlers ( #5764 )
...
* Added https handler for filer
* Added example for security.toml
2024-07-11 07:53:18 -07:00
chrislu
2addcd1623
fix test
2024-07-11 07:53:02 -07:00
chrislu
590397229e
fix heap init
2024-07-11 00:26:18 -07:00
chrislu
06e3dd306f
clean up
2024-07-11 00:19:39 -07:00
chrislu
c59517e284
upgrade to github.com/redis/go-redis/v9
...
fix https://github.com/seaweedfs/seaweedfs/issues/4532
2024-07-11 00:15:19 -07:00
chrislu
33b11f92bc
Squashed commit of the following:
...
commit 465b47b8f72668dfc12eaa72befb232513c9e1db
Merge: f39272b54 9c440d472
2024-07-11 00:04:24 -07:00
Konstantin Lebedev
9c440d472c
[s3] revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function ( #5760 )
...
* revert skip deletion error, since the error file was not found is already skipped on the side of the grpc function
* fix response error
* fix test_lifecycle_get
* Revert "fix test_lifecycle_get"
This reverts commit 8f991bdcf9
2024-07-10 08:47:43 -07:00
chrislu
182f6d00a9
always copy ecj files
2024-07-10 01:25:57 -07:00
Dan
bd54669d58
Detect underflow when calculating unused space ( #5758 )
...
* Detect underflow when calculating unused space
* Detect underflow when calculating unused space
2024-07-10 00:30:28 -07:00
chrislu
b8121e4b24
simplify
2024-07-08 11:10:41 -07:00
chrislu
f618121e63
Fix S3 deletion in deep folders, and names with empty spaces
...
fix https://github.com/seaweedfs/seaweedfs/issues/5748
2024-07-08 10:20:09 -07:00
chrislu
f3d708d3b6
add a test case
2024-07-07 23:44:36 -07:00
chrislu
d29fccc521
skip checking if it is the root directory
2024-07-07 23:44:36 -07:00
chrislu
114ac39917
add a simple test
2024-07-07 23:44:36 -07:00
Konstantin Lebedev
f77eee667d
add s3test for sql ( #5718 )
...
* add s3test for sql
* fix test test_bucket_listv2_delimiter_basic for s3
* fix action s3tests
* regen s3 api xsd
* rm minor s3 test test_bucket_listv2_fetchowner_defaultempty
* add docs
* without xmlns
2024-07-04 11:00:41 -07:00
chrislu
7c06306857
fix breadcrumb
2024-07-03 17:40:55 -07:00
rhysm
7ee1f520a4
volume.tier.upload: Fix deleting replicated volumes ( #5743 )
...
Co-authored-by: Rhys M <rhysm@protonmail.com>
2024-07-03 15:51:25 -07:00
chrislu
8943a572d8
log error message
2024-07-03 09:50:47 -07:00
chrislu
62c29b2398
Revert "directory can paginate"
...
This reverts commit 69003d0e90
2024-07-02 23:55:48 -07:00
chrislu
761a1555de
fix ordering
2024-07-02 23:55:40 -07:00
chrislu
69003d0e90
directory can paginate
2024-07-02 21:36:03 -07:00
Konstantin Lebedev
5ffacbb6ea
refactor all methods strings to const ( #5726 )
2024-07-01 01:00:39 -07:00
chrislu
0fdf7eca48
avoid possible nil case during other error cases
2024-07-01 00:54:37 -07:00
chrislu
ac6fd36c06
using fixed bootstrap from peer filer
2024-06-30 23:16:54 -07:00
chrislu
fbe1316c92
3.69
2024-06-30 23:08:49 -07:00
chrislu
d8dfadb617
ParallelProcessDirectoryStructure
...
need to use this for simpler parallel processing
2024-06-28 15:02:46 -07:00
chrislu
1c0718f26d
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-06-28 14:57:23 -07:00
chrislu
c030cb3ce9
bootstrap filer from one peer
2024-06-28 14:57:20 -07:00
chrislu
00f87e5bb5
remove unused
2024-06-28 14:54:39 -07:00
Taehyung Lim
4d0bf6ddd4
fixed fail to initialize existing ec volume when volume server has separate index dictory ( #5723 )
2024-06-28 13:04:51 -07:00
chrislu
cb67137a03
optionally open the leveldb in readonly mode
2024-06-25 21:07:04 -07:00
chrislu
90182e97fe
remove unused function
2024-06-25 09:18:11 -07:00
chrislu
9ac1023362
check bucket name in sql backend
...
fix https://github.com/seaweedfs/seaweedfs/issues/5710
2024-06-25 08:54:19 -07:00
chrislu
ded761ffc5
adjust visibility
2024-06-24 17:14:52 -07:00
chrislu
07f47776c3
minor
2024-06-23 11:47:26 -07:00
Nikita Borzykh
64756c2d79
fix: correct command flag for master.raftBootstrap option in server.go ( #5701 )
2024-06-22 19:28:37 -07:00
chrislu
82ef66cc19
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-06-22 08:30:18 -07:00
chrislu
622448b993
update advancedtls
2024-06-22 08:30:16 -07:00
Numblgw
73baf82f05
bugfix: unregister ec shards when volume server disconnected ( #5697 )
...
bugfix unregister ec shards when volume server disconnected
Co-authored-by: liguowei <liguowei@xinye.com>
2024-06-20 15:29:36 -07:00
Konstantin Lebedev
34165013c8
chore: hot fix panic fs.verify ( #5696 )
2024-06-20 15:24:31 -07:00
Konstantin Lebedev
7988ee0805
[fs.verify] skip failed files if entry not found on filerStore ( #5693 )
2024-06-19 06:50:55 -07:00
Konstantin Lebedev
ece018b1a9
[fs.verify] getmeta data from filer log ( #5678 )
...
* initial
* fix: print entry path
2024-06-18 22:56:47 -07:00
chrislu
a829f36d17
adjust error handling
...
fix https://github.com/seaweedfs/seaweedfs/issues/5677
2024-06-17 10:59:41 -07:00
chrislu
49893267e9
Merge branch 'master' into mq
2024-06-15 11:53:19 -07:00
chrislu
464611f614
optionally skip deleting file chunks
2024-06-15 11:39:48 -07:00
chrislu
64fd233a59
DeleteUncommittedChunks
2024-06-15 09:55:22 -07:00
chrislu
843e778875
refactor
2024-06-15 09:41:08 -07:00
chrislu
677cfb8ad1
refactor
2024-06-15 09:39:49 -07:00
chrislu
cb1ba0275a
sort imports
2024-06-14 11:44:42 -07:00
chrislu
32dbb6a2ce
Merge branch 'master' into mq
2024-06-14 11:42:05 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops ( #5628 )
...
* Added context for the MasterClient's methods to avoid endless loops
* Returned WithClient function. Added WithClientCustomGetMaster function
* Hid unused ctx arguments
* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions
* Changed the context termination check in the tryConnectToMaster function
* Added a child context to the tryConnectToMaster function
* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
2024-06-14 11:40:34 -07:00
chrislu
70b31e3a28
Merge branch 'master' into mq
2024-06-14 09:17:58 -07:00
chrislu
2a169dde9a
minor
2024-06-14 09:17:46 -07:00
chrislu
8ffda89ed6
Merge branch 'master' into mq
2024-06-13 21:48:57 -07:00
chrislu
b03c831ad2
more logs
2024-06-13 21:48:44 -07:00
chrislu
458dda56ae
Merge branch 'master' into mq
2024-06-12 00:32:04 -07:00
Taehyung Lim
4744889973
fix issue: sometimes volume.fsck report 'volume not found' ( #5537 )
...
* fix issue: sometimes volume.fsck report 'volume not found' when a volume server has multiple disk types
* rename variable
* adjust counters
---------
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-06-11 22:22:57 -07:00
MeenakshiSachdeva
d2516d99cc
Meenakshi fix filer sync ( #5672 )
...
* Fix Filer Sync Issue: 5455
* Fix Filer Sync Issue: 5455
2024-06-10 11:16:29 -07:00
MeenakshiSachdeva
1caa012a76
Fix Filer Sync Issue: 5455 ( #5663 )
2024-06-10 09:19:38 -07:00
chrislu
0cd1c4be48
Merge branch 'master' into mq
2024-06-09 21:21:43 -07:00
chrislu
a9cd9b0542
3.68
2024-06-09 21:10:46 -07:00
chrislu
b5d36541d9
Merge branch 'master' into mq
2024-06-09 12:14:57 -07:00
chrislu
dbfbabac55
simplify
2024-06-09 12:14:10 -07:00
steve.wei
2150289442
fix: Ensure that the clientAddress is unique ( #5655 )
2024-06-07 09:13:03 -07:00
chrislu
25b2850e7d
refactor out FilerClientAccessor
2024-06-06 19:44:19 -07:00
chrislu
4093115ca9
Merge branch 'master' into mq
2024-06-06 19:27:07 -07:00
Gaspare Iengo
dc6b750424
Fix panic ( #5654 )
2024-06-06 18:59:50 -07:00
Gaspare Iengo
fce8fc1e16
Fix Issue #5649 ( #5652 )
2024-06-06 12:49:33 -07:00
steve.wei
d8da4bbaa7
Set the capacity of clientChan to 10000 ( #5647 )
2024-06-05 05:41:46 -07:00
chrislu
9f02bf4ede
Merge branch 'master' into mq
2024-06-04 21:12:30 -07:00
Lê Hoàng Nam
1f2dc54647
fix(s3api): make post api upload object default return 204 ( #5648 )
2024-06-04 06:18:47 -07:00
LHHDZ
36b5b713ba
fix deadlock caused by message chan blocked ( #5639 )
2024-06-03 07:42:40 -07:00
Dominic Pearson
d8bde9b96e
Solaris disk support ( #5638 )
2024-06-02 22:10:28 -07:00
NyaMisty
579ebbdf60
Support concurrent volume.fsck & support disabling -cutoffTimeAgo to improve speed ( #5636 )
2024-06-02 14:25:42 -07:00
NyaMisty
0c62d591e2
Ignore remote volume when selecting volumes in operation (ec.encode/volume.tier.upload) ( #5635 )
2024-06-02 14:16:05 -07:00
NyaMisty
e1c671c028
Fix volume.tier.upload nil pointer panic ( #5634 )
2024-06-02 14:15:39 -07:00
NyaMisty
f2a37933b1
Fix sorted needle file's write mode for (*SortedFileNeedleMap).Delete() ( #5633 )
2024-06-02 14:12:25 -07:00
chrislu
41cc825ddb
Merge branch 'master' into mq
2024-05-30 12:03:22 -07:00
chrislu
d1f0c40402
remove per-message debug logs
2024-05-30 09:49:08 -07:00
chrislu
df9d889489
fix size calculation
2024-05-30 09:41:04 -07:00
chrislu
ff14dfa0ee
debug
2024-05-30 09:15:50 -07:00
chrislu
b485b9695c
fix passing sub follower offset
2024-05-30 09:15:23 -07:00
chrislu
f7b818483a
fix cases when buffer wraps around
2024-05-30 09:10:30 -07:00
chrislu
9a5cc065b0
fix
2024-05-30 08:29:06 -07:00
chrislu
23a72db1df
stop partitionOffsetChan if closed
2024-05-30 00:27:44 -07:00
chrislu
af19256dbf
skip control messages
2024-05-29 23:34:39 -07:00
chrislu
ad2b878130
fix inflight message tracker
2024-05-29 23:33:37 -07:00
chrislu
f8a22038c5
passing along publisher name
2024-05-29 23:31:48 -07:00
chrislu
f0b9e163a0
avoid nil stopChan
2024-05-29 16:18:02 -07:00
chrislu
a5874954dc
less logs
2024-05-29 16:17:37 -07:00
chrislu
3ddfd15ec1
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-05-29 10:41:38 -07:00
chrislu
3e7a92061b
pass along volume server grpc port
...
fix https://github.com/seaweedfs/seaweedfs/issues/5617
2024-05-29 10:41:33 -07:00
chrislu
edc87c16f9
balance subscribers
...
need to ensure the offsets are persisted
2024-05-27 17:30:16 -07:00
dependabot[bot]
722a14ef57
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.2.0 to 0.4.2 ( #5621 )
...
* chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ
Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ ) from 0.2.0 to 0.4.2.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md )
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.2.0...v0.4.2 )
---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-05-27 12:36:00 -07:00
Vlad
72f7b4e9b1
Replaces volume.unmount in the help of the volumeServer.leave ( #5619 )
2024-05-27 01:13:06 -07:00
chrislu
6366898af2
add market
2024-05-26 14:35:22 -07:00
chrislu
96b326a304
use type ConsumerGroupInstanceId
2024-05-26 14:09:51 -07:00
chrislu
ba778539d9
Merge branch 'master' into mq
2024-05-24 17:15:27 -07:00
chrislu
364bb6c7b4
avoid ticker leak
2024-05-24 17:15:12 -07:00
chrislu
37d1ee562d
refactor
2024-05-23 09:21:48 -07:00
chrislu
cdeaaf95b4
go fmt
2024-05-23 08:25:53 -07:00
chrislu
77a7c5c2a0
Merge branch 'master' into mq
2024-05-23 08:25:27 -07:00
chrislu
ba98f02d02
go fmt
2024-05-23 08:25:16 -07:00
chrislu
d40b350f54
coordinator receives unassignment ack
2024-05-23 08:23:35 -07:00
chrislu
73d008fafa
Merge branch 'master' into mq
2024-05-22 20:39:31 -07:00
Chris Lu
cfcf383ee7
fix wrong prefix example
2024-05-21 21:45:29 -07:00
Vlad
a9b057389f
chore: add missing concurrency help in fs.meta.load command ( #5609 )
2024-05-21 21:42:48 -07:00
chrislu
554ae09f82
rename
2024-05-21 10:02:07 -07:00
chrislu
6ef2d010aa
rename file
2024-05-21 09:57:59 -07:00
chrislu
67e6051585
rename Coordinator to SubCoordinator
2024-05-21 09:57:45 -07:00
chrislu
b5099263a4
rename file
2024-05-21 09:56:46 -07:00
chrislu
d5abffa42c
rename Balancer to PubBalancer
2024-05-21 09:56:30 -07:00
chrislu
fa98ecf71e
client side stop partition subscribing if unassigned
2024-05-21 08:42:04 -07:00
chrislu
6634b42981
fix compilation
2024-05-21 08:26:55 -07:00
chrislu
2190d59bc3
refactor
...
ToDo
1. handle unassignment on client side
2. get notified of completed unassignment
3. in consumer_group.go, balance and send unassignment
2024-05-21 08:05:56 -07:00
chrislu
3148dec1f6
avoid concurrent processing for the same key
2024-05-20 13:29:37 -07:00
chrislu
a7d30d0705
refactor
2024-05-20 12:32:12 -07:00
chrislu
33aecb7bb4
remove ProcessorConfiguration
2024-05-20 12:28:01 -07:00
chrislu
65dd5ac6fb
fix
2024-05-20 12:27:45 -07:00
chrislu
3b7f24b1fc
unused
2024-05-20 12:26:54 -07:00
chrislu
f13475ac2e
sending keyed offset
2024-05-20 11:09:48 -07:00
chrislu
d45b1d058d
minor
2024-05-20 11:07:54 -07:00
chrislu
3622196881
go fmt
2024-05-20 11:05:18 -07:00
chrislu
5038577f7e
Merge branch 'master' into mq
2024-05-20 11:04:53 -07:00
chrislu
aba212dedd
Merge branch 'master' of https://github.com/seaweedfs/seaweedfs
2024-05-20 11:03:58 -07:00
chrislu
d218fe54fa
go fmt
2024-05-20 11:03:56 -07:00
chrislu
7a4ace5dfb
adjust logs
2024-05-20 09:34:07 -07:00
chrislu
d8ab22012a
track offset
2024-05-20 09:33:37 -07:00
chrislu
2b07a40da5
add InflightMessageTracker
2024-05-20 09:19:39 -07:00
chrislu
a2885512e1
fix
2024-05-19 15:06:33 -07:00
chrislu
8d02712c63
ConcurrentPartitionLimit
2024-05-19 14:55:03 -07:00
chrislu
8ce2cced47
less logs
2024-05-19 14:54:46 -07:00
chrislu
c6db3f31a1
start consuming ASAP
2024-05-19 14:52:38 -07:00
chrislu
c21b32dd20
logs
2024-05-19 11:02:17 -07:00
chrislu
b1871427c3
persist consumer group offset
...
1. use one follower
2. read write consumer group offset
2024-05-19 00:46:12 -07:00
sb
a3a00d9499
Feature/mongodb security ( #5602 )
2024-05-18 12:19:10 -07:00
sb
3fae87632f
add filer configuration options for username/password and tls ( #5601 )
2024-05-18 09:02:03 -07:00
chrislu
8d5bb7420d
rename
2024-05-17 18:21:21 -07:00
chrislu
94742b8ace
add subscribe follower
2024-05-17 18:17:23 -07:00
Riccardo Bertossa
f6e8a9bf9c
added s3 iam DeleteBucket permission management ( #5599 )
2024-05-17 04:54:09 -07:00
chrislu
279cb2b85e
consumer acks received messages
2024-05-16 20:28:19 -07:00
chrislu
faffb2973c
adjust errors
2024-05-16 11:02:48 -07:00
chrislu
759a8e50cf
remove logs
2024-05-14 23:22:51 -07:00
chrislu
1f20178ded
subscriber receives partitions and dispatch to processors
2024-05-14 23:22:43 -07:00
chrislu
972e9faaa2
move initial assignment to rebalance logic
2024-05-14 09:09:36 -07:00
chrislu
6e5075e14e
move read write topic config into filer client accessor
2024-05-14 08:50:17 -07:00
Martin Stiborský
54f3913bed
[s3] Fixed s3 replication by sending content-md as base64 ( #5596 )
2024-05-14 06:48:24 -07:00
chrislu
ac63f2b5a1
refactor
2024-05-13 22:03:57 -07:00
chrislu
844a1a0192
rename
2024-05-13 19:24:47 -07:00
chrislu
372bd8d71d
consumer instance passing MaxPartitionCount to coordinator
2024-05-13 18:30:10 -07:00
chrislu
2142842f82
refactor
2024-05-13 18:25:38 -07:00
Konstantin Lebedev
d389c5b27e
fix: recreate index include deleted files ( #5579 )
...
* fix: recreate index include deleted files
https://github.com/seaweedfs/seaweedfs/issues/5508
* fix: counting the number of files
* fix: log
2024-05-12 11:31:34 -07:00
Dave St.Germain
731b3aadbe
Add support for OpenBSD ( #5578 )
...
Co-authored-by: Dave St.Germain <dcs@adullmoment.com>
2024-05-10 14:35:41 -07:00
chrislu
8cc5298a08
clean up unused variables
2024-05-10 13:10:45 -07:00
chrislu
d3032d1e80
3.67
2024-05-07 08:47:30 -07:00
chrislu
1917f70f78
also delete parent folder if empty
...
fix https://github.com/seaweedfs/seaweedfs/issues/5567
2024-05-07 08:46:19 -07:00
Patrick Schmidt
96c48bd0da
Fix deadlock in lock table locks ( #5566 )
2024-05-07 05:44:53 -07:00
chrislu
3682eb929f
3.66
2024-05-05 22:23:18 -07:00
chrislu
fcedf97e94
fix test
2024-05-05 22:19:59 -07:00
chrislu
664b704e30
add delay
2024-05-05 14:17:41 -07:00
chrislu
4e3701df77
reset last error
2024-05-05 14:16:54 -07:00
chrislu
658d302aa4
init candidate servers when setting a starting snapshot
2024-05-05 14:16:26 -07:00
chrislu
a5645d50a7
ensure head index is within range
2024-05-05 12:20:08 -07:00
chrislu
253ec73a17
log
2024-05-05 12:19:31 -07:00
chrislu
30f8600a24
working
2024-05-03 21:27:06 -07:00
chrislu
7a9b115cc2
rename
2024-05-02 11:14:58 -07:00
chrislu
4a3e8869bb
subscriber can unmarshal the record
2024-05-02 09:56:11 -07:00
chrislu
1c5043baf7
rename
2024-05-02 09:47:29 -07:00
chrislu
f2f4630029
rename field types
2024-05-02 08:59:22 -07:00
chrislu
b8af997e41
consistent with protobuf data types
2024-05-02 08:52:10 -07:00
chrislu
0f35b3a4ea
add example to publish a record
2024-05-02 08:48:51 -07:00
chrislu
928a4e8dff
rename
2024-05-02 08:35:06 -07:00
chrislu
5bc1b70a4c
rename to WithField()
2024-05-02 08:32:15 -07:00
chrislu
a568d128fb
Merge branch 'master' into mq-subscribe
2024-05-02 08:13:53 -07:00
chrislu
f80fb1c02e
Merge branch 'master' into mq-subscribe
2024-04-29 21:57:31 -07:00
shenxingwuying
ee25ada732
reduce ambiguity about use memory_sequencer ( #5555 )
2024-04-29 21:51:00 -07:00
dependabot[bot]
36a1cf0361
chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 ( #5552 )
...
* chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1
Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie ) from 0.3.1 to 1.0.1.
- [Release notes](https://github.com/viant/ptrie/releases )
- [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md )
- [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1 )
---
updated-dependencies:
- dependency-name: github.com/viant/ptrie
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
* fix compilation
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
2024-04-29 18:42:31 -07:00
chrislu
5e8d788311
Merge branch 'master' into mq-subscribe
2024-04-29 06:44:03 -07:00
chrislu
e6bfd061c7
AllowEmptyFolder checks during object deletion
2024-04-29 06:25:06 -07:00
chrislu
d521466a37
split file
2024-04-29 06:23:42 -07:00
chrislu
a1b59948cc
rename files
2024-04-29 05:33:56 -07:00
chrislu
b570e015ce
Update write_parquet_test.go
2024-04-28 13:56:55 -07:00
chrislu
5dc186ef01
record value API
2024-04-28 13:42:45 -07:00
chrislu
b166597c83
fix test
2024-04-28 13:03:01 -07:00
chrislu
189a7fc90e
record type begin and end
2024-04-28 13:00:52 -07:00
chrislu
0da1b7b0f9
improve api
2024-04-28 12:42:58 -07:00
chrislu
1a2332c257
rename api
2024-04-28 12:40:00 -07:00
chrislu
5ce8f50546
Delete schema_builder_test.go
2024-04-28 12:39:27 -07:00
chrislu
8a0e5b6d54
struct to schema correctly
2024-04-28 12:23:54 -07:00
chrislu
f37862bffe
compile test
2024-04-28 12:23:33 -07:00
chrislu
7310d022ad
recursive struct to schema
2024-04-28 11:14:13 -07:00
Konstantin Lebedev
6f9267ebdf
fix: skip file size checking for volume with TTL ( #5541 )
...
fix: skip file size checking for volume with tll
https://github.com/seaweedfs/seaweedfs/issues/5508
2024-04-28 00:21:57 -07:00
chrislu
a2a872ca03
rename
2024-04-27 23:44:11 -07:00
chrislu
60103171b6
rename
2024-04-27 23:30:15 -07:00
chrislu
111a4e1126
add field index
2024-04-27 23:30:06 -07:00
Konstantin Lebedev
3d3ee04eb9
[s3] Put bucket lifecycle configuration ( #5510 )
2024-04-27 07:39:22 -07:00
Kevin Bulteel
a3dab1fcd0
feat: upload files directly without multipart/form-data ( #5539 )
2024-04-26 16:03:40 -07:00
Nikita Borzykh
0d04264494
Move file deletion logging to v2 verbisity ( #5538 )
2024-04-26 06:41:06 -07:00
chrislu
72b50980f4
a little bit more efficient
2024-04-25 23:59:30 -07:00
chrislu
8948fb51e3
Merge branch 'master' into mq-subscribe
2024-04-25 23:42:19 -07:00
chrislu
63036133ff
minor
2024-04-25 22:02:03 -07:00
chrislu
2a9b0a9332
use parquet levels to construct row
2024-04-25 21:29:45 -07:00
M@
abf01a0eb7
Fixes unlocked read from logBuffer.LastTsNs that is racey. ( #5536 )
2024-04-25 15:46:12 -07:00
chrislu
977e7988e6
toRow conversion with levels info
2024-04-25 09:14:37 -07:00
chrislu
9cb9d27b5b
file rename
2024-04-25 08:56:31 -07:00
chrislu
313097aca0
printout test values
2024-04-25 08:54:57 -07:00
chrislu
827e293010
add visitor
2024-04-25 08:54:41 -07:00
chrislu
138a027a2e
add field types but without value
2024-04-24 23:33:11 -07:00
chrislu
30b5aef62e
record type is also optional
2024-04-24 23:32:35 -07:00
chrislu
ce4597c623
delete the generated file
2024-04-24 23:29:55 -07:00
chrislu
f91484ca07
add some notes
2024-04-24 23:28:20 -07:00
chrislu
f93c4cf48c
adjust test
2024-04-24 23:24:53 -07:00
M@
855607c536
Fixes race with unlocked setter. ( #5532 )
2024-04-24 23:18:28 -07:00
M@
607927da60
Migrates from disintegration/imaging c2019 to cognusion/imaging c2024. ( #5533 )
2024-04-24 23:16:04 -07:00
chrislu
d88c1872ac
works with single level of list
2024-04-24 23:04:47 -07:00
Konstantin Lebedev
2f3fee9bb9
filer read chunk retry if status code 499 ( #5528 )
2024-04-24 06:51:05 -07:00
chrislu
d7e5f6b2a5
fix ordering
2024-04-24 01:32:02 -07:00
skycope
6e4b9181f5
fix "volume.fix.replication" move many replications only to one volumeServer ( #5522 )
2024-04-23 06:33:50 -07:00
chrislu
9ef238b610
has a bug
2024-04-22 00:42:18 -07:00
chrislu
cc1ce3de87
all record fields are optional by default
2024-04-18 23:17:52 -07:00
chrislu
384ed7a381
add test
2024-04-18 22:41:20 -07:00
chrislu
57949f9959
support list type
2024-04-18 22:41:12 -07:00
chrislu
05d18130b6
add schema builder
2024-04-18 22:27:18 -07:00
chrislu
20666bdf81
Merge branch 'master' into mq-subscribe
2024-04-18 21:33:06 -07:00
