mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-12-26 02:37:55 +08:00
Code Issues Packages Projects Releases Wiki Activity
10,971 Commits 32 Branches 296 Tags 153 MiB
4f2bdebe49
Commit Graph

791 Commits

Author SHA1 Message Date
Eugeniy E. Mikhailov
dab0bb8097
Feature limit caching to prescribed number of bytes per file (#6009) 
* feature: we can check if a fileId is already in the cache

We using this to protect cache from adding the same needle to
the cache over and over.

* fuse mount: Do not start dowloader if needle is already in the cache

* added maxFilePartSizeInCache property to ChunkCache

If file very large only first maxFilePartSizeInCache bytes
are going to be put to the cache (subject to the needle size
constrains).

* feature: for large files put in cache no more than prescribed number of bytes

Before this patch only the first needle of a large file was intended for
caching. This patch uses maximum prescribed amount of bytes to be put in
cache. This allows to bypass default 2MB maximum for a file part stored
in the cache.

* added dummy mock methods to satisfy interfaces of ChunkCache
 2024-09-11 21:09:20 -07:00
chrislu
f23015ac5b weed mount caching 2024-09-10 10:30:05 -07:00
chrislu
93decd1114 remove unused 2024-09-10 10:29:25 -07:00
chrislu
69fcdd0840 adjust logging 2024-09-10 10:28:49 -07:00
zouyixiong
d8e18a8eb7
bug fixed to filer store elastic deleteEntry (#5988) 2024-09-09 02:15:25 -07:00
chrislu
18afdb15b6 Revert "weed mount, weed dav add option to force cache" 
This reverts commit 7367b976b0.
 2024-09-04 01:38:29 -07:00
chrislu
7367b976b0 weed mount, weed dav add option to force cache 2024-09-04 01:19:14 -07:00
chrislu
d49ecde535 rename functions 2024-08-01 23:27:23 -07:00
chrislu
4fee496b49 conditional delete 2024-08-01 13:00:54 -07:00
wyang
0581ce6096
fix delete chunk failed if volumeSever specified grpc.port (#5820) 
Co-authored-by: Yang Wang <yangwang@weride.ai>
 2024-07-26 11:14:19 -07:00
chrislu
aed2a18223 get, set and add path conf 2024-07-20 11:27:11 -07:00
chrislu
a8fa78b892 refactoring 2024-07-17 23:14:36 -07:00
vadimartynov
86d92a42b4
Added tls for http clients (#5766) 
* Added global http client

* Added Do func for global http client

* Changed the code to use the global http client

* Fix http client in volume uploader

* Fixed pkg name

* Fixed http util funcs

* Fixed http client for bench_filer_upload

* Fixed http client for stress_filer_upload

* Fixed http client for filer_server_handlers_proxy

* Fixed http client for command_fs_merge_volumes

* Fixed http client for command_fs_merge_volumes and command_volume_fsck

* Fixed http client for s3api_server

* Added init global client for main funcs

* Rename global_client to client

* Changed:
- fixed NewHttpClient;
- added CheckIsHttpsClientEnabled func
- updated security.toml in scaffold

* Reduce the visibility of some functions in the util/http/client pkg

* Added the loadSecurityConfig function

* Use util.LoadSecurityConfiguration() in NewHttpClient func
 2024-07-16 23:14:09 -07:00
wusong
ce61a66b65
Fix mmap write fail (#5791) 
* [mount] fix GetAttr blocks count

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

* [mount] fix mmap read

Signed-off-by: wang wusong <wangwusong@virtaitech.com>

---------

Signed-off-by: wang wusong <wangwusong@virtaitech.com>
Co-authored-by: wang wusong <wangwusong@virtaitech.com>
 2024-07-16 07:59:55 -07:00
chrislu
4e7d8eb3f1 small refactoring 2024-07-11 14:50:11 -07:00
chrislu
2addcd1623 fix test 2024-07-11 07:53:02 -07:00
chrislu
590397229e fix heap init 2024-07-11 00:26:18 -07:00
chrislu
06e3dd306f clean up 2024-07-11 00:19:39 -07:00
chrislu
c59517e284 upgrade to github.com/redis/go-redis/v9 
fix https://github.com/seaweedfs/seaweedfs/issues/4532
 2024-07-11 00:15:19 -07:00
chrislu
33b11f92bc Squashed commit of the following: 
commit 465b47b8f72668dfc12eaa72befb232513c9e1db
Merge: f39272b54 9c440d472
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:56:51 2024 -0700

    Merge branch 'master' into merged-persisted-logs

commit f39272b5416bdd4d1714ce3e99aded0fe62e7d89
Author: chrislu <chris.lu@gmail.com>
Date:   Wed Jul 10 23:49:26 2024 -0700

    merge meta logs from multiple files
 2024-07-11 00:04:24 -07:00
chrislu
d29fccc521 skip checking if it is the root directory 2024-07-07 23:44:36 -07:00
chrislu
0fdf7eca48 avoid possible nil case during other error cases 2024-07-01 00:54:37 -07:00
chrislu
d8dfadb617 ParallelProcessDirectoryStructure 
need to use this for simpler parallel processing
 2024-06-28 15:02:46 -07:00
chrislu
c030cb3ce9 bootstrap filer from one peer 2024-06-28 14:57:20 -07:00
chrislu
cb67137a03 optionally open the leveldb in readonly mode 2024-06-25 21:07:04 -07:00
chrislu
9ac1023362 check bucket name in sql backend 
fix https://github.com/seaweedfs/seaweedfs/issues/5710
 2024-06-25 08:54:19 -07:00
chrislu
464611f614 optionally skip deleting file chunks 2024-06-15 11:39:48 -07:00
chrislu
843e778875 refactor 2024-06-15 09:41:08 -07:00
chrislu
677cfb8ad1 refactor 2024-06-15 09:39:49 -07:00
vadimartynov
8aae82dd71
Added context for the MasterClient's methods to avoid endless loops (#5628) 
* Added context for the MasterClient's methods to avoid endless loops

* Returned WithClient function. Added WithClientCustomGetMaster function

* Hid unused ctx arguments

* Using a common context for the KeepConnectedToMaster and WaitUntilConnected functions

* Changed the context termination check in the tryConnectToMaster function

* Added a child context to the tryConnectToMaster function

* Added a common context for KeepConnectedToMaster and WaitUntilConnected functions in benchmark
 2024-06-14 11:40:34 -07:00
chrislu
dbfbabac55 simplify 2024-06-09 12:14:10 -07:00
dependabot[bot]
722a14ef57
chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ from 0.2.0 to 0.4.2 (#5621) 
* chore(deps): bump github.com/ydb-platform/ydb-go-sdk-auth-environ

Bumps [github.com/ydb-platform/ydb-go-sdk-auth-environ](https://github.com/ydb-platform/ydb-go-sdk-auth-environ) from 0.2.0 to 0.4.2.
- [Changelog](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ydb-platform/ydb-go-sdk-auth-environ/compare/v0.2.0...v0.4.2)

---
updated-dependencies:
- dependency-name: github.com/ydb-platform/ydb-go-sdk-auth-environ
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix compilation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2024-05-27 12:36:00 -07:00
sb
a3a00d9499
Feature/mongodb security (#5602) 2024-05-18 12:19:10 -07:00
sb
3fae87632f
add filer configuration options for username/password and tls (#5601) 2024-05-18 09:02:03 -07:00
chrislu
f80fb1c02e Merge branch 'master' into mq-subscribe 2024-04-29 21:57:31 -07:00
dependabot[bot]
36a1cf0361
chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1 (#5552) 
* chore(deps): bump github.com/viant/ptrie from 0.3.1 to 1.0.1

Bumps [github.com/viant/ptrie](https://github.com/viant/ptrie) from 0.3.1 to 1.0.1.
- [Release notes](https://github.com/viant/ptrie/releases)
- [Changelog](https://github.com/viant/ptrie/blob/master/CHANGELOG.md)
- [Commits](https://github.com/viant/ptrie/compare/v0.3.1...v1.0.1)

---
updated-dependencies:
- dependency-name: github.com/viant/ptrie
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix compilation

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
Co-authored-by: chrislu <chris.lu@gmail.com>
 2024-04-29 18:42:31 -07:00
chrislu
5e8d788311 Merge branch 'master' into mq-subscribe 2024-04-29 06:44:03 -07:00
Nikita Borzykh
0d04264494
Move file deletion logging to v2 verbisity (#5538) 2024-04-26 06:41:06 -07:00
chrislu
cc1c69f312 Merge branch 'master' into mq-subscribe 2024-04-08 11:03:14 -07:00
Henco Appel
5c8e6014ba
fix: filer authenticate with with volume server (#5480) 2024-04-08 07:27:00 -07:00
chrislu
ccdd9cd8de refactor 2024-04-02 16:25:43 -07:00
Nikita Borzykh
829c0da381
fix: adjust condition in prefixFilterEntries prevent infinite loop (#5440) 2024-03-30 14:47:00 -07:00
chrislu
36fec34c47 print only adapted url 
fix https://github.com/seaweedfs/seaweedfs/issues/5424
 2024-03-25 12:50:43 -07:00
Nikita Borzykh
9ca871af53
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries in etcd meta storage backend (#5416) 
Move ListDirectoryEntries logic to ListDirectoryPrefixedEntries
 2024-03-24 14:08:54 -07:00
Nikita Borzykh
1f3742850d
Add TLS support, initial etcd connection check, fix key_prefix handling for etcd backend meta storage (#5403) 2024-03-24 08:36:21 -07:00
chrislu
56ce040ab6 log errors 2024-03-15 21:05:40 -07:00
chrislu
6f615c62ae Merge branch 'master' into mq-subscribe 2024-03-12 09:19:06 -07:00
chrislu
604091a480 use stopChan to close previous filer peer meta subscription instances 2024-03-12 09:18:54 -07:00
chrislu
1efa502dbb go fmt 2024-03-12 09:17:20 -07:00
chrislu
34f2b600ac each log function adds a "done" return parameter 2024-03-07 10:50:09 -08:00