Commit Graph

1199 Commits

Author SHA1 Message Date
Mesar Hameed
65484e80ef
Feat: support username/password authentication for etcd filer store s… (#4477)
Feat: support username/password authentication for etcd filer store seaweedfs/seaweedfs#4262

Co-authored-by: Mesar Hameed <mesar.hameed@gmail.com>
2023-05-15 17:45:14 -07:00
chrislu
efef6e94bf use UPSERT for postgres style databases 2023-04-04 11:52:36 -07:00
renweijun
bbd461b2c6
weed mount default EnableACL,Support chmod chown (#4335) 2023-03-25 04:47:53 -07:00
chrislu
5db9fcccd4 refactoring 2023-03-21 23:01:49 -07:00
Stewart Miles
dd71f54c6b
Fix -raftHashicorp and -raftBootstrap flag propagation. (#4309)
`weed server` was not correctly propagating
`-master.raftHashicorp` and `-master.raftBootstrap` flags when
starting the master server.

Related to #4307
2023-03-15 13:03:20 -07:00
lfhy
1976ca9160
add -disk to filer command (#4247)
* add -disk to filer command

* add diskType to filer.grpc

* use filer.disk when filerWebDavOptions.disk is empty

* add filer.disk to weed server command.

---------

Co-authored-by: 三千院羽 <3000y@MacBook-Pro.lan>
2023-02-25 09:48:59 -08:00
Zachary Walters
ef2f741823
Updated the deprecated ioutil dependency (#4239) 2023-02-21 19:47:33 -08:00
Muhammad Hallaj bin Subery
9bd422d2c9
adding support for B2 region (#4177)
Co-authored-by: Muhammad Hallaj bin Subery <hallaj@tuta.io>
2023-02-05 21:24:21 -08:00
chrislu
81fdf3651b grpc connection to filer add sw-client-id header 2023-01-20 01:48:12 -08:00
chrislu
4614e85efa adjust help message 2023-01-15 21:28:36 -08:00
chrislu
469c959e6a add missing options for webdav 2023-01-08 18:03:22 -08:00
zemul
6b4c033431
add mount log (#4101)
* filer.backup use replication.source.filer

* add mount log

* Revert "filer.backup use replication.source.filer"

This reverts commit 07bf6f956c.

* fix

Co-authored-by: zemul <zhouzemiao@ihuman.com>
2023-01-03 00:00:45 -08:00
Chris Lu
d4566d4aaa
more solid weed mount (#4089)
* compare chunks by timestamp

* fix slab clearing error

* fix test compilation

* move oldest chunk to sealed, instead of by fullness

* lock on fh.entryViewCache

* remove verbose logs

* revert slat clearing

* less logs

* less logs

* track write and read by timestamp

* remove useless logic

* add entry lock on file handle release

* use mem chunk only, swap file chunk has problems

* comment out code that maybe used later

* add debug mode to compare data read and write

* more efficient readResolvedChunks with linked list

* small optimization

* fix test compilation

* minor fix on writer

* add SeparateGarbageChunks

* group chunks into sections

* turn off debug mode

* fix tests

* fix tests

* tmp enable swap file chunk

* Revert "tmp enable swap file chunk"

This reverts commit 985137ec47.

* simple refactoring

* simple refactoring

* do not re-use swap file chunk. Sealed chunks should not be re-used.

* comment out debugging facilities

* either mem chunk or swap file chunk is fine now

* remove orderedMutex  as *semaphore.Weighted

not found impactful

* optimize size calculation for changing large files

* optimize performance to avoid going through the long list of chunks

* still problems with swap file chunk

* rename

* tiny optimization

* swap file chunk save only successfully read data

* fix

* enable both mem and swap file chunk

* resolve chunks with range

* rename

* fix chunk interval list

* also change file handle chunk group when adding chunks

* pick in-active chunk with time-decayed counter

* fix compilation

* avoid nil with empty fh.entry

* refactoring

* rename

* rename

* refactor visible intervals to *list.List

* refactor chunkViews to *list.List

* add IntervalList for generic interval list

* change visible interval to use IntervalList in generics

* cahnge chunkViews to *IntervalList[*ChunkView]

* use NewFileChunkSection to create

* rename variables

* refactor

* fix renaming leftover

* renaming

* renaming

* add insert interval

* interval list adds lock

* incrementally add chunks to readers

Fixes:
1. set start and stop offset for the value object
2. clone the value object
3. use pointer instead of copy-by-value when passing to interval.Value
4. use insert interval since adding chunk could be out of order

* fix tests compilation

* fix tests compilation
2023-01-02 23:20:45 -08:00
CommanderRoot
c7c9d22f37
filer.store.mysql: Use utf8mb4 instead of 3 byte UTF8 (#4094) 2023-01-01 05:07:53 -08:00
CommanderRoot
c2280e94cf
filer.store.mysql: Replace deprecated upsert syntax (#4096) 2023-01-01 05:06:57 -08:00
chrislu
f9fb3cd1fc Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-12-19 11:32:00 -08:00
lfhy
038bab3726
add -filer.path to webdav command (#4061) 2022-12-17 20:07:02 -08:00
chrislu
8a40fa8993 more detailed logs 2022-12-17 13:18:35 -08:00
Jiffs Maverick
4b0430e71d
[metrics] Add the ability to control bind ip (#4012) 2022-11-24 10:22:59 -08:00
chrislu
70a4c98b00 refactor filer_pb.Entry and filer.Entry to use GetChunks()
for later locking on reading chunks
2022-11-15 06:33:36 -08:00
Guo Lei
5b905fb2b7
Lazy loading (#3958)
* types packages is imported more than onece

* lazy-loading

* fix bugs

* fix bugs

* fix unit tests

* fix test error

* rename function

* unload ldb after initial startup

* Don't load ldb when starting volume server if ldbtimeout is set.

* remove uncessary unloadldb

* Update weed/command/server.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

* Update weed/command/volume.go

Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>

Co-authored-by: guol-fnst <goul-fnst@fujitsu.com>
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-11-14 00:19:27 -08:00
zemul
cec55feeb0
[filer.backup] add retention_duration (#3941) 2022-11-04 00:34:40 -07:00
Konstantin Lebedev
f9f499b8d5
[fix] add param for ignore error (#3918) 2022-10-28 06:59:39 -07:00
chrislu
5294b6dd4d remove volume server tcp client 2022-10-23 13:15:03 -07:00
chrislu
9a339a9cfb default hasSlowRead to true 2022-10-12 21:15:10 -07:00
famosss
cacc3e883b
volume server:set the default value of "hasSlowRead" to true (#3710)
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit

* volume server:set the default value of "hasSlowRead" to true
2022-10-12 21:13:26 -07:00
Konstantin Lebedev
401315f337
master fix interruption through ctrl+c (#3834) 2022-10-12 07:18:40 -07:00
BakerBunker
b9d8a837af
Fix mount -t weed with extra options (#3807)
fuse: Ignore "-o" command
2022-10-08 11:25:19 -07:00
Konstantin Lebedev
b9933d5589
master server graceful stop (#3797) 2022-10-06 09:30:30 -07:00
Konstantin Lebedev
6fa3d0cc46
fix delete key panic of remote sync dir (#3770) 2022-10-01 02:33:47 -07:00
chrislu
c3487d3da7 better fix for option format 2022-09-30 08:49:38 -07:00
chrislu
ba0db28dbb mount: fix extra option format 2022-09-30 08:44:41 -07:00
chrislu
8e81619d02 mount: accept all extra mount options
fix https://github.com/seaweedfs/seaweedfs/issues/3767
2022-09-30 08:40:37 -07:00
Konstantin Lebedev
301b678147
[volume] Add new volumes to HUP(reload) signal (#3755)
Add new volumes to HUP(reload) signal
2022-09-28 12:44:13 -07:00
Konstantin Lebedev
7ae51d1ec6
fix set master options for single server (#3708)
Co-authored-by: Chris Lu <chrislusf@users.noreply.github.com>
2022-09-27 12:06:35 -07:00
chrislu
10d5b4b32b volume server: rename readBufferSize to readBufferSizeMB 2022-09-17 10:56:28 -07:00
famosss
d949a238b8
volume: add "readBufSize" option to customize read optimization (#3702)
* simplify a bit

* feat: volume: add "readBufSize" option to customize read optimization

* refactor : redbufSIze -> readBufferSize

* simplify a bit

* simplify a bit
2022-09-16 00:30:40 -07:00
chrislu
cf90f76a35 mark "hasSlowRead" as experimental 2022-09-15 23:33:46 -07:00
chrislu
f9e3e7d1c4 Merge branch 'master' of https://github.com/seaweedfs/seaweedfs 2022-09-15 03:11:34 -07:00
chrislu
896a85d6e4 volume: add "hasSlowRead" option to customize read optimization 2022-09-15 03:11:32 -07:00
Ryan Russell
f6d391c9da
docs(command): readability fixes (#3686)
Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-15 01:33:04 -07:00
chrislu
21c0587900 go fmt 2022-09-14 23:06:44 -07:00
Ryan Russell
8efe1db01a
refactor(various): Listner -> Listener readability improvements (#3672)
* refactor(net_timeout): `listner` -> `listener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(s3): `s3ApiLocalListner` -> `s3ApiLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(filer): `localPublicListner` -> `localPublicListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(command): `masterLocalListner` -> `masterLocalListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

* refactor(net_timeout): `ipListner` -> `ipListener`

Signed-off-by: Ryan Russell <git@ryanrussell.org>

Signed-off-by: Ryan Russell <git@ryanrussell.org>
2022-09-14 11:59:55 -07:00
Ryan Russell
19652c1b83
refactor(socket mount): Update socket mount pattern to `/tmp/seaweedf… (#3662) 2022-09-14 09:14:44 -07:00
Ryan Russell
b6a1b84a00
docs: orignial -> original (#3661) 2022-09-14 09:13:59 -07:00
LHHDZ
6d2000e8da
fix 'localFilerSocket' nil pointer (#3654) 2022-09-14 06:33:00 -07:00
Ryan Russell
2c92a9ff74
refactor: DefaultConcurrencyLimit var rename (#3658) 2022-09-14 06:30:32 -07:00
Ryan Russell
60fa26ef45
refactor: concurrentFiles and concurrentChunks var name fix (#3655) 2022-09-14 04:53:04 -07:00
chrislu
b834027c5a refactor 2022-09-11 21:53:15 -07:00
chrislu
2c6b68b40e simplify a bit 2022-09-10 14:18:28 -07:00
Konstantin Lebedev
b64674018a
[sync] override amz storage class, None to delete (#3639)
* override amz storage class, None to delete
https://github.com/seaweedfs/seaweedfs/issues/3636

* use empty string to delete

* without nil check
2022-09-10 14:15:42 -07:00
chrislu
205ecb5d03 skip directories under ".uploads" directory
related to https://github.com/seaweedfs/seaweedfs/issues/3636

skipping all directories under ".uploads" directory.
2022-09-09 22:57:38 -07:00
chrislu
9b084d4c88 purge tcp implementation 2022-09-08 18:03:43 -07:00
Konstantin Lebedev
f4f6855c61
avoid SIGSEGV on localFilerSocket (#3609)
https://github.com/seaweedfs/seaweedfs/issues/3607
2022-09-06 08:20:23 -07:00
bernardx
228b133afa
new 'concurrency' parameter for filer.sync (#3579)
Co-authored-by: XIAOYQ <xiaoyq@eudic.net>
2022-09-02 23:03:23 -07:00
Konstantin Lebedev
8c3040db81
avoid DATA RACE on S3Options.localFilerSocket (#3571)
* avoid DATA RACE on S3Options.localFilerSocket
https://github.com/seaweedfs/seaweedfs/issues/3552

* copy localSocket
2022-09-01 10:33:23 -07:00
chrislu
bcf35876d1 add more logs 2022-08-31 23:16:05 -07:00
chrislu
3d8b8ffced refactor 2022-08-31 00:24:29 -07:00
Konstantin Lebedev
903a06a160
avoid data race access to startDelay (#3548) 2022-08-30 00:02:09 -07:00
chrislu
3f3a1341d8 make CodeQL happy 2022-08-26 17:09:11 -07:00
chrislu
7394f7feee minor: adjust ordering 2022-08-25 00:20:23 -07:00
chrislu
dbf0de4ce1 minor clean up 2022-08-25 00:19:08 -07:00
Patrick Schmidt
7b424a54dc
Add raft server access mutex to avoid races (#3503) 2022-08-24 09:49:05 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398)
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
2022-08-23 23:18:21 -07:00
chrislu
7ea2c24c5d metadata processing use retry instead of retryForEver 2022-08-22 11:25:40 -07:00
chrislu
b4131a3a72 filer.remote.sync: parallelize uploading to remove object storage 2022-08-22 00:01:48 -07:00
chrislu
f6b92b8561 refactor 2022-08-22 00:00:51 -07:00
chrislu
973f6dd162 refactoring 2022-08-20 22:38:15 -07:00
chrislu
6c8822f269 filer.copy: retryable file part upload 2022-08-20 18:59:57 -07:00
chrislu
a3553da7f7 add fileId to UploadWithRetry return result 2022-08-20 18:54:59 -07:00
chrislu
d49d0a9fc2 filer.copy: retryable upload 2022-08-20 18:50:57 -07:00
chrislu
689b4ecdcc remove unused collection and replication from upload result 2022-08-20 18:14:57 -07:00
chrislu
9f50b310a0 remove unused code 2022-08-20 17:51:12 -07:00
chrislu
2b580a7566 also migrate jsonpb 2022-08-17 12:42:03 -07:00
chrislu
eaeb141b09 move proto package 2022-08-17 12:05:07 -07:00
qzh
7fcfaf7bc9
fix(filer.sync): offset may be set to 0 (#3451)
* fix(filer.sync): initializing the offset is related to the path

* fix(filer.sync): the offset maybe to be set to 0.

Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 23:43:52 -07:00
qzh
400f0c3e5d
fix(filer.sync): initializing the offset is related to the path (#3450)
Co-authored-by: zhihao.qu <zhihao.qu@ly.com>
2022-08-15 21:56:47 -07:00
chrislu
67814a5c79 refactor and fix strings.Split 2022-08-07 01:34:32 -07:00
chrislu
1a4bf0dcb5 filer.sync: parallelize the filer.sync 2022-08-07 00:56:15 -07:00
chrislu
0e9478488d filer.sync: fix when excluded paths is empty 2022-08-07 00:55:34 -07:00
LHHDZ
84ec68e11a
Add download speed limit support (#3408) 2022-08-05 01:16:42 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405)
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
2022-08-04 17:35:00 -07:00
Chris Lu
0bb3ba17ae
Merge pull request #3385 from Woellchen/feature/cluster-status-return-healthy
Add healthy indicator for raft status
2022-07-30 11:55:30 -07:00
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu
0423d50106 wip note 2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a moving 2022-07-29 23:39:11 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c clean up 2022-07-28 23:24:38 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd mq broker adds data center and rack 2022-07-28 23:22:07 -07:00
chrislu
87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17 rename to SeaweedMQ 2022-07-28 23:22:06 -07:00
chrislu
ca836568ac Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
duanhongyi
1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu
b7de5c6c43 shell: remove unused filer parameter from shell.toml 2022-07-07 15:01:23 -07:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
guol-fnst
74c9f39d48 remove hdfs, add tikv for full version update 2022-06-28 12:35:07 +08:00
chrislu
4a65159250 fix reading time 2022-06-27 12:40:47 -07:00
yulai.li
af23e63e3f Improve filer command help, add supported filer store list 2022-06-27 12:09:16 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
chrislu
492da3dbce master: put metadata under instance specific folder 2022-06-20 19:04:49 -07:00
chrislu
2295766e2c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page 2022-06-20 22:36:07 +02:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5 remove "weed" from UsageLine, or weed will not show usage of weed update 2022-06-17 12:47:04 +08:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4 fix path bug on windows
fix usage
2022-06-16 17:24:56 +08:00
guol-fnst
995e221e97 remove -output, add -dir,-name for more flexibility 2022-06-16 16:07:32 +08:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
guol-fnst
4bc8792643 support specific version of weed 2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29 remove unused function
remove log when weed is up to date

add check whether weed is full version
2022-06-16 11:40:20 +08:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37 Fix: provide filer.ui.deleteDir options for server command 2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update 2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555 fix minor bug 2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6 fix(filer.sync): modify clientName format : from -> to 2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87 Change ui.deleteDir default to true 2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b feat(filer.sync): add metricsServer in filer.sync.
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
2022-06-15 11:33:18 +08:00
yulai.li
28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea feat(filer.sync): add offset to path. 2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0 add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate" 2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372 add update feature 2022-06-14 19:01:57 +08:00
zhihao.qu
cd5cca36a4 feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges 2022-06-09 10:53:19 +08:00
chrislu
8a1166c021 auto remove previous leftover socket 2022-06-08 00:32:09 -07:00
chrislu
633a4aed18 filer: fix customizable local socket file name
fix https://github.com/chrislusf/seaweedfs/issues/3147
2022-06-08 00:15:28 -07:00
chrislu
78be3b0f6c fix compilation 2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
ningfd
f32142f6f5 add disableXAttr in mount option 2022-06-06 14:09:01 +08:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
490e0850bf filer.meta.tail add one example of using untilTimeAgo 2022-05-30 20:24:59 -07:00
chrislu
968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
liubaojiang
076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around 2022-05-16 22:11:33 +08:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
chrislu
735038b2c1 backup do not need to use preallocation
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb
# Conflicts:
#	go.mod
#	go.sum
2022-05-03 00:13:57 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
chrislu
192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00