Commit Graph

1116 Commits

Author SHA1 Message Date
Patrick Schmidt
1a4a36d510
Add healthy indicator for raft status 2022-07-30 19:34:26 +02:00
chrislu
0423d50106 wip note 2022-07-30 09:36:27 -07:00
chrislu
0f50fcb96a moving 2022-07-29 23:39:11 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
chrislu
74f60f246f dynamically connect to a filer 2022-07-28 23:24:38 -07:00
chrislu
94b8c42b2c clean up 2022-07-28 23:24:38 -07:00
chrislu
68065128b8 add dc and rack 2022-07-28 23:22:51 -07:00
chrislu
8d31e73ffd mq broker adds data center and rack 2022-07-28 23:22:07 -07:00
chrislu
87ccef72db cluster.ps can list brokers 2022-07-28 23:22:07 -07:00
chrislu
21b6b07dd8 renaming 2022-07-28 23:22:06 -07:00
chrislu
8ca7d1ef17 rename to SeaweedMQ 2022-07-28 23:22:06 -07:00
chrislu
ca836568ac Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-07-27 12:12:43 -07:00
chrislu
3828b8ce87 "github.com/chrislusf/raft" => "github.com/seaweedfs/raft" 2022-07-27 12:12:40 -07:00
Konstantin Lebedev
7e09a548a6 exclude directories to sync on filer 2022-07-27 19:22:57 +05:00
chrislu
64f3d6fb6e metadata subscription uses client epoch 2022-07-23 10:50:28 -07:00
duanhongyi
1ceab96aba filer tikv support tls 2022-07-08 14:23:06 +08:00
chrislu
b7de5c6c43 shell: remove unused filer parameter from shell.toml 2022-07-07 15:01:23 -07:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
guol-fnst
74c9f39d48 remove hdfs, add tikv for full version update 2022-06-28 12:35:07 +08:00
chrislu
4a65159250 fix reading time 2022-06-27 12:40:47 -07:00
yulai.li
af23e63e3f Improve filer command help, add supported filer store list 2022-06-27 12:09:16 +08:00
yulai.li
46e0b629e5 Update tikv client version and add one PC support 2022-06-26 22:43:37 +08:00
chrislu
492da3dbce master: put metadata under instance specific folder 2022-06-20 19:04:49 -07:00
chrislu
2295766e2c Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-06-20 14:16:18 -07:00
chrislu
1d0c53ea56 remote storage: stop supporting hdfs as a remote storage 2022-06-20 14:15:59 -07:00
Stephan
1eb7826909
Fix link to osxfuse github page 2022-06-20 22:36:07 +02:00
Chris Lu
d28f8bb88f
Merge pull request #3192 from guol-fnst/weed_update
remove "weed" from UsageLine, or weed will not show usage of weed update
2022-06-16 22:58:29 -07:00
guol-fnst
c7c20881f5 remove "weed" from UsageLine, or weed will not show usage of weed update 2022-06-17 12:47:04 +08:00
Chris Lu
5f8d6b5f13
Merge pull request #3184 from guol-fnst/weed_update
Weed update
2022-06-16 10:02:45 -07:00
guol-fnst
cd078471c4 fix path bug on windows
fix usage
2022-06-16 17:24:56 +08:00
guol-fnst
995e221e97 remove -output, add -dir,-name for more flexibility 2022-06-16 16:07:32 +08:00
Chris Lu
82f3bcc65e
Merge pull request #3081 from paochiang/volume_upload_limit_fix
exclude replication from the concurrentUploadLimitMB
2022-06-15 23:16:44 -07:00
guol-fnst
4bc8792643 support specific version of weed 2022-06-16 11:40:23 +08:00
guol-fnst
1dbb925d29 remove unused function
remove log when weed is up to date

add check whether weed is full version
2022-06-16 11:40:20 +08:00
Chris Lu
4eeeb5f50d
Merge pull request #3180 from blacktear23/issue-3149 2022-06-15 00:55:20 -07:00
yulai.li
358c0c0b37 Fix: provide filer.ui.deleteDir options for server command 2022-06-15 15:53:31 +08:00
Guo Lei
3df84cabf6
Merge branch 'chrislusf:master' into weed_update 2022-06-15 15:28:40 +08:00
guol-fnst
695e8c7555 fix minor bug 2022-06-15 15:26:04 +08:00
zhihao.qu
4d0d1848c6 fix(filer.sync): modify clientName format : from -> to 2022-06-15 13:33:20 +08:00
yulai.li
b90d213a87 Change ui.deleteDir default to true 2022-06-15 11:37:37 +08:00
zhihao.qu
42d04c581b feat(filer.sync): add metricsServer in filer.sync.
Metrics include:
(1) the offset of the filer.sync
(2) the last send timestamp of the filer subscription
2022-06-15 11:33:18 +08:00
yulai.li
28cda854eb Add filer command line parameter to let Filer UI show/hide directory delete button 2022-06-15 00:30:49 +08:00
zhihao.qu
14d82c3dea feat(filer.sync): add offset to path. 2022-06-14 19:46:02 +08:00
guol-fnst
78fc9cdfa0 add "copied from https://github.com/restic/restic/tree/master/internal/selfupdate" 2022-06-14 19:10:54 +08:00
guol-fnst
322dbcf372 add update feature 2022-06-14 19:01:57 +08:00
zhihao.qu
cd5cca36a4 feat(filer.sync): add fromTsMs. Extract signature from doSubscribeFilerMetaChanges 2022-06-09 10:53:19 +08:00
chrislu
8a1166c021 auto remove previous leftover socket 2022-06-08 00:32:09 -07:00
chrislu
633a4aed18 filer: fix customizable local socket file name
fix https://github.com/chrislusf/seaweedfs/issues/3147
2022-06-08 00:15:28 -07:00
chrislu
78be3b0f6c fix compilation 2022-06-06 10:09:21 -07:00
chrislu
4fd5f96598 filer: remove replication, collection, disk_type info from entry metadata
these metadata can change and are not used
2022-06-06 00:39:35 -07:00
ningfd
f32142f6f5 add disableXAttr in mount option 2022-06-06 14:09:01 +08:00
creeew
02ae102731 fix filer.sync missing source srv uploaded files to target when target down 2022-06-02 01:28:47 +08:00
chrislu
490e0850bf filer.meta.tail add one example of using untilTimeAgo 2022-05-30 20:24:59 -07:00
chrislu
968ca95b49 filer.meta.tail: support untilTimeAgo for a range 2022-05-30 16:16:23 -07:00
chrislu
a2b101a737 subscribe metadata between a range 2022-05-30 15:04:19 -07:00
liubaojiang
076e48a676 add inflight upload data wait timeout 2022-05-21 10:38:08 +08:00
Chris Lu
8efe55f981
Merge pull request #3059 from guol-fnst/avoid_dup_vol
avoid loading duplicated volume directory
2022-05-17 23:48:57 -07:00
guol-fnst
076595fbdd just exit in case of duplicated volume directories were loaded 2022-05-17 15:41:49 +08:00
Nat Makarevitch
b72f0634d7
minor (typos...), done while reading around 2022-05-16 22:11:33 +08:00
chrislu
29198720f2 s3: add grpc server to accept configuration changes 2022-05-15 00:43:37 -07:00
chrislu
735038b2c1 backup do not need to use preallocation
fix https://github.com/chrislusf/seaweedfs/issues/3044
2022-05-13 13:46:52 -07:00
Konstantin Lebedev
f127b326bf add options to scaffold 2022-05-03 22:54:31 +05:00
Konstantin Lebedev
d8925b4e83 Merge branch 'new_master' into ydb
# Conflicts:
#	go.mod
#	go.sum
2022-05-03 00:13:57 +05:00
chrislu
94635e9b5c filer: add filer group 2022-05-01 21:59:16 -07:00
Konstantin Lebedev
50c4f62ed4 ydb-go-sdk move to v3 2022-05-02 02:07:47 +05:00
chrislu
192983b464 s3 backend support customizing storage class 2022-04-30 17:36:40 -07:00
chrislu
3885374edf conditionally build elastic, gocdk to reduce binary size 2022-04-21 01:10:46 -07:00
chrislu
6994e88192 fix typo 2022-04-21 00:02:18 -07:00
a
1d6a9e66b6 Merge branch 'master' into a 2022-04-20 14:01:42 -07:00
chrislu
a69f0b62c2 start filer local socket only on non windows
fix https://github.com/chrislusf/seaweedfs/issues/2930
2022-04-18 00:08:31 -07:00
leyou240
89eb87c1d1
Merge branch 'master' into slices.SortFunc 2022-04-18 10:39:29 +08:00
justin
3551ca2fcf enhancement: replace sort.Slice with slices.SortFunc to reduce reflection 2022-04-18 10:35:43 +08:00
chrislu
1384529eb7 Fix filer.backup deletes files in backup folder in incremental mode
fix https://github.com/chrislusf/seaweedfs/issues/2919
2022-04-14 13:35:01 -07:00
a
846858fb43 merge master 2022-04-12 14:42:29 -07:00
a
41d396edc4 Merge branch 'master' into a 2022-04-07 08:22:59 -07:00
Konstantin Lebedev
f5246b748d Merge branch 'new_master' into hashicorp_raft
# Conflicts:
#	weed/pb/master_pb/master.pb.go
2022-04-07 18:50:27 +05:00
chrislu
86ed27f602 mount: remove leftover socket file 2022-04-07 00:33:13 -07:00
eddy-gfx
ec53eec94f
Merge branch 'master' into a 2022-04-06 18:45:55 -05:00
Konstantin Lebedev
17c6e8e39f Merge branch 'new_master' into hashicorp_raft
# Conflicts:
#	go.mod
#	go.sum
2022-04-05 13:29:46 +05:00
Konstantin Lebedev
622297f1a7 add stats raft handler 2022-04-04 19:16:06 +05:00
Konstantin Lebedev
14dd971890 hashicorp raft with state machine 2022-04-04 17:51:51 +05:00
Konstantin Lebedev
c514710b7b initial add hashicorp raft 2022-04-04 13:50:56 +05:00
chrislu
6a2bcd03aa configure mount quota 2022-04-02 21:34:26 -07:00
chrislu
958f880b70 mount: add grpc method to adjust quota 2022-04-02 15:14:37 -07:00
chrislu
daca2d22a5 use original server address string as map key 2022-04-01 17:34:42 -07:00
a
7e92517571 change user and pass to username and password 2022-04-01 14:09:25 -05:00
Konstantin Lebedev
b6a2b43efe allowDeleteBucketNotEmpty 2022-03-30 22:46:13 +05:00
a
bc603e534f diff 2022-03-28 16:50:28 +00:00
chrislu
21e0898631 refactor: change masters from a slice to a map 2022-03-26 13:33:17 -07:00
chrislu
4ba7127ab1 refactor 2022-03-26 13:13:19 -07:00
Konstantin Lebedev
849a2c1f68 fix cmdFiler 2022-03-23 14:30:46 +05:00
Konstantin Lebedev
105bcfdbd4 set iam ip from filer 2022-03-23 13:45:51 +05:00
elee
423ce57cde prefix search, bucket implemented 2022-03-17 21:12:25 -05:00
Berck Nash
7ee38fa3a4 The fixes for https://github.com/chrislusf/seaweedfs/issues/1937 had a few problems:
(1) The help file says that in the absence of a ipBind being specified, that it will bind to the "ip" specified.  Instead, it bound to localhost which broke the default configuration.  This change implements the documented behavior instead.

(2) The new IAM filer ip address has no default. This instantiates it to the same as the filer IP.  I'm not sure if there should be a corresponding iam.ip or iam.ipBind option added to the filer command?
2022-03-17 15:30:23 -06:00
elee
921535001a arangodb adapter 2022-03-17 04:49:26 -05:00
Berck Nash
9b14f0c81a Add mTLS support for both master and volume http server. 2022-03-16 09:52:17 -06:00
chrislu
3639cad69c master, filer, s3: also listen to "localhost" in addition to specific ip address
related to https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-15 22:28:18 -07:00
chrislu
3a6eb8ca5f default bind to one ip address
fix https://github.com/chrislusf/seaweedfs/issues/1937
2022-03-11 14:02:39 -08:00
banjiaojuhao
b9ff7723dd [bugfix] filer: nil pointer dereference 2022-03-07 23:26:25 +08:00
chrislu
da3d330616 s3 and filer transport using unix domain socket instead of tcp 2022-03-07 02:00:14 -08:00
chrislu
c7e8ac18f0 mount: quota for one mounted collection
related to https://github.com/seaweedfs/seaweedfs-csi-driver/issues/48
2022-03-06 02:44:40 -08:00
chrislu
f51e20028a mount: avoid comma in mount options
fix https://github.com/chrislusf/seaweedfs/issues/2719
2022-03-03 03:42:29 -08:00
chrislu
18543c6e8b minor 2022-02-27 23:11:09 -08:00
chrislu
c3792c8352 remove dead code 2022-02-27 03:03:19 -08:00
chrislu
aa9eef81e6 retire mount v1 2022-02-27 02:57:27 -08:00
chrislu
6e1ab97988 use debug option to see operations 2022-02-27 02:02:30 -08:00
chrislu
2112d99140 mount2: add back readonly mode 2022-02-27 01:13:32 -08:00
chrislu
fc7a4957ea fix mount2 options 2022-02-25 21:22:44 -08:00
chrislu
202a29d014 refactoring 2022-02-25 01:17:26 -08:00
Tuan Vuong
d2ec62656d initialize master address in iam options 2022-02-23 12:01:54 +07:00
banjiaojuhao
b5ec346700 FilerStore: add redis_lua 2022-02-15 20:54:57 +08:00
chrislu
4e72863ba5 mount2 add debug mode 2022-02-15 00:26:30 -08:00
chrislu
4e181db21a mount: default disable cache
* Prevent cases as https://github.com/seaweedfs/seaweedfs-csi-driver/issues/43
* Improve read write benchmarks
* Improve AI training performance. Most of the files are just read once.
2022-02-14 20:42:33 -08:00
chrislu
377bf31445 Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-02-14 13:48:51 -08:00
chrislu
e8420aaed7 fix building for windows freebsd 2022-02-14 13:48:48 -08:00
Chris Lu
d3ee621fce
Merge pull request #2661 from garenchan/ck-dev1 2022-02-14 10:08:37 -08:00
Konstantin Lebedev
526094d2da StopTimeout 30 sec 2022-02-14 21:42:27 +05:00
Konstantin Lebedev
275e9a4e86 reduce to default http server KillTimeout and StopTimeout 2022-02-14 21:38:24 +05:00
garenchan
bd032eabe7 [UPDATE] Make heartbeat interval and election timeout of masters configurable. 2022-02-14 21:09:07 +08:00
chrislu
05724a68d4 skip other OS 2022-02-14 02:59:51 -08:00
chrislu
b9c2bff931 clean up 2022-02-14 02:14:26 -08:00
chrislu
dbeeda8123 listen for metadata updates 2022-02-14 01:09:31 -08:00
chrislu
180445f5a8 change to use fuse file system 2022-02-11 21:35:09 -08:00
chrislu
f87da798a4 to be re-written following fuse virtual file system 2022-02-11 03:09:30 -08:00
chrislu
b6143de52a mount with name 2022-02-10 22:43:55 -08:00
chrislu
7a0c35674c clean up previously mounted folder 2022-02-10 20:46:53 -08:00
chrislu
c3f9d9fa2e initial setup 2022-02-10 20:32:13 -08:00
chrislu
85c1615b43 filer read empty file may cause OOM in some cases
fix https://github.com/chrislusf/seaweedfs/issues/2641
2022-02-07 23:08:54 -08:00
chrislu
affe3c2c12 change to util.WriteFile 2022-02-04 21:32:27 -08:00
chrislu
84c9bc4389 edge case: old entry was not replicated to remote storage 2022-01-30 20:23:24 -08:00
Chris Lu
7c66f3b5fb
Merge pull request #2602 from kmlebedev/master_metrics
master metricsHttpPort
2022-01-20 09:26:25 -08:00
chrislu
b3e526ba95 url should be always using forward slash 2022-01-19 22:16:26 -08:00
Konstantin Lebedev
77c98b657e master metricsHttpPort 2022-01-19 21:43:22 +05:00
guol-fnst
da9540e666 add gocql timeout setting 2022-01-18 15:21:13 +08:00
chrislu
fc0628c038 working 2022-01-17 01:53:56 -08:00
chrislu
2bfeb5d1c8 add filer to iam option 2022-01-15 03:37:52 -08:00
chrislu
b17c426e99 weed server: optionally start IAM service
related to https://github.com/chrislusf/seaweedfs/issues/2560
2022-01-13 22:49:49 -08:00
chrislu
8907e6a40a add more help messages 2022-01-13 13:03:04 -08:00
chrislu
826a7b307e master: remove hard coded filer settings in master.toml
fix https://github.com/chrislusf/seaweedfs/issues/2529
2022-01-12 01:11:25 -08:00
Kyle Sanderson
9e012001be
filer.copy: don't crash when volume creation fails
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7482 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x2)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1d58247]

goroutine 7480 [running]:
github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks.func1(0x0)
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:488 +0x2a7
created by github.com/chrislusf/seaweedfs/weed/command.(*FileCopyWorker).uploadFileInChunks
        /go/src/github.com/chrislusf/seaweedfs/weed/command/filer_copy.go:455 +0x225
2022-01-11 22:22:39 -08:00
chrislu
1a7d5b5b5e Merge branch 'master' of https://github.com/chrislusf/seaweedfs 2022-01-11 12:24:56 -08:00
chrislu
41daecfdca Update mount_std.go 2022-01-11 12:23:12 -08:00
Chris Lu
abe5da7d2c
Merge pull request #2575 from Radtoo/fix_paths2
Fix paths2
2022-01-11 12:04:30 -08:00
chrislu
6a12520a96 fix logging 2022-01-10 01:00:11 -08:00
Radtoo
389002f195 Using positional arguments rather than option flag to enable better shell usage 2022-01-08 16:52:12 +01:00
Radtoo
fba1efb77a Now works with a single file too
Parsing removed from doFixOneVolume

Needle init removed from runFix
2022-01-08 16:31:53 +01:00
chrislu
110d5a5233 support fixing a collection of volumes, or volumes under one directory 2022-01-07 14:52:16 -08:00
Chris Lu
42c849e0df
Merge branch 'master' into metadata_follow_with_client_id 2022-01-02 01:07:30 -08:00
Chris Lu
9b94177380
Merge pull request #2543 from skurfuerst/seaweedfs-158
FEATURE: add JWT to HTTP endpoints of Filer and use them in S3 Client
2022-01-01 22:34:13 -08:00
Sebastian Kurfuerst
1cd3b6b4e1 BUGFIX: security.toml contained wrong keys 2021-12-31 22:05:41 +01:00