Commit Graph

125 Commits

Author SHA1 Message Date
chrislu
cc0c8c5f81 simplify 2022-08-27 00:21:57 -07:00
chrislu
87b70a6809 clean up 2022-08-27 00:09:04 -07:00
chrislu
c839ce1b19 s3 sink use s3 upload manager
fix https://github.com/seaweedfs/seaweedfs/issues/3531
2022-08-26 23:47:12 -07:00
askeipx
2e78a522ab
remove old raft servers if they don't answer to pings for too long (#3398)
* remove old raft servers if they don't answer to pings for too long

add ping durations as options

rename ping fields

fix some todos

get masters through masterclient

raft remove server from leader

use raft servers to ping them

CheckMastersAlive for hashicorp raft only

* prepare blocking ping

* pass waitForReady as param

* pass waitForReady through all functions

* waitForReady works

* refactor

* remove unneeded params

* rollback unneeded changes

* fix
2022-08-23 23:18:21 -07:00
chrislu
4081d50607 filer sink: retryable data chunk uploading 2022-08-20 19:09:15 -07:00
chrislu
e3f40d538d cleaner code 2022-08-20 17:51:30 -07:00
chrislu
11f99836c3 filer.backup: backup small files if the file is saved in filer (saveToFilerLimit > 0)
fix https://github.com/seaweedfs/seaweedfs/issues/3468
2022-08-19 23:00:56 -07:00
Konstantin Lebedev
4d08393b7c
filer prefer volume server in same data center (#3405)
* initial prefer same data center
https://github.com/seaweedfs/seaweedfs/issues/3404

* GetDataCenter

* prefer same data center for ReplicationSource

* GetDataCenterId

* remove glog
2022-08-04 17:35:00 -07:00
chrislu
26dbc6c905 move to https://github.com/seaweedfs/seaweedfs 2022-07-29 00:17:28 -07:00
Konstantin Lebedev
bcbdc4cb37 use const multipart uploads folder
avoid error bucket NotEmpty if multipart uploads folder exist
2022-06-29 16:21:16 +05:00
chrislu
bff1ccc1de fix compilation 2022-05-11 00:52:15 -07:00
chrislu
139e039c44 filer.sync: pass attributes for mount
fix https://github.com/chrislusf/seaweedfs/issues/3012
2022-05-06 03:54:12 -07:00
elee
881a0fe806 ensure compatibility 2022-02-27 04:50:59 -06:00
elee
954ad98e0d set canned acl on replication create 2022-02-27 04:49:31 -06:00
chrislu
9405eaefdb filer.sync: fix replicating partially updated file
Run two servers with volumes and fillers:
server -dir=Server1alpha -master.port=11000 -filer -filer.port=11001 -volume.port=11002
server -dir=Server1sigma -master.port=11006 -filer -filer.port=11007 -volume.port=11008

Run Active-Passive filler.sync:
filer.sync -a localhost:11007 -b localhost:11001 -isActivePassive

Upload file to 11007 port:
curl -F file=@/Desktop/9.xml "http://localhost:11007/testFacebook/"

If we request a file on two servers now, everything will be correct, even if we add data to the file and upload it again:
curl "http://localhost:11007/testFacebook/9.xml"
EQUALS
curl "http://localhost:11001/testFacebook/9.xml"

However, if we change the already existing data in the file (for example, we change the first line in the file, reducing its length), then this file on the second server will not be valid and will not be equivalent to the first file

Снимок экрана 2022-02-07 в 14 21 11

This problem occurs on line 202 in the filer_sink.go file. In particular, this is due to incorrect mapping of chunk names in the DoMinusChunks function. The names of deletedChunks do not match the chunks of existingEntry.Chunks, since the first chunks come from another server and have a different addressing (name) compared to the addressing on the server where the file is being overwritten.

Deleted chunks are not actually deleted on the server to which the file is replicated.
2022-02-07 03:46:28 -08:00
chrislu
9f9ef1340c use streaming mode for long poll grpc calls
streaming mode would create separate grpc connections for each call.
this is to ensure the long poll connections are properly closed.
2021-12-26 00:15:03 -08:00
Chris Lu
e5fc35ed0c change server address from string to a type 2021-09-12 22:47:52 -07:00
Chris Lu
6923af7280 refactoring 2021-09-06 16:20:49 -07:00
Chris Lu
7ce97b59d8 go fmt 2021-09-01 02:45:42 -07:00
Chris Lu
c08ac536ed cloud drive: add support for Wasabi
* disable md5, sha256 checking to avoid reading one chunk twice
* single threaded upload to avoid chunk swapping (to be enhanced later)
2021-08-25 17:34:29 -07:00
Chris Lu
7c39a18ba5 update azure library 2021-08-24 00:32:35 -07:00
Chris Lu
00c4e06caa cloud drive: s3 configurable force path style 2021-08-23 03:30:41 -07:00
Chris Lu
f0cc130849 do not force path style for better compatibility 2021-08-23 03:09:41 -07:00
Chris Lu
99b599aa8a remote.mount 2021-07-26 22:53:44 -07:00
Chris Lu
7359193e97 go fmt 2021-07-21 14:38:12 -07:00
Chris Lu
7ab389e7ec optimization: improve random range query for large files 2021-07-19 23:07:22 -07:00
Chris Lu
450222dd64 add remote to filer.Entry and filer_pb entry, add RemoteConf 2021-07-19 02:47:27 -07:00
Chris Lu
45bffc92a8 filer.backup: fix cloud sinks when updating entry 2021-05-26 14:53:11 -07:00
Chris Lu
1a70cb9b63 local sink: write and update files
fix issues with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-26 14:42:21 -07:00
Chris Lu
dce1f02c9e filer.backup: backup to local directory optionally is incremental
fixed one issue with https://github.com/chrislusf/seaweedfs/issues/2084
2021-05-25 17:19:20 -07:00
Chris Lu
8f8738867f add retry to assign volume
fix https://github.com/chrislusf/seaweedfs/issues/2056
2021-05-07 07:29:26 -07:00
Chris Lu
540441fd38 go fmt 2021-02-28 20:34:14 -08:00
Chris Lu
c9722dceb2 adjust log level 2021-02-28 20:26:55 -08:00
Chris Lu
f2fcb77808 local file sink: create backup files with permission 0755 2021-02-28 16:20:47 -08:00
Chris Lu
984fdd6192 always use non bucket prefixing url 2021-02-28 16:20:13 -08:00
Chris Lu
9abb041763 filer source: support filerProxy mode 2021-02-28 16:19:47 -08:00
Chris Lu
678c54d705 data sink: add incremental mode 2021-02-28 16:19:03 -08:00
Chris Lu
a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu
821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu
990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu
da08402ba2 replicate: use creation time for local incremental file sink
related to https://github.com/chrislusf/seaweedfs/pull/1762
2021-01-28 02:17:41 -08:00
Konstantin Lebedev
02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Konstantin Lebedev
6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Konstantin Lebedev
3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Konstantin Lebedev
612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu
00707ec00f mount: outsideContainerClusterMode proxy through filer
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
2021-01-24 19:01:58 -08:00
Chris Lu
6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers"
This reverts commit 096e088d7b.
2021-01-24 03:15:19 -08:00
Chris Lu
096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu
80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu
2b76854641 add "weed filer.cat" to read files directly from volume servers 2021-01-06 04:22:00 -08:00