mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-23 18:49:17 +08:00
Code Issues Packages Projects Releases Wiki Activity
Page: rclone with SeaweedFS
Pages
AWS CLI with SeaweedFS AWS IAM CLI Actual Users Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Components Configure Remote Storage Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Directories and Files Docker Compose for S3 Docker Image Registry with SeaweedFS Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Hobbyest Tinkerer scale on premises tutorial Home Independent Benchmarks Kubernetes Backups and Recovery with K8up Large File Handling Load Command Line Options from a file Master Server API Migrate to Filer Store Mount Remote Storage Optimization Path Specific Configuration Path Specific Filer Store Production Setup Replication Run Blob Storage on Public Internet Run Presto on SeaweedFS S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Quota S3 Nginx Proxy SRV Service Discovery SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Startup Setup Store file with a Time To Live Super Large Directories System Metrics TensorFlow with SeaweedFS Tiered Storage UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users fstab nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
3 rclone with SeaweedFS
Er2 edited this page 2024-10-03 18:23:22 +03:00
Table of Contents

Installation

See https://rclone.org/install/

On mac: brew install rclone

Configuration

See https://rclone.org/s3/

Set config ~/.config/rclone/rclone.conf:

[swfs]
type = s3
provider = Other
access_key_id = any-key-id
secret_access_key = any-access-key
endpoint = http://localhost:8333
upload_cutoff = 50Mi
chunk_size = 50Mi
force_path_style = true

Reverse proxy sub-path configuration

This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths.
Use this only if you can't create (sub-)domain and use other port!

rclone (like AWS CLI) appends sub-path before actual path so need to add X-Forwarded-Prefix header (set to /s3 for example)

Example for Caddy web server

redir /s3 /s3/
handle_path /s3/* {
	reverse_proxy localhost:8333 {
		header_up X-Forwarded-Prefix /s3
	}
}

Execute commands

copy files

rclone  --log-level INFO copy  --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files

Client-side encryption

Installation local KMS API

git clone https://github.com/kmlebedev/local-kms.git
cd local-kms
go install

run local-kms

local-kms 
INFO[2023-07-24 15:34:23.876] Local KMS Version Unknown (Commit Hash Unknown) 
INFO[2023-07-24 15:34:23.992] No file found at path /init/seed.yaml; skipping seeding. 
INFO[2023-07-24 15:34:23.992] Data will be stored in /tmp/local-kms        
INFO[2023-07-24 15:34:23.992] Local KMS started on 0.0.0.0:8080

create master key

aws kms create-key --endpoint=http://localhost:8080
{
    "KeyMetadata": {
        "AWSAccountId": "111122223333",
        "KeyId": "5beb0309-d1ec-45ea-895a-52bbecbc8bde",
        "Arn": "arn:aws:kms:eu-west-2:111122223333:key/5beb0309-d1ec-45ea-895a-52bbecbc8bde",
        "CreationDate": "2023-07-03T14:24:36+05:00",
        "Enabled": true,
        "KeyUsage": "ENCRYPT_DECRYPT",
        "KeyState": "Enabled",
        "Origin": "AWS_KMS",
        "KeyManager": "CUSTOMER",
        "CustomerMasterKeySpec": "SYMMETRIC_DEFAULT",
        "KeySpec": "SYMMETRIC_DEFAULT",
        "EncryptionAlgorithms": [
            "SYMMETRIC_DEFAULT"
        ]
    }
}

Copy files with encryption

rclone  --log-level INFO copy --cse-kms-master-key-id 5beb0309-d1ec-45ea-895a-52bbecbc8bde --kms-endpoint http://localhost:8080 --ignore-size --checksum --fast-list /Users/kmlebedev/files swfs:/bucket-name/files

Introduction

API

Configuration

Filer

Filer Stores

Advanced Filer Configurations

FUSE Mount

WebDAV

Cloud Drive

AWS S3 API

AWS IAM

Machine Learning

HDFS

Replication and Backup

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples