5 s3cmd with SeaweedFS

Yichuan Sun edited this page 2024-11-05 22:30:50 +08:00

Table of Contents

• Installation
• Configuration
• Reverse proxy sub-path configuration
• Execute commands

Installation

See https://s3tools.org/download

1. Download the zip, unzip, cd into the folder.
2. python setup.py install

Configuration

Answer questions with the following answers:

s3cmd --configure

...

New settings:
  Access Key: any_no_empty_key
  Secret Key: any_no_empty_key
  Default Region: US
  S3 Endpoint: localhost:8333
  DNS-style bucket+hostname:port template for accessing a bucket: localhost:8333
  Encryption password:
  Path to GPG program: /usr/local/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Make sure the .s3cfg file has these values

# Setup endpoint
host_base = localhost:8333
host_bucket = localhost:8333
use_https = False
# Enable S3 v4 signature APIs
signature_v2 = False

Reverse proxy sub-path configuration

This is undefined behavior as AWS S3 servers always have sub-domains instead of sub-paths.
Use this only if you can't create (sub-)domain and use other port!

For s3cmd, X-Forwarded-Host is required to be your.server/s3 but not for other utils so it's better to use signature_v2 = True in s3cfg

Example for Caddy web server

redir /s3 /s3/
handle_path /s3/* {
	reverse_proxy localhost:8333 {
		header_up X-Forwarded-Host your.server/s3
		header_up X-Forwarded-Prefix /s3
	}
}

Execute commands

$ s3cmd mb s3://newbucket
Bucket 's3://newbucket/' created

$ s3cmd ls s3://
2019-01-01 01:30  s3://newbucket

$ s3cmd put /etc/motd s3://newbucket
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: '/etc/motd' -> 's3://newbucket/motd'  [1 of 1]
 6804 of 6804   100% in    0s    87.93 kB/s  done

$ s3cmd get s3://newbucket/motd
download: 's3://newbucket/motd' -> './motd'  [1 of 1]
 6804 of 6804   100% in    0s   595.33 kB/s  done

# change the file
$ vi motd

$ s3cmd sync motd s3://newbucket/
WARNING: Module python-magic is not available. Guessing MIME types based on file extensions.
upload: 'motd' -> 's3://newbucket/motd'  [1 of 1]
 22 of 22   100% in    0s     6.45 kB/s  done
Done. Uploaded 22 bytes in 1.0 seconds, 22.00 B/s.

$ s3cmd ls s3://newbucket/
2019-01-01 01:32         22   s3://newbucket/motd

$ s3cmd del s3://newbucket/motd
delete: 's3://newbucket/motd'

$ s3cmd rb s3://newbucket
Bucket 's3://newbucket/' removed

Introduction

• Components
• Getting Started
• Production Setup
• Benchmarks
• FAQ
• Applications

API

• Master Server API
• Volume Server API
• Filer Server API
• Client Libraries
• SeaweedFS Java Client

Configuration

• Replication
• Store file with a Time To Live
• Failover Master Server
• Erasure coding for warm storage
• Server Startup Setup
• Environment Variables

Filer

• Filer Setup
• Directories and Files
• Data Structure for Large Files
• Filer Data Encryption
• Filer Commands and Operations
• Filer JWT Use

Filer Stores

• Filer Cassandra Setup
• Filer Redis Setup
• Super Large Directories
• Path-Specific Filer Store
• Choosing a Filer Store
• Customize Filer Store

Advanced Filer Configurations

• Migrate to Filer Store
• Add New Filer Store
• Filer Store Replication
• Filer Active Active cross cluster continuous synchronization
• Filer as a Key-Large-Value Store
• Path Specific Configuration
• Filer Change Data Capture

FUSE Mount

• FIO benchmark
• fstab

WebDAV

Cloud Drive

• Cloud Drive Benefits
• Cloud Drive Architecture
• Configure Remote Storage
• Mount Remote Storage
• Cache Remote Storage
• Cloud Drive Quick Setup
• Gateway to Remote Object Storage

AWS S3 API

• Amazon S3 API
• AWS CLI with SeaweedFS
• s3cmd with SeaweedFS
• rclone with SeaweedFS
• restic with SeaweedFS
• nodejs with Seaweed S3
• S3 API Benchmark
• S3 API FAQ
• S3 Bucket Quota
• S3 API Audit log
• S3 Nginx Proxy
• Docker Compose for S3

AWS IAM

• Amazon IAM API
• AWS IAM CLI

Machine Learning

• TensorFlow with SeaweedFS

HDFS

• Hadoop Compatible File System
• run Spark on SeaweedFS
• run HBase on SeaweedFS
• run Presto on SeaweedFS
• Hadoop Benchmark
• HDFS via S3 connector

Replication and Backup

• Async Replication to another Filer [Deprecated]
• Async Backup
• Async Filer Metadata Backup
• Async Replication to Cloud [Deprecated]
• Kubernetes Backups and Recovery with K8up

Messaging

• Filer Metadata Events

Use Cases

• Use Cases
• Actual Users

Operations

• System Metrics
• weed shell
• Data Backup
• Deployment to Kubernetes and Minikube

Advanced

• Large File Handling
• Optimization
• Volume Management
• Tiered Storage
• Cloud Tier
• Cloud Monitoring
• Load Command Line Options from a file
• SRV Service Discovery
• Volume Files Structure

Security

• Security Overview
• Security Configuration
• Run Blob Storage on Public Internet

Misc Use Case Examples

• UrBackup with SeaweedFS
• Docker Image Registry with SeaweedFS
• SeaweedFS in Docker Swarm
• Words from SeaweedFS Users
• Independent Benchmarks
• Hardware