For filer,
- The metadata is stored in filer store.
- The actual data is stored in volume servers.
However, there could be many volume servers. And the volumes may be tiered to the cloud. What if some of them are hacked?
Encrypt data on volume servers
weed filer -encryptVolumeData is an option to encrypt the data on volume servers.
The encryption keys are randomly generated during write time, and are different for different files. The encryption keys are stored as metadata in filer store.
So the volume data on the volume servers are encrypted. As long as the filer store is not exposed, it is nearly impossible to guess the encryption keys for all the files.
Safe Data Storage
Actually the volume servers do not have any concept of encryption. The encrypted data blocks are treated just as any other data blocks. The volume servers are not visible to any unencrypted data, for either storage or transmission. With the file content encrypted, it is safe to put volume servers any where you want.
Safely Forget Data
Another side is, with GDPR, companies are required to "forget" customer data after some time. If the volume data is stored on a glacial storage system, it is cumbersome to dig them out and destroy them. It is much easier to just delete the metadata, and the volume data is automatically "destroyed".
Encryption Algorithm
The encryption is through AES256-GCM https://en.wikipedia.org/wiki/Galois/Counter_Mode
There is one randomly generated cipher key of 256 bits for each file chunk. The cipher code is here https://github.com/seaweedfs/seaweedfs/blob/master/weed/util/cipher.go
Introduction
API
Configuration
- Replication
- Store file with a Time To Live
- Failover Master Server
- Erasure coding for warm storage
- Server Startup Setup
- Environment Variables
Filer
- Filer Setup
- Directories and Files
- Data Structure for Large Files
- Filer Data Encryption
- Filer Commands and Operations
- Filer JWT Use
Filer Stores
- Filer Cassandra Setup
- Filer Redis Setup
- Super Large Directories
- Path-Specific Filer Store
- Choosing a Filer Store
- Customize Filer Store
Advanced Filer Configurations
- Migrate to Filer Store
- Add New Filer Store
- Filer Store Replication
- Filer Active Active cross cluster continuous synchronization
- Filer as a Key-Large-Value Store
- Path Specific Configuration
- Filer Change Data Capture
FUSE Mount
WebDAV
Cloud Drive
- Cloud Drive Benefits
- Cloud Drive Architecture
- Configure Remote Storage
- Mount Remote Storage
- Cache Remote Storage
- Cloud Drive Quick Setup
- Gateway to Remote Object Storage
AWS S3 API
- Amazon S3 API
- AWS CLI with SeaweedFS
- s3cmd with SeaweedFS
- rclone with SeaweedFS
- restic with SeaweedFS
- nodejs with Seaweed S3
- S3 API Benchmark
- S3 API FAQ
- S3 Bucket Quota
- S3 API Audit log
- S3 Nginx Proxy
- Docker Compose for S3
AWS IAM
Machine Learning
HDFS
- Hadoop Compatible File System
- run Spark on SeaweedFS
- run HBase on SeaweedFS
- run Presto on SeaweedFS
- Hadoop Benchmark
- HDFS via S3 connector
Replication and Backup
- Async Replication to another Filer [Deprecated]
- Async Backup
- Async Filer Metadata Backup
- Async Replication to Cloud [Deprecated]
- Kubernetes Backups and Recovery with K8up
Messaging
Use Cases
Operations
Advanced
- Large File Handling
- Optimization
- Volume Management
- Tiered Storage
- Cloud Tier
- Cloud Monitoring
- Load Command Line Options from a file
- SRV Service Discovery
- Volume Files Structure