Introduction
Some user will ask for no single point of failure. Although google runs its file system with a single master for years, no SPOF seems becoming a criteria for architects to pick solutions.
Luckily, it's not too difficult to enable Weed File System with failover master servers.
Cheat Sheet: Startup multiple servers
This section is a quick way to start 3 master servers and 3 volume servers. All done!
weed server -master.ip=localhost -master.port=9333 -dir=./1 -volume.port=8080 \
-master.peers=localhost:9333,localhost:9334,localhost:9335
weed server -master.ip=localhost -master.port=9334 -dir=./2 -volume.port=8081 \
-master.peers=localhost:9333,localhost:9334,localhost:9335
weed server -master.ip=localhost -master.port=9335 -dir=./3 -volume.port=8082 \
-master.peers=localhost:9333,localhost:9334,localhost:9335
How it works
The master servers are coordinated by Raft protocol, to elect a leader. The leader takes over all the work to manage volumes, assign file ids. All other master servers just simply forward requests to the leader.
If the leader dies, another leader will be elected. And all the volume servers will send their heartbeat together with their volumes information to the new leader. The new leader will take the full responsibility.
During the transition, there could be moments where the new leader has partial information about all volume servers. This just means those yet-to-heartbeat volume servers will not be writable temporarily.
Master Identifier
When specifying the peers, the peers are identified by the hostname or IP, and port. The hostname or IP should be consistent with the value from "-ip" option when starting the master.
Startup multiple master servers
Now let's start the master and volume servers separately, the usual way.
Usually you would start several (3 or 5) master servers, then start the volume servers:
weed master -ip=localhost -port=9333 -mdir=./1 -peers=localhost:9333,localhost:9334,localhost:9335
weed master -ip=localhost -port=9334 -mdir=./2 -peers=localhost:9333,localhost:9334,localhost:9335
weed master -ip=localhost -port=9335 -mdir=./3 -peers=localhost:9333,localhost:9334,localhost:9335
# now start the volume servers, specifying any one of the master server
weed volume -dir=./1 -port=8080 -mserver=localhost:9333,localhost:9334,localhost:9335
weed volume -dir=./2 -port=8081 -mserver=localhost:9333,localhost:9334,localhost:9335
weed volume -dir=./3 -port=8082 -mserver=localhost:9333,localhost:9334,localhost:9335
# The full list of masters is recommended, but not required.
# You can only specify any master servers.
weed volume -dir=./3 -port=8082 -mserver=localhost:9333
weed volume -dir=./3 -port=8082 -mserver=localhost:9333,localhost:9334
These 6 commands will actually functioning the same as the previous 3 commands from the cheatsheet.
It is a best practice for volume servers to include as many master servers as possible. So when one of the masters failed, the volume server can switch to another one.
FAQ
-
Q: Is it possible to add an additional master server to an already running SeaweedFS cluster?
A: Treat the masters as zookeeper nodes, and do not need to change often. To add a master server, you need to stop all existing masters, and start the new set of masters with the new master list. For volume servers, it is OK to just keep using with existing masters when adding new volume servers.
Introduction
API
Configuration
- Replication
- Store file with a Time To Live
- Failover Master Server
- Erasure coding for warm storage
- Server Startup Setup
- Environment Variables
Filer
- Filer Setup
- Directories and Files
- Data Structure for Large Files
- Filer Data Encryption
- Filer Commands and Operations
- Filer JWT Use
Filer Stores
- Filer Cassandra Setup
- Filer Redis Setup
- Super Large Directories
- Path-Specific Filer Store
- Choosing a Filer Store
- Customize Filer Store
Advanced Filer Configurations
- Migrate to Filer Store
- Add New Filer Store
- Filer Store Replication
- Filer Active Active cross cluster continuous synchronization
- Filer as a Key-Large-Value Store
- Path Specific Configuration
- Filer Change Data Capture
FUSE Mount
WebDAV
Cloud Drive
- Cloud Drive Benefits
- Cloud Drive Architecture
- Configure Remote Storage
- Mount Remote Storage
- Cache Remote Storage
- Cloud Drive Quick Setup
- Gateway to Remote Object Storage
AWS S3 API
- Amazon S3 API
- AWS CLI with SeaweedFS
- s3cmd with SeaweedFS
- rclone with SeaweedFS
- restic with SeaweedFS
- nodejs with Seaweed S3
- S3 API Benchmark
- S3 API FAQ
- S3 Bucket Quota
- S3 API Audit log
- S3 Nginx Proxy
- Docker Compose for S3
AWS IAM
Machine Learning
HDFS
- Hadoop Compatible File System
- run Spark on SeaweedFS
- run HBase on SeaweedFS
- run Presto on SeaweedFS
- Hadoop Benchmark
- HDFS via S3 connector
Replication and Backup
- Async Replication to another Filer [Deprecated]
- Async Backup
- Async Filer Metadata Backup
- Async Replication to Cloud [Deprecated]
- Kubernetes Backups and Recovery with K8up
Messaging
Use Cases
Operations
Advanced
- Large File Handling
- Optimization
- Volume Management
- Tiered Storage
- Cloud Tier
- Cloud Monitoring
- Load Command Line Options from a file
- SRV Service Discovery
- Volume Files Structure