mirror/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-11-23 18:49:17 +08:00
Code Issues Packages Projects Releases Wiki Activity
Page: TensorFlow with SeaweedFS
Pages
AWS CLI with SeaweedFS AWS IAM CLI Actual Users Amazon IAM API Amazon S3 API Applications Async Backup Async Filer Metadata Backup Async Replication to Cloud Async Replication to another Filer Benchmark SeaweedFS as a GlusterFS replacement Benchmarks from jinleileiking Benchmarks Cache Remote Storage Choosing a Filer Store Client Libraries Cloud Drive Architecture Cloud Drive Benefits Cloud Drive Quick Setup Cloud Monitoring Cloud Tier Components Configure Remote Storage Customize Filer Store Data Backup Data Structure for Large Files Deployment to Kubernetes and Minikube Directories and Files Docker Compose for S3 Docker Image Registry with SeaweedFS Environment Variables Erasure Coding for warm storage Error reporting to sentry FAQ FIO benchmark FUSE Mount Failover Master Server Filer Active Active cross cluster continuous synchronization Filer Cassandra Setup Filer Change Data Capture Filer Commands and Operations Filer Data Encryption Filer JWT Use Filer Metadata Events Filer Redis Setup Filer Server API Filer Setup Filer Store Replication Filer Stores Filer as a Key Large Value Store Gateway to Remote Object Storage Getting Started HDFS via S3 connector Hadoop Benchmark Hadoop Compatible File System Hardware Hobbyest Tinkerer scale on premises tutorial Home Independent Benchmarks Kubernetes Backups and Recovery with K8up Large File Handling Load Command Line Options from a file Master Server API Migrate to Filer Store Mount Remote Storage Optimization Path Specific Configuration Path Specific Filer Store Production Setup Replication Run Blob Storage on Public Internet Run Presto on SeaweedFS S3 API Audit log S3 API Benchmark S3 API FAQ S3 Bucket Quota S3 Nginx Proxy SRV Service Discovery SeaweedFS Java Client SeaweedFS in Docker Swarm Security Configuration Security Overview Server Startup Setup Store file with a Time To Live Super Large Directories System Metrics TensorFlow with SeaweedFS Tiered Storage UrBackup with SeaweedFS Use Cases Volume Files Structure Volume Management Volume Server API WebDAV Words from SeaweedFS Users fstab nodejs with Seaweed S3 rclone with SeaweedFS restic with SeaweedFS run HBase on SeaweedFS run Spark on SeaweedFS s3cmd with SeaweedFS weed shell
3 TensorFlow with SeaweedFS
Jack edited this page 2024-05-17 13:56:07 +01:00
Table of Contents

Simplest Example

import tensorflow as tf
import os

os.environ["S3_ENDPOINT"] = "http://localhost:8333"

# ...

train_dataset = tf.data.TFRecordDataset(filenames=[
    "s3://bucketname/path/to/file1.tfrecord",
    "s3://bucketname/path/to/file2.tfrecord",
]).map(record_parser).batch(BATCH_SIZE)

# ...

model.fit(train_dataset, ...)

TensorFlow on SeaweedFS S3

TensorFlow already supports S3 (Old Link)

Here is an adaption of it with unnecessary content removed.

Configuration

When reading or writing data on S3 with your TensorFlow program, the behavior can be controlled by various environmental variables:

  • S3_ENDPOINT: The endpoint could be overridden explicitly with S3_ENDPOINT specified.

To read or write objects in a bucket that is not publicly accessible, AWS credentials must be provided through one of the following methods:

  • Set credentials in the AWS credentials profile file on the local system, located at: ~/.aws/credentials on Linux, macOS, or Unix, or C:\Users\USERNAME\.aws\credentials on Windows.
  • Set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.

Example Setup

Using the above information, we can configure Tensorflow to communicate to an S3 endpoint by setting the following environment variables:

S3_ENDPOINT=http://localhost:8333
AWS_ACCESS_KEY_ID=XXXXX                 # Credentials if configured
AWS_SECRET_ACCESS_KEY=XXXXX

Usage

Once setup is completed, Tensorflow can interact with S3 in a variety of ways. Anywhere there is a Tensorflow IO function, an S3 URL can be used.

Smoke Test

To test your setup, stat a file:

from tensorflow.python.lib.io import file_io
print file_io.stat('s3://bucketname/path/')

You should see output similar to this:

<tensorflow.python.pywrap_tensorflow_internal.FileStatistics; proxy of <Swig Object of type 'tensorflow::FileStatistics *' at 0x10c2171b0> >

Reading Data

filenames = ["s3://bucketname/path/to/file1.tfrecord",
             "s3://bucketname/path/to/file2.tfrecord"]
dataset = tf.data.TFRecordDataset(filenames)

Tensorflow Tools

Many Tensorflow tools, such as Tensorboard or model serving, can also take S3 URLS as arguments:

tensorboard --logdir s3://bucketname/path/to/model/
tensorflow_model_server --port=9000 --model_name=model --model_base_path=s3://bucketname/path/to/model/export/

This enables an end to end workflow using S3 for all data needs.

Introduction

API

Configuration

Filer

Filer Stores

Advanced Filer Configurations

FUSE Mount

WebDAV

Cloud Drive

AWS S3 API

AWS IAM

Machine Learning

HDFS

Replication and Backup

Messaging

Use Cases

Operations

Advanced

Security

Misc Use Case Examples