Use s3 as backend

These docs provide a detailed examples for using s3 as backend.

access_key_id and secret_access_key could be loaded from ~/.aws/config automatically:

OPENDAL_S3_BUCKET=opendal OPENDAL_S3_REGION=test cargo run --example s3

Or specify manually:

OPENDAL_S3_BUCKET=opendal OPENDAL_S3_ACCESS_KEY_ID=minioadmin OPENDAL_S3_SECRET_ACCESS_KEY=minioadminx OPENDAL_S3_ENDPOINT=http://127.0.0.1:9900 OPENDAL_S3_REGION=test cargo run --example s3

All config could be passed via environment:

  • OPENDAL_S3_ROOT: root path, default: /
  • OPENDAL_S3_BUCKET: bucket name, required.
  • OPENDAL_S3_ENDPOINT: endpoint of s3 service, default: https://s3.amazonaws.com
  • OPENDAL_S3_REGION: region of s3 service, could be auto detected.
  • OPENDAL_S3_ACCESS_KEY_ID: access key id of s3 service, could be auto detected.
  • OPENDAL_S3_SECRET_ACCESS_KEY: secret access key of s3 service, could be auto detected.
  • OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE: Enable virtual host style for API request.

Compatible Services

AWS S3

AWS S3 is the default implementations of s3 services. Only bucket is required.

builder.bucket("<bucket_name>");

Alibaba Object Storage Service (OSS)

OSS is a s3 compatible service provided by Alibaba Cloud.

To connect to OSS, we need to set:

  • endpoint: The endpoint of oss, for example: https://oss-cn-hangzhou.aliyuncs.com
  • bucket: The bucket name of oss.

OSS provide internal endpoint for used at alibabacloud internally, please visit OSS Regions and endpoints for more details.

OSS only supports the virtual host style, users could meet errors like:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>SecondLevelDomainForbidden</Code>
 <Message>The bucket you are attempting to access must be addressed using OSS third level domain.</Message>
 <RequestId>62A1C265292C0632377F021F</RequestId>
 <HostId>oss-cn-hangzhou.aliyuncs.com</HostId>
</Error>

In that case, please enable virtual host style for requesting.

builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com");
builder.region("<region>");
builder.bucket("<bucket_name>");
builder.enable_virtual_host_style();

Minio

minio is an open-source s3 compatible services.

To connect to minio, we need to set:

  • endpoint: The endpoint of minio, for example: http://127.0.0.1:9000
  • region: The region of minio. If not specified, it could be ignored.
  • bucket: The bucket name of minio.
builder.endpoint("http://127.0.0.1:9000");
builder.region("<region>");
builder.bucket("<bucket_name>");

QingStor Object Storage

QingStor Object Storage is a S3-compatible service provided by QingCloud.

To connect to QingStor Object Storage, we need to set:

  • endpoint: The endpoint of QingStor s3 compatible endpoint, for example: https://s3.pek3b.qingstor.com
  • bucket: The bucket name.

Scaleway Object Storage

Scaleway Object Storage is a S3-compatible and multi-AZ redundant object storage service.

To connect to Scaleway Object Storage, we need to set:

  • endpoint: The endpoint of scaleway, for example: https://s3.nl-ams.scw.cloud
  • region: The region of scaleway.
  • bucket: The bucket name of scaleway.

Tencent Cloud Object Storage (COS)

COS is a s3 compatible service provided by Tencent Cloud.

To connect to COS, we need to set:

  • endpoint: The endpoint of cos, for example: https://cos.ap-beijing.myqcloud.com
  • bucket: The bucket name of cos.

Wasabi Object Storage

Wasabi is a s3 compatible service.

Cloud storage pricing that is 80% less than Amazon S3.

To connect to wasabi, we need to set:

  • endpoint: The endpoint of wasabi, for example: https://s3.us-east-2.wasabisys.com
  • bucket: The bucket name of wasabi.

Refer to What are the service URLs for Wasabi's different storage regions? for more details.

Example

Via Environment

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::S3)?;
}

Via Builder

//! Example for initiating a s3 backend.
use std::env;

use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL s3 Example.

Available Environment Values:

- OPENDAL_S3_ROOT: root path, default: /
- OPENDAL_S3_BUCKET: bukcet name, required.
- OPENDAL_S3_ENDPOINT: endpoint of s3 service, default: https://s3.amazonaws.com
- OPENDAL_S3_REGION: region of s3 service, could be auto detected.
- OPENDAL_S3_ACCESS_KEY_ID: access key id of s3 service, could be auto detected.
- OPENDAL_S3_SECRET_ACCESS_KEY: secret access key of s3 service, could be auto detected.
- OPENDAL_S3_SECURITY_TOKEN: temporary credentials of s3 service, optional.
    "#
    );

    // Create s3 backend builder.
    let mut builder: Builder = s3::Builder::default();
    // Set the root for s3, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_S3_ROOT").unwrap_or_default());
    // Set the bucket name, this is required.
    builder.bucket(&env::var("OPENDAL_S3_BUCKET").expect("env OPENDAL_S3_BUCKET not set"));
    // Set the endpoint.
    //
    // For examples:
    // - "https://s3.amazonaws.com"
    // - "http://127.0.0.1:9000"
    // - "https://oss-ap-northeast-1.aliyuncs.com"
    // - "https://cos.ap-seoul.myqcloud.com"
    //
    // Default to "https://s3.amazonaws.com"
    builder.endpoint(
        &env::var("OPENDAL_S3_ENDPOINT").unwrap_or_else(|_| "https://s3.amazonaws.com".to_string()),
    );
    // Set the region in we have this env.
    if let Ok(region) = env::var("OPENDAL_S3_REGION") {
        builder.region(&region);
    }
    // Set the credential.
    //
    // OpenDAL will try load credential from the env.
    // If credential not set and no valid credential in env, OpenDAL will
    // send request without signing like anonymous user.
    builder.access_key_id(
        &env::var("OPENDAL_S3_ACCESS_KEY_ID").expect("env OPENDAL_S3_ACCESS_KEY_ID not set"),
    );
    builder.secret_access_key(
        &env::var("OPENDAL_S3_SECRET_ACCESS_KEY")
            .expect("env OPENDAL_S3_SECRET_ACCESS_KEY not set"),
    );

    // Set the temporary credentials for s3 backend.
    //
    // Temporary credentials expires in a short period of time and OpenDAL will *not* take care of it.
    // Please make sure that the credential is valid.
    if let Ok(token) = &env::var("OPENDAL_S3_SECURITY_TOKEN") {
        builder.security_token(token);
    }

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}