OpenDAL   Build Status Latest Version Crate Downloads

Open Data Access Layer: Access data freely, painlessly, and efficiently


You may be looking for:

Services

Features

Access data freely

  • Access different storage services in the same way
  • Behavior tests for all services

Access data painlessly

  • 100% documents covered
  • Powerful Layers
  • Automatic retry support
  • Full observability support: logging, tracing, metrics.
  • Native decompress support
  • Native service-side encryption support

Access data efficiently

  • Zero cost: mapping to underlying API calls directly
  • Auto metadata reuse: avoid extra metadata calls

Quickstart

use anyhow::Result;
use futures::StreamExt;
use futures::TryStreamExt;
use opendal::ObjectStreamer;
use opendal::Object;
use opendal::ObjectMetadata;
use opendal::ObjectMode;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator
    let op = Operator::from_env(Scheme::Fs)?;

    // Create object handler.
    let o = op.object("test_file");

    // Write data info object;
    o.write("Hello, World!").await?;

    // Read data from object;
    let bs = o.read().await?;

    // Read range from object;
    let bs = o.range_read(1..=11).await?;

    // Get object's path
    let name = o.name();
    let path = o.path();

    // Fetch more meta about object.
    let meta = o.metadata().await?;
    let mode = meta.mode();
    let length = meta.content_length();
    let content_md5 = meta.content_md5();
    let etag = meta.etag();

    // Delete object.
    o.delete().await?;

    // List dir object.
    let o = op.object("test_dir/");
    let mut os = o.list().await?;
    while let Some(entry) = os.try_next().await? {
        let path = entry.path();
        let mode = entry.mode().await?;
    }

    Ok(())
}

More examples could be found at Documentation.

Projects

  • Databend: A modern Elasticity and Performance cloud data warehouse.
  • GreptimeDB: An open-source, cloud-native, distributed time-series database.
  • deepeth/mars: The powerful analysis platform to explore and visualize data from blockchain.

Contributing

Check out the CONTRIBUTING.md guide for more details on getting started with contributing to this project.

Getting help

Submit issues for bug report or asking questions in discussion.

License

Licensed under Apache License, Version 2.0.

Concepts

OpenDAL provides a unified abstraction for all storage services.

Accessor

First, let's start with Accessor. Accessor is the underlying trait that communicate with different storage backends. We use Accessor to abstract the same operations sets on different backends.

Operator

To make our users life easier, we build a struct Operator which can be used anyway.

Operator itself is a simple wrapper of Accessor and is very cheap to be cloned. We build our public APIs upon Operator instead.

Object

Object is the smallest unit in OpenDAL which could be a File Object, Dir Object or others. Object is constructed via Operator::object().

It's a bit like File and Handle in Rust, but we don't need to open or close them. Every API is very straight:

For detailed APIs, please refer to API Documents

Services

This section will demonstrate how to init and start using a service.

OpenDAL can init services via environment or builder.

  • Via Builder: use different backends' Builder API.
  • Via Environment: use Operator::from_env() API.

OpenDAL supports the following services:

  • azblob: Azure blob storage service
  • fs: POSIX alike file system
  • gcs: Google Cloud Storage service
  • hdfs: Hadoop Distributed File System
  • s3: AWS S3 compatible storage service

Azblob

These docs provide a detailed examples for using azblob as backend.

This example has native support for Azure Storage Simulator Azurite. All value will fall back to Azurite default settings.

We can start a mock services like:

docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite
az storage container create --name test --connection-string "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;"

Then start our azblob examples

OPENDAL_AZBLOB_CONTAINER=test cargo run --example azblob

Example

Via Environment

All config could be passed via environment:

  • OPENDAL_AZBLOB_ROOT: root path, default: /
  • OPENDAL_AZBLOB_CONTAINER: container name
  • OPENDAL_AZBLOB_ENDPOINT: endpoint of your container
  • OPENDAL_AZBLOB_ACCOUNT_NAME: account name
  • OPENDAL_AZBLOB_ACCOUNT_KEY: account key
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::Azblob)?;
}

Via Builder

//! Example for initiating a azblob backend.

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::azblob;
use opendal::services::azblob::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL azblob example.

Available Environment Values:

- OPENDAL_AZBLOB_ROOT: root path, default: /
- OPENDAL_AZBLOB_CONTAINER: container name
- OPENDAL_AZBLOB_ENDPOINT: endpoint of your container
- OPENDAL_AZBLOB_ACCOUNT_NAME: account name
- OPENDAL_AZBLOB_ACCOUNT_KEY: account key

    "#
    );

    // Create fs backend builder.
    let mut builder: Builder = azblob::Builder::default();
    // Set the root, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_AZBLOB_ROOT").unwrap_or_else(|_| "/".to_string()));
    // Set the container name
    builder.container(
        &env::var("OPENDAL_AZBLOB_CONTAINER").expect("env OPENDAL_AZBLOB_CONTAINER not set"),
    );
    // Set the endpoint
    //
    // For examples:
    // - "http://127.0.0.1:10000/devstoreaccount1"
    // - "https://accountname.blob.core.windows.net"
    builder.endpoint(
        &env::var("OPENDAL_AZBLOB_ENDPOINT")
            .unwrap_or_else(|_| "http://127.0.0.1:10000/devstoreaccount1".to_string()),
    );
    // Set the account_name and account_key.
    builder.account_name(
        &env::var("OPENDAL_AZBLOB_ACCOUNT_NAME").unwrap_or_else(|_| "devstoreaccount1".to_string()),
    );
    builder.account_key(
        &env::var("OPENDAL_AZBLOB_ACCOUNT_KEY")
            .unwrap_or_else(|_| "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==".to_string()),
    );

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use fs as backend

These docs provide a detailed examples for using fs as backend.

We can run this example via:

cargo run --example fs

Example

Via Environment

All config could be passed via environment:

  • OPENDAL_FS_ROOT: root path, default: /tmp
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::Fs)?;
}

Via Builder

//! Example for initiating a fs backend.

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::fs;
use opendal::services::fs::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL fs Example.

Available Environment Values:

- OPENDAL_FS_ROOT: root path, default: /tmp
    "#
    );

    // Create fs backend builder.
    let mut builder: Builder = fs::Builder::default();
    // Set the root for fs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_FS_ROOT").unwrap_or_else(|_| "/tmp".to_string()));

    // Use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use ftp as backend

These docs provide a detailed examples for using ftp as backend.

We can run this example via:

cargo run --example ftp --features services-ftp

Example

Via Environment

All config could be passed via environment:

  • OPENDAL_FTP_ENDPOINT endpoint to ftp services, required.
  • OPENDAL_FTP_ROOT root dir of this ftp services, default to /
  • OPENDAL_FTP_USER
  • OPENDAL_FTP_PASSWORD
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::Ftp)?;
}

Via Builder

//! Example for initiating a ftp backend.

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::ftp;
use opendal::services::ftp::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL ftp Example.

Available Environment Values:
    - OPENDAL_FTP_ENDPOINT=endpoint     # required
    - OPENDAL_FTP_ROOT=/path/to/dir/   # if not set, will be seen as "/"
    - OPENDAL_FTP_USER=user    # default with empty string ""
    - OPENDAL_FTP_PASSWORD=password    # default with empty string ""
    "#
    );

    // Create fs backend builder.
    let mut builder: Builder = ftp::Builder::default();
    builder.root(&env::var("OPENDAL_FTP_ROOT").unwrap_or_else(|_| "/".to_string()));
    // Set the root for ftp, all operations will happen under this root.

    // NOTE: the root must be absolute path.
    builder
        .endpoint(&env::var("OPENDAL_FTP_ENDPOINT").unwrap_or_else(|_| "127.0.0.1:21".to_string()));
    builder.user(&env::var("OPENDAL_FTP_USER").unwrap_or_else(|_| "".to_string()));
    builder.password(&env::var("OPENDAL_FTP_PASSWORD").unwrap_or_else(|_| "".to_string()));

    // Use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    info!("try to create file: {}", &path);
    op.object(&path).create().await?;
    info!("create file successful!");

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!".as_bytes()).await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to write to file: {}", &path);
    op.object(&path).write("write test".as_bytes()).await?;
    info!("write to file successful!",);

    info!("try to read file content between 5-10: {}", &path);
    let content = op.object(&path).range_read(..5).await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    let dir = "/ftptestfolder/";
    info!("try to create directory: {}", &dir);
    op.object("/ftptestfolder/").create().await?;
    info!("create folder successful!",);

    info!("try to delete directory: {}", &dir);
    op.object(dir).delete().await?;
    info!("delete directory successful");

    Ok(())
}

Use GCS as backend

This page provides some examples for using Google Cloud Storage as backend.

Example

Via Environment Variables

All config could be passed via environment variables:

  • OPENDAL_GCS_BUCKET bucket used for storing data, required
  • OPENDAL_GCS_ROOT working directory inside the bucket, default is "/"
  • OPENDAL_GCS_CREDENTIAL base64 OAUTH2 token used for authentication, required
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // init operator from env vars
    let _op = Operator::from_env(Scheme::Gcs)?;
}

Via Builder

//! Example for initiating a Google Cloud Storage backend.
use std::env;

use anyhow::Result;
use log::info;
use opendal::services::gcs::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"
    OpenDAL GCS example.

    Available Environment Variables:
    - OPENDAL_GCS_ENDPOINT: endpoint to GCS or GCS alike services, default is "https://storage.googleapis.com", optional
    - OPENDAL_GCS_BUCKET: bucket name, required
    - OPENDAL_GCS_ROOT: working directory of OpenDAL, default is "/", optional
    - OPENDAL_GCS_CREDENTIAL: OAUTH2 token for authentication, required
    "#
    );

    // create builder
    let mut builder = Builder::default();

    // set the endpoint for GCS or GCS alike services
    builder.endpoint(&env::var("OPENDAL_GCS_ENDPOINT").unwrap_or_else(|_| "".to_string()));
    // set the bucket used for OpenDAL service
    builder.bucket(&env::var("OPENDAL_GCS_BUCKET").expect("env OPENDAL_GCS_BUCKET not set"));
    // set the working directory for OpenDAL service
    //
    // could be considered as a fixed prefix for all paths you past into the backend
    builder.root(&env::var("OPENDAL_GCS_ROOT").unwrap_or_else(|_| "".to_string()));
    // OAUTH2 base64 credentials
    builder.credential(
        &env::var("OPENDAL_GCS_CREDENTIAL").expect("env OPENDAL_GCS_CREDENTIAL not set"),
    );

    let op = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Hdfs

These docs provide a detailed examples for using hdfs as backend.

We can run this example via:

cargo run --example hdfs --features services-hdfs

All config could be passed via environment:

  • OPENDAL_HDFS_ROOT: root path, default: /tmp
  • OPENDAL_HDFS_NAME_NODE: name node for hdfs, default: default

Example

Before running this example, please make sure the following env set correctly:

  • JAVA_HOME
  • HADOOP_HOME

Via Environment

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::Hdfs)?;
}

Via Builder


use std::env;

use anyhow::Result;
use log::info;
use opendal::services::hdfs;
use opendal::services::hdfs::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL HDFS Example.

Available Environment Values:

- OPENDAL_HDFS_ROOT: root path, default: /tmp
- OPENDAL_HDFS_NAME_NODE: name node for hdfs, default: default
    "#
    );

    // Create fs backend builder.
    let mut builder: Builder = hdfs::Builder::default();
    // Set the root for hdfs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_HDFS_ROOT").unwrap_or_else(|_| "/tmp".to_string()));
    // Set the name node for hdfs.
    //
    // Use `default` as default value.
    builder
        .name_node(&env::var("OPENDAL_HDFS_NAME_NODE").unwrap_or_else(|_| "default".to_string()));

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use IPFS as backend

This page provides some examples for using IPFS as backend.

We can run this example via:

cargo run --example ipfs --features services-ipfs

Example

Via Environment Variables

Available environment variables:

  • OPENDAL_IPFS_ROOT: root path, like /ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ
  • OPENDAL_IPFS_ENDPOINT: endpoint of ipfs, like https://ipfs.io
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // init operator from env vars
    let _op = Operator::from_env(Scheme::Ipfs)?;
}

Via Builder

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::ipfs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL IPFS Example.

Available Environment Values:

- OPENDAL_IPFS_ROOT: root path, like /ipfs/QmPpCt1aYGb9JWJRmXRUnmJtVgeFFTJGzWFYEEX7bo9zGJ
- OPENDAL_IPFS_ENDPOINT: ipfs endpoint, like https://ipfs.io
"#
    );

    let mut builder = ipfs::Builder::default();
    // root must be absolute path in MFS.
    builder.root(&env::var("OPENDAL_IPFS_ROOT").unwrap_or_else(|_| "/".to_string()));
    builder.endpoint(
        &env::var("OPENDAL_IPFS_ENDPOINT").unwrap_or_else(|_| "https://ipfs.io".to_string()),
    );

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = "normal_file";

    info!("try to read file: {}", &path);
    let content = op.object(path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    Ok(())
}

Use IPMFS as backend

This page provides some examples for using IPMFS as backend.

We can run this example via:

cargo run --example ipmfs

Example

Via Environment Variables

Available environment variables:

  • OPENDAL_IPMFS_ROOT: root path, default: /
  • OPENDAL_IPMFS_ENDPOINT: endpoint of ipfs.
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // init operator from env vars
    let _op = Operator::from_env(Scheme::Ipmfs)?;
}

Via Builder

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::ipmfs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL IPFS Example.

Available Environment Values:

- OPENDAL_IPMFS_ROOT: root path in mutable file system, default: /
- OPENDAL_IPMFS_ENDPOINT: ipfs endpoint, default: localhost:5001
"#
    );

    let mut builder = ipmfs::Builder::default();
    // root must be absolute path in MFS.
    builder.root(&env::var("OPENDAL_IPMFS_ROOT").unwrap_or_else(|_| "/".to_string()));
    builder.endpoint(
        &env::var("OPENDAL_IPMFS_ENDPOINT").unwrap_or_else(|_| "http://localhost:5001".to_string()),
    );

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use OBS as backend

This page provides some examples for using Huaweicloud OBS as backend.

We can run this example via:

cargo run --example obs

Example

Via Environment Variables

Available environment variables:

  • OPENDAL_OBS_ROOT: root path, default: /
  • OPENDAL_OBS_BUCKET: bukcet name, required.
  • OPENDAL_OBS_ENDPOINT: endpoint of obs service
  • OPENDAL_OBS_ACCESS_KEY_ID: access key id of obs service, could be auto detected.
  • OPENDAL_OBS_SECRET_ACCESS_KEY: secret access key of obs service, could be auto detected.
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // init operator from env vars
    let _op = Operator::from_env(Scheme::Obs)?;
}

Via Builder

//! Example for initiating an obs backend.
use std::env;

use anyhow::Result;
use log::info;
use opendal::services::obs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL obs Example.

Available Environment Values:

- OPENDAL_OBS_ROOT: root path, default: /
- OPENDAL_OBS_BUCKET: bukcet name, required.
- OPENDAL_OBS_ENDPOINT: endpoint of obs service
- OPENDAL_OBS_ACCESS_KEY_ID: access key id of obs service, could be auto detected.
- OPENDAL_OBS_SECRET_ACCESS_KEY: secret access key of obs service, could be auto detected.
    "#
    );

    // Create s3 backend builder.
    let mut builder = obs::Builder::default();
    // Set the root for obs, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_OBS_ROOT").unwrap_or_default());
    // Set the bucket name, this is required.
    builder.bucket(&env::var("OPENDAL_OBS_BUCKET").expect("env OPENDAL_OBS_BUCKET not set"));
    // Set the endpoint.
    //
    // For examples:
    // - `https://obs.cn-north-4.myhuaweicloud.com`
    // - `https://custom.obs.com`
    builder.endpoint(&env::var("OPENDAL_OBS_ENDPOINT").expect("env OPENDAL_OBS_BUCKET not set"));

    // Set the credential.
    //
    // OpenDAL will try load credential from the env.
    // If credential not set and no valid credential in env, OpenDAL will
    // send request without signing like anonymous user.
    builder.access_key_id(
        &env::var("OPENDAL_OBS_ACCESS_KEY_ID").expect("env OPENDAL_OBS_ACCESS_KEY_ID not set"),
    );
    builder.secret_access_key(
        &env::var("OPENDAL_OBS_SECRET_ACCESS_KEY")
            .expect("env OPENDAL_OBS_SECRET_ACCESS_KEY not set"),
    );

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use redis as backend

These docs provide a detailed examples for using redis as backend.

We can run this example via:

cargo run --example redis --features services-redis

Example

Via Environment

All config could be passed via environment:

  • OPENDAL_REDIS_ENDPOINT endpoint to redis services, required.
  • OPENDAL_REDIS_ROOT root dir of this ftp services, default to /
  • OPENDAL_REDIS_USERNAME
  • OPENDAL_REDIS_PASSWORD
  • OPENDAL_REDIS_DB
use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::Redis)?;
}

Via Builder

//! example for initiating a Redis backend

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::redis;
use opendal::services::redis::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();
    println!(
        r#"
        OpenDAL redis example.

        Available Environment Variables:

        - OPENDAL_REDIS_ENDPOINT: network address of redis server, default is "tcp://127.0.0.1:6379"
        - OPENDAL_REDIS_ROOT: working directory of opendal, default is "/"
        - OPENDAL_REDIS_USERNAME: username for redis, default is no username
        - OPENDAL_REDIS_PASSWORD: password to log in. default is no password
        - OPENDAL_REDIS_DB: Redis db to use, default is 0.
        "#
    );

    // Create redis backend builder
    let mut builder: Builder = redis::Builder::default();

    // Set the root, all operations will happen under this directory, or prefix, more accurately.
    //
    // NOTE: the root must be absolute path
    builder.root(&env::var("OPENDAL_REDIS_ROOT").unwrap_or_else(|_| "/".to_string()));

    // Set the endpoint, the address of remote redis server.
    builder.endpoint(
        &env::var("OPENDAL_REDIS_ENDPOINT").unwrap_or_else(|_| "tcp://127.0.0.1:6379".to_string()),
    );

    // Set the username
    builder.username(&env::var("OPENDAL_REDIS_USERNAME").unwrap_or_else(|_| "".to_string()));

    // Set the password
    builder.password(&env::var("OPENDAL_REDIS_PASSWORD").unwrap_or_else(|_| "".to_string()));

    // Set the db
    builder.db(env::var("OPENDAL_REDIS_DB")
        .map(|s| s.parse::<i64>().unwrap_or_default())
        .unwrap_or_else(|_| 0));

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use rocksdb as backend

These docs provide a detailed examples for using rocksdb as backend.

We can run this example via:

OPENDAL_ROCKSDB_DATADIR=/tmp/rocksdb cargo run --example=rocksdb --features=services-rocksdb

Example

Via Environment

All config could be passed via environment:

  • OPENDAL_ROCKSDB_DATADIR the path to the rocksdb data directory (required)
  • OPENDAL_ROCKSDB_ROOT working directory of opendal, default is /
use anyhow::Result;
use opendal::Object;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    let op = Operator::from_env(Scheme::Rocksdb)?;

    // create an object handler to start operation on rocksdb!
    let _op: Object = op.object("hello_rocksdb!");

    Ok(())
}

Via Builder

//! example for initiating a Rocksdb backend

use std::env;

use anyhow::Result;
use log::info;
use opendal::services::rocksdb;
use opendal::services::rocksdb::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();
    println!(
        r#"
        OpenDAL rocksdb example.

        Available Environment Variables:

        - OPENDAL_ROCKSDB_DATADIR: the path to the rocksdb data directory (required)
        - OPENDAL_ROCKSDB_ROOT:    working directory of opendal, default is "/"
        "#
    );

    // Create rocksdb backend builder
    let mut builder: Builder = rocksdb::Builder::default();

    // Set the root, all operations will happen under this directory, or prefix, more accurately.
    //
    // NOTE: the root must be absolute path
    builder.root(&env::var("OPENDAL_ROCKSDB_ROOT").unwrap_or_else(|_| "/".to_string()));

    // Set the path to the rocksdb data directory
    builder.datadir(
        &env::var("OPENDAL_ROCKSDB_DATADIR").expect("env OPENDAL_ROCKSDB_DATADIR is not set"),
    );

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Use s3 as backend

These docs provide a detailed examples for using s3 as backend.

access_key_id and secret_access_key could be loaded from ~/.aws/config automatically:

OPENDAL_S3_BUCKET=opendal OPENDAL_S3_REGION=test cargo run --example s3

Or specify manually:

OPENDAL_S3_BUCKET=opendal OPENDAL_S3_ACCESS_KEY_ID=minioadmin OPENDAL_S3_SECRET_ACCESS_KEY=minioadminx OPENDAL_S3_ENDPOINT=http://127.0.0.1:9900 OPENDAL_S3_REGION=test cargo run --example s3

All config could be passed via environment:

  • OPENDAL_S3_ROOT: root path, default: /
  • OPENDAL_S3_BUCKET: bucket name, required.
  • OPENDAL_S3_ENDPOINT: endpoint of s3 service, default: https://s3.amazonaws.com
  • OPENDAL_S3_REGION: region of s3 service, could be auto detected.
  • OPENDAL_S3_ACCESS_KEY_ID: access key id of s3 service, could be auto detected.
  • OPENDAL_S3_SECRET_ACCESS_KEY: secret access key of s3 service, could be auto detected.
  • OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE: Enable virtual host style for API request.

Compatible Services

AWS S3

AWS S3 is the default implementations of s3 services. Only bucket is required.

builder.bucket("<bucket_name>");

Alibaba Object Storage Service (OSS)

OSS is a s3 compatible service provided by Alibaba Cloud.

To connect to OSS, we need to set:

  • endpoint: The endpoint of oss, for example: https://oss-cn-hangzhou.aliyuncs.com
  • bucket: The bucket name of oss.

OSS provide internal endpoint for used at alibabacloud internally, please visit OSS Regions and endpoints for more details.

OSS only supports the virtual host style, users could meet errors like:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
 <Code>SecondLevelDomainForbidden</Code>
 <Message>The bucket you are attempting to access must be addressed using OSS third level domain.</Message>
 <RequestId>62A1C265292C0632377F021F</RequestId>
 <HostId>oss-cn-hangzhou.aliyuncs.com</HostId>
</Error>

In that case, please enable virtual host style for requesting.

builder.endpoint("https://oss-cn-hangzhou.aliyuncs.com");
builder.region("<region>");
builder.bucket("<bucket_name>");
builder.enable_virtual_host_style();

Minio

minio is an open-source s3 compatible services.

To connect to minio, we need to set:

  • endpoint: The endpoint of minio, for example: http://127.0.0.1:9000
  • region: The region of minio. If not specified, it could be ignored.
  • bucket: The bucket name of minio.
builder.endpoint("http://127.0.0.1:9000");
builder.region("<region>");
builder.bucket("<bucket_name>");

QingStor Object Storage

QingStor Object Storage is a S3-compatible service provided by QingCloud.

To connect to QingStor Object Storage, we need to set:

  • endpoint: The endpoint of QingStor s3 compatible endpoint, for example: https://s3.pek3b.qingstor.com
  • bucket: The bucket name.

Scaleway Object Storage

Scaleway Object Storage is a S3-compatible and multi-AZ redundant object storage service.

To connect to Scaleway Object Storage, we need to set:

  • endpoint: The endpoint of scaleway, for example: https://s3.nl-ams.scw.cloud
  • region: The region of scaleway.
  • bucket: The bucket name of scaleway.

Tencent Cloud Object Storage (COS)

COS is a s3 compatible service provided by Tencent Cloud.

To connect to COS, we need to set:

  • endpoint: The endpoint of cos, for example: https://cos.ap-beijing.myqcloud.com
  • bucket: The bucket name of cos.

Wasabi Object Storage

Wasabi is a s3 compatible service.

Cloud storage pricing that is 80% less than Amazon S3.

To connect to wasabi, we need to set:

  • endpoint: The endpoint of wasabi, for example: https://s3.us-east-2.wasabisys.com
  • bucket: The bucket name of wasabi.

Refer to What are the service URLs for Wasabi's different storage regions? for more details.

Example

Via Environment

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator from env.
    let op = Operator::from_env(Scheme::S3)?;
}

Via Builder

//! Example for initiating a s3 backend.
use std::env;

use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    if env::var("RUST_LOG").is_err() {
        env::set_var("RUST_LOG", "debug");
    }
    env_logger::init();

    println!(
        r#"OpenDAL s3 Example.

Available Environment Values:

- OPENDAL_S3_ROOT: root path, default: /
- OPENDAL_S3_BUCKET: bukcet name, required.
- OPENDAL_S3_ENDPOINT: endpoint of s3 service, default: https://s3.amazonaws.com
- OPENDAL_S3_REGION: region of s3 service, could be auto detected.
- OPENDAL_S3_ACCESS_KEY_ID: access key id of s3 service, could be auto detected.
- OPENDAL_S3_SECRET_ACCESS_KEY: secret access key of s3 service, could be auto detected.
- OPENDAL_S3_SECURITY_TOKEN: temporary credentials of s3 service, optional.
    "#
    );

    // Create s3 backend builder.
    let mut builder: Builder = s3::Builder::default();
    // Set the root for s3, all operations will happen under this root.
    //
    // NOTE: the root must be absolute path.
    builder.root(&env::var("OPENDAL_S3_ROOT").unwrap_or_default());
    // Set the bucket name, this is required.
    builder.bucket(&env::var("OPENDAL_S3_BUCKET").expect("env OPENDAL_S3_BUCKET not set"));
    // Set the endpoint.
    //
    // For examples:
    // - "https://s3.amazonaws.com"
    // - "http://127.0.0.1:9000"
    // - "https://oss-ap-northeast-1.aliyuncs.com"
    // - "https://cos.ap-seoul.myqcloud.com"
    //
    // Default to "https://s3.amazonaws.com"
    builder.endpoint(
        &env::var("OPENDAL_S3_ENDPOINT").unwrap_or_else(|_| "https://s3.amazonaws.com".to_string()),
    );
    // Set the region in we have this env.
    if let Ok(region) = env::var("OPENDAL_S3_REGION") {
        builder.region(&region);
    }
    // Set the credential.
    //
    // OpenDAL will try load credential from the env.
    // If credential not set and no valid credential in env, OpenDAL will
    // send request without signing like anonymous user.
    builder.access_key_id(
        &env::var("OPENDAL_S3_ACCESS_KEY_ID").expect("env OPENDAL_S3_ACCESS_KEY_ID not set"),
    );
    builder.secret_access_key(
        &env::var("OPENDAL_S3_SECRET_ACCESS_KEY")
            .expect("env OPENDAL_S3_SECRET_ACCESS_KEY not set"),
    );

    // Set the temporary credentials for s3 backend.
    //
    // Temporary credentials expires in a short period of time and OpenDAL will *not* take care of it.
    // Please make sure that the credential is valid.
    if let Ok(token) = &env::var("OPENDAL_S3_SECURITY_TOKEN") {
        builder.security_token(token);
    }

    // `Accessor` provides the low level APIs, we will use `Operator` normally.
    let op: Operator = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    let path = uuid::Uuid::new_v4().to_string();

    // Create an object handle to start operation on object.
    info!("try to write file: {}", &path);
    op.object(&path).write("Hello, world!").await?;
    info!("write file successful!");

    info!("try to read file: {}", &path);
    let content = op.object(&path).read().await?;
    info!(
        "read file successful, content: {}",
        String::from_utf8_lossy(&content)
    );

    info!("try to get file metadata: {}", &path);
    let meta = op.object(&path).metadata().await?;
    info!(
        "get file metadata successful, size: {}B",
        meta.content_length()
    );

    info!("try to delete file: {}", &path);
    op.object(&path).delete().await?;
    info!("delete file successful");

    Ok(())
}

Examples

This section will demonstrate how to use OpenDAL provided features.

Retry

OpenDAL has native support for retry.

Retry Layer

RetryLayer will add retry for OpenDAL operations.

use anyhow::Result;
use backon::ExponentialBackoff;
use opendal::layers::RetryLayer;
use opendal::Operator;
use opendal::Scheme;

let _ = Operator::from_env(Scheme::Fs)
    .expect("must init")
    .layer(RetryLayer::new(ExponentialBackoff::default()));

Retry Logic

For more information about retry design, please refer to RFC-0247: Retryable Error.

Services will return io::ErrorKind::Interrupt if the error is retryable. And operator will retry io::ErrorKind::Interrupt errors until retry times reached.

Logging

OpenDAL has native support for logging.

Logging Layer

LoggingLayer will print logs for every operation with the following rules:

  • OpenDAL will log in structural way.
  • Every operation will start with a started log entry.
  • Every operation will finish with the following status:
    • finished: the operation is successful.
    • errored: the operation returns an expected error like NotFound.
    • failed: the operation returns an unexpected error.

Enable logging layer is easy:

use anyhow::Result;
use opendal::layers::LoggingLayer;
use opendal::Operator;
use opendal::Scheme;

let _ = Operator::from_env(Scheme::Fs)
    .expect("must init")
    .layer(LoggingLayer);

Logging Output

OpenDAL is using log for logging internally.

To enable logging output, please set RUST_LOG:

RUST_LOG=debug ./app

To config logging output, please refer to Configure Logging:

RUST_LOG="info,opendal::services=debug" ./app

Metrics

OpenDAL has native support for metrics.

Metrics Layer

MetricsLayer will add metrics for every operation.

Enable metrics layer requires enable feature layer-metrics:

use anyhow::Result;
use opendal::layers::MetricsLayer;
use opendal::Operator;
use opendal::Scheme;

let _ = Operator::from_env(Scheme::Fs)
    .expect("must init")
    .layer(MetricsLayer);

Metrics Output

OpenDAL is using metrics for metrics internally.

To enable metrics output, please enable one of the exporters that metrics supports.

Take metrics_exporter_prometheus as an example:

let builder = PrometheusBuilder::new();
builder.install().expect("failed to install recorder/exporter");
let handle = builder.install_recorder().expect("failed to install recorder");
let (recorder, exporter) = builder.build().expect("failed to build recorder/exporter");
let recorder = builder.build_recorder().expect("failed to build recorder");

Tracing

OpenDAL has native support for tracing.

Tracing Layer

TracingLayer will add tracing for OpenDAL operations.

Enable tracing layer requires enable feature layer-tracing:

use anyhow::Result;
use opendal::layers::TracingLayer;
use opendal::Operator;
use opendal::Scheme;

let _ = Operator::from_env(Scheme::Fs)
    .expect("must init")
    .layer(TracingLayer);

Tracing Output

OpenDAL is using tracing for tracing internally.

To enable tracing output, please init one of the subscribers that tracing supports.

For example:

extern crate tracing;

let my_subscriber = FooSubscriber::new();
tracing::subscriber::set_global_default(my_subscriber)
    .expect("setting tracing default failed");

For real-world usage, please take a look at tracing-opentelemetry.

Walk Dir

OpenDAL has native walk dir support via BatchOperator.

OpenDAL supports two ways to walk a dir:

  • bottom up
  • top down

Bottom up

Bottom up will list from the most inner dirs.

Given the following file tree:

.
├── dir_x/
│   ├── dir_y/
│   │   ├── dir_z/
│   │   └── file_c
│   └── file_b
└── file_a

The output could be:

dir_x/dir_y/dir_z/file_c
dir_x/dir_y/dir_z/
dir_x/dir_y/file_b
dir_x/dir_y/
dir_x/file_a
dir_x/

Refer to BottomUpWalker for more information.

let mut ds = op.batch().walk_bottom_up();

while let Some(de) = ds.try_next().await? {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

Top down

Top down will list from the most outer dirs.

Given the following file tree:

.
├── dir_x/
│   ├── dir_y/
│   │   ├── dir_z/
│   │   └── file_c
│   └── file_b
└── file_a

The output could be:

dir_x/
dir_x/file_a
dir_x/dir_y/
dir_x/dir_y/file_b
dir_x/dir_y/dir_z/
dir_x/dir_y/dir_z/file_c

Refer to TopDownWalker for more information.

let mut ds = op.batch().walk_top_down();

while let Some(de) = ds.try_next().await? {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

Remove a dir recursively

OpenDAL has native support for remove a dir recursively via BatchOperator.

op.batch().remove_all("path/to/dir").await?

Use this function in cautions to avoid unexpected data loss.

Read compressed files

OpenDAL has native decompress support.

To enable decompress features, we need to specify it in Cargo.toml

opendal = { version = "0.7", features = ["compress"]}

Read into bytes

Use decompress_read to read compressed file into bytes.

use opendal::services::memory;
use std::io::Result;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    // let op = Operator::new(memory::Backend::build().finish().await?);
    let o = op.object("path/to/file.gz");
    let bs: Option<Vec<u8>> = o.decompress_read().await?;
    Ok(())
}

Or, specify the compress algorithm instead by decompress_read_with:

use opendal::services::memory;
use std::io::Result;
use opendal::Operator;
use opendal::io_util::CompressAlgorithm;

#[tokio::main]
async fn main() -> Result<()> {
    // let op = Operator::new(memory::Backend::build().finish().await?);
    let o = op.object("path/to/file.gz");
    let bs: Vec<u8> = o.decompress_read_with(CompressAlgorithm::Gzip).await?;
    Ok(())
}

Read as reader

Use decompress_reader to read compressed file as reader.

use opendal::services::memory;
use std::io::Result;
use opendal::Operator;
use opendal::BytesReader;

#[tokio::main]
async fn main() -> Result<()> {
    // let op = Operator::new(memory::Backend::build().finish().await?);
    let o = op.object("path/to/file.gz");
    let r: Option<BytesReader> = o.decompress_reader().await?;
    Ok(())
}

Or, specify the compress algorithm instead by decompress_reader_with:

use opendal::services::memory;
use std::io::Result;
use opendal::Operator;
use opendal::BytesReader;
use opendal::io_util::CompressAlgorithm;

#[tokio::main]
async fn main() -> Result<()> {
    // let op = Operator::new(memory::Backend::build().finish().await?);
    let o = op.object("path/to/file.gz");
    let bs: BytesReader = o.decompress_reader_with(CompressAlgorithm::Gzip).await?;
    Ok(())
}

Presign

OpenDAL can presign an operation to generate a presigned URL.

Refer to RFC-0413: Presign for more information.

Download

let op = Operator::from_env(Scheme::S3).await?;
let signed_req = op.object("test").presign_read(Duration::hours(1))?;
  • signed_req.method(): GET
  • signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
  • signed_req.headers(): { "host": "s3.amazonaws.com" }

We can download this object via curl or other tools without credentials:

curl "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -O /tmp/test.txt

Upload

let op = Operator::from_env(Scheme::S3).await?;
let signed_req = op.object("test").presign_write(Duration::hours(1))?;
  • signed_req.method(): PUT
  • signed_req.uri(): https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>
  • signed_req.headers(): { "host": "s3.amazonaws.com" }

We can upload file as this object via curl or other tools without credential:

curl -X PUT "https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>" -d "Hello, World!"

Enable Server Side Encryption

OpenDAL has native support for server side encryption.

S3

NOTE: they can't be enabled at the same time.

Enable SSE-KMS with aws managed KMS key:

//! Example for initiating a s3 backend with SSE-KMS and AWS managed key.
use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder: Builder = s3::Builder::default();

    // Setup builders

    // Enable SSE-KMS with aws managed kms key
    builder.server_side_encryption_with_aws_managed_kms_key();

    let op = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Enable SSE-KMS with customer managed KMS key:

//! Example for initiating a s3 backend with SSE-KMS and customer managed key.
use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder: Builder = s3::Builder::default();

    // Setup builders

    // Enable SSE-KMS with customer managed kms key
    builder.server_side_encryption_with_customer_managed_kms_key("aws_kms_key_id");

    let op = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Enable SSE-S3:

//! Example for initiating a s3 backend with SSE-S3
use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder: Builder = s3::Builder::default();

    // Setup builders

    // Enable SSE-S3
    builder.server_side_encryption_with_s3_key();

    let op = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Enable SSE-C:

//! Example for initiating a s3 backend with SSE-C.
use anyhow::Result;
use log::info;
use opendal::services::s3;
use opendal::services::s3::Builder;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let mut builder: Builder = s3::Builder::default();

    // Setup builders

    // Enable SSE-C
    builder.server_side_encryption_with_customer_key("AES256", "customer_key".as_bytes());

    let op = Operator::new(builder.build()?);
    info!("operator: {:?}", op);

    // Writing your testing code here.

    Ok(())
}

Comparisons

In this section, we will compare opendal with other projects to find out the differences and areas that opendal can improve.

All documents listed should be treated as highly biased. Because:

  • OpenDAL's maintainers and contributors write them.
  • Writers may not be familiar with the compared projects (at least not as familiar as with OpenDAL)

Let's see OpenDAL:

OpenDAL vs object_store

NOTE: This document is written by OpenDAL's maintainers and not reviewed by object_store's maintainers. So it could not be very objective.

About object_store

object_store is

A focused, easy to use, idiomatic, high performance, async object store library interacting with object stores.

It was initially developed for InfluxDB IOx and later split out and donated to Apache Arrow.

Similarities

Language

Yes, of course. Both opendal and object_store are developed in Rust, a language empowering everyone to build reliable and efficient software.

License

Both opendal and object_store are licensed under Apache 2.0.

Domain

Both opendal and object_store can be used to access data stored on object storage services. The primary users of those projects are both cloud-native databases too:

  • opendal is mainly used by databend: A modern Elasticity and Performance cloud data warehouse
  • object_store is mainly used by:
    • datafusion: Apache Arrow DataFusion SQL Query Engine
    • Influxdb IOx: The new core of InfluxDB is written in Rust on top of Apache Arrow.

Differences

Owner

object_store is a part of Apache Arrow which means it's hosted and maintained by Apache Software Foundation.

opendal is now hosted and maintained by Datafuse Labs, a new start-up focusing on building data cloud.

opendal has a plan to be donated to ASF. We are at the very early stage of preparing the donation. Welcome any suggestions and help!

Vision

opendal is Open Data Access Layer that accesses data freely, painlessly, and efficiently. object_store is more focused on async object store support.

You will see the different visions lead to very different routes.

Design

object_store exposed a trait called ObjectStore to users.

Users need to build a dyn ObjectStore and operate on it directly:

let object_store: Arc<dyn ObjectStore> = Arc::new(get_object_store());
let path: Path = "data/file01.parquet".try_into().unwrap();
let stream = object_store
    .get(&path)
    .await
    .unwrap()
    .into_stream();

opendal has a similar trait called Accessor.

But opendal don't expose this trait to end users directly. Instead, opendal expose a new struct called Operator and builds public API on it.

let op: Operator = Operator::from_env(Scheme::S3)?;
let r = op.object("data/file01.parquet").reader().await.unwrap();

Interception

Both object_store and opendal provide a mechanism to intercept operations.

object_store called Adapters:

let object_store = ThrottledStore::new(get_object_store(), ThrottleConfig::default())

opendal called Layer:

let op = op.layer(TracingLayer).layer(MetricsLayer);

At the time of writing:

object_store (v0.5.0) supports:

  • ThrottleStore: Rate Throttling
  • LimitStore: Concurrent Request Limit

opendal supports:

  • ImmutableIndexLayer: immutable in-memory index.
  • LoggingLayer: logging.
  • MetadataCacheLayer: metadata cache.
  • ContentCacheLayer: content data cache.
  • MetricsLayer: metrics
  • RetryLayer: retry
  • SubdirLayer: Allow switch directory without changing original operator.
  • TracingLayer: tracing

Services

opendal and object_store have different visions, so they have other services support:

serviceopendalobject_store
azblobYY
fsYY
ftpYN
gcsYY
hdfsYY (via datafusion-objectstore-hdfs)
httpY (read only)N
ipfsY (read only)N
ipmfsYN
memoryYY
obsYN
s3YY

opendal has an idea called AccessorCapability, so it's services may have different capability sets. For example, opendal's http and ipfs are read only.

Features

opendal and object_store have different visions, so they have different feature sets:

opendalobject_storenotes
metadata-get some metadata from underlying storage
createput-
readget-
readget_range-
-get_rangesopendal doesn't support read multiple ranges
writeput-
stathead-
deletedelete-
-listopendal doesn't support list with prefix
listlist_with_delimiter-
-copy-
-copy_if_not_exists-
-rename-
-rename_if_not_exists-
presign-get a presign URL of object
multipartmultipartboth support, but API is different
blocking-opendal supports blocking API

Demo show

The most straightforward complete demo how to read a file from s3:

opendal

let mut builder = s3::Builder::default();
builder.bucket("example");
builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");

let store = Operator::new(builder.build()?);
let r = store.object("data.parquet").reader().await?;

object_store

let mut builder = AmazonS3Builder::new()
.with_bucket_name("example")
.with_access_key_id("access_key_id")
.with_secret_access_key("secret_access_key");

let store = Arc::new(builder.build()?);
let path: Path = "data.parquet".try_into().unwrap();
let stream = store.get(&path).await()?.into_stream();

Upgrade

This document intends to record upgrade and migrate procedures while OpenDAL meets breaking changes.

Upgrade to v0.21

v0.21 is an internal refactor version of OpenDAL. In this version, we refactored our error handling and our Accessor APIs. Thanks to those internal changes, we added an object-level metadata cache, making it nearly zero cost to reuse existing metadata continuously.

Let's start with our errors.

Error Handling

As described in RFC-0977: Refactor Error, we refactor opendal error by a new error called opendal::Error.

This change will affect all APIs that are used to return io::Error.

To migrate this, please replace std::io::Error with opendal::Error:

- use std::io::Result;
+ use opendal::Result;

And the following error kinds should be updated:

  • std::io::ErrorKind::NotFound => opendal::ErrorKind::ObjectNotFound
  • std::io::ErrorKind::PermissionDenied => opendal::ErrorKind::ObjectPermissionDenied

And since v0.21, we will return errors ObjectIsADirectory and ObjectNotADirectory instead of anyhow::Error.

Accessor API

In v0.21, we refactor the whole Accessor's API:

- async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<u64>
+ async fn write(&self, path: &str, args: OpWrite, r: BytesReader) -> Result<RpWrite>

Since v0.21, we will return a reply struct for different operations called RpWrite instead of an exact type. We can split OpenDAL's public API and raw API with this change.

ObjectList and ObjectPage

Since v0.21, Accessor will return ObjectPager for List:

- async fn list(&self, path: &str, args: OpList) -> Result<ObjectStreamer>
+ async fn list(&self, path: &str, args: OpList) -> Result<(RpList, ObjectPager)>

And Object will return an ObjectLister which is built upon ObjectPage:

pub async fn list(&self) -> Result<ObjectLister> { ... }

ObjectLister can be used as an object stream as before. It also provides the function next_page to get the underlying pages directly:

impl ObjectLister {
    pub async fn next_page(&mut self) -> Result<Option<Vec<Object>>>;
}

Code Layout

Since v0.21, we have categorized all APIs into public and raw.

Public APIs are exposed under opendal::Xxx; they are user-face APIs that are easy to use and understand.

Raw APIs are exposed under opendal::raw::Xxx; they are implementation details for underlying services and layers.

Please replace all usage of opendal::io_util::* and opendal::http_util::* to opendal::raw::* instead.

With this change, new users of OpenDAL maybe be it easier to get started.

Summary

Sorry for introducing too much breaking change in a single version. This version can be a solid version for preparing OpenDAL v1.0.

Upgrade to v0.20

v0.20 is a big release that we introduce a lot of performance related changes.

To make the best of information from read operation, we propose and implemented RFC-0926: Object Reader. By this RFC, we can fetch content length from ObjectReader now!

pub struct ObjectReader {
    inner: BytesReader
    meta: ObjectMetadata,
}

impl ObjectReader {
    pub fn content_length(&self) -> u64 {}
    pub fn last_modified(&self) -> Option<OffsetDateTime> {}
    pub fn etag(&self) -> Option<String> {}
}

To make this happen, we changed our Accessor API:

- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}

All layers should be updated to meet this change. Also, it's required to return content_length while building ObjectReader. Please make sure the returning ObjectMetadata is used correctly.

Upgrade to v0.19

OpenDAL deprecate some features:

  • serde: We will enable it by default.
  • layers-retry: We will enable retry support by default.
  • layers-metadata-cache: We will enable it by default.

Deprecated types like DirEntry has been removed.

Upgrade to v0.18

OpenDAL v0.18 introduces the following breaking changes:

  • Deprecated feature flag services-http has been removed.
  • All DirXxx items have been renamed to ObjectXxx to make them more consistent.
    • DirEntry -> ObjectEntry
    • DirStream -> ObjectStream
    • DirStreamer -> ObjectStream
    • DirIterate -> ObjectIterate
    • DirIterator -> ObjectIterator

Besides, we also make a big change to our ObjectEntry API. Since v0.18, we can fully reuse the metadata that fetched during list. Take entry.content_length() for example:

  • If content_lenght is already known, we will return directly.
  • If not, we will check if the object entry is complete:
    • If complete, the entry already fetched all metadata that it could have, return directly.
    • If not, we will send a stat call to get the metadata and refresh our cache.

This change means:

  • All API like content_length will be changed into async functions.
  • metadata and blocking_metadata will not return errors anymore.
  • To retrieve the latest meta, please use entry.into_object().metadata() instead.

Upgrade to v0.17

OpenDAL v0.17 refactor the Accessor to make space for future features.

We move path String out of the OpXxx to function args so that we don't need to clone twice.

- async fn read(&self, args: OpRead) -> Result<BytesReader>
+ async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader>

For more information about this change, please refer to RFC-0661: Path In Accessor.

And since OpenDAL v0.17, we will use rustls as default tls engine for our underlying http client. Since this release, we will not depend on openssl anymore.

Upgrade to v0.16

OpenDAL v0.16 refactor the internal implementation of http service. Since v0.16, http service can be used directly without enabling services-http feature. Accompany by these changes, http service has the following breaking changes:

  • services-http feature has been deprecated. Enabling services-http is a no-op now.
  • http service is read only services and can't be used to list or write.

OpenDAL introduces a new layer ImmutableIndexLayer that can add list capability for services:

use opendal::layers::ImmutableIndexLayer;
use opendal::Operator;
use opendal::Scheme;

async fn main() {
    let mut iil = ImmutableIndexLayer::default();

    for i in ["file", "dir/", "dir/file", "dir_without_prefix/file"] {
        iil.insert(i.to_string())
    }

    let op = Operator::from_env(Scheme::Http)?.layer(iil);
}

For more information about this change, please refer to RFC-0627: Split Capabilities.

Upgrade to v0.14

OpenDAL v0.14 removed all deprecated APIs in previous versions, including:

  • Operator::with_backoff in v0.13
  • All services Builder::finish() in v0.12
  • All services Backend::build() in v0.12

Please visit related version's upgrade guide for migration.

And in OpenDAL v0.14, we introduce a break change for write operations.

pub trait Accessor {
    - async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
    + async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
}

The following APIs have affected by this change:

  • Object::write now accept impl Into<Vec<u8>> instead of AsRef<&[u8]>
  • Object::writer has been removed.
  • Object::write_from has been added to support write from a reader.
  • All layers should be refactored to adapt new Accessor trait.

For more information about this change, please refer to RFC-0554: Write Refactor.

Upgrade to v0.13

OpenDAL deprecate Operator::with_backoff since v0.13.

Please use RetryLayer instead:

use anyhow::Result;
use backon::ExponentialBackoff;
use opendal::layers::RetryLayer;
use opendal::Operator;
use opendal::Scheme;

let _ = Operator::from_env(Scheme::Fs)
    .expect("must init")
    .layer(RetryLayer::new(ExponentialBackoff::default()));

Upgrade to v0.12

OpenDAL introduces breaking changes for services initiation.

Since v0.12, Operator::new will accept impl Accessor + 'static instead of Arc<dyn Accessor>:

impl Operator {
    pub fn new(accessor: impl Accessor + 'static) -> Self { .. }
}

Every service's Builder now have a build() API which can be run without async:

let mut builder = fs::Builder::default();
let op: Operator = Operator::new(builder.build()?);

Along with these changes, Operator::from_iter and Operator::from_env now is a blocking API too.

For more information about this change, please refer to RFC-0501: New Builder.

The following APIs have been deprecated:

  • All services Builder::finish() (replaced by Builder::build())
  • All services Backend::build() (replace by Builder::default())

The following APIs have been removed:

  • public struct Metadata (deprecated in v0.8, replaced by ObjectMetadata)

Upgrade to v0.8

OpenDAL introduces a breaking change of list related operations in v0.8.

Since v0.8, list will return DirStreamer instead:

pub trait Accessor: Send + Sync + Debug {
    async fn list(&self, args: &OpList) -> Result<DirStreamer> {}
}

DirStreamer streams DirEntry which carries ObjectMode, so that we don't need an extra call to get object mode:

impl DirEntry {
    pub fn mode(&self) -> ObjectMode {
        self.mode
    }
}

And DirEntry can be converted into Object without overhead:

let o: Object = de.into()

Since v0.8, opendal::Metadata has been deprecated by opendal::ObjectMetadata.

Upgrade to v0.7

OpenDAL introduces a breaking change of decompress_read related in v0.7.

Since v0.7, decompress_read and decompress_reader will return Ok(None) while OpenDAL can't detect the correct compress algorithm.

impl Object {
    pub async fn decompress_read(&self) -> Result<Option<Vec<u8>>> {}
    pub async fn decompress_reader(&self) -> Result<Option<impl BytesRead>> {}
}

So users should match and check the None case:

let bs = o.decompress_read().await?.expect("must have valid compress algorithm");

Upgrade to v0.4

OpenDAL introduces many breaking changes in v0.4.

Object::reader() is not AsyncSeek anymore

Since v0.4, Object::reader() will return impl BytesRead instead of Reader that implements AsyncRead and AsyncSeek. Users who want AsyncSeek please wrapped with opendal::io_util::seekable_read:

use opendal::io_util::seekable_read;

let o = op.object("test");
let mut r = seekable_read(&o, 10..);
r.seek(SeekFrom::Current(10)).await?;
let mut bs = vec![0;10];
r.read(&mut bs).await?;

Use RangeBounds instead

Since v0.4, the following APIs will be removed.

  • Object::limited_reader(size: u64)
  • Object::offset_reader(offset: u64)
  • Object::range_reader(offset: u64, size: u64)

Instead, OpenDAL is providing a more general range_reader powered by RangeBounds:

pub async fn range_reader(&self, range: impl RangeBounds<u64>) -> Result<impl BytesRead>

Users can use their familiar rust range syntax:

let r = o.range_reader(1024..2048).await?;

Return io::Result instead

Since v0.4, all functions in OpenDAL will return std::io::Result instead.

Please check via std::io::ErrorKind directly:

use std::io::ErrorKind;

if let Err(e) = op.object("test_file").metadata().await {
    if e.kind() == ErrorKind::NotFound {
        println!("object not exist")
    }
}

Removing Credential

Since v0.4, Credential has been removed, please use the API provided by Builder directly.

builder.access_key_id("access_key_id");
builder.secret_access_key("secret_access_key");

Write returns BytesWriter instead

Since v0.4, Accessor::write will return a BytesWriter instead accepting a BoxedAsyncReader.

Along with this change, the old Writer has been replaced by a new set of write functions:

pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}

Users can write into an object more easily:

let _ = op.object("path/to/file").write("Hello, World!").await?;

io_util replaces readers

Since v0.4, mod io_util will replace readers. In io_utils, OpenDAL provides helpful functions like:

  • into_reader: Convert BytesStream into BytesRead
  • into_sink: Convert BytesWrite into BytesSink
  • into_stream: Convert BytesRead into BytesStream
  • into_writer: Convert BytesSink into BytesWrite
  • observe_read: Add callback for BytesReader
  • observe_write: Add callback for BytesWrite

New type alias

For better naming, types that OpenDAL returns have been renamed:

  • AsyncRead + Unpin + Send => BytesRead
  • BoxedAsyncReader => BytesReader
  • AsyncWrite + Unpin + Send => BytesWrite
  • BoxedAsyncWriter => BytesWriter
  • ObjectStream => ObjectStreamer

Internal

This section is used to host internal docs for services/layers implementer.

Implement service

Implement a service mainly two parts:

  • Backend: the struct that implement Accessor trait.
  • Builder: the struct that provide fn build(&mut self) -> Result<Backend>.

Backend

Backend needs to implement Accessor:

#[async_trait]
impl Accessor for Backend {}

Metadata

The only function that required to implement is metadata:

#[async_trait]
impl Accessor for Backend {
    fn metadata(&self) -> AccessorMetadata {
        let mut am = AccessorMetadata::default();
        am.set_scheme(Scheme::Xxx)
            .set_root(&self.root)
            .set_capabilities(AccessorCapability::Read);
        am
    }
}

In this function, backend needs to return:

  • scheme: The Scheme of this backend.
  • root: The root path of current backend.
  • capabilities: The capabilities that this backend have.
  • name: The name of this backend (for object storage services, it's bucket name)

Available capabilities including:

  • Read: Set this capability if this backend can read and stat.
  • Write: Set this capability if this backend can write and delete.
  • List: Set this capability if this backend can list.
  • Presign: Set this capability if this backend can presign.
  • Multipart: Set this capability if this backend can maintain multipart.
  • Blocking: Set this capability if this backend can be used in blocking context.

Builder

Builder must implement Default and the following functions:

impl Builder {
    pub fn build(&mut self) -> Result<Backend> {}
}

Builder's API is part of OpenDAL public API, please don't mark function as pub unless needed.

RFCs of OpenDAL

RFCs power OpenDAL's development.

To add new features and big refactors:

  • Start a new RFCs with the template 0000-example.
  • Submit PR and assign the RFC number with the PR number.
  • Adding into index of rfcs for better rendered.
  • Request reviews from OpenDAL maintainers.
  • Create a tracking issue and update links in RFC after approval.

To find the complete list of approved RFCs, please visit here.

Some useful tips:

  • Start a pre-propose in discussion to communicate quickly.
  • The proposer of RFC may not be the same person as the implementor. Try to include enough information in RFC itself.

Summary

Refactor API in object native way to make it easier to user.

Motivation

opendal is not easy to use.

In our early adoption project databend, we can see a lot of code looks like:

let data_accessor = self.data_accessor.clone();
let path = self.path.clone();
let reader = SeekableReader::new(data_accessor, path.as_str(), stream_len);
let reader = BufReader::with_capacity(read_buffer_size as usize, reader);
Self::read_column(reader, &col_meta, data_type.clone(), arrow_type.clone()).await

And

op.stat(&path).run().await

Conclusion

So in this proposal, I expect to address those problems. After implementing this proposal, we have a faster and easier-to-use opendal.

Guide-level explanation

To operate on an object, we will use Operator::object() to create a new handler:

let o = op.object("path/to/file");

All operations that are available for Object for now includes:

  • metadata: get object metadata (return an error if not exist).
  • delete: delete an object.
  • reader: create a new reader to read data from this object.
  • writer: create a new writer to write data into this object.

Here is an example:

use anyhow::Result;
use futures::AsyncReadExt;

use opendal::services::fs;
use opendal::Operator;

#[tokio::main]
async fn main() -> Result<()> {
    let op = Operator::new(fs::Backend::build().root("/tmp").finish().await?);

    let o = op.object("test_file");

    // Write data info file;
    let w = o.writer();
    let n = w
        .write_bytes("Hello, World!".to_string().into_bytes())
        .await?;
    assert_eq!(n, 13);

    // Read data from file;
    let mut r = o.reader();
    let mut buf = vec![];
    let n = r.read_to_end(&mut buf).await?;
    assert_eq!(n, 13);
    assert_eq!(String::from_utf8_lossy(&buf), "Hello, World!");

    // Get file's Metadata
    let meta = o.metadata().await?;
    assert_eq!(meta.content_length(), 13);

    // Delete file.
    o.delete().await?;

    Ok(())
}

Reference-level explanation

Native Reader support

We will provide a Reader (which implement both AsyncRead + AsyncSeek) for user instead of just a AsyncRead. In this Reader, we will:

  • Not maintain internal buffer: caller can decide to wrap into BufReader.
  • Only rely on accessor's read and stat operations.

To avoid the extra cost for stat, we will:

  • Allow user specify total_size for Reader.
  • Lazily Send stat while the first time SeekFrom::End()

To avoid the extra cost for poll_read, we will:

  • Keep the underlying BoxedAsyncRead open, so that we can reuse the same connection/fd.

With these change, we can improve the Reader performance both on local fs and remote storage:

  • fs, before
Benchmarking fs/bench_read/64226295-b7a7-416e-94ce-666ac3ab037b:
                        time:   [16.060 ms 17.109 ms 18.124 ms]
                        thrpt:  [882.82 MiB/s 935.20 MiB/s 996.24 MiB/s]

Benchmarking fs/bench_buf_read/64226295-b7a7-416e-94ce-666ac3ab037b:
                        time:   [14.779 ms 14.857 ms 14.938 ms]
                        thrpt:  [1.0460 GiB/s 1.0517 GiB/s 1.0572 GiB/s]
  • fs, after
Benchmarking fs/bench_read/df531bc7-54c8-43b6-b412-e4f7b9589876:
                        time:   [14.654 ms 15.452 ms 16.273 ms]
                        thrpt:  [983.20 MiB/s 1.0112 GiB/s 1.0663 GiB/s]

Benchmarking fs/bench_buf_read/df531bc7-54c8-43b6-b412-e4f7b9589876:
                        time:   [5.5589 ms 5.5825 ms 5.6076 ms]
                        thrpt:  [2.7864 GiB/s 2.7989 GiB/s 2.8108 GiB/s]
  • s3, before
Benchmarking s3/bench_read/72025a81-a4b6-46dc-b485-8d875d23c3a5:
                        time:   [4.8315 ms 4.9331 ms 5.0403 ms]
                        thrpt:  [3.1000 GiB/s 3.1674 GiB/s 3.2340 GiB/s]

Benchmarking s3/bench_buf_read/72025a81-a4b6-46dc-b485-8d875d23c3a5:
                        time:   [16.246 ms 16.539 ms 16.833 ms]
                        thrpt:  [950.52 MiB/s 967.39 MiB/s 984.84 MiB/s]
  • s3, after
Benchmarking s3/bench_read/6971c464-15f7-48d6-b69c-c8abc7774802:
                        time:   [4.4222 ms 4.5685 ms 4.7181 ms]
                        thrpt:  [3.3117 GiB/s 3.4202 GiB/s 3.5333 GiB/s]

Benchmarking s3/bench_buf_read/6971c464-15f7-48d6-b69c-c8abc7774802:
                        time:   [5.5598 ms 5.7174 ms 5.8691 ms]
                        thrpt:  [2.6622 GiB/s 2.7329 GiB/s 2.8103 GiB/s]

Object API

Other changes are just a re-order of APIs.

  • Operator::read() -> BoxedAsyncRead => Object::reader() -> Reader
  • Operator::write(r: BoxedAsyncRead, size: u64) => Object::writer() -> Writer
  • Operator::stat() -> Object => Object::stat() -> Metadata
  • Operator::delete() => Object::delete()

Drawbacks

None.

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

  • Implement AsyncWrite for Writer so that we can use Writer easier.
  • Implement Operator::objects() to return an object iterator.

Summary

Enhanced error handling for OpenDAL.

Motivation

OpenDAL didn't handle errors correctly.

fn parse_unexpect_error<E>(_: SdkError<E>, path: &str) -> Error {
    Error::Unexpected(path.to_string())
}

Most time, we return a path that is meaningless for debugging.

There are two issues about this shortcoming:

First, we can't check ErrorKind quickly. We have to use matches for the help:

assert!(
    matches!(
        result.err().unwrap(),
        opendal::error::Error::ObjectNotExist(_)
    ),
);

Then, we didn't bring enough information for users to debug what happened inside OpenDAL.

So we must handle errors correctly, so that:

  • We can check the Kind to know what error happened.
  • We can read context to know more details.
  • We can get the source of this error to know more details.

Guide-level explanation

Now we are trying to get an object's metadata:

let meta = o.metadata().await;

Unfortunately, the Object does not exist, so we can check out what happened.

if let Err(e) = meta {
    if e.kind() == Kind::ObjectNotExist {
        // Handle this error
    }
}

It's possible that we don't care about other errors. It's OK to log it out:

if let Err(e) = meta {
    if e.kind() == Kind::ObjectNotExist {
        // Handle this error
    } else {
        error!("{e}");
    }
}

For a backend implementor, we can provide as much information as possible. For example, we can return bucket is empty to let the user know:

return Err(Error::Backend {
    kind: Kind::BackendConfigurationInvalid,
    context: HashMap::from([("bucket".to_string(), "".to_string())]),
    source: anyhow!("bucket is empty"),
});

Or, we can return an underlying error to let users figure out:

Error::Object {
    kind: Kind::Unexpected,
    op,
    path: path.to_string(),
    source: anyhow::Error::from(err),
}

So our application users will get enough information now:

Object { kind: ObjectNotExist, op: "stat", path: "/tmp/998e4dec-c84b-4164-a7a1-1f140654934f", source: No such file or directory (os error 2) }

Reference-level explanation

We will split Error into Error and Kind.

Kind is an enum organized by different categories.

Every error will map to a kind, which will be in the error message.

pub enum Kind {
    #[error("backend not supported")]
    BackendNotSupported,
    #[error("backend configuration invalid")]
    BackendConfigurationInvalid,

    #[error("object not exist")]
    ObjectNotExist,
    #[error("object permission denied")]
    ObjectPermissionDenied,

    #[error("unexpected")]
    Unexpected,
}

In Error, we will have different struct to carry different contexts:

pub enum Error {
    #[error("{kind}: (context: {context:?}, source: {source})")]
    Backend {
        kind: Kind,
        context: HashMap<String, String>,
        source: anyhow::Error,
    },

    #[error("{kind}: (op: {op}, path: {path}, source: {source})")]
    Object {
        kind: Kind,
        op: &'static str,
        path: String,
        source: anyhow::Error,
    },

    #[error("unexpected: (source: {0})")]
    Unexpected(#[from] anyhow::Error),
}

Every one of them will carry a source: anyhow::Error so that users can get the complete picture of this error. We have implemented Error::kind(), other helper functions are possible, but they are out of this RFC's scope.

pub fn kind(&self) -> Kind {
    match self {
        Error::Backend { kind, .. } => *kind,
        Error::Object { kind, .. } => *kind,
        Error::Unexpected(_) => Kind::Unexpected,
    }
}

The implementor should do their best to carry as much context as possible. Such as, they should return Error::Object to carry the op and path, instead of just returns Error::Unexpected(anyhow::Error::from(err)).

Error::Object {
    kind: Kind::Unexpected,
    op,
    path: path.to_string(),
    source: anyhow::Error::from(err),
}

Drawbacks

None

Rationale and alternatives

Why don't we implement backtrace?

backtrace is not stable yet, and OpenDAL must be compilable on stable Rust.

This proposal doesn't erase the possibility to add support once backtrace is stable.

Prior art

None

Unresolved questions

None

Future possibilities

  • Backtrace support.

Summary

Automatically detecting user's s3 region.

Motivation

Current behavior for region and endpoint is buggy. endpoint=https://s3.amazonaws.com and endpoint="" are expected to be the same, because endpoint="" means take the default value https://s3.amazonaws.com. However, they aren't.

S3 SDK has a mechanism to construct the correct API endpoint. It works like format!("s3.{}.amazonaws.com", region) internally. But if we specify the endpoint to https://s3.amazonaws.com, SDK will take this endpoint static.

So users could meet errors like:

attempting to access must be addressed using the specified endpoint

Automatically detecting the user's s3 region will help resolve this problem. Users don't need to care about the region anymore, OpenDAL will figure it out. Everything works regardless of whether the input is s3.amazonaws.com or s3.us-east-1.amazonaws.com.

Guide-level explanation

OpenDAL will remove region option, and users only need to set the endpoint now.

Valid input including:

  • https://s3.amazonaws.com
  • https://s3.us-east-1.amazonaws.com
  • https://oss-ap-northeast-1.aliyuncs.com
  • http://127.0.0.1:9000

OpenDAL will handle the region internally and automatically.

Reference-level explanation

S3 services support mechanism to indicate the correct region on itself.

Sending a HEAD request to <endpoint>/<bucket> will get a response like:

:) curl -I https://s3.amazonaws.com/databend-shared
HTTP/1.1 301 Moved Permanently
x-amz-bucket-region: us-east-2
x-amz-request-id: NPYSWK7WXJD1KQG7
x-amz-id-2: 3FJSJ5HACKqLbeeXBUUE3GoPL1IGDjLl6SZx/fw2MS+k0GND0UwDib5YQXE6CThiQxpYBWZjgxs=
Content-Type: application/xml
Date: Thu, 24 Feb 2022 05:15:13 GMT
Server: AmazonS3

x-amz-bucket-region: us-east-2 will be returned, and we can use this region to construct the correct endpoint for this bucket:

:) curl -I https://s3.us-east-2.amazonaws.com/databend-shared
HTTP/1.1 403 Forbidden
x-amz-bucket-region: us-east-2
x-amz-request-id: 98CN5MYV3GQ1XMPY
x-amz-id-2: Tdxy36bRRP21Oip18KMQ7FG63MTeXOpXdd5/N3izFH0oalPODVaRlpCkDU3oUN0HIE24/ezX5Dc=
Content-Type: application/xml
Date: Thu, 24 Feb 2022 05:16:57 GMT
Server: AmazonS3

It also works for S3 compilable services like minio:

# Start minio with `MINIO_SITE_REGION` configured
:) MINIO_SITE_REGION=test minio server .
# Sending request to minio bucket
:) curl -I 127.0.0.1:9900/databend
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Content-Length: 0
Content-Security-Policy: block-all-mixed-content
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Bucket-Region: test
X-Amz-Request-Id: 16D6A12DCA57E0FA
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
Date: Thu, 24 Feb 2022 05:18:51 GMT

We can use this mechanism to detect region automatically. The algorithm works as follows:

  • If endpoint is empty, fill it will https://s3.amazonaws.com and the corresponding template: https://s3.{region}.amazonaws.com.
  • Sending a HEAD request to <endpoint>/<bucket>.
  • If got 200 or 403 response, the endpoint works.
    • Use this endpoint directly without filling the template.
    • Take the header x-amz-bucket-region as the region to fill the endpoint.
    • Use the fallback value us-east-1 to make SDK happy if the header not exists.
  • If got a 301 response, the endpoint needs construction.
    • Take the header x-amz-bucket-region as the region to fill the endpoint.
    • Return an error to the user if not exist.
  • If got 404, the bucket could not exist, or the endpoint is incorrect.
    • Return an error to the user.

Drawbacks

None.

Rationale and alternatives

Use virtual style <bucket>.<endpoint>?

The virtual style works too. But not all services support this kind of API endpoint. For example, using http://testbucket.127.0.0.1 is wrong, and we need to do extra checks.

Using <endpoint>/<bucket> makes everything easier.

Use ListBuckets API?

ListBuckets requires higher permission than normal bucket read and write operations. It's better to finish the job without requesting more permission.

Misbehavior S3 Compilable Services

Many services didn't implement S3 API correctly.

Aliyun OSS will return 404 for every bucket:

:) curl -I https://aliyuncs.com/<my-existing-bucket>
HTTP/2 404
date: Thu, 24 Feb 2022 05:32:57 GMT
content-type: text/html
content-length: 690
ufe-result: A6
set-cookie: thw=cn; Path=/; Domain=.taobao.com; Expires=Fri, 24-Feb-23 05:32:57 GMT;
server: Tengine/Aserver

QingStor Object Storage will return 307 with the Location header:

:) curl -I https://s3.qingstor.com/community
HTTP/1.1 301 Moved Permanently
Server: nginx/1.13.6
Date: Thu, 24 Feb 2022 05:33:55 GMT
Connection: keep-alive
Location: https://pek3a.s3.qingstor.com/community
X-Qs-Request-Id: 05b83b615c801a3d

In this proposal, we will not figure them out. It's easier for the user to fill the correct endpoint instead of automatically detecting them.

Prior art

None

Unresolved questions

None

Future possibilities

None

Summary

Allow user to read dir via ObjectStream.

Motivation

Users need readdir support in OpenDAL: Implement List support. Take databend for example, with List support, we can implement copy from s3://bucket/path/to/dir instead of only s3://bucket/path/to/file.

Guide-level explanation

Operator supports new action called objects("path/to/dir") which returns a ObjectStream, we can iterator current dir like std::fs::ReadDir:

let mut obs = op.objects("").map(|o| o.expect("list object"));
while let Some(o) = obs.next().await {
    // Do something upon `Object`.
}

To better support different file modes, there is a new object meta called ObjectMode:

let meta = o.metadata().await?;
let mode = meta.mode();
if mode.contains(ObjectMode::FILE) {
    // Do something on a file object.
} else if mode.contains(ObjectMode::DIR) {
    // Do something on a dir object.
}

We will try to cache some object metadata so that users can reduce stat calls:

let meta = o.metadata_cached().await?;

o.metadata_cached() will return local cached metadata if available.

Reference-level explanation

First, we will add a new API in Accessor:

pub type BoxedObjectStream = Box<dyn futures::Stream<Item = Result<Object>> + Unpin + Send>;

async fn list(&self, args: &OpList) -> Result<BoxedObjectStream> {
    let _ = args;
    unimplemented!()
}

To support options in the future, we will wrap this call via ObjectStream:

pub struct ObjectStream {
    acc: Arc<dyn Accessor>,
    path: String,

    state: State,
}

enum State {
    Idle,
    Sending(BoxFuture<'static, Result<BoxedObjectStream>>),
    Listing(BoxedObjectStream),
}

So the public API to end-users will be:

impl Operator {
    pub fn objects(&self, path: &str) -> ObjectStream {
        ObjectStream::new(self.inner(), path)
    }
}

For cached metadata support, we will add a flag in Metadata:

#[derive(Debug, Clone, Default)]
pub struct Metadata {
    complete: bool,

    path: String,
    mode: Option<ObjectMode>,

    content_length: Option<u64>,
}

And add new API Objbct::metadata_cached():

pub async fn metadata_cached(&mut self) -> Result<&Metadata> {
    if self.meta.complete() {
        return Ok(&self.meta);
    }

    let op = &OpStat::new(self.meta.path());
    self.meta = self.acc.stat(op).await?;

    Ok(&self.meta)
}

The backend implementor must make sure complete is correctly set.

Metadata will be immutable outsides, so all set_xxx APIs will be set to crate public only:

pub(crate) fn set_content_length(&mut self, content_length: u64) -> &mut Self {
    self.content_length = Some(content_length);
    self
}

Drawbacks

None

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

  • More precise field-level metadata cache so that user can send stat only when needed.

Summary

Native support for the limited reader.

Motivation

In proposal object-native-api we introduced Reader, in which we will send request like:

let op = OpRead {
    path: self.path.to_string(),
    offset: Some(self.current_offset()),
    size: None,
};

In this implementation, we depend on the HTTP client to drop the request when we stop reading. However, we always read too much extra data, which decreases our reading performance.

Here is a benchmark around reading the whole file and only reading half:

s3/read/1c741003-40ef-43a9-b23f-b6a32ed7c4c6
                        time:   [7.2697 ms 7.3521 ms 7.4378 ms]
                        thrpt:  [2.1008 GiB/s 2.1252 GiB/s 2.1493 GiB/s]
s3/read_half/1c741003-40ef-43a9-b23f-b6a32ed7c4c6
                        time:   [7.0645 ms 7.1524 ms 7.2473 ms]
                        thrpt:  [1.0780 GiB/s 1.0923 GiB/s 1.1059 GiB/s]

So our current behavior is buggy, and we need more clear API to address that.

Guide-level explanation

We will remove Reader::total_size() from public API instead of adding the following APIs for Object:

pub fn reader(&self) -> Reader {}
pub fn range_reader(&self, offset: u64, size: u64) -> Reader {}
pub fn offset_reader(&self, offset: u64) -> Reader {}
pub fn limited_reader(&self, size: u64) -> Reader {}
  • reader: returns a new reader who can read the whole file.
  • range_reader: returns a ranged reader which read [offset, offset+size).
  • offset_reader: returns a reader from offset [offset:]
  • limited_reader: returns a limited reader [:size]

Take parquet's actual logic as an example. We can rewrite:

async fn _read_single_column_async<'b, R, F>(
    factory: F,
    meta: &ColumnChunkMetaData,
) -> Result<(&ColumnChunkMetaData, Vec<u8>)>
where
    R: AsyncRead + AsyncSeek + Send + Unpin,
    F: Fn() -> BoxFuture<'b, std::io::Result<R>>,
{
    let mut reader = factory().await?;
    let (start, len) = meta.byte_range();
    reader.seek(std::io::SeekFrom::Start(start)).await?;
    let mut chunk = vec![0; len as usize];
    reader.read_exact(&mut chunk).await?;
    Result::Ok((meta, chunk))
}

into

async fn _read_single_column_async<'b, R, F>(
    factory: F,
    meta: &ColumnChunkMetaData,
) -> Result<(&ColumnChunkMetaData, Vec<u8>)>
where
    R: AsyncRead + AsyncSeek + Send + Unpin,
    F: Fn(usize, usize) -> BoxFuture<'b, std::io::Result<R>>,
{
    let (start, len) = meta.byte_range();
    let mut reader = factory(start, len).await?;
    let mut chunk = vec![0; len as usize];
    reader.read_exact(&mut chunk).await?;
    Result::Ok((meta, chunk))
}

So that:

  • No extra data will be read.
  • No extra seek/stat operation is needed.

Reference-level explanation

Inside Reader, we will correctly maintain offset, size, and pos.

  • If offset is None, we will use 0 instead.
  • If size is None, we will use meta.content_length() - self.offset.unwrap_or_default() instead.

We will calculate Reader current offset and size easily:

fn current_offset(&self) -> u64 {
    self.offset.unwrap_or_default() + self.pos
}

fn current_size(&self) -> Option<u64> {
    self.size.map(|v| v - self.pos)
}

Instead of constantly requesting the entire object content, we will set the size:

let op = OpRead {
    path: self.path.to_string(),
    offset: Some(self.current_offset()),
    size: self.current_size(),
};

After this change, we will have a similar throughput for read_all and read_half:

s3/read/6dd40f8d-7455-451e-b510-3b7ac23e0468
                        time:   [4.9554 ms 5.0888 ms 5.2282 ms]
                        thrpt:  [2.9886 GiB/s 3.0704 GiB/s 3.1532 GiB/s]
s3/read_half/6dd40f8d-7455-451e-b510-3b7ac23e0468
                        time:   [3.1868 ms 3.2494 ms 3.3052 ms]
                        thrpt:  [2.3637 GiB/s 2.4043 GiB/s 2.4515 GiB/s]

Drawbacks

None

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

  • Refactor the parquet reading logic to make the most use of range_reader.

Summary

Implement path normalization to enhance user experience.

Motivation

OpenDAL's current path behavior makes users confused:

They are different bugs that reflect the exact root cause: the path is not well normalized.

On local fs, we can read the same path with different path: abc/def/../def, abc/def, abc//def, abc/./def.

There is no magic here: our stdlib does the dirty job. For example:

  • std::path::PathBuf::canonicalize: Returns the canonical, absolute form of the path with all intermediate components normalized and symbolic links resolved.
  • std::path::PathBuf::components: Produces an iterator over the Components of the path. When parsing the path, there is a small amount of normalization...

But for s3 alike storage system, there's no such helpers: abc/def/../def, abc/def, abc//def, abc/./def refers entirely different objects. So users may confuse why I can't get the object with this path.

So OpenDAL needs to implement path normalization to enhance the user experience.

Guide-level explanation

We will do path normalization automatically.

The following rules will be applied (so far):

  • Remove // inside path: op.object("abc/def") and op.object("abc//def") will resolve to the same object.
  • Make sure path under root: op.object("/abc") and op.object("abc") will resolve to the same object.

Other rules still need more consideration to leave them for the future.

Reference-level explanation

We will build the absolute path via {root}/{path} and replace all // into / instead.

Drawbacks

None

Rationale and alternatives

If we build an actual path via {root}/{path}, the link object may be inaccessible.

I don't have good ideas so far. Maybe we can add a new flag to control the link behavior. For now, there's no feature request for link support.

Let's leave for the future to resolve.

S3 URI Clean

For s3, abc//def is different from abc/def indeed. To make it possible to access not normalized path, we can provide a new flag for the builder:

let builder = Backend::build().disable_path_normalization()

In this way, the user can control the path more precisely.

Prior art

None

Unresolved questions

None

Future possibilities

None

Reverted

Summary

Use Stream/Sink instead of AsyncRead in Accessor.

Motivation

Accessor intends to be the underlying trait of all backends for implementors. However, it's not so underlying enough.

Over-wrapped

Accessor returns a BoxedAsyncReader for read operation:

pub type BoxedAsyncReader = Box<dyn AsyncRead + Unpin + Send>;

pub trait Accessor {
    async fn read(&self, args: &OpRead) -> Result<BoxedAsyncReader> {
        let _ = args;
        unimplemented!()
    }
}

And we are exposing Reader, which implements AsyncRead and AsyncSeek to end-users. For every call to Reader::poll_read(), we need:

  • Reader::poll_read()
  • BoxedAsyncReader::poll_read()
  • IntoAsyncRead<ByteStream>::poll_read()
  • ByteStream::poll_next()

If we could return a Stream directly, we can transform the call stack into:

  • Reader::poll_read()
  • ByteStream::poll_next()

In this way, we operate on the underlying IO stream, and the caller must keep track of the reading states.

Inconsistent

OpenDAL's read and write behavior is not consistent.

pub type BoxedAsyncReader = Box<dyn AsyncRead + Unpin + Send>;

pub trait Accessor: Send + Sync + Debug {
    async fn read(&self, args: &OpRead) -> Result<BoxedAsyncReader> {
        let _ = args;
        unimplemented!()
    }
    async fn write(&self, r: BoxedAsyncReader, args: &OpWrite) -> Result<usize> {
        let (_, _) = (r, args);
        unimplemented!()
    }
}

For read, OpenDAL returns a BoxedAsyncReader which users can decide when and how to read data. But for write, OpenDAL accepts a BoxedAsyncReader instead, in which users can't control the writing logic. How large will the writing buffer size be? When to call flush?

Service native optimization

OpenDAL knows more about the service detail, but returning BoxedAsyncReader makes it can't fully use the advantage.

For example, most object storage services use HTTP to transfer data which is TCP stream-based. The most efficient way is to return a full TCP buffer, but users don't know about that. First, users could have continuous small reads on stream. To overcome the poor performance, they have to use BufReader, which adds a new buffering between reading. Then, users don't know the correct (best) buffer size to set.

Via returning a Stream, users could benefit from it in both ways:

  • Users who want underlying control can operate on the Stream directly.
  • Users who don't care about the behavior can use OpenDAL provided Reader, which always adopts the best optimization.

Guide-level explanation

Within the async_streaming_io feature, we will add the following new APIs to Object:

impl Object {
    pub async fn stream(&self, offset: Option<u64>, size: Option<u64>) -> Result<BytesStream> {}
    pub async fn sink(&self, size: u64) -> Result<BytesSink> {}
}

Users can control the underlying logic of those bytes, streams, and sinks.

For example, they can:

  • Read data on demand: stream.next().await
  • Write data on demand: sink.feed(bs).await; sink.close().await;

Based on stream and sink, Object will provide more optimized helper functions like:

  • async read(offset: Option<u64>, size: Option<u64>) -> Result<bytes::Bytes>
  • async write(bs: bytes::Bytes) -> Result<()>

Reference-level explanation

read and write in Accessor will be refactored into streaming-based:

pub type BytesStream =  Box<dyn Stream + Unpin + Send>;
pub type BytesSink =  Box<dyn Sink + Unpin + Send>;

pub trait Accessor: Send + Sync + Debug {
    async fn read(&self, args: &OpRead) -> Result<BytesStream> {
        let _ = args;
        unimplemented!()
    }
    async fn write(&self, args: &OpWrite) -> Result<BytesSink> {
        let _ = args;
        unimplemented!()
    }
}

All other IO functions will be adapted to fit these changes.

For fs, it's simple to implement Stream and Sink for tokio::fs::File.

We will return a BodySinker instead for all HTTP-based storage services. In which we maintain a put_object ResponseFuture that construct by hyper and a sender part of the channel. All data sent by users will be passed to ResponseFuture via the unbuffered channel.

struct BodySinker {
    fut: ResponseFuture,
    sender: Sender<bytes::Bytes>
}

Drawbacks

Performance regression on fs

fs is not stream based backend, and convert from Reader to Stream is not zero cost. Based on benchmark over IntoStream, we can get nearly 70% performance drawback (pure memory):

into_stream/into_stream time:   [1.3046 ms 1.3056 ms 1.3068 ms]
                        thrpt:  [2.9891 GiB/s 2.9919 GiB/s 2.9942 GiB/s]
into_stream/raw_reader  time:   [382.10 us 383.52 us 385.16 us]
                        thrpt:  [10.142 GiB/s 10.185 GiB/s 10.223 GiB/s]

However, real fs is not as fast as memory and most overhead will happen at disk side, so that performance regression is allowed (at least at this time).

Rationale and alternatives

Performance for switching from Reader to Stream

Before

read_full/4.00 KiB      time:   [455.70 us 466.18 us 476.93 us]
                        thrpt:  [8.1904 MiB/s 8.3794 MiB/s 8.5719 MiB/s]
read_full/256 KiB       time:   [530.63 us 544.30 us 557.84 us]
                        thrpt:  [448.16 MiB/s 459.30 MiB/s 471.14 MiB/s]
read_full/4.00 MiB      time:   [1.5569 ms 1.6152 ms 1.6743 ms]
                        thrpt:  [2.3330 GiB/s 2.4184 GiB/s 2.5090 GiB/s]
read_full/16.0 MiB      time:   [5.7337 ms 5.9087 ms 6.0813 ms]
                        thrpt:  [2.5693 GiB/s 2.6444 GiB/s 2.7251 GiB/s]

After

read_full/4.00 KiB      time:   [455.67 us 466.03 us 476.21 us]
                        thrpt:  [8.2027 MiB/s 8.3819 MiB/s 8.5725 MiB/s]
                 change:
                        time:   [-2.1168% +0.6241% +3.8735%] (p = 0.68 > 0.05)
                        thrpt:  [-3.7291% -0.6203% +2.1625%]
                        No change in performance detected.
read_full/256 KiB       time:   [521.04 us 535.20 us 548.74 us]
                        thrpt:  [455.59 MiB/s 467.11 MiB/s 479.81 MiB/s]
                 change:
                        time:   [-7.8470% -4.7987% -1.4955%] (p = 0.01 < 0.05)
                        thrpt:  [+1.5182% +5.0406% +8.5152%]
                        Performance has improved.
read_full/4.00 MiB      time:   [1.4571 ms 1.5184 ms 1.5843 ms]
                        thrpt:  [2.4655 GiB/s 2.5725 GiB/s 2.6808 GiB/s]
                 change:
                        time:   [-5.4403% -1.5696% +2.3719%] (p = 0.44 > 0.05)
                        thrpt:  [-2.3170% +1.5946% +5.7533%]
                        No change in performance detected.
read_full/16.0 MiB      time:   [5.0201 ms 5.2105 ms 5.3986 ms]
                        thrpt:  [2.8943 GiB/s 2.9988 GiB/s 3.1125 GiB/s]
                 change:
                        time:   [-15.917% -11.816% -7.5219%] (p = 0.00 < 0.05)
                        thrpt:  [+8.1337% +13.400% +18.930%]
                        Performance has improved.

Performance for the extra channel in write

Based on the benchmark during research, the unbuffered channel does improve the performance a bit in some cases:

Before:

write_once/4.00 KiB     time:   [564.11 us 575.17 us 586.15 us]
                        thrpt:  [6.6642 MiB/s 6.7914 MiB/s 6.9246 MiB/s]
write_once/256 KiB      time:   [1.3600 ms 1.3896 ms 1.4168 ms]
                        thrpt:  [176.46 MiB/s 179.90 MiB/s 183.82 MiB/s]
write_once/4.00 MiB     time:   [11.394 ms 11.555 ms 11.717 ms]
                        thrpt:  [341.39 MiB/s 346.18 MiB/s 351.07 MiB/s]
write_once/16.0 MiB     time:   [41.829 ms 42.645 ms 43.454 ms]
                        thrpt:  [368.20 MiB/s 375.19 MiB/s 382.51 MiB/s]

After:

write_once/4.00 KiB     time:   [572.20 us 583.62 us 595.21 us]
                        thrpt:  [6.5628 MiB/s 6.6932 MiB/s 6.8267 MiB/s]
                 change:
                        time:   [-6.3126% -3.8179% -1.0733%] (p = 0.00 < 0.05)
                        thrpt:  [+1.0849% +3.9695% +6.7380%]
                        Performance has improved.
write_once/256 KiB      time:   [1.3192 ms 1.3456 ms 1.3738 ms]
                        thrpt:  [181.98 MiB/s 185.79 MiB/s 189.50 MiB/s]
                 change:
                        time:   [-0.5899% +1.7476% +4.1037%] (p = 0.15 > 0.05)
                        thrpt:  [-3.9420% -1.7176% +0.5934%]
                        No change in performance detected.
write_once/4.00 MiB     time:   [10.855 ms 11.039 ms 11.228 ms]
                        thrpt:  [356.25 MiB/s 362.34 MiB/s 368.51 MiB/s]
                 change:
                        time:   [-6.9651% -4.8176% -2.5681%] (p = 0.00 < 0.05)
                        thrpt:  [+2.6358% +5.0614% +7.4866%]
                        Performance has improved.
write_once/16.0 MiB     time:   [38.706 ms 39.577 ms 40.457 ms]
                        thrpt:  [395.48 MiB/s 404.27 MiB/s 413.37 MiB/s]
                 change:
                        time:   [-10.829% -8.3611% -5.8702%] (p = 0.00 < 0.05)
                        thrpt:  [+6.2363% +9.1240% +12.145%]
                        Performance has improved.

Add complexity on the services side

Returning Stream and Sink make it complex to implement. At first glance, it does. But in reality, it's not.

Note: HTTP (especially for hyper) is stream-oriented.

  • Returning a stream is more straightforward than reader.
  • Returning Sink is covered by the global shared BodySinker struct.

Other helper functions will be covered at the Object-level which services don't need to bother.

Prior art

Returning a Writer

The most natural extending is to return BoxedAsyncWriter:

pub trait Accessor: Send + Sync + Debug {
    /// Read data from the underlying storage into input writer.
    async fn read(&self, args: &OpRead) -> Result<BoxedAsyncReader> {
        let _ = args;
        unimplemented!()
    }
    /// Write data from input reader to the underlying storage.
    async fn write(&self, args: &OpWrite) -> Result<BoxedAsyncWriter> {
        let _ = args;
        unimplemented!()
    }
}

But it only fixes the Inconsistent concern and can't help with other issues.

Slice based API

Most rust IO APIs are based on slice:

pub trait Accessor: Send + Sync + Debug {
    /// Read data from the underlying storage into input writer.
    async fn read(&self, args: &OpRead, bs: &mut [u8]) -> Result<usize> {
        let _ = args;
        unimplemented!()
    }
    /// Write data from input reader to the underlying storage.
    async fn write(&self, args: &OpWrite, bs: &[u8]) -> Result<usize> {
        let _ = args;
        unimplemented!()
    }
}

The problem is Accessor doesn't have states:

  • If we require all data must be passed at one time, we can't support large files read & write
  • If we allow users to call read/write multiple times, we need to implement another Reader and Writer alike logic.

Accept Reader and Writer

It's also possible to accept Reader and Writer instead.

pub trait Accessor: Send + Sync + Debug {
    /// Read data from the underlying storage into input writer.
    async fn read(&self, args: &OpRead, w: BoxedAsyncWriter) -> Result<usize> {
        let _ = args;
        unimplemented!()
    }
    /// Write data from input reader to the underlying storage.
    async fn write(&self, args: &OpWrite, r: BoxedAsyncReader) -> Result<usize> {
        let _ = args;
        unimplemented!()
    }
}

This API design addressed all concerns but made it hard for users to use. Primarily, we can't support futures::AsyncRead and tokio::AsyncRead simultaneously.

For example, we can't accept a Box::new(Vec::new()), user can't get this vec from OpenDAL.

Unresolved questions

None.

Future possibilities

  • Implement Object::read_into(w: BoxedAsyncWriter)
  • Implement Object::write_from(r: BoxedAsyncReader)

Summary

Remove the concept of credential.

Motivation

Credential intends to carry service credentials like access_key_id and secret_access_key. At OpenDAL, we designed a global Credential enum for services and users to use.

pub enum Credential {
    /// Plain refers to no credential has been provided, fallback to services'
    /// default logic.
    Plain,
    /// Basic refers to HTTP Basic Authentication.
    Basic { username: String, password: String },
    /// HMAC, also known as Access Key/Secret Key authentication.
    HMAC {
        access_key_id: String,
        secret_access_key: String,
    },
    /// Token refers to static API token.
    Token(String),
}

However, every service only supports one kind of Credential with different Credential load methods covered by reqsign. As a result, only HMAC is used. Both users and services need to write the same logic again and again.

Guide-level explanation

Credential will be removed, and the services builder will provide native credential representation directly.

For s3:

pub fn access_key_id(&mut self, v: &str) -> &mut Self {}
pub fn secret_access_key(&mut self, v: &str) -> &mut Self {}

For azblob:

pub fn account_name(&mut self, account_name: &str) -> &mut Self {}
pub fn account_key(&mut self, account_key: &str) -> &mut Self {}

All builders must implement Debug by hand and redact sensitive fields to avoid credentials being a leak.

impl Debug for Builder {
    fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
        let mut ds = f.debug_struct("Builder");

        ds.field("root", &self.root);
        ds.field("container", &self.container);
        ds.field("endpoint", &self.endpoint);

        if self.account_name.is_some() {
            ds.field("account_name", &"<redacted>");
        }
        if self.account_key.is_some() {
            ds.field("account_key", &"<redacted>");
        }

        ds.finish_non_exhaustive()
    }
}

Reference-level explanation

Simple change without reference-level explanation needs.

Drawbacks

API Breakage.

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

None

Summary

Add creating dir support for OpenDAL.

Motivation

Interoperability between OpenDAL services requires dir support. The object storage system will simulate dir operations with / via object ends. But we can't share the same behavior with fs, as mkdir is a separate syscall.

So we need to unify the behavior about dir across different services.

Guide-level explanation

After this proposal got merged, we will treat all paths that end with / as a dir.

For example:

  • read("abc/") will return an IsDir error.
  • write("abc/") will return an IsDir error.
  • stat("abc/") will be guaranteed to return a dir or a NotDir error.
  • delete("abc/") will be guaranteed to delete a dir or NotDir / NotEmpty error.
  • list("abc/") will be guaranteed to list a dir or a NotDir error.

And we will support create an empty object:

// create a dir object "abc/"
let _ = op.object("abc/").create().await?;
// create a file object "abc"
let _ = op.object("abc").create().await?;

Reference-level explanation

And we will add a new API called create to create an empty object.

struct OpCreate {
    path: String,
    mode: ObjectMode,
}

pub trait Accessor: Send + Sync + Debug {
    async fn create(&self, args: &OpCreate) -> Result<Metadata>;
}

Object will expose API like create which will call Accessor::create() internally.

Drawbacks

None

Rationale and alternatives

None

Prior art

None

Unresolved questions

When writing this proposal, io_error_more is not stabilized yet. We can't use NotADirectory nor IsADirectory directly.

Using from_raw_os_error is unacceptable because we can't carry our error context.

use std::io;

let error = io::Error::from_raw_os_error(22);
assert_eq!(error.kind(), io::ErrorKind::InvalidInput);

So we will use ErrorKind::Other for now, which means our users can't check the following errors:

  • IsADirectory
  • DirectoryNotEmpty
  • NotADirectory

Until they get stabilized.

Future possibilities

None

Summary

Treat io::ErrorKind::Interrupt as retryable error.

Motivation

Supports retry make our users' lives easier:

Feature request: Custom retries for the s3 backend

While the reading/writing from/to s3, AWS occasionally returns errors that could be retried (at least 5xx?). Currently, in the databend, this will fail the whole execution of the statement (which may have been running for an extended time).

Most users may need this retry feature, like decompress. Implementing it in OpenDAL will make users no bother, no backoff logic.

Guide-level explanation

With the retry feature enabled:

opendal = {version="0.5.2", features=["retry"]}

Users can configure the retry behavior easily:

let backoff = ExponentialBackoff::default();
let op = op.with_backoff(backoff);

All requests sent by op will be automatically retried.

Reference-level explanation

We will implement retry features via adding a new Layer.

In the retry layer, we will support retrying all operations. To do our best to keep retrying read & write, we will implement RetryableReader and RetryableWriter, which will support retry while no actual IO happens.

Retry operations

Most operations are safe to retry, like list, stat, delete and create.

We will retry those operations via input backoff.

Retry IO operations

Retry IO operations are a bit complex because IO operations have side effects, especially for HTTP-based services like s3. We can't resume an operation during the reading process without sending new requests.

This proposal will do the best we can: retry the operation if no actual IO happens.

If we meet an internal error before reading/writing the user's buffer, it's safe and cheap to retry it with precisely the same argument.

Retryable Error

  • Operator MAY retry io::ErrorKind::Interrupt errors.
  • Services SHOULD return io::ErrorKind::Interrupt kind if the error is retryable.

Drawbacks

Write operation can't be retried

As we return Writer to users, there is no way for OpenDAL to get the input data again.

Rationale and alternatives

Implement retry at operator level

We need to implement retry logic for every operator function, and can't address the same problem:

  • Reader / Writer can't be retired.
  • Intrusive design that users cannot expand on their own

Prior art

None

Unresolved questions

  • read and write can't be retried during IO.

Future possibilities

None

Summary

Allow getting id from an object.

Motivation

Allow get id from an object will make it possible to operate across different operators. Users can store objects' IDs locally and refer to them with different settings. This proposal will make tasks like backup, restore, and migration possible.

Guide-level explanation

Users can fetch an object id via:

let o = op.object("test_object");
let id = o.id();

The id is unique and permanent inside the underlying storage.

For example, if we have an s3 bucket with the root /workdir/, the object's id test_object will be /workdir/test_object.

Reference-level explanation

id() and path() will be added as functions of object:

impl Object {
    pub fn id(&self) -> String {}
    pub fn path(&self) -> String {}
}
  • path is a re-export of call to Metadata::path().
  • id will be generated by Operator's root and Metadata::path().

Drawbacks

None

Rationale and alternatives

Why not add a new field in Metadata?

Adding a new field inside Metadata requires every service to handle the id separately. And every metadata will need to store a complete id with the operators' root.

Why not provide a full URI like s3://path/to/object?

Because we can't.

A full and functional URI towards an object will need the operator's endpoint and credentials. It's better to provide the mechanism and allow users to construct them based on their own business.

Prior art

None

Unresolved questions

None

Future possibilities

None

Summary

Returning DirEntry instead of Object in list.

Motivation

In Object Stream, we introduce read_dir support via:

pub trait ObjectStream: futures::Stream<Item = Result<Object>> + Unpin + Send {}
impl<T> ObjectStream for T where T: futures::Stream<Item = Result<Object>> + Unpin + Send {}

pub struct Object {
    acc: Arc<dyn Accessor>,
    meta: Metadata,
}

However, the meta inside Object is not well-used:

pub(crate) fn metadata_ref(&self) -> &Metadata {}
pub(crate) fn metadata_mut(&mut self) -> &mut Metadata {}
pub async fn metadata_cached(&mut self) -> Result<&Metadata> {}

Users can't know an object's mode after the list, so they have to send metadata every time they get an object:

let o = op.object("path/to/dir/");
let mut obs = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(o) = obs.next().await {
    let mut o = o?;
    // It's highly possible that OpenDAL already did metadata during list.
    // Use `Object::metadata_cached()` to get cached metadata at first.
    let meta = o.metadata_cached().await?;
    match meta.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

This behavior doesn't make sense as we already know the object's mode after the list.

Introducing a separate DirEntry could reduce an extra call for metadata most of the time.

let o = op.object("path/to/dir/");
let mut ds = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(de) = ds.try_next().await {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

Guide-level explanation

Within this RFC, Object::list() will return DirStreamer instead.

pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
pub type DirStreamer = Box<dyn DirStream>;

DirStreamer will stream DirEntry, which carries information already known during the list. So we can:

let id = de.id();
let path = de.path();
let name = de.name();
let mode = de.mode();
let meta = de.metadata().await?;

With DirEntry support, we can reduce an extra metadata call if we only want to know the object's mode:

let o = op.object("path/to/dir/");
let mut ds = o.list().await?;
// ObjectStream implements `futures::Stream`
while let Some(de) = ds.try_next().await {
    match de.mode() {
        ObjectMode::FILE => {
            println!("Handling file")
        }
        ObjectMode::DIR => {
            println!("Handling dir like start a new list via meta.path()")
        }
        ObjectMode::Unknown => continue,
    }
}

We can convert this DirEntry into Object without overhead:

let o = de.into();

Reference-level explanation

This proposal will introduce a new struct, DirEntry:

struct DirEntry {}

impl DirEntry {
    pub fn id() -> String {}
    pub fn path() -> &str {}
    pub fn name() -> &str {}
    pub fn mode() -> ObjectMode {}
    pub async fn metadata() -> ObjectMetadata {}
}

impl From<DirEntry> for Object {}

And use DirStream to replace ObjectStream:

pub trait DirStream: futures::Stream<Item = Result<DirEntry>> + Unpin + Send {}
pub type DirStreamer = Box<dyn DirStream>;

With the addition of DirEntry, we will remove meta from Object:

#[derive(Clone, Debug)]
pub struct Object {
    acc: Arc<dyn Accessor>,
    path: String,
}

After this change, Object will become a thin wrapper of Accessor with path. And metadata related APIs like metadata_ref() and metadata_mut() will also be removed.

Drawbacks

We are adding a new concept to our core logic.

Rationale and alternatives

Rust fs API design

Rust also provides abstractions like File and DirEntry:

use std::fs;

fn main() -> std::io::Result<()> {
    for entry in fs::read_dir(".")? {
        let dir = entry?;
        println!("{:?}", dir.path());
    }
    Ok(())
}

Users can open a file with entry.path().

Prior art

None.

Unresolved questions

None.

Future possibilities

None.

Summary

Add support for accessor capabilities so that users can check if a given accessor is capable of a given ability.

Motivation

Users of OpenDAL are requesting advanced features like the following:

It's meaningful for OpenDAL to support them in a unified way. Of course, not all storage services have the same feature sets. OpenDAL needs to provide a way for users to check if a given accessor is capable of a given capability.

Guide-level explanation

Users can check an Accessor's capability via Operator::metadata().

let meta = op.metadata();
let _: bool = meta.can_presign();
let _: bool = meta.can_multipart(); 

Accessor will return io::ErrorKind::Unsupported for not supported operations instead of panic as unimplemented().

Users can check before operations or the Unsupported error kind after operations.

Reference-level explanation

We will introduce a new enum called AccessorCapability, which includes AccessorMetadata.

This enum is private and only accessible inside OpenDAL, so it's not part of our public API. We will expose the check API via AccessorMetadata:

impl AccessorMetadata {
    pub fn can_presign(&self) -> bool { .. }
    pub fn can_multipart(&self) -> bool { .. }
}

Drawbacks

None.

Rationale and alternatives

None.

Prior art

go-storage

Unresolved questions

None.

Future possibilities

None.

Summary

Add presign support in OpenDAL so users can generate a pre-signed URL without leaking serect_key.

Motivation

By default, all S3 objects are private. Only the object owner has permission to access them. However, the object owner can optionally share objects with others by creating a presigned URL, using their own security credentials, to grant time-limited permission to download the objects.

From Sharing objects using presigned URLs

We can use this presigned URL for:

  • Download the object within the expired time from a bucket directly
  • Upload content to the bucket on client-side

Adding this feature in OpenDAL will make users' lives easier to generate presigned URLs across different storage services.

The whole process would be:

Guide-level explanation

With this feature, our users can:

Generate presigned URL for downloading

let req = op.presign_read("path/to/file")?;
// req.method: GET
// req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>  

Users can download this object directly from the s3 bucket. For example:

curl <generated_url> -O test.txt

Generate presigned URL for uploading

let req = op.presign_write("path/to/file")?;
// req.method: PUT
// req.url: https://s3.amazonaws.com/examplebucket/test.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=access_key_id/20130721/us-east-1/s3/aws4_request&X-Amz-Date=20130721T201207Z&X-Amz-Expires=86400&X-Amz-SignedHeaders=host&X-Amz-Signature=<signature-value>  

Users can upload content directly to the s3 bucket. For example:

curl -X PUT <generated_url> -T "/tmp/test.txt"

Reference-level explanation

Accessor will add a new API presign:

pub trait Accessor {
    fn presign(&self, args: &OpPresign) -> Result<PresignedRequest> {..}
}

presign accepts OpPresign and returns Result<PresignedRequest>:

struct OpPresign {
    path: String,
    op: Operation,
    expire: time::Duration,
}

struct PresignedRequest {}

impl PresignedRequest {
    pub fn method(&self) -> &http::Method {..}
    pub fn url(&self) -> &http::Uri {..}
}

We are building a new struct to avoid leaking underlying implementations like hyper::Request<T> to users.

This feature will be a new capability in AccessorCapability as described in RFC-0409: Accessor Capabilities

Based on Accessor::presign, we will export public APIs in Operator:

impl Operator {
    fn presign_read(&self, path: &str) -> Result<PresignedRequest> {}
    fn presign_write(&self, path: &str) -> Result<PresignedRequest> {}
}

Although it's possible to generate URLs for create, delete, stat, and list, there are no obvious use-cases. So we will not add them to this proposal.

Drawbacks

None.

Rationale and alternatives

Query Sign Support Status

Prior art

awscli presign

AWS CLI has native presign support

> aws s3 presign s3://DOC-EXAMPLE-BUCKET/test2.txt
https://DOC-EXAMPLE-BUCKET.s3.us-west-2.amazonaws.com/key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAEXAMPLE123456789%2F20210621%2Fus-west-2%2Fs3%2Faws4_request&X-Amz-Date=20210621T041609Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=EXAMBLE1234494d5fba3fed607f98018e1dfc62e2529ae96d844123456

Refer to AWS CLI Command Reference for more information.

Unresolved questions

None.

Future possibilities

  • Add stat/list/delete support

Summary

Add command line interface for OpenDAL.

Motivation

Q: There are so many cli out there, why we still need a cli for OpenDAL?

A: Because there are so many cli out there.

To manipulate our date store in different could service, we need to install different clis:

Those clis provide native and seamless experiences for their own products but also lock us and our data.

However, for 80% cases, we just want to do simple jobs like cp, mv and rm. It's boring to figure out how to use them:

  • aws --endpoint-url http://127.0.0.1:9900/ s3 cp data s3://testbucket/data --recursive
  • azcopy copy 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive
  • gsutil cp data gs://testbucket/

Can we use them in the same way? Can we let the data flow freely?

Let's look back OpenDAL's slogan:

Open Data Access Layer that connect the whole world together

This is a natural extension for OpenDAL: providing a command line interface!

Guide-level explanation

OpenDAL will provide a new cli called: oli. It's a shortcut of OpenDAL Command Line Interface.

Users can install this cli via:

cargo install oli

Or using they favourite package management:

# Archlinux
pacman -S oli
# Debian / Ubuntu
apt install oli
# Rocky Linux / Fedora
dnf install oli
# macOS
brew install oli

With oli, users can:

  • Upload files to s3: oli cp books.csv s3://bucket/books.csv
  • Download files from azblob: oli cp azblob://bucket/books.csv /tmp/books.csv
  • Move data between storage services: oli mv s3://bucket/dir azblob://bucket/dir
  • Delete all files: oli rm -rf s3://bucket

oli also provide alias to make cloud data manipulating even natural:

  • ocp for oli cp
  • ols for oli ls
  • omv for oli mv
  • orm for oli rm
  • ostat for oli stat

oli will provide profile management so users don't need to provide credential every time:

  • oli profile add my_s3 --bucket test --access-key-id=example --secret-access-key=example
  • ocp my_s3://dir /tmp/dir

Reference-level explanation

oli will be a separate crate apart from opendal so we will not pollute the dependencies of opendal. But oli will be releases at the same time with the same version of opendal. That means oli will always use the same (latest) version of opendal.

Most operations of oli should be trivial, we will propose new RFCs if requiring big changes.

oli won't keep configuration. All config will go through environment, for example:

  • OIL_COLOR=always
  • OIL_CONCURRENCY=16

Besides, oil will read profile from env like cargo:

  • OIL_PROFILE_TEST_TYPE=s3
  • OIL_PROFILE_TEST_ENDPOINT=http://127.0.0.1:1090
  • OIL_PROFILE_TEST_BUCKET=test_bucket
  • OIL_PROFILE_TEST_ACCESS_KET_ID=access_key_id
  • OIL_PROFILE_TEST_SECRET_ACCESS_KEY=secret_access_key

With those environments, we can:

ocp path/to/dir test://test/to/dir

Drawbacks

None

Rationale and alternatives

s3cmd

s3cmd is a command line s3 client for Linux and Mac.

Usage: s3cmd [options] COMMAND [parameters]

S3cmd is a tool for managing objects in Amazon S3 storage. It allows for
making and removing "buckets" and uploading, downloading and removing
"objects" from these buckets.

Commands:
  Make bucket
      s3cmd mb s3://BUCKET
  Remove bucket
      s3cmd rb s3://BUCKET
  List objects or buckets
      s3cmd ls [s3://BUCKET[/PREFIX]]
  List all object in all buckets
      s3cmd la 
  Put file into bucket
      s3cmd put FILE [FILE...] s3://BUCKET[/PREFIX]
  Get file from bucket
      s3cmd get s3://BUCKET/OBJECT LOCAL_FILE
  Delete file from bucket
      s3cmd del s3://BUCKET/OBJECT
  Delete file from bucket (alias for del)
      s3cmd rm s3://BUCKET/OBJECT
  Restore file from Glacier storage
      s3cmd restore s3://BUCKET/OBJECT
  Synchronize a directory tree to S3 (checks files freshness using 
       size and md5 checksum, unless overridden by options, see below)
      s3cmd sync LOCAL_DIR s3://BUCKET[/PREFIX] or s3://BUCKET[/PREFIX] LOCAL_DIR
  Disk usage by buckets
      s3cmd du [s3://BUCKET[/PREFIX]]
  Get various information about Buckets or Files
      s3cmd info s3://BUCKET[/OBJECT]
  Copy object
      s3cmd cp s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
  Modify object metadata
      s3cmd modify s3://BUCKET1/OBJECT
  Move object
      s3cmd mv s3://BUCKET1/OBJECT1 s3://BUCKET2[/OBJECT2]
  Modify Access control list for Bucket or Files
      s3cmd setacl s3://BUCKET[/OBJECT]
  Modify Bucket Policy
      s3cmd setpolicy FILE s3://BUCKET
  Delete Bucket Policy
      s3cmd delpolicy s3://BUCKET
  Modify Bucket CORS
      s3cmd setcors FILE s3://BUCKET
  Delete Bucket CORS
      s3cmd delcors s3://BUCKET
  Modify Bucket Requester Pays policy
      s3cmd payer s3://BUCKET
  Show multipart uploads
      s3cmd multipart s3://BUCKET [Id]
  Abort a multipart upload
      s3cmd abortmp s3://BUCKET/OBJECT Id
  List parts of a multipart upload
      s3cmd listmp s3://BUCKET/OBJECT Id
  Enable/disable bucket access logging
      s3cmd accesslog s3://BUCKET
  Sign arbitrary string using the secret key
      s3cmd sign STRING-TO-SIGN
  Sign an S3 URL to provide limited public access with expiry
      s3cmd signurl s3://BUCKET/OBJECT <expiry_epoch|+expiry_offset>
  Fix invalid file names in a bucket
      s3cmd fixbucket s3://BUCKET[/PREFIX]
  Create Website from bucket
      s3cmd ws-create s3://BUCKET
  Delete Website
      s3cmd ws-delete s3://BUCKET
  Info about Website
      s3cmd ws-info s3://BUCKET
  Set or delete expiration rule for the bucket
      s3cmd expire s3://BUCKET
  Upload a lifecycle policy for the bucket
      s3cmd setlifecycle FILE s3://BUCKET
  Get a lifecycle policy for the bucket
      s3cmd getlifecycle s3://BUCKET
  Remove a lifecycle policy for the bucket
      s3cmd dellifecycle s3://BUCKET
  List CloudFront distribution points
      s3cmd cflist 
  Display CloudFront distribution point parameters
      s3cmd cfinfo [cf://DIST_ID]
  Create CloudFront distribution point
      s3cmd cfcreate s3://BUCKET
  Delete CloudFront distribution point
      s3cmd cfdelete cf://DIST_ID
  Change CloudFront distribution point parameters
      s3cmd cfmodify cf://DIST_ID
  Display CloudFront invalidation request(s) status
      s3cmd cfinvalinfo cf://DIST_ID[/INVAL_ID]

aws-cli

aws-cli is the official cli provided by AWS.

$ aws s3 ls s3://mybucket
        LastWriteTime            Length Name
        ------------             ------ ----
                                PRE myfolder/
2013-09-03 10:00:00           1234 myfile.txt

$ aws s3 cp myfolder s3://mybucket/myfolder --recursive
upload: myfolder/file1.txt to s3://mybucket/myfolder/file1.txt
upload: myfolder/subfolder/file1.txt to s3://mybucket/myfolder/subfolder/file1.txt

$ aws s3 sync myfolder s3://mybucket/myfolder --exclude *.tmp
upload: myfolder/newfile.txt to s3://mybucket/myfolder/newfile.txt

azcopy

azcopy is the new Azure Storage data transfer utility.

azcopy copy 'C:\myDirectory\myTextFile.txt' 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt'

azcopy copy 'https://mystorageaccount.blob.core.windows.net/mycontainer/myTextFile.txt' 'C:\myDirectory\myTextFile.txt'

azcopy sync 'C:\myDirectory' 'https://mystorageaccount.blob.core.windows.net/mycontainer' --recursive

gsutil

gsutil is a Python application that lets you access Cloud Storage from the command line.

gsutil cp [OPTION]... src_url dst_url
gsutil cp [OPTION]... src_url... dst_url
gsutil cp [OPTION]... -I dst_url

gsutil mv [-p] src_url dst_url
gsutil mv [-p] src_url... dst_url
gsutil mv [-p] -I dst_url

gsutil rm [-f] [-r] url...
gsutil rm [-f] [-r] -I

Unresolved questions

None.

Future possibilities

None.

Summary

Allow initing opendal operators from an iterator.

Motivation

To init OpenDAL operators, users have to init an accessor first.

let root = &env::var("OPENDAL_S3_ROOT").unwrap_or_else(|_| "/".to_string());
let root = format!("/{}/{}", root, uuid::Uuid::new_v4());

let mut builder = opedal::services::s3::Backend::build();
builder.root(&root);
builder.bucket(&env::var("OPENDAL_S3_BUCKET").expect("OPENDAL_S3_BUCKET must set"));
builder.endpoint(&env::var("OPENDAL_S3_ENDPOINT").unwrap_or_default());
builder.access_key_id(&env::var("OPENDAL_S3_ACCESS_KEY_ID").unwrap_or_default());
builder.secret_access_key(&env::var("OPENDAL_S3_SECRET_ACCESS_KEY").unwrap_or_default());
builder
    .server_side_encryption(&env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION").unwrap_or_default());
builder.server_side_encryption_customer_algorithm(
    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_ALGORITHM").unwrap_or_default(),
);
builder.server_side_encryption_customer_key(
    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY").unwrap_or_default(),
);
builder.server_side_encryption_customer_key_md5(
    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_CUSTOMER_KEY_MD5").unwrap_or_default(),
);
builder.server_side_encryption_aws_kms_key_id(
    &env::var("OPENDAL_S3_SERVER_SIDE_ENCRYPTION_AWS_KMS_KEY_ID").unwrap_or_default(),
);
if env::var("OPENDAL_S3_ENABLE_VIRTUAL_HOST_STYLE").unwrap_or_default() == "on" {
    builder.enable_virtual_host_style();
}
Ok(Some(builder.finish().await?))

We can simplify this logic if opendal has its native from_iter support.

Guide-level explanation

Users can init an operator like the following:

// OPENDAL_S3_BUCKET = <bucket>
// OPENDAL_S3_ENDPOINT = <endpoint>
let op = Operator::from_env(Scheme::S3)?;

Or from a prefixed env:

// OIL_PROFILE_<name>_S3_BUCKET = <bucket>
// OIL_PROFILE_<name>_S3_ENDPOINT = <endpoint>
let op = Operator::from_env(Scheme::S3, "OIL_PROFILE_<name>")?;

Also, we call the underlying function directly:

// var it: impl Iterator<Item=(String, String)>
let op = Operator::from_iter(Scheme::S3, it)?;

Reference-level explanation

Internally, every service's backend will implement the following functions:

fn from_iter(it: impl Iterator<Item=(String, String)>) -> Backend {}

Note: it's not a public API of Accessor, and it will never be. Instead, we will use this function inside the crate to keep the ability to refactor or even remove it.

Drawbacks

None.

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

Connection string

It sounds a good idea to implement something like:

let op = Operator::open("s3://bucket?region=test")?

But there are no valid use cases. Let's implement this in the future if needed.

Summary

Add multipart support in OpenDAL.

Motivation

Multipart Upload APIs are widely used in object storage services to upload large files concurrently and resumable.

A successful multipart upload includes the following steps:

  • CreateMultipartUpload: Start a new multipart upload.
  • UploadPart: Upload a single part with the previously uploaded id.
  • CompleteMultipartUpload: Complete a multipart upload to get a regular object.

To cancel a multipart upload, users need to call AbortMultipartUpload.

Apart from those APIs, most object services also provide a list API to get the current multipart uploads status:

  • ListMultipartUploads: List current ongoing multipart uploads
  • ListParts: List already uploaded parts.

Before CompleteMultipartUpload has been called, users can't read already uploaded parts.

After CompleteMultipartUpload or AbortMultipartUpload has been called, all uploaded parts will be removed.

Object storage services commonly allow 10000 parts, and every part will allow up to 5 GiB. This way, users can upload a file up to 48.8 TiB.

OpenDAL users can upload objects larger than 5 GiB via supporting multipart uploads.

Guide-level explanation

Users can start a multipart upload via:

let mp = op.object("path/to/file").create_multipart().await?;

Or build a multipart via already known upload id:

let mp = op.object("path/to/file").into_multipart("<upload_id>");

With Multipart, we can upload a new part:

let part = mp.write(part_number, content).await?;

After all parts have been uploaded, we can finish this upload:

let _ = mp.complete(parts).await?;

Or, we can abort already uploaded parts:

let _ = mp.abort().await?;

Reference-level explanation

Accessor will add the following APIs:

pub trait Accessor: Send + Sync + Debug {
    async fn create_multipart(&self, args: &OpCreateMultipart) -> Result<String> {
        let _ = args;
        unimplemented!()
    }

    async fn write_multipart(&self, args: &OpWriteMultipart) -> Result<PartWriter> {
        let _ = args;
        unimplemented!()
    }

    async fn complete_multipart(&self, args: &OpCompleteMultipart) -> Result<()> {
        let _ = args;
        unimplemented!()
    }

    async fn abort_multipart(&self, args: &OpAbortMultipart) -> Result<()> {
        let _ = args;
        unimplemented!()
    }
}

While closing a PartWriter, a Part will be generated.

Operator will build APIs based on Accessor:

impl Object {
    async fn create_multipart(&self) -> Result<Multipart> {}
    fn into_multipart(&self, upload_id: &str) -> Multipart {}
}

impl Multipart {
    async fn write(&self, part_number: usize, bs: impl AsRef<[u8]>) -> Result<Part> {}
    async fn writer(&self, part_number: usize, size: u64) -> Result<impl PartWrite> {}
    async fn complete(&self, ps: &[Part]) -> Result<()> {}
    async fn abort(&self) -> Result<()> {}
}

Drawbacks

None.

Rationale and alternatives

Why not add new object modes?

It seems natural to add a new object mode like multipart.

pub enum ObjectMode {
    FILE,
    DIR,
    MULTIPART,
    Unknown,
}

However, to make this work, we need big API breaks that introduce mode in Object.

And we need to change every API call to accept mode as args.

For example:

let _ = op.object("path/to/dir/").list(ObjectMODE::MULTIPART);
let _ = op.object("path/to/file").stat(ObjectMODE::MULTIPART)

Why not split Object into File and Dir?

We can split Object into File and Dir to avoid requiring mode in API. There is a vast API breakage too.

Prior art

None.

Unresolved questions

None.

Future possibilities

Support list multipart uploads

We can support listing multipart uploads to list ongoing multipart uploads so we can resume an upload or abort them.

Support list part

We can support listing parts to list already uploaded parts for an upload.

Summary

Add Gateway for OpenDAL.

Motivation

Our users want features like S3 Proxy and minio Gateway so that they can access all their data in the same way.

By providing a native gateway, we can empower users to access different storage in the same API.

Guide-level explanation

OpenDAL will provide a new binary called: oay. It's a shortcut of OpenDAL Gateway.

Uses can install this binary via:

cargo install oay

Or using they favourite package management:

# Archlinux
pacman -S oay
# Debian / Ubuntu
apt install oay
# Rocky Linux / Fedora
dnf install oay
# macOS
brew install oay

With oay, users can:

  • Serve fs backend with S3 compatible API.
  • Serve s3 backend with Azblob API
  • Serve as a s3 signing services

Reference-level explanation

oay will be a separate crate apart from opendal so we will not pollute the dependencies of opendal. But oay will be releases at the same time with the same version of opendal. That means oay will always use the same (latest) version of opendal.

Most operations of oay should be trivial, we will propose new RFCs if requiring big changes.

oay won't keep configuration. All config will go through environment.

Drawbacks

None

Rationale and alternatives

None

Prior art

Unresolved questions

None

Future possibilities

None

Summary

Allow users to build services without async.

Motivation

Most services share a similar builder API to construct backends.

impl Builder {
    pub async fn finish(&mut self) -> Result<Arc<dyn Accessor>> {}
}

We have async here so that every user who wants to build services backend must go through an async runtime. Even for memory backend:

impl Builder {
    /// Consume builder to build a memory backend.
    pub async fn finish(&mut self) -> Result<Arc<dyn Accessor>> {
        Ok(Arc::new(Backend::default()))
    }
}

Only s3 services need to call async functions detect_region to get the correct region.

So, we can provide blocking Builder APIs and move async-related logic out for users to call out. This way, our users can build services without playing with async runtime.

Guide-level explanation

After this change, all our services builder will add a new API:

impl Builder {
    pub fn build(&mut self) -> Result<Backend> {}
}

Along with this change, our Operator will accept impl Accessor + 'static instead of Arc<dyn Accessor> anymore:

impl Operator {
    pub fn new(accessor: impl Accessor + 'static) -> Self {}
}

Also, we will implement From<impl Accessor + 'static> for Operator:

impl<A> From<A> for Operator
where
    A: Accessor + 'static,
{
    fn from(acc: A) -> Self {
        Operator::newx(acc)
    }
}

We can initiate an operator quicker:

- let op: Operator = Operator::new(fs::Backend::build().finish().await?);
+ let op: Operator = fs::Builder::new().build()?.into();

Reference-level explanation

We will add the following APIs:

  • All builders will add build(&mut self) -> Result<Backend>
  • impl<A> From<A> for Operator where A: Accessor + 'static

We will deprecate the following APIs:

  • All builders finish() API (should be replaced by build())
  • All services build() API (should be replaced by Builder::new() or Builder::default())

We will change the following APIs:

  • Operator: new(accessor: Arc<dyn Accessor>) -> fn new(accessor: impl dyn Accessor + 'static)
  • Operator: async fn from_iter() -> fn from_iter()
  • Operator: async fn from_env() -> fn from_env()

Most services will work the same, except for s3: s3 depends on detect_region to check the correct region if the user doesn't input. After this change, s3::Builder.build() will return error if region is missing. Users should call detect_region by themselves to get the region.

Drawbacks

None.

Rationale and alternatives

None.

Prior art

None.

Unresolved questions

None.

Future possibilities

None.

Summary

Refactor write operation to accept a BytesReader instead.

Motivation

To simulate the similar operation like POSIX fs, OpenDAL returns BytesWriter for users to write, flush and close:

pub trait Accessor {
    async fn write(&self, args: &OpWrite) -> Result<BytesWriter> {}
}

Operator builds the high level APIs upon this:

impl Object {
    pub async fn write(&self, bs: impl AsRef<[u8]>) -> Result<()> {}
    
    pub async fn writer(&self, size: u64) -> Result<impl BytesWrite> {}
}

However, we are meeting the following problems:

  • Performance: HTTP body channel is mush slower than read from Reader directly.
  • Complicity: Service implementer have to deal with APIs like new_http_channel.
  • Extensibility: Current design can't be extended to multipart APIs.

Guide-level explanation

Underlying write implementations will be replaced by:

pub trait Accessor {
    async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
}

Existing API will have no changes, and we will add a new API:

impl Object {
    pub async fn write_from(&self, size: u64, r: impl BytesRead) -> Result<u64> {}
}

Reference-level explanation

Accessor's write API will be changed to accept a BytesReader:

pub trait Accessor {
    async fn write(&self, args: &OpWrite, r: BytesReader) -> Result<u64> {}
}

We will provide Operator::writer based on this new API instead.

RFC-0438: Multipart will also be updated to:

pub trait Accessor {
    async fn write_multipart(&self, args: &OpWriteMultipart, r: BytesReader) -> Result<u64> {}
}

In this way, we don't need to introduce a PartWriter.

Drawbacks

Layer API breakage

This change will introduce break changes to layers.

Rationale and alternatives

None.

Prior art

Unresolved questions

None.

Future possibilities

None.

Summary

Reuse metadata returned during listing, by extending DirEntry with some metadata fields.

Motivation

Users may expect to browse metadata of some directories' child files and directories. Using walk() of BatchOperator seems to be an ideal way to complete this job.

Thus, they start iterating on it, but soon they realized the DirEntry, could only offer the name (or path, more precisely) and access mode of the object, and it's not enough.

So they have to call metadata() for each name they extracted from the iterator.

The final example looks like:

let op = Operator::from_env(Scheme::Gcs)?.batch();

// here is a network request
let mut dir_stream = op.walk("/dir/to/walk")?;

while let Some(Ok(file)) = dir_stream.next().await {
    let path = file.path();

    // here is another network request
    let size = file.metadata().await?.content_length();
    println!("size of file {} is {}B", path, size);
}

But...wait! many storage-services returns object metadata when listing, like HDFS, AWS and GCS. The rust standard library returns metadata when listing local file systems, too.

In the previous versions of OpenDAL those fields were just get ignored. This wastes users' time on requesting on metadata.

Guide-level explanation

The loop in main will be changed to the following code with this RFC:

while let Some(Ok(file)) = dir_stream.next().await {
    let size = if let Some(len) = file.content_length() {
        len
    } else {
        file.metadata().await?.content_length();
    };
    let name = file.path();
    println!("size of file {} is {}B", path, size);
}

Reference-level explanation

Extend DirEntry with metadata fields:

pub struct DirEntry {
    acc: Arc<dyn Accessor>,

    mode: ObjectMode,
    path: String,

    // newly add metadata fields
    content_length: Option<u64>,  // size of file
    content_md5: Option<String>,
    last_modified: Option<OffsetDateTime>,
}

impl DirEntry {
    pub fn content_length(&self) -> Option<u64> {
        self.content_length
    }
    pub fn last_modified(&self) -> Option<OffsetDateTime> {
        self.last_modified
    }
    pub fn content_md5(&self) -> Option<OffsetDateTime> {
        self.content_md5
    }
}

For all services that supplies metadata during listing, like AWS, GCS and HDFS. Those optional fields will be filled up; Meanwhile for those services doesn't return metadata during listing, like in memory storages, just left them as None.

As you can see, for those services returning metadata when listing, the operation of listing metadata will save many unnecessary requests.

Drawbacks

Add complexity to DirEntry. To use the improved features of DirEntry, users have to explicitly check the existence of metadata fields.

The size of DirEntry increased from 40 bytes to 80 bytes, a 100% percent growth requires more memory.

Rational and alternatives

The largest drawback of performance usually comes from network or hard disk operations. By letting DirEntry storing some metadata, many redundant requests could be avoided.

Embed a Structure Containing Metadata

Define a MetaLite structure containing some metadata fields, and embed it in DirEntry

struct MetaLite {
    pub content_length: u64,  // size of file
    pub content_md5: String,
    pub last_modified: OffsetDateTime,
}

pub struct DirEntry {
    acc: Arc<dyn Accessor>,
    
    mode: ObjectMode,
    path: String,
    
    // newly add metadata struct
    metadata: Option<MetaLite>,
}

impl DirEntry {
    // get size of file
    pub fn content_length(&self) -> Option<u64> {
        self.metadata.as_ref().map(|m| m.content_length)
    }
    // get the last modified time
    pub fn last_modified(&self) -> Option<OffsetDateTime> {
        self.metadata.as_ref().map(|m| m.last_modified)
    }
    // get md5 message digest
    pub fn content_md5(&self) -> Option<String> {
        self.metadata.as_ref().map(|m| m.content_md5)
    }
}

The existence of those newly added metadata fields is highly correlated. If one field does not exist, the others neither.

By wrapping them together in an embedded structure, 8 bytes of space for each DirEntry object could be saved. In the future, more metadata fields may be added to DirEntry, then a lot more space could be saved.

This approach could be slower because some intermediate functions are involved. But it's worth sacrificing rarely used features' performance to save memory.

Embed a ObjectMetadata into DirEntry

  • Embed a ObjectMetadata struct into DirEntry
  • Remove the ObjectMode field in DirEntry
  • Change ObjectMetadata's content_length field's type to Option<u64>.
pub struct DirEntry {
    acc: Arc<dyn Accessor>,

    // - mode: ObjectMode, removed
    path: String,

    // newly add metadata struct
    metadata: ObjectMetadata,
}

impl DirEntry {
    pub fn mode(&self) -> ObjectMode {
        self.metadata.mode()
    }
    pub fn content_length(&self) -> Option<u64> {
        self.metadata.content_length()
    }
    pub fn content_md5(&self) -> Option<&str> {
        self.metadata.content_md5()
    }
    // other metadata getters...
}

In the degree of memory layout, it's the same as proposed way in this RFC. This approach offers more metadata fields and fewer changes to code.

Prior art

None.

Unresolved questions

None.

Future possibilities

Switch to Alternative Implement Approaches

As the growing of metadata fields, someday the alternatives could be better. And other RFCs will be raised then.

More Fields

Add more metadata fields to DirEntry, like:

  • accessed: the last access timestamp of object

Simplified Get

Users have to explicitly check if those metadata fields actual present in the DirEntry. This may be done inside the getter itself.

let path = file.path();

// if content_length is not exist
// this getter will automatically fetch from the storage service.
let size = file.content_length().await?;

// the previous getter can cache metadata fetched from service
// so this function could return instantly.
let md5 = file.content_md5().await?;
println!("size of file {} is {}B, md5 outcome of file is {}", path, size, md5);

Summary

We are adding a blocking API for OpenDAL.

Motivation

Blocking API is the most requested feature inside the OpenDAL community: Opendal support sync read/write API

Our users want blocking API for:

  • Higher performance for local IO
  • Using OpenDAL in a non-async environment

However, supporting sync and async API in current Rust is a painful job, especially for an IO library like OpenDAL. For example:

impl Object {
    pub async fn reader(&self) -> Result<impl BytesRead> {}
}

Supporting blocking API doesn't mean removing the async from the function. We should also handle the returning Reader:

impl Object {
    pub fn reader(&self) -> Result<impl Read> {}
}

Until now, I still don't know how to handle them correctly. But we need to have a start: not perfect, but enough for our users to have a try.

So this RFC is an experiment try to introduce blocking API support. I expect the OpenDAL community will evaluate those APIs and keep improving them. And finally, we will pick up the best one for stabilizing.

Guide-level explanation

With this RFC, we can call blocking API with the blocking_ prefix:

fn main() -> Result<()> {
    // Init Operator
    let op = Operator::from_env(Scheme::Fs)?;

    // Create object handler.
    let o = op.object("test_file");

    // Write data info object;
    o.blocking_write("Hello, World!")?;

    // Read data from object;
    let bs = o.blocking_read()?;

    // Read range from the object;
    let bs = o.blocking_range_read(1..=11)?;

    // Get the object's path
    let name = o.name();
    let path = o.path();

    // Fetch more meta about the object.
    let meta = o.blocking_metadata()?;
    let mode = meta.mode();
    let length = meta.content_length();
    let content_md5 = meta.content_md5();
    let etag = meta.etag();

    // Delete object.
    o.blocking_delete()?;

    // List dir object.
    let o = op.object("test_dir/");
    let mut ds = o.blocking_list()?;
    while let Some(entry) = ds.try_next()? {
        let path = entry.path();
        let mode = entry.mode();
    }

    Ok(())
}

All async public APIs of Object and Operator will have a sync version with blocking_ prefix. And they will share precisely the same semantics.

The differences are:

  • They will be executed and blocked on the current thread.
  • Input and output's Reader will become the blocking version like std::io::Read.
  • Output's DirStreamer will become the blocking version like Iterator.

Thanks to RFC-0501: New Builder, all our builder-related APIs have been transformed into blocking APIs, so we don't change our initiation logic.

Reference-level explanation

Under the hood, we will add the following APIs in Accessor:

trait Accessor {
    fn blocking_create(&self, args: &OpCreate) -> Result<()>;
    
    fn blocking_read(&self, args: &OpRead) -> Result<BlockingBytesReader>;
    
    fn blocking_write(&self, args: &OpWrite, r: BlockingBytesReader) -> Result<u64>;
    
    fn blocking_stat(&self, args: &OpStat) -> Result<ObjectMetadata>;
    
    fn blocking_delete(&self, args: &OpDelete) -> Result<()>;
    
    fn blocking_list(&self, args: &OpList) -> Result<DirIterator>;
}

Notes:

  • BlockingBytesReader is a boxed std::io::Read.
  • All blocking operations are happening on the current thread.
  • Blocking operation is implemented natively, no futures::block_on.

Drawbacks

Two sets of APIs

This RFC will add a new set of APIs, adding complicity for OpenDAL.

And users may misuse them. For example: using blocking_read in an async context could block the entire thread.

Rationale and alternatives

Use features to switch async and sync

Some crates provide features to switch the async and sync versions of API.

In this way:

  • We can't provide two kinds of API at the same time.
  • Users must decide to use async or sync at compile time.

Use blocking IO functions in local fs services

Can we use blocking IO functions in local fs services to implement Accessor's asynchronous functions directly? What is the drawback of our current non-blocking API?

We can't run blocking IO functions inside the async context. We need to let the local thread pool execute them and use mio to listen to the events. If we do so, congrats, we are building tokio::fs again!

Prior art

None

Unresolved questions

None

Future possibilities

None

Summary

Use redis as a service of OpenDAL.

Motivation

Redis is a fast, in-memory cache with persistent and distributed storage functionalities. It's widely used in production.

Users also demand more backend support. Redis is a good candidate.

Guide-level explanation

Users only need to provide the network address, username and password to create a Redis Operator. Then everything will act as other operators do.

use opendal::services::redis::Builder;
use opendal::Operator;

let builder = Builder::default();

// set the endpoint of redis server
builder.endpoint("tcps://domain.to.redis:2333");
// set the username of redis
builder.username("example");
// set the password
builder.password(&std::env::var("OPENDAL_REDIS_PASSWORD").expect("env OPENDAL_REDIS_PASSWORD not set"));
// root path
builder.root("/example/");

let op = Operator::new(
    builder.build()?    // services::redis::Backend
);

// congratulations, you can use `op` just like any other operators!

Reference-level explanation

To ease the development, redis-rs will be used.

Redis offers a key-value view, so the path of files could be represented as the key of the key-value pair.

The content of file will be represented directly as String, and metadata will be encoded as bincode before storing as String.

+------------------------------------------+
|Object: /home/monika/                     |
|                                          |           SET
|child: Key: v0:k:/home/monika/           -+---------> 1) /home/monika/poem0.txt
|                                          |
|/* directory has no content  */           |
|                                          |
|metadata: Key: v0:m:/home/monika/         |
+------------------------------------------+

+------------------------------------------+
|Object: /home/monika/poem0.txt            |
|                                          |
| /*      file has no children        */   |
|                                          |
|content: Key: v0:c:/home/monika/poem0.txt-+--+
|                                          |  |
|metadata: Key: v0:m:/home/monika/poem0.txt|  |
|  |                                       |  |
+--+---------------------------------------+  |
   |                                          v
   +> STRING                                STRING
     +----------------------+              +--------------------+
     |\x00\x00\x00\x00\xe6\a|              |1JU5T3MON1K413097321|
     |\x00\x00\xf8\x00\a4)!V|              |&JU5$T!M0N1K4$%#@#$%|
     |\x81&\x00\x00\x00Q\x00|              |3231J)U_ST#MONIKA@#$|
     |         ...          |              |1557(m0N1ka3just4M  |
     +----------------------+              |      ...           |
                                           +--------------------+

The redis-rs's high level APIs is preferred.

const VERSION: usize = 0;

/// meta_key will produce the key to object's metadata
/// "/path/to/object/" -> "v{VERSION}:m:/path/to/object"
fn meta_key(path: &str) -> String {
    format!("v{}:m:{}", VERSION, path);
}

/// content_key will produce the key to object's content
/// "/path/to/object/" -> "v{VERSION}:c:/path/to/object"
fn content_key(path: &str) -> String {
    format!("v{}:c:{}", VERSION, path);
}

let client = redis::Client::open("redis://localhost:6379")?;
let con = client.get_async_connection()?;

Forward Compatibility

All keys used will have a v0 prefix, indicating it's using the very first version of OpenDAL Redis API.

When there are changes to the layout, like refactoring the layout of storage, the version number should be updated, too. Further versions should take the compatibility with former implementations into consideration.

Create File

If user is creating a file with root /home/monika/, and relative path poem0.txt.

// mode: ObjectMode
// path: relative path string
let path = get_abs_path(path);  // /home/monika/ <> /poem.txt -> /home/monika/poem.txt
let m_path = meta_key(path);  // path of metadata
let c_path = content_key(path);  // path of content
let last_modified = OffsetDatetime::now_utc().to_string();

let mut meta = ObjectMeta::default();
meta.set_mode(ObjectMode::FILE);
meta.set_last_modified(OffsetDatetime::now_utc());

let encoded = bincode::encode_to_vec(meta)?;

con.set(c_path, "".to_string())?;
con.set(m_path, encoded.as_slice())?;

This will create two key-value pair. For object content, its key is v0:c:/home/monika/poem0.txt, the value is an empty String; For metadata, the key is v0:m:/home/monika/poem0.txt, the value is a bincode encoded ObjectMetadata structure binary string.

On creating a file or directory, the backend should also create its all parent directories if not present.

// create a file under `PATH`
let mut path = std::path::PathBuf::new(PATH);
let mut con = client.new_async_connection().await?;

while let Some(parent) = path.parent() {
    let (p, c): (String, String) = (parent.display(), path.display());
    let to_children = format!("v0:ch:{}", p);
    con.sadd(to_children, c).await?;
    path = parent;
}

Read File

Opendal empowers users to read with the path object, offset of the cursor and size to read. Redis provided a GETRANGE command which perfectly fit into it.

// path: "poem0.txt"
// offset: Option<u64>, the offset of reading
// size: Option<u64>, the size of reading
let path = get_abs_path(path);
let c_path = content_key(path); 
let (mut start, mut end) = (0, -1);
if let Some(offset) = offset {
    start = offset;
}
if let Some(size) = size {
    end = start + size;
}
let buf: Vec<u8> = con.getrange(c_path, start, end).await?;
Box::new(buf)
GET v0:c:/home/monika/poem0.txt

Write File

Redis ensures the writing of a single entry to be atomic, no locking is required.

What needs to take care by opendal, besides the content of object, is its metadata. For example, though offering a OBJECT IDLETIME command, redis cannot record the last modified time of a key, so this should be done in opendal.

use redis::AsyncCommands;
// args: &OpWrite
// r: BytesReader
let mut buf = vec![];
let content_length: u64 = futures::io::copy(r, &mut buf).await?;
let last_modified: String = OffsetDateTime::now().to_string();

// content md5 will not be offered

let mut meta = ObjectMetadata::default();
meta.set_content_length(content_length);
meta.set_last_modified(last_modified);

// `ObjectMetadata` has implemented the `Serialize` and `Deserialize` trait of `Serde`
// so bincode could serialize and deserialize it.
let bytes = bincode::encode_to_vec(&meta)?;

let path = get_abs_path(args.path());
let m_path = meta_key(path);
let c_path = content_key(path);

con.set(c_path, content).await?;
con.set(m_path, bytes).await?;
SET v0:c:/home/monika/poem.txt content_string
SET v0:m:/home/monika/poem.txt <bincode encoded metadata>

Stat

To get the metadata of an object, using the GET command and deserialize from bincode.

let path = get_abs_path(args.path());
let meta = meta_key(path);
let bin: Vec<u8> = con.get(meta).await?;
let meta: ObjectMeta = bincode::deserialize(bin.as_slice())?;
GET v0:m:/home/monika/poem.txt

List

For listing directories, just SSCAN through the child list of the directory, nice and correct.

// print all sub-directories of `PATH`

let s_key = format!("v0:k:{}", PATH);
let mut con = client.new_async_connection().await?;
let mut it = con.sscan::<&str, String>(s_key).await?;

while let Some(child) = it.next_item().await {
    println!("get sub-dir: {}", child);
}

Delete

All subdirectories of path will be listed and removed.

On deleting a file or directory, the backend should remove the entry from its parent's SET, and remove all children of entry.

This could be done by postorder deleting.

async fn remove_entry(con: &mut redis::aio::AsyncConnection, entry: String) {
    let skey = format!("v0:ch:{}", entry);
    let it = con.sscan::<&str, String>(skey).await?;
    while let Some(child) = it.next_item().await {
        remove_entry(&mut con, child).await;
    }
    if let Some(parent) = std::PathBuf::new(entry).parent() {
        let p: String = parent.display();
        let parent_skey = format!("v0:ch:{}", p);
        let _ = con.srem(parent_skey, entry).await;
    }
    // remove metadata and content
}

Blocking APIs

redis-rs also offers a synchronous version of API, just port the functions above to its synchronous version.

Drawbacks

  1. New dependencies is introduced: redis-rs and bincode;
  2. Some calculations have to be done in client side, this will affect the performance;
  3. Grouping atomic operations together doesn't promise transactional access, this may lead to data racing issues.
  4. Writing large binary strings requiring copying all data from pipe(or BytesReader in opendal) to RAM, and then send to redis.

Rationale and alternatives

RedisJSON module

The RedisJSON module provides JSON support for Redis, and supports depth up to 128. Working on a JSON api could be easier than manually parsing or deserializing from HASH.

Since bincode also offers the ability of deserializing and serializing, RedisJSON won't be used.

Prior art

None

Unresolved questions

None

Future possibilities

The implementation proposed here is far from perfect.

  • The data organization could be optimized to make it acts more like a filesystem
  • Making a customized redis module to calculate metadata on redis side
  • Wait for stable of bincode 2.0, and bump to it.

Summary

Split basic operations into read, write, and list capabilities.

Motivation

In RFC-0409: Accessor Capabilities, we introduce the ideas of Accessor Capabilities. Services could have different capabilities, and users can check them via:

let meta = op.metadata();
let _: bool = meta.can_presign();
let _: bool = meta.can_multipart(); 

If users call not supported capabilities, OpenDAL will return io::ErrorKind::Unsupported instead.

Along with that RFC, we also introduce an idea about Basic Operations: the operations that all services must support, including:

  • metadata
  • create
  • read
  • write
  • delete
  • list

However, not all storage services support them. In our existing services, exception includes:

  • HTTP services don't support write, delete, and list.
  • IPFS HTTP gateway doesn't support write and delete.
    • NOTE: ipfs has a writable HTTP gateway, but there is no available instance.
  • fs could be read-only if mounted as RO.
  • object storage like s3 and gcs could not have enough permission.
  • cache services may not support list.

So in this RFC, we want to remove the idea about Basic Operations and convert them into different capabilities:

  • read: read and stat
  • write: write and delete
  • list: list

Guide-level explanation

No public API changes.

Reference-level explanation

This RFC will add three new capabilities:

  • read: read and stat
  • write: write and delete
  • list: list

After this change, all services must declare the features they support.

Most of this RFCs work is to refactor the tests. This RFC will refactor the behavior tests into several parts based on capabilities.

Drawbacks

None

Rationale and alternatives

None

Prior art

None

Unresolved questions

None

Future possibilities

Read-only Services

OpenDAL can implement read-only services after this change:

  • HTTP Service
  • IPFS HTTP Gateway

Add new capabilities with Layers

We can implement a layer that can add list capability for underlying storage services. For example, IndexLayer for HTTP services.

Summary

Move the path from OpXxx to Accessor directly.

Motivation

Accessor uses OpXxx to carry path input:

impl Accessor {
    async fn read(&self, args: &OpRead) -> Result<BytesReader> {
        let _ = args;
        unimplemented!()
    }
}

#[derive(Debug, Clone, Default)]
pub struct OpRead {
    path: String,
    offset: Option<u64>,
    size: Option<u64>,
}

However, nearly all operation requires a path. And the path is represented in String, which means we have to clone it:

impl OpRead {
    pub fn new(path: &str, range: impl RangeBounds<u64>) -> Result<Self> {
        let br = BytesRange::from(range);

        Ok(Self {
            path: path.to_string(),
            offset: br.offset(),
            size: br.size(),
        })
    }
}

Besides, we can't expose low-level APIs like:

impl Object {
    pub async fn read_with(&self, op: OpRead) -> Result<Vec<u8>> {
        ..
    }
}

Because users can't build the required OpRead.

Guide-level explanation

With this RFC, users can use low-level APIs can control the OpXxx directly:

impl Object {
    pub async fn read_with(&self, op: OpRead) -> Result<Vec<u8>> {
        ..
    }

    pub async fn write_with(&self, op: OpWrite, bs: impl Into<Vec<u8>>) -> Result<()> {
        ..
    }
}

So we can add more args in requests like:

o.write_with(OpWrite::new().with_content_md5("xxxxx"), bs).await;

Reference-level explanation

All path in OpXxx will be moved to Accessor directly:

pub trait Accessor: Send + Sync + Debug {
    async fn create(&self, path: &str, args: OpCreate) -> Result<()> {}
    
    async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
    
    ...
}
  • All functions that accept OpXxx requires ownership instead of reference.
  • All OpXxx::new() will introduce breaking changes:
    - pub fn new(path: &str, range: impl RangeBounds<u64>) -> Result<Self>
    + pub fn new(range: impl RangeBounds<u64>) -> Self
    

Drawbacks

Breaking Changes

This RFC may break users' code in the following ways:

  • Code that depends on Accessor:
    • Self-implemented Services
    • Self-implemented Layers
  • Code that depends on OpXxx

Rationale and alternatives

None.

Prior art

None.

Unresolved questions

None.

Future possibilities

We can add more fields in OpXxx.

Summary

Add generic kv services support OpenDAL.

Motivation

OpenDAL now has some kv services support:

  • memory
  • redis

However, maintaining them is complex and very easy to be wrong. We don't want to implement similar logic for every kv service. This RFC intends to introduce a generic kv service so that we can:

  • Implement OpenDAL Accessor on this generic kv service
  • Add new kv service support via generic kv API.

Guide-level explanation

No user-side changes.

Reference-level explanation

OpenDAL will introduce a generic kv service:

trait KeyValueAccessor {
    async fn get(&self, key: &[u8]) -> Result<Option<Vec<u8>>>;
    async fn set(&self, key: &[u8], value: &[u8]) -> Result<()>;
}

We will implement the OpenDAL service on KeyValueAccessor. To add new kv service support, users only need to implement it against KeyValueAccessor.

Spec

This RFC is mainly inspired by TiFS: FUSE based on TiKV. We will use the same ScopedKey idea in TiFS.

pub enum ScopedKey {
    Meta,
    Inode(u64),
    Block {
        ino: u64,
        block: u64,
    },
    Entry {
        parent: u64,
        name: String,
    },
}

We can encode a scoped key into a byte array as a key. Following is the common layout of an encoded key.

+ 1byte +<----------------------------+ dynamic size +------------------------------------>+
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       v                                                                                  v
+------------------------------------------------------------------------------------------+
|       |                                                                                  |
| scope |                                       body                                       |
|       |                                                                                  |
+-------+----------------------------------------------------------------------------------+

Meta

There is only one key in the meta scope. The meta key is designed to store metadata of our filesystem. Following is the layout of an encoded meta key.

+ 1byte +
|       |
|       |
|       |
|       |
|       |
|       |
|       v
+-------+
|       |
|   0   |
|       |
+-------+

This key will store data:

pub struct Meta {
    inode_next: u64,
}

The meta-structure contains only an auto-increasing counter inode_next, designed to generate an inode number.

Inode

Keys in the inode scope are designed to store attributes of files. Following is the layout of an encoded inode key.

+ 1byte +<-------------------------------+ 8bytes +--------------------------------------->+
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       |                                                                                  |
|       v                                                                                  v
+------------------------------------------------------------------------------------------+
|       |                                                                                  |
|   1   |                               inode number                                       |
|       |                                                                                  |
+-------+----------------------------------------------------------------------------------+

This key will store data:

pub struct Inode {
    meta: Metadata,
    blocks: HashMap<u64, u32>,
}

blocks is the map from block_id -> size. We will use this map to calculate the correct blocks to read.

Block

Keys in the block scope are designed to store blocks of a file. Following is the layout of an encoded block key.

+ 1byte +<----------------- 8bytes ---------------->+<------------------- 8bytes ----------------->+
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       v                                           v                                              v
+--------------------------------------------------------------------------------------------------+
|       |                                           |                                              |
|   2   |              inode number                 |                  block index                 |
|       |                                           |                                              |
+-------+-------------------------------------------+----------------------------------------------+

Entry

Keys in the file index scope are designed to store the entry of the file. Following is the layout of an encoded file entry key.

+ 1byte +<----------------- 8bytes ---------------->+<-------------- dynamic size ---------------->+
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       |                                           |                                              |
|       v                                           v                                              v
+--------------------------------------------------------------------------------------------------+
|       |                                           |                                              |
|   3   |     inode number of parent directory      |         file name in utf-8 encoding          |
|       |                                           |                                              |
+-------+-------------------------------------------+----------------------------------------------+

Store the correct inode number for this file.

pub struct Index {
    pub ino: u64,
}

Drawbacks

None.

Rationale and alternatives

None.

Prior art

None.

Unresolved questions

None.

Future possibilities

None.

Summary

Returning reading related object meta in the reader.

Motivation

Some services like s3 could return object meta while issuing reading requests.

In GetObject, we could get:

  • Last-Modified
  • Content-Length
  • ETag
  • Content-Range
  • Content-Type
  • Expires

We can avoid extra HeadObject calls by reusing that meta wisely, which could take 50ms. For example, Content-Range returns the content range of this read in the whole object: <unit> <range-start>-<range-end>/<size>. By using the content range, we can avoid HeadObject to get this object's total size, which means a lot for the content cache.

Guide-level explanation

reader and all its related API will return ObjectReader instead:

- pub async fn reader(&self) -> Result<impl BytesRead> {}
+ pub async fn reader(&self) -> Result<ObjectReader> {}

ObjectReader impls BytesRead too, so existing code will keep working. And ObjectReader will provide similiar APIs to ObjectEntry, for example:

pub async fn content_length(&self) -> Option<u64> {}
pub async fn last_modified(&self) -> Option<OffsetDateTime> {}
pub async fn etag(&self) -> Option<String> {}

Note:

  • All fields are optional, as services like fs could not return them.
  • content_length here is this read request's length, not the object's length.

Reference-level explanation

We will change the API signature of Accessor:

- async fn read(&self, path: &str, args: OpRead) -> Result<BytesReader> {}
+ async fn read(&self, path: &str, args: OpRead) -> Result<ObjectReader> {}

ObjectReader is a wrapper of BytesReader and ObjectMeta:

pub struct ObjectReader {
    inner: BytesReader
    meta: ObjectMetadata,
}

impl ObjectReader {
    pub async fn content_length(&self) -> Option<u64> {}
    pub async fn last_modified(&self) -> Option<OffsetDateTime> {}
    pub async fn etag(&self) -> Option<String> {}
}

Services can decide whether or not to fill them.

Drawbacks

None.

Rationale and alternatives

None.

Prior art

None.

Unresolved questions

None.

Future possibilities

Add content-range support

We can add content-range in ObjectMeta so that users can fetch and use them.

Summary

Use a separate error instead of std::io::Error.

Motivation

OpenDAL is used to use std::io::Error for all functions. This design is natural and easy to use. But there are many problems with the usage:

Not friendly for retry

io::Error can't carry retry-related information. In RFC-0247: Retryable Error, we use io::ErrorKind::Interrupt to indicate this error is retryable. But this change will hide the real error kind from the underlying. To mark this error has been retried, we have to add another new error wrapper:

#[derive(thiserror::Error, Debug)]
#[error("permanent error: still failing after retry, source: {source}")]
struct PermanentError {
    source: Error,
}

ErrorKind is inaccurate

std::io::ErrorKind is used to represent errors returned from system io, which is unsuitable for mistakes that have business semantics. For example, users can't distinguish ObjectNotFound or BucketNotFound from ErrorKind::NotFound.

ErrorKind is incompelete

OpenDAL has been waiting for features io_error_more to be stabilized for a long time. But there is no progress so far, which makes it impossible to return ErrorKind::IsADirectory or ErrorKind::NotADirectory on stable rust.

Error is not easy to carry context

To carry context inside std::io::Error, we have to check and make sure all functions are constructed ObjectError or BacknedError:

#[derive(Error, Debug)]
#[error("object error: (op: {op}, path: {path}, source: {source})")]
pub struct ObjectError {
    op: Operation,
    path: String,
    source: anyhow::Error,
}

To make everything worse, we can't prevent opendal returns raw io errors directly. For example, in Object::range_read:

pub async fn range_read(&self, range: impl RangeBounds<u64>) -> Result<Vec<u8>> {
    ...

    io::copy(s, &mut bs).await?;

    Ok(bs.into_inner())
}

We leaked the io::Error without any context.

Guide-level explanation

Thus, I propose to add opendal::Error back with everything improved.

Users will have similar usage as before:

if let Err(e) = op.object("test_file").metadata().await {
    if e.kind() == ErrorKind::ObjectNotFound {
        println!("object not exist")
    }
}

Users can check if this error a temporary:

if err.is_temporary() {
    // retry the operation
}

Users can print error messages via Display:

> println!("{}", err);

ObjectNotFound (permanent) at read, context: { service: S3, path: path/to/file } => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: ""

Also, users can choose to print the more verbose message via Debug:

> println!("{:?}", err);

ObjectNotFound (permanent) at read => status code: 404, headers: {"x-amz-request-id": "GCYDTQX51YRSF4ZF", "x-amz-id-2": "EH0vV6lTwWk+lFXqCMCBSk1oovqhG4bzALU9+sUudyw7TEVrfWm2o/AFJKhYKpdGqOoBZGgMTC0=", "content-type": "application/xml", "date": "Mon, 21 Nov 2022 05:26:37 GMT", "server": "AmazonS3"}, body: ""

Context:
    service: S3
    path: path/to/file

Source: <source error>

Backtrace: <backtrace if we have>

Reference-level explanation

We will add new Error and ErrorKind in opendal:

pub struct Error {
    kind: ErrorKind,
    message: String,

    status: ErrorStatus,`
    operation: &'static str,
    context: Vec<(&'static str, String)>,
    source: Option<anyhow::Error>,
}
  • status: the status of this error, which indicates if this error is temporary
  • operation: the operation which generates this error
  • context: the context related to this error
  • source: the underlying source error

Drawbacks

Breaking changes

This RFC will lead to a breaking at user side.

Rationale and alternatives

None.

Prior art

None.

Unresolved questions

None.

Future possibilities

More ErrorKind

We can add more error kinds to make it possible for users to check.