NOTE: This document is written by OpenDAL's maintainers and not reviewed by object_store's maintainers. So it could not be very objective.
A focused, easy to use, idiomatic, high performance,
asyncobject store library interacting with object stores.
Yes, of course. Both
object_store are developed in Rust, a language empowering everyone to build reliable and efficient software.
object_store are licensed under Apache 2.0.
object_store can be used to access data stored on object storage services. The primary users of those projects are both cloud-native databases too:
opendalis mainly used by databend: A modern Elasticity and Performance cloud data warehouse
object_storeis mainly used by:
object_store is a part of
Apache Arrow which means it's hosted and maintained by Apache Software Foundation.
opendal is now hosted and maintained by Datafuse Labs, a new start-up focusing on building data cloud.
opendalhas a plan to be donated to
ASF. We are at the very early stage of preparing the donation. Welcome any suggestions and help!
opendal is Open Data Access Layer that accesses data freely, painlessly, and efficiently.
object_store is more focused on async object store support.
You will see the different visions lead to very different routes.
object_store exposed a trait called
ObjectStore to users.
Users need to build a
dyn ObjectStore and operate on it directly:
let object_store: Arc<dyn ObjectStore> = Arc::new(get_object_store()); let path: Path = "data/file01.parquet".try_into().unwrap(); let stream = object_store .get(&path) .await .unwrap() .into_stream();
opendal has a similar trait called
opendal don't expose this trait to end users directly. Instead,
opendal expose a new struct called
Operator and builds public API on it.
let op: Operator = Operator::from_env(Scheme::S3)?; let r = op.object("data/file01.parquet").reader().await.unwrap();
opendal provide a mechanism to intercept operations.
let object_store = ThrottledStore::new(get_object_store(), ThrottleConfig::default())
let op = op.layer(TracingLayer).layer(MetricsLayer);
At the time of writing:
- ThrottleStore: Rate Throttling
- LimitStore: Concurrent Request Limit
- ImmutableIndexLayer: immutable in-memory index.
- LoggingLayer: logging.
- MetadataCacheLayer: metadata cache.
- ContentCacheLayer: content data cache.
- MetricsLayer: metrics
- RetryLayer: retry
- SubdirLayer: Allow switch directory without changing original operator.
- TracingLayer: tracing
object_store have different visions, so they have other services support:
|hdfs||Y||Y (via datafusion-objectstore-hdfs)|
|http||Y (read only)||N|
|ipfs||Y (read only)||N|
opendal has an idea called
AccessorCapability, so it's services may have different capability sets. For example, opendal's
ipfs are read only.
object_store have different visions, so they have different feature sets:
|metadata||-||get some metadata from underlying storage|
|-||get_ranges||opendal doesn't support read multiple ranges|
|-||list||opendal doesn't support list with prefix|
|presign||-||get a presign URL of object|
|multipart||multipart||both support, but API is different|
|blocking||-||opendal supports blocking API|
The most straightforward complete demo how to read a file from s3:
let mut builder = s3::Builder::default(); builder.bucket("example"); builder.access_key_id("access_key_id"); builder.secret_access_key("secret_access_key"); let store = Operator::new(builder.build()?); let r = store.object("data.parquet").reader().await?;
let mut builder = AmazonS3Builder::new() .with_bucket_name("example") .with_access_key_id("access_key_id") .with_secret_access_key("secret_access_key"); let store = Arc::new(builder.build()?); let path: Path = "data.parquet".try_into().unwrap(); let stream = store.get(&path).await()?.into_stream();