Object store#

The objects plugin stores binary payloads in the database file and names them in a tree of files and directories. Enable it at open with plugins {objects {}}. It has no settings. See plugin selection for the open options. Calls without it report the unavailable status.

The typed wrappers use plugin_call or plugin_stream with plugin objects. Request members use lowerCamelCase, such as expectedRevision.

Payloads#

A payload is an immutable byte sequence. Its digest is the SHA-256 of all its bytes. Write a payload through an upload stream: write appends bytes in order and finish publishes the payload atomically. Equal bytes publish one payload: a second finish of the same bytes reports deduplicated true and stores nothing new. The database splits payloads into content-defined chunks of 256 KiB to 4 MiB, so edits to a large payload share most chunks with earlier versions. manifest lists the chunks, and an open stream reads any byte range: seek moves its position and read copies from it.

Published payloads are never deleted. Removing a file or pointing it at another payload keeps the earlier payload, and history keeps every earlier version of the file. An upload that does not finish publishes nothing. Closing the stream abandons it, and the next open removes what an interrupted upload left behind.

Each upload holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes. An upload that cannot fit reports out_of_memory. Calls on one stream are serialized; separate streams can run at the same time.

Digests cross the API as 64 hexadecimal digits in either case. Results show them as {#sha256 <hex>} values.

Files and directories#

Every node lives under the root directory, entity 0. A directory holds named children; a file names one published payload. Names are non-empty UTF-8 text without / or NUL, and are not . or ... Names are unique within a directory and hold at most 4096 bytes.

A record describes one node:

create creates up to 4096 nodes in one commit. A node names its parent by entity, or with under by the position of an earlier directory in the same request:

[
  {
    "name": "notes"
  },
  {
    "name": "hello.txt",
    "under": 1,
    "digest": {
      "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
    }
  }
]

A file can name only a published payload. A merge patch can also write stardust/object/digest on an entity; the commit then needs a published payload too.

Revisions#

Each write commits atomically through the commit queue. A refused write changes nothing. update, move and remove take an expected revision. Pass the revision of a record you read to refuse the write when another writer changed the node since: the call reports conflict. Pass 0 to skip the check. An update to the current payload, or a move to the current name and parent, changes nothing and keeps the revision.

With replace true, a move can replace a destination node of the same kind. A file replaces a file. A directory replaces only an empty directory. Remove deletes a file or an empty directory.

Reads#

Every tree read uses one snapshot of the database. Concurrent writes do not change a read in progress. children lists a directory in name order. path returns the names from the root to a node. files finds the files that hold a payload. A stream read verifies every chunk it reads against its digest and reports evaluation_failed when stored bytes do not match.

Without the plugin#

On a database that does not enable objects, a write that sets stardust/object/digest fails with the unavailable status and the code plugin_required. Payloads and files that the plugin stored stay in the database file and return when the plugin is enabled again.

Operations#

Objects::get#

pub fn get(&self, entity: i64) -> Result<Document>

Parameters#

Errors#

Examples#

Read the record of a file#

let mut upload = db.objects().upload()?;
upload.write(b"Hello, objects")?;
upload.finish()?;
upload.close();
let nodes = stardust::parse(r#"[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]"#, Format::DUST)?;
db.objects().create(&nodes)?;
let result = db.objects().get(1)?;

{
  "entity": 1,
  "parent": 0,
  "revision": 2,
  "name": "hello.txt",
  "kind": "file",
  "size": 14,
  "digest": {
    "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
  }
}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::lookup#

pub fn lookup(&self, parent: i64, name: &str) -> Result<Document>

Parameters#

Errors#

Examples#

Find a child of the root by name#

db.objects().create(&stardust::parse("[{name notes}]", Format::DUST)?)?;
// Entity 0 is the root directory.
let result = db.objects().lookup(0, "notes")?;

{
  "entity": 1,
  "parent": 0,
  "revision": 2,
  "name": "notes",
  "kind": "directory"
}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::children#

pub fn children(&self, parent: i64) -> Result<Document>

Parameters#

Errors#

Examples#

List the root directory#

db.objects().create(&stardust::parse("[{name notes} {name drafts}]", Format::DUST)?)?;
let result = db.objects().children(0)?;

[
  {
    "entity": 2,
    "parent": 0,
    "revision": 3,
    "name": "drafts",
    "kind": "directory"
  },
  {
    "entity": 1,
    "parent": 0,
    "revision": 3,
    "name": "notes",
    "kind": "directory"
  }
]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::path#

pub fn path(&self, entity: i64) -> Result<Document>

Parameters#

Errors#

Examples#

Read the path of a nested directory#

db.objects().create(&stardust::parse("[{name notes} {name drafts under 1}]", Format::DUST)?)?;
let result = db.objects().path(2)?;

[
  "notes",
  "drafts"
]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::files#

pub fn files(&self, digest: &str) -> Result<Document>

Parameters#

Errors#

Examples#

Find the files that hold a payload#

let mut upload = db.objects().upload()?;
upload.write(b"Hello, objects")?;
upload.finish()?;
upload.close();
let nodes = stardust::parse(
    r#"[{name a.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}} {name b.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]"#,
    Format::DUST,
)?;
db.objects().create(&nodes)?;
let result = db.objects().files("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")?;

[
  1,
  2
]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::manifest#

pub fn manifest(&self, digest: &str) -> Result<Document>

Parameters#

Errors#

Notes#

The manifest is {digest size chunks}. chunks lists {digest size} in byte order and is omitted for an empty payload.

Examples#

Read the manifest of a published payload#

let mut upload = db.objects().upload()?;
upload.write(b"Hello, objects")?;
upload.finish()?;
upload.close();
let result = db.objects().manifest("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")?;

{
  "digest": {
    "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
  },
  "size": 14,
  "chunks": [
    {
      "digest": {
        "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
      },
      "size": 14
    }
  ]
}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::create#

pub fn create(&self, nodes: &Document) -> Result<Document>

Parameters#

Errors#

Notes#

Nodes are created in order. A node names its parent by entity, or by the one-based position of an earlier directory in the same array with under.

Examples#

Create a directory and a file in one commit#

let mut upload = db.objects().upload()?;
upload.write(b"Hello, objects")?;
upload.finish()?;
upload.close();
// under 1 places hello.txt in the first node of the request.
let nodes = stardust::parse(r#"[{name notes} {name hello.txt under 1 digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]"#, Format::DUST)?;
let result = db.objects().create(&nodes)?;

[
  {
    "entity": 1,
    "parent": 0,
    "revision": 3,
    "name": "notes",
    "kind": "directory"
  },
  {
    "entity": 2,
    "parent": 1,
    "revision": 3,
    "name": "hello.txt",
    "kind": "file",
    "size": 14,
    "digest": {
      "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
    }
  }
]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::update#

pub fn update(&self, entity: i64, digest: &str, expected_revision: i64) -> Result<Document>

Parameters#

Errors#

Examples#

Point a file at a new payload#

for text in [&b"Hello, objects"[..], &b"Revised notes"[..]] {
    let mut upload = db.objects().upload()?;
    upload.write(text)?;
    upload.finish()?;
}
let nodes = stardust::parse(r#"[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]"#, Format::DUST)?;
db.objects().create(&nodes)?;
// 0 skips the revision check; pass a record's revision to refuse a stale write.
let result = db.objects().update(1, "2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2", 0)?;

{
  "entity": 1,
  "parent": 0,
  "revision": 3,
  "name": "hello.txt",
  "kind": "file",
  "size": 13,
  "digest": {
    "#sha256": "2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2"
  }
}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::move#

pub fn r#move(&self, entity: i64, parent: i64, name: &str, expected_revision: i64, replace: bool) -> Result<Document>

Parameters#

Errors#

Notes#

With replace, a node of the same kind at the destination is replaced; a directory only when it is empty. A move into the node's own subtree is invalid.

Examples#

Move a directory into another#

db.objects().create(&stardust::parse("[{name notes} {name drafts}]", Format::DUST)?)?;
// false refuses to replace a node that already has the name.
let result = db.objects().r#move(2, 1, "archive", 0, false)?;

{
  "entity": 2,
  "parent": 1,
  "revision": 4,
  "name": "archive",
  "kind": "directory"
}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::remove#

pub fn remove(&self, entity: i64, expected_revision: i64) -> Result<()>

Parameters#

Errors#

Notes#

The payload stays published.

Examples#

Remove an empty directory#

db.objects().create(&stardust::parse("[{name drafts}]", Format::DUST)?)?;
db.objects().remove(1, 0)?;

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects::upload#

pub fn upload(&self) -> Result<PluginStream>

Parameters#

Errors#

Notes#

The stream holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes. Write appends bytes; finish publishes the payload and returns {digest size deduplicated}.

Examples#

Upload a payload and publish it#

// Dropping the upload abandons it unless it finished.
let mut upload = db.objects().upload()?;
// The upload stream implements std::io::Write, so std::io::copy can stream into it.
std::io::copy(&mut &b"Hello, objects"[..], &mut upload)?;
let result = upload.finish()?;

{
  "digest": {
    "#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
  },
  "size": 14,
  "deduplicated": false
}

C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release

Objects::open#

pub fn open(&self, digest: &str) -> Result<PluginStream>

Parameters#

Errors#

Notes#

Read copies bytes from the position and verifies each chunk it touches; seek moves the position, from the end with the payload size. The reader charges its state to the database memory budget until it is released or the database closes.

Examples#

Read a range of a payload#

let mut upload = db.objects().upload()?;
upload.write(b"Hello, objects")?;
upload.finish()?;
upload.close();
let mut reader = db.objects().open("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")?;
// The reader also implements std::io::Read and std::io::Seek.
reader.seek(7, 0)?;
let mut data = [0u8; 7];
let read = reader.read(&mut data)?;
// data holds b"objects".
let result = read as i64;

7

C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release