Object store

The objects plugin stores binary payloads in the database file and names them in a tree of files and directories. Enable it at open with plugins {objects {}}. It has no settings. See plugin selection for the open options. Calls without it report the unavailable status.

The typed wrappers use plugin_call or plugin_stream with plugin objects. Request members use lowerCamelCase, such as expectedRevision.

Payloads

A payload is an immutable byte sequence. Its digest is the SHA-256 of all its bytes. Write a payload through an upload stream: write appends bytes in order and finish publishes the payload atomically. Equal bytes publish one payload: a second finish of the same bytes reports deduplicated true and stores nothing new. The database splits payloads into content-defined chunks of 256 KiB to 4 MiB, so edits to a large payload share most chunks with earlier versions. manifest lists the chunks, and an open stream reads any byte range: seek moves its position and read copies from it.

Published payloads are never deleted. Removing a file or pointing it at another payload keeps the earlier payload, and history keeps every earlier version of the file. An upload that does not finish publishes nothing. Closing the stream abandons it, and the next open removes what an interrupted upload left behind.

Each upload holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes. An upload that cannot fit reports out_of_memory. Calls on one stream are serialized; separate streams can run at the same time.

Digests cross the API as 64 hexadecimal digits in either case. Results show them as {#sha256 <hex>} values.

Files and directories

Every node lives under the root directory, entity 0. A directory holds named children; a file names one published payload. Names are non-empty UTF-8 text without / or NUL, and are not . or ... Names are unique within a directory and hold at most 4096 bytes.

A record describes one node:

Member	Value
`entity`	The node's entity.
`parent`	The parent directory's entity, `0` for the root.
`name`	The node's name in its parent.
`kind`	`file` or `directory`.
`revision`	The transaction that last changed the node.
`size`	A file's payload size in bytes.
`digest`	A file's payload digest.

create creates up to 4096 nodes in one commit. A node names its parent by entity, or with under by the position of an earlier directory in the same request:

dust

136 B

[
  {name notes}
  {name   hello.txt
   under  1
   digest {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}}]

A file can name only a published payload. A merge patch can also write stardust/object/digest on an entity; the commit then needs a published payload too.

Revisions

Each write commits atomically through the commit queue. A refused write changes nothing. update, move and remove take an expected revision. Pass the revision of a record you read to refuse the write when another writer changed the node since: the call reports conflict. Pass 0 to skip the check. An update to the current payload, or a move to the current name and parent, changes nothing and keeps the revision.

With replace true, a move can replace a destination node of the same kind. A file replaces a file. A directory replaces only an empty directory. Remove deletes a file or an empty directory.

Reads

Every tree read uses one snapshot of the database. Concurrent writes do not change a read in progress. children lists a directory in name order. path returns the names from the root to a node. files finds the files that hold a payload. A stream read verifies every chunk it reads against its digest and reports evaluation_failed when stored bytes do not match.

Without the plugin

On a database that does not enable objects, a write that sets stardust/object/digest fails with the unavailable status and the code plugin_required. Payloads and files that the plugin stored stay in the database file and return when the plugin is enabled again.

Operations

Operation	Returns	Summary
Objects.get	document	objects.get(entity) returns the owned record document of an object file or directory.
Objects.lookup	document	objects.lookup(parent, name) returns the owned record document of a directory's child.
Objects.children	document	objects.children(parent) returns an owned array document of a directory's child records in name order.
Objects.path	document	objects.path(entity) returns an owned array document of the names from the root to an object node.
Objects.files	document	objects.files(digest) returns an owned ascending array document of the file entities that hold a payload.
Objects.manifest	document	objects.manifest(digest) returns a published payload's owned manifest document.
Objects.create	document	objects.create(nodes) creates files and directories in one commit and returns an owned array document of their records.
Objects.update	document	objects.update(entity, digest, expected_revision) points a file at another published payload and returns its owned record document.
Objects.move	document	objects.move(entity, parent, name, expected_revision, replace) renames or moves a node and returns its owned record document.
Objects.remove	none	objects.remove(entity, expected_revision) removes a file or an empty directory.
Objects.upload	plugin_stream	objects.upload() opens an owned plugin stream that uploads one payload.
Objects.open	plugin_stream	objects.open(digest) opens an owned plugin stream that reads a published payload.

Objects.get

python

38 B

def get(self, entity: int) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`entity`	`int`	Value	Required	The node's entity.

Errors

Status	When
`"invalid_argument"`	entity is not a user entity
`"not_found"`	no object node has the entity
`"evaluation_failed"`	stored tree metadata or the manifest of a named payload is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

Read the record of a file

python

377 B

async with await db.objects().upload() as upload:
    await upload.write(b"Hello, objects")
    await (await upload.finish()).close()
async with await stardust.parse('[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]', "dust") as nodes:
    await (await db.objects().create(nodes)).close()
result = await db.objects().get(1)

Result / 169 B

{entity   1
 parent   0
 revision 2
 name     hello.txt
 kind     file
 size     14
 digest   {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.lookup

python

52 B

def lookup(self, parent: int, name: str) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`parent`	`int`	Value	Required	The directory's entity. 0 selects the root.
`name`	`str`	Value	Required	The child's name.

Errors

Status	When
`"invalid_argument"`	the parent or the name is invalid, or the parent is a file
`"not_found"`	the parent has no child with the name, or the parent does not exist
`"evaluation_failed"`	stored tree metadata or the manifest of a named payload is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

Find a child of the root by name

python

201 B

async with await stardust.parse("[{name notes}]", "dust") as nodes:
    await (await db.objects().create(nodes)).close()
# Entity 0 is the root directory.
result = await db.objects().lookup(0, "notes")

Result / 72 B

{entity   1
 parent   0
 revision 2
 name     notes
 kind     directory}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.children

python

43 B

def children(self, parent: int) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`parent`	`int`	Value	Required	The directory's entity. 0 selects the root.

Errors

Status	When
`"invalid_argument"`	the parent is invalid or is a file
`"not_found"`	the parent does not exist
`"evaluation_failed"`	stored tree metadata or the manifest of a named payload is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

List the root directory

python

174 B

async with await stardust.parse("[{name notes} {name drafts}]", "dust") as nodes:
    await (await db.objects().create(nodes)).close()
result = await db.objects().children(0)

Result / 157 B

[{entity   2
  parent   0
  revision 3
  name     drafts
  kind     directory}
 {entity   1
  parent   0
  revision 3
  name     notes
  kind     directory}]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.path

python

39 B

def path(self, entity: int) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`entity`	`int`	Value	Required	The node's entity. 0 selects the root, whose path is empty.

Errors

Status	When
`"invalid_argument"`	entity is not a user entity
`"not_found"`	the node or one of its ancestors does not exist
`"evaluation_failed"`	stored tree metadata or the manifest of a named payload is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

Read the path of a nested directory

python

178 B

async with await stardust.parse("[{name notes} {name drafts under 1}]", "dust") as nodes:
    await (await db.objects().create(nodes)).close()
result = await db.objects().path(2)

Result / 14 B

[notes drafts]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.files

python

40 B

def files(self, digest: str) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`digest`	`str`	Value	Required	The payload's SHA-256 digest as 64 hexadecimal digits.

Errors

Status	When
`"invalid_argument"`	the digest is not 64 hexadecimal digits
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

Find the files that hold a payload

python

537 B

async with await db.objects().upload() as upload:
    await upload.write(b"Hello, objects")
    await (await upload.finish()).close()
async with await stardust.parse('[{name a.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}} {name b.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]', "dust") as nodes:
    await (await db.objects().create(nodes)).close()
result = await db.objects().files("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")

Result / 5 B

[1 2]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.manifest

python

43 B

def manifest(self, digest: str) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`digest`	`str`	Value	Required	The payload's SHA-256 digest as 64 hexadecimal digits.

Errors

Status	When
`"invalid_argument"`	the digest is not 64 hexadecimal digits
`"not_found"`	no payload with the digest is published
`"evaluation_failed"`	the stored manifest is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Notes

The manifest is {digest size chunks}. chunks lists {digest size} in byte order and is omitted for an empty payload.

Examples

Read the manifest of a published payload

python

238 B

async with await db.objects().upload() as upload:
    await upload.write(b"Hello, objects")
    await (await upload.finish()).close()
result = await db.objects().manifest("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")

Result / 208 B

{digest {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}
 size   14
 chunks [{digest {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}
          size   14}]}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.create

python

45 B

def create(self, nodes: Document) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`nodes`	`Document`	Borrowed for the call	Required	A borrowed array of at most 4096 nodes. Each has name, and parent (entity, default 0) or under (one-based position of an earlier directory). A file also has digest {#sha256 hex}. Without digest, the node is a directory.

Errors

Status	When
`"invalid_argument"`	the array is empty or malformed, a name is invalid, or a parent is a file
`"not_found"`	a parent does not exist or a digest is not published
`"conflict"`	a name is already taken in its parent or appears twice, or the database holds writes
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Notes

Nodes are created in order. A node names its parent by entity, or by the one-based position of an earlier directory in the same array with under.

Examples

Create a directory and a file in one commit

python

417 B

async with await db.objects().upload() as upload:
    await upload.write(b"Hello, objects")
    await (await upload.finish()).close()
# under 1 places hello.txt in the first node of the request.
async with await stardust.parse('[{name notes} {name hello.txt under 1 digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]', "dust") as nodes:
    result = await db.objects().create(nodes)

Result / 255 B

[{entity   1
  parent   0
  revision 3
  name     notes
  kind     directory}
 {entity   2
  parent   1
  revision 3
  name     hello.txt
  kind     file
  size     14
  digest   {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}}]

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.update

python

78 B

def update(self, entity: int, digest: str, expected_revision: int) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`entity`	`int`	Value	Required	The file's entity.
`digest`	`str`	Value	Required	The payload's SHA-256 digest as 64 hexadecimal digits.
`expected_revision`	`int`	Value	Required	The transaction that last touched the node, from its record. 0 skips the check.

Errors

Status	When
`"invalid_argument"`	the entity is a directory or invalid, the digest is malformed, or `expected_revision` is negative
`"not_found"`	the file does not exist or the digest is not published
`"conflict"`	`expected_revision` is stale, or the database holds writes
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Examples

Point a file at a new payload

python

581 B

for text in (b"Hello, objects", b"Revised notes"):
    async with await db.objects().upload() as upload:
        await upload.write(text)
        await (await upload.finish()).close()
async with await stardust.parse('[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]', "dust") as nodes:
    await (await db.objects().create(nodes)).close()
# 0 skips the revision check; pass a record's revision to refuse a stale write.
result = await db.objects().update(1, "2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2", 0)

Result / 169 B

{entity   1
 parent   0
 revision 3
 name     hello.txt
 kind     file
 size     13
 digest   {#sha256 2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2}}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.move

python

102 B

def move(self, entity: int, parent: int, name: str, expected_revision: int, replace: bool) -> Document

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`entity`	`int`	Value	Required	The node to move.
`parent`	`int`	Value	Required	The destination directory. 0 selects the root.
`name`	`str`	Value	Required	The node's new name.
`expected_revision`	`int`	Value	Required	The transaction that last touched the node, from its record. 0 skips the check.
`replace`	`bool`	Value	Required	Replace an existing node of the same kind at the destination. False reports conflict for an occupied name.

Errors

Status	When
`"invalid_argument"`	the request is malformed, the destination is a file or inside the node, the kinds differ, or a replaced directory is not empty
`"not_found"`	the node or the destination does not exist
`"conflict"`	the name is occupied without replace, `expected_revision` is stale, or the database holds writes
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Notes

With replace, a node of the same kind at the destination is replaced; a directory only when it is empty. A move into the node's own subtree is invalid.

Examples

Move a directory into another

python

255 B

async with await stardust.parse("[{name notes} {name drafts}]", "dust") as nodes:
    await (await db.objects().create(nodes)).close()
# False refuses to replace a node that already has the name.
result = await db.objects().move(2, 1, "archive", 0, False)

Result / 74 B

{entity   2
 parent   1
 revision 4
 name     archive
 kind     directory}

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.remove

python

61 B

def remove(self, entity: int, expected_revision: int) -> None

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`entity`	`int`	Value	Required	The node to remove.
`expected_revision`	`int`	Value	Required	The transaction that last touched the node, from its record. 0 skips the check.

Errors

Status	When
`"invalid_argument"`	the entity is invalid, the directory is not empty, or `expected_revision` is negative
`"not_found"`	the node does not exist
`"conflict"`	`expected_revision` is stale, or the database holds writes
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Notes

The payload stays published.

Examples

Remove an empty directory

python

153 B

async with await stardust.parse("[{name drafts}]", "dust") as nodes:
    await (await db.objects().create(nodes)).close()
await db.objects().remove(1, 0)

C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release

Objects.upload

python

32 B

def upload(self) -> PluginStream

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.

Errors

Status	When
`"unavailable"`	the database does not enable the objects plugin
`"conflict"`	the database holds writes
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

Notes

The stream holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes. Write appends bytes; finish publishes the payload and returns {digest size deduplicated}.

Examples

Upload a payload and publish it

python

202 B

# Close abandons the upload unless it finished.
async with await db.objects().upload() as upload:
    await upload.write(b"Hello, ")
    await upload.write(b"objects")
    result = await upload.finish()

Result / 126 B

{digest       {#sha256 95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb}
 size         14
 deduplicated false}

C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release

Objects.open

python

43 B

def open(self, digest: str) -> PluginStream

Parameters

Parameter	Type	Ownership	Default	Meaning
`self`	`Objects`	Borrowed for the call	Required	The objects owner for this operation. Keep it alive during the call.
`digest`	`str`	Value	Required	The payload's SHA-256 digest as 64 hexadecimal digits.

Errors

Status	When
`"invalid_argument"`	the digest is not 64 hexadecimal digits
`"not_found"`	no payload with the digest is published
`"evaluation_failed"`	the stored manifest is corrupted
`"unavailable"`	the database does not enable the objects plugin
`"closed"`	the database is closed
`"out_of_memory"`	native memory is exhausted

async with await db.objects().upload() as upload:
    await upload.write(b"Hello, objects")
    await (await upload.finish()).close()
# The reader is an async file-like object.
async with await db.objects().open("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb") as reader:
    await reader.seek(7)
    data = await reader.read(7)
# data holds b"objects".
result = len(data)

Result / 1 B

C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release