Object store
The objects plugin stores binary payloads in the database file and names them in a tree of files and directories.
Enable it at open with plugins {objects {}}. It has no settings.
See plugin selection for the open options.
Calls without it report the unavailable status.
The typed wrappers use plugin_call or plugin_stream with plugin objects.
Request members use lowerCamelCase, such as expectedRevision.
Payloads
A payload is an immutable byte sequence. Its digest is the SHA-256 of all its bytes.
Write a payload through an upload stream: write appends bytes in order and finish publishes the payload atomically.
Equal bytes publish one payload: a second finish of the same bytes reports deduplicated true and stores nothing new.
The database splits payloads into content-defined chunks of 256 KiB to 4 MiB, so edits to a large payload share most chunks with earlier versions.
manifest lists the chunks, and an open stream reads any byte range: seek moves its position and read copies from it.
Published payloads are never deleted. Removing a file or pointing it at another payload keeps the earlier payload, and history keeps every earlier version of the file. An upload that does not finish publishes nothing. Closing the stream abandons it, and the next open removes what an interrupted upload left behind.
Each upload holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes.
An upload that cannot fit reports out_of_memory. Calls on one stream are serialized; separate streams can run at the same time.
Digests cross the API as 64 hexadecimal digits in either case. Results show them as {#sha256 <hex>} values.
Files and directories
Every node lives under the root directory, entity 0. A directory holds named children; a file names one published payload.
Names are non-empty UTF-8 text without / or NUL, and are not . or ... Names are unique within a directory and hold at most 4096 bytes.
A record describes one node:
| Member | Value |
|---|---|
entity | The node's entity. |
parent | The parent directory's entity, 0 for the root. |
name | The node's name in its parent. |
kind | file or directory. |
revision | The transaction that last changed the node. |
size | A file's payload size in bytes. |
digest | A file's payload digest. |
create creates up to 4096 nodes in one commit.
A node names its parent by entity, or with under by the position of an earlier directory in the same request:
[
{
"name": "notes"
},
{
"name": "hello.txt",
"under": 1,
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
}
}
]
A file can name only a published payload.
A merge patch can also write stardust/object/digest on an entity; the commit then needs a published payload too.
Revisions
Each write commits atomically through the commit queue. A refused write changes nothing.
update, move and remove take an expected revision.
Pass the revision of a record you read to refuse the write when another writer changed the node since: the call reports conflict.
Pass 0 to skip the check.
An update to the current payload, or a move to the current name and parent, changes nothing and keeps the revision.
With replace true, a move can replace a destination node of the same kind.
A file replaces a file. A directory replaces only an empty directory.
Remove deletes a file or an empty directory.
Reads
Every tree read uses one snapshot of the database. Concurrent writes do not change a read in progress.
children lists a directory in name order.
path returns the names from the root to a node.
files finds the files that hold a payload.
A stream read verifies every chunk it reads against its digest and reports evaluation_failed when stored bytes do not match.
Without the plugin
On a database that does not enable objects, a write that sets stardust/object/digest fails with the unavailable status and the code plugin_required.
Payloads and files that the plugin stored stay in the database file and return when the plugin is enabled again.
Operations
| Operation | Returns | Summary |
|---|---|---|
| Objects.Get | document | objects.get(entity) returns the owned record document of an object file or directory. |
| Objects.Lookup | document | objects.lookup(parent, name) returns the owned record document of a directory's child. |
| Objects.Children | document | objects.children(parent) returns an owned array document of a directory's child records in name order. |
| Objects.Path | document | objects.path(entity) returns an owned array document of the names from the root to an object node. |
| Objects.Files | document | objects.files(digest) returns an owned ascending array document of the file entities that hold a payload. |
| Objects.Manifest | document | objects.manifest(digest) returns a published payload's owned manifest document. |
| Objects.Create | document | objects.create(nodes) creates files and directories in one commit and returns an owned array document of their records. |
| Objects.Update | document | objects.update(entity, digest, expected_revision) points a file at another published payload and returns its owned record document. |
| Objects.Move | document | objects.move(entity, parent, name, expected_revision, replace) renames or moves a node and returns its owned record document. |
| Objects.Remove | none | objects.remove(entity, expected_revision) removes a file or an empty directory. |
| Objects.Upload | plugin_stream | objects.upload() opens an owned plugin stream that uploads one payload. |
| Objects.Open | plugin_stream | objects.open(digest) opens an owned plugin stream that reads a published payload. |
Objects.Get
func (v Objects) Get(entity int64) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
entity | int64 | Value | Required | The node's entity. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | entity is not a user entity |
StatusNotFound | no object node has the entity |
StatusEvaluationFailed | stored tree metadata or the manifest of a named payload is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
Read the record of a file
upload, err := db.Objects().Upload()
if err != nil {
return err
}
defer upload.Close()
if _, err := upload.Write([]byte("Hello, objects")); err != nil {
return err
}
published, err := upload.Finish()
if err != nil {
return err
}
defer published.Close()
nodes, err := stardust.Parse([]byte(`[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
result, err := db.Objects().Get(1)
if err != nil {
return err
}
defer result.Close(){
"entity": 1,
"parent": 0,
"revision": 2,
"name": "hello.txt",
"kind": "file",
"size": 14,
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
}
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Lookup
func (v Objects) Lookup(parent int64, name string) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
parent | int64 | Value | Required | The directory's entity. 0 selects the root. |
name | string | Value | Required | The child's name. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the parent or the name is invalid, or the parent is a file |
StatusNotFound | the parent has no child with the name, or the parent does not exist |
StatusEvaluationFailed | stored tree metadata or the manifest of a named payload is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
Find a child of the root by name
nodes, err := stardust.Parse([]byte(`[{name notes}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
// Entity 0 is the root directory.
result, err := db.Objects().Lookup(0, "notes")
if err != nil {
return err
}
defer result.Close(){
"entity": 1,
"parent": 0,
"revision": 2,
"name": "notes",
"kind": "directory"
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Children
func (v Objects) Children(parent int64) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
parent | int64 | Value | Required | The directory's entity. 0 selects the root. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the parent is invalid or is a file |
StatusNotFound | the parent does not exist |
StatusEvaluationFailed | stored tree metadata or the manifest of a named payload is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
List the root directory
nodes, err := stardust.Parse([]byte(`[{name notes} {name drafts}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
result, err := db.Objects().Children(0)
if err != nil {
return err
}
defer result.Close()[
{
"entity": 2,
"parent": 0,
"revision": 3,
"name": "drafts",
"kind": "directory"
},
{
"entity": 1,
"parent": 0,
"revision": 3,
"name": "notes",
"kind": "directory"
}
]C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Path
func (v Objects) Path(entity int64) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
entity | int64 | Value | Required | The node's entity. 0 selects the root, whose path is empty. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | entity is not a user entity |
StatusNotFound | the node or one of its ancestors does not exist |
StatusEvaluationFailed | stored tree metadata or the manifest of a named payload is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
Read the path of a nested directory
nodes, err := stardust.Parse([]byte(`[{name notes} {name drafts under 1}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
result, err := db.Objects().Path(2)
if err != nil {
return err
}
defer result.Close()[ "notes", "drafts" ]
C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Files
func (v Objects) Files(digest string) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
digest | string | Value | Required | The payload's SHA-256 digest as 64 hexadecimal digits. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the digest is not 64 hexadecimal digits |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
Find the files that hold a payload
upload, err := db.Objects().Upload()
if err != nil {
return err
}
defer upload.Close()
if _, err := upload.Write([]byte("Hello, objects")); err != nil {
return err
}
published, err := upload.Finish()
if err != nil {
return err
}
defer published.Close()
nodes, err := stardust.Parse([]byte(`[{name a.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}} {name b.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
result, err := db.Objects().Files("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")
if err != nil {
return err
}
defer result.Close()[ 1, 2 ]
C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Manifest
func (v Objects) Manifest(digest string) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
digest | string | Value | Required | The payload's SHA-256 digest as 64 hexadecimal digits. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the digest is not 64 hexadecimal digits |
StatusNotFound | no payload with the digest is published |
StatusEvaluationFailed | the stored manifest is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
The manifest is {digest size chunks}. chunks lists {digest size} in byte order and is omitted for an empty payload.
Examples
Read the manifest of a published payload
upload, err := db.Objects().Upload()
if err != nil {
return err
}
defer upload.Close()
if _, err := upload.Write([]byte("Hello, objects")); err != nil {
return err
}
published, err := upload.Finish()
if err != nil {
return err
}
defer published.Close()
result, err := db.Objects().Manifest("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")
if err != nil {
return err
}
defer result.Close(){
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
},
"size": 14,
"chunks": [
{
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
},
"size": 14
}
]
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Create
func (v Objects) Create(nodes *Document) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
nodes | *Document | Borrowed for the call | Required | A borrowed array of at most 4096 nodes. Each has name, and parent (entity, default 0) or under (one-based position of an earlier directory). A file also has digest {#sha256 hex}. Without digest, the node is a directory. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the array is empty or malformed, a name is invalid, or a parent is a file |
StatusNotFound | a parent does not exist or a digest is not published |
StatusConflict | a name is already taken in its parent or appears twice, or the database holds writes |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
Nodes are created in order. A node names its parent by entity, or by the one-based position of an earlier directory in the same array with under.
Examples
Create a directory and a file in one commit
upload, err := db.Objects().Upload()
if err != nil {
return err
}
defer upload.Close()
if _, err := upload.Write([]byte("Hello, objects")); err != nil {
return err
}
published, err := upload.Finish()
if err != nil {
return err
}
defer published.Close()
// under 1 places hello.txt in the first node of the request.
nodes, err := stardust.Parse([]byte(`[{name notes} {name hello.txt under 1 digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
result, err := db.Objects().Create(nodes)
if err != nil {
return err
}
defer result.Close()[
{
"entity": 1,
"parent": 0,
"revision": 3,
"name": "notes",
"kind": "directory"
},
{
"entity": 2,
"parent": 1,
"revision": 3,
"name": "hello.txt",
"kind": "file",
"size": 14,
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
}
}
]C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Update
func (v Objects) Update(entity int64, digest string, expectedRevision int64) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
entity | int64 | Value | Required | The file's entity. |
digest | string | Value | Required | The payload's SHA-256 digest as 64 hexadecimal digits. |
expectedRevision | int64 | Value | Required | The transaction that last touched the node, from its record. 0 skips the check. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the entity is a directory or invalid, the digest is malformed, or expected_revision is negative |
StatusNotFound | the file does not exist or the digest is not published |
StatusConflict | expected_revision is stale, or the database holds writes |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Examples
Point a file at a new payload
for _, text := range []string{"Hello, objects", "Revised notes"} {
upload, err := db.Objects().Upload()
if err != nil {
return err
}
if _, err := upload.Write([]byte(text)); err != nil {
upload.Close()
return err
}
published, err := upload.Finish()
upload.Close()
if err != nil {
return err
}
published.Close()
}
nodes, err := stardust.Parse([]byte(`[{name hello.txt digest {#sha256 "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"}}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
// 0 skips the revision check; pass a record's revision to refuse a stale write.
result, err := db.Objects().Update(1, "2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2", 0)
if err != nil {
return err
}
defer result.Close(){
"entity": 1,
"parent": 0,
"revision": 3,
"name": "hello.txt",
"kind": "file",
"size": 13,
"digest": {
"#sha256": "2b6d3a4299dac5900171112d4fd569b48e54711d8937f6ff5ca00528d47afad2"
}
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Move
func (v Objects) Move(entity int64, parent int64, name string, expectedRevision int64, replace bool) (*Document, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
entity | int64 | Value | Required | The node to move. |
parent | int64 | Value | Required | The destination directory. 0 selects the root. |
name | string | Value | Required | The node's new name. |
expectedRevision | int64 | Value | Required | The transaction that last touched the node, from its record. 0 skips the check. |
replace | bool | Value | Required | Replace an existing node of the same kind at the destination. False reports conflict for an occupied name. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the request is malformed, the destination is a file or inside the node, the kinds differ, or a replaced directory is not empty |
StatusNotFound | the node or the destination does not exist |
StatusConflict | the name is occupied without replace, expected_revision is stale, or the database holds writes |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
With replace, a node of the same kind at the destination is replaced; a directory only when it is empty. A move into the node's own subtree is invalid.
Examples
Move a directory into another
nodes, err := stardust.Parse([]byte(`[{name notes} {name drafts}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
// false refuses to replace a node that already has the name.
result, err := db.Objects().Move(2, 1, "archive", 0, false)
if err != nil {
return err
}
defer result.Close(){
"entity": 2,
"parent": 1,
"revision": 4,
"name": "archive",
"kind": "directory"
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Remove
func (v Objects) Remove(entity int64, expectedRevision int64) errorParameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
entity | int64 | Value | Required | The node to remove. |
expectedRevision | int64 | Value | Required | The transaction that last touched the node, from its record. 0 skips the check. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the entity is invalid, the directory is not empty, or expected_revision is negative |
StatusNotFound | the node does not exist |
StatusConflict | expected_revision is stale, or the database holds writes |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
The payload stays published.
Examples
Remove an empty directory
nodes, err := stardust.Parse([]byte(`[{name drafts}]`), stardust.DUST)
if err != nil {
return err
}
defer nodes.Close()
created, err := db.Objects().Create(nodes)
if err != nil {
return err
}
created.Close()
if err := db.Objects().Remove(1, 0); err != nil {
return err
}C exports: stardust_db_plugin_call, stardust_result_take_document, stardust_result_error, stardust_result_release
Objects.Upload
func (v Objects) Upload() (*PluginStream, error)Errors
| Status | When |
|---|---|
StatusUnavailable | the database does not enable the objects plugin |
StatusConflict | the database holds writes |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
The stream holds a working buffer of up to 4 MiB from the database memory budget until it finishes or closes. Write appends bytes; finish publishes the payload and returns {digest size deduplicated}.
Examples
Upload a payload and publish it
upload, err := db.Objects().Upload()
if err != nil {
return err
}
// Close abandons the upload unless it finished.
defer upload.Close()
// The upload stream implements io.Writer, so io.Copy can stream into it.
if _, err := io.Copy(upload, strings.NewReader("Hello, objects")); err != nil {
return err
}
result, err := upload.Finish()
if err != nil {
return err
}
defer result.Close(){
"digest": {
"#sha256": "95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb"
},
"size": 14,
"deduplicated": false
}C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release
Objects.Open
func (v Objects) Open(digest string) (*PluginStream, error)Parameters
| Parameter | Type | Ownership | Default | Meaning |
|---|---|---|---|---|
digest | string | Value | Required | The payload's SHA-256 digest as 64 hexadecimal digits. |
Errors
| Status | When |
|---|---|
StatusInvalidArgument | the digest is not 64 hexadecimal digits |
StatusNotFound | no payload with the digest is published |
StatusEvaluationFailed | the stored manifest is corrupted |
StatusUnavailable | the database does not enable the objects plugin |
StatusClosed | the database is closed |
StatusOutOfMemory | native memory is exhausted |
Notes
Read copies bytes from the position and verifies each chunk it touches; seek moves the position, from the end with the payload size. The reader charges its state to the database memory budget until it is released or the database closes.
Examples
Read a range of a payload
upload, err := db.Objects().Upload()
if err != nil {
return err
}
defer upload.Close()
if _, err := upload.Write([]byte("Hello, objects")); err != nil {
return err
}
published, err := upload.Finish()
if err != nil {
return err
}
defer published.Close()
reader, err := db.Objects().Open("95e84da38cbefdf3f9f6ef02201074c73124f7cf2025d3e1d69e591a410856fb")
if err != nil {
return err
}
defer reader.Close()
// The reader is an io.ReadSeeker and an io.ReaderAt.
if _, err := reader.Seek(7, io.SeekStart); err != nil {
return err
}
data := make([]byte, 7)
read, err := io.ReadFull(reader, data)
if err != nil {
return err
}
// data holds "objects".
result := int64(read)7
C exports: stardust_db_plugin_stream_open, stardust_result_error, stardust_result_release