# Q Metadata Specification

## 1. Introduction

This document lists the standard metadata keys for Q data items, discussing the rules of metadata insertion, processing and validation.

### 1.1. Definitions

To avoid confusions in terminology, this document will strictly abide the following definitions:

Term | Definition
`key` | A metadata category name, technically a `key` as the word is used with Java `Hashtable` and Python `dict` objects.

`uri`

A Uniform Resource Indicator for an item of content stored within the Q network.
Q URIs have the form: `Q:<basename>[,<cryptoKey>][<path>]`
Some examples:

- `Q:fhvnr3HFSK234khsf90sdh42fsh` (a plain hash uri, no cryptoKey)
- `Q:e54fhjeo39schr2kcy4osEH478D/files/johnny.mp3` (a secure space URI, no cryptoKey)
- `Q:vhfh4se987WwfkhwWFEwkh3234S,47fhh2dkhseiyu` (a plain hash URI, with a cryptoKey)

`basename` | The basic element of a Q uri. This will be a base64-encoded hash - refer below to URI calculation procedures
`cryptoKey` | An optional session encryption key for the stored data, encoded as base64. This affords some protection to server node operators, and gives them a level of plausible deniability for whatever gets stored in their server's datastore without their direct human awareness.

`path`

Whever an item of content is inserted in `secure space` mode, this path serves as a pseudo-pathname, and is conceptually similar to the `path` component in (for example) standard HTTP URLs `http://<domainname>[:<port>][<path>]`, such as `http://slashdot.org/faq/editorial.shtml` (whose `path` is `/faq/editorial.shtml`).
Paths, if not empty, should contain a leading slash ("/"). If an application specifies a non-empty `path` that doesn't begin with a leading '/', a '/' will be automatically prepended by the receiving node.

`plain hash` | A mode of inserting items, whereby the security of the resulting URI comes from computing the URI from a hash of the item's data and metadata (and imposing a mathematical barrier against spoofing content under a given URI). Corresponds to Freenet's `CHK@` keys.

`secure space`

A mode of inserting items where the security of the URI is based not on a hash of the item's data and metadata (as with `plain hash` mode), but on the `privateKey` provided by the application, and a content signature created from that private key. Corresponds to Freenet's `SSK@` keys. Within a secure space, you can insert any number of items under different pseudo-pathnames (as is the case with Freenet SSK keys).

### 2.1. Keys Inserted By Application Before sending `putItem` RPCs

As the heading suggests, this is a list of metadata keys which should be inserted by a Q application prior to invoking a `putItem` RPC on the local Q client node.

Key | Data Type | Description
`title` | String | Optional but strongly recommended. A free-text short description of the item, should be less than 80 characters. The idea is that applications should support a 'view' of catalogue data that shows item titles. (Prior Q convention of titles expressed as valid filename syntax has been abandoned).

`path`

String

Optional but strongly recommended. A virtual 'pathname' for the item, which should be in valid *nix absolute pathname syntax (beginning with '/', containing no '//', consisting only of alphanumerics, '-', '_', '.' and '/'.
In Q web interfaces, the `filename` component of this path will serve as the recommended filename when downloading/saving the item.
If the application also provides a `privateKey` key, the path is used in conjunction with the private key to generate `publicKey` and `signature` keys (see below), and ultimately the final `uri` under which the item can be retrieved by others.
Refer also to `mimetype` below.

`encrypt`

String

Optional. If this key is present, and has a value "1", "yes" or "true", this indicates that the application wishes the data to be stored on servers in encrypted form.
If this key is present and set to a positive value, the Q node, on receiving the `putItem` RPC, will:

1. Generate a random symmetric encryption key
2. Encrypt the item's data using this encryption key
3. Delete the `encrypt` key from the metadata
4. Enclose a base64 representation of this encryption key in the RPC response it sends back to the application (embedded in the `uri`

`type` | String | Optional but strongly recommended. A standard ed2k specifier, one of `text html image audio video archive other`
`mimetype` | String | Optional but moderately recommended. Mimetype designation of data, eg `text/html`, `image/jpeg` etc. If not specified, an attempt will be made to guess a mometype from the value of the `path` key. If this attempt fails, then this key will be set to `application/x-octet-stream` by the node receiving the `putItem` RPC.
`keywords` | String | Optional but moderately recommended. A set of keywords, under which the inserting app would like this item to be discoverable. Keywords should be entirely lower case and comma-separated. Content inserts should consider keywords carefully, and only use space characters inside keywords when necessary (eg, for flagging a distinctive phrase containing very common words).

`privateKey`

String

Optional. A Base64-encoded signing private key, in cases where the application wishes to insert an item in `signed space` mode. This can be accompanied by another key, `path`, indicating a 'path' within the signed space. If 'path' is not given, it will default to '/'.
Either way, when a node receives a `putItem` RPC containing a `privateKey` in its metadata, it removes this key and replaces it with `publicKey` and `signature`.

`path`

String

Optional. The virtual pathname, within signed space, under which to store the item. This gets ignored and deleted unless the application also provides a `privateKey` as well. But if the private key is given, the path is used in conjunction with the private key to generate `publicKey` and `signature` keys (see below).
`path` should be a 'unix-style pathname', ie, containing only slashes as (pseudo) directory delimiters, and alphanumeric, '-', '_' and '.' characters, and preferably ending in a meaningful file extension such as `.html`

`expiry`

int

Unixtime at which the inserted item should expire. When this expiry time is reached, the item won't necessarily be deleted straight away, but may be deleted whenever a node's data store is full.
If this is not provided, it will default to a given duration according to the client node's configuration.
If it is provided, by an application, then the client node will transparently generate the required 'rent payment' before caching the data item and uploading it to servers.

### 2.2. Keys Inserted By Node Upon Receipt Of `putItem` RPC

Key | Data Type | Description
`size` | Integer | Size of the data to be inserted, in bytes.
`dataHash` | String | base64-encoded SHA256 hash of data.

`uri`

String

This depends on whether the item is being inserted in plain or signed space mode.
If inserting in plain mode, then the uri is in the form `Q:somebase64hash`, where the hash is computed according to the plain hash calculation procedure.
If inserting in signed space mode, then the uri will be in the form `Q:somebase64hash/path.ext`, where the hash is computed as per the signed space hash calculation procedure, and the `/path.ext` is the verbatim value of the app-supplied `path` key.

`publicKey` | String | Base64-encoded signing public key. In cases where app provides `privateKey`, a node will derive the signing public key from the private key, delete the private key from the metadata, and replace it with its corresponding public key key.
`signature` | String | Base64-encoded signature of `path+dataHash`, created using the app-provided `privateKey`.

`rent`

String

A rent payment for the data's accommodation on the server.
Intention is to support a variety of payment tokens. Initially, the only acceptable form of payment will be a hashcash-like token, in the form `hashcash:base64string`. The `hashcash:` prefix indicates that this payment is in hashcash currency, in which case the `base64String` should decode to a 16-byte string whose SHA256 hash partially collides with `dataHash`. The greater the number of bits in the collision, the longer the data's accommodation will be 'paid up for'.
If this key is already present, a Q node will verify the hashcash, and adjust the `expiry` key value to the time the item's accommodation is paid up till.
If the key is not present:

- A client node will generate a value for this key with enough collision bits to pay the accommodation up till the given app-specified `expiry` date.
- A server node will grant temporary free accommodation, and adjust the `expiry` key to the end of the free accommodation period.

## 3. URI Determination Procedures

### 3.1. Plain Hash URI Calculation Procedure

When items are inserted in `plain` mode, the final URI is determined from a hash of the data and metadata. Security of the item is based on the mathematical difficulty of creating an arbitrary data+metadata set whose hash collides with the target URI.
Specifically, the recipe for calculating plain hash URIs is:

1. If the key `size` is missing, set this to the size of the data, in bytes
2. If the key `dataHash` is missing, set this to the base64-encoded SHA256(data)
3. If the key `title` is missing, set this to the value of `dataHash`
4. From the metadata, create a set of strings, each in the form `key=value`, where each line contains a metadata `key` and its `value`, and is terminated by an ASCII linefeed (\n, 0x10).
5. Ensure that key `uri` is omitted
6. Sort the strings into ascending ASCII sort order
7. Concatenate the strings together into one big string
8. Calculate the SHA256 hash of this string
9. Encode the hash into Base64
10. Prepend the string `Q:` to this

### 3.2. Signed Space URI Calculation Procedure

This is much simpler than determining plain hash URI, since the security of the URI is based not on hashes of data and metadata, but on the cryptographic `privateKey` given by the application.
Calculation recipe for Signed Space URIs is:

1. Calculate the SHA256 hash of the private key's binary data (not its base64 representation)
2. Encode this hash into base64, dropping any trailing '=' characters
3. Append to this the value of metadata item `path` (recall that `path`, if not empty, must begin with a '/')
4. Prepend the string `Q:` to this

The resulting URI then is in the form `Q:pubkeyHash/path.ext`

Last modified: Wed Apr 6 00:36:37 NZST 2005