# Q Protocol Specification

(first draft by aum)
Return to Q Homepage
Introduction | XML-RPC | Architecture | Commands | Example Code | Metadata | Security | Contact us

## 1. Introduction

This document describes details of the interfaces between the various entities in the Q network - server nodes, client nodes and client applications.
Purpose is to:

- Assist with people writing user client applications, such as GUI apps, command-line apps, or integrate Q in to existing apps.
- Permit alternative implementations of any of these entities, in any programming language.
- Help interested parties to gain a quick understanding of Q's architecture, perhaps with a view to contributing ideas for improvement.

## 2. XML-RPC Interface

### 2.1. WTF? All those ugly complicated angle-brackets?!?

If you haven't come across XML-RPC before, the whole concept might seem frightening, like you've gotta write thousands of lines of code for parsing and encoding XML, and negotiate some mind-numbingly complex multi-layered protocol.
This is most certainly not the case. XML-RPC libraries are way simple to use.
XML-RPC client and server libraries are available for all major (and most minor) programming languages, and are structured in a way that hides all the intricate details and presents an extremely simple and quickly learnable API over the top.

### 2.2. Why XML-RPC??

I've chosen XML-RPC as the node interface framework because:

- It's easy and quick to learn, regardless of programming language
- It's supported by free libraries in all major programming languages
- It avoids the maintenance problems of home-brew interfaces (people writing implementations in several languages, some falling into disuse then breaking)
- It reduces the opportunity for writing vulnerable client code (compare to writing raw socket handlers in C, and inadvertently opening oneself up to buffer overruns etc)
- It allows for rapid client development

## 3. Architectural Overview

The Q network is structured as a two-level hierarchy of server nodes and client nodes. Additionally, client applications are run by users, and form the human interface to Q.
Let's quickly overview the difference between these three entities:

- Server nodes:
  - Are exptected to stay up all or most of the time
  - Are suited for running on permanently-up I2P routers
  - Run an XML-RPC server, listening exclusively within the I2P network for commands from other peer server nodes as well as from client nodes
  - Run XML-RPC clients, for sending commands via I2P to other server nodes
  - When joining the network, announce themselves as peers to other server nodes
  - Usually have no direct contact with client applications
  - Receive and execute commands from client nodes, as well as from other peer server nodes.
  - Will never send commands to client nodes.
  - Store content, which is served up by request to client nodes
  - Send catalogues of their stored content on request to client nodes
  - Store lists of their known peer server nodes, and send these lists on request to client nodes
  - Manage load by advising client nodes, and peer server nodes, in command replies, of the next advisable time for contact
  - Should preferably be implemented in platform-independent code
- Client nodes:
  - May run as continuously or as intermittently as desired without causing disruption to the network
  - Run an XML-RPC server, listening exclusively within the user's local TCP/IP network (usually a localhost port), as opposed to server nodes which run their XML-RPC server listening within I2P
  - Run XML-RPC clients, for sending commands via I2P to server nodes
  - Never announce themselves as peers to server nodes
  - Never have contact with other client nodes
  - Are suited for use over permanent or transient I2P routers
  - Periodically contact servers requesting differential updates to content catalogues, as well as peer lists. From this info, they maintain a local mirror of what's available globally
  - When receiving any command reply from a given server, are expected to honour the next advised contact time specified by that server
  - Form the official point of access to the Q network for client applications
  - Should preferably be implemented in platform-independent code
- Client applications:
  - Form the point of human (or third-party program) access to the Q network
  - Offer the user a means of searching for content, inserting content and retrieving content
  - Include GUI apps, CLI apps, web apps, and apps with other user or program interfaces.
  - Usually never run an XML-RPC server at all
  - Run a single XML-RPC client, for sending commands via TCP to a local client node
  - Are implemented and maintained separately to the core Q framework, though at any time might be included in official Q distributions
  - Can be freely implemented in platform-independent or platform-dependent code. For instance, Macintosh-only, or Windows-only implementations are perfectly acceptable (but not quite as welcome as platform-independent implementations)

## 4. Q Command Interface Description

### 4.1. Overview

As mentioned earlier, communication between all Q entities takes place via an XML-RPC mechanism.
This chapter describes the actual primitives which are supported by both server nodes and client nodes.
Although the primitives are the same for both server and client, the way they are actioned internally may vary.

> For example, with the getItem primitive, server nodes will only look in their local content store for the item, returning either that item's data and metadata, or a failure reply. On the other hand, client nodes will try their local content store first, and if the item is not found, will look in their peer catalogues. If the item is found in a peer catalogue, the client node will then on-send getItem calls to all server nodes believed to hold that item, until or unless it retrieves a verifiable copy of that item

### 4.2. XML-RPC Data Types

> > It's possibly a good idea here to get a hold of the XML-RPC library for your favourite programming language, as well as the manual, and look up the description of data types. Also, if you're especially keen, you might like to read up on XML-RPC in general:
> > - XML-RPC Information
> > - XML-RPC Howto

XML-RPC supports a canonical set of data types, which are seamlessly integrated into all its high level language implementations. A quick overview of the XML-RPC data types used in Q appears below.

XML-RPC Data Type | Description
int | Plain 32-bit integer
string | Sequence of ASCII bytes, viewed as java.lang.String objects in java, and str objects (strings) in Python. Note that ASCII control chars, and high-bit-set chars, are highly illegal and will cause failure.
binary data | Raw binary data, viewed as byte [] in java, and xmlrpclib.Binary objects in Python. This is the format used for raw content data.
list | Sequence of objects, viewed as java.util.Vector in java, and list objects in Python.
struct | An unordered set of (key, value) pairs. Represented as java.util.Hashtable objects in java, and dict objects in Python, (associative array in perl, ...)

### 4.3. General Command/Response Format

With Q's XML-RPC usage, all commands are a sequence of zero or more arguments. All responses are a struct with at least the key status, whose value, a string, is one of:

- "ok" - the command was successful; any additional data is included under other keys, depending on the command
- "error" - the command failed, and an additional key error contains a terse description of the error

Note that all commands are also implemented with an alternative entry point, one which takes a single Hashtable (struct/dict/assoc-array) argument. Refer to the javadocs for further info:

#### 4.4. Exceptions - XML-RPC and Otherwise

> In certain cases, XML-RPC calls to Q nodes may return an exception.
> For instance, any attempt to invoke any primitive other than those listed below will most definitely cause an exception, because in the Q XML-RPC implementation, no provision is made for default handlers.
> Apart from this, it's possible that calls to known legal methods may trigger an exception. This is not supposed to happen, and the author will be working over time to intercept all such exceptions and wrap them in appropriate response structures. But in the meantime, client app developers should catch any exceptions resulting from their XML-RPC calls and recover appropriately.

### 4.5. Overview of Q XML-RPC Primitives

The XML-RPC primitives supported by Q server and client nodes include:

- i2p.q.ping - test if a server node is alive
- i2p.q.hello - one new server node introduces itself to another server node
- i2p.q.getItem - retrieve an item of content
- i2p.q.putItem - insert an item of content
- i2p.q.getUpdate - retrieve a differential update of peers list (and optionally, catalog update)
- i2.q.search - search a client node for data items matching certain patterns

### 4.6. i2p.q.ping

> #### Overview

> > The i2p.q.ping primitive is used to test if a given server or client node is presently online. It can be sent by server nodes, client nodes and client apps.

> #### Arguments

> > This primitive accepts no arguments, and will fail if any arguments are given.

> #### Server Behaviour

> > No action on the part of the receiving server is required, apart from sending back:

> Key | Type | Description
> status | string | "ok"
> id | string | The node's nodeId, as a base64 string
> dest | string | Node's destination, represented as base64 string
> uptime | int | The number of seconds that this node has been running for
> load | float | Current load this node is experiencing, as a float between 0.0 (no load) to 1.0 (impossibly flatlined)

> #### Client Behaviour

> > Same as server.

### 4.7. i2p.q.hello

> #### Overview

> > The i2p.q.hello primitive is sent by new server nodes to advise other existing server nodes of their existence. It is only sent by server nodes to other server nodes. It is considered an abuse for a client node to send this command.

> #### Arguments

> - destination (string) - the base64 representation of the calling node's I2P destination (on which the calling node's in-I2P XML-RPC server may be subsequently reached). Same format as the I2P hosts.txt listing.

> #### Server Behaviour

> > If the destination is valid, the receiving server will reply with:

> Key | Type | Description
> status | string | "ok"

> > If the destination is invalid, the receiving server will send back:

> Key | Type | Description
> status | string | "error"
> error | string | "baddest"

> #### Client Behaviour

> > i2p.q.hello calls to clients are illegal. Client nodes receiving such calls will respond with:

> Key | Type | Description
> status | string | "error"
> error | string | "unimplemented"

### 4.8. i2p.q.getItem

> #### Overview

> > The i2p.q.getItem primitive is used to attempt retrieval of an item of content from a client or server node.

> #### Arguments

> - key (string) - the base64 key under which the item in question is stored

> #### Server Behaviour

> > Servers receiving this command will only search their own datastore for the item. They will never attempt to on-request this item from other servers.
> > If the server possesses the requested item in its datastore, it will respond with:

> Key | Type | Description
> status | string | "ok"
> metadata | struct | A nested struct, containing the metadata for the key. (Refer section on metadata).
> data | binary data | The raw data.

> > If the server doesn't possess the data, it will respond with:

> Key | Type | Description
> status | string | "error"
> error | string | "notfound"

> #### Client Behaviour

> > If the client possesses the key in its own local datastore, it will send back the full data immediately:

> Key | Type | Description
> status | string | "ok"
> metadata | struct | A nested struct, containing the metadata for the key. (Refer section on metadata).
> data | binary data | The raw data.

> > If the client doesn't possess the key, it will search its internal catalogues for a server which does have the key.
> > If one or more servers possessing the key are found, the client will on-send an i2p.q.getItem command to each of those servers in turn, until it either successfully retrieves the data, or fails.
> > If the client successfully retrieves the data from one or more of its servers, it will add the data to its internal cache, and reply with the above success response.
> > If the client was unable to source the complete data from any of its servers, it will reply with:

> Key | Type | Description
> status | string | "error"
> error | string | "notfound"

### 4.9. i2p.q.putItem

> #### Overview

> > The i2p.q.putItem primitive is used by client nodes to insert a new item of content onto a server node.
> > It is also used by client apps to insert a new item onto their client node.
> > Also, if a server node is receiving a high traffic of requests for a given item, it may at its discretion send i2p.q.putItem commands to peer servers to mirror the item on those servers, and spread the load.

> #### Arguments

> - data - (binary) - the raw data to insert. Refer earlier - the compatible Java datatype is byte[], and Python datatype is xmlrpclib.Binary.
> - metadata - (struct) - optional - a struct of metadata to insert alongside the data. If this is not given, a minimal metadata set will be automatically created by the recipient. See the section on metadata.

> #### Server Behaviour

> > If the server successfully received and stored the data (and optionally provided metadata), it will reply with:

> Key | Type | Description
> status | string | "ok"
> key | string | The base64 key under which this item has been stored, and which should be used for any subsequent i2p.q.getItem requests for that item within the Q network.

> > However, if the server's datastore is full, the server will not be able to store this item, in which case it will respond with:

> Key | Type | Description
> status | string | "error"
> error | string | "storefull"

> #### Client Behaviour

> > Client nodes receiving this command will attempt to store the item in their own datastore, and respond immediately with one of the above server responses.
> > In addition, client nodes will enqueue a background job to upload this item to one or more selected server nodes.

### 4.10. i2p.q.getUpdate

> #### Overview

> > The i2p.q.getUpdate primitive is used to request a differential peers list update (which optionally can include a catalog update as well).
> > Client apps invoke this primitive on client nodes to get up-to-date listings of items available in the network. Note that client apps will not hand over any peers list.
> > Client nodes periodically schedule a background job to invoke this primitive on their known servers, such that they keep the most recent possible view of available data and other servers.

> #### Arguments

> - since - (int) - unix time in seconds to update from. The recipient will send back a list of all content it has become aware of since this time.
> - includePeers - (int) - set to 1 to include peer list update in the return data, 0 to omit.
> - includeCatalog - (int) - set to 1 to include catalog update in the return data, 0 to omit.

> #### Server Behaviour

> > On receiving this command, a server node will send back lists of metadata records for all new content (and/or all new peers) it has become aware of since the given date. The full response is formatted as follows:

> Key | Type | Description
> status | string | "ok"
> items | list | A list of metadata records for new items. Refer to the section on metadata for more information. If the server has not become aware of any new data since the given date (or if the includeCatalog argument was 0), this list will be empty.
> peers | list | A list of destinations of new peers. If the server has not discovered any new peers since the given date (or if the includePeers argument was 0), this list will be empty.
> timeUpdateEnds | int | unixtime in secs that this update ends. The peer receiving this response should note this time, and quote it as the since argument in the next getUpdate request
> timeNextContact | int | Advised time (unixtime in sec) for sending the next getUpdate command. The sending peer should not issue any getCatalog commands before this time, but is welcome to issue them after this time. The actual time value is guesstimated by the server node, depending on its current load.

> ### 4.11. i2p.q.search

> > #### Overview

> > > The i2p.q.search primitive is invoked by client apps to search a client node for data items matching a set of criteria.
> > > Only client nodes support this primitive. Server nodes will return an empty result set and an error response.

> > #### Arguments

> > - criteria - (hashtable) - a set of metadata criteria to match. Each key in this hashtable is a metadata key (eg title, type etc), and the corresponding value is a regular expression string to match. Regular expression syntax is documented in the java API in the section on class 'Pattern'.
> >   The search criteria work 'AND-style', in that if more than one metadata key match pattern is given, then only items matching all of the given criteria will be returned.
> >   Python example (using XML-RPC proxy - see code samples below):

> >   > `

> >   >     result = mynode.i2p.q.search({"type":"text", "summary":"^War.*"})
> >   >     metaRecs = result['items']

> >   > `

> >   Java Example (using XML-RPC proxy - see code examples below):

> >   > `

> >   >     Hashtable criteria = new Hashtable();
> >   >     criteria.put("type", "text");
> >   >     criteria.put("summary", "^War.*");
> >   >     Vector args = new Vector();
> >   >     args.addElement(criteria);
> >   >     Hashtable result = (Hashtable)mynode.execute("i2p.q.search", args);
> >   >     Vector metaRecs = (Vector)result.get("items");

> >   > `

> >   Note that if the criteria argument is empty (no keys/values), then the client node will send back metadata for every item of content it knows of, which (depending on the size of the Q network), could be quite a resource-hungry operation.

> > #### Server Behaviour

> > > Servers receiving this command will send back an error response:

> > Key | Type | Description
> > status | string | "error"
> > error | string | "unimplemented"

> > #### Client Behaviour

> > > Client nodes receiving this command will send back the following response:

> > Key | Type | Description
> > status | string | "ok"
> > items | vector | A list of metadata records (Hashtables) for items which match the given search criteria, and are retrievable through this client node (ie, the client node either possesses the item, or knows one or more servers which possess the item).

> ## 5. Client Program Examples

> ### 5.1. Overview

> This section provides a couple of simple examples of client app programming.
> At present, only Python and Java examples are given.
> (If you don't know either of these languages, you should be able to get the general drift by studying the examples, sufficient to map the concepts to the XML-RPC API available to your preferred language.)
> The examples below communicate with a client node XML-RPC server (running on the local machine and listening on its default port of 7651), and perform simple operations of data insertion, catalog fetching and data retrieval.

> ### 5.2. Java Example

> To run this example, you'll need:
> - A running I2P installation, with an instance of a Q client node.
> - The I2P standard jarfiles declared in your java CLASSPATH
> - The standard Apache XML-RPC library jarfile in your CLASSPATH (which you will already have on your CLASSPATH, because this is part of installing Q). Recall that you can get a copy of Apache java XML-RPC lib jarfile from http://ws.apache.org/xmlrpc).
> Now for the code (heavily annotated, so you don't necessarily need to know or understand Java), which should be written to a source file called QDemo.java. Note that this client would be a significantly shorter if it instantiated a QClientNode class directly and invoked its methods, but that is not what we're showing here - we're demonstrating the use of the client node's XML-RPC interface.

> > `

> >     // QDemo.java
> >     //
> >     // A simple demo example of a Q client application, which
> >     // communicates with a running Q client node on the local
> >     // machine via its TCP XML-RPC interface
> >     //
> >     // If your client node is not running on localhost, or
> >     // if it's listening on a port other than the default
> >     // 7651, you'll need to change the code below.
> >     //
> >     // Note that this demo is bloated by the fact we're using
> >     // raw XML-RPC.
> >     //
> >     // The following exercises are left to the reader:
> >     //  1. Modify this app so that instead of using the XML-RPC
> >     //     interface, it instantiates a QClientNode, and
> >     //     invokes its methods directly.
> >     //  2. Write a thin wrapper class which instantiates an XML-RPC
> >     //     client, and offers simpler access methods (thus avoiding
> >     //     the need to create and populate Vectors of args before
> >     //     calling, and pick through a reply Hashtable after the call),
> >     //     and create a version of this demo which uses the wrapper.

> >     // pull in some standard java stuff
> >     import java.*;
> >     import java.lang.*;
> >     import java.util.*;
> >     import java.net.*;
> >     import java.io.*;

> >     // pull in some xml-rpc stuff
> >     import org.apache.xmlrpc.*;

> >     // since we're talking to the node via xmlrpc, and talking to
> >     // it in a separate VM, we don't need to import any Q packages

> >     // Define a minimal demo class, which kust defines a
> >     // main method enabling us to run the demo from a shell.
> >     //
> >     // For the purposes of this demo, we're assuming that your Q client node is
> >     // running on your local machine, and that you haven't altered the
> >     // listening port (default 7651) for the client's XML-RPC interface.

> >     public class QDemo {

> >         // just define a main so we can run this from a shell
> >         static public void main(String [] args)
> >             throws MalformedURLException, XmlRpcException, IOException
> >         {
> >             // for getting and analysing replies from node
> >             Hashtable result;
> >             String status;

> >             // Create a new client app object
> >             XmlRpcClient myClient = new XmlRpcClient("http://127.0.0.1:7651");

> >             // -------------------------------------
> >             // First action - execute a 'ping' on this peer
> >             // -------------------------------------

> >             Vector noArgs = new Vector();
> >             result = (Hashtable)myClient.execute("i2p.q.ping", noArgs);
> >             print("ping: result=" + result);

> >             // -------------------------------------
> >             // Second action - insert an item of data
> >             // -------------------------------------

> >             // mark the current time, we'll use this later
> >             Integer then = new Integer((int)(new Date().getTime() / 1000));

> >             // create metadata
> >             // (note from previous chapter that metadata is optional)
> >             Hashtable meta = new Hashtable();
> >             meta.put("type", "text");
> >             meta.put("abstract", "a simple piece of demo data");
> >             meta.put("mimetype", "text/plain");

> >             // create some data
> >             String data = "Hello, world";

> >             // set up the arguments list
> >             Vector insertArgs = new Vector();
> >             insertArgs.addElement(meta);
> >             insertArgs.addElement(data.getBytes()); // must insert data as byte[]

> >             // and do the insert
> >             result = (Hashtable)myClient.execute("i2p.q.putItem", insertArgs);
> >             print("putItem: result=" + result);

> >             // check what happened
> >             status = (String)result.get("status");
> >             String key;
> >             if (status.equals("ok")) {
> >                 // insert succeeded
> >                 key = (String)result.get("key");
> >                 print("Insert successful");
> >             } else {
> >                 // insert failed, bail
> >                 print("Insert failed: error=" + (String)result.get("error"));
> >                 return;
> >             }

> >             // -------------------------------------
> >             // Third action - check for catalog updates
> >             // (which should include what we've just inserted)
> >             // -------------------------------------

> >             // create an args list, with just the date we noted before the insert
> >             Vector updateArgs = new Vector();
> >             updateArgs.addElement(then);
> >             // add the flags
> >             updateArgs.addElement(new Integer(0));   // 'includePeers'
> >             updateArgs.addElement(new Integer(1));   // 'includeCatalog'

> >             // execute the 'getCatalog'
> >             result = (Hashtable)myClient.execute("i2p.q.getUpdate", updateArgs);
> >             print("getUpdate: result="+result);

> >             // pick out the results, and search for what we just inserted
> >             int i;
> >             Vector items = (Vector)result.get("items");
> >             int nitems = items.size();
> >             boolean foundit = false;
> >             for (i = 0; i < nitems; i++) {
> >                 // get the nth item
> >                 Hashtable metaRec = (Hashtable)items.get(i);
> >                 String thisKey = (String)metaRec.get("key");
> >                 if (thisKey.equals(key)) {
> >                     // yay, got it!
> >                     foundit = true;
> >                     break;
> >                 }
> >             }

> >             // did we get it?
> >             if (!foundit) {
> >                 print("wtf? we inserted it but it's not in the catalog!");
> >                 return;
> >             }

> >             // yep, we got it, so try to retrieve it back
> >             Vector getArgs = new Vector();
> >             getArgs.addElement(key);
> >             result = (Hashtable)myClient.execute("i2p.q.getItem", getArgs);
> >             print("getItem: result=" + result);

> >             // did we get it?
> >             status = (String)result.get("status");
> >             if (!status.equals("ok")) {
> >                 print("getItem failed: " + (String)result.get("error"));
> >                 return;
> >             }

> >             // yep, got it
> >             byte [] binData = (byte [])result.get("data");
> >             String strData = new String(binData);
> >             print("getItem: success, data='"+strData+"'");

> >             print("--- END OF Q CLIENT DEMO ---");
> >         }

> >         // a convenient shorthand method for printing stuff to stdout
> >         static void print(String msg) {
> >             System.out.println(msg);
> >         }
> >     }

> > `

> ### 5.3. Python Example

> To run this example, you will need a running I2P installation, including a running instance of a Q client node.
> Note that, in contrast to Java, Python 2.3 and later have all the necessary XML-RPC libraries built in.
> Now for some code (again, heavily annotated). This, together with the previous example, present an interesting comparison between some of Java and Python's ways of doing things.

> > `

> >     #!/usr/bin/env python
> >     """
> >     QDemo.py

> >     A simple demo example of a Q client application, which
> >     communicates with a running Q client node on the local
> >     machine via its TCP XML-RPC interface

> >     If your client node is not running on localhost, or
> >     if it's listening on a port other than the default
> >     7651, you'll need to change the code below.

> >     Note that this demo is bloated by the fact we're using
> >     raw XML-RPC.

> >     The following exercise is left to the reader:
> >      * Write a thin wrapper class which instantiates an XML-RPC
> >        client, and offers simpler access methods (thus avoiding
> >        the need to pick through a reply dict after the call),
> >        and create a version of this demo which uses the wrapper.
> >     """

> >     # a coupla needed imports
> >     from time import time
> >     from xmlrpclib import ServerProxy, Binary

> >     # For the purposes of this demo, we're assuming that your Q client node is
> >     # running on your local machine, and that you haven't altered the
> >     # listening port (default 7651) for the client's XML-RPC interface.

> >     def qdemo():
> >         # Create a new client app object
> >         myClient = ServerProxy("http://127.0.0.1:7651")

> >         # -------------------------------------
> >         # First action - execute a 'ping' on this peer
> >         # -------------------------------------

> >         result = myClient.i2p.q.ping()
> >         print "ping: result=%s" % result

> >         # -------------------------------------
> >         # Second action - insert an item of data
> >         # -------------------------------------

> >         # mark the current time, we'll use this later
> >         then = int(time())

> >         # create metadata
> >         # (note from previous chapter that metadata is optional)
> >         meta = {
> >             "type" : "text",
> >             "abstract" : "a simple piece of demo data",
> >             "mimetype" : "text/plain",
> >             }

> >         # create some data, and binary-wrap it
> >         data = "Hello, world"
> >         binData = Binary(data)

> >         # and do the insert
> >         result = myClient.i2p.q.putItem(meta, binData)
> >         print "putItem: result=%s" % result

> >         # check what happened
> >         if result["status"] == "ok":
> >             # insert succeeded
> >             key = result["key"]
> >             print "Insert successful"
> >         else:
> >             # insert failed, bail
> >             print "Insert failed: error=%s" % result['error']
> >             return;

> >         # -------------------------------------
> >         # Third action - check for catalog updates
> >         # (which should include what we've just inserted)
> >         # -------------------------------------

> >         # execute the 'getUpdate'
> >         result = myClient.i2p.q.getUpdate(then, 0, 1)
> >         print "getUpdate: result=%s" % result

> >         # pick out the results, and search for what we just inserted
> >         foundit = False
> >         for metaRec in result['items']:
> >             if metaRec['key'] == key:
> >                 # yay, got it!
> >                 foundit = True
> >                 break

> >         # did we get it?
> >         if not foundit:
> >             print "wtf? we inserted it but it's not in the catalog!"
> >             return;

> >         # yep, we got it, so try to retrieve it back
> >         print "getCatalog: found the item we just inserted"
> >         result = myClient.i2p.q.getItem(key)
> >         print "getItem: result=%s" % result

> >         # did we get it?
> >         if result["status"] != "ok":
> >             print "getItem failed: %s" + result["error"]
> >             return;

> >         # yep, got it (note that data is an xmlrpclib.Binary object,
> >         # and the raw data we want is in its .data attribute)
> >         print "getItem: success, data='%s'" % result['data'].data

> >         print "--- END OF Q CLIENT DEMO ---"

> >     # run the demo func if this script is executed directly
> >     if __name__ == '__main__':
> >         qdemo()

> > `

> ## 6. Keys and Metadata

> ### 6.1. Overview

> Like Freenet, content is stored in Q as (data, metadata) pairs.
> However, there's a difference. On Freenet, metadata is stored as a string of up to 32k length, and must be parsed (and sometimes executed) by client code. On the other hand, metadata is exposed in Q as an XML-RPC struct (Java Hashtable or Properties object, or Python dict, or Perl associative array etc).
> If a content item gets inserted to the Q network without metadata, a minimal metadata set will be transparently generated, and is guaranteed to contain at least the following elements:

> Key | Type | Description
> size | int | Size of the stored data item, in bytes
> dataHash | string | a base64 representation of the SHA256 hash of the full raw data, using the I2P base64 alphabet

> ### 6.2. Node IDs

> When Q nodes are first created, they generate themselves a random I2P privKey/dest keypair using the in-I2P services.
> The I2P destination gets converted to what we call a Q Node ID, as follows:
> - Start with binary destination (not base64)
> - Determine the SHA256 binary digest of this dest
> - Encode the resulting binary string via I2P's base64 alphabet

> ### 6.3. Keys

> Here, 'key' means the unique short string, by which items of content can be retrieved, and which is returned from an i2p.q.putItem command.
> Like Freenet's CHK@ keytype, Q keys are hashes of the key's content and metadata.
> The recipe for calculating the 'key' of a particular item of metadata+data is:
> 1. If no metadata is submitted with the data, create a minimal metadata as per above
> 2. Serialise out the metadata into a string representation, with the fieldnames in alphanumeric order. The format of such string is one line per metadata field/value pair per line, in the format:

>    > `metadatakeyname=metadatakeyvalue\n`

> 3. Calculate the binary SHA1 digest of this serialised metadata string
> 4. Base64-encode this binary digest via the I2P Base64 alphabet

> ### 6.4. Q Metadata Conventions

> Additional to the core metadata defined above, there is a convention in Q that the following optional extra metadata keys be provided on insert, and recognised and honoured on retrieve.
> It is highly recommended that these keys be included in metadata when content is inserted:

> Key | Type | Description

> title

> string

> A short and descriptive title for the item, preferably formatted as a filename which is legal and convenient on all main operating systems, ie, containing only alphanumerics, '-', '_' and '.'.
> It is highly advisable that an appropriate file extension appear at the end of the title. Refer to the Security Considerations section below.
> It is expected that client applications will use this title field when displaying available content lists to users.

> type

> string

> Generic type of material, using the following superset of the eMule/Donkey classifications:
> - text
> - html
> - image
> - audio
> - video
> - software
> - archive
> - misc

> mimetype

> string

> A recognised mime-type, as per RFC1341, RFC1521, RFC1522, such as audio/mpeg, text/plain etc.
> This will help client app developers devise ways of disposing with data items they request from client nodes.
> For instance, client apps with http front ends may send back this mimetype as the value of the Content-type: header, (and possibly take preventative action with potentially hazardous mimetypes, such as those which some browsers such as IE might trust and execute blindly as binary code).
> Alternatively, gui-based or cli-based client apps may convert this mimetype to an appropriate benign file extension (such as .txt, .ogg, .jpg etc). See Security Considerations below.

> keywords | string | A set of space-separated keywords describing this item, intended for human reading, as well as automatic parsing by client apps.
> abstract | string | A short descriptive summary of the nature of the data, intended for human reading, as well as automatic pattern matching searches by client apps.

> ### 6.5. One Data Item, Many Metadata Sets?

> It is perfectly possible, and legal, for one item of data to be referenced by two completely different items of metadata.
> Since content keys are a hash of metadata, which in turn contains a hash of the data, then two pieces of metadata referencing the same data item, but containing different metadata values, will end up with different keys.
> So as far as key addresses go, there will be a many-to-1 relationship between raw content keys, and the data returned under these keys.

> ## 7. Security Considerations

> All Peer2Peer software (as with all networked software in general) carries with it a set of devastating security risks which should be respected to the utmost.
> This applies in no small part to Q.
> So this brief sermon is addressed to anyone writing any client applications or APIs talking to the Q network.
> Any material which involves the execution of code on a client machine is risky. However, much of the risk can be managed if the code is open source and peer-reviewed.
> Perhaps the biggest issue as far as Q is concerned is this:

> > Client app developers should never, NEVER implicitly trust incoming content, and should always assume that malicious remote users will insert content which attempts to compromise other users' systems.

> If a Q client app wants to offer filetype-specific support, then perhaps a good strategy is for the client app to use a whitelist of known low-risk file extensions, such as .txt, or (possibly) .ogg, .png etc. Recall that in some Windows configurations, even .jpg can carry an arbitrary code execution attack!
> Note that .html (text/html) is especially dangerous, and should be respected accordingly.
> Support for .html could be a real boon. For instance, it could allow I2P users to publish an I2P equivalent of freenet's freesites - static HTML websites which are accessible even when the author goes offline.
> However, if a client app chooses to recognise .html, it should either use a code-screening mechanism like freenet's fproxy and keep it up to date with all the latest advisories, or use a mandatory-proxy mechanism like I2P's eepProxy.
> One possiblility is to serve up such content via a totally in-I2P http interface, such that Joe can view the content via his regular eeproxy-configured browser.
> This is a typical case where security and ease/convenience can end up in direct conflict. Automatic handling of content according to data type is great from a Joe Sixpack Windows User point of view, but it is a snake-pit of risks that can potentially result in any of the following (or worse):
> - Set up Joe's computer as a spambot
> - Get Joe's personal credit card and other info, and use this criminally
> - Download child pornography or terrorist information onto Joe's PC, use an exploit to get Joe's IP address and/or identity details, and report this to authorities, thus framing Joe and sending him off undeservedly to Club Fed or Her Majesty's
> - Mount a DDoS, anonymity or other attack on the I2P network
> - Further spread additional content for achieving more of the above on other unsuspecting users.
> The crux of this lecture is that client app writers have a huge responsibility to ensure their apps are safe against malicious content.
> Perhaps the best and most practical solution is to just store downloaded material into a directory known to and owned by the user, and make it the user's task and responsibility to manually copy materials out of this directory and take responsibility for how s/he uses this content thereafter.

> ## 8. Contacting the Author

> I am aum, and can be reached as aum on in-I2P IRC networks, and also at the in-I2P email address of aum@mail.i2p.

> Introduction | XML-RPC | Architecture | Commands | Example Code | Metadata | Security | Contact us

> Last modified: Sat Apr 2 13:31:08 NZST 2005