# Q User/Programmer Manual

A brief but hopefully easy guide to installing and using the Q distributed file store within the I2P network
(Return to Q Homepage)
Introduction | Checklist | Server?orClient? | Walkthrough | Server Nodes | About QMgr | Contact us

## 1. Introduction

> Q is a distributed Peer2Peer file storage/retrieval network that aims to deliver optimal performance by respecting the properties of the I2P network.
> This manual serves as a 'walkthrough' guide, to take you through the steps from initial download, to everyday usage. It also provides information for the benefit of higher-level client application authors.

## 2. Preliminary Checklist

> OK, we assume here that you've already cracked the tarball, and are looking at the distribution files.
> In order to get Q set up and running, you'll need:
> 1. An I2P router installed, set up and (permanently or transiently) running
> 2. Your system shell set up with at the environment variables:
>    - CLASSPATH - this should include:
>      - The regular I2P jar files and 3rd party support jar files (eg i2p.jar, i2ptunnel.jar, streaming.jar, mstreaming.jar, jbigi.jar)
>      - Apache's XML-RPC support jarfile - included in this Q distro as xmlrpc.jar
>      - Aum's jarfile aum.jar, which includes Q and all needed support code
>    - PATH - your execution search path must include the directory in which your main java VM execution program (java, or on windows systems, java.exe) resides.
>      NOTE - if java[.exe] is not on your PATH, then Q will not run.

## 3. Q Server or Q Client?

> Nearly everyone will want to run a Q Client Node.
> It is only client nodes which provide users with full access to the Q network.
> However, if you have a (near-) permanently running I2P Router, and you're a kind and generous soul, you might also be willing to run a Q Server Node in addition to your Q Client Node.
> If you do choose to run a server node, you'll be expected to keep it running as near as possible to 24/7. While transience of client nodes - frequent entering and leaving the Q network - causes little or no disruption, transience of server nodes can significantly impair Q's usability for everyone, particularly if this transience occurs frequently amongst more than the smallest percentage of the server node pool.
> Until you're feeling well "settled in" with Q, your best approach is to just run a client node for now, and add a server node later when you feel ready.

## 4. Q Walkthrough

### 4.1. Introduction

> This chapter discusses the deployment and usage of a Q Client Node, and will take you through the steps of:
> 1. Double-checking that you've met the installation requirements
> 2. Launching a Q Client Node
> 3. Verifying that your Q Client Node is running
> 4. If your node fails to launch, figuring out why
> 5. Importing one or more noderefs into your node
> 6. Observing that your node is discovering other nodes on the network
> 7. Observing that your node is discovering content on the network
> 8. Searching for items of content that match chosen criteria
> 9. Retrieving stuff from the network
> 10. Inserting stuff to the network
> 11. Shutting down your client node
> Setup and running of Q Server Nodes will be discussed in a later chapter.

### 4.2. Verify Your Q Installation Is Correct

> Ensure that all the needed I2P jarfiles, as well as xmlrpc.jar and Q's very own aum.jar are correctly listed in your CLASSPATH environment varaible, and your main java launcher is correctly listed in your PATH environment variable.
> Typically, you will likely copy the jarfiles aum.jar and xmlrpc.jar into the lib/ subdirectory of your I2P router installation, along with all the other I2P jar files. Wherever you choose to put these files, make sure they're correctly listed in your CLASSPATH.
> Also, you'll want to add execute permission to your qmgr (or qmgr.bat) wrapper script, and copy it into one of the directories listed in your PATH environment variable.

### 4.3. Get Familiar With qmgr

> qmgr (or qmgr.bat) is a convenience wrapper script to save your sore fingers from needless typing. It's just a wrapper which passes arguments to the java command `java net.i2p.aum.q.QMgr`
> You can verify you've set up qmgr correctly with the command:

> > `

> >     qmgr help

> > `

> This displays a brief summary of qmgr commands. On the other hand, the command:

> > `

> >     qmgr help verbose

> > `

> floods your terminal window with a detailed explanation of all the qmgr commands and their arguments.

### 4.4. Running A Q Client Node For The First Time

> Provided you've successfully completed the preliminaries, you can launch your Q Client Node with the command:

> > `

> >     qmgr start

> > `

> All going well, you should have a Q Client Node now running in background.

### 4.5. Verify that your Q Client Node is actually Running

> After typed the qmgr start command, you will see little or no evidence that Q is actually running.
> You can test if the node is actually up by typing the command:

> > `

> >     qmgr status

> > `

> If your Q Client Node is running, this status command should produce something like:

> > `

> >     Pinging node at '/home/myusername/.quartermaster_client'...
> >     Node Ping:
> >       status=ok
> >       numPeers=0
> >       dest=-3LQaE215uIYwl-DsirnzXGQBI31EZQj9u~xx45E823WqjN5i2Umi37GPFTWc8KyislDjF37J7jy5newLUp-qrDpY7BZum3bRyTXo3Udl8a3sUjuu4qR5oBEWFfoghQiqDGYDQyJV9Rtz7DEGaKHGlhtoGsAYRXGXEa8a43T2llqZx2fqaXs~836g8t6sLZjryA5A9fpq98nE5lT0hcTalPieFpluJVairZREXpUiAUmGHG7wAIjF6iszXLEHSZ8Qc622Xgwy0d1yrPojL2yhZ64o05aueYcr~xNCiFxYoHyEJO3XYmkx~q-W-mzS3nn6pRevRda74MnX1~3fFDZ0u~OG6cLZoFkWgnxrwrWGFUUVMR87Yz251xMCKJAX6zErcoGjGFpqGZsWxl4~yq7yfkjPnq3GuTxp2cB75bRAOZRIAieqBOVJDEodFYW5amCinu4AxYE7G1ezz4ghqHFe~0yaAdO74Q1XoUny138YT6P33oNOOlISO1cAAAA
> >       uptime=4952
> >       load=0.0
> >       id=6LVZ9-~GgJJ52WUF1fLHt3UnH50TnXSoPQXy7WZ4GA=
> >       numLocalItems=47
> >       numRemoteItems=2173

> > `

> If you see something like this, then smile, because Q is now up on your system.
> If the node launch failed, you might see something like:

> > `

> >     Pinging node at '/home/myusername/.quartermaster_client'...
> >     java.io.IOException: Connection refused
> >             at org.apache.xmlrpc.XmlRpcClient$Worker.execute(Unknown Source)
> >             at org.apache.xmlrpc.XmlRpcClient.execute(Unknown Source)
> >             at net.i2p.aum.q.QMgr.doStatus(QMgr.java:310)
> >             at net.i2p.aum.q.QMgr.execute(QMgr.java:813)
> >             at net.i2p.aum.q.QMgr.main(QMgr.java:869)
> >     Failed to ping node

> > `

> This indicates that your Q client node has either crashed, or failed to launch in the first place.
> If you're having trouble like this, you might like to try running your Q client node in foreground, instead of spawning it off into background.
> The command to run a Q client node in foreground is:

> > `

> >     qmgr foreground

> > `

> You should see some meaningless startup messages, and no return to your shell prompt.

### 4.6. Diversion - Q Storage Directories

> By default, when you run a Q Client Node, it creates a datastore directory tree at ~/.quartermaster_client. (Windows users note - you'll find this directory wherever your user home directory is - this depends on what version of Windows you have installed).
> Within this directory tree, you should see a file called node.log, which will contain various debug log messages, and can help you to rectify any problems with your Q installation. If you hit a wall and can't rectify the problems yourself, you should send this file to the Q author (aum).
> It's possible to run your Q node from another directory, by passing that directory as a -dir <path> argument to the qmgr start, foreground and stop commands. See qmgr help verbose for more information.

### 4.7. Importing a Noderef

> Note from the prior qmgr status command the line:

> > `

> >     numPeers=0

> > `

> This means that your Q client node is running standalone, and doesn't have any contact with any Q network. As such, your node is effectively useless. We need to hook up your node with other nodes in the Q network.
> Q doesn't ship with any means for new client nodes to automatically connect to any Q server nodes. This is deliberate.
> In all likelihood, there will be one 'main' Q network running within I2P, largely based around the author's own Q server node, and most people will likely want to use this Q network. But the author doesn't want to stop other people running their own private Q networks, for whatever purpose has meaning for them.

> > This is especially relevant for Q as opposed to Freenet. With Freenet, there's no way for a user to know of the existence of any item of content without first being given its 'key'. However, since Q works with published catalogs, any user can know everything that's available on a Q network, which might not be desirable to those wishing to share content in a private situation.
> > The Q author anticipates, and warmly supports, people running their own private Q networks within I2P, in addition to accessing the mainstream 'official' Q network.
> > The way Q is designed and implemented, there is no way for anyone, including Q's author, to know of the existence of anyone else's private Q network. It is beyond the author's control, (and thus arguably the author's legal responsibility), what private Q networks people set up, and what kind of content is trafficked on these networks. This claim of plausible deniability on the part of Q's author parallels that of a hardware retailer denying responsibility for what people do with tools that they purchase.

> Ok, getting back on topic - your brand new virgin Q client node is useless and lonely, and desperately needs some Q server nodes to talk to. So let's hook up your node to the mainstream Q network.
> You'll need to get one or more 'noderefs' for Q server nodes.
> There's nothing fancy about a Q noderef. It's just a regular I2P 'destination', with which your Q Client Node can connect with a Q Server Node.
> A 'semi-official' list of noderefs for the mainstream Q network can be downloaded from the url: http://aum.i2p/q/qnoderefs.txt.
> Download this file, save it as (say) qnoderefs.txt. (Alternatively, if you're wanting to subscribe into a private Q network, then get a noderef for at least one of that network's server nodes from someone on that network who trusts you).
> Import these noderefs into your Q client node via the command:

> > `

> >     qmgr addref qnoderefs.txt

> > `

> If all goes well, you should see no output from this command, or (possibly) a brief line or two suggesting success.
> Your client node is now subscribed into the Q network of your choice. Verify this with the command:

> > `

> >     qmgr status

> > `

> In the output from that command, you should see the numPeers= line showing at least 1 peer.
> If there is more than one Q Server Node on the Q network you've just subscribed to, then your local node should sooner or later discover all these server nodes, and the numPeers value should increase over time.

> > While Q is in its early development and testing stages, the author may abdicate the mainstream Q network, and publish nodrefs for a whole new mainstream Q network. This will especially happen if the author makes any substantial changes to the inter-node protocol, and/or releases incompatible new versions of Q client/server nodes. Remember that http://aum.i2p/q/qnoderefs.txt will serve as the authoritative source for noderefs for the mainstream Q network within the mainstream I2P network.

> When your client node gets its noderefs to a Q network, it will periodically, from then on, retrieve differential peer list and catalog updates from servers it knows about.
> Even if you only feed your client just one ref for a single server node, it will in time discover all other operating server nodes on that Q network, and will build up a full local catalog of everything that's available on that Q network.
> Provided that your client is running ok, and has been fed with at least one ref for a live Q network that contains content, then over time, successive:

> > `

> >     qmgr status

> > `

> commands should report increasing values in the fields:
> - numPeers - number of peers this client node knows about
> - numLocalItems - number of locally stored content items, ie items which you have either inserted to, or retrieved from, your client node
> - numRemoteItems - number of unique data items which are available on remote server nodes in the Q network, and which can be retrieved through your local client node.

> > #### 4.7.1. One Big Warning

> > If you are participating in more than one distinct Q network, then do not insert noderefs for different networks into the same running instance of a local Q client, unless you don't plan on inserting content via that client.
> > For instance, let's say you are participating in two different Q networks:
> > - The 'mainstream' Q netowrk
> > - A secret Q network - "My friends' teen angst diaries"
> > If you get a noderef for both these networks, and insert both of these into the same running Q client node, then this local client node will be transparently connected to both networks.
> > If you only ever plan on retrieving content, and never inserting content, this won't be a problem, except that you won't be able to tell which content resides on the mainstream Q network, and which resides in the secret Q network.
> > The big problem arises from inserting content. Whenever you insert data through this 'contaminated' Q client node, this node picks 3 different servers to which upload a copy of this data. You won't have any control over whether the data gets inserted to the mainstream Q network, or your secret Q network. You might insert something sensitive, intending it to go only into the secret Q network, where in fact it also ends up in the mainstream network, with consequences you might not want.

### 4.8. Content Data and Metadata

> Whenever content gets stored on Q, it is actually stored as two separate items:
> - The raw data - whether a text file, or the raw bytes of image files, audio files etc
> - The metadata, which contains human-readable and machine-readable descriptions of the data
> Metadata consists of a set of category=value pairs.
> Confused yet? Don't worry, I'm confused as well. Let's illustrate this with an example of metadata for an MP3 audio recording:
> - title=Fight_Last_Thursday.mp3
> - type=audio
> - mimetype=audio/mpeg
> - abstract=upcoming single recorded in our garage last April
> - keywords=grunge,country,indie
> - artist=Ring of Fire
> - size=4379443
> - contact=ring-of-fire@mail.i2p
> - key=blah37blah24-yada23hfhyada
> All metadata categories are optional. In fact, you can insert content with no metadata at all.
> If you fail to provide metadata when inserting an item, a blank set of metadata will be created with at least the following categories:
> - key - the derived key, under which the item will later be retrievable by yourself and others
> - title - if not provided at insert time, this will be set to the key
> - size - size of the item's raw data, in bytes
> Within Q, there is a convention to supply a minimal amount of metadata. While this is not expected or enforced, including all these categories is most strongly recommended. These core categories are:
> - title - a meaningful title for the data item, consisting only of characters which are legal in filenames on all platforms, and which ends with a file extension.
> - type - one of a superset of eMule classifiers, such as:
>   - text - plain text
>   - html - HTML content
>   - image - content is in an image format, such as .png, .jpg, .gif etc
>   - audio - content is an audio sample, such as .ogg, .mp3, .wav etc
>   - video - due to the sheer size of video files, and Q's present design, it's unlikely people will be inserting video content anytime soon (unless it's very short)
>   - archive - packed file collections, such as .tar.gz, .zip, .rar etc
>   - misc - content does not fit into any of the above categories
> - mimetype - not as important as the type category, but providing this category in your metadata is still strongly encouraged. Value for this category should be one of the standard mimetypes, eg text/html, audio/ogg etc.
> - abstract - a short description (<255 characters), intended for human reading
> - keywords - a comma-separated list of keywords, intended for machine-readability, should be all lowercase, no spaces
> Note that you can supply extra metadata categories in addition to the above, and that people searching for content can search on these extra categories if they know about them.

### 4.9. Searching For Content

> As mentioned earlier - in constrast with Freenet, local Q nodes build up a complete catalog of all available content on whatever Q network they are connected to.
> This is a design decision, based on the choice to eliminate query traffic.
> The author hopes that this will result in a distributed storage network with a high retrievability guarantee, in contrast with freenet which offers no such guarantee.
> With Freenet, you only ever know of the existence of something if someone tells you about it.
> But with Q, your local client node builds up a global catalog of everything that's available within the whole network.
> The QMgr client has a command for searching your Q client node:

> > `

> >     qmgr search -m category1=pattern1 category2=pattern2 ...

> > `

> For example:

> > `

> >     qmgr search -m type=audio artist=Mozart keywords=symphony

> > `

> or:

> > `

> >     qmgr search -m type=text title="bible|biblical|(Nag Hammadi)" keywords="apocrypha|Magdalene"

> > `

> As implied in the latter example, search patterns are regular expressions. This example will locate all text items, whose title metadata category contains one of bible, biblical or Nag Hammadi, and whose keywords category contains either or both the words apocrypha or Magdalene.
> Please use the search function carefully, otherwise (if and when Q usage grows) you could be inundated with thousands or even millions of entries.
> If a search turns up nothing, qmgr will simply exit. But if it turns up one or more items, it will the items out one at a time, with the key first, then each metadata entry on an indented line following.

### 4.10. Retrieving Content

> Now, we're actually going to retrieve something.
> Presumably, after following the previous section, you will have seen one or more search results come up, with the 'keys' under which the items can be accessed.
> Now, choose one of the keys, preferably for a short text item. Try either of the following commands:

> > `

> >     qmgr get <keystring> something.txt

> > `

> or:

> > `

> >     qmgr get <keystring> > something.txt

> > `

> (both have the same effect - the first one explicitly writes to the named file, the second one dumps the raw data to stdout, which we shell-redirect into the file.
> Note - redirection of fetched data to a file via shell is not working at present. Use only the first form till we fix the bug.

### 4.11. Inserting Content

> Our last example in this walkthrough relates to inserting content.
> Firstly, create a small text file with 2-3 lines of text, and save it as (say) myqinsert.txt.
> Now, think of some metadata to insert along with the file. Or, you can just use the set:

> > `

> >     type=text
> >     keywords=test
> >     abstract=My simple test of inserting into Q
> >     title=myqinsert.txt

> > `

> Now, let's insert the file. Ensure your Q client node is running, then type:

> > `

> >     qmgr put myqinsert.txt -m type=text keywords=test title="myqinsert.txt" \
> >      abstract="My simple test of inserting into Q"

> > `

> If all went well, this command should produce half a line of gibberish, followed immediately by your shell prompt, eg:

> > `

> >     aRoFC~9MU~pM2C-uCTDBp5B7j79spFD8gUeu~BNkUf0=$

> > `

> The '$' at the end is your shell prompt, and all the characters before it are the 'key' which was derived from the content you just inserted.
> To avoid the hassle of copying/pasting the key, you could just add output redirection to the above command, eg:

> > `

> >     qmgr put myqinsert.txt -m type=text keywords=test title="myqinsert.txt" \
> >      abstract="My simple test of inserting into Q" \
> >      > myqinsert.key

> > `

> This will cause the generated key to be written safe and sound into the file myqinsert.key.
> You can verify that this insert worked by a 'get' command, as in:

> > `

> >     qmgr get `cat myqinsert.key` somefilename.ext

> > `

> (Note that this won't work on windows because the DOS shell is irredeemably brain-damaged. If you're using Windows, you will have to cut/paste the key.

### 4.12. Shutting Down your Node

> If you've worked through to here, then congratulations! You've got your Q Client Node set up and working, and ready to meet all your distributed file storage and retrieval needs.
> You can leave your client node running 24/7 if you want. In fact, we recommend you keep your client node running as much of the time as possible, so that you get prompt catalog updates, and can more quickly stay in touch with new content.
> However, if you need to shut down your node, the command for doing this is:

> > `

> >     qmgr stop

> > `

> This command will take a while to complete (since the node has to wait for the I2P java shutdown hooks to complete before it can rest in peace). But once your node is shut down, you can start it up again at any time and pick up where you left off.

## 5. Running a Q Server Node

### 5.1. Introduction

> This section describes the requirements for, and procedures involved with, running a Q Server Node.
> We'll use a similar 'walkthrough' style to that which we used in the previous section on client nodes.

### 5.2. Requirements and Choices

> Running a Q server is a generous thing to do, and helps substantially with making Q work at its best for everyone. However, please do make sure you can meet some basic requirements:
> - You are running a permanent (24/7) I2P Router, on a box with at least (say) 98% uptime.
> - You have a little bandwidth to spare, and don't mind the extra memory, disk and CPU-usage footprint of running a fulltime Q server node
> - You have already been able to successfully run a Q client node.
> Also, please decide whether you want your server node to contribute to the mainstream Q network, or whether you want to create your own private Q network, or join someone else's private network. Your contribution will be most appreciated, though, if you can run a server within the mainstream Q network.

### 5.3. Starting Your Server Node

> Starting up a Q Server node is very similar to starting up a Q client node, except that with the qmgr command line, you must put the keyword arg server before the command word. So the command is:

> > `

> >     qmgr server start

> > `

> Similar to Q client nodes, you can check the status of a running Q server node with the command:

> > `

> >     qmgr server status

> > `

> (Note that this command will take longer to complete than with client nodes, because the communication passes through a multi-hop I2P tunnel, rather than just through localhost TCP).
> If the status command succeeds, then you'll know your new Q Server Node is happily running in background.

### 5.4. Joining A Q Network

> When a Q Server node starts up for the first time, it is in a private network all by itself.
> If you want to link your server into an existing Q network, you'll have to add a noderef for at least one other server on that network. The command to do this is similar to that for subscribing a client node to a network:

> > `

> >     qmgr server addref <noderef-file>

> > `

> where <noderef-file> is a file into which you've saved the noderef for the network you want to join.

> > Recall from the section on client nodes that the authoritative noderefs for the mainstream Q network can be downloaded from http://aum.i2p/q/qnoderefs.txt.

> After you've added the noderef, subsequent qmgr server status commands should show numPeers having a value of at least 1 (and growing, as more server nodes come online in the mainstream Q network.)

### 5.5. Private Networks - Exporting Your Server's Noderef

> If you're planning to start your own private Q network, and want to include other server operators in this network, then you'll have to export your server's noderef and make it available to the others you want to invite into your network.
> The command to export your Q Server noderef is:

> > `

> >     qmgr server getref <noderef-file>

> > `

> This will extract the I2P Destination of your running server node, and write it into <noderef-file>. You can then privately share this file with others who you want to invite into your private network. Each recipient of this file will do a qmgr server addref <noderef-file> command to import your ref into their servers.
> Don't forget that if you're running, or participating in, a private Q network, then you'll need to run a separate client for accessing this network, separate from any mainstream Q network client you may already be running.
> To start this extra client, you'll have to choose a directory where you want this client to reside, a port number you want your client to listen on locally for user commands, and run the command:

> > `

> >     qmgr -dir /path/to/my/new/client -port <portnum> start

> > `

> You need the -port <portnum> command, because otherwise it'll fail to launch (if you already have a client node running off the mainstream Q network).
> This will create, and launch, a new instance of a Q client, accessing your private Q network. Don't forget to import your server's noderef into this client. Also, note that you'll have to use this same -port <portnum> argument when doing any operation on this client instance, such as get, put, status, search.

## 6. About the qmgr Utility

qmgr (or, to people fluent in Java, net.i2p.aum.q.QMgr), is just one simple Q client application, that happens to be bundled in with the Q distro.
It is by no means the only, or even main facility for accessing the Q network. We anticipate that folks will write all manner of client apps, including fancy GUI apps.
Anyway, qmgr does give you a rudimentary yet workable client for basic access to the Q network. Until fancy apps get written, qmgr will have to do.
Don't forget that qmgr has very detailed inbuilt help. Run:

> `

>     qmgr help

> `

for a quick help summary, or:

> `

>     qmgr help verbose

> `

for the 'War and Peace' treatise.

> One crucial concept to remember with qmgr is that client and server node instances are uniquely identified by the directories at which they reside. If you are running multiple server and/or client instances, you can specify an instance with the -dir <dirpath> option - see the help for details.

One last note - we strongly discourage any writing of client apps that spawn a qmgr process, pass it arguments and parse its results. This is most definitely a path to pain, since qmgr's shell interface is subject to radical change at any time without notice.
qmgr is for human usage, or at most, inclusion in init/at/cron scripts. Please respect this.
If you want to write higher-level clients, your best course of action is to use the official client api library, which we anticipate will have versions available in Java, Python, Perl and C++. If you want to write in another language, such as OCaml, Scheme etc, then the existing api lib implementations should serve as an excellent reference to support you in writing a native port for your own language.

## 8. Contacting the Author

I am aum, and can be reached as aum on in-I2P IRC networks, and also at the in-I2P email address of aum@mail.i2p.

Return to Q Homepage
Introduction | Checklist | Server?orClient? | Walkthrough | Server Nodes | About QMgr | Contact us

Last modified: Sun Apr 3 20:06:53 NZST 2005