# TreeCache: a Tree Structured Replicated Transactional Cache

Bela
Ban
bela@jboss.com
Ben
Wang
ben.wang@jboss.com
August, 2004

## Introduction

This and a companion documents describe the TreeCache, a tree-structured replicated transactional cache, and the TreeCacheAop, an "object-oriented" cache that is AOP-enabled subclass of TreeCache, allowing for Plain Old Java Objects (POJOs) to be inserted and replicated transactionally between nodes in a cluster. The TreeCache is configurable, i.e. aspects of the system such as replication (async, sync, or none), transaction isolation levels, eviction policy, and transactional manager are all configurable.

TreeCache can also be used independently, but TreeCacheAop requires both TreeCache and the JBoss4.0 AOP standalone subsystem.

## Overview

The structure of the cache is A tree with nodes. Each node has a name and zero or more children. A node can only have 1 parent; there is currently no support for graphs. A node can be reached by navigating from the root recursively through children, until the node is found. It can also be accessed by giving a fully qualified name (FQN), which consists of the concatenation of all node names from the root to this particular node.

A TreeCache can have multiple roots, allowing for a number of different trees to be present in the cache. Note that a one level tree is essentially a hashtable. Each node in the tree has a hashmap of keys and values. For a replicated cache, all keys and values have to be serializable. Serializability is not a requirement for TreeCacheAop, where reflection and AOP is used to replicate any type.

A TreeCache can be either local or replicated. Local trees exist only inside the VM in which they are created, whereas replicated trees propagate any changes to all other replicated trees in the same cluster.

The first version of a cache for JBoss was a hashmap. However, the decision was taken to go with a tree structured cache because (a) it is more flexible and efficient and (b) a tree can always be reduced to a hashmap, thereby offering both possibilities. The efficiency argument was driven by concerns over replication overhead, and was that a value itself can be a rather sophisticated object, with aggregation pointing to other objects, or an object containing many fields. A small change in the object would therefore trigger the entire object (possibly the transitive closure over the object graph) to be serialized and propagated to the other nodes in the cluster. With a tree, only the modified nodes in the tree need to be serialized. This is not necessarily a concern for TreeCache, but is a vital requirement for TreeCacheAop (as we will see in the TreeCacheAop documentation).

When a change is made to the TreeCache, and that change is done in the context of a transaction, then we wait with replication until the TX commits successfully. All modifications are kept in a list associated with the transaction for the caller. When the TX commits, we replicate the changes. Otherwise, (on a rollback) we simply undo the changes and release the locks, resulting in no replication traffic. For example, if a caller makes 100 modifications and then rolls back the TX, we will not replicate anything, resulting in no network traffic.

If a caller has no TX associated with it (and isolation level is not NONE), we will replicate right after each modification, e.g. in the above case we would send 100 messages, plus an additional message for the rollback. In this sense, no transaction can be think of autocommit is on in JDBC terminology where each operation is committed automatically.

There is an API for plugging in different transaction managers: all it requires is to get the TX associated with the caller’s thread. There is a dummy TransactionManager implementation, and one for JBoss.

Finally, we use pessimistic locking of the cache by default (optimistic locking is on the todo list). As mentioned previously, we can configure the local locking policy corresponding to the JDBC style transaction isolation level, i.e., SERIALIZABLE, REPEATABLE, READ_COMMITTED, READ_UNCOMMITTED and NONE. More on the transaction isolation level will be discussed later. Note that the cluster-wide isolation level is by default read-uncommitted because we don’t acquire a cluster-wide lock on touching an object for which we don’t yet have a lock (this would result in too much overhead for messaging) We will offer a cluster-wide serializable policy once we implement optimistic locking, which will be more efficient than implementing a serializable policy over a pessimistic locking scheme. .

### Schematic TreeCache architecture

## Architecture

The architecture is shown above. The example shows 2 Java VMs, each has created an instance of TreeCache. These VMs can be located on the same machine, or on 2 different machines. The setup of the underlying group communication subsystem is done using JGroups (http://www.jgroups.org).

Any modification (see API below) in one cache will be replicated to the other cache Note that you can have more than 2 caches in a cluster. and vice versa. Depending on the TX settings, this will be done either after each modification or at the end of a TX (at commit time). When a new cache is created, it can optionally acquire the contents from one of the existing caches.

## API

Here's some sample code before we dive into the API itself:

    TreeCache tree = new TreeCache();
    tree.setClusterName("demo-cluster");
    tree.setClusterProperties("default.xml");
    tree.setCacheMode(TreeCache.REPL_SYNC);
    tree.createService(); // not necessary, but is same as MBean lifecycle
    tree.startService(); // kick start tree cache
    tree.put("/a/b/c", "name", "Ben");
    tree.put("/a/b/c/d", "uid", new Integer(322649));
    Integer tmp=(Integer)tree.get("/a/b/c/d", "uid");
    tree.remove("/a/b");
    tree.stopService();
    tree.destroyService(); // not necessary, but is same as MBean lifecycle

The sample code first creates a TreeCache instance and then configures it. There is another constructor which accepts a number of configuration options. However, the TreeCache can be configured entirely from an XML file (shown later), so we don't recommend manual configuration as shown in the sample.

The cluster name, properties of the underlying JGroups stack, and cache mode (synchronous replication) are configured first (a list of configuration options is shown later). Then we start the TreeCache. If replication is enabled, this will make the TreeCache join the cluster, and (optionally) acquire initial state from an existing node.

Then we add 2 items into the cache: the first element creates a node "a" with a child node "b" that has a child node "c". (TreeCache by default creates intermediary nodes that don't exist). The key "name" is then inserted into the "/a/b/c" node, with a value of "Ben".

The other element will create just the subnode "d" of "c" because "/a/b/c" already exists. It binds the integer 322649 under key "uid".

The resulting tree looks like this:

### Sample Tree Nodes

The TreeCache has 4 nodes "a", "b", "c" and "d". Nodes "/a/b/c" has values "name" associated with "Ben" in its hashmap, and node "/a/b/c/d" has values "uid" and 322649.

Each node can be retrieved by its absolute name (e.g. "/a/b/c") or by navigating from parent to children (e.g. navigate from "a" to "b", then from "b" to "c").

The next method in the example gets the value associated with key="uid" in node "/a/b/c/d", which is the integer 322649.

The remove() method then removes node "/a/b" and all subnodes recursively from the cache. In this case, nodes "/a/b/c/d", "/a/b/c" and "/a/b" will be removed, leaving only "/a".

Finally, the TreeCache is stopped. This will cause it to leave the cluster, and every node in the cluster will be notified. Note that TreeCache can be stopped and started again. When it is stopped, all contents will be deleted. And when it is restarted, if it joins a cache group, the state will be replicated initially. So potentially you can recreate the contents.

In the sample, replication was enabled, which caused the 2 put() and the 1 remove() methods to replicated their changes to all nodes in the cluster. The get() method was executed on the local cache only.

Keys into the cache can be either strings separated by slashes ('/'), e.g. "/a/b/c", or they can be fully qualified names Fqns . An Fqn is essentially a list of Objects that need to implement hashCode() and equals(). All strings are actually transformed into Fqns internally. Fqns are more efficient than strings, for example:

    String n1 = "/300/322649";
    Fqn n2 = new Fqn(new Object{new Integer(300), new Integer(322649)});

In this example, we want to access a node that has information for employee with id=322649 in department with id=300. The string version needs 2 hashmap lookups on Strings, whereas the Fqn version needs to hashmap lookups on Integer. In a large hashtable, the hashCode() method for String may have collisions, leading to actual string comparisons. Also, clients of the cache may already have identifiers for their objects in Object form, and don't want to transform between Object and Strings, preventing unnecessary copying.

Note that the modification methods are put() and remove(). The only get method is get().

There are 2 put() methods Plus their equivalent helper methods taking a String as node name. : `put(Fqn node, Object key, Object key)` and `put(Fqn node, Hashmap values)`. The former takes the node name, creates it if it doesn't yet exist, and put the key and value into the node's hashmap, returning the previous value. The latter takes a hashmap of keys and values and adds them to the node's hashmap, overwriting existing keys and values. Content that is not in the new hashmap remains in the node's hashmap.

There are 3 remove() methods: `remove(Fqn node, Object key)`, `remove(Fqn node)` and `removeData(Fqn node)`. The first removes the given key from the node. The second removes the entire node and all subnodes, and the third removes all elements from the given node's hashmap.

The get methods are: `get(Fqn node)` and `get(Fqn node, Object key)`. The former returns a Node This is mainly used internally, and we may decide to remove public access to the Node in a future release. object, allowing for direct navigation, the latter returns the value for the given key for a node.

Also, the TreeCache has a number of getters and setters. Since the API may change at any time, we recommend the Javadoc for up-to-date information.

## Replication

The TreeCache can be configured to be either local or replicated .

Local caches don't join a cluster and don't replicate changes to other nodes in a cluster. Therefore their elements don't need to be serializable However, we recommend making them serializable, enabling a user to change the cache mode at any time. . On the other hand, replicated caches replicate all changes to the other TreeCaches (node) in the cluster. Replication can either happen after each modification (no transactions), or at the end of a transaction (commit time).

Replication can be synchronous or asynchronous . Use of either one of the options is application dependent. Synchronous replication blocks the caller (e.g. on a put()) until the modifications have been replicated successfully to all nodes in a cluster. Asynchronous replication performs replication in the background (the put() returns immediately). TreeCache also offers a replication queue, where modifications are replicated periodically (i.e. interval-based), or when the queue size exceeds a number of elements, or a combination thereof).

Asynchronous replication is faster (no caller blocking), because synchronous replication requires acknowledgments from all nodes in a cluster that they received and applied the modification successfully (round-trip time). However, when a synchronous replication returns successfully, the caller knows for sure that all modifications have been applied at all nodes, whereas this may or may not be the case with asynchronous replication. With asynchronous replication, errors are simply written to a log.

## Transaction Isolation Level

TreeCache currently uses pessimistic locking to prevent concurrent access to the same data. Locking is not exposed directly to user. Instead, a transaction isolation level which provides different locking behavior is configurable.

Locking internally is done on a node-level, so for example when we want to access "/a/b/c", a lock will be acquired for nodes "a", "b" and "c". When the same transaction wants to access "/a/b/c/d", since we already hold locks for "a", "b" and "c", we only need to acquire a lock for "d".

Lock owners are either transactions (call comes in on a transaction) or threads (no transaction associated with incoming transaction). Regardless, a local transaction or a thread is internally transformed into an instance of GlobalTransaction, which is used as a globally unique ID for modifications across a cluster. E.g. when we run a two-phase commit protocol (see below) across the cluster, the GlobalTransaction uniquely identifies the unit of work across a cluster.

TreeCache supports transaction isolation level in the manner of the JDBC model. A user can configure per instance-wide isolation level (as those defined in java.sql.Connection), i.e., NONE, READ_UNCOMMITTED, READ_COMMITTED, REPEATABLE_READ, and SERIALIZABLE. The mapping is summarized here.

1. NONE. No transaction support is needed. There is no locking at this level, e.g., users will have to manage the data integrity. Implementations use no locks.

2. READ_UNCOMMITTED. Data can be read anytime while write operations are exclusive. Note that this level doesn't prevent the so-called "dirty read" where data modified in Tx1 can be read in Tx2 before Tx1 commits. In other words, if you have the following sequence,

       Tx1   Tx2
        W
              R

   using this isolation level will not Tx2 read operation. Implementations typically use an exclusive lock for writes while reads don't need to acquire a lock.

3. READ_COMMITTED. Data can be read any time as long as there is no write. This level prevents the dirty read. But it doesn’t prevent the so-called ‘non-repeatable read’ where one thread reads the data twice can produce different results. For example, if you have the following sequence,

       Tx1   Tx2
        R
              W
        R

   where the second read in Tx1 thread will produce different result.

   Implementations usually use a read-write lock; reads succeed acquiring the lock when there are only reads, writes have to wait until there are no more readers holding the lock, and readers are blocked acquiring the lock until there are no more writers holding the lock. Reads typically release the read-lock when done, so that a subsequent read to the same data has to re-acquire a read-lock; this leads to nonrepeatable reads, where 2 reads of the same data might return different values. Note that, the write only applies regardless of transaction state (whether it has been committed or not).

4. REPEATABLE_READ. Data can be read while there is no write and vice versa. This level prevents "non-repeatable read" but it does not prevent the so-called "phantom read" where new data can be inserted into the tree from the other transaction. Implementations typically use a read-write lock.

5. SERIALIZABLE. Data access is synchronized with exclusive locks. Only 1 writer or reader can have the lock at any given time. Locks are released at the end of the transaction.

Locks can be read or write locks. Write locks serialize read and write access, whereas read-only locks only serialize read access. When a write lock is held, no other write or read locks can be acquired. When a read lock is held, others can acquire read locks. However, to acquire write locks, one has to wait until all read locks have been released. When scheduled concurrently, write locks always have precedence over read locks. Note that (if enabled) read locks can be upgraded to write locks.

Using read-write locks helps in the following scenario: consider a tree with entries "/a/b/n1" and "/a/b/n2". With write-locks, when Tx1 accesses "/a/b/n1", Tx2 cannot access "/a/b/n2" until Tx1 has completed and released its locks. However, with read-write locks this is possible, because Tx1 acquires read-locks for "/a/b" and a read-write lock for "/a/b/n1". Tx2 is then able to acquire read-locks for "/a/b" as well, plus a read-write lock for "/a/b/n2". This allows for more concurrency in accessing the cache.

## Transactional Support

A TreeCache can be configured to use transactions to bundle units of work, which can then be replicated as one unit. Alternatively, if transaction support is disabled, it is equivalent to setting AutoCommit to on where modifications are potentially Depending on whether interval-based replication is used replicated after every change (if replication is enabled).

What TreeCache needs to do on every incoming call (e.g. put()) is (a) get the transaction associated with the thread and (b) register (if not already done) with the transaction manager to be notified when a transaction commits or is rolled back. In order to do this, the cache has to be configured with an instance of TransactionManagerLookup which returns a javax.transaction.TransactionManager.

There are currently 2 implementations of TransactionManagerLookup available: DummyTransactionManagerLookup and JBossTransactionManagerLookup. The former is a dummy implementation of a TransactionManager which can be used for standalone TreeCache applications (running outside an appserver). This is just for demo purposes (e.g. the standalone demo runs with this one). The latter is to be used when TreeCache is used inside JBoss, and its implementation looks like this:

    public class JBossTransactionManagerLookup implements TransactionManagerLookup {

        public JBossTransactionManagerLookup() {;}

        public TransactionManager getTransactionManager() throws Exception {
           Object tmp=new InitialContext().lookup("java:/TransactionManager");
           return (TransactionManager)tmp;
        }
    }

The implementation looks up the JBoss TransactionManager from the JNDI and returns it.

When a call comes in, the TreeCache gets the current transaction and records the modification under the transaction as key. (If there is no transaction, the modification is applied immediately and possibly replicated). So over the lifetime of the transaction all modifications will be recorded and associated with the transaction. Also, the TreeCache registers with the transaction to be notified of transaction committed or aborted when it first encounters the transaction.

When a transaction rolls back, we undo the changes in the cache and release all locks.

When the transaction commits, we initiate a two-phase commit protocol Only with synchronous replication. : in the first phase, a PREPARE containing all modifications for the current transaction is sent to all nodes in the cluster. Each node acquires all necessary locks and applies the changes, and then sends back a success message. If a node in a cluster cannot acquire all locks, or fails otherwise, it sends back a failure message.

The coordinator of the two-phase commit protocol waits for all responses (or a timeout, whichever occurs first). If one of the nodes in the cluster responds with FAIL (or we hit the timeout), then a rollback phase is initiated: a ROLLBACK message is sent to all nodes in the cluster. On reception of the ROLLBACK message, every node undoes the changes for the given transaction, and releases all locks held for the transaction.

If all responses are OK, a COMMIT message is sent to all nodes in the cluster. On reception of a COMMIT message, each node applies the changes for the given transaction and releases all locks associated with the transaction.

When we referred to 'transaction', we actually mean a global representation of a local transaction, which uniquely identifies a transaction across a cluster.

### Example

Let's look at an example of how to use the standalone (e.g. outside an appserver) TreeCache with dummy transactions:

    Properties prop = new Properties();
    prop.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.cache.transaction.DummyContextFactory");
    User Transaction tx=(UserTransaction)new InitialContext(prop).lookup("UserTransaction");
    TreeCache tree = new TreeCache();
    PropertyConfigurator config = new PropertyConfigurator();
    config.configure(tree, "META-INF/replSync-service.xml");
    tree.createService(); // not necessary
    tree.startService(); // kick start tree cache

    try {
       tx.begin();
       tree.put("/classes/cs-101", "description", "the basics");
       tree.put("/classes/cs-101", "teacher", "Ben");
       tx.commit();
    }
    catch(Throwable ex) {
       try { tx.rollback(); } catch(Throwable t) {}
    }

The first lines obtain a user transaction using the 'J2EE way' via JNDI. Note that we could also say

    UserTransaction tx = new DummyUserTransaction(DummyTransactionManager.getInstance());

Then we create a new TreeCache and configure it using a PropertyConfigurator class and a configuration XML file (see below for a list of all configuration options).

Next we start the cache. Then, we start a transaction (and associate it with the current thread internally). Any methods invoked on the cache will now be collected and only applied when the transaction is committed. In the above case, we create a node "/classes/cs-101" and add 2 elements to its hashmap. Assuming that the cache is configured to use synchronous replication, on transaction commit the modifications are replicated. If there is an exception in the methods (e.g. lock acquisition failed), or in the two-phase commit protocol applying the modifications to all nodes in the cluster, the transaction is rolled back.

## Eviction policy

Eviction policy specifies the behavior of a node residing inside the cache, e.g., life time and maximum numbers allowed.

### Eviction policy plugin

The design in the TreeCache of eviction policy framework is based on the loosely coupled observable pattern (albeit still synchronous) where the eviction policy provider will implement one listener interface. It's the policy provider responsibility to decide when to call back the cache "evict" operation.

In order to implement an eviction policy, the provider will need to implement a listener interface that listens to the node events, namely,

    public interface TreeCacheListener {

       /**
        * Called when a node is created
        * @param fqn
        */
       void nodeCreated(Fqn fqn);

       /**
        * Called when a node is removed.
        * @param fqn
        */
       void nodeRemoved(Fqn fqn);

       /**
        * Called when a node is loaded into memory via the CacheLoader. This is not the same
        * as {@link #nodeCreated(Fqn)}.
        */
       void nodeLoaded(Fqn fqn);

       /**
        * Called when a node is evicted (not the same as remove()).
        * @param fqn
        */
       void nodeEvicted(Fqn fqn);

       /**
        * Called when a node is modified, e.g., one (key, value) pair
        * in the internal map storage has been modified.
        * @param fqn
        */
       void nodeModified(Fqn fqn);

       /**
        * Called when a node is visisted, i.e., get().
        * @param fqn
        */
       void nodeVisited(Fqn fqn);

       /**
        * Called when the cache is started.
        * @param cache
        */
       void cacheStarted(TreeCache cache);

       /**
        * Called when the cache is stopped.
        * @param cache
        */
       void cacheStopped(TreeCache cache);

       void viewChange(View new_view);  // might be MergeView after merging
    }

Note that the last two methods are used during cache life cycle. Eviction policy provider can use these two to kick start the policy.

In addition, TreeCache has two additional APIs that relate to eviction 1) to evict a node that similar to remove, and 2) to get a node value, but both without triggering a node event,

       /**
        * Called by eviction policy provider. Note that eviction is done only in local mode,
        * that is, it doesn't replicate the node removal.
        * @param fqn Will remove everything associated with this fqn.
        * @throws Exception
        */
       public void evict(Fqn fqn) throws Exception;

       /**
        * Like <code>get</code> method but without triggering a node visit event. This is used
        * to prevent refresh of the cache data in the eviction policy.
        * @param fqn
        * @param key
        * @return
        * @throws LockingException
        * @throws TimeoutException
        */
       public Object peek(Fqn fqn, Object key) throws LockingException, TimeoutException;

Note that:

- Eviction policy is only applied locally. I.e., if the mode is replicated, only the local node is evicted and it won't propagate the eviction to other participating caches. This is necessary because different caches may have different get operations that will touch the timestamp in the eviction policy algorithm. However, get operation is not replicated. Therefore, node in `cache2` may get evicted even when it is accessed often in cache1! There is always the question of state synchronization. This is OK since if user in cache1 finds out a node does not exist, it should get it from the underlying database and then put it into the cache. This operation will replicate across the group and the data will then be synchronized.

- When a node is evicted, it will check if it has children nodes. If it has, it will only remove the data value in the hashmap (without removing the whole node). If not, the whole node is removed.

### TreeCache LRU eviction policy implementation

TreeCache has implemented a LRU eviction policy, `org.jboss.cache.eviction.LRUPolicy`, that controls both the node lifetime and age. It has the following configuration parameters:

- `wakeUpIntervalSeconds`. This is the interval (in seconds) to process the node events and also to perform sweeping for the size limit and age-out nodes.

- `Region`. Region is a group of nodes where they posses the same eviction policy, e.g., same expired time. In TreeCache, region is denoted by a fqn, e.g., `/company/personnel`, and it is recursive. In specifying the region, the order is important. For example, if `/org/jboss/test` specified before `/org/jboss/test/data`, then any node under `/org/jboss/test/data` belongs to the first region rather than the second. Note also that whenever eviction policy is activated, there should be always a `/_default_` region where it covers all the eviction policy not specify by the user. In addition, the region configuration is not programmable, i.e., all the polciy has to be specified via XML configuration.

  - `maxNodes`. This is the maximum number of nodes allowed in this region. 0 denotes no limit.

  - `timeToIdleSeconds`. Time to idle (in seconds) before the node is swept away. 0 denotes no limit.

Please read the following section for an example.

## CacheLoader

JBossCache can use a CacheLoader to access a backend datastore. If JBossCache is configured with a CacheLoader, then the following properties are provided:

- whenever an element is accessed, and that element is not in the cache (e.g. due to eviction), then the CacheLoader transparently loads the element into the cache if found in the backend store
- whenever an element is modified, added or removed, then that modification is persisted in the backend store via the CacheLoader. If transactions are used, all modifications created within a transaction are persisted. To this end, the CacheLoader takes part in the two phase commit protocol run by the transaction manager

Currently, the CacheLoader looks similar to the TreeCache interface. In the future, it will actually implement the same interface. The goal is to be able to form hierarchical cache topologies, where one cache can delegate to another, which in turn may delegate to yet another cache.

The CacheLoader interface is defined in org.jboss.cache.loader.CacheLoader as follows (edited for brevity):

    public interface CacheLoader extends Service {

       /**
        * Sets the configuration. Will be called before {@link #create()} and {@link #start()}
        * @param url The name of the resource holding the configuration information, e.g. a URL, or a file name
        */
       void setConfig(String url);

       void setCache(TreeCache c);

       /**
        * Returns a list of children names, all names are <em>relative</em>. Returns null if the parent node is not found.
        * The returned set must not be modified, e.g. use Collections.unmodifiableSet(s) to return the result
        * @param fqn The FQN of the parent
        * @return Set<String>. A list of children. Returns null if no children nodes are present, or the parent is
        * not present
        */
       Set getChildrenNames(Fqn fqn) throws Exception;

       /**
        * Returns the value for a given key. Returns null if the node doesn't exist, or the value is not bound
        */
       Object get(Fqn name, Object key) throws Exception;

       /**
        * Returns all keys and values from the persistent store, given a fully qualified name
        * @param name
        * @return Map<Object,Object> of keys and values for the given node. Returns null if the node was not found
        * @throws Exception
        */
       Map get(Fqn name) throws Exception;

       /**
        * Checks whether the CacheLoader has a node with Fqn
        * @return True if node exists, false otherwise
        */
       boolean exists(Fqn name) throws Exception;

       /**
        * Inserts key and value into the attributes hashmap of the given node. If the node does not exist, all
        * parent nodes from the root down are created automatically
        */
       void put(Fqn name, Object key, Object value) throws Exception;

       /**
        * Inserts all elements of attributes into the attributes hashmap of the given node, overwriting existing
        * attributes, but not clearing the existing hashmap before insertion (making it a union of existing and
        * new attributes)
        * If the node does not exist, all parent nodes from the root down are created automatically
        * @param name The fully qualified name of the node
        * @param attributes A Map of attributes. Can be null
        */
       void put(Fqn name, Map attributes) throws Exception;

       /**
        * Inserts all modifications to the backend store. Overwrite whatever is already in
        * the datastore.
        * @param modifications A List<Modification> of modifications
        * @throws Exception
        */
       void put(List modifications) throws Exception;

       /** Removes the given key and value from the attributes of the given node. No-op if node doesn't exist */
       void remove(Fqn name, Object key) throws Exception;

       /**
        * Removes the given node. If the node is the root of a subtree, this will recursively remove all subnodes,
        * depth-first
        */
       void remove(Fqn name) throws Exception;

       /** Removes all attributes from a given node, but doesn't delete the node itself */
       void removeData(Fqn name) throws Exception;

       /**
        * Prepare the modifications. For example, for a DB-based CacheLoader:
        * <ol>
        * <li>Create a local (JDBC) transaction
        * <li>Associate the local transaction with <code>tx</code> (tx is the key)
        * <li>Execute the coresponding SQL statements against the DB (statements derived from modifications)
        * </ol>
        * For non-transactional CacheLoader (e.g. file-based), this could be a null operation
        * @param tx            The transaction, just used as a hashmap key
        * @param modifications List<Modification>, a list of all modifications within the given transaction
        * @param one_phase     Persist immediately and (for example) commit the local JDBC transaction as well. When true,
        *                      we won't get a {@link #commit(Object)} or {@link #rollback(Object)} method call later
        */
       void prepare(Object tx, List modifications, boolean one_phase) throws Exception;

       /**
        * Commit the transaction. A DB-based CacheLoader would look up the local JDBC transaction asociated
        * with <code>tx</code> and commit that transaction<br/>
        * Non-transactional CacheLoaders could simply write the data that was previously saved transiently under the
        * given <code>tx</code> key, to (for example) a file system (note this only holds if the previous prepare() did
        * not define one_phase=true
        */
       void commit(Object tx) throws Exception;

       /**
        * Roll the transaction back. A DB-based CacheLoader would look up the local JDBC transaction asociated
        * with <code>tx</code> and roll back that transaction
        */
       void rollback(Object tx);

       /**
        * Fetch the entire state for this cache from secondary storage (disk, DB) and return it as a byte buffer.
        * This is for initialization of a new cache from a remote cache. The new cache would then call
        * storeEntireState()
        * todo: define binary format for exchanging state
        */
       byte[] loadEntireState() throws Exception;

       /** Store the given state in secondary storage. Overwrite whatever is currently in storage */
       void storeEntireState(byte[] state) throws Exception;
    }

### CacheLoader interfaces

The interaction between JBossCache and a CacheLoader implementation is as follows. When `CacheLoaderClass` (see below) is non-null, an instance of it is created when the cache is created. Since `CacheLoader` extends `Service`,

    public interface Service {
       void create() throws Exception;

       void start() throws Exception;

       void stop();

       void destroy();
    }

`CacheLoader.create()` and `CacheLoader.start()` are called when the cache is started. Correspondingly, `stop()` and `destroy()` are called when the cache is stopped.

Next, `setConfig()` and `setCache()` are called. The latter can be used to store a reference to the cache, the former is used to configure this instance of the CacheLoader. For example, here a database CacheLoader could establish a connection to the database.

The CacheLoader interface has a set of methods that are called when no transactions are used: `get()`, `put()`, `remove()` and `removeData()`: they get/set/remove the value immediately. These methods are described as javadoc comments in the above interface.

Then there are three methods that are used with transactions: `prepare()`, `commit()` and `rollback()`. The `prepare()` method is called when a transaction is to be committed. It has a transaction object and a list of modfications as argument. The transaction object can be used as a key into a hashmap of transactions, where the values are the lists of modifications. Each modification list has a number of `Modification` elements, which represent the changes made to a cache for a given transaction. When `prepare()` returns successfully, then the CacheLoader must be able to commit (or rollback) the transaction successfully.

The `commit()` method tells the CacheLoader to commit the transaction, and the `rollback()` method tells the CacheLoader to discard the changes associated with that transaction.

The last two methods are `loadEntireState()` and `storeEntireState()`. The first method asks the CacheLoader to get the entire state the backend store manages and return it as a byte buffer, and the second tells a CacheLoader to replace its entire state with the byte buffer argument. These methods are used for scenarios where each JBossCache node in a cluster has its own local data store, e.g. a local DB, and - when a new node starts - we have to initialize its backend store with the contents of the backend store of an existing member. See below for deails.

### Configuration via XML

The CacheLoader is configured as follows in the JBossCache XML file:

        <!-- ==================================================================== -->
        <!-- Defines TreeCache configuration                                      -->
        <!-- ==================================================================== -->

        <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
           <attribute name="CacheLoaderClass">org.jboss.cache.loader.bdbje.BdbjeCacheLoader</attribute>
           <!-- attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute -->
           <attribute name="CacheLoaderConfig">c:\tmp\bdbje</attribute>
           <attribute name="CacheLoaderShared">true</attribute>
           <attribute name="CacheLoaderPreload">/</attribute>
           <attribute name="CacheLoaderFetchTransientState">false</attribute>
           <attribute name="CacheLoaderFetchPersistentState">true</attribute>
        </mbean>

The `CacheLoaderClass` attribute defines the class of the CacheLoader implementation. The currently available implementations shipped with JBossCache are `FileCacheLoader`, which is a simple filesystem-based implementation, and `BdbjeCacheLoader`, which is a CacheLoader implementation based on the Sleepycat DB Java Edition. Note that the Sleepycat implementation is much more efficient than the filesystem-based implemantation, and provides transactional guarantees, but requires a commercial license if distributed with an application (see http://www.sleepycat.com/jeforjbosscache for details). An implementation of CacheLoader has to have an empty constructor.

The `CacheLoaderConfig` attribute defines configuration specific to the given implementation. The filesystem-based implementation for example defines the root directory to be used, whereas a database implementation might define the database URL, name and password to establish a database connection. This configuration is passed to the CacheLoader implementation via `CacheLoader.setConfig(String)` as a string.

`CacheLoaderShared` determines whether each JBossCache node in a cluster is to use its own CacheLoader, or a shared CacheLoader. See discussion below for details.

`CacheLoaderPreload` allows us to define a list of nodes, or even entire subtrees, that are visited by the cache on startup, in order to preload the data associated with those nodes. The default ("/") loads the entire data available in the backend store into the cache, which is probably not a good idea given that the data in the backend store might be potentially large. As an example, `/a, /product/catalogue` loads the subtrees `/a` and `/product/catalogue` into the cache, but nothing else. Anything else is loaded lazily (when accessed). Preloading makes sense when one anticipates using elements under a given subtree frequently. Note that preloading loads all nodes and associated attributes from the given node, recursively up until the root node.

`CacheLoaderFetchTransientState` determines whether or not to fetch the in-memory state of a cache when joining a cluster.

`CacheLoaderFetchPersistentState` determines whether or not to fetch the persistent state of a cache when joining a cluster. This only applies if (a) a CacheLoader has been configured and (b) `CacheLoaderShared` is `false`.

Note that an attempt to fetch the transient and/or persistent state is only made if `FetchStateOnStartup` is true.

### CacheLoader use cases

#### Local cache with store

This is the simplest case. We have a JBossCache instance, whose mode is `LOCAL`, therefore no replication is going on. The CacheLoader simply loads non-existing elements from the store and stores modifications back to the store. When the cache is started, depending on the `CacheLoaderPreload` attribute, certain data can be preloaded, so that the cache is partly warmed up.

When using TreeCacheAop, this means that entire POJOs can be stored to a database or a filesystem, and when accessing fields of a POJO, they will be lazily loaded using the CacheLoader to access a backend store. This feature effectively provides simple persistency for any POJO.

#### Replicated caches with all nodes sharing the same store

The following figure shows 2 JBossCache nodes sharing the same backend store:

##### 2 nodes sharing a backend store

Both nodes have a CacheLoader that accesses a common shared backend store. This could for example be a shared filesystem (using the FileCacheLoader), or a shared database. Because both nodes access the same store, they don't necessarily need state transfer on startup. Of course they can enable state transfer, if they want to have a warm or hot cache after startup. Rather, the`FetchStateOnStartup` attribute could be set to false, resulting in a 'cold' cache, that gradually warms up as elements are accessed and loaded for the first time. This would mean that individual caches in a cluster might have different in-memory state at any given time (largely depending on their preloading and eviction strategies).

When storing a value, the writer takes care of storing the change in the backend store. For example, if node1 made change C1 and node2 C2, then node1 would tell its CacheLoader to store C1, and node2 would tell its CacheLoader to store C2.

#### Replicated caches with only one node having a store

##### 2 nodes but only one accesses the backend store

This is a similar case as the previous one, but here only one node in the cluster interacts with a backend store via its CacheLoader. All other nodes perform in-memory replication. A use case for this is HTTP session replication, where all nodes replicate sessions in-memory, and - in addition - one node saves the sessions to a persistent backend store. Note that here it may make sense for the CacheLoader to store changes asynchronously, that is not on the caller's thread, in order not to slow down the cluster by accessing (for example) a database. This is a non-issue when using asynchronous replication.

#### Replicated caches with each node having its own store

##### 2 nodes each having its own backend store

Here, each node has its own datastore. Modifications to the cache are (a) replicated across the cluster and (b) persisted using the CacheLoader. This means that all datastores have exactly the same state. When replicating changes synchronously and in a transaction, the two phase commit protocol takes care that all modifications are replicated and persisted in each datastore, or none is replicated and persisted (atomic updates).

Note that currently JBossCache is not an XAResource, that means it doesn't implement recovery. When used with a TransactionManager that supports recovery, this functionality is not available.

The challenge here is state transfer: when a new node starts it needs to do the following:

1. Tell the coordinator (oldest node in a cluster) to send it the state

2. The coordinator then needs to wait until all in-flight transactions have completed. During this time, it will not allow for new transactions to be started.

3. Then the coordinator asks its CacheLoader for the entire state using `loadEntireState()`. It then sends back that state to the new node.

4. The new node then tells its CacheLoader to store that state in its store, overwriting the old state. This is the `CacheLoader.storeEntireState()` method

5. As an option, the transient (in-memory) state can be transferred as well during the state transfer.

6. The new node now has the same state in its backend store as everyone else in the cluster, and modifications received from other nodes will now be persisted using the local CacheLoader.

## Configuration

All properties of the cache are configured via setters and can be retrieved via getters. This can be done either manually, or via the PropertyConfigurator and an XML file. A sample configuration file is shown below (stripped of comments etc):

    <?xml version="1.0" encoding="UTF-8" ?>
    <server>
      <classpath codebase="./lib" archives="jboss-cache.jar, jgroups.jar" />

      <!--  ====================================================================  -->
      <!--  Defines TreeCache configuration                                       -->
      <!--  ====================================================================  -->
      <mbean code="org.jboss.cache.TreeCache" name="jboss.cache:service=TreeCache">
        <depends>jboss:service=Naming</depends>
        <depends>jboss:service=TransactionManager</depends>

        <!-- Configure the TransactionManager -->
        <attribute name="TransactionManagerLookupClass">org.jboss.cache.DummyTransactionManagerLookup</attribute>

        <!--
                Node locking level : SERIALIZABLE
                                     REPEATABLE_READ (default)
                                     READ_COMMITTED
                                     READ_UNCOMMITTED
                                     NONE
        -->
        <attribute name="IsolationLevel">REPEATABLE_READ</attribute>

        <!--     Valid modes are LOCAL
                                 REPL_ASYNC
                                 REPL_SYNC
        -->
        <attribute name="CacheMode">LOCAL</attribute>

        <!-- Name of cluster. Needs to be the same for all clusters, in order
                 to find each other -->
        <attribute name="ClusterName">TreeCache-Cluster</attribute>

        <attribute name="ClusterConfig">
          <config>
            <!-- UDP: if you have a multihomed machine,
                    set the bind_addr attribute to the appropriate NIC IP address
            -->
            <!-- UDP: On Windows machines, because of the media sense feature
                     being broken with multicast (even after disabling media sense)
                     set the loopback attribute to true
            -->
            <UDP mcast_addr="228.1.2.3" mcast_port="45566" ip_ttl="64" ip_mcast="true"
               mcast_send_buf_size="150000" mcast_recv_buf_size="80000" ucast_send_buf_size="150000"
               ucast_recv_buf_size="80000" loopback="false" />
            <PING timeout="2000" num_initial_members="3" up_thread="false" down_thread="false" />
            <MERGE2 min_interval="10000" max_interval="20000" />
            <FD shun="true" up_thread="true" down_thread="true" />
            <VERIFY_SUSPECT timeout="1500" up_thread="false" down_thread="false" />
            <pbcast.NAKACK gc_lag="50" max_xmit_size="8192" retransmit_timeout="600,1200,2400,4800" up_thread="false"
               down_thread="false" />
            <UNICAST timeout="600,1200,2400" window_size="100" min_threshold="10" down_thread="false" />
            <pbcast.STABLE desired_avg_gossip="20000" up_thread="false" down_thread="false" />
            <FRAG frag_size="8192" down_thread="false" up_thread="false" />
            <pbcast.GMS join_timeout="5000" join_retry_timeout="2000" shun="true" print_local_addr="true" />
            <pbcast.STATE_TRANSFER up_thread="false" down_thread="false" />
          </config>
        </attribute>

        <!--    The max amount of time (in milliseconds) we wait until the
                initial state (ie. the contents of the cache) are retrieved from
                existing members in a clustered environment
        -->
        <attribute name="InitialStateRetrievalTimeout">5000</attribute>

        <!--    Number of milliseconds to wait until all responses for a
                synchronous call have been received.
        -->
        <attribute name="SyncReplTimeout">10000</attribute>

        <!--  Max number of milliseconds to wait for a lock acquisition -->
        <attribute name="LockAcquisitionTimeout">15000</attribute>

        <!--  Name of the eviction policy class. -->
        <attribute name="EvictionPolicyClass">org.jboss.cache.eviction.LRUPolicy</attribute>

        <!--  Specific eviction policy configurations. This is LRU -->
        <attribute name="EvictionPolicyConfig">
          <config>
            <attribute name="wakeUpIntervalSeconds">5</attribute>
            <!--  Cache wide default -->
            <region name="/_default_">
             <attribute name="maxNodes">5000</attribute>
             <attribute name="timeToIdleSeconds">1000</attribute>
           </region>

           <region name="/org/jboss/data">
             <attribute name="maxNodes">5000</attribute>
             <attribute name="timeToIdleSeconds">1000</attribute>
           </region>

           <region name="/org/jboss/test/data">
             <attribute name="maxNodes">5</attribute>
             <attribute name="timeToIdleSeconds">4</attribute>
           </region>
          </config>
        </attribute>

        <attribute name="CacheLoaderClass">org.jboss.cache.loader.bdbje.BdbjeCacheLoader</attribute>
        <!--       <attribute name="CacheLoaderClass">org.jboss.cache.loader.FileCacheLoader</attribute>-->
        <attribute name="CacheLoaderConfig">c:\\tmp#bela.dat</attribute>
        <attribute name="CacheLoaderShared">true</attribute>
        <attribute name="CacheLoaderPreload">/a/b/c,/all/my/objects</attribute>
        <attribute name="CacheLoaderFetchTransientState">false</attribute
        <attribute name="CacheLoaderFetchPersistentState">true</attribute

      </mbean>
    </server>

The PropertyConfigurator.configure() method needs to have as argument a filename that needs to be found on the classpath, and will use it to configure the TreeCache from the properties defined in it. Note that this configuration file is used to configure the TreeCache both as a standalone cache, and as an MBean if run inside the JBoss container Actually, we will use an XMBean in the next release. .

A list of properties is shown below:

Name | Description
CacheLoaderClass | The fully qualified name of the CacheLoader implementation
CacheLoaderConfig | A string from which the specific CacheLoader implementation can configure itself
CacheLoaderFetchPersistentState | {only used if FetchStateOnStartup is true) Fetches the persistent state from another node, only if CacheLoaderShared is false
CacheLoaderFetchTransientState | (only used if FetchStateOnStartup is true) Fetches the in-memory state from another node
CacheLoaderPreload | A list of comma-separate nodes that need to be preloaded, e.g. `/aop, /productcatalogue`
CacheLoaderShared | Whether we want to shared a datastore, or whether each node wants to have its own local datastore
CacheMode | LOCAL, REPL_SYNC or REPL_ASYNC
ClusterName | Name of cluster. Needs to be the same for all nodes in a cluster in order to find each other
ClusterConfig | The configuration of the underlying JGroups stack. See cluster-service.xml for an example.
EvictionPolicyClass | The name of a class implementing EvictionPolicy. If empty, no eviction policy is enabled.
EvictionPolicyConfig | Configuration parameter for the specified eviction policy. Note that the content is provider specific.
FetchStateOnStartup | Whether or not to acquire the initial state from existing members. Allows for warm/hot caches (true/false). This can be further defined by CacheLoaderFetchTransientState and CacheLoaderFetchPersistentState
InitialStateRetrievalTimeout | Time in milliseconds to wait for initial state retrieval
IsolationLevel | Node locking level : SERIALIZABLE, REPEATABLE_READ (default), READ_COMMITTED, READ_UNCOMMITTED, and NONE. Case doesn't matter. See documentation on locking for details.
LockAcquisitionTimeout | Time in milliseconds to wait for a lock to be acquired. If a lock cannot be acquired an exception will be thrown
ReplQueueInterval | Time in milliseconds for elements from the replication queue to be replicated.
SyncReplTimeout | For synchronous replication: time in milliseconds to wait until replication acks have been received from all nodes in the cluster
ReplQueueMaxElements | Max number of elements in the replication queue until replication kicks in
TransactionManagerLookupClass | The fully qualified name of a class implementing TransactionManagerLookup. Default is JBossTransactionManagerLookup. There is also an option of DummyTransactionManagerLookup for example.
UseReplQueue | For asynchronous replication: whether or not to use a replication queue (true/false).