<!-- $Id: logshipping.sgml,v 1.9 2005/04/15 22:27:05 cbbrowne Exp $ -->
<sect1 id="logshipping">
<title>Log Shipping - &slony1; with Files</title>

<para> One of the new features for 1.1 is the ability to serialize the
updates to go out into log files that can be kept in a spool
directory.</para>

<para> The spool files could then be transferred via whatever means
was desired to a <quote>slave system,</quote> whether that be via FTP,
rsync, or perhaps even by pushing them onto a 1GB <quote>USB
key</quote> to be sent to the destination by clipping it to the ankle
of some sort of <quote>avian transport</quote> system.</para>

<para> There are plenty of neat things you can do with a data stream
in this form, including:

<itemizedlist>
  
  <listitem><para> Replicating to nodes that
  <emphasis>aren't</emphasis> securable</para></listitem>

  <listitem><para> Replicating to destinations where it is not
  possible to set up bidirection communications</para></listitem>

  <listitem><para> Supporting a different form of <acronym>PITR</acronym>
  (Point In Time Recovery) that filters out read-only transactions and
  updates to tables that are not of interest.</para></listitem>

  <listitem><para> If some disaster strikes, you can look at the logs
  of queries in detail</para>

  <para> This makes log shipping potentially useful even though you
  might not intend to actually create a log-shipped node.</para></listitem>

  <listitem><para> This is a really slick scheme for building load for
  doing tests</para></listitem>

  <listitem><para> We have a data <quote>escrow</quote> system that
  would become incredibly cheaper given log shipping</para></listitem>

  <listitem><para> You may apply triggers on the <quote>disconnected
  node </quote> to do additional processing on the data</para>

  <para> For instance, you might take a fairly <quote>stateful</quote>
  database and turn it into a <quote>temporal</quote> one by use of
  triggers that implement the techniques described in
  <citation>Developing Time-Oriented Database Applications in SQL
  </citation> by <ulink url="http://www.cs.arizona.edu/people/rts/">
  Richard T. Snodgrass</ulink>.</para></listitem>

</itemizedlist></para>

<qandaset>
<qandaentry>

<question> <para> Where are the <quote>spool files</quote> for a
subscription set generated?</para>
</question>

<answer> <para> Any <link linkend="slon"> slon </link> subscriber node
can generate them by adding the <option>-a</option> option.</para>

<note><para> Notice that this implies that in order to use log
shipping, you must have at least one subscriber node. </para></note>
</answer>

</qandaentry>

<qandaentry> <question> <para> What takes place when a <xref
linkend="stmtfailover">/ <xref linkend="stmtmoveset"> takes
place?</para></question>

<answer><para> Nothing special.  So long as the archiving node remains
a subscriber, it will continue to generate logs.</para></answer>
</qandaentry>

<qandaentry> <question> <para> What if we run out of <quote>spool
space</quote>?  </para></question>

<answer><para> The node will stop accepting <command>SYNC</command>s
until this problem is alleviated.  The database being subscribed to
will also fall behind.  </para></answer>
</qandaentry>

<qandaentry>
<question> <para> How do we set up a subscription?  </para></question>

<answer><para> The script in <filename>tools</filename> called
<application>slony1_dump.sh</application> is a shell script that dumps
the <quote>present</quote> state of the subscriber node.</para>

<para> You need to start the <application><link linkend="slon"> slon
</link></application> for the subscriber node with logging turned on.
At any point after that, you can run
<application>slony1_dump.sh</application>, which will pull the state
of that subscriber as of some <command>SYNC</command> event.  Once the dump completes,
all the <command>SYNC</command> logs generated from the time that dump
<emphasis>started</emphasis> may be added to the dump in order to get
a <quote>log shipping subscriber.</quote> </para></answer>
</qandaentry>

<qandaentry> <question><para> What are the limitations of log
shipping? </para>
</question>

<answer><para> In the initial release, there are rather a lot of
limitations.  As releases progress, hopefully some of these
limitations may be alleviated/eliminated. </para> </answer>

<answer><para> The log shipping functionality amounts to
<quote>sniffing</quote> the data applied at a particular subscriber
node.  As a result, you must have at least one <quote>regular</quote>
node; you cannot have a cluster that consists solely of an origin and
a set of <quote>log shipping nodes.</quote>. </para></answer>

<answer><para> The <quote>log shipping node</quote> tracks the
entirety of the traffic going to a subscriber.  You cannot separate
things out if there are multiple replication sets.  </para></answer>

<answer><para> The <quote>log shipping node</quote> presently tracks
only <command>SYNC</command> events.  This should be sufficient to cope with
<emphasis>some</emphasis> changes in cluster configuration, but not
others.  </para>

<para> Log shipping does <emphasis>not</emphasis> process certain
additional events, with the implication that the introduction of any
of the following events can invalidate the relationship between the
<command>SYNC</command>s and the dump created using
<application>slony1_dump.sh</application> so that you'll likely need
to rerun <application>slony1_dump.sh</application>:

<itemizedlist>
<listitem><para><command> SUBSCRIBE_SET </command></para></listitem> 

</itemizedlist></para>

<para> A number of event types <emphasis> are </emphasis> handled in
such a way that log shipping copes with them:

<itemizedlist>

<listitem><para><command>SYNC </command> events are, of course,
handled.</para></listitem>

<listitem><para><command>DDL_SCRIPT</command> is handled.</para></listitem>

<listitem><para><command> UNSUBSCRIBE_SET </command></para> 

<para> This event, much like <command>SUBSCRIBE_SET</command> is not
handled by the log shipping code.  But its effect is, namely that
<command>SYNC</command> events on the subscriber node will no longer
contain updates to the set.</para>

<para> Similarly, <command>SET_DROP_TABLE</command>,
<command>SET_DROP_SEQUENCE</command>,
<command>SET_MOVE_TABLE</command>,
<command>SET_MOVE_SEQUENCE</command> <command>DROP_SET</command>,
<command>MERGE_SET</command>, will be handled
<quote>appropriately</quote>.</para></listitem>

<listitem><para> The various events involved in node configuration are
irrelevant to log shipping:

<command>STORE_NODE</command>,
<command>ENABLE_NODE</command>,
<command>DROP_NODE</command>,
<command>STORE_PATH</command>,
<command>DROP_PATH</command>,
<command>STORE_LISTEN</command>,
<command>DROP_LISTEN</command></para></listitem>

<listitem><para> Events involved in describing how particular sets are
to be initially configured are similarly irrelevant:

<command>STORE_SET</command>,
<command>SET_ADD_TABLE</command>,
<command>SET_ADD_SEQUENCE</command>,
<command>STORE_TRIGGER</command>,
<command>DROP_TRIGGER</command>,</para></listitem>

</itemizedlist>
</para>
</answer>

<answer><para> It would be nice to be able to turn a <quote>log
shipped</quote> node into a fully communicating &slony1; node that you
could failover to.  This would be quite useful if you were trying to
construct a cluster of (say) 6 nodes; you could start by creating one
subscriber, and then use log shipping to populate the other 4 in
parallel.</para>

<para> This usage is not supported, but presumably one could add the
&slony1; configuration to the node, and promote it into being a new
node.  Again, a Simple Matter Of Programming (that might not
necessarily be all that simple)... </para></answer>
</qandaentry>
</qandaset>

<sect2><title> Usage Hints </title>

<note> <para> Here are some more-or-less disorganized notes about how
you might want to use log shipping...</para> </note>

<itemizedlist>

<listitem><para> You <emphasis>don't</emphasis> want to blindly apply
<command>SYNC</command> files because any given <command>SYNC</command> file may <emphasis>not</emphasis> be
the right one.  If it's wrong, then the result will be that the call
to <function> setsyncTracking_offline() </function> will fail, and
your <application> psql</application> session will <command> ABORT
</command>, and then run through the remainder of that <command>SYNC</command> file
looking for a <command>COMMIT</command> or <command>ROLLBACK</command>
so that it can try to move on to the next transaction.</para>

<para> But we <emphasis> know </emphasis> that the entire remainder of
the file will fail!  It is futile to go through the parsing effort of
reading the remainder of the file.</para>

<para> Better idea: 

<itemizedlist>

<listitem><para> Read the first few lines of the file, up to and
including the <function> setsyncTracking_offline() </function> call.</para></listitem>  

<listitem><para> Try to apply it that far.</para></listitem>

<listitem><para> If that failed, then it is futile to continue;
<command>ROLLBACK</command> the transaction, and perhaps consider
trying the next file.</para></listitem>

<listitem><para> If the <function> setsyncTracking_offline()
</function> call succeeds, then you have the right next
<command>SYNC</command> file, and should apply it.  You should
probably <command>ROLLBACK</command> the transaction, and then use
<application>psql</application> to apply the entire file full of
updates.</para></listitem>

</itemizedlist></para>

<para> In order to support the approach of grabbing just the first few
lines of the sync file, the format has been set up to have a line of
dashes at the end of the <quote>header</quote> material:

<programlisting>
-- Slony-I sync log
-- Node 11, event 745
start transaction;
select "_T1".setsyncTracking_offline(1, '744', '745');
--------------------------------------------------------------------
</programlisting></para></listitem>

</itemizedlist>
</sect2>
</sect1>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"book.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->
