<!-- $Id: maintenance.sgml,v 1.15 2005/05/13 21:24:03 cbbrowne Exp $ -->
<sect1 id="maintenance"> <title>&slony1; Maintenance</title>

<para>&slony1; actually does a lot of its necessary maintenance
itself, in a <quote>cleanup</quote> thread:

<itemizedlist>

<listitem><para> Deletes old data from various tables in the
<productname>Slony-I</productname> cluster's namespace, notably
entries in <xref linkend="table.sl-log-1">, <xref
linkend="table.sl-log-2"> (not yet used), and <xref
linkend="table.sl-seqlog">.</para></listitem>

<listitem><para> Vacuum certain tables used by &slony1;.  As of 1.0.5,
this includes pg_listener; in earlier versions, you must vacuum that
table heavily, otherwise you'll find replication slowing down because
&slony1; raises plenty of events, which leads to that table having
plenty of dead tuples.</para>

<para> In some versions (1.1, for sure; possibly 1.0.5) there is the
option of not bothering to vacuum any of these tables if you are using
something like <application>pg_autovacuum</application> to handle
vacuuming of these tables.  Unfortunately, it has been quite possible
for <application>pg_autovacuum</application> to not vacuum quite
frequently enough, so you probably want to use the internal vacuums.
Vacuuming <envar>pg_listener</envar> <quote>too often</quote> isn't
nearly as hazardous as not vacuuming it frequently enough.</para>

<para>Unfortunately, if you have long-running transactions, vacuums
cannot clear out dead tuples that are newer than the eldest
transaction that is still running.  This will most notably lead to
<envar>pg_listener</envar> growing large and will slow
replication.</para></listitem>


<listitem> <para> The <link linkend="dupkey"> Duplicate Key Violation
</link> bug has helped track down some &postgres; race conditions.
One remaining issue is that it appears that is a case where
<command>VACUUM</command> is not reclaiming space correctly, leading
to corruption of B-trees. </para>

<para> It may be helpful to run the command <command> REINDEX TABLE
sl_log_1;</command> periodically to avoid the problem
occurring. </para> </listitem>

</itemizedlist>
</para>

<sect2><title> Watchdogs: Keeping Slons Running</title>

<para>There are a couple of <quote>watchdog</quote> scripts available that
monitor things, and restart the <application>slon</application> processes should
they happen to die for some reason, such as a network <quote>glitch</quote>
that causes loss of connectivity.</para>

<para>You might want to run them...</para>
</sect2>
<sect2><title>Parallel to Watchdog: generate_syncs.sh</title>

<para>A new script for &slony1; 1.1 is
<application>generate_syncs.sh</application>, which addresses the following kind of
situation.</para>

<para>Supposing you have some possibly-flakey server where the
<application>slon</application> daemon that might not run all the time, you might
return from a weekend away only to discover the following situation.</para>

<para>On Friday night, something went <quote>bump</quote> and while the
database came back up, none of the <application>slon</application> daemons
survived.  Your online application then saw nearly three days worth of
reasonably heavy transaction load.</para>

<para>When you restart <application>slon</application> on Monday, it
hasn't done a SYNC on the master since Friday, so that the next
<quote>SYNC set</quote> comprises all of the updates between Friday
and Monday.  Yuck.</para>

<para>If you run <application>generate_syncs.sh</application> as a cron job every
20 minutes, it will force in a periodic <command>SYNC</command> on the origin, which
means that between Friday and Monday, the numerous updates are split
into more than 100 syncs, which can be applied incrementally, making
the cleanup a lot less unpleasant.</para>

<para>Note that if <command>SYNC</command>s <emphasis>are</emphasis> running
regularly, this script won't bother doing anything.</para>
</sect2>

<sect2><title>Replication Test Scripts </title>

<para> In the directory <filename>tools</filename> may be found four
scripts that may be used to do monitoring of &slony1; instances:

<itemizedlist>

<listitem><para> <command>test_slony_replication</command> is a
Perl script to which you can pass connection information to get to a
&slony1; node.  It then queries <xref linkend="table.sl-path"> and
other information on that node in order to determine the shape of the
requested replication set.</para>

<para> It then injects some test queries to a test table called
<envar>slony_test</envar> which is defined as follows, and which needs to be
added to the set of tables being replicated:

<programlisting>
CREATE TABLE slony_test (
    description text,
    mod_date timestamp with time zone,
    "_Slony-I_testcluster_rowID" bigint DEFAULT nextval('"_testcluster".sl_rowid_seq'::text) NOT NULL
);
</programlisting></para>

<para> The last column in that table was defined by &slony1; as one
lacking a primary key...</para>

<para> This script generates a line of output for each &slony1; node
that is active for the requested replication set in a file called
<filename>cluster.fact.log</filename>.</para>

<para> There is an additional <option>finalquery</option> option that allows
you to pass in an application-specific SQL query that can determine
something about the state of your application.</para></listitem>

<listitem><para><command>log.pm</command> is a Perl module that manages logging
for the Perl scripts.</para></listitem>

<listitem><para><command>run_rep_tests.sh</command> is a <quote>wrapper</quote> script
that runs <command>test_slony_replication</command>.</para>

<para> If you have several &slony1; clusters, you might set up
configuration in this file to connect to all those
clusters.</para></listitem>

<listitem><para><command>nagios_slony_test</command> is a script
that was constructed to query the log files so that you might run the
replication tests every so often (we run them every 6 minutes), and
then a system monitoring tool such as <ulink
url="http://www.nagios.org/"> <productname>Nagios</productname>
</ulink> can be set up to use this script to query the state indicated
in those logs.</para>

<para> It seemed rather more efficient to have a
<application>cron</application> job run the tests and have
<productname>Nagios</productname> check the results rather than having
<productname>Nagios</productname> run the tests directly.  The tests
can exercise the whole &slony1; cluster at once rather than
<productname>Nagios</productname> invoking updates over and over
again.</para></listitem>

</itemizedlist></para>

</sect2>

<sect2><title> Log Files</title>

<para><xref linkend="slon"> daemons generate some more-or-less verbose
log files, depending on what debugging level is turned on.  You might
assortedly wish to:

<itemizedlist>

<listitem><para> Use a log rotator like <productname>Apache</productname> 
<application>rotatelogs</application> to have a sequence of log files so that no
one of them gets too big;</para></listitem>

<listitem><para> Purge out old log files, periodically.</para></listitem>

</itemizedlist>
</para>
</sect2>
</sect1>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:book.sgml
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->

