<!-- $Id: monitoring.sgml,v 1.20 2005/06/22 17:32:27 devrim Exp $ -->
<sect1 id="monitoring">
<title>Monitoring</title>

<para>Here are some of things that you may find in your &slony1; logs,
and explanations of what they mean.</para>

<sect2><title>CONFIG notices</title>

<para>These entries are pretty straightforward. They are informative
messages about your configuration.</para>

<para>Here are some typical entries that you will probably run into in
your logs:

<screen>
CONFIG main: local node id = 1
CONFIG main: loading current cluster configuration
CONFIG storeNode: no_id=3 no_comment='Node 3'
CONFIG storePath: pa_server=5 pa_client=1 pa_conninfo="host=127.0.0.1 dbname=foo user=postgres port=6132" pa_connretry=10
CONFIG storeListen: li_origin=3 li_receiver=1 li_provider=3
CONFIG storeSet: set_id=1 set_origin=1 set_comment='Set 1'
CONFIG main: configuration complete - starting threads
</screen></para></sect2>

<sect2><title>DEBUG Notices</title>

<para>Debug notices are always prefaced by the name of the thread that
the notice originates from. You will see messages from the following
threads:

<variablelist>
<varlistentry><term>localListenThread</term> 

<listitem><para> This is the local thread that listens for events on
the local node.</para></listitem></varlistentry>

<varlistentry><term>remoteWorkerThread-X</term> 

<listitem><para> The thread processing remote events.  You can expect
to see one of these for each node that this node communicates
with.</para></listitem></varlistentry>

<varlistentry><term>remoteListenThread-X</term>

<listitem><para>Listens for events on a remote node database.  You may
expect to see one of these for each node in the
cluster.</para></listitem></varlistentry>

<varlistentry><term>cleanupThread</term> <listitem><para> Takes care
of things like vacuuming, cleaning out the confirm and event tables,
and deleting old data.</para></listitem></varlistentry>

<varlistentry><term>syncThread</term> <listitem><para> Generates SYNC
events.</para></listitem></varlistentry>

</variablelist>
</para>
</sect2>

<sect2> <title> How to read &slony1; logs </title>

<para> Note that as far as slon is concerned, there is no "master" or
"slave."  They are just nodes. </para>

<para>What you can expect, initially, is to see, on both nodes, some
events propagating back and forth.  Firstly, there should be some
events published to indicate creation of the nodes and paths.  If you
don't see those, then the nodes aren't likely to be able to
communicate with one another, and nothing else will happen... </para>

<itemizedlist>

<listitem><para>Create the two nodes.</para> 

<para> No slons are running yet, so there are no logs to look at.</para>
</listitem>

<listitem><para> Start the two slons</para>

<para> The logs for each will start out very quiet, as neither node
has much to say, and neither node knows how to talk to any other node.
</para>

</listitem>

<listitem><para> Do the <command>STORE PATH</command> for the
communications paths.  That will allow the nodes to start to become
aware of one another.</para>

<para> In version 1.0, <xref linkend="table.sl-listen"> is not set up
automatically, so things still remain quiet until you explicitly
submit <command>STORE LISTEN</command> requests. In version 1.1, the
<quote>listen paths</quote> are set up automatically, which will much
more quickly get the communications network up and running.  </para>

<para> If you look at the contents of the tables <xref
linkend="table.sl-node"> and <xref linkend="table.sl-path"> and <xref
linkend="table.sl-listen">, on each node, that should give a good idea
as to where things stand.  Until the <xref linkend="slon"> starts,
each node may only be partly configured.  If there are two nodes,
there should be two entries in all three of these tables once the
communications configuration is set up properly.  If there are fewer
entries than that, well, that should give you some idea of what is
missing.</para>
</listitem>

<listitem><para> If needed (<emphasis>e.g.</emphasis> - before version
1.1), submit <command>STORE LISTEN</command> requests to indicate how
the nodes will use the communications paths. </para>

<para> Once this has been done, the nodes' logs should show a greater
level of activity, with events periodically being initiated on one
node or the other, and propagating to the other. </para>
</listitem>

<listitem> <para> You'll set up the set (<command>CREATE
SET</command>), add tables (<command>SET ADD TABLE</command>), and
sequences (<command>SET ADD SEQUENCE</command>), and will see relevant
events only on the origin node for the set. </para></listitem>

<listitem><para> Then, when you submit the <command>SUBSCRIBE SET</command>
request, the event should go to both nodes. </para>

<para> The origin node has little more to do, after that...  The
subscriber will then have a <command>COPY_SET</command> event, which
will lead to logging information about adding each table and copying
its data.</para></listitem>

</itemizedlist>

<para>After that, you'll mainly see two sorts of behaviour:</para>

<itemizedlist>

<listitem><para> On the origin, there won't be much logged, just
indication that some <command>SYNC</command> events are being
generated and confirmed by other nodes.</para></listitem>

<listitem><para> On the subscriber, there will be reports of
<command>SYNC</command> events, and that the subscriber pulls data
from the provider for the relevant set(s).  This will happen
infrequently if there are no updates going to the origin node; it will
happen frequently when the origin sees heavy updates. </para>
</listitem>

</itemizedlist>

<para> WriteMe: I can't decide the format for the rest of this. I
think maybe there should be a "how it works" page, explaining more
about how the threads work, what to expect in the logs after you run a
</para>
</sect2>

<sect2> <title> Nagios Replication Checks </title>

<para> The script in the <filename>tools</filename> directory called
<command> pgsql_replication_check.pl </command> represents about best
answers yet arrived at in several attempts to build replication tests
to plug into the <ulink url="http://www.nagios.org/"> Nagios </ulink>
system monitoring tool.</para>

<para> A former script, <filename>
test_slony_replication.pl</filename>, took a <quote>clever</quote>
approach where a <quote>test script</quote> is periodically run, which
rummages through the &slony1; configuration to find origin and
subscribers, injects a change, and watches for its propagation through
the system.  It had two problems:</para>
<itemizedlist>

<listitem><para> Connectivity problems to the
<emphasis>single</emphasis> host where the test ran would make it look
as though replication was destroyed.  Overall, this monitoring
approach has been fragile to numerous error conditions.</para>
</listitem>

<listitem><para> Nagios has no ability to benefit from the
<quote>cleverness</quote> of automatically exploring the set of nodes.
</para> </listitem>
</itemizedlist>

<para> The new script, <command>pgsql_replication_check.pl</command>,
takes the minimalist approach of assuming that the system is an online
system that sees regular <quote>traffic,</quote> so that you can
define a view specifically for the replication test called
<envar>replication_status</envar> which is expected to see regular
updates.  The view simply looks for the youngest
<quote>transaction</quote> on the node, and lists its timestamp, age,
and some bit of application information that might seem useful to see.
</para>

<itemizedlist>

<listitem><para> In an inventory system, that might be the order
number for the most recently processed order. </para> </listitem>

<listitem><para> In a domain registry, that might be the name of the
most recently created domain.</para> </listitem>

</itemizedlist>

<para> An instance of the script will need to be run for each node
that is to be monitored; that is the way
<application>Nagios</application> works. </para>

</sect2>

<sect2 id="testslonystate"> <title> test_slony_state</title>

<para> This script is in preliminary stages, and may be used to do
some analysis of the state of a &slony1; cluster.</para>

<para> You specify arguments including <option>database</option>,
<option>host</option>, <option>user</option>,
<option>cluster</option>, <option>password</option>, and
<option>port</option> to connect to any of the nodes on a cluster.
You also specify a <option>mailprog</option> command (which should be
a program equivalent to <productname>Unix</productname>
<application>mailx</application>) and a recipient of email. </para>

<para> The script then rummages through <xref linkend="table.sl-path">
to find all of the nodes in the cluster, and the DSNs to allow it to,
in turn, connect to each of them.</para>

<para> For each node, the script examines the state of things,
including such things as:

<itemizedlist>
<listitem><para> Checking <xref linkend="table.sl-listen"> for some
<quote>analytically determinable</quote> problems.  It lists paths
that are not covered.</para></listitem>

<listitem><para> Providing a summary of events by origin node</para>

<para> If a node hasn't submitted any events in a while, that likely
suggests a problem.</para></listitem>

<listitem><para> Summarizes the <quote>aging</quote> of table <xref
linkend="table.sl-confirm"> </para>

<para> If one or another of the nodes in the cluster hasn't reported
back recently, that tends to lead to cleanups of tables like <xref
linkend="table.sl-log-1"> and <xref linkend="table.sl-seqlog"> not
taking place.</para></listitem>

<listitem><para> Summarizes what transactions have been running for a
long time</para>

<para> This only works properly if the statistics collector is
configured to collect command strings, as controlled by the option
<option> stats_command_string = true </option> in <filename>
postgresql.conf </filename>.</para>

<para> If you have broken applications that hold connections open,
this will find them.</para>

<para> If you have broken applications that hold connections open,
that has several unsalutory effects as <link
linkend="longtxnsareevil"> described in the
FAQ</link>.</para></listitem>

</itemizedlist></para>

<para> The script does some diagnosis work based on parameters in the
script; if you don't like the values, pick your favorites!</para>

</sect2>

</sect1>
<!-- Keep this comment at the end of the file
Local variables:
mode:sgml
sgml-omittag:nil
sgml-shorttag:t
sgml-minimize-attributes:nil
sgml-always-quote-attributes:t
sgml-indent-step:1
sgml-indent-data:t
sgml-parent-document:"book.sgml"
sgml-exposed-tags:nil
sgml-local-catalogs:("/usr/lib/sgml/catalog")
sgml-local-ecat-files:nil
End:
-->
