<?xml version="1.0" encoding="UTF-8"?>

<!-- ====================================================================== -->
<!-- = chapter-1.xml                                                      = -->
<!-- = standard text processing module guide - chapter 1                  = -->
<!-- ====================================================================== -->
<!-- = This  program  is  free  software; you  can redistribute it and/or = -->
<!-- = modify it provided that this copyright notice is kept intact.      = -->
<!-- = This program is distributed in the hope that it will be useful but = -->
<!-- = without  any  warranty;  without  even  the  implied  warranty  of = -->
<!-- = merchantability or fitness for  a  particular purpose. In no event = -->
<!-- = shall  the  copyright  holder be liable for any  direct, indirect, = -->
<!-- = incidental  or special  damages arising  in any way out of the use = -->
<!-- = of this software.                                                  = -->
<!-- ====================================================================== -->
<!-- = copyright (c) 2005-2007 - amaury darsch                            = -->
<!-- ====================================================================== -->

<chapter module="txt" number="1">
  <title>Text scanning</title>
  
  <p>
  This chapter is dedicated to the <afnix/> text scanning, a
  subpart of the <em>standard text processing</em> module. Text
  scanning refers to the ability to constructs <em>lexemes</em> from a
  text stream. The lexemes are built from the stream with the help of
  regular expressions. Eventually some special operations like
  <em>text transliteration</em> is also possible.
  </p>

  <p>
  All <afnix/> text processing objects are located in the
  <code>afnix-txt</code> module. This module must be loaded prior
  any operation. Multiple calls to the module initialization routine
  are harmless. The interpreter method <code>module</code> loads a
  specific module by name. When the module has been loaded, the
  object are available in the <code>afnix:txt</code> nameset.
  </p>

  <example>
  interp:library "afnix-txt"
  </example>

  <!-- scanning capabilities -->
  <section>
    <title>Scanning concepts</title>

    <p>
    Text scanning is the ability to extract lexical elements or
    <em>lexemes</em> from a stream. A scanner or lexical analyzer is
    the principal object used to perform this task. A scanner is
    created by adding special object that acts as a pattern
    matcher. When a pattern is matched, a special object called a
    <em>lexeme</em> is returned. 
    </p>
    
    <!-- pattern object -->
    <subsect>
      <title>Pattern object</title>

      <p>
      A <code>Pattern</code> object is a special object that acts as
      model for the string to match. There are several ways to build a
      pattern. The simplest way to build it is with a regular
      expression. Another type of pattern is a balanced pattern. In
      its first form, a pattern object can be created with a regular
      expression object.
      </p>

      <example>
      # load the text module
      interp:library "afnix-txt"

      # create a pattern object
      const pat (afnix:txt:Pattern "$d+")
      </example>

      <p>
      In this example, the pattern object is built to detect integer
      objects.
      </p>

      <example>
      pat:check "123" # true
      pat:match "123" # 123
      </example>

      <p>
      The <code>check</code> method return true if the input string
      matches the pattern. The <code>match</code> method returns the
      string that matches the pattern. Since the pattern object can
      also operates with stream object, the <code>match</code> method
      is appropriate to match a particular string. The pattern object
      is, as usual, available with the appropriate predicate.
      </p>

      <example>
      afnix:txt:pattern-p pat # true
      </example>

      <p>
      Another form of pattern object is the balanced pattern. A
      balanced pattern is determined by a starting string and an
      ending string. There are two types of balanced pattern. One is a
      single balanced pattern and the other one is the recursive
      balanced pattern. The single balanced pattern is appropriate for
      those lexical element that are defined by a character. For
      example, the classical C-string is a single balanced pattern
      with the double quote character.
      </p>

      <example>
      # create a balanced pattern
      const pat (afnix:txt:Pattern "ELEMENT" "&lt;" "&gt;")
      pat:check "&lt;xml&gt;" # true
      pat:match "&lt;xml&gt;" # xml
      </example>

      <p>
      In the case of the C-string, the pattern might be more
      appropriately defined with an additional <em>escape
      character</em>. Such character is used by the pattern matcher to
      grab characters that might be part of the pattern definition.
      </p>

      <example>
      # create a balanced pattern
      const pat (afnix:txt:Pattern "STRING" "&quot;" '\\')
      pat:check "&quot;hello&quot;" # true
      pat:match "&quot;hello&quot;" # "hello"
      </example>

      <p>
      In this form, a balanced pattern with an escape character is
      created. The same string is used for both the starting and
      ending string. Another constructor that takes two strings can be
      used if the starting and ending strings are different.

      The last pattern form is the balanced recursive form. In this
      form, a starting and ending string are used to delimit the
      pattern. However, in this mode, a recursive use of the starting
      and ending strings is allowed. In order to have an exact match,
      the number of starting string must equal the number of ending
      string. For example, the C-comment pattern can be viewed as
      recursive balanced pattern.
      </p>

      <example>
      # create a c-comment pattern
      const pat (afnix:txt:Pattern "STRING" "/*" "*/" )
      </example>
    </subsect>

    <!-- lexeme object -->
    <subsect>
      <title>Lexeme object</title>

      <p>
      The <code>Lexeme</code> object is the object built by a scanner
      that contains the matched string. A lexeme is therefore a tagged
      string. Additionally, a lexeme can carry additional information
      like a source name and index.
      </p>

      <example>
      # create an empty lexeme
      const lexm (afnix:txt:Lexeme)
      afnix:txt:lexeme-p lexm # true
      </example>

      <p>
      The default lexeme is created with any value. A value can be set
      with the <code>set-value</code> method and retrieved with the
      <code>get-value</code> methods.
      </p>

      <example>
      lexm:set-value "hello"
      lexm:get-value # hello
      </example>

      <p>
      Similar are the <code>set-tag</code> and <code>get-tag</code>
      methods which operate with an integer. The source name and index
      are defined as well with the same methods.
      </p>

      <example>
      # check for the source
      lexm:set-source "world"
      lexm:get-source # world

      # check for the source index
      lexm:set-index 2000
      lexm:get-index # 2000
      </example>
    </subsect>
  </section>

  <!-- text scanning -->
  <section>
    <title>Text scanning</title>

    <p>
    Text scanning is the ability to extract lexical elements or
    lexemes from an input stream. Generally, the lexemes are the
    results of a matching operation which is defined by a pattern
    object. As a result, the definition of a scanner object is the
    object itself plus one or several pattern object.
    </p>

    <!-- scanner construction -->
    <subsect>
      <title>Scanner construction</title>

      <p>
      By default, a scanner is created without pattern objects. The
      <code>length</code> method returns the number of pattern
      objects. As usual, a predicate is associated with the scanner
      object.
      </p>

      <example>
      # the default scanner
      const  scan (afnix:txt:Scanner)
      afnix:txt:scanner-p scan # true

      # the length method
      scan:length # 0
      </example>

      <p>
      The scanner construction proceeds by adding pattern
      objects. Each pattern can be created independently, and later
      added to the scanner. For example, a scanner that reads real,
      integer and string can be defined as follow:
      </p>

      <example>
      # create the scanner pattern
      const REAL    (afnix:txt:Pattern "REAL"    [$d+.$d*])
      const STRING  (afnix:txt:Pattern "STRING"  "&quot;" '\\')
      const INTEGER (afnix:txt:Pattern "INTEGER" [$d+|"0x"$x+])
      # add the pattern to the scanner
      scanner:add INTEGER REAL STRING
      </example>

      <p>
      The order of pattern integration defines the priority at which a
      token is recognized. The symbol name for each pattern is
      optional since the functional programming permits the creation
      of patterns directly. This writing style makes the scanner
      definition easier to read.
      </p>
    </subsect>

    <!-- unsing the scanner -->
    <subsect>
      <title>Using the scanner</title>

      <p>
      Once constructed, the scanner can be used <em>as is</em>. A
      stream is generally the best way to operate. If the scanner
      reaches the end-of-stream or cannot recognize a lexeme, the nil
      object is returned. With a loop, it is easy to get all lexemes.
      </p>

      <example>
      while (trans valid (is:valid-p)) {
        # try to get the lexeme
	trans lexm (scanner:scan is)
	# check for nil lexeme and print the value
	if (not (nil-p lexm)) (println (lexm:get-value))
	# update the valid flag
	valid:= (and (is:valid-p) (not (nil-p lexm)))
      }
      </example>

      <p>
      In this loop, it is necessary first to check for the end of the
      stream. This is done with the help of the special loop construct
      that initialize the <code>valid</code> symbol. As soon as the
      the lexeme is built, it can be used. The lexeme holds the value
      as well as it tag.
      </p>
    </subsect>
  </section>
</chapter>

