# Text scanning

This chapter is dedicated to the text scanning, a subpart of the standard text processing module. Text scanning refers to the ability to constructs lexemes from a text stream. The lexemes are built from the stream with the help of regular expressions. Eventually some special operations like text transliteration is also possible.

All text processing objects are located in the `afnix-txt` module. This module must be loaded prior any operation. Multiple calls to the module initialization routine are harmless. The interpreter method `module` loads a specific module by name. When the module has been loaded, the object are available in the `afnix:txt` nameset.

interp:library "afnix-txt"

## Scanning concepts

Text scanning is the ability to extract lexical elements or lexemes from a stream. A scanner or lexical analyzer is the principal object used to perform this task. A scanner is created by adding special object that acts as a pattern matcher. When a pattern is matched, a special object called a lexeme is returned.

### Pattern object

A `Pattern` object is a special object that acts as model for the string to match. There are several ways to build a pattern. The simplest way to build it is with a regular expression. Another type of pattern is a balanced pattern. In its first form, a pattern object can be created with a regular expression object.

# load the text module interp:library "afnix-txt"

# create a pattern object const pat (afnix:txt:Pattern "$d+")

In this example, the pattern object is built to detect integer objects.

pat:check "123" # true pat:match "123" # 123

The `check` method return true if the input string matches the pattern. The `match` method returns the string that matches the pattern. Since the pattern object can also operates with stream object, the `match` method is appropriate to match a particular string. The pattern object is, as usual, available with the appropriate predicate.

afnix:txt:pattern-p pat # true

Another form of pattern object is the balanced pattern. A balanced pattern is determined by a starting string and an ending string. There are two types of balanced pattern. One is a single balanced pattern and the other one is the recursive balanced pattern. The single balanced pattern is appropriate for those lexical element that are defined by a character. For example, the classical C-string is a single balanced pattern with the double quote character.

# create a balanced pattern const pat (afnix:txt:Pattern "ELEMENT" "<" ">") pat:check "<xml>" # true pat:match "<xml>" # xml

In the case of the C-string, the pattern might be more appropriately defined with an additional escape character. Such character is used by the pattern matcher to grab characters that might be part of the pattern definition.

# create a balanced pattern const pat (afnix:txt:Pattern "STRING" """ '\\') pat:check ""hello"" # true pat:match ""hello"" # "hello"

In this form, a balanced pattern with an escape character is created. The same string is used for both the starting and ending string. Another constructor that takes two strings can be used if the starting and ending strings are different.

The last pattern form is the balanced recursive form. In this form, a starting and ending string are used to delimit the pattern. However, in this mode, a recursive use of the starting and ending strings is allowed. In order to have an exact match, the number of starting string must equal the number of ending string. For example, the C-comment pattern can be viewed as recursive balanced pattern.

# create a c-comment pattern const pat (afnix:txt:Pattern "STRING" "/*" "*/" )

### Lexeme object

The `Lexeme` object is the object built by a scanner that contains the matched string. A lexeme is therefore a tagged string. Additionally, a lexeme can carry additional information like a source name and index.

# create an empty lexeme const lexm (afnix:txt:Lexeme) afnix:txt:lexeme-p lexm # true

The default lexeme is created with any value. A value can be set with the `set-value` method and retrieved with the `get-value` methods.

lexm:set-value "hello" lexm:get-value # hello

Similar are the `set-tag` and `get-tag` methods which operate with an integer. The source name and index are defined as well with the same methods.

# check for the source lexm:set-source "world" lexm:get-source # world

# check for the source index lexm:set-index 2000 lexm:get-index # 2000

## Text scanning

Text scanning is the ability to extract lexical elements or lexemes from an input stream. Generally, the lexemes are the results of a matching operation which is defined by a pattern object. As a result, the definition of a scanner object is the object itself plus one or several pattern object.

### Scanner construction

By default, a scanner is created without pattern objects. The `length` method returns the number of pattern objects. As usual, a predicate is associated with the scanner object.

# the default scanner const scan (afnix:txt:Scanner) afnix:txt:scanner-p scan # true

# the length method scan:length # 0

The scanner construction proceeds by adding pattern objects. Each pattern can be created independently, and later added to the scanner. For example, a scanner that reads real, integer and string can be defined as follow:

# create the scanner pattern const REAL (afnix:txt:Pattern "REAL" [$d+.$d*]) const STRING (afnix:txt:Pattern "STRING" """ '\\') const INTEGER (afnix:txt:Pattern "INTEGER" [$d+|"0x"$x+]) # add the pattern to the scanner scanner:add INTEGER REAL STRING

The order of pattern integration defines the priority at which a token is recognized. The symbol name for each pattern is optional since the functional programming permits the creation of patterns directly. This writing style makes the scanner definition easier to read.

### Using the scanner

Once constructed, the scanner can be used as is. A stream is generally the best way to operate. If the scanner reaches the end-of-stream or cannot recognize a lexeme, the nil object is returned. With a loop, it is easy to get all lexemes.

while (trans valid (is:valid-p)) { # try to get the lexeme trans lexm (scanner:scan is) # check for nil lexeme and print the value if (not (nil-p lexm)) (println (lexm:get-value)) # update the valid flag valid:= (and (is:valid-p) (not (nil-p lexm))) }

In this loop, it is necessary first to check for the end of the stream. This is done with the help of the special loop construct that initialize the `valid` symbol. As soon as the the lexeme is built, it can be used. The lexeme holds the value as well as it tag.