# Data file details

## Basic native format

In gretl's native data format, a data set is stored in XML (extensible mark-up language). Data files correspond to the simple DTD (document type definition) given in `gretldata.dtd`, which is supplied with the gretl distribution and is installed in the system data directory (e.g. `/usr/share/gretl/data` on Linux.) Data files may be plain text or gzipped. They contain the actual data values plus additional information such as the names and descriptions of variables, the frequency of the data, and so on.

Most users will probably not have need to read or write such files other than via gretl itself, but if you want to manipulate them using other software tools you should examine the DTD and also take a look at a few of the supplied practice data files: `data4-1.gdt` gives a simple example; `data4-10.gdt` is an example where observation labels are included.

## Traditional ESL format

For backward compatibility, gretl can also handle data files in the traditional format inherited from Ramanathan's ESL program. In this format (which was the default in gretl prior to version 0.98) a data set is represented by two files. One contains the actual data and the other information on how the data should be read. To be more specific:

1. Actual data: A rectangular matrix of white-space separated numbers. Each column represents a variable, each row an observation on each of the variables (spreadsheet style). Data columns can be separated by spaces or tabs. The filename should have the suffix `.gdt`. By default the data file is ASCII (plain text). Optionally it can be gzip-compressed to save disk space. You can insert comments into a data file: if a line begins with the hash mark (`#`) the entire line is ignored. This is consistent with gnuplot and octave data files.

2. Header: The data file must be accompanied by a header file which has the same basename as the data file plus the suffix `.hdr`. This file contains, in order:

   - (Optional) comments on the data, set off by the opening string `(*` and the closing string `*)`, each of these strings to occur on lines by themselves.

   - (Required) list of white-space separated names of the variables in the data file. Names are limited to 8 characters, must start with a letter, and are limited to alphanumeric characters plus the underscore. The list may continue over more than one line; it is terminated with a semicolon, `;`.

   - (Required) observations line of the form `1 1 85`. The first element gives the data frequency (1 for undated or annual data, 4 for quarterly, 12 for monthly). The second and third elements give the starting and ending observations. Generally these will be 1 and the number of observations respectively, for undated data. For time-series data one can use dates of the form `1959.1` (quarterly, one digit after the point) or `1967.03` (monthly, two digits after the point). See for special use of this line in the case of panel data.

   - The keyword `BYOBS`.

Here is an example of a well-formed data header file.

    	  (*
    	  DATA9-6:
    	  Data on log(money), log(income) and interest rate from US.
    	  Source: Stock and Watson (1993) Econometrica
    	  (unsmoothed data) Period is 1900-1989 (annual data).
    	  Data compiled by Graham Elliott.
    	  *)
    	  lmoney lincome intrate ;
    	  1 1900 1989 BYOBS

The corresponding data file contains three columns of data, each having 90 entries.

Three further features of the traditional data format may be noted.

1. If the `BYOBS` keyword is replaced by `BYVAR`, and followed by the keyword `BINARY`, this indicates that the corresponding data file is in binary format. Such data files can be written from gretlcli using the `store` command with the `-s` flag (single precision) or the `-o` flag (double precision).

2. If `BYOBS` is followed by the keyword `MARKERS`, gretl expects a data file in which the first column contains strings (8 characters maximum) used to identify the observations. This may be handy in the case of cross-sectional data where the units of observation are identifiable: countries, states, cities or whatever. It can also be useful for irregular time series data, such as daily stock price data where some days are not trading days — in this case the observations can be marked with a date string such as `10/01/98`. (Remember the 8-character maximum.) Note that `BINARY` and `MARKERS` are mutually exclusive flags. Also note that the markers are not considered to be a variable: this column does not have a corresponding entry in the list of variable names in the header file.

3. If a file with the same base name as the data file and header files, but with the suffix `.lbl`, is found, it is read to fill out the descriptive labels for the data series. The format of the label file is simple: each line contains the name of one variable (as found in the header file), followed by one or more spaces, followed by the descriptive label. Here is an example: `price New car price index, 1982 base year`

If you want to save data in traditional format, use the `-t` flag with the `store` command, either in the command-line program or in the console window of the GUI program.

## Binary database details

A gretl database consists of two parts: an ASCII index file (with filename suffix `.idx`) containing information on the series, and a binary file (suffix `.bin`) containing the actual data. Two examples of the format for an entry in the `idx` file are shown below:

          G0M910  Composite index of 11 leading indicators (1987=100)
          M 1948.01 - 1995.11  n = 575
          currbal Balance of Payments: Balance on Current Account; SA
          Q 1960.1 - 1999.4 n = 160

The first field is the series name. The second is a description of the series (maximum 128 characters). On the second line the first field is a frequency code: `M` for monthly, `Q` for quarterly, `A` for annual, `B` for business-daily (daily with five days per week) and `D` for daily (seven days per week). No other frequencies are accepted at present. Then comes the starting date (N.B. with two digits following the point for monthly data, one for quarterly data, none for annual), a space, a hyphen, another space, the ending date, the string `n =` and the integer number of observations. In the case of daily data the starting and ending dates should be given in the form `YYYY/MM/DD`. This format must be respected exactly.

Optionally, the first line of the index file may contain a short comment (up to 64 characters) on the source and nature of the data, following a hash mark. For example:

          # Federal Reserve Board (interest rates)

The corresponding binary database file holds the data values, represented as floats, that is, single-precision floating-point numbers, typically taking four bytes apiece. The numbers are packed by variable, so that the first n numbers are the observations of variable 1, the next m the observations on variable 2, and so on.

# Technical notes

Gretl is written in the C programming language. I have abided as far as possible by the ISO/ANSI C Standard (C89), although the graphical user interface and some other components necessarily make use of platform-specific extensions.

gretl was developed under Linux. The shared library and command-line client should compile and run on any platform that (a) supports ISO/ANSI C, (b) has the following libraries installed: zlib (data compression), libxml (XML manipulation), and LAPACK (linear algebra support). The homepage for zlib can be found at info-zip.org; libxml is at xmlsoft.org; LAPACK is at netlib.org. If the GNU readline library is found on the host system this will be used for gretcli, providing a much enhanced editable command line. See the readline homepage.

The graphical client program should compile and run on any system that, in addition to the above requirements, offers GTK version 1.2.3 or higher (see gtk.org). As of this writing there are two main variants of the GTK libraries: the 1.2 series and the 2.0 series which was launched in summer 2002. These variants are mutually incompatible. gretl can be built using either one — the source code package includes two sub-directories, `gui` for GTK 1.2 and `gui2` for GTK 2.0. I recommend use of GTK 2.0 if it is available, since it offers many enhancements over GTK 1.2.

gretl calls gnuplot for graphing. You can find gnuplot at gnuplot.info. As of this writing the most recent official release is 4.0 (of April, 2004). The MS Windows version of gretl comes with a Windows version gnuplot 4.0; the gretl website also offers an rpm of gnuplot 3.8j0 for x86 Linux systems.

Some features of gretl make use of Adrian Feguin's gtkextra library. You can find gtkextra at gtkextra.sourceforge.net. The relevant parts of this package are included (in slightly modified form) with the gretl source distribution.

A binary version of the program is available for the Microsoft Windows platform (32-bit version, i.e. Windows 95 or higher). This version was cross-compiled under Linux using mingw (the GNU C compiler, gcc, ported for use with win32) and linked against the Microsoft C library, `msvcrt.dll`. It uses Tor Lillqvist's port of GTK 2.0 to win32. The (free, open-source) Windows installer program is courtesy of Jordan Russell (jrsoftware.org).

I'm hopeful that some users with coding skills may consider gretl sufficiently interesting to be worth improving and extending. The documentation of the libgretl API is by no means complete, but you can find some details by following the link Libgretl API docs on the gretl homepage.

# Numerical accuracy

gretl uses double-precision arithmetic throughout — except for the multiple-precision plugin invoked by the menu item Model/High precision OLS which represents floating-point values using a number of bits given by the environment variable `GRETL_MP_BITS` (default value 256). The normal equations of Least Squares are by default solved via Cholesky decomposition, which is accurate enough for most purposes (with the option of using QR decomposition instead). The program has been tested rather thoroughly on the statistical reference datasets provided by NIST (the U.S. National Institute of Standards and Technology) and a full account of the results may be found on the gretl website (follow the link Numerical accuracy).

In October 2002 I had a useful exchange with Giovanni Baiocchi and Walter Distaso, who were writing a review of gretl for the Journal of Applied Econometrics, and James MacKinnon, software review editor for the journal. This review has since been published; see Baiocchi and Distaso (2003). and I am grateful to Baiocchi and Distaso for their careful examination of the program, which prompted the following modifications.

1. The reviewers pointed out that there was a bug in gretl's p-value finder, whereby the program printed the complement of the correct probability for negative values of z. This was fixed in version 0.998 of the program (released July 9, 2002).

2. They also noted that the p-value finder produced inaccurate results for extreme values of x (e.g. values of around 8 to 10 in the t distribution with 100 degrees of freedom). This too was fixed in gretl version 0.998, with a switch to more accurate probability distribution code.

3. The reviewers noted a flaw in the presentation of regression coefficients in gretl, whereby some coefficients could be printed to an unacceptably small number of significant figures. This was fixed in version 0.999 (released August 25, 2002): now all the statistics associated with a regression are printed to 6 significant figures.

4. It transpired from the reviewer's tests that the numerical accuracy of gretl on MS Windows was less than on Linux, where I had done my testing. For example, on the Longley data — a well-known ill-conditioned dataset often used for testing econometrics programs — the Windows version of gretl was getting some coefficients wrong at the 7th digit while the same coefficients were correct on Linux. This anomaly was fixed in gretl version 1.0pre3 (released October 10, 2002).

The current version of gretl includes a plugin that runs the NIST linear regression test suite. You can find this under the Utilities menu in the main window. When you run this test, the introductory text explains the expected result. If you run this test and see anything other than the expected result, please send a bug report to `cottrell@wfu.edu`.

As mentioned above, all regression statistics are printed to 6 significant figures in the current version of gretl (except when the multiple-precision plugin is used, then results are given to 12 figures). If you want to examine a particular value more closely, first save it (see ) then print it using `print --ten` (see ). This will show the value to 10 digits.

# Advanced econometric analysis with free software

As mentioned in the main text, gretl offers a reasonably full selection of least-squares based estimators, plus a few additional estimators such as (binomial) logit and probit and Least Absolute Deviations. Advanced users may, however, find gretl's menu of statistical routines restrictive.

No doubt some advanced users will prefer to write their own statistical code in a fundamental computer language such as C, C++ or Fortran. Another option is to use a relatively high-level language that offers easy matrix manipulation and that already has numerous statistical routines built in, or available as add-on packages. If the latter option sounds attractive, and you are interested in using free, open source software, I would recommend taking a look at either GNU R (r-project.org) or GNU Octave (www.octave.org). These programs are very close to the commercial programs S and Matlab respectively.

Also as mentioned above, gretl offers the facility of exporting data in the formats of both Octave and R. In the case of Octave, the gretl data set is saved as a single matrix, `X`. You can pull the `X` matrix apart if you wish, once the data are loaded in Octave; see the Octave manual for details. As for R, the exported data file preserves any time series structure that is apparent to gretl. The series are saved as individual structures. The data should be brought into R using the `source()` command.

Of these two programs, R is perhaps more likely to be of immediate interest to econometricians since it offers more in the way of statistical routines (e.g. generalized linear models, maximum likelihood estimation, time series methods). I have therefore supplied gretl with a convenience function for moving data quickly into R. Under gretl's Session menu, you will find the entry Start GNU R. This writes out an R version of the current gretl data set (`Rdata.tmp`, in the user's gretl directory), and sources it into a new R session. A few details on this follow.

First, the data are brought into R by writing a temporary version of `.Rprofile` in the current working directory. (If such a file exists it is referenced by R at startup.) In case you already have a personal `.Rprofile` in place, the original file is temporarily moved to `.Rprofile.gretltmp`, and on exit from gretl it is restored. (If anyone can suggest a cleaner way of doing this I'd be happy to hear of it.)

Second, the particular way R is invoked depends on the internal gretl variable `Rcommand`, whose value may be set under the File, Preferences menu. The default command is `RGui.exe` under MS Windows. Under X it is either `R --gui=gnome` if an installation of the Gnome desktop (gnome.org) was detected at compile time, or `xterm -e R` if Gnome was not found. Please note that at most three space-separated elements in this command string will be processed; any extra elements are ignored.

# Listing of URLs

Below is a listing of the full URLs of websites mentioned in the text.

Census Bureau, Data Extraction Service

  `http://www.census.gov/ftp/pub/DES/www/welcome.html`

Estima (RATS)

  `http://www.estima.com/`

Gnome desktop homepage

  `http://www.gnome.org/`

GNU Multiple Precision (GMP) library

  `http://swox.com/gmp/`

GNU Octave homepage

  `http://www.octave.org/`

GNU R homepage

  `http://www.r-project.org/`

GNU R manual

  `http://cran.r-project.org/doc/manuals/R-intro.pdf`

Gnuplot homepage

  `http://www.gnuplot.info/`

Gnuplot manual

  `http://ricardo.ecn.wfu.edu/gnuplot.html`

Gretl data page

  `http://gretl.sourceforge.net/gretl_data.html`

Gretl homepage

  `http://gretl.sourceforge.net/`

GTK+ homepage

  `http://www.gtk.org/`

GTK+ port for win32

  `http://www.gimp.org/~tml/gimp/win32/`

Gtkextra homepage

  `http://gtkextra.sourceforge.net/`

InfoZip homepage

  `http://www.info-zip.org/pub/infozip/zlib/`

JRSoftware

  `http://www.jrsoftware.org/`

Mingw (gcc for win32) homepage

  `http://www.mingw.org/`

Minpack

  `http://www.netlib.org/minpack/`

Penn World Table

  `http://pwt.econ.upenn.edu/`

Readline homepage

  `http://cnswww.cns.cwru.edu/~chet/readline/rltop.html`

Readline manual

  `http://cnswww.cns.cwru.edu/~chet/readline/readline.html`

Xmlsoft homepage

  `http://xmlsoft.org/`