\chapter{Special functions in genr}
\label{chap-genr}

\section{Introduction}
\label{genr-intro}

The \verb+genr+ command provides a flexible means of defining new
variables.  It is documented in the \GCR.  This chapter offers a more
expansive discussion of some of the special functions available via
\verb+genr+ and some of the finer points of the command.
    

\section{Time-series filters}
\label{genr-filter}

One sort of specialized function in \verb+genr+ is the time-series
filter.  Two such filters are currently available, the
Hodrick--Prescott filter and the Baxter--King bandpass filter. These
are accessed using \verb+hpfilt()+ and \verb+bkfilt()+ respectively:
in each case the function takes one argument, the name of the variable
to be processed.
    

\subsection{The Hodrick--Prescott filter}
\label{hodrick-prescott}

A time series $y_t$ may be decomposed into a trend or growth
component $g_t$ and a cyclical component $c_t$.  
%
\[
y_t = g_t + c_t, \quad t = 1,2,\dots,T
\]
%
The Hodrick--Prescott filter effects such a decomposition by
minimizing the following:
%
\[
    \sum_{t = 1}^T {(y_t - g_t )^2 } + \lambda \sum_{t = 2}^{T -
      1} \left((g_{t+1} - g_t) - (g_t - g_{t - 1} )\right)^2 .
\]
%
The first term above is the sum of squared cyclical components $c_t =
y_t - g_t$. The second term is a multiple $\lambda$ of the sum of
squares of the trend component's second differences. This
second term penalizes variations in the growth rate of the trend
component: the larger the value of $\lambda$, the higher is the
penalty and hence the smoother the trend series.

Note that the \cmd{hpfilt} function in \app{gretl} produces the
cyclical component, $c_t$, of the original series.  If you want the
smoothed trend you can subtract the cycle from the original:

\begin{code}
genr ct = hpfilt(yt)
genr gt = yt - ct
\end{code}

Hodrick and Prescott (1997) suggest that a value of $\lambda = 1600$
is reasonable for quarterly data.  The default value in \app{gretl} is
100 times the square of the data frequency (which, of course, yields
1600 for quarterly data).  The value can be adjusted using the
\cmd{set} command, with a parameter of \cmd{hp\_lambda}.  For example,
\cmd{set hp\_lambda 1200}.


\subsection{The Baxter and King filter}
\label{baxter-king}

Consider the spectral representation of a time series $y_t$:
%       
\[ y_t = \int_{-\pi}^{\pi} e^{i\omega} \mathrm{d} Z(\omega) \]
%
To extract the component of $y_t$ that lies between the frequencies
$\underline{\omega}$ and $\overline{\omega}$ one could apply a
bandpass filter:
%       
\[ c^*_t = \int_{-\pi}^{\pi} F^*(\omega) e^{i\omega} \mathrm{d}
Z(\omega) \]
%
where $F^*(\omega) = 1$ for $\underline{\omega} < |\omega| <
\overline{\omega}$ and 0 elsewhere. This would imply, in the time
domain, applying to the series a filter with an infinite number of
coefficients, which is undesirable. The Baxter and King bandpass
filter applies to $y_t$ a finite polynomial in the lag
operator $A(L)$:
%       
\[ c_t = A(L) y_t \]
%
where $A$($L$) is defined as
%       
\[ A(L) = \sum_{i=-k}^{k} a_i L^i \]

The coefficients $a_i$ are chosen such that $F(\omega)
= A(e^{i\omega})A(e^{-i\omega})$ is the best approximation to
$F^*(\omega)$ for a given $k$. Clearly, the higher $k$ the better the
approximation is, but since $2k$ observations have to be discarded, a
compromise is usually sought. Moreover, the filter has also other
appealing theoretical properties, among which the property that $A(1)
= 0$, so a series with a single unit root is made stationary by
application of the filter.

In practice, the filter is normally used with monthly or quarterly
data to extract the ``business cycle'' component, namely the component
between 6 and 36 quarters. Usual choices for $k$ are 8 or 12 (maybe
higher for monthly series).  The default values for the frequency
bounds are 8 and 32, and the default value for the approximation
order, $k$, is 8. You can adjust these values using the \cmd{set}
command. The keyword for setting the frequency limits is
\verb+bkbp_limits+ and the keyword for $k$ is \verb+bkbp_k+.  Thus for
example if you were using monthly data and wanted to adjust the
frequency bounds to 18 and 96, and $k$ to 24, you could do

\begin{code}
    set bkbp_limits 18 96
    set bkbp_k 24
\end{code}

These values would then remain in force for calls to the \verb+bkfilt+
function until changed by a further use of \verb+set+.
      

\section{Resampling and bootstrapping}
\label{genr-resample}

Another specialized function is the resampling, with replacement, of a
series.  Given an original data series \varname{x}, the command
\begin{code}
    genr xr = resample(x)
\end{code}
creates a new series each of whose elements is drawn at random from
the elements of \varname{x}.  If the original series has 100
observations, each element of \varname{x} is selected with probability
$1/100$ at each drawing.  Thus the effect is to ``shuffle'' the
elements of \varname{x}, with the twist that each element of
\varname{x} may appear more than once, or not at all, in \varname{xr}.

The primary use of this function is in the construction of bootstrap
confidence intervals or p-values.  Here is a simple example.  Suppose
we estimate a simple regression of $y$ on $x$ via OLS and find that
the slope coefficient has a reported $t$-ratio of 2.5 with 40 degrees
of freedom.  The two-tailed p-value for the null hypothesis that the
slope parameter equals zero is then 0.0166, using the $t(40)$
distribution.  Depending on the context, however, we may doubt whether
the ratio of coefficient to standard error truly follows the $t(40)$
distribution.  In that case we could derive a bootstrap p-value as
shown in Example~\ref{resampling-loop}.  

Under the null hypothesis that the slope with respect to $x$ is zero,
$y$ is simply equal to its mean plus an error term.  We simulate $y$
by resampling the residuals from the initial OLS and re-estimate the
model.  We repeat this procedure a large number of times, and record
the number of cases where the absolute value of the $t$-ratio is
greater than 2.5: the proportion of such cases is our bootstrap
p-value.  For a good discussion of simulation-based tests and
bootstrapping, see Davidson and MacKinnon (2004, chapter 4).

\begin{script}[htbp]
  \caption{Calculation of bootstrap p-value}
  \label{resampling-loop}
\begin{code}
    ols y 0 x
    # save the residuals
    genr ui = $uhat
    scalar ybar = mean(y)
    # number of replications for bootstrap
    scalar replics = 10000
    scalar tcount = 0
    series ysim = 0
    loop replics --quiet
      # generate simulated y by resampling
      ysim = ybar + resample(ui)
      ols ysim 0 x
      scalar tsim = abs($coeff(x) / $stderr(x))
      tcount += (tsim > 2.5)
    endloop      
    printf "proportion of cases with |t| > 2.5 = %g\n", \
       tcount / replics
\end{code}
\end{script}
    

\section{Handling missing values}
\label{genr-missing}

Four special functions are available for the handling of missing
values.  The boolean function \verb+missing()+ takes the name of a
variable as its single argument; it returns a series with value 1 for
each observation at which the given variable has a missing value, and
value 0 otherwise (that is, if the given variable has a valid value at
that observation).  The function \verb+ok()+ is complementary to
\verb+missing+; it is just a shorthand for \verb+!missing+ (where
\verb+!+ is the boolean NOT operator).  For example, one can count the
missing values for variable \verb+x+ using

\begin{code}
    genr nmiss_x = sum(missing(x))
\end{code}

The function \verb+zeromiss()+, which again takes a single series as
its argument, returns a series where all zero values are set to the
missing code.  This should be used with caution --- one does not want
to confuse missing values and zeros --- but it can be useful in some
contexts.  For example, one can determine the first valid observation
for a variable \verb+x+ using

\begin{code}
    genr time
    genr x0 = min(zeromiss(time * ok(x)))
\end{code}

The function \verb+misszero()+ does the opposite of \verb+zeromiss+,
that is, it converts all missing values to zero.  

It may be worth commenting on the propagation of missing values within
\verb+genr+ formulae.  The general rule is that in arithmetical
operations involving two variables, if either of the variables has a
missing value at observation $t$ then the resulting series will also
have a missing value at $t$.  The one exception to this rule is
multiplication by zero: zero times a missing value produces zero
(since this is mathematically valid regardless of the unknown value).
    

\section{Retrieving internal variables}
\label{genr-internal}

The \verb+genr+ command provides a means of retrieving various values
calculated by the program in the course of estimating models or
testing hypotheses.  The variables that can be retrieved in this way
are listed in the \GCR; here we say a bit more about the special
variables \verb+$test+ and \verb+$pvalue+.

These variables hold, respectively, the value of the last test
statistic calculated using an explicit testing command and the p-value
for that test statistic.  If no such test has been performed at the
time when these variables are referenced, they will produce the
missing value code.  The ``explicit testing commands'' that work in
this way are as follows: \cmd{add} (joint test for the significance of
variables added to a model); \cmd{adf} (Augmented Dickey--Fuller test,
see below); \cmd{arch} (test for ARCH); \cmd{chow} (Chow test for a
structural break); \cmd{coeffsum} (test for the sum of specified
coefficients); \cmd{cusum} (the Harvey--Collier $t$-statistic);
\cmd{kpss} (KPSS stationarity test, no p-value available);
\cmd{lmtest} (see below); \cmd{meantest} (test for difference of
means); \cmd{omit} (joint test for the significance of variables
omitted from a model); \cmd{reset} (Ramsey's RESET); \cmd{restrict}
(general linear restriction); \cmd{runs} (runs test for randomness);
\cmd{testuhat} (test for normality of residual); and \cmd{vartest}
(test for difference of variances). In most cases both a \verb+$test+
and a \verb+$pvalue+ are stored; the exception is the KPSS test, for
which a p-value is not currently available.
    
An important point to notice about this mechanism is that the internal
variables \verb+$test+ and \verb+$pvalue+ are over-written each time
one of the tests listed above is performed.  If you want to reference
these values, you must do so at the correct point in the sequence of
\app{gretl} commands.  

A related point is that some of the test commands generate, by
default, more than one test statistic and p-value; in these cases only
the last values are stored. To get proper control over the retrieval
of values via \verb+$test+ and \verb+$pvalue+ you should formulate the
test command in such a way that the result is unambiguous.  This
comment applies in particular to the \verb+adf+ and \verb+lmtest+
commands.

\begin{itemize}
\item By default, the \cmd{adf} command generates three variants of
  the Dickey--Fuller test: one based on a regression including a
  constant, one using a constant and linear trend, and one using a
  constant and a quadratic trend.  When you wish to reference
  \verb+$test+ or \verb+$pvalue+ in connection with this command, you
  can control the variant that is recorded by using one of the flags
  \verb+--nc+, \verb+--c+, \verb+--ct+ or \verb+--ctt+ with
  \verb+adf+.
\item By default, the \cmd{lmtest} command (which must follow an OLS
  regression) performs several diagnostic tests on the regression in
  question.  To control what is recorded in \verb+$test+ and
  \verb+$pvalue+ you should limit the test using one of the flags
  \verb+--logs+, \verb+--autocorr+, \verb+--squares+ or
  \verb+--white+.
\end{itemize}

As an aid in working with values retrieved using \verb+$test+ and
\verb+$pvalue+, the nature of the test to which these values relate is
written into the descriptive label for the generated variable.  You
can read the label for the variable using the \cmd{label} command
(with just one argument, the name of the variable), to check that you
have retrieved the right value.  The following interactive session
illustrates this point.

\begin{code}
    ? adf 4 x1 --c

    Augmented Dickey-Fuller tests, order 4, for x1
    sample size 59
    unit-root null hypothesis: a = 1

      test with constant
      model: (1 - L)y = b0 + (a-1)*y(-1) + ... + e
      estimated value of (a - 1): -0.216889
      test statistic: t = -1.83491
      asymptotic p-value 0.3638

    P-values based on MacKinnon (JAE, 1996)
    ? genr pv = $pvalue
    Generated scalar pv (ID 13) = 0.363844
    ? label pv    
      pv=Dickey-Fuller pvalue (scalar)
\end{code}



%%% Local Variables: 
%%% mode: latex
%%% TeX-master: "gretl-guide"
%%% End: 

