# Nonlinear least squares

## Introduction and examples

As of version 1.0.9, gretl supports nonlinear least squares (NLS) using a variant of the Levenberg–Marquandt algorithm. The user must supply a specification of the regression function; prior to giving this specification the parameters to be estimated must be declared and given initial values. Optionally, the user may supply analytical derivatives of the regression function with respect to each of the parameters. The tolerance (criterion for terminating the iterative estimation procedure) can be set using the `genr` command.

The syntax for specifying the function to be estimated is the same as for the `genr` command. Here are two examples, with accompanying derivatives.

### Consumption function from Greene

    	  nls C = alpha + beta * Y^gamma
    	  deriv alpha = 1
    	  deriv beta = Y^gamma
    	  deriv gamma = beta * Y^gamma * log(Y)
    	  end nls

### Nonlinear function from Russell Davidson

    	  nls y = alpha + beta * x1 + (1/beta) * x2
    	  deriv alpha = 1
    	  deriv beta = x1 - x2/(beta*beta)
    	  end nls

Note the command words `nls` (which introduces the regression function), `deriv` (which introduces the specification of a derivative), and `end nls`, which terminates the specification and calls for estimation. If the `--vcv` flag is appended to the last line the covariance matrix of the parameter estimates is printed.

## Initializing the parameters

The parameters of the regression function must be given initial values prior to the `nls` command. This can be done using the `genr` command (or, in the GUI program, via the menu item Define new variable). In some cases, where the nonlinear function is a generalization of (or a restricted form of) a linear model, it may be convenient to run an `ols` and initialize the parameters from the OLS coefficient estimates. In relation to the first example above, one might do:

          ols C 0 Y
          genr alpha = coeff(0)
          genr beta = coeff(Y)
          genr gamma = 1

And in relation to the second example one might do:

          ols y 0 x1 x2
          genr alpha = coeff(0)
          genr beta = coeff(x1)

## NLS dialog window

It is probably most convenient to compose the commands for NLS estimation in the form of a gretl script but you can also do so interactively, by selecting the item Nonlinear Least Squares under the Model menu. This opens a dialog box where you can type the function specification (possibly prefaced by `genr` lines to set the initial parameter values) and the derivatives, if available. An example of this is shown in . Note that in this context you do not have to supply the `nls` and `end nls` tags.

### NLS dialog box

Nonlinear least squares dialog

## Analytical and numerical derivatives

If you are able to figure out the derivatives of the regression function with respect to the parameters, it is advisable to supply those derivatives as shown in the examples above. If that is not possible, gretl will compute approximate numerical derivatives. The properties of the NLS algorithm may not be so good in this case (see ).

If analytical derivatives are supplied, they are checked for consistency with the given nonlinear function. If the derivatives are clearly incorrect estimation is aborted with an error message. If the derivatives are suspicious a warning message is issued but estimation proceeds. This warning may sometimes be triggered by incorrect derivatives, but it may also be triggered by a high degree of collinearity among the derivatives.

Note that you cannot mix analytical and numerical derivatives: you should supply expressions for all of the derivatives or none.

## Controlling termination

The NLS estimation procedure is an iterative process. Iteration is terminated when a convergence criterion is met or when a set maximum number of iterations is reached, whichever comes first. The maximum number of iterations is `100*(k+1)` when analytical derivatives are given and `200*(k+1)` when numerical derivatives are used, where `k` denotes the number of parameters being estimated. The convergence criterion is that the relative error in the sum of squares, and/or the relative error between the the coefficient vector and the solution, is estimated to be no larger than some small value. This small value is by default the machine precision to the power 3/4, but it can be set with the `genr` command using the special variable `toler`. For example

    genr toler = .0001

will relax the tolerance to 0.0001.

## Details on the code

The underlying engine for NLS estimation is based on the minpack suite of functions, available from netlib.org. Specifically, the following minpack functions are called:

`lmder` | Levenberg–Marquandt algorithm with analytical derivatives
`chkder` | Check the supplied analytical derivatives
`lmdif` | Levenberg–Marquandt algorithm with numerical derivatives
`fdjac2` | Compute final approximate Jacobian when using numerical derivatives
`dpmpar` | Determine the machine precision

On successful completion of the Levenberg–Marquandt iteration, a Gauss–Newton regression is used to calculate the covariance matrix for the parameter estimates. Since NLS results are asymptotic, there is room for debate over whether or not a correction for degrees of freedom should be applied when calculating the standard error of the regression (and the standard errors of the parameter estimates). For comparability with OLS, and in light of the reasoning given in Davidson and MacKinnon (1993), the estimates shown in gretl do use a degrees of freedom correction.

## Numerical accuracy

shows the results of running the gretl NLS procedure on the 27 Statistical Reference Datasets made available by the U.S. National Institute of Standards and Technology (NIST) for testing nonlinear regression software. For a discussion of gretl's accuracy in the estimation of linear models, see . For each dataset, two sets of starting values for the parameters are given in the test files, so the full test comprises 54 runs. Two full tests were performed, one using all analytical derivatives and one using all numerical approximations. In each case the default tolerance was used. The data shown in the table were gathered from a pre-release build of gretl version 1.0.9, compiled with gcc 3.3, linked against glibc 2.3.2, and run under Linux on an i686 PC (IBM ThinkPad A21m).

Out of the 54 runs, gretl failed to produce a solution in 4 cases when using analytical derivatives, and in 5 cases when using numeric approximation. Of the four failures in analytical derivatives mode, two were due to non-convergence of the Levenberg–Marquandt algorithm after the maximum number of iterations (on `MGH09` and `Bennett5`, both described by NIST as of Higher difficulty) and two were due to generation of range errors (out-of-bounds floating point values) when computing the Jacobian (on `BoxBOD` and `MGH17`, described as of Higher difficulty and Average difficulty respectively). The additional failure in numerical approximation mode was on `MGH10` (Higher difficulty, maximum number of iterations reached).

The table gives information on several aspects of the tests: the number of outright failures, the average number of iterations taken to produce a solution and two sorts of measure of the accuracy of the estimates for both the parameters and the standard errors of the parameters.

For each of the 54 runs in each mode, if the run produced a solution the parameter estimates obtained by gretl were compared with the NIST certified values. We define the minimum correct figures for a given run as the number of significant figures to which the least accurate gretl estimate agreed with the certified value, for that run. The table shows both the average and the worst case value of this variable across all the runs that produced a solution. The same information is shown for the estimated standard errors. For the standard errors, I excluded one outlier from the statistics shown in the table, namely `Lanczos1`. This is an odd case, using generated data with an almost-exact fit: the standard errors are 9 or 10 orders of magnitude smaller than the coefficients. In this instance gretl could reproduce the certified standard errors to only 3 figures (analytical derivatives) and 2 figures (numerical derivatives).

The second measure of accuracy shown is the percentage of cases, taking into account all parameters from all successful runs, in which the gretl estimate agreed with the certified value to at least the 6 significant figures which are printed by default in the gretl regression output.

### Nonlinear regression: the NIST tests

| Analytical derivatives | Numerical derivatives
--- | --- | ---
Failures in 54 tests | 4 | 5
Average iterations | 32 | 127
Avg. of min. correct figures, parameters | 8.120 | 6.980
Worst of min. correct figures, parameters | 4 | 3
Avg. of min. correct figures, standard errors | 8.000 | 5.673
Worst of min. correct figures, standard errors | 5 | 2
Percent correct to at least 6 figures, parameters | 96.5 | 91.9
Percent correct to at least 6 figures, standard errors | 97.7 | 77.3

Using analytical derivatives, the worst case values for both parameters and standard errors were improved to 6 correct figures on the test machine when the tolerance was tightened to 1.0e−14. Using numerical derivatives, the same tightening of the tolerance raised the worst values to 5 correct figures for the parameters and 3 figures for standard errors, at a cost of one additional failure of convergence.

Note the overall superiority of analytical derivatives: on average solutions to the test problems were obtained with substantially fewer iterations and the results were more accurate (most notably for the estimated standard errors). Note also that the six-digit results printed by gretl are not 100 percent reliable for difficult nonlinear problems (in particular when using numerical derivatives). Having registered this caveat, the percentage of cases where the results were good to six digits or better seems high enough to justify their printing in this form.