# Reusable components and solver driver API¶

This section highlights the major library components, ranging from basic building blocks necessary for any AMPL driver, to more advanced utilities. The choice of API for a driver may depend on Solver driver setups.

## Reading NL and writing SOL files¶

Any AMPL solver driver currenlty needs to input an NL file and report results in a SOL file. The details are taken care of in the recommended driver setup. However, a minimal driver setup might address the corresponding APIs directly, as described below.

### NL format¶

NL is a technical format for representing optimization problems in discrete or continuous variables. The format provides linear constraints, as well as non-linear expression trees. It is described in the technical report Writing .nl Files.

The NL format supports a wide range of problem types including but not limited to the following areas of optimization:

Mixed-integer quadratic programming with or without convex quadratic constraints

Mixed-integer nonlinear programming

Semidefinite programming problems with bilinear matrix inequalities

Complementarity problems (MPECs) in discrete or continuous variables

### NL file reader¶

MP provides a high-performance nl file reader which is up to 6x faster than the one provided by the traditional AMPL Solver Library (ASL).

This section describes the C++ API of an NL reader which is

Reusable: the reader can be used to process NL files in different ways and not limited to a single problem representation

High performance: fast mmap-based reader with SAX-like API and no dynamic memory allocations in the common case

Easy to use: clean, modern code base and simple API

Complete: supports all NL constructs including extensions implemented in AMPL Solver Library

Reliable: extensively and continuously tested on a variety of platforms

#### Easy-to-use functions¶

The `mp/nl.h`

header only contains declarations of
`mp::ReadNLFile()`

and `mp::ReadNLString()`

, and can be used to read the standard optimization problem
object of class `mp::Problem`

, for example:

```
#include "mp/nl.h"
#include "mp/problem.h"
mp::Problem p;
ReadNLFile("diet.nl", p);
```

#### Full NL-reader API¶

If you want to provide a custom NL handler, include `mp/nl-reader.h`

instead.
Class `mp::NLHandler`

can be customized for most efficient translation of NL format into
solver API using the ProblemBuilder concept.

Note that `mp/nl.h`

is a much smaller header than `mp/nl-reader.h`

so prefer
it unless you need access to the full NL reader API, described below.

`mp::ReadNLFile()`

,`mp::ReadNLString()`

read NL model`mp::NLHandler`

,`mp::NullNLHandler`

provide interface for a custom NL handler`mp::NLHeader`

stores problem information`mp::NameProvider`

reads variable and constraint names (see AMPL options`auxfiles`

,`(solver)_auxfiles`

), or provides generic names otherwise`mp::READ_BOUNDS_FIRST`

can be passed as a flag to`mp::ReadNLFile()`

`mp::MAX_AMPL_OPTIONS`

is the maximum number of options reserved for AMPL use in NL and SOL formats

### SOL file writer¶

Writing solution/results output is easiest as part of the general workflow, see Model manager.

A standalone .sol file writer could be implemented by parameterizing the
`mp::internal::AppSolutionHandlerImpl`

(or `mp::internal::SolutionWriterImpl`

)
templates by minimal implementations of the `mp::BasicSolver`

and
`mp::ProblemBuilder`

interfaces.

## Writing NL and reading SOL files¶

For modeling systems and applications using AMPL solvers, MP provides a library to write NL and read SOL files. The library is virtually zero-overhead: it does not store the model, nor does it require any intermediate objects to represent model information.

### NL Writer design¶

The NL writer library provides two approaches:

“Straightforward” API via

`NLModel`

and`NLSolver`

. This is currently limited to (MI)LP and (MI)QP models.Full NL API via callbacks into custom

`NLFeeder`

and`SOLHandler`

classes, optionally via`NLSolver`

.This approach allows a most efficient conversion of a user model’s internal representation to NL format by providing NL component writer callbacks (and solution reader callbacks to receive solutions.)

In turn, the library is flexible to use because various components of the user model are provided on the library’s request from a user-specialized

`NLFeeder`

object (for solution input, solution components are received by methods of a custom`SOLHandler`

class.)

### NL Writer API languages¶

See the NL Writer example folder and tests.

**C++ API**is provided by classes`NLSolver`

,`NLModel`

,`NLFeeder`

,`SOLHandler`

.**C API**is provided by structs`NLW2_NLSolver_C`

,`NLW2_NLModel_C`

,`NLW2_NLFeeder_C`

,`NLW2_SOLHandler_C`

.Currently only (MI)LP/(MI)QP (via

`NLW2_NLModel_C`

) or linear models (via`NLW2_NLFeeder_C`

,`NLW2_SOLHandler_C`

) are supported.

**Python API**only supports (MI)LP/(MI)QP (via`nlwpy.NLModel`

), see NLWPY example.Available on PyPI as the nlwpy package.

## Recommended driver logic¶

Using the mp::StdBackend and the derived classes is the recommended approach to building a new solver interface. They provide a convenient API for common solver driver actions, options and suffixes. The high-level application structure is suggested as follows:

BackendApp –> Custom Backend –> Solver.

In the recommended driver setup, the interaction of the Backend with the solver API is separated in two channels: model manipulation is delegated to Model manager. ModelManager addresses solver API via a separate modeling API wrapper:

Custom Backend –> Model manager –> … –> Flat model API –> Solver.

More details are given in Model/solution I/O and reformulations.

Thus, solver API is wrapped by two separate classes specializing in model manipulation vs. process logic. A reason for this design is maintainability and recompilation speed. Creating such a driver is described in the HowTo.

### BackendApp¶

`mp::BackendApp`

supports basic application functions, such as screen output
and interrupts handling. It calls a CustomBackend which should implement
the `mp::BasicBackend`

interface.

### The Backend classes¶

`mp::StdBackend`

and `mp::MIPBackend`

implement the `mp::BasicBackend`

interface and
standardize some common AMPL app behaviour, such as
solver messages and status reporting,
LP basis statuses, and other suffix I/O.
Their solver-specific subclasses can be customized for a particular solver.
They rely on the Model manager interface
for model and solution I/O.

As an example, if the driver should read and write simplex basis status suffixes, the derived Backend class can declare

```
ALLOW_STD_FEATURE( BASIS, true )
SolutionBasis GetBasis() override;
void SetBasis(SolutionBasis ) override;
```

and define the `GetBasis`

, `SetBasis`

methods.
See Implement standard & custom features
for further details.

### Solver, SolverImpl [deprecated]¶

Classes `mp::SolverApp`

, `mp::Solver`

and `mp::SolverImpl`

enable very basic
standard behaviour (e.g., multiobj, solution output). They are deprecated
in favor of the BackendApp/Backend classes and
can be discontinued in future.

## Model/solution I/O and reformulations¶

The tools presented in this section standardize model/solution I/O (currently relying on NL file input and SOL file output) and conversion for a particular solver.

### Overview¶

While the components can be theoretically used in isolation, for example just the Model manager, in the Recommended setup the model handling is implemented according to the following scheme:

- Model manager –>
Problem builder –> Problem converter / flattener –> Flat Converter –> Flat model API –> Solver.

To give an example, consider the following model:

```
var x >=0, <=7;
var y >=0, <=4, integer;
s.t. Con01: x + log(y) <= 5;
s.t. Con02: numberof 2 in (x, y) <= 1;
```

The nonlinear expressions `log`

and `numberof`

are received from AMPL
in expression trees which are input from
an NL file by a problem builder. At the next step,
problem flattener replaces nonlinear expressions
by auxiliary variables:

```
var t1 = log(y);
var t2 = numberof 2 in (x, y);
s.t. Con01_: x + t1 <= 5;
s.t. Con02_: t2 <= 1;
```

Then, the defining constraints of `t1`

and `t2`

are either passed to the solver
which accepts them via the model API, or become reformulated
into more simple entities by Flat Converter. If the solver
natively accepts a nonlinear constraint, it is possible to still apply automatic
reformulation via a solver option, for example *acc:log* for logarithm. Run the driver
with *-=* or *-c* for a list of natively accepted constraints and options.

An in-depth treatment of some automatic reformulations is given in [CLModernArch], [SOCTransform], [MOI], and [CP2MIP]. Customization for a new solver driver is sketched in Configure automatic model reformulations.

- CP2MIP
G. Belov, P. J. Stuckey, G. Tack, M. Wallace. Improved Linearization of Constraint Programming Models. In: Rueher, M. (eds) Principles and Practice of Constraint Programming. CP 2016. LNCS, vol 9892. Springer, Cham.

*https://doi.org/10.1007/978-3-319-44953-1_4*.- CLModernArch
J. J. Dekker. A Modern Architecture for Constraint Modelling Languages. PhD thesis. Monash University, 2021.

- SOCTransform
R. Fourer and J. Erickson. Detection and Transformation of Second-Order Cone Programming Problems in a General-Purpose Algebraic Modeling Language. Optimization Online, 2019.

- MOI
B. Legat, O. Dowson, J. D. Garcia, M. Lubin. MathOptInterface: A Data Structure for Mathematical Optimization Problems. INFORMS Journal on Computing 34 (2), 2021.

*https://doi.org/10.1287/ijoc.2021.1067*.

### Model manager¶

Class `mp::BasicModelManager`

standardizes the interface for
model input and results output. This interface is used by the
Backend classes.

Current suggested implementations rely on

`mp::ModelManagerWithProblemBuilder`

which uses NL file input and SOL file output as well as a model converter. The model converter should implement the`mp::BasicConverter`

interface and provide a Problem Builder.

### Problem builders¶

Basic Model/solution I/O and
model managers rely on a `mp::ProblemBuilder`

concept.

A custom builder can pass the NL model directly into the solver. A few examples are in nl-example.cc,

`mp::Problem`

, SCIP 8.0 NL file reader.Alternatively, standard classes

`mp::Problem`

and`mp::ColProblem`

provide intermediate storage for a problem instance. From`mp::Problem`

, conversion tools can be customized to reformulate the instance for a particular solver.

`mp::Problem`

converters¶

Given a problem instance in the standard format `mp::Problem`

, several
tools can be adapted to convert the instance for a particular solver.

For ‘flat’ (expression-less) solvers,

`mp::ProblemFlattener`

can walk the NL forest, passing flattened expressions as constraints to Flat model converters. In turn, these facilitate conversion of flat constraints which are not natively accepted by a solver into simpler forms.For expression-tree supporting solvers,

`mp::ExprVisitor`

and`mp::ExprConverter`

are efficient type-safe templates which can be customized to transform instances for a particular expression-based solver API.

### Flat model converters¶

`mp::FlatConverter`

and `mp::MIPFlatConverter`

represent and
convert flat models (i.e., models without expression trees).
Typically, a flat model
is produced from an NL model by Problem Flattener.
As a next step, constraints which are not natively accepted by a
solver, are transformed into simpler forms. This behavior
can be flexibly parameterized for a particular solver, preferably
via the solver’s modeling API wrapper:

Flat model API is the interface via which

`mp::FlatConverter`

addresses the underlying solver.Value presolver transforms solutions and suffixes between the original NL model and the flat model.

Reformulation graph discusses exporting of the flattening / reformulation graph.

### Flat model API¶

`mp::BasicFlatModelAPI`

is the interface via which Flat model converters address
the underlying solver.

#### Constraint acceptance¶

A subclassed wrapper, such as `mp::GurobiModelAPI`

,
signals its accepted constraints and which model reformulations are preferable.
For example, `GurobiModelAPI`

declares the following in order to natively
receive the logical OR constraint:

```
ACCEPT_CONSTRAINT(OrConstraint,
Recommended, // Driver recommends native form
CG_General) // Constraint group for suffix exchange
void AddConstraint(const OrConstraint& );
```

By default, if the driver does not mark a constraint as acceptable,
`mp::FlatConverter`

and its descendants attempt to simplify it. See
Configure automatic model reformulations for further details.

#### Model query API¶

To obtain a summary information on the flat model, for example the number
of constraints of a particular type or group, use the helper object of type
`mp::FlatModelInfo`

obtainable by `mp::BasicFlatModelAPI::GetFlatModelInfo()`

.

To preallocate memory for a class of constraints, the implementation can
redefine method `mp::BasicFlatModelAPI::InitProblemModificationPhase()`

:

```
void MosekModelAPI::InitProblemModificationPhase(const FlatModelInfo* info) {
/// Preallocate linear and quadratic constraints.
/// CG_Linear, CG_Quadratic, etc. are the constraint group indexes
/// provided in ACCEPT_CONSTRAINT macros.
MOSEK_CCALL(MSK_appendcons(lp(),
info->GetNumberOfConstraintsOfGroup(CG_Linear) +
info->GetNumberOfConstraintsOfGroup(CG_Quadratic)));
}
```

### Value presolver¶

Class `mp::pre::ValuePresolver`

manages transformations of solutions and suffixes
between the original NL model and the converted model. For driver architectures
with Model manager, the value presolver object must be shared between
the model converter and the Backend to enable
solution/suffix transformations corresponding to those on the model, see
`mp::CreateGurobiModelMgr`

as an example.

#### Invocation API¶

To use the ValuePresolver API, the following classes are needed:

`mp::pre::BasicValuePresolver`

defines an interface for`mp::pre::ValuePresolver`

.`mp::pre::ValueNode`

temporarily stores values corresponding to a single type of model item (variables, constraints, objectives).`mp::pre::ValueMap`

is a map of node values, where the key usually corresponds to an item subcategory. For example, Gurobi distinguishes attributes for the following constraint categories: linear, quadratic, SOS, general. Thus, the reformulation graph needs to have these four types of target nodes for constraint values:pre::ValueMapInt GurobiBackend::ConsIIS() { ...... return {{{ CG_Linear, iis_lincon }, { CG_Quadratic, iis_qc }, { CG_SOS, iis_soscon }, { CG_General, iis_gencon }}}; }

`mp::pre::ModelValues`

is a class joining value maps for variables, constraints, and objectives. It is useful when the conversions connect items of different types: for example, when converting an algebraic range constraint to an equality constraint with a bounded slack variable, the constraint’s basis status is mapped to that of the slack. Similarly, the range constraint should be reported infeasible if either the slack’s bounds or the equality are:IIS GurobiBackend::GetIIS() { pre::ModelValuesInt mv = GetValuePresolver(). PostsolveIIS( pre::ModelValuesInt{ VarsIIS(), ConsIIS() } ); return { mv.GetVarValues()(), mv.GetConValues()() }; }

#### Implementation API¶

To implement value pre- / postsolving, the following API is used:

`mp::pre::ValuePresolver`

implements the interface of`mp::pre::BasicValuePresolver`

. It calls the individual pre- and postsolve routines.`mp::pre::BasicLink`

is the interface to various implementations of links between nodes, such as`mp::pre::CopyLink`

,`mp::pre::One2ManyLink`

, and`mp::pre::RangeCon2Slack`

.Expression tree flattenings into new constraints and variables, as well as subsequent conversions, are by default automatically linked by

`mp::pre::One2ManyLink`

. To implement a specific link, see the example of`mp::pre::RangeCon2Slack`

. In particular, autolinking should normally be turned off.

### Reformulation graph¶

Explore the flattening and reformulation graph for your model, as described in Reformulation explorer.

## C++ ASL adapter¶

An efficient type-safe C++ adapter for the traditional ASL library for connecting solvers to AMPL and other systems. ASL has many additional functions, such as writing NL files and automatic differentiation.

## More details¶

This section overviews some more details of the API.

For a complete API reference, see the index.

### Problem representation¶

A standard representation of a model, convenient for intermediate storage.

`mp::ProblemInfo`

,`mp::var::Type`

,`mp::obj::Type`

,`mp::func::Type`

,`mp::ComplInfo`

### Expression forest walkers¶

Typesafe expression walkers for models stored in memory.

`mp::ProblemFlattener`

### Solution status¶

### Suffixes¶

Standard suffix value enums:

`mp::IISStatus`

,`mp::BasicStatus`