Testing#

Ensure the major functionality of your driver is tested by running end-to-end tests. Add your own test cases for essential functions. Unit tests should be used as well.

End-to-end tests#

End-to-end tests have proven most important.

Run end-to-end tests#

  • To run all tests for one or several solvers, enter the command:

    python3 test/end2end/run.py solver [another_solver [...]] [--options 'acc:max=0']

    The solver and ampl executables are expected to be on the system path, or use options --ampl, --bin_path.

  • Detailed results are saved into a CSV report file, see --reportstub.

  • To run a subset of the test cases, cd into the corresponding subfolder of test/end2end/cases, or use the --dir or --nonrecursive options.

Compulsory configurations#

The following test configurations are compulsory for a release.

  • All important/representative solvers. While some tests use acc:... options to enforce linearization etc., open-source solvers use it everywhere. If desired, such options can be passed to all test cases using the --options switch, e.g., --options 'acc:indle=0 acc:indge=0 acc:indeq=0 acc:and=0 acc:or=0'.

  • Using solution checker with --options "chk:fail chk:feastol=1e-2[ chk:mode=1023]".

Add solver to the test library#

  • To add a new solver to the test library, derive a corresponding class in Solver.py and list it in SolverCollection.py.

    • The stags parameter of the solver definition base class constructor in Solver.py should contain the tags describing available driver features.

Add new test cases#

The major functionality of a solver driver should be end-to-end tested. Folder test/end2end/cases/categorized/fast contains test cases which can be run in a few seconds for this purpose, which should be done frequently.

To add new test cases, add the model/data/AMPL script/NL files in a subfolder of test/end2end/cases/categorized/ and describe them in the local modellist.json having the following format.

Test specification#

In modellist.json, the top JSON object is an array of test cases. Each element is a dictionary with the following items, where non-compulsory items are italicized:

  • “name”: “<name>”: case name. Unless “files” is present (see below), the first word of <name> must coincide with the model / script filename stem. For example, a test case using case01.mod only could be called case01 objno=5.

  • “tags”: [“linear”, “continuous”]: tags specifying model type, the case is executed only if the tags are a subset of the solver’s ones. Except the tag “run” which means the test case is an AMPL script.

  • “files”: [“diet.mod”, “diet.dat”]

  • “objective”: value: expected objective value.

  • “options”: { “ANYSOLVER_options”: “iisfind=1”, “baron_options”: “iisfind=12”, “send_statuses”: “0” }. Option key ending with ANYSOLVER_options is for any solver, except when a solver-specific key is present (like baron_options). Moreover, mp_options applies to all MP drivers.

  • “values”: { “X[0].iis”: “upp”, … }. Expected values or expressions, in the form AMPL display command would accept. Only available for AMPL models / scripts. Example:

    "values": {
  "if color['Belgium'] != color['France'] then 1": 1,
  "if (forall {s in NS} (VAL_U_F2[s] = 1  ==>  U_F2[s] >= 0.0001  else  U_F2[s] = 0)) then 1": 1,
  "abs(x)": { "max": 1e-3 },
  "solve_result_num": 0
}

    In particular:

    • The expected value can be a dictionary of the form { "min": ..., "max": ... }:

      "max{i in 1 .. _nccons} abs(_ccon[i])": { "max": 1e-5 },
"min{i in 1 .. _ncons} _con[i].slack": { "min": -1e-5 }

      This checks all complementarity and algebraic constraints.

    • To check logical expressions, use if/then:

      "if forall {i in 1.._nlogcons} _logcon[i] then 1": 1,

      This checks all logical constraints.

  • “output”: [“Presolved model has 500 variables”, “RHS range: [1e-2, 1e4]”]. Output chunks expected in the solver / AMPL log.

Checking reformulations#

It is possible to check characteristics of the models received by an MP driver from the NL file, as well as of the model submitted to the solver. For that, add option [tech:]debug=1. Then, the driver outputs problem suffixes of the form flat<N>__[<C>__]<itemtype> giving the number of model items of various types and statuses. The specifiers:

  • <N> is 0 for the initial flat model obtained from the NL file, 1 for the reformulated model.

  • If present, <C> can be A for the total number of items, U for the number of total items still relevant, but possibly reformulated. If not present, it is the number of active items at the end of step <N>.

  • <itemtype> can be a constraint type name, variable type, or objective type.

Run an AMPL session to see all such suffixes, or check existing test cases.

Unit tests#

Unit tests are in the folder test. Compulsory are tests converter-flat-test and converter-mip-test.