Ensure the major functionality of your driver is tested by running end-to-end tests. Add your own test cases for essential functions. Unit tests can be used as well.

End-to-end tests

End-to-end tests have proven most important.

Run end-to-end tests

  • To run all tests for one or several solvers, enter the command:

    python3 test/end2end/run.py solver [another_solver [...]]

    The solver and ampl executables are expected to be on the system path.

  • Detailed results are saved into a CSV report file, see --reportstub.

  • To run a subset of the test cases, cd into the corresponding subfolder of test/end2end/cases, or use the --dir or --nonrecursive options.

Add solver to the test library

  • To add a new solver to the test library, derive a corresponding class in Solver.py and list it in SolverCollection.py.

    • The stags parameter of the solver definition base class constructor in Solver.py should contain the tags describing available driver features.

Add new test cases

The major functionality of a solver driver should be end-to-end tested. Folder test/end2end/cases/categorized/fast contains test cases which can be run in a few seconds for this purpose, which should be done frequently.

To add new test cases, add the model/data/AMPL script files in a subfolder of test/end2end/cases/categorized/ and describe them in the local modellist.json having the following format. The top JSON object is an array of test cases. Each element is a dictionary with the following items, where non-compulsory items are italicized:

  • “name”: “<name>”: case name. Unless “files” is present (see below), the first word of <name> must coincide with the model / script filename stem. For example, a test case using case01.mod only could be called case01 objno=5.

  • “tags”: [“linear”, “continuous”]: tags specifying model type, the case is executed only if the tags are a subset of the solver’s ones. Except the tag “run” which means the test case is an AMPL script.

  • “files”: [“diet.mod”, “diet.dat”]

  • “objective”: value: expected objective value.

  • “options”: { “ANYSOLVER_options”: “iisfind=1”, “baron_options”: “iisfind=12”, “send_statuses”: “0” }. Option key ending with ANYSOLVER_options is for any solver, except when a solver-specific key is present (like baron_options.)

  • “values”: { “X[0].iis”: “upp”, … }. Expected values or expressions, in the form AMPL display command would accept.

    • For example, to check logical expressions, use if/then:

      "values": {
        "if color['Belgium'] != color['France'] then 1": 1,
        "if (forall {s in NS} (VAL_U_F2[s] = 1  ==>  U_F2[s] >= 0.0001  else  U_F2[s] = 0)) then 1": 1,
        "if abs(x) < 1e-3 then 1": 1,
        "solve_result_num": 0

Unit tests

You can employ unit tests as well, see folder test.