+LatexCommand formatted
+reference "subsec:New-layouts"
+
+\end_inset
+
+.
+ No lyx2lyx conversion is then required when a new major version is released.
+\end_layout
+
+\begin_layout Standard
+The user can move an existing document to the new version simply by selecting
+ a new document class.
+ This step is well supported by \SpecialChar LyX
+, with established methods for handling
+ unsupported styles and other changes.
+ This way, no lyx2lyx code is required.
+\end_layout
+
+\begin_layout Standard
+The steps to support a new version of an existing document class are thus:
+\end_layout
+
+\begin_layout Itemize
+Create a new layout file including the upstream version in the name (avoid
+ special characters like spaces and dots), e.g.
+
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+acmsiggraph-v0-92.layout
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Itemize
+Include the name of the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+*.cls
+\end_layout
+
+\end_inset
+
+ file as an optional argument in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+
+\backslash
+DeclareLaTeXClass
+\end_layout
+
+\end_inset
+
+ line and include the version number in the GUI name:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+
+\backslash
+DeclareLaTeXClass[acmsiggraph]{ACM SIGGGRAPH (v.
+\begin_inset space ~
+\end_inset
+
+0.92)}
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+Update the GUI name in the old layout file (whose name should not be changed),
+ e.g.:
+\begin_inset Newline newline
+\end_inset
+
+
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+
+\backslash
+DeclareLaTeXClass{ACM SIGGRAPH (<= v.
+\begin_inset space ~
+\end_inset
+
+0.91, obsolete)}
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+To avoid duplicate definitions, the new layout can
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+Input
+\end_layout
+
+\end_inset
+
+ the old layout file and add\SpecialChar breakableslash
+remove\SpecialChar breakableslash
+obsolete\SpecialChar breakableslash
+modify settings and styles (similar
+ to the inclusion of
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+*.inc
+\end_layout
+
+\end_inset
+
+ files).
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+It may be tempting to let the new layout be the
+\begin_inset Quotes eld
+\end_inset
+
+master version
+\begin_inset Quotes erd
+\end_inset
+
+ and have the old layout import it.
+ However, this should not be done because any changes to the new layout
+ would need undo steps in the importing old layout.
+\end_layout
+
+\end_deeper
+\begin_layout Itemize
+If the new LaTeX document class obsoletes the old one, update the example
+ and template files to use the new layout.
+ Add a note about the changes (preferably with a pointer to the documentation
+ of the changes).
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+This way, new documents based on the template or example will use the up-to-date
+ document class version.
+\end_layout
+
+\end_deeper
+\begin_layout Standard
+\begin_inset Newpage newpage
+\end_inset
+
+
+\end_layout
+
+\begin_layout Section
+Tests
+\end_layout
+
+\begin_layout Standard
+Automated tests are an important tool to detect bugs and regressions in
+ software development.
+ Some projects like gcc even require each bug fix to be accompanied by a
+ test case for the automatic test suite, that would detect this bug.
+ Testing interactive features automatically is of course very hard, but
+ core functionality like document import and export can be tested quite
+ easily, and some tests of this kind exist.
+\end_layout
+
+\begin_layout Subsection
+unit tests
+\end_layout
+
+\begin_layout Standard
+There are attempts to set up a suite of unit tests for LyX.
+\end_layout
+
+\begin_layout Standard
+TODO: describe what is done and what is still to do.
+\end_layout
+
+\begin_layout Subsection
+tex2lyx tests
+\end_layout
+
+\begin_layout Standard
+The tex2lyx tests are located in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test
+\end_layout
+
+\end_inset
+
+ subfolder of the \SpecialChar LyX
+ source code distribution.
+ The actual testing is performed by the simple python script
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test/runtests.py
+\end_layout
+
+\end_inset
+
+.
+ Each test consists of two files:
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+<test name>.tex
+\end_layout
+
+\end_inset
+
+ contains the \SpecialChar LaTeX
+ code that should be tested.
+
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+<test name>.lyx.lyx
+\end_layout
+
+\end_inset
+
+ contains the expected output of tex2lyx.
+ When a test is run, the actual produced output is compared with the stored
+ reference output.
+ The test passes if both are identical.
+ The test machinery is also able to generate a file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+<test name>.lyx.tex
+\end_layout
+
+\end_inset
+
+ by exporting the produced .lyx file with \SpecialChar LyX
+ again.
+ This may be useful for roundtrip comparisons.
+\end_layout
+
+\begin_layout Subsubsection
+Running the tests
+\end_layout
+
+\begin_layout Standard
+The tex2lyx tests can be run in several ways.
+ When in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx
+\end_layout
+
+\end_inset
+
+ subfolder of the build directory, the commands
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+ (cmake, all platforms),
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+make test
+\end_layout
+
+\end_inset
+
+ (cmake, when using a make based build system and not MSVC) or
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+make alltests
+\end_layout
+
+\end_inset
+
+ (autotools) will run the tex2lyx tests.
+ Alternatively, in the root of the build directory, the command
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -R tex2lyx
+\end_layout
+
+\end_inset
+
+ runs all tests whose names match the regex
+\begin_inset Quotes eld
+\end_inset
+
+tex2lyx
+\begin_inset Quotes erd
+\end_inset
+
+.
+ Another way to run the tex2lyx tests in the root build directory is to
+ instead use the command
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L '(cmplyx|roundtrip)'
+\end_layout
+
+\end_inset
+
+, which runs all tests categorized with the label
+\begin_inset Quotes eld
+\end_inset
+
+roundtrip
+\begin_inset Quotes erd
+\end_inset
+
+ or
+\begin_inset Quotes eld
+\end_inset
+
+cmplyx
+\begin_inset Quotes erd
+\end_inset
+
+.
+ If a test fails, the differences between the expected and actual results
+ are output in unified diff format.
+\end_layout
+
+\begin_layout Subsubsection
+Updating test references
+\begin_inset CommandInset label
+LatexCommand label
+name "sec:Updating-test-references"
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+In some cases a changed tex2lyx output is not a test failure, but wanted,
+ e.
+\begin_inset space \thinspace{}
+\end_inset
+
+g.
+\begin_inset space \space{}
+\end_inset
+
+if a tex2lyx bug was fixed, or a new feature was added.
+ In these cases the stored references need to be updated.
+ To do so if using autotools, call
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+make updatetests
+\end_layout
+
+\end_inset
+
+ in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx
+\end_layout
+
+\end_inset
+
+ subdirectory of the build directory.
+ If instead using CMake, call
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+make updatetex2lyxtests
+\end_layout
+
+\end_inset
+
+ in the build directory or in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test
+\end_layout
+
+\end_inset
+
+ subdirectory of the build directory.
+\begin_inset Foot
+status collapsed
+
+\begin_layout Plain Layout
+Note that this is a case where a make target in the build directory can
+ affect the source directory, which might not be advisable.
+\end_layout
+
+\end_inset
+
+ On Windows do the following:
+\end_layout
+
+\begin_layout Itemize
+Assure that the path to the python.exe is in your system PATH variable.
+\end_layout
+
+\begin_layout Itemize
+Double-click on the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+updatetex2lyxtests.vcxproj
+\end_layout
+
+\end_inset
+
+ in the build directory or in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test
+\end_layout
+
+\end_inset
+
+ subdirectory of your build directory.
+\end_layout
+
+\begin_layout Itemize
+In the appearing MSVC program right-click on the project
+\family sans
+updatetex2lyxtests
+\family default
+ in the project explorer and chose
+\family sans
+Create
+\family default
+.
+\end_layout
+
+\begin_layout Standard
+For convenience, these commands also produce re-exported roundtrip .lyx.tex
+ files.
+ Please examine the changed output carefully before committing the changed
+ files to the repository: Since the test machinery does not do a roundtrip
+ test .tex
+\begin_inset Formula $\Rightarrow$
+\end_inset
+
+ .lyx
+\begin_inset Formula $\Rightarrow$
+\end_inset
+
+ .tex, and does not compare the produced dvi or pdf output, it assumes that
+ the stored .lyx reference produces correct output if processed by \SpecialChar LyX
+.
+ There is only one chance to detect wrong output: before committing a new
+ reference.
+ Once it is committed, it is quite difficult to verify whether it is correct.
+\end_layout
+
+\begin_layout Standard
+Please
+\emph on
+do not
+\emph default
+ update the test references by opening them with \SpecialChar LyX
+ or directly running lyx2lyx
+ on them.
+ This would not work, since lyx2lyx and \SpecialChar LyX
+ produce slightly different files
+ regarding insignificant whitespace and line breaks.
+\end_layout
+
+\begin_layout Subsubsection
+Adding a new test
+\end_layout
+
+\begin_layout Standard
+In many cases tests for new features may be added to one of the existing
+ test files, but sometimes this is not possible or not wanted.
+ Then a new test file needs to be added:
+\end_layout
+
+\begin_layout Enumerate
+Create the new file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test/<test name>.tex
+\end_layout
+
+\end_inset
+
+ and run tex2lyx in roundtrip mode to produce the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test/<test name>.lyx.lyx
+\end_layout
+
+\end_inset
+
+.
+ This file will be the new reference.
+\end_layout
+
+\begin_layout Enumerate
+Once you confirmed that the tex2lyx output is correct, add the new files
+ to the corresponding lists in
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test/runtests.py
+\end_layout
+
+\end_inset
+
+,
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/Makefile.am
+\end_layout
+
+\end_inset
+
+ and
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+src/tex2lyx/test/CMakeLists.txt
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Enumerate
+Commit the changes to the repository, or send a patch to the development
+ list and ask for committing if you do not have commit rights.
+\end_layout
+
+\begin_layout Subsection
+ctest automatic tests
+\end_layout
+
+\begin_layout Standard
+Some tests are located in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/
+\end_layout
+
+\end_inset
+
+ subfolder of the \SpecialChar LyX
+ source code distribution.
+
+\begin_inset Foot
+status open
+
+\begin_layout Plain Layout
+The README document in this folder only describes the
+\begin_inset Quotes eld
+\end_inset
+
+keytests
+\begin_inset Quotes erd
+\end_inset
+
+ subset of autotests!
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+These tests can be run by the commands
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+ in the
+\emph on
+ build directory
+\emph default
+ (all platforms) or (when using a make based build system and not MSVC)
+
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+make test
+\end_layout
+
+\end_inset
+
+ in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+autotests/
+\end_layout
+
+\end_inset
+
+ subfolder of the
+\emph on
+ build directory
+\emph default
+.
+ The test logs are written to the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+Testing/Temporary/
+\end_layout
+
+\end_inset
+
+ subfolder of the
+\emph on
+
+\emph default
+build directory.
+
+\end_layout
+
+\begin_layout Subsubsection
+Export tests
+\end_layout
+
+\begin_layout Standard
+The export tests are integration tests.
+ They take longer to run and are more likely to break than the tex2lyx tests.
+ Nevertheless, they have caught many regressions and without a better alternativ
+e it is important to keep them up-to-date and understand how they work.
+\end_layout
+
+\begin_layout Standard
+The export tests
+\begin_inset Quotes eld
+\end_inset
+
+reuse
+\begin_inset Quotes erd
+\end_inset
+
+ documentation, template, and example documents.
+ In addition, there are a number of dedicated sample documents in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+autotests/export/
+\end_layout
+
+\end_inset
+
+ subfolder of the \SpecialChar LyX
+ source code distribution.
+ All samples are (after copying and eventual processing by scripts) exported
+ to various output formats via the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+—export-to
+\end_layout
+
+\end_inset
+
+ command line option.
+ The tests checks for errors reported by LyX.
+ (However, error-free export is no guarantee for an error-free output document.)
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:when-to-run-an-export-test"
+
+\end_inset
+
+Expectations of LyX developers
+\end_layout
+
+\begin_layout Standard
+Because the export tests are integration tests and take a long time to run,
+ LyX developers are rarely expected to run all of the tests.
+ Here are some good practices to follow by developers:
+\end_layout
+
+\begin_layout Itemize
+When making a non-trivial change to a .layout file, run the export and layout
+ tests corresponding with that .layout file.
+\end_layout
+
+\begin_layout Itemize
+When making non-trivial changes to a .lyx file, run the export tests correspondin
+g to that .lyx file.
+
+\begin_inset Foot
+status collapsed
+
+\begin_layout Plain Layout
+This rule is due to revision.
+
+\end_layout
+
+\begin_layout Plain Layout
+There is an objection from the documentation maintainer that working on
+ the documentation must not be complicated by having to consider non-standard
+ exports.
+\end_layout
+
+\begin_layout Itemize
+successful compiling/testing an edited documentation file with pdflatex
+ suffices to ensure it can be commited, not tests with other exports are
+ required.
+\end_layout
+
+\begin_layout Plain Layout
+If sudden failures with other exports due to “half-tested” documentation
+ updates are a problem for the test maintainer, the test suite should use
+ copies that are
+\end_layout
+
+\begin_layout Itemize
+copied to a cache dir (autotests/samples/doc/, say) but not changed,
+\end_layout
+
+\begin_layout Itemize
+updated regularely (but on a time chosen by the test suite maintainer) from
+ the originals in lib/doc/
+\end_layout
+
+\begin_layout Plain Layout
+This way,
+\end_layout
+
+\begin_layout Itemize
+no test will fail due to ongoing work on documentation,
+\end_layout
+
+\begin_layout Itemize
+the documentation is still tested in full (with some delay),
+\end_layout
+
+\begin_layout Itemize
+failures with non-default export can be examined and handled accordingly
+ in one run with the cache update,
+\end_layout
+
+\begin_layout Itemize
+“interesting failures” (like the nested-language+polyglossia problem in
+ es/Customization can be separated and moved into dedicated test samples.
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+When making non-trivial changes to LyX's \SpecialChar LaTeX
+ export code (e.g.
+ touching the encoding code or package handling code that you expect will
+ change the exported \SpecialChar LaTeX
+ in some way):
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+\paragraph_spacing single
+Consider running all of the export tests before and after your change.
+ If there are differences, please reconcile these (i.e.
+ fix the bug or fix the tests)
+\emph on
+before
+\emph default
+ committing.
+ Ask for help if you're not sure what to.
+\end_layout
+
+\begin_layout Standard
+If you do not want to run the tests,
+\end_layout
+
+\begin_layout Itemize
+post the patch on the list and others will run the tests and eventually
+ ask for fixes, or
+\end_layout
+
+\begin_layout Itemize
+commit, but be prepared to fix eventually arising problems or to revert
+ the commit if there is no easy fix.
+\end_layout
+
+\end_deeper
+\begin_layout Itemize
+Understand how to interpret test failures.
+ If your commit is found to have broken a test, you should be able to interpret
+ the test results when made aware of them.
+ See Section
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "subsec:Interpreting-export-tests"
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:export-test-output-formats"
+
+\end_inset
+
+Output formats
+\end_layout
+
+\begin_layout Standard
+The following output formats are currently tested for each sample document
+ (see
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Export-test-filtering"
+
+\end_inset
+
+ for exceptions):
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+LyX:
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+lyx16 LyX 1.6 file format (lyx2lyx)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+lyx21 LyX 2.1 file format (lyx2lyx)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+xhtml LyXHTML (native LyX HTML export)
+\end_layout
+
+\end_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+LyX
+\begin_inset space ~
+\end_inset
+
++
+\begin_inset space ~
+\end_inset
+
+LaTeX:
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+dvi DVI (8-bit latex)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+dvi3_texF DVI (LuaTeX with 8-bit TeX fonts)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+dvi3_systemF DVI (LuaTeX with Unicode fonts)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf2 PDF (pdflatex)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf4_texF PDF (XeTeX with 8-bit TeX fonts)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf4_systemF PDF (XeTeX with Unicode fonts)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf5_texF PDF (LuaTeX with 8-bit TeX fonts)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf5_systemF PDF (LuaTeX with Unicode fonts)
+\end_layout
+
+\end_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+LyX
+\begin_inset space ~
+\end_inset
+
++
+\begin_inset space ~
+\end_inset
+
+LaTeX
+\begin_inset space ~
+\end_inset
+
++
+\begin_inset space ~
+\end_inset
+
+postprocessing:
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf DVI -> PS (dvips) -> PDF (ps2pdf)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdf3 DVI -> PDF (dvipdfm)
+\end_layout
+
+\end_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+not
+\begin_inset space ~
+\end_inset
+
+tested: (or only if set as default output format in the document source)
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+latex LaTeX (plain)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+luatex LaTeX (LuaTeX)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+dviluatex LaTeX (dviluatex)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+pdflatex LaTeX (pdflatex)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+platex LaTeX (pLaTeX)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+xetex LaTeX (XeTeX)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+eps3 EPS (encapsulated Postscript) (cropped)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+ps DVI -> Postscript (dvips)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+odf
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+text (nor text2, ..., text4)
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+textparagraph
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+word
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+word2
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring pdf5msystemFM
+wordhtml
+\end_layout
+
+\end_deeper
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:Configuring-ctests"
+
+\end_inset
+
+Configuring the tests
+\end_layout
+
+\begin_layout Standard
+To enable the export autotests, add the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-DLYX_ENABLE_EXPORT_TESTS=ON
+\end_layout
+
+\end_inset
+
+ flag.
+ For example:
+\end_layout
+
+\begin_layout Standard
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+cmake -DLYX_ENABLE_EXPORT_TESTS=ON /path/to/source
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+\noindent
+This flag will increase the time for the cmake command by several seconds,
+ mainly because of the process of inverting tests (see Section
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "subsec:Interpreting-export-tests"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:ctest-options"
+
+\end_inset
+
+Running the tests
+\end_layout
+
+\begin_layout Standard
+To run all tests, in the build directory simply run the command
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+.
+ A full, up-to-date TeXLive installation is recommended to run the tests.
+ Otherwise, some tests will fail.
+ Tests with additional requirements are labeled
+\begin_inset Quotes eld
+\end_inset
+
+unreliable:nonstandard
+\begin_inset Quotes erd
+\end_inset
+
+.
+
+\end_layout
+
+\begin_layout Standard
+To run only some of the tests, use command line options (see examples below):
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-R <pattern>
+\end_layout
+
+\end_inset
+
+ Run only the tests whose names match the given regular expression.
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-L <pattern>
+\end_layout
+
+\end_inset
+
+ Run only the tests whose labels match the given regular expression.
+ A test may have more that one label.
+
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-E <pattern>
+\end_layout
+
+\end_inset
+
+ Exclude the tests whose names match the given regular expression.
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-LE <pattern>
+\end_layout
+
+\end_inset
+
+ Exclude the tests whose labels match the given regular expression.
+ Cannot be combined with
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-L
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+The following options help to find good selection patterns:
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-N
+\end_layout
+
+\end_inset
+
+ List the tests that would be run but not actually run them.
+
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+Useful in conjunction with the -R, -L, -E and -LE options, e.g., if you want
+ to know how many tests there are or whether your
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+<pattern>
+\end_layout
+
+\end_inset
+
+ regular expression did what you expected.
+\end_layout
+
+\end_deeper
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+\SpecialChar nobreakdash
+\SpecialChar nobreakdash
+print-labels
+\end_layout
+
+\end_inset
+
+ print the list of all labels associated with the test set.
+ Can also be combined with -R, -L, -E, ...
+
+\end_layout
+
+\begin_layout Standard
+Other useful options are:
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-j <jobs>
+\end_layout
+
+\end_inset
+
+ Run the tests in parallel using the given number of jobs.
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+We are still working on getting the tests to run in parallel.
+ However, when running the tests in parallel, sometimes tests fail that
+ pass when run sequentially.
+ A reasonable approach is to first run the tests in parallel and then run
+ the failed tests sequentially.
+\end_layout
+
+\begin_layout Standard
+For example, to run 8 jobs at a time:
+\end_layout
+
+\begin_layout Standard
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -j8
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest \SpecialChar nobreakdash
+\SpecialChar nobreakdash
+rerun-failed
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+When specifying a subset of the tests (e.g.
+ using
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+\SpecialChar nobreakdash
+R <pattern>
+\end_layout
+
+\end_inset
+
+), the same subset must be specified when using the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+\SpecialChar nobreakdash
+\SpecialChar nobreakdash
+rerun-failed
+\end_layout
+
+\end_inset
+
+ option because it is the test numbers that are used to index which tests
+ failed on the previous run.
+\end_layout
+
+\begin_layout Standard
+\noindent
+Note that some tests cannot be run in parallel.
+ These tests are marked in the code with the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+\noindent
+RUN_SERIAL ON
+\end_layout
+
+\end_inset
+
+ CMake property.
+\end_layout
+
+\end_deeper
+\begin_layout Labeling
+\labelwidthstring -R
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+\SpecialChar nobreakdash
+\SpecialChar nobreakdash
+timeout <seconds>
+\end_layout
+
+\end_inset
+
+ Set a global timeout on all tests that do not already have a timeout set
+ on them.
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+There have been bugs in LyX and in \SpecialChar LaTeX
+ which cause compilation to hang, and
+ without a timeout a test might never stop (in one case there was even a
+ memory leak).
+ If a test times out, the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+ command exits with error, but you can distinguish between a timed out test
+ and a failed test in the output reported at the end of the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+ command.
+\end_layout
+
+\end_deeper
+\begin_layout Standard
+See the manual (
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+man ctest
+\end_layout
+
+\end_inset
+
+) the full list of command line options.
+\end_layout
+
+\begin_layout Paragraph
+Examples
+\end_layout
+
+\begin_layout Itemize
+run only the export tests:
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L export
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+run inverted tests:
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L "inverted|suspended"
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+list all export tests which match any of the labelling patterns:
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -N -R "
+\backslash
+..*_export/"
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Itemize
+exclude rarely used output formats and post-processing tests
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L export -E "_(texF|dvi3|pdf3?)"
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "subsec:Interpreting-export-tests"
+
+\end_inset
+
+Interpreting the export test results
+\end_layout
+
+\begin_layout Standard
+A test can fail for several reasons, not all of them bad.
+\end_layout
+
+\begin_layout Enumerate
+A new or edited sample document may be incompatible with some output formats.
+\end_layout
+
+\begin_layout Enumerate
+A dependency is not met (e.g.
+ the \SpecialChar LaTeX
+ class file).
+ One hint that this is the case is that the corresponding
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+check_load
+\end_layout
+
+\end_inset
+
+ test will likely also fail.
+\end_layout
+
+\begin_layout Enumerate
+An inverted test fails to fail (i.e.
+ export that previously failed now works).
+\end_layout
+
+\begin_layout Enumerate
+An external dependency was updated (e.g.
+ \SpecialChar TeX
+ Live).
+\end_layout
+
+\begin_layout Enumerate
+A recent code change introduced a bug.
+\end_layout
+
+\begin_layout Enumerate
+\begin_inset CommandInset label
+LatexCommand label
+name "enu:exposed"
+
+\end_inset
+
+A change in a document exposed a previously unknown bug or an incompatibility
+ with an export format (e.g.
+ Lua\SpecialChar LaTeX
+).
+\end_layout
+
+\begin_layout Standard
+Because the .lyx files are exported in several formats, it is not surprising
+ that many of the exports fail.
+ This expectation of failure is addressed by
+\begin_inset Quotes eld
+\end_inset
+
+inverting
+\begin_inset Quotes erd
+\end_inset
+
+ the tests, that is, by marking the test as
+\begin_inset Quotes eld
+\end_inset
+
+passing
+\begin_inset Quotes erd
+\end_inset
+
+ if the export exits with error and as
+\begin_inset Quotes eld
+\end_inset
+
+failing
+\begin_inset Quotes erd
+\end_inset
+
+ if the export succeeds
+\emph on
+.
+
+\emph default
+ It follows that these expected failures will not show up as failed tests
+ in the test results and thus will not pollute the
+\begin_inset Quotes eld
+\end_inset
+
+good
+\begin_inset Quotes erd
+\end_inset
+
+ tests.
+ If the export actually succeeds, then the test will fail.
+ The purpose of this failure is to get your attention—something has changed,
+ possibly for the better.
+\end_layout
+
+\begin_layout Standard
+We try to document why a test is inverted or ignored.
+ See the comment (prefixed with
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+#
+\end_layout
+
+\end_inset
+
+) above the block in which the test is listed as inverted or ignored in
+ the files
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/invertedTests
+\end_layout
+
+\end_inset
+
+,
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/unreliableTests
+\end_layout
+
+\end_inset
+
+ and
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/ignoredTests
+\end_layout
+
+\end_inset
+
+.
+
+\end_layout
+
+\begin_layout Standard
+A good question is why do we enable the tests for non-default formats? The
+ answer is that if a non-default route is broken it is often because a bug
+ was introduced in LyX and not because a document-specific change was made
+ that is not supported by the route.
+ In other words, there is a high signal/noise ratio in the export tests
+ for some non-default formats.
+
+\end_layout
+
+\begin_layout Standard
+When a test or several tests fail, consider checking the files in the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+Testing/Temporary/
+\end_layout
+
+\end_inset
+
+ subdirectory of your build directory.
+ In this subdirectory are three files: the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+LastTestsFailed.log
+\end_layout
+
+\end_inset
+
+ simply lists the tests that failed on your last
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest
+\end_layout
+
+\end_inset
+
+ command; the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+LastTest.log
+\end_layout
+
+\end_inset
+
+ file contains the output from the tests (and often has details explaining
+ why a test failed); and the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+CTestCostData.txt
+\end_layout
+
+\end_inset
+
+ file lists the times that it took to run the tests.
+\end_layout
+
+\begin_layout Paragraph
+What action should you take if a test fails?
+\end_layout
+
+\begin_layout Standard
+\paragraph_spacing single
+It is always good to check manually why something fails and if it passes
+ if the PDF output is good.
+\end_layout
+
+\begin_layout Itemize
+Generally, if a change breaks compilation for the target format (for the
+ manuals pdf2) without solving some important other issue,
+\emph on
+fix or revert the commit
+\emph default
+ that led to failure.
+\end_layout
+
+\begin_layout Itemize
+If it is not possible to (immediately) fix the failure but there are reasons
+ not to revert the commit (e.g.
+ it fixes another more important issue),
+\emph on
+invert
+\emph default
+ the failing test case (see
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Inverted-tests"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_layout Itemize
+If an
+\emph on
+inverted
+\emph default
+ test case fails because the export now works,
+\emph on
+uninvert
+\emph default
+ the test by removing the pattern from the
+\begin_inset Quotes eld
+\end_inset
+
+invertedTests
+\begin_inset Quotes erd
+\end_inset
+
+ file) (see
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Inverted-tests"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_layout Itemize
+If the export did not fail previously but led to wrong output (PDF, say),
+\begin_inset Foot
+status collapsed
+
+\begin_layout Plain Layout
+Non-failing test with wrong output should be labeledas
+\begin_inset Quotes eld
+\end_inset
+
+unreliable:wrong_output
+\begin_inset Quotes erd
+\end_inset
+
+ (
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Unreliable-tests"
+
+\end_inset
+
+).
+\end_layout
+
+\end_inset
+
+ it is in fact an improvement when the test now fails.
+
+\emph on
+Invert
+\emph default
+ the failing test case (see
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Inverted-tests"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_layout Itemize
+In case of tests failing due to missing requirements (tests labeled
+\begin_inset Quotes eld
+\end_inset
+
+unreliable:nonstandard
+\begin_inset Quotes erd
+\end_inset
+
+ or testing on a system withonly a subset of TeXLive installed), ignore
+ the failure, ask for someone else to run the test, or install the missing
+ ressources and try again.
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:Inverted-tests"
+
+\end_inset
+
+Inverted tests
+\end_layout
+
+\begin_layout Standard
+Test cases whose name matches a pattern in the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/invertedTests
+\end_layout
+
+\end_inset
+
+ get the label
+\emph on
+inverted
+\emph default
+.
+ They get also the test property
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+WILL_FAIL
+\end_layout
+
+\end_inset
+
+, i.e.
+ they are reported as failing if the export works without error
+\begin_inset Flex URL
+status collapsed
+
+\begin_layout Plain Layout
+
+https://cmake.org/cmake/help/v3.0/command/set_tests_properties.html
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+Add failing cases to this file, if they cannot be solved
+\begin_inset Quotes eld
+\end_inset
+
+immediately
+\begin_inset Quotes erd
+\end_inset
+
+ but it is expected that the export will work in a foreseeable future, e.g.
+ low priority issues like failures to export to a non-target format (for
+ the manuals everything except pdf2).
+\end_layout
+
+\begin_layout Standard
+The following sublabels are currently present in
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+invertedTests
+\end_layout
+
+\end_inset
+
+:
+\end_layout
+
+\begin_layout Description
+todo test failures that require attention:
+\end_layout
+
+\begin_deeper
+\begin_layout Itemize
+minor issues to explore and properly sort later,
+\end_layout
+
+\begin_layout Itemize
+easyfix issues,
+\end_layout
+
+\begin_layout Itemize
+LyX bugs to report at trac (move pattern to section "lyxbugs" once done).
+\end_layout
+
+\end_deeper
+\begin_layout Description
+lyxbugs LyX bugs with a Trac number.
+\end_layout
+
+\begin_layout Description
+ert Export failures due to "raw" LaTeX use in ERT or preamble code.
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+"Wontfix" if demonstrating correct use and OK in the default output format.
+\end_layout
+
+\end_deeper
+\begin_layout Description
+texissues Export fails due to LaTeX limitations like non-ASCII characters
+ in verbatim or listings, incompatible packages, ...
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+"Wontfix" if documents demonstrate correct use in the default output format:
+\end_layout
+
+\begin_layout Itemize
+If the source can be made more robust without becoming "hackish", fix the
+ source,
+\end_layout
+
+\begin_layout Itemize
+if LyX could be enhanced to care for a permanent TeX limitation, file a
+ ticket at trac and add a pattern under lyxbugs,
+\end_layout
+
+\begin_layout Itemize
+otherwise, add a pattern here.
+\end_layout
+
+\end_deeper
+\begin_layout Description
+attic Documents in the attic (kept for reference and format conversion test).
+ Usually
+\begin_inset Quotes eld
+\end_inset
+
+Wontfix
+\begin_inset Quotes erd
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Subparagraph
+suspended tests
+\end_layout
+
+\begin_layout Standard
+Test cases whose name additionally matches a pattern in the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/suspendedTests
+\end_layout
+
+\end_inset
+
+ get the label
+\emph on
+suspended
+\emph default
+(instead of
+\emph on
+export
+\emph default
+and
+\emph on
+ inverted
+\emph default
+).
+ This means they are not executed using
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L export
+\end_layout
+
+\end_inset
+
+ or
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L inverted
+\end_layout
+
+\end_inset
+
+.
+ However, they also get the test property
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+WILL_FAIL
+\end_layout
+
+\end_inset
+
+, i.e.
+ they are reported as failing if the export works without error.
+ From time to time they still have to be checked using
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L suspended
+\end_layout
+
+\end_inset
+
+.
+\end_layout
+
+\begin_layout Standard
+These tests are suspended, because the export fails for known reasons which
+ cannot ATM be resolved.
+ But it is expected the reason might disappear in the future.
+ Be it new TL or better handling in \SpecialChar LyX
+.
+\end_layout
+
+\begin_layout Standard
+For ctest commands without the
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+-L
+\end_layout
+
+\end_inset
+
+ parameter nothing changes.
+ Suspended or not, tests will be executed depending only on the selecting
+ regular expression given to the ctest command (see
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:ctest-options"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:Unreliable-tests"
+
+\end_inset
+
+Unreliable tests
+\end_layout
+
+\begin_layout Standard
+Test cases whose name matches a pattern in the file
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+development/autotests/unreliableTests
+\end_layout
+
+\end_inset
+
+ get the label
+\emph on
+unreliable
+\emph default
+.
+\end_layout
+
+\begin_layout Standard
+These tests are not executed using
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L export
+\end_layout
+
+\end_inset
+
+ or
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+ctest -L inverted
+\end_layout
+
+\end_inset
+
+.
+
+\end_layout
+
+\begin_layout Standard
+They pass or fail for various reasons not related to LyX (nonstandard, erratic)
+ or pass but should rather fail (wrong output).
+
+\begin_inset Note Note
+status collapsed
+
+\begin_layout Plain Layout
+*invalid* tests (wrong output) are not *unreliable*.
+ # Use "unfit" or "unapplicable" as better label and name of pattern file?
+
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Standard
+The following sublabels are currently present in
+\begin_inset Flex Code
+status collapsed
+
+\begin_layout Plain Layout
+unreliableTests
+\end_layout
+
+\end_inset
+
+:
+\end_layout
+
+\begin_layout Description
+nonstandard Documents with additional requirements, e.g.
+ a class or package file not in TeXLive.
+
+\begin_inset Note Note
+status open
+
+\begin_layout Plain Layout
+TODO: rename to
+\begin_inset Quotes eld
+\end_inset
+
+extra
+\begin_inset Quotes erd
+\end_inset
+
+ or
+\begin_inset Quotes eld
+\end_inset
+
+exotic
+\begin_inset Quotes erd
+\end_inset
+
+?
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\begin_layout Description
+erratic Tests depending on local configuration or the phase of the moon.
+
+\end_layout
+
+\begin_layout Description
+varying_versions Test depending on TeX distribution, package versions or
+ OS.
+\end_layout
+
+\begin_layout Description
+wrong
+\begin_inset space ~
+\end_inset
+
+output Export does not fail but the resulting document has (undetected)
+ errors.
+\end_layout
+
+\begin_deeper
+\begin_layout Standard
+\paragraph_spacing single
+\begin_inset Note Note
+status open
+
+\begin_layout Plain Layout
+\paragraph_spacing single
+These tests are in a strict sense not unreliable but
+\emph on
+invalid
+\emph default
+ (not measuring what they should measure).
+\end_layout
+
+\end_inset
+
+
+\end_layout
+
+\end_deeper
+\begin_layout Paragraph
+\begin_inset CommandInset label
+LatexCommand label
+name "par:Export-test-filtering"
+
+\end_inset
+
+Export test filtering
+\end_layout
+
+\begin_layout Standard
+The assignment of a label to a test is controlled by a set of files with
+ regular expressions that are matched against the test names.
+\end_layout
+
+\begin_layout Description
+ignoredTests (small file)
+\begin_inset Newline newline
+\end_inset
+
+Tests selected here are withdrawn in the configuration step (cf.
+
+\begin_inset CommandInset ref
+LatexCommand ref
+reference "par:Configuring-ctests"
+
+\end_inset
+
+).
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Input Test of any export combination
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Output Stop if tests not selected here
+\end_layout
+
+\end_deeper
+\begin_layout Description
+unreliableTests: Tests selected pass or fail dependent on the system where
+ the test is run.
+ Selected tests gain the label 'unreliable'.
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Input Each test which passed 'ignoredTests'
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Output Stop if test selected, gain label 'unreliable'.
+\end_layout
+
+\end_deeper
+\begin_layout Description
+invertedTests
+\begin_inset space \space{}
+\end_inset
+
+
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Input Each test which passed 'unreliableTests'
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Output Stop if not selected, gain test-property 'WILL_FAIL' (i.e.
+ tests are reported as failing if the export works without error.) If no
+ subselection applies, gain labels 'export' and 'inverted'.
+\end_layout
+
+\begin_layout Standard
+The following filter perfoms a subselection of 'invertedTests':
+\end_layout
+
+\begin_layout Description
+suspendedTests Tests selected here gain the label 'suspended' but _not_
+ 'export' or 'inverted', although in ctest they remain inverted.
+ ('ctest' knows only 'inverted' or not, labels are used only for test selection)
+\end_layout
+
+\begin_deeper
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Input Each test selected by 'invertedTests'
+\end_layout
+
+\begin_layout Labeling
+\labelwidthstring 00.00.0000
+Output Selected test gains label 'suspended'.
+
+\end_layout
+
+\end_deeper
+\end_deeper
+\begin_layout Standard
+The following table may clarify label assignement
+\end_layout
+
+\begin_layout Standard
+\begin_inset Tabular
+<lyxtabular version="3" rows="7" columns="9">
+<features tabularvalignment="middle">
+<column alignment="left" valignment="top" width="0pt">
+<column alignment="left" valignment="top" width="0pt">
+<column alignment="left" valignment="top" width="0pt">
+<column alignment="left" valignment="top" width="0pt">
+<column alignment="center" valignment="top">
+<column alignment="center" valignment="top">
+<column alignment="center" valignment="top">
+<column alignment="center" valignment="top">
+<column alignment="center" valignment="top">
+<row>
+<cell multicolumn="1" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Test matching pattern in file:
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="1" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Assigned label
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multicolumn="2" alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+test property
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+ignoredTests
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+unreliableTests
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+invertedTests
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+suspendedTests
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+export
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+unreliable
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+inverted
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+suspended
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Yes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell multirow="3" alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+No
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Yes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
++
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell multirow="4" alignment="left" valignment="middle" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+No
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="3" alignment="left" valignment="middle" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Yes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+Yes
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+-
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
++
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" rightline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+WILL_FAIL
+\end_layout
+
+\end_inset
+</cell>
+</row>
+<row>
+<cell multirow="4" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="left" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell multirow="4" alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="left" valignment="top" topline="true" bottomline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
+No
+\end_layout
+
+\end_inset
+</cell>
+<cell alignment="center" valignment="top" topline="true" leftline="true" usebox="none">
+\begin_inset Text
+
+\begin_layout Plain Layout
++
+\end_layout