X-Git-Url: https://git.lyx.org/gitweb/?a=blobdiff_plain;f=lib%2Fdoc%2FDevelopment.lyx;h=eb2917a6362ff6fa7ba0e8473c95b60d9a1564a5;hb=693ca643fc1e6e9d797d8a074285136987b82add;hp=ab79cc7fcd6e71e0cb8e13147aac34ae145e2eb8;hpb=1f9640e71ffb2c5f7184dd2378960af16736cab1;p=lyx.git diff --git a/lib/doc/Development.lyx b/lib/doc/Development.lyx index ab79cc7fcd..eb2917a636 100644 --- a/lib/doc/Development.lyx +++ b/lib/doc/Development.lyx @@ -1,8 +1,9 @@ #LyX 2.2 created this file. For more info see http://www.lyx.org/ -\lyxformat 502 +\lyxformat 504 \begin_document \begin_header -\origin unavailable +\save_transient_properties true +\origin /systemlyxdir/doc/ \textclass scrartcl \options BCOR8mm,captions=tableheading \use_default_options false @@ -10,21 +11,6 @@ logicalmkup \end_modules \maintain_unincluded_children false -\begin_local_layout -Format 7 -InsetLayout CharStyle:MenuItem -LyxType charstyle -LabelString menu -LatexType command -LatexName menuitem -Font -Family Sans -EndFont -Preamble -\newcommand*{\menuitem}[1]{{\sffamily #1}} -EndPreamble -End -\end_local_layout \language english \language_package default \inputencoding auto @@ -40,8 +26,8 @@ End \font_sf_scale 100 100 \font_tt_scale 100 100 \graphics default -\default_output_format default -\output_sync 0 +\default_output_format pdf2 +\output_sync 1 \bibtex_command default \index_command default \paperfontsize 12 @@ -174,7 +160,7 @@ development source code distribution. This document is not translated, since the development language of \SpecialChar LyX is - english. + English. If you want to use \SpecialChar LyX you don't need to read this manual. However, if you want to learn more about how \SpecialChar LyX @@ -301,7 +287,7 @@ math package Any new math package that is automatically loaded needs a file format update. The reason for this is that there is no true ERT inset for math formulas: - Each command is parsed, and if a user happens to defne a local command + Each command is parsed, and if a user happens to define a local command with the same name as a command that triggers an automatic load of a package, he needs to be able to switch off the automatic loading of that package. This switch is stored by the @@ -323,6 +309,13 @@ If you are still unsure, please ask on the development list. \begin_layout Subsection How to update the file format number of .lyx files +\begin_inset CommandInset label +LatexCommand label +name "subsec:update_lyx_files" + +\end_inset + + \end_layout \begin_layout Standard @@ -339,6 +332,12 @@ Implement and test the new feature, including the reading and writing of \end_layout \begin_layout Enumerate +\begin_inset CommandInset label +LatexCommand label +name "enu:Describe_format" + +\end_inset + Describe the new format in \begin_inset Flex Code status collapsed @@ -392,6 +391,12 @@ lib/lyx2lyx/LyX.py \end_layout \begin_layout Enumerate +\begin_inset CommandInset label +LatexCommand label +name "enu:Add-an-entry" + +\end_inset + Add an entry to both format lists (for conversion and reversion) in \begin_inset Newline newline \end_inset @@ -510,1263 +515,3455 @@ It would be nice if you could create a .lyx test file which contains instances so please ask on the development list if you want to create one. \end_layout -\begin_layout Subsection -Backporting new styles to the stable version -\end_layout +\begin_layout Enumerate +\begin_inset CommandInset label +LatexCommand label +name "enu:updatefiles" -\begin_layout Standard -Starting with the stable \SpecialChar LyX - 2.1 branch, there is a mechanism in place to backport - new styles to the stable version without the need to update the file format. - The basic idea is that the new style definition is automatically copied - to the document preamble, so that it can even be used by older minor revisions - that did not yet include the style. - To backport a new style to the stable version, the following steps are - needed: -\end_layout +\end_inset -\begin_layout Enumerate -Add the line -\begin_inset Flex Code +Update LyX's .lyx documentation files to the new format. + The developer who makes the change knows best what changes to expect when + inspecting the resulting diff. + Because of this, you might be able to catch a bug in the lyx2lyx code that + updates the format just by taking a quick scan through the large diff that + is the result +\begin_inset Note Note status collapsed \begin_layout Plain Layout -ForceLocal -1 +Another advantage is that if later we suspect a bug in lyx2lyx we can easily + see which layout update made an unexpected change by looking at the git + log of a .lyx file that suffers the problem. \end_layout \end_inset - to the style definition in the development version. -\end_layout - -\begin_layout Enumerate -Copy the style definition to the stable version, but use +. + To do this, first make sure that there are no changes to the git repository + that you will not want to commit (this is needed because it will be convenient + to commit with the command \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ForceLocal 1 +git commit -a \end_layout \end_inset - instead. - If needed adjust the format to the one used by the stable version (see - the customization manual for details of the layout file format). -\end_layout - -\begin_layout Enumerate -For each update of the style in a later stable version, increase the argument - of +). + Then run the following command in the root folder of the source: \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ForceLocal +python development/tools/updatedocs.py \end_layout \end_inset - by one (in the stable version, the development version should not be touched). -\end_layout - -\begin_layout Standard -For details about the +. + Look at the resulting changes using the command \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ForceLocal +git diff \end_layout \end_inset - flag see the customization manual. - No +. + If anything looks surprising, please investigate. + Keep in mind that the case of \begin_inset Flex Code status collapsed \begin_layout Plain Layout -lyx2lyx +LFUNs.lyx \end_layout \end_inset - support is needed for backported styles: Since the style of the development - version has an infinite version number, it will always be used. - Furthermore, since its version number is less than one, the style will - not be written anymore to the document header for files saved by the new - version. + is special, because it is first generated with +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +gen_lfuns.py \end_layout -\begin_layout Standard -\begin_inset Newpage newpage \end_inset + before being converted to the latest format. + Finally, commit using +\begin_inset Flex Code +status collapsed +\begin_layout Plain Layout +git commit -a \end_layout -\begin_layout Section -Tests -\end_layout +\end_inset -\begin_layout Standard -Automated tests are an important tool to detect bugs and regressions in - software development. - Some projects like gcc even require each bug fix to be accompanied by a - test case for the automatic test suite, that would detect this bug. - Testing interactive features automatically is of course very hard, but - core functionality like document import and export can be tested quite - easily, and some tests of this kind exist. +. \end_layout \begin_layout Subsection -\SpecialChar LyX - tests +Updating the file format number of layout files \end_layout \begin_layout Standard -Some tests are located in the +See step +\begin_inset CommandInset ref +LatexCommand ref +reference "enu:updatefiles" + +\end_inset + + in section +\begin_inset CommandInset ref +LatexCommand ref +reference "subsec:update_lyx_files" + +\end_inset + + but instead of the \begin_inset Flex Code status collapsed \begin_layout Plain Layout -development/autotests +updatedocs.py \end_layout \end_inset - subfolder of the \SpecialChar LyX - source code distribution. -\end_layout - -\begin_layout Subsubsection -Running the tests -\end_layout - -\begin_layout Standard -cmake is required to run the \SpecialChar LyX - tests, running them is not implemented for - autotools. - The \SpecialChar LyX - tests can be run by the commands + command, use this command: \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ctest +python development/tools/updatelayouts.py \end_layout \end_inset - (all platforms) or +. +\end_layout + +\begin_layout Standard +Note that we do not update the local layout of our \begin_inset Flex Code status collapsed \begin_layout Plain Layout -make test +.lyx \end_layout \end_inset - (when using a make based build system and not MSVC) in the + files because users would then not be able to export to older formats. + For example, if a 2.2.0 user exported a template to 2.1.x format and tried + to open the file in LyX 2.1.x, there would be an error because the file would + contain a local layout whose format is too new. + The root reason for this is that we do not support converting layouts to + older layout formats, as we do for the \begin_inset Flex Code status collapsed \begin_layout Plain Layout -autotests +.lyx \end_layout \end_inset - subfolder of the build directory. + file format. \end_layout \begin_layout Subsection -tex2lyx tests +Updating the file format number of bind/ui files \end_layout \begin_layout Standard -The tex2lyx tests are located in the +A change to the functionality of existing LFUNs can require a conversion + of \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx/test +.bind \end_layout \end_inset - subfolder of the \SpecialChar LyX - source code distribution. - The actual testing is performed by the simple python script + and \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx/test/runtests.py +.ui \end_layout \end_inset -. - Each test consists of two files: + files, and therefore an increment of the LFUN format, as well as a conversion + of Info insets in \begin_inset Flex Code status collapsed \begin_layout Plain Layout -.tex +.lyx \end_layout \end_inset - contains the \SpecialChar LaTeX - code that should be tested. - -\begin_inset Flex Code -status collapsed + files for manuals. + The latter cannot be done automatically and requires also a \SpecialChar LyX + format increase + (think of e.g. + someone who might have made a set of \SpecialChar LyX + teaching manuals for use in their + own group) +\begin_inset Foot +status open \begin_layout Plain Layout -.lyx.lyx +\begin_inset Flex URL +status open + +\begin_layout Plain Layout + +http://www.lyx.org/trac/ticket/9794 \end_layout \end_inset - contains the expected output of tex2lyx. - When a test is run, the actual produced output is compared with the stored - reference output. - The test passes if both are identical. - The test machinery is also able to generate a file -\begin_inset Flex Code -status collapsed -\begin_layout Plain Layout -.lyx.tex \end_layout \end_inset - by exporting the produced .lyx file with \SpecialChar LyX - again. - This may be useful for roundtrip comparisons. -\end_layout - -\begin_layout Subsubsection -Running the tests +. \end_layout -\begin_layout Standard -The tex2lyx tests can be run in several ways. - When in the +\begin_layout Enumerate +Increment the LFUN file format number in \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx +src/LyXAction.h \end_layout \end_inset - subfolder of the build directory, the commands +. +\end_layout + +\begin_layout Enumerate +Implement the LFUN conversion in \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ctest +lib/scripts/prefs2prefs_lfuns.py \end_layout \end_inset - (cmake, all platforms), -\begin_inset Flex Code -status collapsed -\begin_layout Plain Layout -make test \end_layout +\begin_layout Enumerate +See step +\begin_inset CommandInset ref +LatexCommand ref +reference "enu:updatefiles" + \end_inset - (cmake, when using a make based build system and not MSVC) or + in section +\begin_inset CommandInset ref +LatexCommand ref +reference "subsec:update_lyx_files" + +\end_inset + + but instead of the \begin_inset Flex Code status collapsed \begin_layout Plain Layout -make alltests +updatedocs.py \end_layout \end_inset - (autotools) will run the tex2lyx tests. - Alternatively, in the root of the build directory, the command + command, use this command: \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ctest -R tex2lyx +bash development/tools/updatelfuns.sh \end_layout -\end_inset - - runs all tests whose names match the regex -\begin_inset Quotes eld -\end_inset - -tex2lyx -\begin_inset Quotes erd \end_inset . - Another way to run the tex2lyx tests in the root build directory is to - instead use the command +\end_layout + +\begin_layout Enumerate +Update Info insets in \begin_inset Flex Code status collapsed \begin_layout Plain Layout -ctest -L '(cmplyx|roundtrip)' +.lyx \end_layout \end_inset -, which runs all tests categorized with the label -\begin_inset Quotes eld -\end_inset + files. + To do so, increment the \SpecialChar LyX + format and proceed as in +\begin_inset CommandInset ref +LatexCommand ref +reference "subsec:update_lyx_files" -roundtrip -\begin_inset Quotes erd \end_inset - or -\begin_inset Quotes eld -\end_inset +, steps +\begin_inset CommandInset ref +LatexCommand ref +reference "enu:Describe_format" -cmplyx -\begin_inset Quotes erd \end_inset -. - If a test fails, the differences between the expected and actual results - are output in unified diff format. -\end_layout - -\begin_layout Subsubsection -Updating test references -\begin_inset CommandInset label -LatexCommand label -name "sec:Updating-test-references" +- +\begin_inset CommandInset ref +LatexCommand ref +reference "enu:updatefiles" \end_inset +. + In the lyx2lyx implementation ( +\begin_inset CommandInset ref +LatexCommand ref +reference "enu:Add-an-entry" -\end_layout - -\begin_layout Standard -In some cases a changed tex2lyx output is not a test failure, but wanted, - e. -\begin_inset space \thinspace{} -\end_inset - -g. -\begin_inset space \space{} \end_inset -if a tex2lyx bug was fixed, or a new feature was added. - In these cases the stored references need to be updated. - To do so if using autotools, call +th step), implement a conversion similar to the one in \begin_inset Flex Code status collapsed \begin_layout Plain Layout -make updatetests +prefs2prefs_lfuns.py \end_layout \end_inset - in the + above, as well as a corresponding reversion; for this one can use \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx +convert_info_insets \end_layout \end_inset - subdirectory of the build directory. - If instead using CMake, call + from \begin_inset Flex Code status collapsed \begin_layout Plain Layout -make updatetex2lyxtests +lib/lyx2lyx/lyx2lyx_tools.py \end_layout \end_inset - in the build directory or in the +. + +\end_layout + +\begin_layout Subsection +Backporting new styles to the stable version +\end_layout + +\begin_layout Standard +Starting with the stable \SpecialChar LyX + 2.1 branch, there is a mechanism in place to backport + new styles to the stable version without the need to update the file format. + The basic idea is that the new style definition is automatically copied + to the document preamble, so that it can even be used by older minor revisions + that did not yet include the style. + To backport a new style to the stable version, the following steps are + needed: +\end_layout + +\begin_layout Enumerate +Add the line \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx/test +ForceLocal -1 \end_layout \end_inset - subdirectory of the build directory. -\begin_inset Foot + to the style definition in the development version. +\end_layout + +\begin_layout Enumerate +Copy the style definition to the stable version, but use +\begin_inset Flex Code status collapsed \begin_layout Plain Layout -Note that this is a case where a make target in the build directory can - affect the source directory, which might not be advisable. +ForceLocal 1 \end_layout \end_inset - On Windows do the following: + instead. + If needed adjust the format to the one used by the stable version (see + the customization manual for details of the layout file format). \end_layout -\begin_layout Itemize -Assure that the path to the python.exe is in your system PATH variable. +\begin_layout Enumerate +For each update of the style in a later stable version, increase the argument + of +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ForceLocal \end_layout -\begin_layout Itemize -Double-click on the file +\end_inset + + by one (in the stable version, the development version should not be touched). +\end_layout + +\begin_layout Standard +For details about the \begin_inset Flex Code status collapsed \begin_layout Plain Layout -updatetex2lyxtests.vcxproj +ForceLocal \end_layout \end_inset - in the build directory or in the + flag see the customization manual. + No \begin_inset Flex Code status collapsed \begin_layout Plain Layout -src/tex2lyx/test +lyx2lyx \end_layout \end_inset - subdirectory of your build directory. + support is needed for backported styles: Since the style of the development + version has an infinite version number, it will always be used. + Furthermore, since its version number is less than one, the style will + not be written anymore to the document header for files saved by the new + version. \end_layout -\begin_layout Itemize -In the appearing MSVC program right-click on the project -\family sans -updatetex2lyxtests -\family default - in the project explorer and chose -\family sans -Create -\family default -. +\begin_layout Standard +\begin_inset Newpage newpage +\end_inset + + +\end_layout + +\begin_layout Section +Tests \end_layout \begin_layout Standard -For convenience, these commands also produce re-exported roundtrip .lyx.tex - files. - Please examine the changed output carefully before committing the changed - files to the repository: Since the test machinery does not do a roundtrip - test .tex -\begin_inset Formula $\Rightarrow$ +Automated tests are an important tool to detect bugs and regressions in + software development. + Some projects like gcc even require each bug fix to be accompanied by a + test case for the automatic test suite, that would detect this bug. + Testing interactive features automatically is of course very hard, but + core functionality like document import and export can be tested quite + easily, and some tests of this kind exist. +\end_layout + +\begin_layout Subsection +unit tests +\end_layout + +\begin_layout Standard +There are attempts to set up a suite of unit tests for LyX. +\end_layout + +\begin_layout Standard +TODO: describe what is done and what is still to do. +\end_layout + +\begin_layout Subsection +tex2lyx tests +\end_layout + +\begin_layout Standard +The tex2lyx tests are located in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test +\end_layout + \end_inset - .lyx -\begin_inset Formula $\Rightarrow$ + subfolder of the \SpecialChar LyX + source code distribution. + The actual testing is performed by the simple python script +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test/runtests.py +\end_layout + \end_inset - .tex, and does not compare the produced dvi or pdf output, it assumes that - the stored .lyx reference produces correct output if processed by \SpecialChar LyX . - There is only one chance to detect wrong output: before committing a new - reference. - Once it is committed, it is quite difficult to verify whether it is correct. + Each test consists of two files: +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +.tex \end_layout -\begin_layout Standard -Please -\emph on -do not -\emph default - update the test references by opening them with \SpecialChar LyX - or directly running lyx2lyx - on them. - This would not work, since lyx2lyx and \SpecialChar LyX - produce slightly different files - regarding insignificant whitespace and line breaks. +\end_inset + + contains the \SpecialChar LaTeX + code that should be tested. + +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +.lyx.lyx +\end_layout + +\end_inset + + contains the expected output of tex2lyx. + When a test is run, the actual produced output is compared with the stored + reference output. + The test passes if both are identical. + The test machinery is also able to generate a file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +.lyx.tex +\end_layout + +\end_inset + + by exporting the produced .lyx file with \SpecialChar LyX + again. + This may be useful for roundtrip comparisons. \end_layout \begin_layout Subsubsection -Adding a new test +Running the tests \end_layout \begin_layout Standard -In many cases tests for new features may be added to one of the existing - test files, but sometimes this is not possible or not wanted. - Then a new test file needs to be added: +The tex2lyx tests can be run in several ways. + When in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx +\end_layout + +\end_inset + + subfolder of the build directory, the commands +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + + (cmake, all platforms), +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +make test +\end_layout + +\end_inset + + (cmake, when using a make based build system and not MSVC) or +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +make alltests +\end_layout + +\end_inset + + (autotools) will run the tex2lyx tests. + Alternatively, in the root of the build directory, the command +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -R tex2lyx +\end_layout + +\end_inset + + runs all tests whose names match the regex +\begin_inset Quotes eld +\end_inset + +tex2lyx +\begin_inset Quotes erd +\end_inset + +. + Another way to run the tex2lyx tests in the root build directory is to + instead use the command +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L '(cmplyx|roundtrip)' +\end_layout + +\end_inset + +, which runs all tests categorized with the label +\begin_inset Quotes eld +\end_inset + +roundtrip +\begin_inset Quotes erd +\end_inset + + or +\begin_inset Quotes eld +\end_inset + +cmplyx +\begin_inset Quotes erd +\end_inset + +. + If a test fails, the differences between the expected and actual results + are output in unified diff format. +\end_layout + +\begin_layout Subsubsection +Updating test references +\begin_inset CommandInset label +LatexCommand label +name "sec:Updating-test-references" + +\end_inset + + +\end_layout + +\begin_layout Standard +In some cases a changed tex2lyx output is not a test failure, but wanted, + e. +\begin_inset space \thinspace{} +\end_inset + +g. +\begin_inset space \space{} +\end_inset + +if a tex2lyx bug was fixed, or a new feature was added. + In these cases the stored references need to be updated. + To do so if using autotools, call +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +make updatetests +\end_layout + +\end_inset + + in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx +\end_layout + +\end_inset + + subdirectory of the build directory. + If instead using CMake, call +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +make updatetex2lyxtests +\end_layout + +\end_inset + + in the build directory or in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test +\end_layout + +\end_inset + + subdirectory of the build directory. +\begin_inset Foot +status collapsed + +\begin_layout Plain Layout +Note that this is a case where a make target in the build directory can + affect the source directory, which might not be advisable. +\end_layout + +\end_inset + + On Windows do the following: +\end_layout + +\begin_layout Itemize +Assure that the path to the python.exe is in your system PATH variable. +\end_layout + +\begin_layout Itemize +Double-click on the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +updatetex2lyxtests.vcxproj +\end_layout + +\end_inset + + in the build directory or in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test +\end_layout + +\end_inset + + subdirectory of your build directory. +\end_layout + +\begin_layout Itemize +In the appearing MSVC program right-click on the project +\family sans +updatetex2lyxtests +\family default + in the project explorer and chose +\family sans +Create +\family default +. +\end_layout + +\begin_layout Standard +For convenience, these commands also produce re-exported roundtrip .lyx.tex + files. + Please examine the changed output carefully before committing the changed + files to the repository: Since the test machinery does not do a roundtrip + test .tex +\begin_inset Formula $\Rightarrow$ +\end_inset + + .lyx +\begin_inset Formula $\Rightarrow$ +\end_inset + + .tex, and does not compare the produced dvi or pdf output, it assumes that + the stored .lyx reference produces correct output if processed by \SpecialChar LyX +. + There is only one chance to detect wrong output: before committing a new + reference. + Once it is committed, it is quite difficult to verify whether it is correct. +\end_layout + +\begin_layout Standard +Please +\emph on +do not +\emph default + update the test references by opening them with \SpecialChar LyX + or directly running lyx2lyx + on them. + This would not work, since lyx2lyx and \SpecialChar LyX + produce slightly different files + regarding insignificant whitespace and line breaks. +\end_layout + +\begin_layout Subsubsection +Adding a new test +\end_layout + +\begin_layout Standard +In many cases tests for new features may be added to one of the existing + test files, but sometimes this is not possible or not wanted. + Then a new test file needs to be added: +\end_layout + +\begin_layout Enumerate +Create the new file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test/.tex +\end_layout + +\end_inset + + and run tex2lyx in roundtrip mode to produce the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test/.lyx.lyx +\end_layout + +\end_inset + +. + This file will be the new reference. +\end_layout + +\begin_layout Enumerate +Once you confirmed that the tex2lyx output is correct, add the new files + to the corresponding lists in +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test/runtests.py +\end_layout + +\end_inset + +, +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/Makefile.am +\end_layout + +\end_inset + + and +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +src/tex2lyx/test/CMakeLists.txt +\end_layout + +\end_inset + +. +\end_layout + +\begin_layout Enumerate +Commit the changes to the repository, or send a patch to the development + list and ask for committing if you do not have commit rights. +\end_layout + +\begin_layout Subsection +ctest automatic tests +\end_layout + +\begin_layout Standard +Some tests are located in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/ +\end_layout + +\end_inset + + subfolder of the \SpecialChar LyX + source code distribution. + +\begin_inset Foot +status open + +\begin_layout Plain Layout +The README document in this folder only describes the +\begin_inset Quotes eld +\end_inset + +keytests +\begin_inset Quotes erd +\end_inset + + subset of autotests! +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +These tests can be run by the commands +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + + in the +\emph on + build directory +\emph default + (all platforms) or (when using a make based build system and not MSVC) + +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +make test +\end_layout + +\end_inset + + in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +autotests/ +\end_layout + +\end_inset + + subfolder of the +\emph on + build directory +\emph default +. + The test logs are written to the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +Testing/Temporary/ +\end_layout + +\end_inset + + subfolder of the +\emph on + +\emph default +build directory. + +\end_layout + +\begin_layout Subsubsection +Export tests +\end_layout + +\begin_layout Standard +The export tests are integration tests. + They take longer to run and are more likely to break than the tex2lyx tests. + Nevertheless, they have caught many regressions and without a better alternativ +e it is important to keep them up-to-date and understand how they work. +\end_layout + +\begin_layout Standard +The export tests +\begin_inset Quotes eld +\end_inset + +reuse +\begin_inset Quotes erd +\end_inset + + documentation, template, and example documents. + In addition, there are a number of dedicated sample documents in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +autotests/export/ +\end_layout + +\end_inset + + subfolder of the \SpecialChar LyX + source code distribution. + All samples are (after copying and eventual processing by scripts) exported + to various output formats via the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +—export-to +\end_layout + +\end_inset + + command line option. + The tests checks for errors reported by LyX. + (However, error-free export is no guarantee for an error-free output document.) +\end_layout + +\begin_layout Paragraph +Expectations of LyX developers +\end_layout + +\begin_layout Standard +Because the export tests are integration tests and take a long time to run, + LyX developers are rarely expected to run all of the tests. + Here are some good practices to follow by developers: +\end_layout + +\begin_layout Itemize +When making a non-trivial change to a .layout file, run the export and layout + tests corresponding with that .layout file. +\end_layout + +\begin_layout Itemize +When making non-trivial changes to a .lyx file, run the export tests correspondin +g to that .lyx file. + +\begin_inset Foot +status collapsed + +\begin_layout Plain Layout +This rule is due to revision. + +\end_layout + +\begin_layout Plain Layout +There is an objection from the documentation maintainer that working on + the documentation must not be complicated by having to consider non-standard + exports. +\end_layout + +\begin_layout Itemize +successful compiling/testing an edited documentation file with pdflatex + suffices to ensure it can be commited, not tests with other exports are + required. +\end_layout + +\begin_layout Plain Layout +If sudden failures with other exports due to “half-tested” documentation + updates are a problem for the test maintainer, the test suite should use + copies that are +\end_layout + +\begin_layout Itemize +copied to a cache dir (autotests/samples/doc/, say) but not changed, +\end_layout + +\begin_layout Itemize +updated regularely (but on a time chosen by the test suite maintainer) from + the originals in lib/doc/ +\end_layout + +\begin_layout Plain Layout +This way, +\end_layout + +\begin_layout Itemize +no test will fail due to ongoing work on documentation, +\end_layout + +\begin_layout Itemize +the documentation is still tested in full (with some delay), +\end_layout + +\begin_layout Itemize +failures with non-default export can be examined and handled accordingly + in one run with the cache update, +\end_layout + +\begin_layout Itemize +“interesting failures” (like the nested-language+polyglossia problem in + es/Customization can be separated and moved into dedicated test samples. +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Itemize +When making non-trivial changes to LyX's \SpecialChar LaTeX + export code (e.g. + touching the encoding code or package handling code that you expect will + change the exported \SpecialChar LaTeX + in some way): +\end_layout + +\begin_deeper +\begin_layout Standard +\paragraph_spacing single +Consider running all of the export tests before and after your change. + If there are differences, please reconcile these (i.e. + fix the bug or fix the tests) +\emph on +before +\emph default + committing. + Ask for help if you're not sure what to. +\end_layout + +\begin_layout Standard +If you do not want to run the tests, +\end_layout + +\begin_layout Itemize +post the patch on the list and others will run the tests and eventually + ask for fixes, or +\end_layout + +\begin_layout Itemize +commit, but be prepared to fix eventually arising problems or to revert + the commit if there is no easy fix. +\end_layout + +\end_deeper +\begin_layout Itemize +Understand how to interpret test failures. + If your commit is found to have broken a test, you should be able to interpret + the test results when made aware of them. + See Section +\begin_inset CommandInset ref +LatexCommand ref +reference "subsec:Interpreting-export-tests" + +\end_inset + +. +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:export-test-output-formats" + +\end_inset + +Output formats +\end_layout + +\begin_layout Standard +The following output formats are currently tested for each sample document + (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Export-test-filtering" + +\end_inset + + for exceptions): +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +LyX: +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +lyx16 LyX 1.6 file format (lyx2lyx) +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +lyx21 LyX 2.1 file format (lyx2lyx) +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +xhtml LyXHTML (native LyX HTML export) +\end_layout + +\end_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +LyX +\begin_inset space ~ +\end_inset + ++ +\begin_inset space ~ +\end_inset + +LaTeX: +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +dvi DVI (8-bit latex) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +dvi3_texF DVI (LuaTeX with 8-bit TeX fonts) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +dvi3_systemF DVI (LuaTeX with Unicode fonts) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf2 PDF (pdflatex) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf4_texF PDF (XeTeX with 8-bit TeX fonts) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf4_systemF PDF (XeTeX with Unicode fonts) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf5_texF PDF (LuaTeX with 8-bit TeX fonts) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf5_systemF PDF (LuaTeX with Unicode fonts) +\end_layout + +\end_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +LyX +\begin_inset space ~ +\end_inset + ++ +\begin_inset space ~ +\end_inset + +LaTeX +\begin_inset space ~ +\end_inset + ++ +\begin_inset space ~ +\end_inset + +postprocessing: +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf DVI -> PS (dvips) -> PDF (ps2pdf) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdf3 DVI -> PDF (dvipdfm) +\end_layout + +\end_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +not +\begin_inset space ~ +\end_inset + +tested: (or only if set as default output format in the document source) +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +latex LaTeX (plain) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +luatex LaTeX (LuaTeX) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +dviluatex LaTeX (dviluatex) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +pdflatex LaTeX (pdflatex) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +platex LaTeX (pLaTeX) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +xetex LaTeX (XeTeX) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +eps3 EPS (encapsulated Postscript) (cropped) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +ps DVI -> Postscript (dvips) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +odf +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +text (nor text2, ..., text4) +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +textparagraph +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +word +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +word2 +\end_layout + +\begin_layout Labeling +\labelwidthstring pdf5msystemFM +wordhtml +\end_layout + +\end_deeper +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Configuring-ctests" + +\end_inset + +Configuring the tests +\end_layout + +\begin_layout Standard +To enable the export autotests, add the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-DLYX_ENABLE_EXPORT_TESTS=ON +\end_layout + +\end_inset + + flag. + For example: +\end_layout + +\begin_layout Standard +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +cmake -DLYX_ENABLE_EXPORT_TESTS=ON /path/to/source +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\noindent +This flag will increase the time for the cmake command by several seconds, + mainly because of the process of inverting tests (see Section +\begin_inset CommandInset ref +LatexCommand ref +reference "subsec:Interpreting-export-tests" + +\end_inset + +). +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:ctest-options" + +\end_inset + +Running the tests +\end_layout + +\begin_layout Standard +To run all tests, in the build directory simply run the command +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + +. + A full, up-to-date TeXLive installation is recommended to run the tests. + Otherwise, some tests will fail. + Tests with additional requirements are labeled +\begin_inset Quotes eld +\end_inset + +unreliable:nonstandard +\begin_inset Quotes erd +\end_inset + +. + +\end_layout + +\begin_layout Standard +To run only some of the tests, use command line options (see examples below): +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-R +\end_layout + +\end_inset + + Run only the tests whose names match the given regular expression. +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-L +\end_layout + +\end_inset + + Run only the tests whose labels match the given regular expression. + A test may have more that one label. + +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-E +\end_layout + +\end_inset + + Exclude the tests whose names match the given regular expression. +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-LE +\end_layout + +\end_inset + + Exclude the tests whose labels match the given regular expression. + Cannot be combined with +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-L +\end_layout + +\end_inset + +. +\end_layout + +\begin_layout Standard +The following options help to find good selection patterns: +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-N +\end_layout + +\end_inset + + List the tests that would be run but not actually run them. + +\end_layout + +\begin_deeper +\begin_layout Standard +Useful in conjunction with the -R, -L, -E and -LE options, e.g., if you want + to know how many tests there are or whether your +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + regular expression did what you expected. +\end_layout + +\end_deeper +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +\SpecialChar nobreakdash +\SpecialChar nobreakdash +print-labels +\end_layout + +\end_inset + + print the list of all labels associated with the test set. + Can also be combined with -R, -L, -E, ... + +\end_layout + +\begin_layout Standard +Other useful options are: +\end_layout + +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-j +\end_layout + +\end_inset + + Run the tests in parallel using the given number of jobs. +\end_layout + +\begin_deeper +\begin_layout Standard +We are still working on getting the tests to run in parallel. + However, when running the tests in parallel, sometimes tests fail that + pass when run sequentially. + A reasonable approach is to first run the tests in parallel and then run + the failed tests sequentially. +\end_layout + +\begin_layout Standard +For example, to run 8 jobs at a time: +\end_layout + +\begin_layout Standard +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -j8 +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest \SpecialChar nobreakdash +\SpecialChar nobreakdash +rerun-failed +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Standard +When specifying a subset of the tests (e.g. + using +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +\SpecialChar nobreakdash +R +\end_layout + +\end_inset + +), the same subset must be specified when using the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +\SpecialChar nobreakdash +\SpecialChar nobreakdash +rerun-failed +\end_layout + +\end_inset + + option because it is the test numbers that are used to index which tests + failed on the previous run. +\end_layout + +\begin_layout Standard +\noindent +Note that some tests cannot be run in parallel. + These tests are marked in the code with the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +\noindent +RUN_SERIAL ON +\end_layout + +\end_inset + + CMake property. +\end_layout + +\end_deeper +\begin_layout Labeling +\labelwidthstring -R +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +\SpecialChar nobreakdash +\SpecialChar nobreakdash +timeout +\end_layout + +\end_inset + + Set a global timeout on all tests that do not already have a timeout set + on them. +\end_layout + +\begin_deeper +\begin_layout Standard +There have been bugs in LyX and in \SpecialChar LaTeX + which cause compilation to hang, and + without a timeout a test might never stop (in one case there was even a + memory leak). + If a test times out, the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + + command exits with error, but you can distinguish between a timed out test + and a failed test in the output reported at the end of the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + + command. +\end_layout + +\end_deeper +\begin_layout Standard +See the manual ( +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +man ctest +\end_layout + +\end_inset + +) the full list of command line options. +\end_layout + +\begin_layout Paragraph +Examples +\end_layout + +\begin_layout Itemize +run only the export tests: +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L export +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Itemize +run inverted tests: +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L "inverted|suspended" +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Itemize +list all export tests which match any of the labelling patterns: +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -N -R " +\backslash +..*_export/" +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Itemize +exclude rarely used output formats and post-processing tests +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L export -E "_(texF|dvi3|pdf3?)" +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "subsec:Interpreting-export-tests" + +\end_inset + +Interpreting the export test results +\end_layout + +\begin_layout Standard +A test can fail for several reasons, not all of them bad. +\end_layout + +\begin_layout Enumerate +A new or edited sample document may be incompatible with some output formats. +\end_layout + +\begin_layout Enumerate +A dependency is not met (e.g. + the \SpecialChar LaTeX + class file). + One hint that this is the case is that the corresponding +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +check_load +\end_layout + +\end_inset + + test will likely also fail. +\end_layout + +\begin_layout Enumerate +An inverted test fails to fail (i.e. + export that previously failed now works). +\end_layout + +\begin_layout Enumerate +An external dependency was updated (e.g. + \SpecialChar TeX + Live). +\end_layout + +\begin_layout Enumerate +A recent code change introduced a bug. +\end_layout + +\begin_layout Enumerate +\begin_inset CommandInset label +LatexCommand label +name "enu:exposed" + +\end_inset + +A change in a document exposed a previously unknown bug or an incompatibility + with an export format (e.g. + Lua\SpecialChar LaTeX +). +\end_layout + +\begin_layout Standard +Because the .lyx files are exported in several formats, it is not surprising + that many of the exports fail. + This expectation of failure is addressed by +\begin_inset Quotes eld +\end_inset + +inverting +\begin_inset Quotes erd +\end_inset + + the tests, that is, by marking the test as +\begin_inset Quotes eld +\end_inset + +passing +\begin_inset Quotes erd +\end_inset + + if the export exits with error and as +\begin_inset Quotes eld +\end_inset + +failing +\begin_inset Quotes erd +\end_inset + + if the export succeeds +\emph on +. + +\emph default + It follows that these expected failures will not show up as failed tests + in the test results and thus will not pollute the +\begin_inset Quotes eld +\end_inset + +good +\begin_inset Quotes erd +\end_inset + + tests. + If the export actually succeeds, then the test will fail. + The purpose of this failure is to get your attention—something has changed, + possibly for the better. +\end_layout + +\begin_layout Standard +We try to document why a test is inverted or ignored. + See the comment (prefixed with +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +# +\end_layout + +\end_inset + +) above the block in which the test is listed as inverted or ignored in + the files +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/suspiciousTests +\end_layout + +\end_inset + +, +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/unreliableTests +\end_layout + +\end_inset + + and +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/ignoredTests +\end_layout + +\end_inset + +. + +\end_layout + +\begin_layout Standard +A good question is why do we enable the tests for non-default formats? The + answer is that if a non-default route is broken it is often because a bug + was introduced in LyX and not because a document-specific change was made + that is not supported by the route. + In other words, there is a high signal/noise ratio in the export tests + for some non-default formats. + +\end_layout + +\begin_layout Standard +When a test or several tests fail, consider checking the files in the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +Testing/Temporary/ +\end_layout + +\end_inset + + subdirectory of your build directory. + In this subdirectory are three files: the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +LastTestsFailed.log +\end_layout + +\end_inset + + simply lists the tests that failed on your last +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest +\end_layout + +\end_inset + + command; the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +LastTest.log +\end_layout + +\end_inset + + file contains the output from the tests (and often has details explaining + why a test failed); and the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +CTestCostData.txt +\end_layout + +\end_inset + + file lists the times that it took to run the tests. +\end_layout + +\begin_layout Paragraph +What action should you take if a test fails? +\end_layout + +\begin_layout Standard +\paragraph_spacing single +It is always good to check manually why something fails and if it passes + if the PDF output is good. +\end_layout + +\begin_layout Itemize +Generally, if a change breaks compilation for the target format (for the + manuals pdf2) without solving some important other issue, +\emph on +fix or revert the commit +\emph default + that led to failure. +\end_layout + +\begin_layout Itemize +If it is not possible to (immediately) fix the failure but there are reasons + not to revert the commit (e.g. + it fixes another more important issue), +\emph on +invert +\emph default + the failing test case (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Inverted-tests" + +\end_inset + +). +\end_layout + +\begin_layout Itemize +If an +\emph on +inverted +\emph default + test case fails because the export now works, +\emph on +uninvert +\emph default + the test by removing the labeling pattern from +\begin_inset Quotes eld +\end_inset + +suspiciousTests +\begin_inset Quotes erd +\end_inset + +) (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Inverted-tests" + +\end_inset + +). +\end_layout + +\begin_layout Itemize +If the export did not fail previously but led to wrong output (PDF, say), + it is in fact an improvement when the test now fails, label it as +\begin_inset Quotes eld +\end_inset + +unreliable:wrong:output +\begin_inset Quotes erd +\end_inset + + ( +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Unreliable-tests" + +\end_inset + +). +\end_layout + +\begin_layout Itemize +In case of tests failing due to missing requirements (when only a subset + of TeXLive is installed or a test labeled +\begin_inset Quotes eld +\end_inset + +unreliable:nonstandard +\begin_inset Quotes erd +\end_inset + + fails), ignore the failure, ask for someone else to run the test, or install + the missing ressources and try again. +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Inverted-tests" + +\end_inset + +Inverted tests +\end_layout + +\begin_layout Standard +Test cases whose name matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/suspiciousTests +\end_layout + +\end_inset + + get the label +\emph on +inverted +\emph default +. + They get also the test property +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +WILL_FAIL +\end_layout + +\end_inset + +, i.e. + they are reported as failing if the export works without error +\begin_inset Flex URL +status collapsed + +\begin_layout Plain Layout + +https://cmake.org/cmake/help/v3.0/command/set_tests_properties.html +\end_layout + +\end_inset + +. +\end_layout + +\begin_layout Standard +Add failing cases to this file, if they cannot be solved +\begin_inset Quotes eld +\end_inset + +immediately +\begin_inset Quotes erd +\end_inset + + but it is expected that the export will work in a foreseeable future, e.g. + low priority issues like failures to export to a non-target format (for + the manuals everything except pdf2). +\end_layout + +\begin_layout Standard +The following sublabels are currently present in +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +suspiciousTests +\end_layout + +\end_inset + +: +\end_layout + +\begin_layout Description +todo test failures that require attention: +\end_layout + +\begin_deeper +\begin_layout Itemize +minor issues to explore and properly sort later, +\end_layout + +\begin_layout Itemize +easyfix issues, +\end_layout + +\begin_layout Itemize +LyX bugs to report at trac (move pattern to section "lyxbugs" once done). +\end_layout + +\end_deeper +\begin_layout Description +lyxbugs LyX bugs with a Trac number. +\end_layout + +\begin_layout Description +ert Export failures due to "raw" LaTeX use in ERT or preamble code. +\end_layout + +\begin_deeper +\begin_layout Standard +"Wontfix" if demonstrating correct use and OK in the default output format. +\end_layout + +\end_deeper +\begin_layout Description +texissues Export fails due to LaTeX limitations like non-ASCII characters + in verbatim or listings, incompatible packages, ... +\end_layout + +\begin_deeper +\begin_layout Standard +"Wontfix" if documents demonstrate correct use in the default output format: +\end_layout + +\begin_layout Itemize +If the source can be made more robust without becoming "hackish", fix the + source, +\end_layout + +\begin_layout Itemize +if LyX could be enhanced to care for a permanent TeX limitation, file a + ticket at trac and add a pattern under lyxbugs, +\end_layout + +\begin_layout Itemize +otherwise, add a pattern here. +\end_layout + +\end_deeper +\begin_layout Description +attic Documents in the attic. + (Kept for reference and format conversion test.) +\end_layout + +\begin_layout Subparagraph +suspended +\end_layout + +\begin_layout Standard +Test cases whose name additionally matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/suspendedTests +\end_layout + +\end_inset + + get the label +\emph on +suspended +\emph default +(instead of +\emph on +export +\emph default +and +\emph on + inverted +\emph default +). + This means they are not executed using +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L export +\end_layout + +\end_inset + + or +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L inverted +\end_layout + +\end_inset + +. + However, they also get the test property +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +WILL_FAIL +\end_layout + +\end_inset + +, i.e. + they are reported as failing if the export works without error. + From time to time they still have to be checked using +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L suspended +\end_layout + +\end_inset + +. +\end_layout + +\begin_layout Standard +These tests are suspended, because the export fails for known reasons which + cannot ATM be resolved. + But it is expected the reason might disappear in the future. + Be it new TL or better handling in \SpecialChar LyX +. +\end_layout + +\begin_layout Standard +For ctest commands without the +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +-L +\end_layout + +\end_inset + + parameter nothing changes. + Suspended or not, tests will be executed depending only on the selecting + regular expression given to the ctest command (see +\begin_inset CommandInset ref +LatexCommand ref +reference "par:ctest-options" + +\end_inset + +). +\end_layout + +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Unreliable-tests" + +\end_inset + +Unreliable tests +\end_layout + +\begin_layout Standard +Test cases whose name matches a pattern in the file +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests/unreliableTests +\end_layout + +\end_inset + + get the label +\emph on +unreliable +\emph default +. +\end_layout + +\begin_layout Standard +These tests are not executed using +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L export +\end_layout + +\end_inset + + or +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +ctest -L inverted +\end_layout + +\end_inset + +. + +\end_layout + +\begin_layout Standard +They pass or fail for various reasons not related to LyX (nonstandard, erratic) + or pass but should rather fail (wrong output). + +\begin_inset Note Note +status collapsed + +\begin_layout Plain Layout +*invalid* tests (wrong output) are not *unreliable*. + # Use "unfit" or "unapplicable" as better label and name of pattern file? + +\end_layout + +\end_inset + + +\end_layout + +\begin_layout Description +nonstandard Documents with additional requirements, e.g. + a class or package file not in TeXLive. + +\begin_inset Note Note +status open + +\begin_layout Plain Layout +TODO: rename to "extra"? +\end_layout + +\end_inset + + +\end_layout + +\begin_deeper +\begin_layout Standard +These tests are labeled as +\family typewriter +'nonstandard'. +\end_layout + +\end_deeper +\begin_layout Description +erratic Tests depending on local configuration, OS, TeX distribution, package + versions, or the phase of the moon. + +\begin_inset Note Note +status open + +\begin_layout Plain Layout +TODO: use +\emph on +erratic +\emph default + only for the phase-of-moon dependency? +\end_layout + +\end_inset + + +\end_layout + +\begin_deeper +\begin_layout Standard +These tests are labeled as +\family typewriter +'erratic'. +\end_layout + +\end_deeper +\begin_layout Description +wrong +\begin_inset space ~ +\end_inset + +output Export does not fail but the resulting document has errors. +\end_layout + +\begin_deeper +\begin_layout Standard +\paragraph_spacing single +\begin_inset Note Note +status open + +\begin_layout Plain Layout +\paragraph_spacing single +These tests are actually not +\emph on +unreliable +\emph default + but +\emph on +invalid +\emph default + (not measuring what they should measure). +\end_layout + +\end_inset + + +\end_layout + +\end_deeper +\begin_layout Paragraph +\begin_inset CommandInset label +LatexCommand label +name "par:Export-test-filtering" + +\end_inset + +Export test filtering +\end_layout + +\begin_layout Standard +The assignment of a label to a test is controlled by a set of files with + regular expressions that are matched against the test names. +\end_layout + +\begin_layout Description +ignoredTests (small file) +\begin_inset Newline newline +\end_inset + +Tests selected here are withdrawn in the configuration step (cf. + +\begin_inset CommandInset ref +LatexCommand ref +reference "par:Configuring-ctests" + +\end_inset + +). +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Input Test of any export combination +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Output Stop if tests not selected here +\end_layout + +\end_deeper +\begin_layout Description +unreliableTests: Tests selected either pass or fail, but that is dependent + on the system where the test is run. + Selected tests gain the label 'unreliable'. +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Input Each test which passed 'ignoredTests' +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Output Stop if test selected, gain label 'unreliable'. +\end_layout + +\end_deeper +\begin_layout Description +suspiciousTests +\begin_inset space \space{} +\end_inset + + +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Input Each test which passed 'unreliableTests' +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Output Stop if not selected. +\end_layout + +\begin_layout Standard +The following file is meant as subselections of 'suspiciousTests'. + If neither subselection applies, test gains labels 'export' and 'inverted' +\end_layout + +\begin_layout Description +suspendedTests Tests selected here gain the label 'suspended' but _not_ + 'export' or 'inverted', although in ctest they remain inverted. + ('ctest' knows only 'inverted' or not, labels are used only for test selection) +\end_layout + +\begin_deeper +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Input Each test selected by 'suspiciousTests' +\end_layout + +\begin_layout Labeling +\labelwidthstring 00.00.0000 +Output Selected test gains label 'suspended'. + +\end_layout + +\end_deeper +\end_deeper +\begin_layout Standard +The following table may clarify label assignement +\end_layout + +\begin_layout Standard +\begin_inset Tabular + + + + + + + + + + + + + + + + +\begin_inset Text + +\begin_layout Plain Layout +Test found in file: +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +Marked in ctest, Assigned label +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + +\end_layout + +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout + \end_layout -\begin_layout Enumerate -Create the new file -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -src/tex2lyx/test/.tex + \end_layout \end_inset - - and run tex2lyx in roundtrip mode to produce the file -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -src/tex2lyx/test/.lyx.lyx + \end_layout \end_inset + + +\begin_inset Text + +\begin_layout Plain Layout -. - This file will be the new reference. \end_layout -\begin_layout Enumerate -Once you confirmed that the tex2lyx output is correct, add the new files - to the corresponding lists in -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -src/tex2lyx/test/runtests.py + \end_layout \end_inset - -, -\begin_inset Flex Code -status collapsed + + + + +\begin_inset Text \begin_layout Plain Layout -src/tex2lyx/Makefile.am +Ignored \end_layout \end_inset - - and -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -src/tex2lyx/test/CMakeLists.txt +Unreliable \end_layout \end_inset + + +\begin_inset Text -. +\begin_layout Plain Layout +Suspicious \end_layout -\begin_layout Enumerate -Commit the changes to the repository, or send a patch to the development - list and ask for committing if you do not have commit rights. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Subsection -Export tests (cmake only) +\begin_layout Plain Layout +Suspended \end_layout -\begin_layout Standard -The export tests are integration tests. - They take longer to run and are more likely to break than the tex2lyx tests. - Nevertheless, they have caught many regressions and without a better alternativ -e it is important to keep them up-to-date and understand how they work. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Subsubsection -Expectations of LyX developers +\begin_layout Plain Layout +export \end_layout -\begin_layout Standard -Because the export tests are integration tests and take a long time to run, - LyX developers are rarely expected to run all of the tests. - Here are some good practices to follow by developers: -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Itemize -When making a non-trivial change to a .layout file, run the export and layout - tests corresponding with that .layout file. -\end_layout +\begin_layout Plain Layout -\begin_layout Itemize -When making non-trivial changes to a .lyx file, run the export tests correspondin -g to that .lyx file. \end_layout -\begin_layout Itemize -When making non-trivial changes to LyX's LaTeX export code (e.g. - touching the encoding code or package handling code that you expect will - change the exported LaTeX in some way), consider running all of the export - tests before and after your change. - If there are differences, please reconcile these (i.e. - fix the bug or fix the tests) -\emph on -before -\emph default - committing. - Ask for help if you're not sure what to do or if you do not want to run - the tests, post the patch on the list and others will run the tests. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Itemize -Understand how to interpret test failures. - If your commit is found to have broken a test, you should be able to interpret - the test results when made aware of them. - See Section -\begin_inset CommandInset ref -LatexCommand ref -reference "subsec:Interpreting-export-tests" +\begin_layout Plain Layout +unreliable +\end_layout \end_inset + + +\begin_inset Text -. -\end_layout +\begin_layout Plain Layout -\begin_layout Subsubsection -Configuring the tests \end_layout -\begin_layout Standard -To enable these tests, add the -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout --DLYX_ENABLE_EXPORT_TESTS=ON +inverted \end_layout \end_inset + + +\begin_inset Text + +\begin_layout Plain Layout - flag. - For example: \end_layout -\begin_layout Standard -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -cmake -DLYX_ENABLE_EXPORT_TESTS=ON /path/to/source +suspended \end_layout \end_inset + + +\begin_inset Text +\begin_layout Plain Layout \end_layout -\begin_layout Standard -\noindent -This flag will increase the time for the cmake command by several seconds, - mainly because of the process of inverting tests (see Section -\begin_inset CommandInset ref -LatexCommand ref -reference "subsec:Interpreting-export-tests" - \end_inset + + + + +\begin_inset Text -). +\begin_layout Plain Layout +Yes \end_layout -\begin_layout Subsubsection -Running the tests +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -To run all tests, in the build directory simply run the command -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -ctest +- \end_layout \end_inset - -. - To run only some of the tests, use the command -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest -R +- \end_layout \end_inset - -, where -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout - +- \end_layout \end_inset - - is a regular expression that matches test names. - To run only the export tests, you can use -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest -L export +- \end_layout \end_inset + + +\begin_inset Text -. - For the list of test categories available in addition to -\begin_inset Quotes eld -\end_inset +\begin_layout Plain Layout +- +\end_layout -export -\begin_inset Quotes erd \end_inset - -, run -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest \SpecialChar nobreakdash -\SpecialChar nobreakdash -print-labels + \end_layout \end_inset - -. - It is often useful to list the tests without running them (e.g. - if you want to know how many tests there are or whether your -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout - +- \end_layout \end_inset - - regular expression did what you expected). - This can be done with the -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout --N +- \end_layout \end_inset - - or -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -\SpecialChar nobreakdash -\SpecialChar nobreakdash -show-only +- \end_layout \end_inset - - argument. - We are still working on getting the tests to run in parallel which is supported - by the -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest +- \end_layout \end_inset - - command with the -\begin_inset Flex Code -status collapsed + + + + +\begin_inset Text \begin_layout Plain Layout --j +No \end_layout \end_inset - - or -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -\SpecialChar nobreakdash -\SpecialChar nobreakdash -parallel +Yes \end_layout \end_inset + + +\begin_inset Text - argument. - However, when running the tests in parallel, sometimes tests fail that - pass when run sequentially. - A reasonable approach is to first run the tests in parallel and then run - the failed tests sequentially. - For example, to run 8 jobs at a time: +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -ctest -j8 +- \end_layout \end_inset + + +\begin_inset Text - +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -ctest \SpecialChar nobreakdash -\SpecialChar nobreakdash -rerun-failed +- \end_layout \end_inset + + +\begin_inset Text - +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -\noindent -Note that some tests cannot be run in parallel. - These tests are marked in the code with the -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -\noindent -RUN_SERIAL ON ++ \end_layout \end_inset + + +\begin_inset Text - CMake property. +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -In some situations the option -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -\SpecialChar nobreakdash -\SpecialChar nobreakdash -timeout +- \end_layout \end_inset - - is useful. - There have been bugs in LyX and in LaTeX which cause compilation to hang, - and without a timeout a test might never stop (in one case there was even - a memory leak). - If a test times out, the -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest +- \end_layout \end_inset - - command exits with error, but you can distinguish between a timed out test - and a failed test in the output reported at the end of the -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest +- \end_layout \end_inset + + + + +\begin_inset Text - command. -\end_layout +\begin_layout Plain Layout -\begin_layout Subsubsection -\begin_inset CommandInset label -LatexCommand label -name "subsec:Interpreting-export-tests" +\end_layout \end_inset + + +\begin_inset Text -Interpreting the export test results +\begin_layout Plain Layout +No \end_layout -\begin_layout Standard -A test can fail for several reasons, not all of them bad. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Enumerate -The .lyx file could have been added recently and some formats are not expected - to work well. +\begin_layout Plain Layout +Yes \end_layout -\begin_layout Enumerate -A dependency is not met (e.g. - the LaTeX class file). - One hint that this is the case is that the corresponding -\begin_inset Flex Code -status open +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -check_load +Yes \end_layout \end_inset + + +\begin_inset Text - test will likely also fail. +\begin_layout Plain Layout +- \end_layout -\begin_layout Enumerate -An export that previously failed to compile now compiles. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Enumerate -An external dependency was updated (e.g. - TeX Live). +\begin_layout Plain Layout +- \end_layout -\begin_layout Enumerate -A recent code change introduced a bug. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Enumerate -\begin_inset CommandInset label -LatexCommand label -name "enu:exposed" +\begin_layout Plain Layout +- +\end_layout \end_inset + + +\begin_inset Text -A change in a document exposed the incompatibility of that class with an - export format (e.g. - LuaTeX). +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -Because the .lyx files are exported in several formats, it is not surprising - that many of the exports fail. - This expectation of failure is addressed by -\begin_inset Quotes eld \end_inset + + +\begin_inset Text -inverting -\begin_inset Quotes erd -\end_inset +\begin_layout Plain Layout ++ +\end_layout - the tests, that is, by marking the test as -\begin_inset Quotes eld \end_inset + + +\begin_inset Text -passing -\begin_inset Quotes erd -\end_inset +\begin_layout Plain Layout +- +\end_layout - if the export exits with error and as -\begin_inset Quotes eld \end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +- +\end_layout -failing -\begin_inset Quotes erd \end_inset + + +\begin_inset Text - if the export succeeds -\emph on -. +\begin_layout Plain Layout ++ +\end_layout -\emph default - It follows that these expected failures will not show up as failed tests - in the test results and thus will not pollute the -\begin_inset Quotes eld \end_inset + + + + +\begin_inset Text -good -\begin_inset Quotes erd -\end_inset +\begin_layout Plain Layout - tests. - If the export actually succeeds, then the test will fail. - The purpose of this failure is to get your attention—something has changed, - possibly for the better. \end_layout -\begin_layout Standard -We try to document why a test is inverted or ignored. - See the comment (prefixed with -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -# + \end_layout \end_inset - -) above the block in which the test is listed as inverted or ignored in - the files -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -development/autotests/revertedTests + \end_layout \end_inset - - and -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -development/autotests/ignoredTests +No \end_layout \end_inset + + +\begin_inset Text -. - It is possible that an export goes from succeeding to failing just because - the document was changed and the document was never expected to work with - a certain export format in the first case. - Once this is confirmed to be the situation, the test can be inverted. +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -A good question is why do we enable the tests for non-default formats? The - answer is that if a non-default route is broken it is often because a bug - was introduced in LyX and not because a document-specific change was made - that is not supported by the route. - In other words, there is a high signal/noise ratio in the export tests - for the non-default formats. - -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Standard -What action should you take if a test fails? First, check manually that - when the compilation succeeded before the resulting PDF was good. - In fact, sometimes it is an improvement when a test fails. - If you check manually, it might be the case that the export was succeeding - before but showing garbled text in a PDF output. - Now it might fail with a clear message of "language xyz not supported". - It is always good to check manually why something fails and if it passes - if the PDF output is good. +\begin_layout Plain Layout ++ \end_layout -\begin_layout Standard -Sometimes a test is fixed by accident. - We should uninvert a test (remove it from the -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -revertedTests +- \end_layout \end_inset + + +\begin_inset Text - file) in order to preserve the fix. +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -When a test or several tests fail, consider checking the files in the -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -Testing/Temporary/ ++ \end_layout \end_inset - - subdirectory of your build directory. - In this subdirectory are three files: the file -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -LastTestsFailed.log ++ \end_layout \end_inset - - simply lists the tests that failed on your last -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest +- \end_layout \end_inset - - command; the -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -LastTest.log +- \end_layout \end_inset - - file contains the output from the tests (and often has details explaining - why a test failed); and the -\begin_inset Flex Code -status collapsed + + + + +\begin_inset Text \begin_layout Plain Layout -CTestCostData.txt + \end_layout \end_inset + + +\begin_inset Text - file lists the times that it took to run the tests. -\end_layout +\begin_layout Plain Layout -\begin_layout Subsubsection -Inverted tests \end_layout -\begin_layout Standard -These tests are expected to always fail. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Description -reverted These tests are expected to fail, but are subject to be examined. - It is expected that they will pass in a foreseeable future. - They are labeled 'reverted'. +\begin_layout Plain Layout +No \end_layout -\begin_layout Description -suspended Some inverted tests are labeled 'suspended'. - This means, they are not executed using -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout -ctest -L export +- \end_layout \end_inset - - or -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest -L reverted +- \end_layout \end_inset - -. - From time to time they still have to be checked using -\begin_inset Flex Code -status collapsed + + +\begin_inset Text \begin_layout Plain Layout -ctest -L suspended ++ \end_layout \end_inset + + +\begin_inset Text -. +\begin_layout Plain Layout +- \end_layout -\begin_deeper -\begin_layout Standard -These tests are suspended, because they fail for known reasons which cannot - ATM be resolved. - But it is expected the reason might disappear in the future. - Be it new TL or better handling in \SpecialChar LyX -. +\end_inset + + +\begin_inset Text + +\begin_layout Plain Layout +- \end_layout -\begin_layout Standard -For ctest commands without the -\begin_inset Flex Code -status collapsed +\end_inset + + +\begin_inset Text \begin_layout Plain Layout --L +- \end_layout \end_inset + + +\begin_inset Text - parameter nothing changes. - Suspended or not, tests will be executed depending only on the regexes - parameters given to the ctest command. +\begin_layout Plain Layout +- \end_layout -\end_deeper -\begin_layout Subsubsection -Unreliable tests -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Description -nonstandard In primary sense such test means "requires non-standard ressources - (LaTeX packages and document classes, fonts, ... - that are not a requirement for running this test suite". +\begin_layout Plain Layout +- \end_layout -\begin_deeper -\begin_layout Standard -In a wider sense, it is currently used also for "not to be expected to succeed - on every site that runs this test suite". - This wider definition includes tests that have "arbitrary" result depending - on local configuration, OS, TeX distribution, package versions, or the - phase of the moon. -\end_layout +\end_inset + + +\begin_inset Text -\begin_layout Standard -These tests are labeled as -\family typewriter -'nonstandard'. +\begin_layout Plain Layout +- \end_layout -\end_deeper -\begin_layout Description -erratic Tests depending on local configuration, OS, TeX distribution, package - versions, or the phase of the moon. -\end_layout +\end_inset + + + + +\end_inset + -\begin_deeper -\begin_layout Standard -These tests are labeled as -\family typewriter -'erratic'. \end_layout -\end_deeper -\begin_layout Subsection +\begin_layout Subsubsection check_load tests \end_layout @@ -1804,8 +4001,38 @@ development/autotests/filterCheckWarnings Under cmake, the tests are labeled as 'load'. \end_layout -\begin_layout Subsection -URL tests (cmake only) +\begin_layout Subsubsection +Keytests +\end_layout + +\begin_layout Standard +Automated tests based on the "MonKey Testing" keytest program. + They are documented in the README document in +\begin_inset Flex Code +status collapsed + +\begin_layout Plain Layout +development/autotests +\end_layout + +\end_inset + + subfolder of the \SpecialChar LyX + source code distribution. + T +\end_layout + +\begin_layout Subsubsection +lyx21 tests +\end_layout + +\begin_layout Standard +These tests combine lyx2lyx tests with check_load tests. + They fail if either fails. +\end_layout + +\begin_layout Subsubsection +URL tests \end_layout \begin_layout Standard @@ -1841,7 +4068,7 @@ LastTest.log 'url'. \end_layout -\begin_layout Subsubsection +\begin_layout Paragraph Running URL tests \end_layout @@ -1864,7 +4091,7 @@ ctest -L url \begin_inset Newline newline \end_inset -runns all tests with label +runs all tests with label \family typewriter 'url' \end_layout @@ -1878,7 +4105,7 @@ ctest -R 'check_.*urls' \begin_inset Newline newline \end_inset -runns the tests 'check_accessible_urls' +runs the tests 'check_accessible_urls' \end_layout \begin_layout Standard @@ -1886,40 +4113,6 @@ Associated test results can be examined in ctest-log directory in files of the form 'LastFailed.*URLS.log' \end_layout -\begin_layout Subsection -Test labels (cmake only) -\end_layout - -\begin_layout Standard -ctest label commands: -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -\SpecialChar nobreakdash -print-labels shows all assigned labels -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -L -\begin_inset space ~ -\end_inset - -labelname executes all tests to which this label is asigned to. - A test may have more that one label. -\end_layout - -\begin_layout Description -\SpecialChar nobreakdash -j -\begin_inset space ~ -\end_inset - -number executes tests in parallel using 'number' simultaneously processes. - Some tests are marked as 'sequencial', for them this parameter has no effect. -\end_layout - \begin_layout Section Development policies \end_layout