1 The ctests are not convenient tests. In this file we can attempt to log the
2 benefits and costs to using them so we can periodically evaluate which tests we
3 should keep and which we should get rid of.
5 Below are some objective costs:
7 - The time it takes to run the ctests.
8 - The email discussion back-and-forth.
10 There are also some non-objective costs. The ctests are annoying and not fun to
11 maintain, discuss, and fix. Although it's hard to measure this "annoyance"
12 cost, it should be taken into account when we review.
14 Below we can try to keep a log of which bugs are found with which ctests. This
15 way we can see which tests are the most useful. It is difficult to objectively
16 differentiate between minor bugs and important bugs.
18 The "Benefit" row gives the amount of days in-between when an issue is found
19 because of the ctests and when an issue is reported by someone not using the
22 The "ctest" row gives a ctest pattern for the tests that were affected by the
25 Examples where convergence tests took time and annoyance to figure out that
26 they were actually "false positives" (i.e., no bug that is worth the time or
29 Report : https://www.mail-archive.com/search?l=mid&q=20210125200655.e27xaaoy2belyxhw%40tallinn
30 Report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218248.html
32 Testing compilation after lyx2lyx roundtrip was not helpful in this case:
33 report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218232.html
34 ctests : de/UserGuide_lyx(22|23)
37 Examples where an upstream issue was reported and ended up wasting time:
39 Report : https://github.com/latex3/latex2e/issues/836#issuecomment-1159826566
40 ctests : export/templates/Obsolete/.*American_Geophysical_Union.*_(lyx22|lyx23|pdf|dvi).*
43 Examples where bugs were fixed (more quickly) because of ctests:
45 Report : https://www.mail-archive.com/search?l=mid&q=20210126190837.pwuxhrhrikzszfrh%40tallinn
48 Report : https://www.mail-archive.com/search?l=mid&q=b0a33192bc5635bd1cb5596e346f9c34615ffeb4.camel%40lyx.org
55 Report : https://www.mail-archive.com/search?l=mid&q=20210218051838.jbwnemfv5ilhlsls%40tallinn
64 Report : https://www.mail-archive.com/search?l=mid&q=20210523014926.xewdr5x62br2okvy%40tallinn
67 Report : https://www.mail-archive.com/search?l=mid&q=20210528001413.56phju7dvfn4dexs%40tallinn
70 Report : https://www.mail-archive.com/search?l=mid&q=20210529174524.idnmt5w4tvhuvyy6%40tallinn
71 Note1 : We are still not sure what to do regarding how to make it more user-friendly
72 for users to realize that "-dALLOWPSTRANSPARENCY" is needed for the ps2pdf converter.
73 Benefit: 14 days (https://www.lyx.org/trac/ticket/12303)
75 Report : https://www.mail-archive.com/search?l=mid&q=20211205190238.edrtqmg3fkdr7gbo%40poole
76 Fix : 489bf46a, e3489d54
78 Report : https://www.mail-archive.com/search?l=mid&q=20220205184128.4clckanaufwmx5z4%40gary
80 Report : https://www.mail-archive.com/search?l=mid&q=20220206142559.rpgwki7wruhtasxp%40gary
83 Report : https://www.mail-archive.com/search?l=mid&q=20220310131706.qdthdjblfpme2wew%40gary
85 ctests : examples/Welcome_pdf2
87 report : https://www.mail-archive.com/lyx-devel@lists.lyx.org/msg218232.html
88 Fix : b8502a3e, e316e21a
89 ctests : de/UserGuide_lyx(16|20|21)
91 report : https://www.mail-archive.com/search?l=mid&q=20221218210348.gkwloeu74zjewqsu%40gary
93 ctests : DEFAULTOUTPUT_export/export/latex/CJK/zh_CN-en-de-el-ru_language-default_pdf2
97 Examples where upstream (e.g., LaTeX packages) bugs were fixed (more quickly)
100 Fix : https://github.com/nrnrnr/noweb/pull/12#event-4892099481
102 Report : (private message to Jürgen: Message-ID:<20211205192054.zbxij5butj3wdyvf@poole>)
103 Fix : some of the polyglossia commits in the range 1c36dcdc^...bf69288c
104 Benefit: Not clear. The regressions were in the development version so they
105 could have easily been caught before release.
107 Report : https://www.mail-archive.com/search?l=mid&q=20211207200100.o27ojaftehrjfcza%40poole
111 Report : (private message to Jürgen: LyXMessage-ID:<20220216020703.la2fd5gu7yz7y7dc@gary>)
112 Fix : https://github.com/reutenauer/polyglossia/commit/c7489ebaf8c66a7e4fe2529de7068ebb79c68812
113 Benefit: Not clear. The regressions were in the development version so they
114 could have easily been caught before release.
115 ctests : export/export/lyx2lyx/revert-languages-polyglossia-primary_lyx22
118 Report : https://github.com/latex3/latex2e/issues/836
119 Fix : https://github.com/latex3/latex2e/commit/878a4715df024fc3425d2bfd5d3d138b9395c178
120 ctests : doc/EmbeddedObjects
123 Report : https://github.com/gsilano/EuropeCV/issues/29
128 Report : https://github.com/schlcht/microtype/issues/21
129 upstream: https://github.com/latex3/latex2e/issues/880
131 ctests : export/templates/Articles/Hebrew_Article_.*(dvi|pdf|pdf2|pdf3)