ctests: log a waste and a success

[lyx.git] / development / autotests / README

diff --git a/development/autotests/README b/development/autotests/README
index 0cdefbd35806e8cdc2d19dcc8c82930c7a07900b..bd28751216f6657fb94e8b88cde1201fa362e63f 100644 (file)
--- a/development/autotests/README
+++ b/development/autotests/README
@@ -1,13 +1,87 @@
 Automated tests based on the "MonKey Testing" keytest program.
 Automated tests based on the "MonKey Testing" keytest program.

+======================================================================

 
 

+WARNING!!
+----------------------------------------------------------------------
+Running the tests in this folder may cause abrupt kill of existing
+LyX instances on your system. In order to avoid any possible data
+loss, quit any running instance of LyX in which you're carrying out
+real work. Also, although precautions have been taken in order to not
+allow the key presses to reach non-LyX windows, this may sometimes
+happen anyway. Please, close any running application that is not
+strictly needed.
+
+
+DEPENDENCIES
+----------------------------------------------------------------------
+The following programs are needed by the testing framework:
+- xvkbd: a modified copy of xvkbd is included in this folder, and should
+         be built before running the tests
+- wmctrl
+- pcregrep
+- Tcl/Tk (wish8.5)
+
+
+USAGE
+----------------------------------------------------------------------

 In order to launch all the tests available in this folder, just run
 
 In order to launch all the tests available in this folder, just run
 

-  ./run-tests.sh
+  make
+
+and do not interact with your computer while the tests are running.

 
 If you wish to launch one or more specific tests, then provide them
 
 If you wish to launch one or more specific tests, then provide them

-as argument to the script, e.g.:
+as argument to the run-tests.sh script, e.g.:

 
   ./run-tests.sh <whatever>*-in.txt
 
 For failed tests you get a folder with the logs that help you identify
 the problem.
 
   ./run-tests.sh <whatever>*-in.txt
 
 For failed tests you get a folder with the logs that help you identify
 the problem.

+
+Despite the efforts to avoid it, sometimes a test fails simply because
+the key presses provided to the process are someway lost due to slow
+hardware, slow window manager, too beautiful and animated desktop,...
+When this happens, the tests should be run with a higher delay among
+key presses. The default of 100 milliseconds can be changed by setting
+the XVKBD_DELAY environment variable.
+
+The standard xvkbd available on your system may be used to run the
+tests. However, a custom xvkbd may optionally be used that forbids the
+testing framework to type into non-LyX windows and may mitigate the
+issue described right above. This option can be activated by defining
+the XVKBD_HACKED environment variable to any value.
+
+
+SYNTAX
+----------------------------------------------------------------------
+Each test-case script should be named as xxx-in.txt. The syntax of the
+script is described in detail in the sample test-case script
+hello-world-in.txt.
+
+Alternatively, a test-case can be an arbitrary executable script with
+name ending in "-in.sh", that is launched inside a dedicated folder,
+and to which the variables LYX_ROOT and LYX_EXE are exported, so that
+you can easily reference the root LyX sources folder and the program
+executable. The test script needs to complete with a no-error return
+value (zero), otherwise a failure is reported (see export-in.sh for
+an example).
+
+Whenever LyX is launched, in both cases, the LYX_USERDIR variable is
+set to autotests/out-home, in order to rely on clean preferences
+settings (use make clean or make without arguments in order to let
+that folder be created again).
+
+
+TODO
+----------------------------------------------------------------------
+Speed-up the execution of the tests as much as possible (i.e., get rid
+of heuristic waits when present and replace them with exact check of
+the condition we're waiting for).
+
+Allow for tests with multiple LyX instances, e.g., for copy&paste
+across LyX instances, or multiple applications, i.e., for testing
+the interaction between LyX and other applications.
+
+The modified xvkbd forces the focus on the specified target window
+(for safety reasons).  Allow for a temporary disable of this within
+the scripts, whenever we need to interact with a pop-up dialog.