Scilab cli: add a "--timeout delay" argument
[scilab.git] / scilab / modules / development_tools / help / en_US / test_run.xml
1 <?xml version="1.0" encoding="UTF-8"?>
2 <!--
3  * Scilab ( http://www.scilab.org/ ) - This file is part of Scilab
4  * Copyright (C) INRIA
5  * Copyright (C) 2009-2011 - DIGITEO - Michael Baudin
6  * Copyright (C) 2013 - Scilab Enterprises - Paul Bignier: added 32/64bits separation
7  *
8  * Copyright (C) 2012 - 2016 - Scilab Enterprises
9  *
10  * This file is hereby licensed under the terms of the GNU GPL v2.0,
11  * pursuant to article 5.3.4 of the CeCILL v.2.1.
12  * This file was originally licensed under the terms of the CeCILL v2.1,
13  * and continues to be available under such terms.
14  * For more information, see the COPYING file which you should have received
15  * along with this program.
16  *
17  -->
18 <refentry xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:svg="http://www.w3.org/2000/svg" xmlns:ns5="http://www.w3.org/1999/xhtml" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:db="http://docbook.org/ns/docbook" xmlns:scilab="http://www.scilab.org" xml:id="test_run" xml:lang="en">
19     <refnamediv>
20         <refname>test_run</refname>
21         <refpurpose>
22             Launches unit tests and non-regression test in a module or directory. The test run checks first that test scripts does not generate an error, then that the test diary (<literal>dia</literal> files) is the same as the reference diary (<literal>dia.ref</literal> files).
23         </refpurpose>
24     </refnamediv>
25     <refsynopsisdiv>
26         <title>Syntax</title>
27         <synopsis>
28             status = test_run()
29             status = test_run(module)
30             status = test_run(module, test_name)
31             status = test_run(module, test_name, options, exportToFile)
32         </synopsis>
33     </refsynopsisdiv>
34     <refsection>
35         <title>Arguments</title>
36         <variablelist>
37             <varlistentry>
38                 <term>module</term>
39                 <listitem>
40                     <para>
41                         A String array or <literal>[]</literal> (or equivalently <literal>"[]"</literal>). Name of the modules or directory for the tests, all internal modules if <literal>[]</literal>
42                     </para>
43                     <itemizedlist>
44                         <listitem>
45                             <para>
46                                 the name of an internal Scilab module ("core", "time", ...), a sub-module (e.g. "optimization|neldermead").
47                             </para>
48                         </listitem>
49                         <listitem>
50                             <para>
51                                 the name of an ATOMS module ("module_lycee", "nisp", ...). To be taken into account, the module must be loaded when test_run() is called.
52                             </para>
53                         </listitem>
54                         <listitem>
55                             <para>
56                                 the absolute directory path of a module containing test/unit_tests or test/nonreg_tests.
57                             </para>
58                         </listitem>
59                     </itemizedlist>
60                 </listitem>
61             </varlistentry>
62             <varlistentry>
63                 <term>test_name</term>
64                 <listitem>
65                     <para>
66                         A string array or <literal>[]</literal> (or equivalently <literal>"[]"</literal>). The names of the tests to execute during this run. If <literal>[]</literal> executes all tests found.
67                     </para>
68                 </listitem>
69             </varlistentry>
70             <varlistentry>
71                 <term>options</term>
72                 <listitem>
73                     <para>
74                         A string array or <literal>[]</literal> or <literal>"[]"</literal>. The options for the tests during this run, default options if <literal>[]</literal> or <literal>"[]"</literal>.
75                     </para>
76                     <variablelist> or
77                         <varlistentry>
78                             <term>"no_check_ref"</term>
79                             <listitem>
80                                 <para>does not check if the .dia and .dia.ref are equal</para>
81                             </listitem>
82                         </varlistentry>
83                         <varlistentry>
84                             <term>"no_check_error_output"</term>
85                             <listitem>
86                                 <para>The error output stream is not checked. This option can be used when Scilab complains about the localization being not available.</para>
87                             </listitem>
88                         </varlistentry>
89                         <varlistentry>
90                             <term>"create_ref"</term>
91                             <listitem>
92                                 <para>create the .dia.ref file and does not check if the .dia and .dia.ref are equal</para>
93                             </listitem>
94                         </varlistentry>
95                         <varlistentry>
96                             <term>"show_error"</term>
97                             <listitem>
98                                 <para>If an error occurs, shows the last 10 lines of the execution</para>
99                             </listitem>
100                         </varlistentry>
101                         <varlistentry>
102                             <term>"show_diff"</term>
103                             <listitem>
104                                 <para>
105                                     If a difference is found, shows the result of the command <literal>diff -u</literal>
106                                 </para>
107                             </listitem>
108                         </varlistentry>
109                         <varlistentry>
110                             <term>"list"</term>
111                             <listitem>
112                                 <para>Does not perform the tests but displays a list of available tests</para>
113                             </listitem>
114                         </varlistentry>
115                         <varlistentry>
116                             <term>"help"</term>
117                             <listitem>
118                                 <para>display some examples about how to use this command</para>
119                             </listitem>
120                         </varlistentry>
121                         <varlistentry>
122                             <term>"mode_nw"</term>
123                             <listitem>
124                                 <para>Add the "-nw" option to the launch</para>
125                             </listitem>
126                         </varlistentry>
127                         <varlistentry>
128                             <term>"mode_nwni"</term>
129                             <listitem>
130                                 <para>Add the "-nwni" option to the launch</para>
131                             </listitem>
132                         </varlistentry>
133                         <varlistentry>
134                             <term>"mode_nwni_profiling"</term>
135                             <listitem>
136                                 <para>Add the "-nwni -profiling" option to the launch for detect valgrind error (Linux only)</para>
137                             </listitem>
138                         </varlistentry>
139                         <varlistentry>
140                             <term>"nonreg_tests"</term>
141                             <listitem>
142                                 <para>runs only the non-regression tests, skipping unit tests</para>
143                             </listitem>
144                         </varlistentry>
145                         <varlistentry>
146                             <term>"unit_tests"</term>
147                             <listitem>
148                                 <para>Runs only the unit tests, skipping non-regression tests</para>
149                             </listitem>
150                         </varlistentry>
151                         <varlistentry>
152                             <term>"skip_tests"</term>
153                             <listitem>
154                                 <para>
155                                     Skip the tests specified in <term>test_name</term>.
156                                 </para>
157                             </listitem>
158                         </varlistentry>
159                         <varlistentry>
160                             <term>"enable_lt"</term>
161                             <listitem>
162                                 <para>Enable long-time execution tests</para>
163                             </listitem>
164                         </varlistentry>
165                         <varlistentry>
166                             <term>"short_summary"</term>
167                             <listitem>
168                                 <para>Does not display statistics nor execution time after execution (only number of executed, passed, failed and skipped tests will be displayed on a single line).</para>
169                             </listitem>
170                         </varlistentry>
171                     </variablelist>
172                 </listitem>
173             </varlistentry>
174             <varlistentry>
175                 <term>exportToFile</term>
176                 <listitem>
177                     <para>
178                         Export to a XML file the result of the test. This file follows the specification of the XUnit format.
179                         Note that the usage of this option enables <literal>show_diff</literal> and <literal>show_error</literal>.
180                     </para>
181                     <para>
182                         If the file pointed by <varname>exportToFile</varname> already exists, the new result will be added to the existing file.
183                     </para>
184                 </listitem>
185             </varlistentry>
186             <varlistentry>
187                 <term>status</term>
188                 <listitem>
189                     <para>
190                         Boolean value
191                         Returns %t if no error has been detected
192                         Returns %f if any error has been detected
193                     </para>
194                 </listitem>
195             </varlistentry>
196         </variablelist>
197     </refsection>
198     <refsection>
199         <title>Description</title>
200         <para>
201             Search for .tst files in the unit test and non-regression test library
202             execute them, and display a report about success of failures.
203             The .tst files are searched in directories SCI+"/modules/*/tests/unit_tests"
204             and SCI+"/modules/*/tests/nonreg_tests".
205         </para>
206         <para>
207             First <literal>test_run</literal> checks that a test does not produce an error.
208         </para>
209         <para>
210             Then <literal>test_run</literal> checks that the output and commands of a script are identical to the reference file. Whenever a test is executed, a <literal>.dia</literal> file is generated which contains
211             the full list of commands executed along with messages that appear in the
212             console output. When the script is done, the <literal>.dia</literal> file is compared with
213             the <literal>.dia.ref</literal> file which is expected to be in the same directory
214             as the .tst file. If the two file are different, the test fails.
215         </para>
216         <para>
217             Special tags may be inserted in the .tst file, which help to
218             control the processing of the corresponding test. These tags
219             are expected to be found in Scilab comments.
220         </para>
221         <para>These are the available tags:</para>
222         <itemizedlist>
223             <listitem>
224                 <para>
225                     &lt;-- INTERACTIVE TEST --&gt;
226                     This test will be skipped because it is interactive.
227                 </para>
228             </listitem>
229             <listitem>
230                 <para>
231                     &lt;-- LONG TIME EXECUTION --&gt;
232                     This test will be skipped because it needs long-time duration. To enable
233                     the test, call test_run with the following option: "enable_lt"
234                 </para>
235             </listitem>
236             <listitem>
237                 <para>
238                     &lt;-- NOT FIXED --&gt;
239                     This test will be skipped because it is a known, but unfixed bug.
240                 </para>
241             </listitem>
242             <listitem>
243                 <para>
244                     &lt;-- TEST WITH GRAPHIC --&gt;
245                     This test will be executed with scilab -nw. (default mode)
246                 </para>
247             </listitem>
248             <listitem>
249                 <para>
250                     &lt;-- NO TRY CATCH --&gt;
251                 </para>
252             </listitem>
253             <listitem>
254                 <para>
255                     &lt;-- NO CHECK ERROR OUTPUT --&gt;
256                     The error output file is not checked
257                 </para>
258             </listitem>
259             <listitem>
260                 <para>
261                     &lt;-- NO CHECK REF --&gt;
262                     The .dia and the .dia.ref files are not compared.
263                 </para>
264             </listitem>
265             <listitem>
266                 <para>
267                     &lt;-- ENGLISH IMPOSED --&gt;
268                     This test will be executed with the -l en_US option.
269                 </para>
270             </listitem>
271             <listitem>
272                 <para>
273                     &lt;-- FRENCH IMPOSED --&gt;
274                     This test will be executed with the -l fr_FR option.
275                 </para>
276             </listitem>
277             <listitem>
278                 <para>
279                     &lt;-- CLI SHELL MODE --&gt;
280                     (was &lt;-- JVM NOT MANDATORY --&gt; and is obsolete)
281                     This test will be executed with scilab -nwni.
282                     All tests without this, will not be executed in "mode_mwni".
283                 </para>
284             </listitem>
285             <listitem>
286                 <para>
287                     &lt;-- WINDOWS ONLY --&gt;
288                     If the operating system isn't Windows, the test is skipped.
289                 </para>
290             </listitem>
291             <listitem>
292                 <para>
293                     &lt;-- UNIX ONLY --&gt;
294                     If the operating system isn't an Unix OS, the test is skipped.
295                 </para>
296             </listitem>
297             <listitem>
298                 <para>
299                     &lt;-- LINUX ONLY --&gt;
300                     If the operating system isn't GNU/Linux, the test is skipped.
301                 </para>
302             </listitem>
303             <listitem>
304                 <para>
305                     &lt;-- MACOSX ONLY --&gt;
306                     If the operating system isn't Mac OS X, the test is skipped.
307                 </para>
308             </listitem>
309             <listitem>
310                 <para>
311                     &lt;-- XCOS TEST --&gt;
312                     This test will launch all the necessary Xcos libs. This test
313                     will be launched in nw mode.
314                 </para>
315             </listitem>
316         </itemizedlist>
317         <para>
318             Each test is executed in a separated process, created with the "host" command.
319             That enables the current command to continue, even if the test as
320             created an unstable environment. It also enables the tests to be
321             independent from one another.
322         </para>
323     </refsection>
324     <refsection>
325         <title>Platform-specific tests</title>
326         <para>
327             It may happen that the output of a test depends on the platform on which it is
328             executed. In this case, the <literal>.ref</literal> file cannot be correct for
329             all platforms and unit tests may fail for some platform. In this case, we can
330             create a default <literal>.ref</literal> and create additional <literal>.ref</literal>
331             file for each platform.
332         </para>
333         <para>
334             The various platform-specific <literal>.ref</literal> files must have one of the following extensions.
335         </para>
336         <itemizedlist>
337             <listitem>
338                 <para>
339                     <literal>.unix.dia.ref</literal> for Unix platform,
340                 </para>
341             </listitem>
342             <listitem>
343                 <para>
344                     <literal>.linux.dia.ref</literal> for GNU/Linux platform,
345                 </para>
346             </listitem>
347             <listitem>
348                 <para>
349                     <literal>.linux32.dia.ref</literal> for GNU/Linux platform with 32bits processors,
350                 </para>
351             </listitem>
352             <listitem>
353                 <para>
354                     <literal>.win.dia.ref</literal> for Windows platform,
355                 </para>
356             </listitem>
357             <listitem>
358                 <para>
359                     <literal>.win32.dia.ref</literal> for Windows platform with 32bits processors,
360                 </para>
361             </listitem>
362             <listitem>
363                 <para>
364                     <literal>.macosx.dia.ref</literal> for Mac OS X platform.
365                 </para>
366             </listitem>
367         </itemizedlist>
368         <para>
369             The algorithm is the following.
370             First, the <literal>.ref</literal> is considered. If this file does not exist,
371             the platform-specific <literal>platform.ref</literal> file is examined depending on the current platform.
372         </para>
373         <itemizedlist>
374             <listitem>
375                 <para>
376                     on Windows platforms: <literal>.win.dia.ref</literal>, <literal>.win32.dia.ref</literal>
377                 </para>
378             </listitem>
379             <listitem>
380                 <para>
381                     on Max OS X platforms: <literal>.unix.dia.ref</literal>, <literal>.macosx.dia.ref</literal>,
382                 </para>
383             </listitem>
384             <listitem>
385                 <para>
386                     on GNU/Linux platforms: <literal>.unix.dia.ref</literal>, <literal>.linux.dia.ref</literal>, <literal>.linux32.dia.ref</literal>.
387                 </para>
388             </listitem>
389         </itemizedlist>
390     </refsection>
391     <refsection>
392         <title>Examples</title>
393         <programlisting role="example"><![CDATA[
394 // Launch all tests
395 // This may take some time
396 // =============================================
397
398 // test_run();
399 // test_run([]);
400 // test_run([],[]);
401 // test_run("[]","[]");
402 // test_run [] [];
403
404 // Test one or several module
405 // =============================================
406
407 // Test one module
408 test_run('time');
409
410 // Test several modules
411 test_run(['time','string']);
412
413 // Test a submodule
414 test_run('optimization|neldermead');
415
416 // Refer to a module by its path
417 test_run(SCI+'/modules/core');
418
419 // Launch a specific test
420 // =============================================
421
422 // One specific test
423 test_run('time','datenum');
424
425 // Several tests
426 test_run('time',['datenum';'calendar']);
427
428 // Skip some tests
429 // =============================================
430
431 test_run('time',['datenum';'calendar'],'skip_tests');
432
433 // Options
434 // =============================================
435
436 // does not check if the .dia and .dia.ref are equal
437 test_run('time','datenum','no_check_ref');
438
439 // Create the .dia.ref file and does not check if the .dia and .dia.ref are equal
440 test_run([],[],'create_ref');
441
442 // Does not perform the tests but displays a list of available tests
443 test_run([],[],'list');
444
445 // Display some examples about how to use this command
446 test_run([],[],'help');
447
448 // Runs only the non-regression tests, skipping unit tests
449 test_run([],[],'nonreg_test');
450
451 // Runs only the unit tests, skipping non-regression tests
452 test_run([],[],'unit_test');
453
454 // Do not check the error output (std err)
455 test_run('boolean','bug_2799','no_check_error_output');
456
457 // Combine several options
458 test_run([],[],['no_check_ref','mode_nw']);
459
460 // Console mode
461 test_run time [] no_check_ref //tests time module with no_check_ref option
462  ]]></programlisting>
463         
464         <programlisting role="example"><![CDATA[
465 // Run unitary tests of an external module (with his path)
466 test_run('SCI/contrib/toolbox_skeleton')
467  ]]></programlisting>
468         
469         <programlisting role="example"><![CDATA[
470 // Export to a XML Xunit file
471 test_run('boolean',[],[],TMPDIR+"/boolean_test_run.xml");
472 test_run('time','datenum',[],TMPDIR+"/time_datenum_test_run.xml");
473  ]]></programlisting>
474     </refsection>
475     <refsection>
476         <title>Internal Design</title>
477         <para>
478             The tests are performed in the temporary directory, not in the directory
479             which originally contain the tests files.
480         </para>
481         <para>
482             The .tst script is not run as is. Instead, a header and a footer are
483             inserted at the beginning and at the end of the .tst at the time
484             the script is copied into the temporary directory.
485             The role of this modification is to redirect the output messages
486             into the .dia file, so that the user can have a log file once the test
487             is performed.
488         </para>
489         <para>
490             An execution timeout delay (watchdog timer) is setup to 5 minutes
491             for each regular test. To ignore this timeout use the long-time
492             execution (<literal>LONG TIME EXECUTION</literal>) flag.
493         </para>
494     </refsection>
495     <refsection>
496         <title>History</title>
497         <revhistory>
498             <revision>
499                 <revnumber>5.4.0</revnumber>
500                 <revdescription>test_run returns a status:
501                     <itemizedlist><listitem>
502                             Returns %t if no error has been detected
503                         </listitem>
504                         <listitem>
505                             Returns %f if any error has been detected
506                         </listitem>
507                     </itemizedlist>
508                     <para>
509                         <literal>show_diff</literal> and <literal>show_error</literal> added as new options
510                     </para>
511                     <para>
512                         <literal>CLI SHELL MODE</literal> tag is added. Replaces <literal>JVM NOT MANDATORY</literal> (still supported)
513                     </para>
514                     <para>
515                         <literal>test_run</literal> can work on an external module.
516                     </para>
517                     <para>
518                         Fourth argument added to export to a XML file
519                     </para>
520                 </revdescription>
521             </revision>
522             <revision>
523                 <revnumber>5.5.0</revnumber>
524                 <revdescription>32/64bits separation available</revdescription>
525             </revision>
526             <revision>
527                 <revnumber>6.0.0</revnumber>
528                 <revdescription>
529                     <para>profiling mode added to profile execution with valgrind (Linux only)</para>
530                     <para>
531                         timeout delay (watchdog timer) set to 5 minutes for single tests without <literal>LONG TIME EXECUTION</literal>
532                     </para>
533                 </revdescription>
534             </revision>
535         </revhistory>
536     </refsection>
537 </refentry>