1 <?xml version="1.0" encoding="UTF-8"?>
3 * Scilab ( http://www.scilab.org/ ) - This file is part of Scilab
5 * Copyright (C) 2009-2011 - DIGITEO - Michael Baudin
6 * Copyright (C) 2013 - Scilab Enterprises - Paul Bignier: added 32/64bits separation
7 * Copyright (C) 2012 - 2016 - Scilab Enterprises
8 * Copyright (C) 2018 - Samuel GOUGEON
10 * This file is hereby licensed under the terms of the GNU GPL v2.0,
11 * pursuant to article 5.3.4 of the CeCILL v.2.1.
12 * This file was originally licensed under the terms of the CeCILL v2.1,
13 * and continues to be available under such terms.
14 * For more information, see the COPYING file which you should have received
15 * along with this program.
18 <refentry xmlns="http://docbook.org/ns/docbook" xmlns:xlink="http://www.w3.org/1999/xlink"
19 xmlns:svg="http://www.w3.org/2000/svg" xmlns:ns5="http://www.w3.org/1999/xhtml"
20 xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:db="http://docbook.org/ns/docbook"
21 xmlns:scilab="http://www.scilab.org" xml:id="test_run" xml:lang="en">
23 <refname>test_run</refname>
25 runs unit tests and non-regression tests of a module or in a directory
32 status = test_run(module)
33 status = test_run(module, test_name)
34 status = test_run(module, test_name, options, exportToFile)
38 <title>Arguments</title>
44 A String array or <literal>[]</literal> (or equivalently <literal>"[]"</literal>). Name of the modules or directory for the tests, all internal modules if <literal>[]</literal>
49 the name of an internal Scilab module ("core", "time", ...), a sub-module (e.g. "optimization|neldermead").
54 the name of an ATOMS module ("module_lycee", "nisp", ...). To be taken into account, the module must be loaded when test_run() is called.
59 the absolute directory path of a module containing test/unit_tests or test/nonreg_tests.
66 <term>test_name</term>
69 A string array or <literal>[]</literal> or <literal>"[]"</literal>:
70 The names of the tests to execute during this run. If <varname>test_name</varname>
71 is <literal>[]</literal>, all tests found in the module or in the directory are executed.
74 The wildcard * can be used, like in <literal>*sin</literal>,
75 <literal>*sin</literal>, or <literal>*sin*</literal>.
83 A string array or <literal>[]</literal> or <literal>"[]"</literal>. The options for the tests during this run, default options if <literal>[]</literal> or <literal>"[]"</literal>.
87 <term>"no_check_ref"</term>
89 <para>does not check if the .dia and .dia.ref are equal</para>
93 <term>"no_check_error_output"</term>
95 <para>The error output stream is not checked. This option can be used when Scilab complains about the localization being not available.</para>
99 <term>"create_ref"</term>
101 <para>creates the .dia.ref file (for tests not having the
102 <literal><-- NO CHECK REF --></literal> flag)
103 and does not check if the .dia and .dia.ref are equal.
108 <term>"show_error"</term>
110 <para>If an error occurs, shows the last 10 lines of the execution</para>
114 <term>"show_diff"</term>
117 If a difference is found, shows the result of the command <literal>diff -u</literal>
124 <para>Does not perform the tests but displays a list of available tests</para>
130 <para>display some examples about how to use this command</para>
134 <term>"mode_nw"</term>
136 <para>Add the "-nw" option to the launch</para>
140 <term>"mode_nwni"</term>
142 <para>Add the "-nwni" option to the launch</para>
146 <term>"mode_nwni_profiling"</term>
148 <para>Add the "-nwni -profiling" option to the launch for detect valgrind error (Linux only)</para>
152 <term>"nonreg_tests"</term>
154 <para>runs only the non-regression tests, skipping unit tests</para>
158 <term>"unit_tests"</term>
160 <para>Runs only the unit tests, skipping non-regression tests</para>
164 <term>"skip_tests"</term>
167 Skip the tests specified in <term>test_name</term>.
172 <term>"enable_lt"</term>
174 <para>Enable long-time execution tests</para>
178 <term>"short_summary"</term>
180 <para>Does not display statistics nor execution time after execution (only number of executed, passed, failed and skipped tests will be displayed on a single line).</para>
187 <term>exportToFile</term>
190 Export to a XML file the result of the test. This file follows the specification of the XUnit format.
191 Note that the usage of this option enables <literal>show_diff</literal> and <literal>show_error</literal>.
194 If the file pointed by <varname>exportToFile</varname> already exists,
195 the new results are appended to the existing file.
204 Returns %t if no error has been detected
205 Returns %f if any error has been detected
212 <title>Description</title>
214 Search for .tst files in the unit test and non-regression test library,
215 execute them, and display a report about success or failures.
216 The .tst files are searched in directories ~/tests/unit_tests"
217 and ~/tests/nonreg_tests" where "~" is the root directory of targeted modules.
220 First <literal>test_run</literal> checks that a test does not produce an error.
223 Then <literal>test_run</literal> checks that the output and commands of a script are
224 identical to the reference file. Whenever a test is executed, a <literal>.dia</literal>
225 file is generated which contains
226 the full list of commands executed along with messages that appear in the
227 console output. When the script is done, the <literal>.dia</literal> file is compared with
228 the <literal>.dia.ref</literal> file which is expected to be in the same directory
229 as the .tst file. If the two file are different, the test fails.
232 Special tags may be inserted in the .tst file, which help to
233 control the processing of the corresponding test. These tags
234 are expected to be found in Scilab comments.
236 <para>These are the available tags:</para>
240 <-- INTERACTIVE TEST -->
241 This test will be skipped because it is interactive.
246 <-- LONG TIME EXECUTION -->
247 This test will be skipped because it needs long-time duration. To enable
248 the test, call test_run with the following option: "enable_lt"
253 <-- NOT FIXED -->
254 This test will be skipped because it is a known, but unfixed bug.
259 <-- TEST WITH GRAPHIC -->
260 This test will be executed with scilab -nw. (default mode)
265 <-- NO TRY CATCH -->
270 <-- NO CHECK ERROR OUTPUT -->
271 The error output file is not checked
276 <-- NO CHECK REF -->
277 The .dia and the .dia.ref files are not compared.
282 <-- ENGLISH IMPOSED -->
283 This test will be executed with the -l en_US option.
288 <-- FRENCH IMPOSED -->
289 This test will be executed with the -l fr_FR option.
294 <-- CLI SHELL MODE -->
295 (was <-- JVM NOT MANDATORY --> and is obsolete)
296 This test will be executed with scilab -nwni.
297 All tests without this, will not be executed in "mode_mwni".
302 <-- WINDOWS ONLY -->
303 If the operating system isn't Windows, the test is skipped.
308 <-- UNIX ONLY -->
309 If the operating system isn't an Unix OS, the test is skipped.
314 <-- LINUX ONLY -->
315 If the operating system isn't GNU/Linux, the test is skipped.
320 <-- MACOSX ONLY -->
321 If the operating system isn't Mac OS X, the test is skipped.
326 <-- XCOS TEST -->
327 This test will launch all the necessary Xcos libs. This test
328 will be launched in nw mode.
333 Each test is executed in a separated process, created with the "host" command.
334 That enables the current command to continue, even if the test as
335 created an unstable environment. It also enables the tests to be
336 independent from one another.
340 <title>Platform-specific tests</title>
342 It may happen that the output of a test depends on the platform on which it is
343 executed. In this case, the <literal>.ref</literal> file cannot be correct for
344 all platforms and unit tests may fail for some platform. In this case, we can
345 create a default <literal>.ref</literal> and create additional <literal>.ref</literal>
346 file for each platform.
349 The various platform-specific <literal>.ref</literal> files must have one of the following extensions.
354 <literal>.unix.dia.ref</literal> for Unix platform,
359 <literal>.linux.dia.ref</literal> for GNU/Linux platform,
364 <literal>.linux32.dia.ref</literal> for GNU/Linux platform with 32bits processors,
369 <literal>.win.dia.ref</literal> for Windows platform,
374 <literal>.win32.dia.ref</literal> for Windows platform with 32bits processors,
379 <literal>.macosx.dia.ref</literal> for Mac OS X platform.
384 The algorithm is the following.
385 First, the <literal>.ref</literal> is considered. If this file does not exist,
386 the platform-specific <literal>platform.ref</literal> file is examined depending on the current platform.
391 on Windows platforms: <literal>.win.dia.ref</literal>, <literal>.win32.dia.ref</literal>
396 on Max OS X platforms: <literal>.unix.dia.ref</literal>, <literal>.macosx.dia.ref</literal>,
401 on GNU/Linux platforms: <literal>.unix.dia.ref</literal>, <literal>.linux.dia.ref</literal>, <literal>.linux32.dia.ref</literal>.
407 <title>Examples</title>
408 <programlisting role="example"><![CDATA[
410 // This may take some time
411 // =============================================
416 // test_run("[]","[]");
419 // Test one or several module
420 // =============================================
425 // Test several modules
426 test_run(['time','string']);
429 test_run('optimization|neldermead');
431 // Refer to a module by its path
432 test_run(SCI+'/modules/core');
434 // Launch a specific test
435 // =============================================
438 test_run('time','datenum');
441 test_run('time',['datenum';'calendar']);
444 // =============================================
446 test_run('time',['datenum';'calendar'],'skip_tests');
449 // =============================================
451 // does not check if the .dia and .dia.ref are equal
452 test_run('time','datenum','no_check_ref');
454 // Create the .dia.ref file and does not check if the .dia and .dia.ref are equal
455 test_run([],[],'create_ref');
457 // Does not perform the tests but displays a list of available tests
458 test_run([],[],'list');
460 // Display some examples about how to use this command
461 test_run([],[],'help');
463 // Runs only the non-regression tests, skipping unit tests
464 test_run([],[],'nonreg_test');
466 // Runs only the unit tests, skipping non-regression tests
467 test_run([],[],'unit_test');
469 // Do not check the error output (std err)
470 test_run('boolean','bug_2799','no_check_error_output');
472 // Combine several options
473 test_run([],[],['no_check_ref','mode_nw']);
476 test_run time [] no_check_ref //tests time module with no_check_ref option
479 <programlisting role="example"><![CDATA[
480 // Run unitary tests of an external module (with his path)
481 test_run('SCI/contrib/toolbox_skeleton')
484 <programlisting role="example"><![CDATA[
485 // Export to a XML Xunit file
486 test_run('boolean',[],[],TMPDIR+"/boolean_test_run.xml");
487 test_run('time','datenum',[],TMPDIR+"/time_datenum_test_run.xml");
490 <emphasis role="bold">Selections with wildcard *:</emphasis>
492 <programlisting role="example"><![CDATA[
493 test_run elementary_functions *space
494 test_run elementary_functions dec2*
495 test_run string *ascii*
498 --> test_run elementary_functions *space
499 TMPDIR = C:\MyPath\AppData\Local\Temp\SCI_TMP_3668_1147
501 001/002 - [elementary_functions] logspace....................passed
502 002/002 - [elementary_functions] linspace....................passed
503 --------------------------------------------------------------------------
507 --> test_run elementary_functions dec2*
508 TMPDIR = C:\MyPath\AppData\Local\Temp\SCI_TMP_3668_1147
510 001/004 - [elementary_functions] dec2oct.....................passed
511 002/004 - [elementary_functions] dec2hex.....................passed
512 003/004 - [elementary_functions] dec2bin.....................passed
513 004/004 - [elementary_functions] dec2base....................passed
514 --------------------------------------------------------------------------
518 --> test_run string *ascii*
519 TMPDIR = C:\MyPath\AppData\Local\Temp\SCI_TMP_3668_1147
521 001/003 - [string] isascii...................................passed
522 002/003 - [string] asciimat..................................passed
523 003/003 - [string] ascii.....................................passed
524 --------------------------------------------------------------------------
530 <title>Internal Design</title>
532 The tests are performed in the temporary directory, not in the directory
533 which originally contain the tests files.
536 The .tst script is not run as is. Instead, a header and a footer are
537 inserted at the beginning and at the end of the .tst at the time
538 the script is copied into the temporary directory.
539 The role of this modification is to redirect the output messages
540 into the .dia file, so that the user can have a log file once the test
544 An execution timeout delay (watchdog timer) is setup to 5 minutes
545 for each regular test. To ignore this timeout use the long-time
546 execution (<literal>LONG TIME EXECUTION</literal>) flag.
550 <title>History</title>
553 <revnumber>5.4.0</revnumber>
554 <revdescription>test_run returns a status:
555 <itemizedlist><listitem>
556 Returns %t if no error has been detected
559 Returns %f if any error has been detected
563 <literal>show_diff</literal> and <literal>show_error</literal> added as new options
566 <literal>CLI SHELL MODE</literal> tag is added. Replaces <literal>JVM NOT MANDATORY</literal> (still supported)
569 <literal>test_run</literal> can work on an external module.
572 Fourth argument added to export to a XML file
577 <revnumber>5.5.0</revnumber>
578 <revdescription>32/64bits separation available</revdescription>
581 <revnumber>6.0.0</revnumber>
583 <para>profiling mode added to profile execution with valgrind (Linux only)</para>
585 timeout delay (watchdog timer) set to 5 minutes for single tests without <literal>LONG TIME EXECUTION</literal>
590 <revnumber>6.0.2</revnumber>
592 <para>Tests names with the * wildcard like sin*, *sin, or *sin* are now allowed</para>