We have recently published videos where RoutineBot does an automatic check of “BSC Designer” software:
What was left behind the scenes is Compare Suite, which was used to compare actual and expected results, and the batch files that were used to automate the scripts. We’ve gotten some questions asking to share the full testing technology used. In this article you will find a script code that use RoutineBot and Compare Suite for test automation. It’s not the work of the art, but it does what it needs to do.
Preparations for testing
Before we will go into the specific details here are general requirements that we had for an automated testing:
- It should be 100% automated, without user iteration; all tests should be executed via a single batch file.
- All test scripts should take into account that something might go wrong; in this case a test script needs to log the situation appropriately and terminate the tested program so that the next test could be started without problems.
- We need to compare expected results to actual results; if the results don’t match 100%, then we need to be able to find out where the problem was; we will use Compare Suite for that.
- We will need RoutineBot to test parts of the GUI that we cannot test by just passing appropriate parameters via a command line.
Test scripts organization
Each test script is an independent test module. It is located in a separate folder and can be started independently from other scripts. To make it easier we’ve called the folders test-001, test-002 etc…
There is a “Run All Tests.bat” file that prints test names and calls test scripts one by one:
echo Test1: KPI Math
call test-002\run_test.cmd
echo Test2: Examples
call test-003\run_test.cmd
As you might have noticed, all CMD files that start the script have the same name – “run_test.cmd,” the difference is only in the name of the folder. It was done intentionally to make the process of test scripts replicating easy. If you need to create a new script, just make a copy of the script folder and change “call test-003\run_test.cmd” to “call test-004\run_test.cmd” and that’s it.
Test script organization
Let’s take the testing process of BSC Designer as an example. BSC Designer supports command line interface, so testing some of the functions will be easy.
We will need to:
- Open (via command line interface) in BSC Designer a project file that we want to use for a test
- Generate an HTML report for this project file using a command line interface of BSC Designer
- Compare the resulting HTML report with an HTML report that was generated and checked manually
Here is the body of the run_test.cmd script, please see the comments to get an idea about why we need certain parts:
rem we want to see just an output that we echo
@echo off
set bsc_designer_folder=C:\Program Files (x86)\BSC Designer\rem We get in MyPath the current folder of the script
rem and in bsc_project we have the full path to the project that we want to use for a test
set “MyPath=%~dpnx0” & call set “MyPath=%%MyPath:\%~nx0=%%”
set bsc_project=”%MyPath%\02-kpis-math.bsc”rem Now we need to delete the results of all test if any
del /Q /S “%MyPath%\cs_reports\*.*” >> log.txt
del /Q /S “%MyPath%\actual_results\*.*” >> log.txt
del /Q /S “%MyPath%\actual_results\bsc-report.files\*.*” >> log.txtrem If this parameter is set then we are not actually testing, but using this script to create files that will be treated as a correct result
IF “%1″==”prepare_correct_results” GOTO prepare_correctrem We run BSC Designer software passing project name as a parameter
rem the resulted report will be generated in “actual_results” folder
“%bsc_designer_folder%bsc_designer.exe” %bsc_project% -html -export “%MyPath%\actual_results\bsc-report.html” -overwrite >> log.txt:compare
rem Now, we run Compare Suite to compare files from “actual_results” folder and “correct_results” folder“C:\Program Files (x86)\Compare Suite\comparesuite.exe” “%MyPath%\actual_results\bsc-report.html” “%MyPath%\correct_results\bsc-report.html” -r:”%MyPath%\cs_reports\cs_report.htm” -AllowMultipleInstance >> log.txt
:checks
rem We do a kind of self-test, that allows one to track situations when a CS report was not created for some reason
find “STRING_DOES_NOT_EXIST” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test0_notfound
echo ….. SELF-TEST FAILED
goto all_done:test0_notfound
rem echo Test %CurrDirName% ….. SELF-TEST PASSED
goto test0_done
:test0_done
find “File comparison Report” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test1_notfound
goto test1_done
:test1_notfound
echo ….. SELF-TEST FAILED
goto all_done:test1_done
rem So, if a self-test was passed, it means that a cs_report file exists and contains some data about the difference between expected and actual results
rem Now, we will try to find the string in the report that corresponds to the Similarity, % record.
rem if we find there 100, then two files are equal and the test was passed.
find “<td class=””val””>100</td>” %MyPath%\cs_reports\cs_report.htm >> log.txt
if %errorlevel% equ 1 goto test2_notfound
echo ….. +
goto test2_done
:test2_notfound
echo ….. FAILED
goto test2_done
:test2_done
goto all_done:prepare_correct
rem We need this part to be able to prepare correct results in the beginning
echo Preparing correct results
del /Q /S “%MyPath%\correct_results\*.*” >> log.txt
del /Q /S “%MyPath%\correct_results\bsc-report.files\*.*” >> log.txt
“%bsc_designer_folder%bsc_designer.exe” %bsc_project% -html -export “%MyPath%\correct_results\bsc-report.html” -overwrite >> log.txt
echo Finished preparing
pause:all_done
If the test was passed correctly, then you’ll see in the batch output “….. +” if not you’ll see “….. FAILED”
Running test with RoutineBot
Now, let me show you how to execute the test with RoutineBot. Instead of running BSC Designer in a command line mode we will need to start a RoutineBot script:
set script_name=script.rbp
set “MyPath=%~dpnx0” & call set “MyPath=%%MyPath:\%~nx0=%%”
“C:\Program Files (x86)\RoutineBot\Interpreter.exe” “%MyPath%\%script_name%” -Warnings “ON” -LogFile
“C:\Tmp\bsc_designer_tests\_log.txt”
xcopy “P:\Project\Projects\BSC_Designer\tests\z_rb_logs\*.*” “%MyPath%\actual_results\*.*” /e /i /h >> log.txt
As you can see, RoutineBot script needs to write its log into C:\Tmp\bsc_designer_tests\_log.txt. Then, we follow the same algorithm, we have the log for the correct results and we have the log for actual results. We pass these two files to Compare Suite (here are the details about Compare Suite’s command line parameters).
Running all test
Each test was written as a separate module. You can always run a selected test script or run all of them using run_test.cmd file.
Hopefully your test log will look something like this:
If not, you can always check cs_reports folder where we saved the results of the Compare Suite comparison to find out what worked wrong.
Feel free to share your thoughts and experience in the comments.
Leave a Reply
You must be logged in to post a comment.