The fundamentals¶

The test framework provides a framework for standardizing the creation of test cases.

The main features are:

the support of test cases with steps
Support extensions to communicate with the system to test or control
automatic generation of test reports.

Test case¶

The creation of a test case in the solution is standardized.

A test case is divided into 4 sections:

description: description of the different stages of the test
prepare: preparation of adapters and libraries for communicating with the system to be tested or piloted
definition: test flow
cleanup: cleaning phase

The result of a test case is automatically calculated by the framework when the test is completed according to the different stages defined.

There are 3 possible results:

PASS: all the steps of the tests have been successfully executed
FAILED: at least one step is in error after execution
UNDEFINED: at least one step of the test has not been executed

Note

The cleanup section is always called, even if there is an error.

Test steps¶

A test case is divided into sub-steps.

A step is defined by:

a summary of the action to be carried out
the detailed description of the action to be carried out
the description of the expected result to validate the step.

The definition of the test steps must be done in the description section:

self.step1 = self.addStep(
                                              expected="Logged in",
                                              description="Login throught the rest api",
                                              summary="Login throught the rest api",
                                              enabled=True
                                      )

The result of a step is to be specified in the definition section

Example to set the result to PASS or FAILED

self.step1.setPassed(actual="step executed as expected")

self.step1.setFailed(actual="error to run the step")

Warning

Do not forget to start a step with the function start otherwise it is not possible to put the result.

Note

Do not forget to specify the result of a step, otherwise it will be considered as UNDEFINED.

Important

A step set to FAILED can not become PASS thereafter in a test.

Cancellation of a test¶

It is possible to force the execution of a test case by using the stop function in the description section of your test.

Test(self).interrupt(err="aborted by tester")

Using the stop feature will stop the test and automatically call the cleanup section of the test case. In this case, the aborted argument is set to True by the framework to indicate the cancellation of the test.

def definition(self):
      Test(self).interrupt("bad response received")

def cleanup(self, aborted):
      if aborted: self.step1.setFailed(actual="%s" % aborted)

Adding trace¶

The framework provides some functions to add messages during the execution of a test.

The following levels are available:

Example to display a message of type info
Trace(self).info(txt="hello world")
Example to display a warning message
Trace(self).warning(txt="hello world")
Example to display an error message
Trace(self).error(txt="hello world")

Note

If an error message is displayed then the result will automatically be set to FAILED.

Note

Messages appear automatically in the basic report.

Data¶

Public¶

A public space is available on the test server. This space makes it possible to provide files that are necessary during the execution of a test.

The files are stored in the [...]/var/public/ directory on the server.

Warning

This space is common to all projects configured on the server.

Private¶

Private vault only exists while running a test. It can save logs generated or recovered during the execution of the test. These logs are automatically made available to the user in a zip file when the test is completed. They can be retrieved from the client or from the server API.

The logs are organized by directory:

TC-TESTCASE directory - # <id_tc>: contains the logs generated by the test case
ADP directory - # <id_id>: contains the logs generated by the different adapters used during the test

Example to save the text hello world in a` my_logs` file from the test case

Private(self).saveFile(destname="my_logs", data="hello world")

Example to add text to an already existing log file

Private(self).appendFile(destname="my_logs", data="hello world2")

Note

It is also possible to save files from an adapter. They will be automatically stored in a directory with the name of the adapter.

Cache¶

The test framework allows caching of data in the key/value form. This function may be necessary to share data between tests when writing a scenario for example.

Example to save a value in the cache

Cache().set(name="my_data", data="hello")

Read a value from the cache

my_data= Cache().get(name="my_data")
Trace(self).warning(my_data)

Example to capture a data with a regular expression and with record in the cache

my_data="March, 25 2017 07:38:58 AM"

Cache().capture(data=my_data, regexp=".* (?P<TIME>\d{2}:\d{2}:\d{2}) .*")

Trace(self).info( txt=Cache().get(name="TIME") )

It is also possible to rely on a custom parameter to supply the regular expression.

.*session_id=[!CAPTURE:SESSIONID:];expires.*

or in greedy mode

.*session_id=[!CAPTURE:SESSIONID:.*?];.*

Important

The cache exists only during the execution of a test.

Put on hold¶

This function allows you to pause while running a test.

Example of holding for 10 seconds:

Time(self).wait(timeout=10)

Standby example until the current date and time match the specified date:

Time(self).waitUntil(dt='2016-09-12 02:00:00', fmt='%Y-%m-%d %H:%M:%S', delta=0)

Interaction with the tester¶

The framework makes it possible to write semi-automatic tests, ie in interaction mode. This function can be interesting for a test in interactive mode (eg configuration of a device)

Example asking the name of the person:

user_rsp = Interact(self).interact(ask="Your name?", timeout=30.0, default=None)

From the client, the Interact tab automatically appears to answer the question asked during the execution of the test. This window is available from the analysis window.

Note

If no response is provided within the interval, it is possible to provide a default value with the default argument.

Parameters of a test¶

Inputs¶

Input parameters are used to add variables to a test. They are configurable from the client.

There are several types of parameters:

The variables are accessible from a test with the input (...) function

input('DEBUG')

The text parameter

The text type is used to construct parameters that use other parameters or the cache. It is therefore possible to use keywords that will be interpreted by the test framework at the time of execution.

List of available keywords:

Keywords	Description
`[! INPUT::]`	Retrieves the value of a parameter present in the test
`[! CACHE::]`	Retrieves a value present in the cache

Note

The name of a parameter is unique and must be capitalized.

The agents¶

The list of agents can be accessed from a test using the () key mode.

self.ADP_REST= SutAdapters.REST.Client(
                                          parent=self,
                                          destinationIp=input('HOST'),
                                          destinationPort=input('PORT'),
                                          debug=input('DEBUG'),
                                          sslSupport=input('USE_SSL'),
                                          agentSupport=input('SUPPORT_AGENT'),
                                          agent=input('AGENT_SOCKET')
                                         )

Define the agent parameter with json like that {‘name’: ‘agent1’, ‘type’: ‘socket’}

Import / export settings¶

The test parameters can be exported to a dedicated testconfig (tcx) file type. It is therefore possible to prepare the parameters without having the test.

It is possible to import a configuration file into a test. The import will overwrite all the parameters if the name is the same.