DocumentationEEM Project with EEM Timer jobs
Function | This Plugin is a specific check for SikuliX and SmartBear TestComplete based end-to-end (E2E) testing. It runs a project on a remote Windows host via its WinAgent, waits for the execution to end, then fetches generated log files, screenshots or videos and displays them in an overview page. |
|---|---|
Alarming | Run time, Response code, Result code (EEM Timer), Known Error (EEM Timer) |
EEM Project general information
The EEM Project plugin is devised to be used on a device of type PC (EEM Robot). EEM (Enduser Experience Monitor) devices are robot devices which carry out end to end software and services testing using SKOOR Robot, a product based on the open source project SikuliX, or SmartBear’s TestComplete / TestExecute testing tools. The plugin was developed to streamline the configuration and management of end-to-end test runs from a SKOOR Engine server. Tests can be programmed to do functional end to end testing of websites, web services, UI applications or mobile apps.
Communication between the EEM robot and SKOOR Engine is done via the SKOOR Engine WinAgent application. The WinAgent needs to be configured to allow execution of SikuliX or TestComplete / TestExecute binaries and fetching the generated output files. Furthermore, WinAgent needs to be running as an application, not as a service.
The EEM Project plugin consists of the EEM Project configuration itself and any number of attached EEM Timer jobs that can be added underneath it.
EEM Project values and alarm limits
Value / Alarm limit | Description |
|---|---|
Run time | The run time of the whole EEM project |
Response code | 0 = Ok |
Error code | Generic job error code (see section Job error codes) |
If an EEM Project job is linked to a schedule (which defines active and inactive times for job execution) the Check now button does not work during the inactive time configured in the schedule.
EEM Robot device
Before creating an EEM Project job, make sure to configure the device the job belongs to as a device of type PC (EEM Robot). When creating such a device the following configuration option screen is shown:
The EEM Project Infos section is provided for reference of the configured defaults of the properties that are used to run an EEM project. Apart from this, a PC (EEM Robot) device is functionally identical to other device types. Override properties by manually setting one or more of the properties in the Properties section.
Device properties
Properties configured on the device level take effect on all EEM project jobs of that device. If required, some of them can be overwritten on the measurement jobs
Property name | Description |
|---|---|
Agent username | Must be set if a user name and password was configured in the WinAgent settings on the EEM host |
Agent password | Must be set if a user name and password was configured in the WinAgent settings on the EEM host |
Agent secure conn | Must be set if the corresponding setting (SSL encryption for the connection) was configured in the WinAgent settings on the EEM host |
EEM binary path | Must be set if it differs from its default. This must be set corresponding to the EEM execution mode property and, of course, the paths to the software components |
EEM execution mode | For runtime devices, used to perform measurements, choose SikuliX or TestExecute For devices used to develop EEM projects, choose SikuliX Development or TestComplete |
EEM fetch MHT logs (TestExecute only) | If enabled (default), the SKOOR Engine will try to transfer the project run summary logs generated on the EEM robot. These are generated for each run in MHT (MHTML) format and can be viewed with Internet Explorer only. MHT logs files can be quite large, sometimes several MB, especially if the project run generated many errors. Turn this off if there is too little space on the SKOOR Engine to store the logs or if the network bandwidth between SKOOR Engine and the EEM robot is too low for the transfer of large files |
EEM log root directory | This is the root directory where the SikuliX or MHT logs are stored on the EEM robot |
EEM project root directory | This is the root directory where the SikuliX / TestComplete projects are stored on the EEM robot |
EEM remote access | Usually the project runs are carried out on the console of the EEM robot (the screen one would see if a physical monitor were connected). In this case VNC or other means such as Teamviewer are used as a remote access method. However, this leaves the console screen of the robot visible while the tests are running. If security is an issue, the tests can be configured to run within a Windows remote desktop (RDP) session. This allows the console screen to be locked. Users will be disconnected when tests are run. If network level authentication is required, the device must be configured with FreeRDP |
RDP domain | Active Directory Domain used for RDP login |
RDP username/password | Active Directory username and password for RDP login |
RDP resolution | Screen resolution used for EEM measurements. This is especially important for image-based measurements (SKOOR Robot) |
RDP options | Some options can be configured for the RDP session. Please consult the EEM Project Infos section on the device object for details |
RDP logout | To save time between measurements, RDP sessions are not logged out after running a project job. If a logout is required for some reason, this property can be set accordingly |
EEM Project jobs
For EEM Project job configuration, please consult the specific subpages for SKOOR Robot and SmartBear.
EEM Project results
Measurement results are presented on the EEM project and timer jobs as well as in the EEM result browser.
EEM Project queue
The SKOOR Engine collector schedules the EEM Project jobs automatically according to their specified execution interval. This means each EEM Project job can have its individual execution interval and the scheduler makes sure that this interval is met. If there are multiple EEM Project jobs on a EEM device and the scheduler decides that it is time for an EEM Project job to run, it is queued if currently one of the other EEM Project jobs is already running. To view this queue, the following prerequisites must be met:
The EEM device needs to have a Default collector defined in its configuration
At least one EEM Project job must exist on the EEM device
If the above criteria are satisfied, the queue and the status of the currently running project can be viewed by choosing Show details on the EEM device:
EEM Project distribution
Developing test projects requires an installation of the TestComplete or SikuliX software. Development is usually done on different hosts than where the tests are actually run. Where the tests are run, only the TestExecute runtime software needs to be installed (SmartBear only). To distribute developed tests, the SKOOR Engine offers a mechanism which involves creating a zip file of the current project directory on the development host, fetching it to SKOOR Engine, then distributing it to the EEM robots. The SikuliX executable is transferred the same way. To achieve this, follow the guidelines below:
Set the EEM execution mode property to SikuliX Development or TestComplete on the development host device
Make sure all EEM robot host devices on the other hand have this property set to TestExecute or SikuliX
On the development host, develop a new EEM project or edit an existing one. Save the project and close the respective IDE.
Below the development device in the SKOOR Engine UI, on the EEM project that should be distributed to EEM robot hosts, select Distribute to test repository from the Execute/Distribute dropdown list in the job's Job execution section:
Choosing Execute measurement job would simply run the EEM project.
Click OK to save the job
Open the new job using the magnifying glass and click Distribute
The Values section should now show Suite distributed successfully:
The zipped project suite directory will now be placed into the /opt/eranger/collector/eem_repository/from_development/test directory of the SKOOR collector that the EEM Project job is configured on
The archive will be synced to the /opt/eranger/collector/eem_repository/to_execute/test directory, thus making it available for auto-updating.
On the EEM robot device, go to the corresponding EEM Project job and check the Auto update suite control. Then select the Test repository from the list that appears:
Run Check now on the EEM Project job. This will first check if there are any updated project files in the SKOOR Engine repository (below the /opt/eranger/collector/eem_repository/to_execute/test directory). If yes, it will transfer them to the EEM robot, replacing the old project suite directory. Then it will run the test and return its results
If it turns out the test runs fine on the TestExecute host which is enabled for the test repository, one can start distributing the project to the production repository. It is good practice to have a dedicated testing robot, where all developed tests are run first before distributing them to the production robots
It can take up to 4 minutes until SKOOR Engine copies newly uploaded (distributed) projects to all SKOOR Engine agents, so they might not be available immediately for the auto-update
If projects are not distributed, check if the SKOOR Engine, where the EEM Project jobs are configured, has public-key authenticated access via ssh to those SKOOR Engine collectors that the EEM Project jobs run on. The SKOOR Engine fetches all compressed project archives from all SKOOR Engine collectors, then distributes them to all collectors’ repository directories.