A Journey Through Performance Testing

What is Performance Testing?

Performance Testing in general, is a name given to tests that are used to check on how a system behaves and performs. It examines the responsiveness, scalability, stability, speed, reliability, and resource usage of your software and infrastructure. Different types of Performance Tests provide you with different data.

Before conducting a Performance Test, it is important to determine your system’s business goals, so that it can be determined if your system behaves in a satisfactory manner or not in accordance to the needs of your customers.

After running Performance Tests, you can analyze different KPIs, such as the number of virtual users, hits per second, errors per second, response time, latency, and bytes per second (throughput), as well as the correlations between them. Through the reports, you can identify bottlenecks, bugs, and errors, then decide what needs to be done.

When should you use Performance Testing?

Performance Tests should be run when you want to check your website and app performance, which may extend to testing servers, databases, networks, etc. If you follow the waterfall methodology, it is important that a test is conducted with each version released. Similarly, if you are shifting left and going agile, you should test continuously.

It should also be noted that a detailed plan should be prepared before starting Performance Testing activities, to help clarify how Performance Testing will be carried out from both technical and business perspectives.

Here are some planning steps and explanations for each which will come in handy for you next time you want to test the waters!

What is the process of doing a Performance Test?

Doing an end to end Performance Test can be divided into three main parts:

● Performance Test Prerequisites

● Performance Test Planning

● Performance Test Execution

In each of these phases, we need to cover some specific areas for performing a perfect Performance Test adhering to the best practices.

Performance Test Prerequisites

Testers should have a clear idea about the project domain as well as the performance needed for the specific project before starting to plan out a Performance Test.

Testers should gain an understanding of performance criteria such as design, normal and peak loads, system resources, dependencies, workload characteristics and time as the prerequisites for Performance Testing which should be part of any strategy.

We can divide Performance Test Prerequisites phase in to four sub categories:

● Domain KT

● NFR Gathering

● Study on API Documentation/Collection

● Test Estimate Creation

Performance Test Planning

In this phase, we are focusing on the creation of a Performance Test plan mainly focused on scripting. We can divide this phase into five sub categories as follows:

● Test Plan Creation

● Test Data Creation

● Environment Preparation

● Script Creation / Update

● Script Review and Trial Execution

Performance Test Execution

Performance Test Execution refers to running tests specific to Performance Testing such as load test, soak test, stress test, spike test etc. using Performance Testing tools. Performance Test Plans contains detailed information of all the tests which need to be executed within the performance-testing window.

The Performance Test Execution Phase has the following activities to be performed:

● Execute the Documented/ Agreed Performance Test

● Analyze the Performance Test Result

● Verify the Result against defined NFRs

● Prepare an Interim Performance Test Report

● Take Decisions to Complete or Repeat the Test Cycle based on the Interim Test Result

This phase has three sub-phases:

● Test Execution: To run the planned Performance Tests

● Performance Test Execution: Back up

● Result Analysis: To analyze the Test Results and Prepare an Interim Test Report

Let’s dive deeper into the area of Performance Testing processes in the next chapter.

To be continued….

Further Reading:

Gihani Naligama

Software QA engineer

Execute Testing on Autonomous Robot Platform

Robotics is an interdisciplinary branch of engineering and science that includes mechanical engineering, electronic engineering, information engineering, computer science, and much more. Robotics deals with the design, construction, operation, and use of robots, as well as computer systems for their control, sensory feedback, and information processing.

A robot is a machine programmable by a computer capable of carrying out a complex series of actions automatically. It can be autonomous or semi-autonomous. An external control device can guide robots or the control may be embedded within.

When we consider about Robot test engineering nowadays-mobile autonomous robots are progressively entering the mass market. Thus, manufacturers have to perform quality assurance tests on a series of robots. Therefore, tests should be repeatable and as automated as possible. Testing activity applied to mobile robotic systems is a challenge because new features, such as non-determinism of inputs, communication among components and time constraints must be considered. Simulations have been used to support the development and validation of these systems. Coverage testing criteria can contribute to this scenario adding mechanisms for measuring quality during the development of systems.

When we move from software test engineering to robot test engineering we mainly focus on the following five areas.

  1. Stress Testing
  2. Performance Testing
  3. Longevity Testing
  4. Security Testing
  5. Functional Testing

Let’s take a look at each, one by one:

Stress Testing

Stress testing is defined as a type of software testing that verifies the stability & reliability of the system. We use the stress test in Robot test engineering to determine the robustness and error handling under extremely heavy load conditions of the robot as well as its backend. Under stress test following areas are addressed.

  • Highly Accelerated Life Test (HALT)

    HALT is a technique designed to discover weak links of a product. It can trigger failure modes faster than traditional testing approaches. Rather than applying low-stress levels over long durations, HALT testing applies high-stress levels for short durations well beyond the expected field environment. Hardware units like electronics, drive systems, sensors used in the robot are tested under the HALT test by driving robot with different speed limits. Software components, for example, the robot’s control system, which is the brain of the robot, communication between the robot to the backend as well as the monitoring and managing tools of the robot are validated under extreme workload or stress.

    When we come to the control node stress test, we stress the different states of the robot and test how it behaves under stress. Failures that are triggered under the above-mentioned tests do not have pass/fail results and it requires a root cause analysis and corrective action to achieve optimum value from testing. It creates the ability to learn more about the design and material limitations as well as the bottlenecks and provides opportunities to continually improve the design before bringing the robot to market. As most items are only as good as their weakest link, the testing should be repeated to systematically improve the overall robustness of the product.

  • Environmental conditions

    Environmental conditions testing is the measurement of the performance of equipment under specified environmental conditions such as temperature, static charge, obstacles, climate. When we run the robot in different environmental conditions, how environmental factors affect the robot’s smooth functioning and how much stress it creates on hardware as well as software of the robot was tested.

  • Benchmarking values

    Considering the Mean Time Between Failures (MTBF) and the performance statistics of each component, the benchmark is derived for the robot.

    A couple of examples are;

    • The battery charge retention time
    • The motor gear wastage
    • RF antenna interference

Performance Testing

Performance testing is the process of determining the speed, responsiveness, and stability of a computer, network, software program or device under a workload.

In Robot test engineering, we do a performance test to ensure the performance of software, network &, hardware.

Under software, QA should ensure the performance of software resources used like MQTT broker, responsiveness of monitoring & troubleshooting software.

To ensure the network performance, QA should validate the teleoperation control, which used to control the robot externally from a remote location. This involves accessing and controlling the robot in real-time (autonomous mode & manual mode) including in the Network performance test.

The QA should ensure that the performance of the sensor system, control System, and drive system of the robot is carried under hardware performance.

Longevity Testing

Longevity test is an operational testing scheme that uses a baseline work efficiency specification to evaluate large enterprise applications and hardware performance. LT is applied for error checking or heavy usage after a live operational period and is contingent on complexity and size. In robot test engineering QA should carry out the Longevity test and ensure;

  • Hardware (sensors & electronics)

    How the hardware performs in the long run and benchmarks the value for maintenance.
  • Drive system (Performance in continuous operation)

    How the robot’s control node and navigation system performs in the long run and benchmarks the value for maintenance.
  • Environment (Parameters change with runtime)

    How these environmental factors (temperature, static charge, obstacles, climate) affected the robot in the long run and benchmark value for maintenance.
  • Software (Performance over time)

    How backend, managing and monitoring systems behave in the long run and identify the bottlenecks to optimize the software.

Security Testing

Security testing is a process intended to reveal flaws in the security mechanisms of an information system that protects data and maintains functionality as intended. When we come to robot test engineering, security is a very sensible and important component to verify.

First, the reasons for an attack needs to be identified. Then, the methods of attacking needs to be thought of, for each reason. Finding out whether precautions have already been taken for the above methods would be the next step. However, a system cannot be secured 100% due to the existence of unavoidable reasons. Such an instance would be, an attacker trying to remove the nuts and bolts of a robot. This will also be identified as a threat that cannot be avoided or take any precautions to. A scope will be laid out in the initial stages and only these threats will be looked into. Some threats that are out of scope will be kept unattended.

Functional Testing

When we consider about functional testing there are 4 main areas, which we consider.

  • Software
  • Navigation/Autonomous functions
  • Use cases
  • Simulation

Under software testing, we test the robot’s control node which is the brain of the robot by testing its every state with all the possible scenarios and also we test the robot’s utilities like RDL (Report Definition Language) which uses to communicate between robots and the backend. In addition, QA should ensure the software backend as well as managing and monitoring units of the robot-like Troubleshooter application.

Robot Navigation stack plays a major role. The ROS (Robot Operating System ) navigation stack is useful for mobile robots to move from place to place reliably. The job of navigation stack is to produce a safe path for the robot to execute, by processing data from odometry, sensors and environment map.

QA validates the Navigation Stack and the autonomous function of the robot to ensure that the robot navigates safely and correctly with avoiding blockers. A robot is designed for specific use cases and in the case of AZIRO, the robot’s primary use case was to scan for RFID inventory whilst navigating the store floor. The QA should be able to thoroughly validate this use case with special emphasis on the algorithm used by the robot for navigation and testing the edge cases.

When we do the robot functional testing QA use a testing environment. In the test field firstly it is required to assure the intractability of hardware and software by running the robot. Upon fine-tuning the robot’s performance, the next step was to create a more advanced testing environment which was very similar to the actual store environment.

Finally, the robot is deployed in the real-world environment, and a final testing round is carried out to ensure the robot works as expected.

Let’s go to the deeper level of robot testing in the next chapter.

To be continued….



A Methodology for Testing Mobile Autonomous Robots

Gihani Naligama

Gihani Naligama is a QA Engineer at Zone24x7