The Unique Complexity of Robotics Software Testing
Testing robotics software is much more complicated than testing standard software! The newly developed flashy features for the robot might work well in controlled office environments, but as soon as the robot is moved into its real environment, things often do not go as planned.
There are many reasons why testing, test automation, and debugging can be difficult in robotics. Based on our experience at Henki Robotics, we’ve compiled a list explaining why.
1. Complex environments
In robotics, every new environment is a new test case! It is impossible to simulate all the different scenarios that your robot will face, and to prepare for everything. There is always something new to encounter: mirrors, windows, new types of obstacles, electrical cords, etc.
💡 Unit testing, integration testing, and system testing are extremely important to prepare for all the most common scenarios and edge cases!
2. Tests are conducted in real-time
Outside of the robotics scope, unit tests and integration tests can usually be run fast and automatically. But testing a real robot system requires taking it to a testing location and running test scenarios manually or semi-automatically. This requires finding or building a test facility, and the whole process can be expensive in terms of time and money.
💡 Having an accurate simulation to run the most common cases will ease this process. However, extensive testing on real robots is still a mandatory step to build a robust and reliable system.
3. Reproducibility
Robotics software is often based heavily on statistical algorithms, which don’t guarantee the same results even when processing the same situations. This can make the testing and debugging difficult.
Also, even a small difference in the stack can lead to widely different outputs; Incorrectly detecting something small as an obstacle with a Lidar sensor can change the whole path of the robot!
💡 To account for the discrepancies in reproducibility, it is important to collect data continuously when the robot is operating. ROS 2 Diagnostics and rosbag recording can be a lifesaver in these situations!
4. Hardware
Hardware is volatile. What if a sensor breaks or doesn’t perform normally? Software needs to account for these cases, which makes error handling and diagnostics an important part of your robotics software stack.
Hardware can also act very differently in different situations. For example, when doing camera-based obstacle detection, the accuracy can differ drastically when taking the robot outdoors and having direct sunlight coming to the cameras, compared to indoor operations.
💡 Invest in production-grade hardware, early diagnostics, fall-back mechanisms, and test the system rigorously. And avoid USB-connectors for sensors (if you know, you know)!
Testing Is Tough — But It’s Worth It
Testing robotics systems can be difficult and time-consuming, but with the right tools and creative solutions, you can build reliable and scalable software. Robust testing is essential for ensuring reliable, safe, and cost-effective robotics deployments in real-world environments.
We would love to hear in the comments what your most memorable moments were while testing robotics systems!
Looking to accelerate your robotics development? Send us a message!