Hardware-in-the-Loop and Continuous-Integration
Continuous Integration for embedded systems
Any small change to an embedded system can have the most malicious effects. Moreover, if you find and fix the problems too late, the fix is very expensive. Therefore, it should be possible to test every change to an embedded system within minutes. This is the only way to ensure that you can detect and fix the problems quickly.
For pure software projects, complete automation of the tests via Continuous Integration (CI) helps. In this case, every code change automatically triggers the construction and testing of the complete software. This allows a full regression test to detect bugs in existing and new codes within minutes. Could it also work for embedded systems?
Continuous Integration for Test Automation
Let’s first look at how Continuous Integration works.
- each developer updates his software changes by committing them to a central source code management system, e.g. GIT
- each commit notifies the continuous integration server application (CI server).
- the CI server checks out the change just made and completely rebuilds the software project, including the tests.
- all tests are now automatically executed for the new software version. 4.1 If a test fails, only the developer who made the last change is notified – e.g. by e-mail. 4.2 If all tests run without problems, no one is notified. However, the test results are logged.
- in many cases a complete release with documentation and release notes is built at the same time. This always ensures that the project is ready for delivery at any time and in an automated way.
What is different on embedded systems?
The goal should be to run the tests as quickly as possible after each software change. For unit tests, which are usually pure software tests, this is easy. But how can you achieve this for system or integration tests on embedded systems?
The necessary hardware-in-the-loop tests (HIL tests) or manual system tests are often difficult to automate. • Real hardware parts in the test setup prevent automation and repeatability • In many cases, only the "happy path" can be tested, but not a misbehavior • The test bench is often too expensive to be available for automation for each project, at any time
The problem using the example of an oil pump
Let’s take a look at an example from a real project. It involves the system test for a safety-relevant oil pump.
Test setup objective: Test temperature shutdown for all relevant system states.
The previous setup: The complete system runs in a large climate chamber.
The disadvantages: • Due to the slow performance of the climatic chamber, the complete test would take about 30 days. This makes the test practically worthless for development. • Due to the complex system structure, the tests can only be partially automated. • Only the temperature error can be tested. For other error situations, the motor, pump or hydraulic system must be manipulated manually. • The expensive climate chamber is not available for other important system tests for a very long time.
The new setup: We only test the microcontroller, the drivers and the application software and simulate the complete environment. The test is done as a black box test via the signals on the pins of the microcontroller.
- One tests what changes most often: The software
- You can test not only the temperature shutdown, but also all other error situations.
- In addition to the good case (happy path), all bad cases (unhappy path) can also be tested by simulation and fault injection
- The test execution can be fully automated
- Full regression testing is performed for EVERY change
- The implementation takes only 50 minutes instead of 30 days.
- The hardware setup is significantly cheaper and smaller.
The hardware test setup
Instead of the complete system, only the microcontroller with the complete unchanged software is tested.
The pins and the required signals of the system under test (SUT, right side) are connected to the test system (left side). The environment simulation (motor simulation, hydraulic simulation, temperature simulation, …) and the automated test cases run on the test system.
Due to the complete environment simulation, the SUT can be set to any required state and thus the tests can be completely automated.
If you have more than one project, or if several developers are working on one project, it is recommended to set up the SUT in a 19 inch cabinet. The example shows 8 different projects in a waist-high cabinet. These are connected to the CI server via USB hub.
The software development environment
To be able to create environment simulations and test cases as efficiently as possible, we use a model-driven approach. Here, the ROOM language and the open source tool eTrice are used to develop the software structure of the test system and the environment simulations. The test cases are also developed model-driven with the language CeGe (Case Generator). C code is generated from the models, compiled and transferred to the test system. The results logged during test execution are transformed into various diagrams and standard formats. This makes it easier for developers, project managers and reviewers to interpret the results, learn and start the next iteration.
By reducing the hardware-in-the-loop test to the microcontroller with the complete software, complete automation of the test setup can be achieved. Thanks to the use of continuous integration, the response times for test execution are in the range of minutes instead of days, weeks or even months. Developers can eliminate errors that occur extremely quickly and without dependencies on other changes. This is an important prerequisite for fast, iterative project development.
- Open source modeling tool and code generator eTrice: https://www.eclipse.org/etrice/
- miniHIL hardware and software: https://www.protos.de/produkte/minihil/
Get a professional advice