Testing variants and product lines for embedded systems

Key questions when testing variants and product lines

When developing and testing variants, you will usually be confronted with the following questions:

• Which types of variability are there?
• How are variants and product lines related?
• How can variability be described?
• How can variability be implemented and configured in the product architecture?
• How can all product variants be tested with manageable effort?

Types of variability

Positive variability

Here, only the features required for a specific variant are integrated into the system.

• Advantages: Simpler systems that are easier to test.
• Disadvantages: Each variant requires its own software version, which complicates maintenance and updates.

Negative variability

Here, the system contains all features, and activation takes place via data configurations.

• Advantages: A single software basis for all variants, easy subsequent changes.
• Disadvantages: High configuration complexity, difficult testing due to many possible combinations.

In addition, a distinction is made between:

• Data variability (different parameter values)
• Behavioural variability (different system responses)
• Structural variability (different system components)

Best practice: Complex behavioural variability can often be replaced by finer structural variability, which simplifies testability.

Product lines and variant management

Product lines help to strategically organise variability. Features are grouped into product families, which significantly reduces testing and production complexity. A product from a product family contains a subset of the family’s features. Examples include platform strategies in the automotive industry or modular household appliance series.

Strategic pre-grouping of dominant features into product families or platforms can significantly reduce variability. This reduces the complexity of software development and testing.

Test strategies for variants

Test levels for embedded systems

1. Unit and software integration tests: Individual software components can be easily tested for different variants. However, concurrency tests are difficult to perform. Interaction with the hardware cannot be tested, nor can different hardware configuration variants. However, test automation is very easy to implement.

2. The microcontroller is considered a black box – including application and drivers. Different hardware configurations can be simulated via the controller pins. The entire software in its respective variant configuration can be tested completely. Thanks to simple hardware, automated testing is possible without any problems.

3. Hardware-in-the-loop tests (HiL): Here, the entire control unit, including sensors, actuators and power components, is tested. The achievable test depth is often limited by specific hardware restrictions, such as the distance between the test system and the software. If the sensors or actuators differ between variants, several test setups are required. Automation is possible, but usually costly.

4. Tests in the real system are usually limited to specific variants, as mechanical and electrical setups are required. Automation is very costly or not possible at all.

Variants can be easily tested at the software level, but it becomes more difficult when different hardware is required. Simulations of hardware environments can help to test different variants at the application level.

Focus on HW/SW integration testing

Since many variants can be tested automatically with minimal hardware effort, we usually concentrate on this level.

The miniHIL hardware platform offers a simple way to flexibly simulate hardware variants. IO connections can be automatically switched via a patch panel to test different wiring variants.

Requirements for test hardware and software

Test hardware

• Direct, flexible control via the microcontroller pins using switchable electrical signals.
• Cost-effective test hardware to enable separate test setups for different microcontrollers.
• Support for different sensor and actuator interfaces to realistically map different hardware configurations.
• MiniHIL platform with automatic rewiring of IOs for different variants (e.g. digital, analogue, SPI, I2C, CAN, LIN, PWM).

Test software

• Switchable IO infrastructure for the different variants.
• Closed- and open-loop simulation for simulating different test scenarios, both for good and error cases (error injection).
• Automatic generation of test cases for many feature combinations.
• Full automation of tests via continuous integration processes.
• Model-based test approaches for systematic validation of variants. Support for variant test models to systematically organise cabling, drivers, simulations and test cases.

Test automation with continuous integration

Through continuous integration (CI), every code change can be tested for all variants within minutes. This includes:

• Automated building, testing and delivery
• Fast feedback within 5 – 60 minutes
• Testing of all variants on all target platforms with every change

Hardware platform for variant testing

Signals can be flexibly wired on the test platform and switched at runtime to test different wiring variants. The miniHIL can be dynamically adapted to different microcontrollers and sensor configurations.

A formal test case model can help to efficiently generate test cases for variants. State transition tests can be used to cover all possible variant paths and data combinations.

Summary

The targeted use of HW/SW integration tests with miniHIL and automated test case generation allows variants to be tested efficiently and cost-effectively. This enables scalable quality assurance, even for complex product lines. In our presentations, these methods are explained in detail and a live demo of a variant test is shown.