Bredereke, J.,
Schlingloff, B.-H.:
Specification Based Testing of the UMTS Protocol Stack.
In:
"Proc. of the 14th Int'l. Software & Internet Quality Week
- QW2001", San Francisco, USA
(29 May - 1 June 2001).
Current methods for testing embedded real-time control software can be classified as structural or specification-based. Structural testing methods try to execute as many different parts of the program code as possible, where coverage is measured in terms of statements, conditionals, branches, function calls, and so on. Specification based methods treat the system under test as a black box and focus on testing the required properties of the system.
We have applied the latter approach in the development of the Radio Link Control (RLC) protocol layer of the Universal Mobile Telecommunication System (UMTS), a new generation of high-speed, multi-media mobile phone systems. The work is part of an ongoing cooperation between Siemens AG, Salzgitter, and Technologie-Zentrum Informatik (TZi), Bremen, where Siemens develops the code for the RLC layer, and TZi provides testing support.
The UMTS user equipment and the base stations are developed at different sites. For such systems, specification based testing is more appropriate than structural testing: in order to ensure inter-operability between devices from different providers, test suites should be based solely on the UMTS international standard by 3GPP (the 3rd Generation Partnership Project) plus additional site-specific requirements, rather than on individual program code.
The 3GPP standard is written partly in English and partly in the specification and description language SDL. The implementation is developed from these sources with the help of suitable tools. Independently from the development of the implementation, we have formalized the requirements for the protocol using the process algebraic language CSP. These formal specifications describe the expected behaviour of the system under test (SUT) at its interfaces. From a CSP test specification of just a few pages, test scripts of arbitrary length are generated automatically by the supporting tool RT-Tester. These test scripts are executed on a separate testing machine automatically and in real time. They may run over long periods of time: hours, days, weeks and more -- without the necessity of manually writing test scripts of an according length. The testing machine and the SUT communicate via TCP/IP sockets, and test results are evaluated on the fly in real time during the run of the SUT, by using the CSP description as a test oracle. To ensure that the tests cover the whole bandwidth of all possible system situations, a mathematically proven test strategy is used. It guarantees that the test coverage increases steadily, approaching a full verification of the specified requirements. Functional properties such as correct transmission and packet routing can be tested together with real-time properties such as correct time slots and sufficient performance within the same test run.
A particular problem for the development of test suites for UMTS is that the standard describing the requirements still is subject to considerable change. Similarly, some details for the machine representation of data at the interfaces are not yet fixed. We therefore designed the testing environment to be highly flexible by
Another preparative to cope with changing requirements is a modular structure of the formal behaviour specification. CSP allows to specify different aspects of a system separately, and to combine these specifications by suitable composition operators. We therefore strive to describe each property of the SUT in a separate requirements module. If a change of requirements affects only one particular aspect of the system, the necessary adaptions in the CSP code are restricted to a single module. This improves the maintainability of the test suites. Furthermore, some crucial aspects of the SUT have to be tested more thoroughly than others. We therefore compose suitably tailored instances of the above modules to complex test suites, and run them using the RT-Tester tool. In all these cases, the actual test scripts are generated automatically on the fly.
In writing the formal requirements specification and comparing its interface to the implementation's interface, we found several ambiguities in the standards document. They can be subject to different, equally legal, incompatible interpretations. These spots will need special care to avoid inter-operability problems with software developed at other sites or by different manufacturers. Furthermore, we also found a few discrepancies between the implementation's interface definition and the interface defined in the standard.
There were several situations in which the SUT did not behave as expected. For example, in a certain state the SUT reacts to a certain signal where no reaction was intended. With structural testing, probably no explicit test script would have been written that checks for a non-reaction in this state. The systematic random exploration of the state space in our approach found this problem automatically. Furthermore, the tests revealed interactions between different instances of the RLC protocol machines. The requirements allow different instances of these machines which behave completely independent. Each instance could be tested separately. But we also performed a test where several protocol machines were tested at the same time, each one against its own copy of the requirements specification. It turned out that in such a setup there was the possibility that the entire SUT could deadlock. Another interesting observation showed up only intermittently, after a certain history of input: even though the SUT always should have gone into the same state, it did not. The on-the-fly generation of the test scripts allowed us to run the test suite for a long period of time, which was necessary to detect this problem.
Our paper is organized as follows: first, we present a brief introduction to specification based testing and to the UMTS protocol stack. Then, we give an overview of the functionality and properties of the RLC layer, and its implementation in SDL. The main part deals with our automated testing environment: formal CSP specifications for the RLC, interfacing between SUT and RT-Tester, and testing of multiple instances in parallel and real-time. Finally, we describe and interpret the testing results and summarize our work.