October 16, 2024
By Patricia Waldron
Owolabi Legunsen, assistant professor of computer science in the Cornell Ann S. Bowers College of Computing and Information Science, received a Distinguished Paper award for the second year in a row at the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), held Sept. 16-20 in Vienna.
His paper, “An In-Depth Study of Runtime Verification Overheads During Software Testing," with co-author Kevin Guan, a graduate student in the field of computer science, was among 11 papers to receive this award out of 143 presented.
Runtime verification is a computing system analysis technique that involves monitoring a running program to see if it is behaving according to spec. It can pick up on bugs missed by traditional software testing, but has not been widely adopted due to its reputation for requiring too much overhead – i.e., time, memory, and processing power.
In the new research, Guan and Legunsen investigated the overhead for runtime verification of 1,544 Java open-source projects. They reported that the required time for runtime verification varied widely. The overhead was "acceptable" for 40.9% of projects, meaning that it took 12.48 seconds or less. But for other projects, runtime verification took up to 28.7 hours.
Their analysis also showed that most monitoring is highly repetitive, which wastes time. Additionally, about a third of the monitoring time was spent in test code or third-party libraries – but these components cannot safely be skipped.
However, they found that most overhead was dominated not by monitoring but by instrumentation, which is the process of modifying software to add code that monitors its behavior. Faster instrumentation could speed up testing, Guan and Legunsen argue. To offer proof of concept of this finding, the researchers showed that keeping the instrumentation offline during runtime verification, when possible, can greatly reduce overhead.
This new work was the first ever in-depth study of runtime verification overhead. The findings offer new directions for future work to further reduce overhead, which may promote adoption of this useful analysis technique.
Patricia Waldron is a writer for the Cornell Ann S. Bowers College of Computing and Information Science.