RT Conference Proceedings
T1 Facing the truth: 												Benchmarking the techniques for the evolution of variant-rich systems
A1 Strüber, Daniel
A1 Mukelabai, Mukelabai
A1 Krüger, Jacob
A1 Fischer, Stefan
A1 Linsbauer, Lukas
A1 Martinez, Jabier
A1 Berger, Thorsten
A2 Berger, Thorsten
A2 Collet, Philippe
A2 Duchien, Laurence
A2 Fogdal, Thomas
A2 Heymans, Patrick
A2 Kehrer, Timo
A2 Martinez, Jabier
A2 Mazo, Raul
A2 Montalvillo, Leticia
A2 Salinesi, Camille
A2 Ternava, Xhevahire
A2 Thum, Thomas
A2 Ziadi, Tewfik
AB The evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone&own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.
PB Association for Computing Machinery
SN 9781450371384
YR 2019
FD 2019-09-09
LK https://hdl.handle.net/11556/1622
UL https://hdl.handle.net/11556/1622
LA eng
NO Strüber , D , Mukelabai , M , Krüger , J , Fischer , S , Linsbauer , L , Martinez , J & Berger , T 2019 , Facing the truth : Benchmarking the techniques for the evolution of variant-rich systems . in T Berger , P Collet , L Duchien , T Fogdal , P Heymans , T Kehrer , J Martinez , R Mazo , L Montalvillo , C Salinesi , X Ternava , T Thum & T Ziadi (eds) , SPLC 2019 - 23rd International Systems and Software Product Line Conference . ACM International Conference Proceeding Series , vol. A , Association for Computing Machinery , 23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019 , Paris , France , 9/09/19 . https://doi.org/10.1145/3336294.3336302
NO conference
NO Publisher Copyright: © 2019 Copyright held by the owner/author(s).
NO Acknowledgments. Supported by ITEA project REVaMP2 funded by Vinnova Sweden (2016-02804). We thank the participants of Dagstuhl seminar 19191, all survey participants, and Tewfik Ziadi for input and comments on earlier versions of this paper. This work is supported by the projects Magic P12-TIC1814, HADAS TIN2015-64841-R (co-financed by FEDER funds), MEDEA RTI2018-099213-B-I00 (co-financed by FEDER funds), and TASOVA MCIU-AEI TIN2017-90644-REDT.
DS TECNALIA Publications
RD 28 jul 2024