Facing the truth: Benchmarking the techniques for the evolution of variant-rich systems

Simple item page
dc.contributor.authorStrüber, Daniel
dc.contributor.authorMukelabai, Mukelabai
dc.contributor.authorKrüger, Jacob
dc.contributor.authorFischer, Stefan
dc.contributor.authorLinsbauer, Lukas
dc.contributor.authorMartinez, Jabier
dc.contributor.authorBerger, Thorsten
dc.contributor.editorBerger, Thorsten
dc.contributor.editorCollet, Philippe
dc.contributor.editorDuchien, Laurence
dc.contributor.editorFogdal, Thomas
dc.contributor.editorHeymans, Patrick
dc.contributor.editorKehrer, Timo
dc.contributor.editorMartinez, Jabier
dc.contributor.editorMazo, Raul
dc.contributor.editorMontalvillo, Leticia
dc.contributor.editorSalinesi, Camille
dc.contributor.editorTernava, Xhevahire
dc.contributor.editorThum, Thomas
dc.contributor.editorZiadi, Tewfik
dc.contributor.institutionSWT
dc.date.accessioned2024-07-24T11:47:03Z
dc.date.available2024-07-24T11:47:03Z
dc.date.issued2019-09-09
dc.descriptionPublisher Copyright: © 2019 Copyright held by the owner/author(s).
dc.description.abstractThe evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone&own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.en
dc.description.sponsorshipAcknowledgments. Supported by ITEA project REVaMP2 funded by Vinnova Sweden (2016-02804). We thank the participants of Dagstuhl seminar 19191, all survey participants, and Tewfik Ziadi for input and comments on earlier versions of this paper. This work is supported by the projects Magic P12-TIC1814, HADAS TIN2015-64841-R (co-financed by FEDER funds), MEDEA RTI2018-099213-B-I00 (co-financed by FEDER funds), and TASOVA MCIU-AEI TIN2017-90644-REDT.
dc.description.statusPeer reviewed
dc.identifier.citationStrüber , D , Mukelabai , M , Krüger , J , Fischer , S , Linsbauer , L , Martinez , J & Berger , T 2019 , Facing the truth : Benchmarking the techniques for the evolution of variant-rich systems . in T Berger , P Collet , L Duchien , T Fogdal , P Heymans , T Kehrer , J Martinez , R Mazo , L Montalvillo , C Salinesi , X Ternava , T Thum & T Ziadi (eds) , SPLC 2019 - 23rd International Systems and Software Product Line Conference . ACM International Conference Proceeding Series , vol. A , Association for Computing Machinery , 23rd International Systems and Software Product Line Conference, SPLC 2019, co-located with the 13th European Conference on Software Architecture, ECSA 2019 , Paris , France , 9/09/19 . https://doi.org/10.1145/3336294.3336302
dc.identifier.citationconference
dc.identifier.doi10.1145/3336294.3336302
dc.identifier.isbn9781450371384
dc.identifier.urihttps://hdl.handle.net/11556/1622
dc.identifier.urlhttp://www.scopus.com/inward/record.url?scp=85123043199&partnerID=8YFLogxK
dc.language.isoeng
dc.publisherAssociation for Computing Machinery
dc.relation.ispartofSPLC 2019 - 23rd International Systems and Software Product Line Conference
dc.relation.ispartofseriesACM International Conference Proceeding Series
dc.relation.projectIDMEDEA, RTI2018-099213-B-I00
dc.relation.projectIDTASOVA, MCIU-AEI TIN2017-90644-REDT
dc.relation.projectIDITEA
dc.relation.projectIDVINNOVA, 2016-02804
dc.relation.projectIDEuropean Regional Development Fund, FEDER
dc.rightsinfo:eu-repo/semantics/restrictedAccess
dc.subject.keywordsBenchmark
dc.subject.keywordsProduct lines
dc.subject.keywordsSoftware evolution
dc.subject.keywordsSoftware variability
dc.subject.keywordsSoftware
dc.subject.keywordsHuman-Computer Interaction
dc.subject.keywordsComputer Vision and Pattern Recognition
dc.subject.keywordsComputer Networks and Communications
dc.titleFacing the truth: Benchmarking the techniques for the evolution of variant-rich systemsen
dc.typeconference output
Files