Browsing by Keyword "Benchmark"
Now showing 1 - 3 of 3
Results Per Page
Sort Options
Item Facing the truth: Benchmarking the techniques for the evolution of variant-rich systems(Association for Computing Machinery, 2019-09-09) Strüber, Daniel; Mukelabai, Mukelabai; Krüger, Jacob; Fischer, Stefan; Linsbauer, Lukas; Martinez, Jabier; Berger, Thorsten; Berger, Thorsten; Collet, Philippe; Duchien, Laurence; Fogdal, Thomas; Heymans, Patrick; Kehrer, Timo; Martinez, Jabier; Mazo, Raul; Montalvillo, Leticia; Salinesi, Camille; Ternava, Xhevahire; Thum, Thomas; Ziadi, Tewfik; SWTThe evolution of variant-rich systems is a challenging task. To support developers, the research community has proposed a range of different techniques over the last decades. However, many techniques have not been adopted in practice so far. To advance such techniques and to support their adoption, it is crucial to evaluate them against realistic baselines, ideally in the form of generally accessible benchmarks. To this end, we need to improve our empirical understanding of typical evolution scenarios for variant-rich systems and their relevance for benchmarking. In this paper, we establish eleven evolution scenarios in which benchmarks would be beneficial. Our scenarios cover typical lifecycles of variant-rich system, ranging from clone&own to adopting and evolving a configurable product-line platform. For each scenario, we formulate benchmarking requirements and assess its clarity and relevance via a survey with experts in variant-rich systems and software evolution. We also surveyed the existing benchmarking landscape, identifying synergies and gaps. We observed that most scenarios, despite being perceived as important by experts, are only partially or not at all supported by existing benchmarks-a call to arms for building community benchmarks upon our requirements. We hope that our work raises awareness for benchmarking as a means to advance techniques for evolving variant-rich systems, and that it will lead to a benchmarking initiative in our community.Item Feature location benchmark for extractive software product line adoption research using realistic and synthetic Eclipse variants(2018-12) Martinez, Jabier; Ziadi, Tewfik; Papadakis, Mike; Bissyandé, Tegawendé F.; Klein, Jacques; Traon, Yves le; SWTContext: It is common belief that high impact research in software reuse requires assessment in non-trivial, comparable, and reproducible settings. However, software artefacts and common representations are usually unavailable. Also, establishing a representative ground truth is a challenging and debatable subject. Feature location in the context of software families, which is key for software product line adoption, is a research field that is becoming more mature with a high proliferation of techniques. Objective: We present EFLBench, a benchmark and a framework to provide a common ground for the evaluation of feature location techniques in families of systems. Method: EFLBench leverages the efforts made by the Eclipse Community which provides feature-based family artefacts and their plugin-based implementations. Eclipse is an active and non-trivial project and thus, it establishes an unbiased ground truth which is realistic and challenging. Results: EFLBench is publicly available and supports all tasks for feature location techniques integration, benchmark construction and benchmark usage. We demonstrate its usage, simplicity and reproducibility by comparing four techniques in Eclipse releases. As an extension of our previously published work, we consider a decade of Eclipse releases and we also contribute an approach to automatically generate synthetic Eclipse variants to benchmark feature location techniques in tailored settings. We present and discuss three strategies for this automatic generation and we present the results using different settings. Conclusion: EFLBench is a contribution to foster the research in feature location in families of systems providing a common framework and a set of baseline techniques and results.Item A literature review and comparison of three feature location techniques using ArgoUML-SPL(Association for Computing Machinery, 2019-02-06) Cruz, Daniel; Figueiredo, Eduardo; Martinez, Jabier; SWTOver the last decades, the adoption of Software Product Line (SPL) engineering for supporting software reuse has increased. An SPL can be extracted from one single product or from a family of related software products, and feature location strategies are widely used for variability mining. Several feature location strategies have been proposed in the literature and they usually aim to map a feature to its source code implementation. In this paper, we present a systematic literature review that identifies and characterizes existing feature location strategies. We also evaluated three different strategies based on textual information retrieval in the context of the ArgoUML-SPL feature location case study. In this evaluation, we compare the strategies based on their ability to correctly identify the source code of several features from ArgoUML-SPL ground truth. We then discuss the strengths and weaknesses of each feature location strategy.