Analyzing Families of Experiments in SE: A Systematic Mapping Study.
Context: Families of experiments -groups of interrelated experiments with the same goal- are on the rise in Software Engineering (SE). The analysis techniques used to aggregate experiments' results within families may impact the reliability of the joint findings.
Objectives: Identifying the techniques used to aggregate experiments' results within families. Raising awareness on the importance of applying suitable analysis techniques to reach valid and reliable conclusions within families.
Method: A systematic mapping study (SMS) is conducted with the aim of identifying the techniques used to analyze SE families and their conditions of applicability. A series of recommendations to analyze and report families are derived from families' limitations with regard to the data analysis of joint results.
Results: Several techniques have been used in SE to analyze families: Narrative synthesis, Aggregated Data (AD), Individual Participant Data (IPD) mega-trial or stratified, and Aggregation of $p$-values. Ad-hoc procedures are followed to select aggregation techniques within families. AD or IPD stratified are the most appropriate techniques to analyze SE families.
Conclusion: Analysis techniques' selection shall be motivated within research articles reporting families. Data analysis' reporting practices shall be improved to facilitate the assessment of the appropriateness of the aggregation technique used to analyze the family.
Publisher URL: http://arxiv.org/abs/1805.09009
Researcher is an app designed by academics, for academics. Create a personalised feed in two minutes.
Choose from over 15,000 academics journals covering ten research areas then let Researcher deliver you papers tailored to your interests each day.
Researcher displays publicly available abstracts and doesn’t host any full article content. If the content is open access, we will direct clicks from the abstracts to the publisher website and display the PDF copy on our platform. Clicks to view the full text will be directed to the publisher website, where only users with subscriptions or access through their institution are able to view the full article.