Toward an automated benchmark management system

摘要: The systematic evaluation of program analyses as well software-engineering tools requires benchmark suites that are representative real-world projects in the domains for which or designed. Such benchmarks currently only exist a few research areas and even where they exist, often not effectively maintained, due to required manual effort. This makes evaluating new on software relies current technologies impossible. We describe ABM, methodology semi-automatically mine repositories extract up-to-date sets applications belonging specific domains. proposed facilitates creation such collections it easier release updated versions suite. Resulting from an instantiation methodology, we present collection Java business web applications. serve starting point creating current, targeted suites, thus helps better evaluate program-analysis tools.