К содержанию
РецензированоEN2008

Wake Up and Smell the Coffee: Evaluation Methodology for the 21st Century

Stephen M. Blackburn · Kathryn S. McKinley · Robin Garner

Научные статьи · 2008

Аннотация

Evaluation methodology underpins all innovation in experimental computer science. It requires relevant workloads, appropriate experimental design, and rigorous analysis. Unfortunately, methodology is not keeping pace with the changes in our field. The rise of managed languages such as Java, C#, and Ruby in the past decade and the imminent rise of commodity multicore architectures for the next decade pose new methodological challenges that are not yet widely understood. This paper explores the consequences of our collective inattention to methodology on innovation, makes recommendations for addressing this problem in one domain, and provides guidelines for other domains. We describe benchmark suite design, experimental design, and analysis for evaluating Java applications. For example, we introduce new criteria for measuring and selecting diverse applications for a benchmark suite. We show that the complexity and nondeterminism of the Java runtime system make experimental design a first-order consideration, and we recommend mechanisms for addressing complexity and nondeterminism.

Доказательный фрагмент

«Evaluation methodology underpins all innovation in experimental computer science.»

Образец извлечения

Демо-превью

Evaluation methodology underpins all innovation in experimental computer science. Полный текст публикации CiteX не хранит — доступ к первоисточнику открывается по DOI согласно лицензии издателя и выбранному плану.

DOI не указан
Цитирований
0
Тип
Журнал
Страна
Узбекистан
Спросить CiteX об этой работе

Смежные источники