Skip to main content

Module 25: Benchmarking & Experimental Methodology - How to Run Fair Experiments and Not Fool Yourself

The difference between science and tinkering is methodology. This module teaches rigorous benchmarking: BBOB/COCO, statistical testing, performance profiles, and how to avoid the common sins of EC experimentation.

Learning Objectives​

  1. Use COCO/BBOB to benchmark an algorithm on 24 functions
  2. Generate ECDF plots and interpret them
  3. Perform statistical tests (Mann-Whitney, Wilcoxon signed-rank)
  4. Understand effect sizes and performance profiles
  5. Know the No Free Lunch theorem and its practical implications

Concept Explanation​

Coming soon.

Code Examples​

Coming soon.

Exercises​

Coming soon.

Milestone Checklist​

  • Ran a COCO benchmark experiment
  • Generated proper ECDF plots
  • Performed statistical comparison of two algorithms
  • Understand why cherry-picked benchmarks are misleading

Was this page helpful?