Reproducible Research for Pattern Recognition
A hands-on course given at UNESP Bauru, Brazil, on reproducible research for engineers working with software in pattern recognition and machine learning.
This is a course on reproducible research for research engineers working with software applications in pattern recognition and machine learning, given at UNESP Bauru, Brazil. It motivates and explains the concepts behind reproducible research, its implications, and the tools for implementing it at an individual or group level. It is hands-on: students are required to create their own workflows for selected problems. By the end of the course, students should understand the basic concepts of reproducibility, its importance in their daily practice, and how to achieve it with freely available tools and environments.
One of the key aspects of modern technological research lies in the use of personal computers, either for the simulation of known phenomena or for the evaluation of data collected from natural observations. Mashups of these data, organised in tables and figures, are attached to textual descriptions leading to scientific publications. In current practice, the data sets, code and actionable software leading to those results are excluded upon recording and preservation of articles. This panorama slows down potential scientific development in at least two major aspects: re-using ideas from different sources normally implies the re-development of the software leading to the original results, and the reviewing process of candidate ideas is based on trust rather than on hard, verifiable evidence that can be thoroughly analysed.
The course introduces the concept of reproducible research, a term that labels scientific work providing not only a description of the effort leading to the stated conclusions, but also pointing to the data, software and instructions that allow readers to reproduce the author’s results locally, with all required details and in a very short time. The promised gains are considerable, but they do not come without a cost: to boost reproducibility, researchers need to re-organise themselves so as to always be doing reproducible research. The course walks students through the tools and practical exercises required to implement it in their daily activities.
Finally, students are introduced to the BEAT platform, a web-based system for reproducible research. BEAT provides tools to graphically create workflows, write algorithms, and run, log and search for results in a socially interactive way. All the complexity of reproducibility and computation is hidden behind a graphical web interface, and experiments designed inside the platform can be transmitted and reproduced in a matter of seconds.
Program
The length of each topic depends on student motivation and discussions. The minimum course time is around 10 hours, and it can be given over two days of at least 5 hours each.
- Introduction and programming background: the need for reproducibility, how to encode databases and protocols, and the tools for reproducible research in the wild
- Python and Bob: building database packages, using Python and Bob for basic machine learning, and putting it all together
- Going social with the BEAT platform: the requirement for a web-based tool, adapting a workflow to the platform, and going from running experiments to preparing a publication using only a web browser
Prerequisites
Participants should understand the basics of pattern recognition, machine learning and programming. Knowing the Python programming language is a plus, as is familiarity with numerical and scientific programming in Python (NumPy and SciPy).
Materials
Cover: Unesp Bauru by Hernani Arruda Monteiro da Silva, resized, CC BY 2.0.
