Verification of Scientific Methods and Results

Crowd sourcing approaches can offer great opportunities to address a variety of challenges facing the scientific community. One such example is Philip Morris International’s (PMI) sbv IMPROVER.

To find out about the project and its aims, we spoke to Dr Julia Hoeng, Director of Systems Toxicology at PMI Research & Development.

Q: Can you tell us a little about the overall aims of sbv IMPROVER?

A: sbv IMPROVER stands for systems biology verification: Industrial Methodology for PROcess VErification in Research. It is a collaborative initiative, led and funded by Philip Morris International (PMI), which aims to develop a robust methodology for verifying scientific methods and results.^1,2 The platform hosts a range of open scientific challenges and events, each addressing a unique problem in systems biology, so as to independently verify methods and results and facilitate the analysis of complex datasets.

Based on the principles of crowd-sourcing and collaborative competition, the platform is actively promoting transparency of research processes, open innovation in scientific discovery and enhanced dialogue within the scientific community. The approach is advancing the credibility of scientific techniques and has the potential to complement the classical peer-review process with a more rigorous evaluation of datasets and verification of conclusions. 

This work not only helps PMI in the assessment of Reduced-Risk Products, products with the potential to reduce individual risk and population harm in comparison to cigarettes, but it should also be relevant to other industries that would benefit from the independent assessment of scientific methods and results: pharmaceuticals, environmental risk management and nanotechnology innovation, to name just a few.

Q: What are the benefits of using a crowd-sourcing approach to address scientific problems?

A: Crowd-sourcing ensures credibility, impartiality and the reproducibility of scientific conclusions. It can achieve this because:

• Many contributors are involved, each bringing independent methods and knowledge

• Different solutions can be applied to various aspects of a complex problem

• Benchmarking is unbiased

• Investigators can draw on ‘The Wisdom of Crowds’: the combination of independent solutions often proves to be more robust than any individual solution alone

It is because of these attributes that crowd-souring is growing in popularity as a means to address scientific issues in prediction and verification. sbv IMPROVER is just one such initiative. Others include:

• CASP (Critical Assessment of Protein Structure Prediction)

• DREAM (Dialogue for Reverse Engineering Assessments and Methods)

• CAPRI (Critical Assessment of Prediction Interactions)

• FlowCAP (Critical Assessment of Population Identification Methods)

• BioCreAtIvE (Critical Assessment of Information Extraction Systems in Biology)

• KDD Cup (Knowledge Discovery and Data Mining)

Two key principles underlie each of these initiatives. Firstly, assessment of a prediction by an impartial, outside party is a more rigorous model for verification than self-assessment. Secondly, responses from the community to a scientific challenge can build consensus regarding the most constructive approach for the task.

Q: You mentioned that sbv IMRPOVER could help complement the peer-review system. Can you explain a little more about this?

A: The classical peer-review system is widely considered to be one of the most important mechanisms for quality control of scientific papers. However, it is also acknowledged that the system has weaknesses in that it is open to error / bias and reviewers can only assess whether a paper’s conclusions are supported by the data and results that are shown in the paper (there is likely to be a great deal of data that the paper’s authors have not explicitly addressed). sbv IMPROVER is able to address these issues since:

• Each member of the crowd is identifiable and their contributions are made publicly

• All data is available at all times

• Conclusions are verified based on full datasets, not just by results presented by authors

sbv IMPROVER fills a gap in research quality assessment that is not addressed by classical peer-review. The scientific conclusions obtained through sbv IMPROVER are fully reproducible and thus can be applied with a high degree of confidence.

Q: What challenges and events has sbv IMPROVER hosted? What is coming next?

A: Since 2012, when the platform was first launched to the community, we have run four sbv IMPROVER Challenges and, most recently, one sbv IMPROVER Datathon. 

The inaugural sbv IMPROVER Datathon was held in Singapore on 23-24 September 2016. Teams of computational biologists, bioinformaticians and data scientists met to pool their knowledge and resources and find new ways of approaching large, complex datasets. The result is a set of powerful, open-source data analysis and visualization tools that will be relevant to research scientists working in a range of different fields, as well as to the life-sciences industries and regulatory bodies. These tools will be made available in due course via the Garuda platform, an open connectivity platform that provides a framework to navigate through different applications, databases and services in biology and medicine.

Further challenges and datathons are currently in planning. 

Further information on all aspects of sbv IMPROVER is available at http://www.sbvimprover.com. 

Julia Hoeng was speaking to Anna MacDonald, Editor for Technology Networks. 

About Dr Julia Hoeng

Julia Hoeng is Director of Systems Toxicology at PMI Research & Development where she leads the Systems Biology Program, covering a portfolio of projects in vitro, in vivo and in silico research for product testing. She holds a PhD and Post-doc from Cambridge University and a MS in Bioinformatics from Georgia Institute of Technology, Atlanta, Georgia, USA as well as an MBA from Business School Lausanne. Julia has published numerous articles and book chapters highlighting the use of systems biology approaches for toxicology.

References

1. Meyer P, Alexopoulos LG, Bonk T, et al. Verification of systems biology research in the age of collaborative competition. Nature Biotechnology, 2011;29(9):811-5

2. Meyer P, Hoeng J, Rice JJ, et al. Industrial methodology for process verification in research (IMPROVER): toward systems biology verification. Bioinformatics 2012;28(9):1193-201

Notes

Reduced-Risk Products (“RRPs”) is the term we use to refer to products that present, are likely to present, or have the potential to present less risk of harm to smokers who switch to these products versus continued smoking.  We have a range of RRPs in various stages of development, scientific assessment and commercialization.  Because our RRPs do not burn tobacco, they produce far lower quantities of harmful and potentially harmful compounds than found in cigarette smoke.