Limitations of microbiome measurement: Prof. Gloor shares insights with ISAPP

February 20, 2019

The number of papers published on the human microbiome is growing exponentially – but not all of the studies are equally well designed or reported. Evaluating the latest research requires a basic understanding of the latest approaches to microbiome methods and data analysis.

To help equip scientists not conducting microbiome research with the tools to understand microbiome-focused publications, ISAPP hosted a webinar titled Understanding microbiome experiments: a critical assessment of methods and data analysis. The webinar featured Gregory Gloor, PhD., Professor, Department of Biochemistry, Schulich School of Medicine and Dentistry, The University of Western Ontario, Canada.

Prof. Gloor’s slides are available here.

Prof. Gloor opened his talk with a sobering perspective: the current body of microbiome publications is fraught with problems. There is a fundamental lack of reproducibility in the microbiome field (Sinha et al. 2017). This is largely due to the large number of tools available and a lack of an a priori established research plan for microbiome analysis, which should be consistently followed throughout a project. At every step of the way, many decisions must be made regarding wet lab methods, bioinformatics toolsets and statistics to use. Different choices lead to different results. Once the biological specimens are assayed, choices for bioinformatics and statistical analyses can greatly influence the conclusions. In short, it’s possible to view the data through so many different lenses that eventually a researcher can find a story worth telling. How close that story comes to the truth is a principle that sometimes is sacrificed for the sake of an interesting story.

Another important challenge to the field is representative sampling. Too few samples are typically taken, often because of cost limitations, so that the samples do not reasonably approximate the truth about the environment being sampled. Conclusions from such studies result in both many false positives and many false negatives.

Prof. Gloor also warned about outsourcing microbiome analysis. Commercial entities often use every metric, hoping the customer will get some outcome they hoped for. Further, their tools are often outdated or proprietary. So caution must be used – there is no substitute for expertise.

Some suggestions for improving outcomes were offered:

• Each project should stipulate a research approach and outcome a priori, which is consistently followed throughout the project.
• Methodological consistency is important within a lab, but analytical methods do not necessarily need to be standardized across all labs. If all labs use the same methods, consistent, but incorrect, outcomes may result. So use of different metrics is good, but methods should be consistent within a project. The value of different research groups using different methods to ask particular research questions is that if the same result emerges from different approaches, it increases confidence that the results are true.
• Gloor cautioned that microbiome datasets are compositional, and compositional data approaches must be used (Gloor et al 2017).
• Functional readouts have less methodological variation than taxonomic readouts. Therefore, functional analysis of shotgun metagenomics or shotgun metatranscriptomics is typically a more reproducible, and also more informative, readout.
• Recent advances have significantly decreased the cost of performing shotgun metagenomics for both taxonomic and functional readouts (Hillmann et al 2018).
• There are now near-complete microbial genomic datasets available for European, North American and Asian populations (Almeida et al 2019) that will make it easier to functionally map datasets.

Prof. Gloor mentioned an interesting aside: prior clinical trial registration, ~60% of large clinical trials showed benefit of the intervention being tested. After the registration process required declaration of primary research outcomes, that number dropped to closer to 10% (Kaplan and Irvin 2015). This suggests that primary outcomes and analysis methods need to be in place to restrict researcher bias. Right now such mechanisms are insufficient in the microbiome field.

Prof. Gloor’s paper, Microbiome Datasets Are Compositional: And This Is Not Optional, provides great background reading for this webinar.

This webinar was developed by ISAPP Industry Advisory Committee representatives as an extension of the annual IAC Learning Forum.

Dr. Gloor is a professor of biochemistry with broad experience in molecular biology, genetics and genomics. His research is focused on the development of tools to examine 16S rRNA gene composition, gene expression of mixed population samples and metabolomic analysis of clinical samples. He is currently working on developing and adapting principled methods to characterize correlation and differential abundance in sparse, high throughput sequencing data as generated in 16S rRNA gene sequencing surveys, meta-genomics and meta-transcriptomics. One of his primary contributions has been the ALDEx2 tool in Bioconductor for the analysis of high-throughput experiments that generate counts per sequence tag: 16S rRNA gene sequencing, metagenomics, transcriptomics and selex-type experiments.