Our methods - backed by evidence

Bayesian Reasoning Tools

Summary: Bayesian reasoning and decision making is widely considered normative because it minimizes prediction error in a coherent way. In a large scale lab experiment, Cruz et al. provide proof of principle that a Bayesian network modelling tool, adapted to provide basic training and guidance on the modelling process to beginners without requiring knowledge of the mathematical machinery working behind the scenes, significantly helps lay people find normative Bayesian solutions to complex problems, compared to generic training on probabilistic reasoning. 

​

Reference: Cruz, N., Connor Desai, S., Dewitt, S., Hahn, U., Lagnado, D., Liefgreen, A., Phillips, K., Pilditch, T., & Tesic, M. (2020). Widening Access to Bayesian Problem Solving. Frontiers in Psychology, 11(660). https://doi.org/10.3389/fpsyg.2020.00660

The value of probabilities and reasoning structure

Summary: The paper outlines the value of breaking down complex reasoning problems into localised structures of conditional probabilistic relations. When variables of the problem (and their relations) are represented correctly, updating can be performed locally, taking advantage of well-established statistical techniques. This can help reduce imprecision over the evaluative process.

​

Reference: Spiegelhalter, D. J., & Lauritzen, S. L. (1990). Sequential updating of conditional probabilities on directed graphical structures. Networks, 20(5), 579-605. https://doi.org/10.1002/net.3230200507

The need for coherence in expert forecasting

Summary: How we should aggregate expert judgments is important for anytime decision-relevant data are scarce, judgments are being made across a heterogeneous group of forecasters, as well as when individuals must make multiple judgments over time. Across two studies, a method focussed on probabilistic coherence (eliciting probabilities and weighting them) outperformed more typical averaging methods by up to 30% forecast accuracy.

​

Reference: Karvetski, C. W., Olson, K. C., Mandel, D. R., & Twardy, C. R. (2013). Probabilistic coherence weighting for optimizing expert forecasts. Decision Analysis, 10(4), 305-326. http://dx.doi.org/10.1287/deca.2013.0279

The importance of standardising uncertainty communication

Summary: Public policymakers routinely receive and communicate information characterized by uncertainty. Decisions based on such information can have important consequences, so it is imperative that uncertainties are communicated effectively. Many organizations have developed dictionaries, or lexicons, that contain specific language (e.g., very likely, almost certain) to express uncertainty. An evidence-based, standardised lexicon was developed and tested in two policy-relevant domains: climate science and intelligence analysis. In both, evidence-based lexicons were better understood than those now used by the Intergovernmental Panel on Climate Change, the U.S. National Intelligence Council, and the U.K. Defence Intelligence.

​

Reference: Ho, E. H., Budescu, D. V., Dhami, M. K., & Mandel, D. R. (2015). Improving the communication of uncertainty in climate science and intelligence analysis. Behavioral Science & Policy, 1(2), 43-55. [Link]

Systematic review of  relevant errors and biases

Summary: A systematic review of errors and biases in probabilistic reasoning from the psychology and behavioural economics literatures. Usefully, the review covers the strength of the evidence for each putative bias, and describing how they relate to each other.

​

Reference: Benjamin, D. J. (2019). Errors in probabilistic reasoning and judgment biases. Handbook of Behavioral Economics: Applications and Foundations 1, 2, 69-186. [Link]