Adaptive behaviour requires agents to build mental representations of their environment and infer adequate choice behaviour based on these representations. Recent computational models of planning and decision-making have cast these processes as Bayesian inference, assuming that the brain develops a generative (probabilistic) model of the environment to infer the latent structure of the world. I will present experimental work within that framework investigating the neural basis of such probabilistic inference in decision-making. Particularly, I will present a paradigm that allowed us to decompose two distinct types of information content with respect to an agent’s model of the world: information-theoretic surprise that reflects the unexpectedness of an observation, and epistemic value or, more formally, Bayesian surprise, that induces actual shifts in beliefs. Using functional magnetic-resonance imaging in humans we found that dopamine-rich midbrain regions encoded shifts in beliefs but not surprise, which links the function of neuromodulators to building probabilistic models of the environment. In a second paradigm, we investigated the comparison between different models of the world, which has been highlighted in the context of arbitrating between model-free and model-based modes of behaviour. We used a computational framework based on active inference, which provides a candidate account for how beliefs about the environment translate into behaviour based on surprise minimisation, and found evidence in favour of Bayesian model averaging over alternative accounts, implying that subjects’ choices are the average of the choices under each model weighted by their relative evidence (uncertainty). Neurally, we found that model uncertainty – the crucial aspect of the winning model - was encoded in the dorsolateral prefrontal cortex. These results provide interesting hypothesis for understanding pathologic choice behaviour in psychiatry as well as structural learning in future work.