October 2020
Volume 20, Issue 11
Open Access
Vision Sciences Society Annual Meeting Abstract  |   October 2020
Evidence that the Brain’s Physics Engine Runs Forward Simulations of What will Happen Next
Author Affiliations & Notes
  • RT Pramod
    Massachusetts Institute of Technology
  • Michael Cohen
    Massachusetts Institute of Technology
    Amherst College, Amherst, MA
  • Kirsten Lydic
    Massachusetts Institute of Technology
  • Josh Tenenbaum
    Massachusetts Institute of Technology
  • Nancy Kanwisher
    Massachusetts Institute of Technology
  • Footnotes
    Acknowledgements  This work was supported by NIH grant Grant DP1HD091947 to N.K and National Science Foundation Science and Technology Center for Brains, Minds, and Machines.
Journal of Vision October 2020, Vol.20, 1521. doi:https://doi.org/10.1167/jov.20.11.1521
  • Views
  • Share
  • Tools
    • Alerts
      This feature is available to authenticated users only.
      Sign In or Create an Account ×
    • Get Citation

      RT Pramod, Michael Cohen, Kirsten Lydic, Josh Tenenbaum, Nancy Kanwisher; Evidence that the Brain’s Physics Engine Runs Forward Simulations of What will Happen Next. Journal of Vision 2020;20(11):1521. https://doi.org/10.1167/jov.20.11.1521.

      Download citation file:

      © ARVO (1962-2015); The Authors (2016-present)

  • Supplements

Human vision enables us not only to recognize what is where, but to understand the physical properties, relations and forces in a scene and use this information to predict what will happen next. Recent work suggests that these intuitive physical inferences are based on probabilistic simulations of a mental physics engine akin to the physics engines used in video games. Indeed, parietal and frontal regions have been implicated as the “brain’s physics engine”, as they are strongly engaged during intuitive physical inference, and they contain information about object mass. Here, we used fMRI to test the hypothesis that these brain regions conduct simulations of what will happen next. Specifically, we predicted a higher response in these regions for static images of real-world scenes that depict a) unstable configurations of objects or of people in precarious positions (expected to induce forward simulation) than b) stable configurations (where less simulation is expected). Six subjects fixated a cross through the experiment (verified via eye-tracking), and performed an orthogonal 1-back task on stimuli arranged in a blocked design. As predicted, we found significantly higher responses in independently-defined parietal “physics regions” when participants viewed unstable vs stable scenes (p=0.004 for a paired t-test across subjects). Moreover, similar effects were found in visual motion area MT, also consistent with greater simulation for unstable than stable stimuli. This increased response is unlikely to reflect differential eye movements, low-level stimulus differences (as stable versus unstable stimuli elicit equal responses in V1 and were not decodable in early layers of a CNN), or differential attention (as no increased response was found for animate rather than physical instability, e.g. a person being chased by a shark). These results suggest that “the brain’s physics engine” computes information about physical stability based on forward simulations of what will happen next.


This PDF is available to Subscribers Only

Sign in or purchase a subscription to access this content. ×

You must be signed into an individual account to use this feature.