Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

Benjamin Lahner ^1,&, Yalda Mohsenzadeh ^2,3,4,&, Caitlin Mullin ^5,&, and Aude Oliva ¹

PLOS Biology paper (2024) | OSF repository

1 Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA.

2 The Brain and Mind Institute, The University of Western Ontario, London, Canada

3 Department of Computer Science, The University of Western Ontario, London, Canada

4 Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada

5 Vision: Science to Application (VISTA), York University, Toronto, Ontario, Canada

& denotes first authorship

Summary

We find the images that humans tend to easily remember (high memorable images) undergo extended processing late in time in multiple visual regions, including Early Visual Cortex. The representation for image memorability arises after the representations for their low level and high level image features and significantly correlates with neural representations in classical memory regions. Together, this works provides a spatially and temporally unified account of the memorability effect and supports the Levels of Processing Theory, which predicts that information that is better remembered has undergone a greater depth of processing.

Motivation

People tend to remember the images on the left (high memorable) much better than the images on the right (low memorable). Here we ask, how is the brain processing these images differently that makes some better remembered than others?

Methods

We compile a set of 78 high memorable images and 78 low memorable images matched for semantic description and low-level features. We show these images to 15 people while recording high temporal resolution (MEG) and high spatial resolution (fMRI) data. We then correlate the MEG and fMRI data together to obtain the spatiotemporal movie below. We see that visual processing is similar between the high memorable (left) and low memorable (right) image sets up until about 250 milliseconds, when we begin to see significant differences emerging (bottom).

Results: The memorability effect is spatially distributed and temporally late

Five of six regions of interest (ROIs) showed significant differences between responses to high memorable vs low memorable images, including Early Visual Cortex. Note how the high memorable images (red curve) and low memorable images (blue curve) are similar until around ~300 seconds when they diverge; high memorable images continue to be significantly represented but the low memorable images drop towards baseline.

Results: The memorability effect occurs after high level perception

We then used a deep neural network (AlexNet) to determine that our brains represent differences in image memorability significantly after our brains represent low level (lines, edges, colors) and high level (shapes, concepts) features in four visual ROIs.

Results: The memorability effect is represented in classic memory regions

Lastly, we find significant similarity between the representation of image memorability and classical memory regions in the medial temporal lobe (but not in a memory region in the Left Prefrontal Cortex). This result suggests that image memorability is shared between both perceptual and memory regions, further highlighting their relationship.

Funding and Acknowledgements

This work was funded by the Vannevar Bush Faculty Fellowship program funded by the Office of Naval Research grant No. N00014-16-1-3116 (to A.O.; https://basicresearch.defense.gov/); National Science Foundation award 1532591 in Neural and Cognitive Systems (to A.O.; https://www.nsf.gov/); Multidisciplinary University Research Initiative (MURI) award by the Army Research Office grant No. W911NF-23-1-0277 (to A.O.: https://arl.devcom.army.mil/who-we706 are/aro/); the EECS MathWorks Fellowship (to B.L.; https://www.mathworks.com/). No funders had any role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

We thank Dimitrios Pantazis for guidance for the MEG experiments and data analyses. We thank the National Institute of Health for their shared instrument grant 1S10OD021569 and Siemens PrismaFit 3T scanner (Erlangen, Germany) (https://www.nih.gov/). The study was conducted at the Athinoula A. Martinos Imaging Center at the McGovern Institute for Brain Research, Massachusetts Institute of Technology.

Citation

Lahner B, Mohsenzadeh Y, Mullin C, Oliva A (2024). Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response. PLoS Biol 22(4): e3002564. https://doi.org/10.1371/journal.pbio.3002564

Other MEG-fMRI Fusion work from the Olivalab

• Cochlea to categories: The spatiotemporal dynamics of semantic auditory representations.
  Matthew X. Lowe*, Yalda Mohsenzadeh*, Benjamin Lahner, Ian Charest, Aude Oliva and Santani Teng.
  Cognitive Neuropsychology, 2022; doi:10.1080/02643294.2022.2085085
  Project Page - PDF - Fusion Video
• Reliability and generalizability of similarity-based fusion of fMRI and MEG data in the ventral and dorsal visual streams.
  Yalda Mohsenzadeh*, Caitlin Mullin*, Benjamin Lahner, Radoslaw Cichy, and Aude Oliva.
  Vision 2019, 3, 8; doi:10.3390/vision3010008
  Project Page - PDF - Fusion Video
• M/EEG-fMRI fusion primer: resolving human brain responses in space and time.
  Radoslaw Cichy and Aude Oliva.
  Neuron 2020, 107(5), 772-781; https://doi.org/10.1016/j.neuron.2020.07.001
• Similarity-based fusion of MEG and fMRI reveals spatio-temporal dynamics in human cortex during visual object recognition.
  Radoslaw Cichy, Dimitrios Pantazis, and Aude Oliva.
  Cerebral Cortex, 26(8), 3563-3579; https://doi.org/10.1093/cercor/bhw135