Deep diffusion models for decoding visual stimuli from brain scans

Faculty: Sridharan Devarajan (CNS & CSA), Ambedkar Dukkipati (CSA)

Description

1. We know little about how the brain’s visual system encodes visual objects and how it pays attention to these objects in cluttered scenes.
2. In this project, we will have subjects view and pay attention to particular objects or object features in cluttered scenes.
3. We will simultaneously be recording function MRI (fMRI) scans in the scanner at IISc we will also use some existing databases with several thousand scans.
4. From these scans, we will try to decode — classify and literally “reconstruct” — what the subject was viewing/paying attention to in the cluttered scene using deep neural networks, and deep generative models.
5. The project will involve advanced generative modeling including instance conditioned GANs and deep diffusion models.

Additional reading

• https://www.nature.com/articles/ncomms15037
• https://proceedings.neurips.cc/paper/2019/file/7d2be41b1bde6ff8fe45150c37488ebb-Paper.pdf
• https://sites.google.com/view/stablediffusion-with-brain/