With the Help of Brain Scanning and AI Researchers Can Decode What You Are Seeing or Imagining


For a while now, intense research into scanning the brain to decode its contents has been ongoing. Through a progression of various studies, scientists are gradually learning how to interpret what participants see, imagine, remember and even dream.

Significant limitations have however hampered progress severely. It is first necessary to catalog a subject’s unique brain patterns extensively, after which the patterns are compared to a small set of pre-programmed images. These requirements mean that each subject has to undergo expensive and lengthy fMRI testing.

A team of researchers in Kyoto has recently managed to decode and predict what a person is imagining or seeing, by referring to a substantially bigger catalog of images using neural network based artificial intelligence.

Team leader Yukiyasu Kamitani of Kyoto University explained that when a person gazes at an object, their brain processes the patterns they see hierarchically, beginning with the simplest and then moving on to more complex features.

The team used an AI that works on the same principle. The AI that was used is called a ‘Deep Neural Network’, or DNN, and it was originally trained by a team that has since moved on to join Google.

The joint team from ATR (Advanced Telecommunications Research) Computational Neuroscience Laboratories and Kyoto University revealed that brain activity patterns can be translated, or decoded, into signal arrangements of simulated neurons in the DNN when both are shown the same image.

The researchers also found that the higher and lower visual areas of the brain were better at decoding the respective layers of the DNN. This shows a homology between the neural network and human brains.

Kamitani explained that they tested to see if a DNN signal pattern that had been decoded from brain activity could be used to identify objects seen or imagined from random categories. The decoder uses neural network patterns to compare with image data from a huge database. As expected, the decoder was able to identify target objects with a high probability.

Kamitani hopes that the advancement of AI development and brain decoding will improve the image identification accuracy of their technique, and concludes that the bringing of brain science and AI research closer together could potentially pave the way for new brain machine interfaces, and perhaps even bring us closer to understanding consciousness itself.

The study results were published in an article in Nature Communications.