Brain-Inspired AI Cuts Video Processing Time – Neuroscience News

Summary: Researchers introduced EMF-dVAE (Efficient Multimodal Fusion with a discrete Variational Autoencoder). Inspired by the selective attention mechanisms of the human brain, the AI framework filters out redundant visual frames in video streams to focus strictly on the most informative moments.

Key Facts

• 65% Reduction in Processing Time: EMF-dVAE reduced the average processing time for a two-minute video clip from 52 seconds to 18 seconds during inference.
• High Efficiency Feature Selection: The framework achieved state-of-the-art predictive accuracy while discarding nearly 85% of visual data, utilizing only 15.42% of total visual features.
• Brain-Inspired Selective Attention: Modeled after human conversational dynamics, the AI uses audio cues to determine when to allocate visual processing resources rather than continuously analyzing every video frame.
• Noise Elimination Enhances Accuracy: Ignoring redundant and uninformative video frames prevented visual noise from degrading model predictions, resulting in higher accuracy alongside faster processing.
• Sustainability & Real-Time Deployment: The framework provides a scalable model for reducing energy consumption in video-heavy multimodal AI applications, making real-time feedback feasible on consumer-grade hardware.

Source: JAIST

Artificial neural networks were originally inspired by the human brain, but they are still far less efficient at processing information. One reason the human brain is so efficient is its ability to focus only on the most relevant information and allocate cognitive effort based on the task.

As artificial intelligence (AI) systems increasingly analyze multiple types of data, such as text, video, audio, and images, this ability to focus on the most important information is becoming increasingly important for reducing computational time and resource use.

Most video frames contain little useful information, yet conventional AI systems still analyze them all, increasing computational cost and sometimes introducing noise that can reduce prediction accuracy.

A research team, led by Professor Shogo Okada from the Japan Advanced Institute of Science and Technology (JAIST), Japan, along with his Doctoral Student Hung Le from JAIST, has developed an AI model that identifies the key moments in a video, reducing the processing time for a 2-minute clip, from 52 seconds to 18 seconds. The findings of the study were made available online on July 11, 2026, and will be published in Volume 137 of the journal Information Fusion on January 01, 2027.

Doctoral Candidate Le, who is the first author of the paper, compares the model to the way humans naturally pay attention during a conversation: “Humans do not constantly keep their eyes on their conversation partner during a conversation. They first notice changes in sound and only direct their gaze at the specific moments they perceive as important.”

The model, called EMF-dVAE (Efficient Multimodal Fusion with a discrete Variational Autoencoder), consists of two main components: a discrete variational autoencoder (dVAE) and a multimodal fusion (MF) network.

During training, the dVAE is given partially corrupted visual data, where the accompanying audio is first used to identify which visual segments should be masked. The model learns to reconstruct the missing visual information and, through this process, learns which visual regions are most informative for the task.

Once training is complete, the dVAE identifies only the most important visual segments during inference. These selected visual features are then combined with the audio and language information by the MF network to produce the final prediction.

Since the model only analyzes a fraction of the video during inference, it requires much less computation while maintaining high accuracy. When tested on the ETS-Interview dataset, which contains 1,891 2-minute job interview videos from 260 participants, EMF-dVAE achieved state-of-the-art performance while using only 15.42% of the available visual features.

It also reduced the processing time per video by approximately 65%, from 52 seconds to 18 seconds. Remarkably, by ignoring nearly 85% of the visual data, the model became both faster and more accurate, as redundant video frames can obscure informative signals while increasing computational cost.

“Much like the human brain, which focuses its attention on the most relevant moments, the AI automatically adjusts how much video data it analyzes for each clip, allowing it to allocate computational resources more efficiently,” says Prof. Okada.

By making video analysis more efficient, the framework could enable AI-powered video interview coaches and communication-training tools that provide users with affordable, real-time feedback at near-expert levels. Because the framework processes only the most informative visual information, it also has the potential to reduce computational cost and energy consumption, supporting the development of more sustainable AI systems.

“Video is becoming the dominant form of data, and AI that must watch everything will not scale—economically or environmentally. Within 5–10 years, AI that budgets its attention the way humans do could make multimodal assistants, interview coaches, tutoring systems, and communication-support robots affordable and responsive on everyday devices,” says Prof. Okada.

Key Questions Answered:

Editorial Notes:

• This article was edited by a Neuroscience News editor.
• Journal paper reviewed in full.
• Additional context added by our staff.

About this AI and visual neuroscience research news

Author: Shogo Okada
Source: Japan Advanced Institute of Science and Technology
Contact: Shogo Okada – Japan Advanced Institute of Science and Technology
Image: The image is credited to Neuroscience News

Original Research: Open access.
“Audio-guided visual selection for efficient multimodal fusion via a discrete variational autoencoder” by Hung Le, Hung-Hsuan Huang, Candy Olivia Mawalim, Chee Wee Leong, Shogo Okada. Information Fusion
DOI:10.1016/j.inffus.2026.104613

---

Abstract

Audio-guided visual selection for efficient multimodal fusion via a discrete variational autoencoder

In multimodal learning, not all parts of all modalities are of equal importance. An additional modality can introduce overlapping signals while requiring a significant increase in computation. As a result, there are many cases where computational requirements for a multimodal model are increasing while its performance is degrading.

Early detecting and subsequently fusing only the complementary segments that exist in the high-resource modality by previewing the low-resource modality is how one can overcome this problem. This work proposes a framework to dynamically learn the selection of such salient segments in the vision feature space by previewing the acoustic features of the audio.

The framework contains a self-supervised discrete variational autoencoder (dVAE) component and a traditional supervised multimodal fusion component. The former is trained to produce a discrete masked vector in the latent space, as opposed to the traditional continuous latent VAE, by using Binary Concrete distributions as the encoder and the prior distributions.

This masked vector removes the non-salient segments in the vision feature space before passing them to the latter component. At inference time, only salient vision features need to be extracted, making the framework very efficient.

The empirical experiment on a large-scale video interview dataset reveals that this approach outperforms SOTA in terms of both efficiency and performance, requiring only 15.42 percent of the original vision features (a 23% relative improvement) while improving correlation from 0.6792 to 0.7196 and F1-score from 74.73% to 76.64%.