Multimodal Machine Learning Applications

NeuroVLM: A generative vision-language framework for human neuroimaging

NeuroVLM generates and evaluates vision-language models for human neuroimaging using 30,826 neuroimage-text pairs to perform tasks like text-to-image generation and image-to-text interpretation.

Conformalized Multiview Learning

This paper introduces Coracle, a conformalized framework for multimodal AI that provides valid uncertainty estimates, outperforming existing methods in uncertainty estimation and calibration.

Assessing the Performance of LLMs in Multimodal Information Extraction for Biological Research: A Case Study on LLPS

This study evaluates a multimodal LLM workflow for extracting biological information from text and figures, achieving an F1-score of 0.847 using Gemini 2.5 Pro.

LabOS: The AI-XR Co-Scientist That Sees and Works With Humans

LabOS is an AI co-scientist that integrates computational reasoning with physical experimentation through multimodal perception and human-AI collaboration, enhancing scientific discovery in biomedical and materials science.

An autonomous AI agent for universal behavior analysis

BehaveAgent automates cross-species behavior analysis from video input using multimodal AI without manual intervention or retraining, enabling zero-shot reasoning and end-to-end analysis.

Comprehensive Neural Representations of Naturalistic Stimuli through Multimodal Deep Learning

This paper introduces a deep learning framework for predicting whole-brain neural responses to naturalistic stimuli by integrating visual and linguistic features, achieving better accuracy than unimodal and static multimodal baselines.

Decoding the Moving Mind: Multi-Subject fMRI-to-Video Retrieval with MLLM Semantic Grounding

This paper introduces a framework for multi-subject fMRI-to-video retrieval that improves retrieval accuracy by addressing temporal mismatches, integrating subject-specific patterns, and using semantic grounding from LLMs.

Monkey See, Model Knew: Large Language Models Accurately Predict Visual Brain Responses in Humans AND Non-Human Primates

This paper shows that large language models can predict visual brain responses in humans and non-human primates, suggesting that language-aligned models capture general visual patterns rather than language-specific features.

Assessing the potential of vision language models for automated phenotyping of Drosophila melanogaster

This study evaluates Vision Language Models for automated Drosophila wing phenotype detection and description, finding GPT-4 excels in generating accurate textual descriptions of complex phenotypes.

Video-Language Models as Flexible Social and Physical Reasoners

This paper introduces a video-language model that integrates spatiotemporal embeddings into LLMs to reason about social and physical scenarios, showing high variance explanation in relevant brain regions.

CT Multi-Task Learning with a Large Image-Text (LIT) Model

This paper explores using a large image-text (LIT) model for multi-task CT analysis in lung cancer diagnosis, showing improved performance in segmentation, nodule detection, and classification.