Abhidip Bhattacharyya

Manning College of Information and Computer Sciences

140 Governors Drive, Room 254

Amherst, MA 01003

I am a post-doc at CICS at University of Massachusetts Amherst working with Prof. Erik G. Learned-Miller. My research interests include Computer vision and Computational Linguistics with a focus on decoding semantics in Vision and Language. In my research I am exploring how language semantic can help in vision task.

Before joining UMass, I earned my PhD in Computer science and Cognitive Science from University of Colorado Boulder. I was advised by Prof. Martha Palmer and Prof. Christofer Heckman. I was part of CLEAR.

Prior to my PhD I worked at TATA Resrach, Development and Design Center on extracting and formalizing bussiness rules from bussiness documents. I recieved my masters from Indian Statistical Institute, Kolkata.

I am interested in Machine Learning, Deep Neural Network, Computer Vision, Computational Linguistics.

I pursue painting and drawing and photography as my main hobbies.

news

Jun 2, 2026	Our paper “For What Reason? Interpreting Models’ Encoding of Causation and Antithesis” is accepted at SIGDIAL 2026.
Nov 13, 2025	https://www.amherststemnetwork.com/2026/02/23/uncovering-the-mysteries-behind-artificial-intelligence-professor-abhidip-bhattacharyya-aims-to-increase-transparency-in-vision-language-models/
Nov 13, 2025	Presented my work at Amherst College
Aug 2, 2023	Earned my PhD in Computer Science and Cognitive Science!!!
Feb 6, 2023	I will be giving talks at CS Openhouse on Feb 24th and CLEAR openhouse on March 8th.

selected publications

CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling

Abhidip Bhattacharyya, Martha Palmer, and Christoffer Heckman

In The 12th Joint Conference on Lexical and Computational Semantics, Co-located with ACL, Jul 2023

Bib PDF

@inproceedings{crapes,
  title = {CRAPES:Cross-modal Annotation Projection for Visual Semantic Role Labeling},
  author = {Bhattacharyya, Abhidip and Palmer, Martha and Heckman, Christoffer},
  booktitle = {The 12th Joint Conference on Lexical and Computational Semantics, Co-located with ACL},
  month = jul,
  year = {2023},
  address = {Toronto, Canada},
  publisher = {Association for Computational Linguistics},
}

ReCAP: Semantic Role Enhanced Caption Generation

Abhidip Bhattacharyya, Martha Palmer, and Christoffer Heckman

In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), May 2024

Abs Bib PDF

Even though current vision language (V+L) models have achieved success in generating image captions, they often lack specificity and overlook various aspects of the image. Additionally, the attention learned through weak supervision operates opaquely and is difficult to control. To address these limitations, we propose the use of semantic roles as control signals in caption generation. Our hypothesis is that, by incorporating semantic roles as signals, the generated captions can be guided to follow specific predicate argument structures. To validate the effectiveness of our approach, we conducted experiments using data and compared the results with a baseline model VL-BART(CITATION). The experiments showed a significant improvement, with a gain of 45% in Smatch score (Standard NLP evaluation metric for semantic representations), demonstrating the efficacy of our approach. By focusing on specific objects and their associated semantic roles instead of providing a general description, our framework produces captions that exhibit enhanced quality, diversity, and controllability.
@inproceedings{bhattacharyya-etal-2024-recap, title = {{R}e{CAP}: Semantic Role Enhanced Caption Generation}, author = {Bhattacharyya, Abhidip and Palmer, Martha and Heckman, Christoffer}, editor = {Calzolari, Nicoletta and Kan, Min-Yen and Hoste, Veronique and Lenci, Alessandro and Sakti, Sakriani and Xue, Nianwen}, booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)}, month = may, year = {2024}, address = {Torino, Italia}, publisher = {ELRA and ICCL}, url = {https://aclanthology.org/2024.lrec-main.1191}, pages = {13633--13649}, }
One More Modality: Does Abstract Meaning Representation Benefit Visual Question Answering?

Abhidip Bhattacharyya, Emma Markle, and Shira Wein

In Findings of the Association for Computational Linguistics: EMNLP 2025, Nov 2025

Abs PDF

Visual Question Answering (VQA) requires a vision-language model to reason over both visual and textual inputs to answer questions about images. In this work, we investigate whether incorporating explicit semantic information, in the form of Abstract Meaning Representation (AMR) graphs, can enhance model performance—particularly in low-resource settings where training data is limited. We augment two vision-language models, LXMERT and BLIP-2, with sentence- and document-level AMRs and evaluate their performance under both full and reduced training data conditions. Our findings show that in well-resourced settings, models (in particular the smaller LXMERT) are negatively impacted by incorporating AMR without specialized training. However, in low-resource settings, AMR proves beneficial: LXMERT achieves up to a 13.1% relative gain using sentence-level AMRs. These results suggest that while addition of AMR can lower the performance in some settings, in a low-resource setting AMR can serve as a useful semantic prior, especially for lower-capacity models trained on limited data.