Vanshpreet Singh Kohli supervised by Dr. Radhika Mamidi received his Master of Science – Dual Degree in Computational Linguistics (CLD) by Research. Here’s a summary of his research work on Choose Your Words Wisely: Domain-adaptive Masking Makes Language Models Learn Faster

This thesis presents a novel approach to domain-specific language model training that significantly reduces computational costs while maintaining or improving performance on downstream tasks. Our work aims to addresses the heavy computational costs in current domain adaptation practices; because of vast corpora available for training models on fields like law and medicine, some popular models like BioBERT often incur higher computational costs for the domain-specific portion of their training than the entire cost of training the foundational models they are initialized from. The key contribution of this work is the development of a domain-adaptive masking strategy that leverages curriculum learning principles to optimize the continual pre-training process. We hypothesize that much of the training is redundant, wherein the model effectively spends time relearning lexical and semantic associations already understood by the foundational language model. To test this hypothesis, rather than employing the traditional random masking approach used in BERT-based models, we create a concept-based curriculum that selectively masks tokens corresponding to domain-relevant concepts while filtering out general-domain knowledge already well-represented in foundational models. We test our approach in the biomedical domain by further pre-training BERT on a corpus of PubMed Abstracts, mirroring the architecture and corpus selection of BioBERT. We evaluate our model against two BioBERT checkpoints across eight Named Entity Recognition tasks and find that it maintains competitive performance at a fraction of the computational cost. Specifically, the model outperforms BioBERT-v1.0 on three-fourths of the evaluated tasks at only two-thirds of the computational cost, and outperforms BioBERT-v1.1 on half of the tasks at just 9% of its compute cost, while also being trained on a smaller subset of the latter’s training corpus. While we demonstrate the effectiveness of our strategy in the biomedical domain, our methodology of intelligent sample selection itself is domain-agnostic, and we posit that it may be adapted to the broader field of transfer learning with minimal modifications. Our findings have significant implications for resource-constrained environments and suggest a promising direction for scaling domain-adaptation across various specialized fields. 

July 2026