Research work on Unsupervised approach to evaluate sentence-level fluency: Do we really need reference? By Dr. Manish Shrivastava and his students Gopichand Kanumolu, MS CSE; Lokesh Madasu, MS CSE; Pavan Baswani, MS CSE and Ananya Mukherjee, Ph.D CSE was given the Best paper award in SEALP2023 Workshop at IJCNLP-AACL 2023 held on Bali, Indonesia on 1 November.

Here is the summary of the research work as explained by the authors:

Fluency is a crucial goal of all Natural Language Generation (NLG) systems. Widely used automatic evaluation metrics fall short in capturing the fluency of machine-generated text. Assessing the fluency of NLG systems poses a challenge since these models are not limited to simply reusing words from the input but may also generate abstractions. Existing reference-based fluency evaluations, such as word overlap measures, often exhibit weak correlations with human judgments. This paper adapts an existing unsupervised technique for measuring text fluency without the need for any reference. Our approach leverages various word embeddings and trains language models using Recurrent Neural Network (RNN) architectures. We also experiment with other available multilingual Language Models (LMs). To assess the performance of the models, we conduct a comparative analysis across \textit{10 Indic languages}, correlating the obtained fluency scores with human judgments.

November 2023