The tool is a simple plugin that lets you select text on a web page producing a 3D avatar that signs in the Indian Sign Language.
For a majority of the hearing impaired, sign language is their first language. Any other language acquired like English, Hindi or Telugu is often a second language. Visual cues like hand gestures, signals, facial expressions, and body movements are used for sign language worldwide. The Indian Sign Language (ISL) uses gestures to represent complex words and sentences and has its own grammar and word order. Hence the hearing impaired find reading online content to be a challenge. To address this gap, Masthan Sharif Shaik and Mohammed Raza, third year BTech students from the Center for Visual Information Technology (CVIT), IIIT-H under the supervision of Profs. CV Jawahar and Vinay Namboodiri (University of Bath, UK) developed a cost-effective plugin that can be added to any website.
The plugin itself is simple enough that it can run inside a browser in real time minus any real hardware or an AI model running on a server. “Websites today support captions, screen readers and high-contrast modes, but almost none of them can show content in sign language. Existing signing avatars are either costly motion-capture systems, heavy AI models that need powerful GPUs, or systems built from sign notation – all of which require linguistic experts and also tend to move robotically,” says Sharif.
What They Did
The researchers first took the official ISLRTC sign-language dictionary videos and used computer vision to recover each signer’s 3D body, hand and finger movements. “Automatic reconstruction however often gets fingers wrong, and in sign language a wrong finger can change the meaning. So, we built an annotation tool, Gizmo, to correct only the few key poses of each sign instead of every frame,” explains Sharif. There is an offline preprocessing that takes place where sentences are converted into ISL word order. For example, a sentence such as “He will come home”, in ISL becomes “He home come”. When a sentence like the former is highlighted and the ‘Click To Sign’ button selected, the stored motion for each sign is fetched, and the signs are blended smoothly allowing the avatar to sign the whole sentence fluidly. Words without a sign are fingerspelled letter by letter.


Advantages
The movements are more natural in this case because the avatar replays motion captured from real signers rather than motion generated from notation. Additionally, what makes this task unique is that it all happens without a GPU. “The heavy computer-vision work happens once, offline, when the sign dictionary is built. Viewing a sentence only replays and blends stored motion together, which an ordinary laptop handles”, says Sharif.
The research has been documented in a paper titled,”SignPlug – ISL plugin for the web” which has been accepted at the 17th Indian Conference on Computer Vision, Graphics, and Image Processing (ICVGIP 2026) to be held in December at, Kolkata.
Next Step

When tested on deaf and hard-of-hearing signers, the ISL signers understood about 89% of sentences signed by the avatar. However, the lack of facial features in the avatar was cited as a drawback because expressions typically include grammar in ISL as well as signs that change form in context. “We will work to incorporate this in our next iteration,” says Sharif who is happy that he got to demo his work in front of Prof. Andrew Zisserman, prominent British computer scientist, a Professor of Computer Vision Engineering at the University of Oxford, and a leading researcher at Google DeepMind at the recently held silver jubilee celebrations of CVIT. “Indian Sign language (ISL) seriously lacks resources and technologies”, explains Prof. Jawahar. The group is working in two directions – Sign Language Synthesis as well as Sign Language Understanding. The plugin is related to sign language synthesis from text on low cost commodity hardware.

Sarita Chebbi is a compulsive early riser. Devourer of all news. Kettlebell enthusiast. Nit-picker of the written word especially when it’s not her own.


Next post