IIITH researchers create a web-based interface that not only helps you play the classic game of Pictionary but also comes with an AI-based automated moderator. A closer look at the novel solution reveals that it’s more than just about drawing and guessing. 

It was during the pandemic that the classic game of Pictionary found new found interest when it went online. Used as a platform to connect with coworkers as well as physically distanced families, the virtual avatar of the game is both heaps of fun and useful – as a bonding tool – all at once. The remote version however is not without its limitations. By virtue of being physically removed from other players, it affords a sense of anonymity leading to rule violations. A research team led by Prof. Ravi Kiran Sarvadevabhatla comprising of Nikhil Bansal and Kiruthika Kannan from the Centre for Visual Information Technology (CVIT), has developed an AI model that can sound an alert by automatically detecting content contravening the rules of the game.

 

Sketch The Word
Though the game mandates the use of only pictorial content to elicit guesses from the Guessing team, it’s common to find players indulging in some amounts of written text or numbers as a means of providing hints towards the target word. While it’s possible to intervene and check such game violations when the game is being played physically, manual intervention is not feasible in the virtual scenario. “Besides written text, we also find arrows and icons such as a circle or even underlining often used to draw attention perhaps to a particular part of the sketch, which is atypical (drawing) content,” elaborates Prof. Ravi Kiran.

 

 

In order to flag such transgressions in real time, the team initially created the first-of-its-kind atypical sketch content dataset. “We began by creating an online Pictionary portal where people were paired up to play as either Drawers or Guessers. Data from these gaming sessions was collected and the atypical content in it was annotated. This dataset in turn was used to train our atypical sketch content detection network,” explains the professor. The AI system essentially serves as a mechanism that instantaneously alerts the players in case of infractions.

More Than Textual Breaches
With the premise of handwritten text comprising a bulk of atypical sketch content, the team began their research by applying handwriting text recognition to their annotated dataset. However, they quickly realised that sketch content was more complex than mere handwritten material. “The way people write, say, with the help of a stylus where the intent is to write as opposed to draw, and the way people write on a drawing canvas, or a whiteboard is very different. Besides in our case, writing is interspersed with the drawing itself. It was important for us to first isolate the writing from the sketch and then detect it,” explains Prof. Ravi Kiran. With the help of an ML model (CanvasNet), the team created a system known as Drawmon (short for.Drawing Board Monitor) that detects and isolates the icons and symbols in addition to the drawing itself, sounding an alert about the former’s presence to the drawers. The research resulted in the publication of a paper titled “Drawmon: A Distributed System for Detection of Atypical Sketch Content In Concurrent Pictionary Games” (https://drawm0n.github.io/) that was presented at the #1 venue for cutting-edge multimedia advances – The ACM (Association for Computing Machinery) Multimedia Conference in Lisbon, Portugal held between 10-14 October 2022.

ACM-W Scholarship
Kiruthika Kannan who is an integral part of the project was awarded the ACM-W scholarship that enabled her to travel to the conference to present the paper. The scholarship expressly provides support to women undergraduates and graduate students in computer science and related programs to attend research computer science conferences. The intent of the award is to inspire a student to continue on to their next level of academic or industry career through such exposure. While applying for the 1200 USD scholarship, Kiruthika who is a Masters student explained about the project at length and expressed her eagerness in connecting with other researchers and professors who were likely to attend the conference. Incidentally, presenting a paper at the conference is not mandatory for a scholarship recipient.

As Good As A Product
Typically, research projects involve proof of concepts and development of prototypes. In this case, there was a lot of engineering and iterative design that went into the background making it more than just a regular prototype. In fact, according to Prof. Ravi Kiran, the solution is as close to a product as it can get. Dual Degree student Nikhil Bansal who was largely instrumental in building the browser-based system recalls the challenges they faced beginning with the complexity of creating a multi-player system. “Aside from creating the GUI and interface, we wanted to have 200-300 players simultaneously accessing the portal where they could be paired against each other. We had to make sure these sessions could run in parallel without interacting with each other. The feedback system we developed in turn had to be able to handle all those hundreds of sessions and send feedback to the appropriate one(s),” he says.

The Bigger Picture
Giving context to their patent-pending solution, Prof. Ravi Kiran says, “Games have had a long history of being used as a proxy to measure progress in AI. Take for instance, AlphaGo developed by Google DeepMind. The work done towards developing that agent went on to be tremendously useful for decoding DNA patterns. In our case, we achieved 3 things – created a unique dataset of atypical drawing content, a system that flags non-conforming content, and third, there’s on-going work on developing automated agents that can both guess and draw. Right now we have baseline agents but we’re headed towards creating more sophisticated versions of those where it will be difficult to tell them apart from humans.” While the solution was developed for a game setting, the team opines that it could be useful for any scenario involving interactive whiteboards which seem to be ubiquitous in today’s workplaces and classrooms.

KCIS-funded Project
The project which was chosen as part of the Kohli Challenge floated by the Kohli Centre on Intelligent Systems (KCIS) was completely funded by the centre. “Unlike other projects, the USP in this case is that the funding did not have constraints of any sort. We were encouraged to go ahead based on the exciting research. It was even presented to the TCS Foundation who greatly appreciated the idea,” sums up Prof. Ravi Kiran.