IIITH students take home the Qualcomm Innovation Fellowship award two years in a row. Last year’s winning team also gets awarded ‘winner of winners’ for the year 2020-2021. 

A team comprising of students Pranay Gupta and Debtanu Gupta under the mentorship of Dr. Ravi Kiran Sarvadevabhatla has won this year’s edition of the prestigious Qualcomm Innovation Fellowship award for their innovative proposal titled “Deep Neural Models For Generalized Synthesis of Multi-avatar Actions”. It is the second consecutive fellowship award for the institute coming on the heels of last year’s win by a faculty team of Prof. Madhava Krishna and Dr. Vineet Gandhi with students Unni Krishnan Nair and Nivedita Rufus for their work on “Language-based Navigation with Object Referenced Spatiotemporal Constraints in Autonomous Driving”.

Industry Collab With Student Research
The Qualcomm Innovation Fellowship (QIF) program was created by the American multinational to drive research and development through a collaborative effort with university students and “their forward thinking ideas”. It focuses not only on recognizing and rewarding innovative ideas proposed by graduate students but also mentoring them with the help of Qualcomm’s engineers across a broad range of technical research areas that are aligned with the company’s own core values. While this initiative kicked off in 2009 at Qualcomm’s Research Centre in Silicon Valley, California and soon expanded to Europe, it wasn’t until 2016 that it was launched for select Indian universities such as the IITs, IISc, Indian Institute of Statistical Science, IIITH and IIIT Delhi. It is a highly competitive fellowship and undergoes a 2-stage selection process. In the first stage, applications are invited from the list of preselected universities. A shortlisted set of teams is then asked to make presentations based on which a handful of winners is announced. This year 13 teams have been declared winners with each of them receiving a financial award of 10 lakhs in addition to the assignment of a Qualcomm researcher as mentor for a period of a year to support the students in their research goals. In India, after the completion of the Fellowship, a super-winner team from the QIF winners for that year is selected who will then be supported by Qualcomm to further the research as stated in the winning proposal.

The (Super) Winner Takes It All
The QIF super winners’ research which marries computer vision and natural language processing deals with level 5 autonomous vehicles. These are cars requiring zero human intervention. “When there is no interface where one can control the vehicle, a nice way to control it is via language,” says Unni Krishnan. He cites an example of stopping en route home from work to run errands, such as halting at a take-away joint to pick up a bite. “You can give commands to the system, like, ‘Do you see a restaurant? Stop near it wherever there is shade. Or, ‘Wait for the children to cross the road and then proceed’.” Essentially the vehicle reacts in response to language-based instructions which have conditions related to space and time.

When the team first began working on this proposal, they tried to identify an object of interest. “If our instruction to the system was to ‘Turn into the right lane in front of the blue car’, our task was to get the system to identify the blue car first,” says Nivedita. With this work, they participated in a challenge that was hosted by the European Conference for Computer Vision (ECCV) 2020 where they stood third globally. “We published the method and open sourced the code for this challenge,” she says. While the research community’s acceptance of the work was gratifying, Nivedita remarks that it didn’t make sense navigationally. “At the end of the day, we needed to move the car. Here, the navigational point of interest lay in front of the blue car and wasn’t the blue car itself.” This spurred them on to better their previous work and resulted in a paper titled ‘Referring Navigable Regions’. Here, they attempted to predict the region of interest such as the space between two cars for parking, or any where in the left lane and so on. For this, voice commands such as ‘Switch to the middle lane” or “Park in front of the red car” are recognised by the system. The work has been accepted at the International Conference On Intelligent Robots and Systems (IROS) 2021 which will be held virtually between September 27 and October 1, 2021.

Having won an additional grant of 10 lakhs, the super winners are finetuning their navigational prediction model. “We now have a black box of a neural network where we feed instructions and an image. It then gives us a set of velocity values, steering angles and pedal control values a car should maintain in order to attain the given goal,” explains Nivedita.

Keeping It Simple
Quiz them on what made their work stand out, and the team answers that it’s because they started from the basics. To make their proposal attractive to those who may have had little to no exposure in the field of autonomous driving, they eschewed jargon and instead focused on explaining key concepts lucidly. “Both Unni and I have been working on self-driving cars but for this proposal, we started from the ground up because of which we were able to present our ideas clearly compared to the other proposals,” says Nivedita. They also acknowledge the efforts of their professors in steering them in the right direction. “Madhav sir and Vineet sir have a great deal of expertise when it comes to computer vision in general and autonomous vehicles in particular,” remarks Unni.

2021 Champs: What They Proposed
This year’s QIF edition saw a large number of innovative proposals from the fields of machine learning, computer vision, telecommunications and more. IIITH’s AI-based proposal straddles deep learning and multimedia. “In layman’s terms, our work deals with the automatic generation of videos of typical everyday activities performed by humans, with just the name of the activity as an input,” says Pranay Gupta, adding that the generated videos are however not the typical RGB or colour videos that one encounters. “They are instead geometrical models of humans.” What makes this effort complex is the fact that there is no upper limit to the number of physical activities that one can perform. Plus, there is no single way a particular physical task can be discharged. This explains why current video-generation methods are limited to automatically generating animations for only 10 or 12 activities such as walking, running and so on. “Our primary milestone is to generate animations for around 120 activities,” says Debtanu. “None of the existing methods auto-generate animations involving two or more persons. So we’ll be the first to do so,” he remarks. There are other firsts that the team is aiming for with these research efforts. For instance, the generation of activities based on variable durations, taking into account the change in relative displacement for human locomotion and so on.

Applications
An immediate application of the research proposal is in the animation industry and in the AR/VR space. “While the main aim is to automate human motion to make it look as natural as possible for animations, there could be potential applications in situations where it is necessary to predict the future move of a person, such as in the case of autonomous driving cars where the system needs to learn and predict moves made by pedestrians,” explains Debtanu.

Faculty Speak
For Dr. Ravi Kiran, the fellowship award has come full circle. “I was a recipient of the award as a PhD student of IISc in its inaugural Indian edition in 2016. To have my student team win it too is really special. I’m happy that Qualcomm continues to be part of my research journey,” he says. He draws attention to the financial grant that accompanies the award terming it unique in that it comes with no strings attached. The grant can be used at the faculty’s discretion to fund any research expense including conference travel expenses that are typically incurred by researchers. The QIF award gives students ample exposure to industry and helps in networking and developing contacts. In addition to this, with the current emphasis by the Centre on liasioning with academia and industry alike, presentations have been made by the winning teams to the Secretary, Ministry of Electronics and Information Technology (MeitY) and other government officials. As one of the mentors of the super winning team, Prof. Madhava Krishna says that they did not think about it (winning) at all when the team was presenting their ideas that shaped up over the course of a year. “We were very pleasantly surprised at being declared as superwinner.”