Generative AI tools for Coding – Why getting all ducks in a row is critical

A roundtable hosted by IIIT Hyderabad on Generative AI tools for the initial phase of the software development life cycle, saw the meeting of industry leaders, technologists, innovators and research minds. What emerged was a white paper, that catalyzed insights into actionable strategies, for organizations to harness the full potential of Gen AI.

Generative AI (Gen AI) tools continue to charm, excite and induce cold fear. Post pandemic, when ChatGPT lowered the boom on an unsuspecting world, it was a veritable IT Gold Rush, with a global scramble for a piece of the action. In the digital world, Gen AI tools have become transformative assets in software development lifecycle, enabling accelerated coding, improved collaboration among developers, and innovative problem solving. But what lies beneath?

IIIT Hyderabad recently held a round-table discussion, where conversation centered specifically on the development phase of the Software Development Life Cycle (SDLC), whose deterministic nature makes it a low-hanging fruit for leveraging Generative AI effectively.

Moderated by IIITH Prof. Raghu Reddy, the brainstorming session represented a balanced mix of researchers with the industry-facing perspectives of tech product and SaaS companies(Ozonetel), Banks, (JPMC, Lloyds, DBS), THub, startups (MontyCloud), service companies (Bosch), healthcare (AIG, Evernorth) and open source (Swecha, TechVedika).

The White Paper that evolved out of the confabulation presents a holistic overview of the current scenario, future potential of Generative AI tools in software development, and offers actionable insights for practitioners and researchers.

Current use of Generative AI tools in development phase of the SDLC
Panelist concurred that LLMs and other Gen AI tools provide relatively higher efficiency compared to traditional search engines and question-and-answer (Q&A) platforms. Development practices have changed, with developers now using Gen AI tools to complete tasks in 1-2 searches against the 5-6 searches on Stack Overflow.

Due to its generic abilities, panelists agreed that ’simple tasks work well’ on most Gen AI tools, with one panelist citing an example of React forms. Using Gen AI tools in a greenfield project to build systems from scratch is much easier than using them on existing systems, noted some industry experts.

Challenges and workarounds
While Gen AI tools offer transformative potential in software development, panelist shared challenges they faced in contextual limitations, privacy concerns, integration complexities and organizational resistance.

• Alternatives to search engines:  While Gen AI tools seamlessly replace search engines and Q&A platforms, the final code has to be copied and modified to be integrated into the system.
• Challenges with existing products and legacy code: The difficulty of Gen AI tools increases with existing products. For example, for language migrations, the tools still pose a challenge for converting bitwise operations to Java code.
• Gaps in context and domain knowledge: Organizations who have explored multiple tools like Cursor, Copilot, Claude observed that AI tools sometimes lack the contextual awareness required for intricate coding tasks. Business specific responses are currently not a strong suit of Gen AI tools. For instance, in the domain of business logic, responses are found incorrect 50%-60% of the time. Grave concerns revolved around data privacy, intellectual property, and compliance with regulatory standards when it came to sharing context, metadata, and the proprietary nature of training datasets and the potential exposure of sensitive information.
• Cost and Training challenges: Integration into existing systems often demands substantial time and resources, as well as a commitment to restructuring workflows to accommodate AI-driven methodologies. The cost of manpower training to use un-ambiguous English in the prompts was also raised.
• Adoption challenges in Organizations: While Investment typically happens at top management levels, usage efficiency in the workforce is around 20%, due to concerns about job displacement and disruptions to familiar workflows, “developer ego” or the notion that they “can do better than the LLMs”.  

Panelists concurred that Gen AI tools “cannot replace” but act to “assist or accelerate” the process; sometimes producing code that is functional but not optimized, leading to potential rework and long-term technical debt(TD). Panelists also discussed TD management for long term sustainability but the discussion on the need to “irradiate TD” remained inconclusive. Tracking changes across large codebases using a Gen AI tool becomes challenging, when modifications span multiple development stages. Organizations were urged to pick up the baton to achieve the most “green” solutions and not wait for it to be enforced as a mandate by an external influencing factor like governments.

Creating solutions and drawing blueprints
With productivity as central focus, Gen AI tools generate detailed, context-aware documentation, making it easier to maintain codebases, enabling collaborative development across diverse stakeholders and geographically dispersed teams. This helps with code explainability, document generation, unit test generation from code and with grunt work assistance.

While some founders dumped it on their engineers to “make things work”, others stated that over time, the diminishing ability of developers was a concern. However, some argued that particular skills would inevitably depreciate over the years, like in the case of Java.

To address concerns of privacy, protection and regulation of intellectual property, panelists pointed out the need for robust governance frameworks and transparency in AI tool usage. While tracking change summaries remains a challenge, ownership and accountability of the code generated and tested, lies with the code reviewers, usually a senior developer, assisted by specific models to ensure that reliable code gets accepted into the system. “Trust but Verify” was a strong undertone of the discussion.

The significant usage of Gen AI tools in initial development stages underlined the importance of inventing roles like a “product manager”, to prevent reworking that was viewed as software development waste.

Best practices for integrating tools into organizational development teams
Organizations need clear strategies and best practices to successfully integrate, implement and make the most of Generative AI tools.

• Consider Gen AI tools as junior developer: The coding agent Gen AI tool is to be treated as a junior developer that probably costs $30 per month which is of great value for startups. With well-defined and limited context, with system instructions, it can be promoted and utilized as a senior developer. For instance, a banking organization achieved 80-85% accuracy for converting text to SQL, with a senior in the loop for validation and reviews. An engineering manufacturing company is conducting experiments in a similar controlled environment for domain specific use cases.

The code generated by AI tools needs to follow organizational coding standards and quality, to give the most deterministic code for the situation. For best results, using an ensemble of LLMs, an agentic system can reduce errors and code smells in the results.

• Small-scale implementations for trust building: To analyze successful adoption, initiating small-scale implementations allows teams to assess the impact and usability of AI tools in a controlled environment. Swecha, an open-source community tested the use of LLMs, using control and experimental teams and found that the team using the LLM, accomplished the task in half the time. For cohesive adoption across diverse teams and to reduce misunderstandings, use clear and precise language when defining AI tool capabilities, expectations, and implementation plans.  

• Developer Training and Skill Building: Comprehensive training programs for Developers with Hands-on workshops, online resources, continuous learning opportunities along with team retrospectives on the hits and misses of AI adoption, promotes a culture of transparency. Cross-functional collaborations enable teams to explore innovative applications and drive continuous improvement. Along with a broader understanding of the big picture, skillsets for prospective employees include proficiency in unambiguous English, system thinking and prior knowledge of Gen AI tools for coding.

On the organizational side, establishing a framework of clear policies on data security, intellectual property and ethical considerations provide a structured approach to risk management and alignment with organizational goals and regulatory requirements, that is crucial for responsible AI usage. Continuous evaluation of performance, Iterative feedback loops and data-driven decision-making enable organizations to refine strategies and optimize outcomes over time.

While several Gen AI-based developer productivity tools, including code generation tools, have entered the mainstream over the past year, they only assist with parts of the overall process—in the design phase, data modeling, utility services code, initial code for major functions, or even troubleshooting and debugging. Though they vastly improve efficiency, the general consensus is that these tools will not necessarily lead to a reduction in the size of developer teams.

And in conclusion
Generative AI tools are revolutionizing software development, offering significant benefits in efficiency, quality, and collaboration. Despite challenges like technical debt and integration complexities, the transformative potential outweighs the drawbacks. Organizations that approach adoption strategically and iteratively can unlock new possibilities in software development. A standardized framework for integrating Generative AI tools into an organization is the need of the hour.