Insight & Analysis | Technology & Business Solutions (TABS)

Proving the Value of Generative AI: THL’s Successful Pilot with Coding Assistants

June 20, 2024

THL’s Implementation of Coding Assistants Across Select Portfolio Companies Led to Productivity Increases of Between 10% and 30% and Enabled Accelerated Product Development

Key Takeaways

Generative AI (GenAI) is a hot topic for its transformative potential and theoretical benefits, but real-world applications are still nascent.
To put theory to practice, THL launched a GenAI pilot with select portfolio companies to determine what value coding assistants might bring to their engineering teams; after four weeks, they achieved a total productivity increase ranging from 10-30%.
THL’s pilot program demonstrates that GenAI can significantly enhance efficiency and position early adopters to gain competitive advantage.

Generative AI (GenAI) is one of the hottest topics in technology and business today, frequently making headlines with bold claims about its transformative potential across industries. Yet despite the widespread media attention, real-world applications of generative AI are still nascent in day-to-day business operations. Much of the media’s coverage focuses on theoretical benefits — increased productivity, more innovation and creativity, better data synthesis and analysis, etc. — rather than concrete implementations and proven outcomes. This gap between promise and practice often leaves businesses at a GenAI standstill: will they implement it or not? And if they do, where and how do they start?

To address these and other questions, in December 2023, THL launched a GenAI pilot program focusing on coding assistants — tools for software developers that give intelligent code suggestions in real-time, automate repetitive coding tasks, identify bugs, and more.

As part of the pilot, engineering teams from a wide range of THL’s portfolio companies across all three of our industry focus areas adopted GitHub Copilot and other coding assistants for four weeks. The tool did not replace any of the companies’ developers — on the contrary, it helped them to do their jobs better and faster. As of May 2024, total productivity improved by between 10% and 30%, depending on the company.

“We spent a lot of time looking at GenAI last year and wanted to put theory to practice and extract meaningful value from it for our portfolio companies,” said Alex Sabel, Research Analyst, THL. “Things are moving so fast with some of these tools, and starting with coding assistants felt like a great way to help them gain efficiency and reduce tech debt without reducing workforce.”

Proving Generative AI’s Value: THL’s Approach, Implementation & Execution

When thinking about ways to put GenAI into practice, coding assistants emerged as a compelling and strategic starting point. For one, they provide immediate and measurable benefits. By tracking metrics such as the number of lines of code produced, the speed of code completion, and the reduction in bugs or errors, organizations can quickly assess the impact of AI on productivity and efficiency. Equally, coding assistants are growing in popularity among developers. GitHub Copilot, the coding assistant that THL selected for its pilot, has a strong reputation.

“It’s like you have a partner next to you helping you code,” said Mark Benaquista, Managing Director, THL and a member of the firm’s operating team, the Strategic Resource Group (SRG). “Only this partner has access to a giant coding library and takes microseconds to pop code into place.”

THL chose Copilot because of its capabilities, simplicity, and familiarity. The portfolio companies that participated — which included software companies with established engineering bases across the technology, healthcare, and financial services industries —primarily use GitHub features as their repository to manage code, so the tool was not foreign to them and did not require extra implementation or integration. Instead, teams had to subscribe to Copilot, enable or install a compatible code editor, and install the Copilot extension. Companies not working in Microsoft coding environments used other coding assistants with similar ease and success.

To help them get started, THL Director Jagjit Singh, also an SRG team member, spent two weeks traveling to participating companies and meeting with their engineering teams in Europe, the Philippines, and India.

“Being there in person made a big difference with onboarding and increasing adoption,” said Singh. “We were able to answer questions in person and get developers up and running in a matter of hours or, at most, days.”

Once teams started using the tool, THL hosted monthly roundtables to allow them to network with each other, ask questions, and share wins and lessons learned. Throughout the onboarding process, some teams required additional engagement to clear up implementation concerns around purpose and privacy.

Overcoming Common Onboarding Challenges

As with any new tool — and especially GenAI — management teams approached the pilot with caution. Common questions arose: Why change a process that’s already working? Are developer jobs at risk? Will proprietary data and code be safe?

The SRG team recognized that they needed to provide additional information about the purpose of a coding assistant and why they were introducing it in the first place. In addition to sending Singh to visit teams in person, THL was also careful in how it introduced the opportunity.

“There was definitely some change management involved, so when we brought the teams together for the first roundtable, we explained that this is not a labor reducer — it is a roadmap accelerant,” said Sabel. “In other words, our partners can complete their roadmaps quicker, and then move onto the next range of needs and opportunities.”

“Think of it like putting a booster engine behind them so they can get off the wheel and back on real work — on builds that will help drive the business forward,” added Benaquista.

Regarding privacy, coding assistants are especially well suited for scripts that involve boilerplate code. No internal data needs to be shared, and teams can use these types of tools however they think will be most helpful.

“Using Copilot has enabled our team to work on more strategic projects because we’re not spending all their time on routine code or bug fixes,” said Mark Chua Chui Hian, MD, inriver. “Our productivity has increased, and we are in the process of getting more of our engineers onboarded.”

Results: Coding Assistant Increases Productivity 10-30% Without Impacting Workforce

One dozen portfolio companies across THL’s investment verticals participated in the firm’s coding assistant pilot.

Over the course of four weeks, teams cataloged each individual task that required code —describing it in detail, approximating how long it takes to complete manually, and logging how long it takes to complete with coding assistants. Participants in the pilot program used the change in time between manual coding and assisted coding to calculate a percentage change in productivity for each individual task on its own and relative to total workload. THL tracked feedback and outcomes across three categories: code testing, code creation, and education. Results demonstrated a productivity increase of between 10% and 83% depending on the task, and an overall increase in productivity of between 10% and 30%. The wide range in productivity outcomes is a function of the specific underlying tasks and their complexity levels. While some tasks proved highly automatable, they did not always represent a large share of developers’ workload and time commitment.

“Tools like Copilot have boosted team engagement and empowered our resources to accelerate our roadmap delivery by automating routine tasks, improving documentation, and streamlining our ability to grasp complex concepts,” said Nathan Maehren, Chief Digital & Technology Officer, Careforth.

“Having a coding assistant is like using a power tool to help with tedious manual work,” added Victor Moreira, CTO, Carpe Data. “Our developers have really embraced the tool, recognizing it as a powerful partner that enhances their efficiency and lets them focus on more complex and innovative projects.”

Adoption rates vary by organization. As of May 2024, there was an overall adoption rate of 44% across 11 portfolio companies, representing 507 users out of a combined eligible user base of 1,147.

The path to full adoption has been faster for smaller organizations and slower for larger organizations. Engineering teams are typically broken up into smaller scrum teams, and rolling out a new tool in all those different business units takes time.

As companies see the benefits of using a coding assistant, we expect adoption to increase.

“We had 20 of our engineers on GitHub Copilot for a three-week experiment with THL and found 11% productivity improvement across developer types,” said Colin Bodell, CTO, BazaarVoice. “Equally important, we received excellent feedback, including a 95 NPS, which helped secure organizational buy-in and increased our adoption to about 60 developers.”

Future Implications & Recommendations

THL’s coding assistant pilot answers several lingering questions about generative AI and opens the door for additional efficiency gains across the firm’s portfolio companies. Such tools are designed to support existing teams so they can accelerate production in a more efficient way and focus on high-value tasks designed to drive business growth. We believe that companies that adopt these AI-driven tools can gain a competitive edge. THL continues to encourage its partners to think creatively about ways to position their teams to capture the full value of future iterations of this technology.

Because of the success of THL’s pilot, the firm plans to roll out coding assistants to additional companies in its portfolio..

“The generative AI implications are far and wide,” said Benaquista. “Coding is one small aspect of it, but it allows our CEOs to understand there’s a lot of impacts across many elements of an organization.”

To learn more about THL’s work with coding assistants and generative AI, contact our team today:

Mark Benaquista, Managing Director, SRG
mbenaquista@thl.com

Jagjit Singh, Director, SRG
jsingh@thl.com

Alex Sabel, Research Analyst
asabel@thl.com

_{^{1 The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. Sida Peng, Eirini Kallaimvakou, Peter Cihon, Mert Demirer. Microsoft Research, GitHub Inc., MIT Sloan School of Management. February 13, 2023.}}

_{Certain executives of THL portfolio companies who make statements about THL may also be limited partners in the THL funds. No such executive was compensated for providing these statements. As a result of the companies’ ownership structure and the executives’ investments in the THL Funds, a conflict of interest exists because the executives have an incentive to make positive statements about THL. Any such statements are subject to change without notice and may be contrary to opinions expressed by others. The statements reflect the subjective views of the speaker and THL has not independently verified such information and makes no representation or warranty, express or implied, as to the accuracy or completeness of the information contained herein.}

_{THL applied a consistent method across all 12 companies participating in the pilot program. This method included identifying tasks falling under three categories (code testing, code creation, and education); tracking the time to completion using Copilot versus a manual process; and weighting these outcomes by the share of workload that these tasks represented for each company’s engineering organization.}

_{Neither THL Partners nor its portfolio companies received financial incentive or benefit from GitHub or its parent company, Microsoft, for using and referencing Copilot in this pilot program.}