Scenario
As one of the world’s leading global law firms, Ashurst has been driving change and innovation throughout their 200-year history. Their ambition has always been to lead the field by championing new ideas and staying ahead of change. For this reason, Ashurst embarked on exploring the advantages AI technologies could have for their business and engaged Arinco’s OpenAI Done Right™ offering to help them build and evaluate a proof-of-concept (PoC) AI application using services provided by Microsoft on the Azure platform.
This strategic project aimed to address the following initiatives:
- Assess the technical feasibility of building custom AI solutions in-house
- Discover, understand and document use cases associated with the client bid/tender process and any potential quick wins using current AI technologies
- Consolidate and map findings against Ashurst’s AI assessment framework to help inform decision-making around build vs. buy and legal market AI solutions vs. more generic product offerings
- Upskill the Office of the Chief Digital Officer (CDO) as a centre of excellence in AI solutions and use cases
Solution
Leveraging the latest in generative AI services, Arinco developed a proof-of-concept knowledge base and chatbot application, to demonstrate the capabilities of Azure services such as OpenAI and Cognitive Search.
Documents used for creating the pilot’s knowledge base were sourced from bids and RFPs in the public domain, to support rapid delivery and ensure that strict confidentiality and governance requirements for sensitive business data were not compromised.
Bid-related documents were systematically organised into collections, allowing chat interactions to be confined within specific bid collections. This was crucial to ensure the accuracy of chat interactions by aligning them with the relevant information. Additionally, this approach established a coherent framework for potential future enhancements, such as user-defined document collections. Defining the right controls to source facts from private knowledge, rather than public sources, reduced chances of hallucinations and improved human trust in chat responses.
Throughout the 5-week engagement, Arinco partnered with the Ashurst teams to design, refine and develop the PoC application. The process involved a range of activities including:
- Deploying Azure infrastructure for private, secure enterprise-grade consumption of AI services such as ChatGPT 3.5, OpenAI text embeddings, and Document Intelligence for text and layout extraction from multiple document types
- Building a searchable, collection-oriented knowledge base utilising the latest semantic and vector-based features of Azure Cognitive Search
- Building a modern Single-Page Application as the front-end chat UI using Blazor WebAssembly, with a secure backend API built using ASP.NET Core
- Providing CI/CD pipelines for continuous delivery of infrastructure and application components to the Azure environment
- Defining an authentication and authorization framework integrated with Ashurst’s Active Directory tenant to ensure private end-user access to a limited audience
- Prompting the chat to generate citation references to individual pages in source documents to enable fact-checking of chat responses, so that business users could build trust
- Prompting the chat to generate follow-up questions to facilitate faster exploratory chat interactions
- Engaging with the wider business in discovery workshops to identify and model key use cases
- Refining outcomes from use case testing to improve results and adopt new strategies where needed, such as document summary grounding and proximity/vector-based near-neighbour grounding
- Performing detailed handover of the pilot application and related services to internal Ashurst technical staff to allow them to maintain and extend the solution after the initial engagement
Outcome
Chris Boulter, Associate Director - Digital Products
Arinco maintained a tight feedback loop with Ashurst, and the solution was able to be refined quickly based on testing performed by Ashurst staff. For example, generating text embeddings on searchable content and adding vector search capabilities to the PoC application was able to be delivered in a couple of days and provided measurable improvement to chat results.
Ashurst was able to use the PoC application to demonstrate the importance of prompt engineering and how a combination of business user training and adoption of specific technological strategies would be important to extracting value and reducing inconsistency from chat responses. The PoC application displayed strengths in Question-and-Answer analysis on narrow fact ranges and the ability to generate summarised content related to those facts. Further research and development are planned to look at how this can be improved on.
The PoC application was built with the intention to support future enhancements, such as upgrading to the latest versions of OpenAI services such as GPT-4, to increase business impact and minimise maintenance costs.
Ashurst are progressing to build on the insights discovered from this initial engagement and are looking to expand to address potential use cases involving data from internal sources, such as bid responses and firm experience history, to build more representative AI applications that can optimise the full bid management lifecycle.
Other types of AI applications such as automated fact extraction and analysis reporting against predetermined questions on incoming bids are also being considered by Ashurst for future development, as the PoC has demonstrated this strategy can be executed successfully and efficiently.
