ISDA Study Highlights AI Benefits to CSA Digitisation

While uncertain remains over the benefits, or otherwise, of AI in trading (as opposed from machine learning), few will argue with the technology’s potential in the workflow, a view that has been reinforced by a study from ISDA looking at how AI can be used to accurately and reliably extract, interpret and digitise key legal clauses in its Credit Support Annexes (CSA) documentation.

The study evaluated the performance of eight large language models (LLMs) on their ability to accurately extract and interpret five clauses from a selection of CSAs and digitise them into Common Domain Model (CDM) representations. The CDM is a machine-readable data model that describes financial products, trades and lifecycle events in a standard way, helping to facilitate straight-through processing.

Based on a benchmarking exercise, several LLMs achieved accuracy levels of over 90% when prompted with CSA-specific information, with the simpler clauses seeing accuracy levels of 100% in some cases, ISDA says.

The study also found that providing LLMs with CSA-specific information, such as the ISDA Documentation Taxonomy and ISDA Clause Library, using prompt engineering techniques consistently enhances performance, especially for clauses that exhibit greater linguistic complexity, such as minimum transfer amounts and threshold clauses.

Clauses in CSAs that typically use standardised phrasing (for example, base and eligible currency) are easier for LLMs to extract accurately, irrespective of whether the LLMs were prompted with CSA-specific information. Equally, another finding of the study was that larger proprietary LLMs typically exhibit better baseline performances. Smaller open-source LLMs also benefit from CSA-specific information, however, offering a viable alternative to financial institutions with stringent data privacy requirements that necessitate on-premises deployment.

Another finding was that 100% accuracy is rarely achieved, especially for more nuanced clauses, due to inherent variations in legal language, subtle distinctions between similar clauses and complex cross-referencing within documents. Further refinements in prompting and additional CSA-specific information may be needed to address these challenges, ISDA states.

“Our benchmarking study shows how generative AI can be used to accurately extract relevant contract information and digitize it into a standardised CDM format, reducing resource requirements and the potential for errors versus traditional contract data extraction,” says Scott O’Malia, chief executive of ISDA, who promises that ISDA will conduct additional research to determine what steps are needed to further improve accuracy levels.