FMSB Publishes Algo Risk Model Transparency Draft
Posted by Colin Lambert. Last updated: July 20, 2023
The Financial Markets Standards Board has published a transparency draft State of Good Practice looking at the application of a model risk management framework to algorithms, the body says the aim is to support firms applying model risk management, by addressing sub-issues associated with model use throughout the risk chain.
The paper lays out key factors in determining whether a risk method used in an algo constitutes a model, as well as factors influenced the risk-tiering, key features for model testing, methodology and performance monitoring, as well as the treatment of material changes to models from a validation and documentation perspective.
The study looks at these issues through the prism of existing regulation from the US (Supervisory Guidance on Model Risk Management ‘SR11-7’) and UK (Supervisory Statement ‘SS1/23’) in particular, but observes that other jurisdictions will have their own requirements for model management and oversight.
The study presents a series of Good Practice Statements for feedback, these are that firms should examine their algos to identify any methods that constitute a model, and categorise each of their models associated with algos into risk-based tiers to help identify and manage model risk.
Model testing in Algos should, in particular, the draft states, assess model performance under a variety of market conditions, including volatile conditions and scenarios where there is limited and/or poor-quality market data; and emphasise testing of both the embedded controls and the mitigating controls as opposed to testing the accuracy of a model, given that the adverse consequences of model inaccuracy can be addressed through an effective control framework.
When considering the residual risk and the depth and frequency of the validation of the methodology of a model, the draft says firms should take into account all mitigating controls. Further, validation activities should be tailored to the context in which the models are deployed. They should also be proportionate to the risks they present, FMSB adds, noting, “For algos, model validation may prioritise reliance upon the effectiveness of mitigating controls over model accuracy”.
The draft says the nature and frequency of ongoing performance monitoring for models associated with Algos should (i) be appropriate to the risk-based tier of the model; and (ii) complement any manual trading supervision of the algos and the associated continuous objective feedback of algo or model performance required in a wholesale markets context.
When considering how to respond to any model issues or errors identified during ongoing performance monitoring, FMSB says firms should consider, using observed data, if such issue or error is likely to lead to materially adverse outcomes.
When determining if a change to a model associated with algos requires validation, and, if so, the extent of such validation, FMSB says a firm should consider (i) the materiality of the change in methodology; (ii) the risk-based tier of the model; and (iii) the extent to which the change impacts the inherent risk of the model.
Firms should also consider whether model, or model change, documentation can be supported with model source code access. A further recommendation is that staff validating the models should be “sufficiently knowledgeable” of their use.
The paper, which can be accessed here, also provides examples to support the good practice statements in the draft. FMSB is inviting comments on the draft by 22 September 2023.
