Navigating Risk

If you work for a mid- to large-sized insurance company, you’ve probably noticed recent activity in your organization in regard to building, maintaining or improving a model governance program. You may have wondered what triggered this. Recently, model governance has captured the attention of supervisory and regulatory authorities, as well as actuarial organizations. This includes the Society of Actuaries (SOA), American Academy of Actuaries (the Academy), Canadian Institute of Actuaries (CIA) and International Association of Actuaries (IAA). To understand the evolution and growth of this field, we must look to how the use of models has changed over the past few decades.

The IAA released a practice note in November 2010 that lists several reasons for increased use of internal models for insurer risk assessment and capital management. One reason is the development of increasingly sophisticated risk-based insurance regulatory capital requirements. Another reason is the availability of inexpensive and fast computers. Additionally, the availability of data has increased. Add to that increased computing power, sophisticated modeling techniques and complex modeling software, and the result is the prevalent use of models that are not easily validated. Nor are the results always easy to analyze or explain.

Additionally, the financial crisis of 2008 affected the way the insurance industry viewed modeling capability and output. A Chief Risk Officer (CRO) Council paper released in 2012 states, “During and after the financial crisis of 2008, models were perceived to be ineffective in producing sufficiently severe outcomes, which has put increased scrutiny on model risk management.”¹ This increased scrutiny has been applied to the banking industry especially. However, it also has influenced the thinking of actuaries and actuarial organizations in their development of model risk management programs.

As we examine the evolution of model validation and governance guidance over the past few decades, there are three areas to note.

• Guidance on model validation grew to include model governance, of which validation is only one component.
• Regulation and supervisory guidance that applies to the banking industry has been considered and incorporated, in part, into recent actuarial guidance.
• We can identify consistent themes of a strong model governance program from this guidance.

Pre-financial Crisis

Much of the model governance activity in the banking and insurance industry occurred in the years following the financial crisis. However, there was some guidance in the banking industry in the years preceding. For example, the Office of the Comptroller of Currency (OCC) released OCC Bulletin 2000-16 in 2000. The topic of this bulletin was model validation. It provided “guidance to help financial institutions mitigate potential risks arising from reliance on computer-based financial models that are improperly validated or tested.”² The bulletin outlined elements of a sound model validation policy. These elements include independent review, defined responsibilities, model documentation, ongoing validation and audit oversight.

A good example of the evolution of model governance guidance comes from the Basel Accords. The Basel Committee on Banking Supervision issued recommendations on banking laws and regulations. The first report was issued in July 1988 and contained no reference to model validation, governance or documentation. The second of the Basel Accords, Basel II, was issued in June 2004. This report refers to validation and documentation dozens of times. In addition, the report includes a reference to “independent review of all elements of the internal modeling process.”³

On the actuarial modeling side, we can look to the description of past practices in a 2016 Milliman white paper. This paper states: “In the past, it was the norm to have each team—pricing, valuation, asset-liability modeling (ALM), etc.—create and manage its own functional models, even when there was overlap among the business needs being served or an absolute need for consistency across assumptions, calculations and results. From time to time, individuals might communicate with other teams to try to reconcile the models, but in the end the organization still had separate models with separate ownership, often spread across a variety of different platforms. The result was typically a costly, confusing and opaque environment.”⁴

Financial Crisis

The financial crisis of 2008 spurred many model validation and governance initiatives. In January 2009, the Federal Reserve Board issued Federal Reserve Bank Supervisory Letter SR Letter 09-01. This letter included a section called “Required Annual Independent Review of Market-Risk Measurement and Management Systems.” This section states, “At a minimum, the annual review should incorporate the following,” and includes several items such as requirements for documentation, independence and validation of any significant changes.⁵ Additionally, SR Letter 09-01 includes guidance on the verification of the model’s accuracy through backtesting.

There is early evidence of the concept of model governance becoming broader than validation. In 2009, the Federal Housing Finance Agency (FHFA), which regulates and supervises Fannie Mae, Freddie Mac and the Federal Home Loan (FHL) banks, released Advisory Bulletin 2009-AB-03, titled “Validation and Documentation of Models and Related Controls on Internal Processes.” It replaced a previous advisory bulletin by the same name issued in 2006, 2006-AB-02. The 2009 bulletin recommended that each FHL bank have policies and procedures to ensure all of its models are documented and validated. The bulletin discussed formal policies, documentation and the elements of a sound validation program.

Then there was the joint guidance from the OCC and the Federal Reserve Board. This guidance centered around model risk management and was released in April 2011. The guidance is listed under OCC 2011-12 as well as SR 11-7. It defined a model and described the need for a model inventory. It also emphasized the need for documenting—including methodologies and processing components that implement the theory—and testing the model. It stated that validation must involve a degree of independence from model development. It covered model development, implementation and use, and it stated that materiality is an important consideration in model risk management. Also, notably, the guidance discussed governance, policies and controls. This included the role of senior management in establishing a strong model risk management framework. It described senior management’s role in setting internal policies and controls and defining roles and responsibilities, including for both business units and internal audit functions.

In 2013, the FHFA issued advisory bulletin AB 2013-07, titled “Model Risk Management Guidance.” The bulletin expanded upon previous guidance issued by the FHFA, replacing 2009-AB-03. It set minimum thresholds for model risk management by outlining governance requirements, included a definition of model and model risk, and discussed model risk management and control frameworks. Additionally, it covered model validation programs and the model lifecycle.

The concept of model governance regulation is not limited to the United States. Solvency II is a European Union legislative program that was implemented on Jan. 1, 2016. One of the three pillars of Solvency II set out requirements for governance and risk management of insurers. Included in the governance requirements is model validation, which is addressed in a Lloyd’s of London document, “Model Validation Guidance.” The stated purpose was, “This document provides guidance to agents in respect of internal model validation requirements under Solvency II.”⁶  The guidance covered the components of validation, including independence, risk coverage and indicators, risk ranking and the validation cycle.

Solvency II also addressed general governance requirements. The following are included among the requirements:

1. Member States shall require all insurance and reinsurance undertakings to have in place an effective system of governance [that] provides for sound and prudent management of the business.
2. The system of governance shall be proportionate to the nature, scale and complexity of the operations of the insurance or reinsurance undertaking.
3. Insurance and reinsurance undertakings shall have written policies in relation to at least risk management, internal control, internal audit and, where relevant, outsourcing.⁷

In the years since the financial crisis, there has been a high degree of activity on the actuarial side as well. In 2010, the IAA released a practice note titled “Note on the Use of Internal Models for Risk and Capital Management Purposes by Insurers.” The note contained a section on model governance. In this section, the topics of testing and validation, documentation and audit review were discussed.

In 2012, the North American CRO Council released an article on applying model validation principles to risk and capital models. The article focused on model validation, specifically for risk and capital models. It covered key principles of model validation. These principles included the following eight points:

• Model build and design need to be consistent with intended use.
• Validation should be independent.
• There should be an owner of model validation.
• The appropriateness of the governance structure should be ensured.
• Validation should be proportional to complexity and materiality.
• Inputs, calculations and outputs should be validated.
• Limitations should be addressed.
• Validation should be documented.

There were two recent Actuarial Standards of Practice (ASOPs) released regarding enterprise risk management.  ASOP 46, “Risk Evaluation in Enterprise Risk Management,” was adopted in September 2012, and ASOP 47, “Risk Treatment in Enterprise Risk Management,” was adopted in December of that same year. Per the background given in ASOP 47, “This standard, along with ASOP 46, ‘Risk Evaluation in Enterprise Risk Management,’ is intended to cover the risk evaluation and risk treatment activities within enterprise risk management work.”⁸ ASOP 46 explicitly addressed model validation, including that “the actuary should disclose whether and how the modeled future economic conditions have been reviewed and tested for reasonableness.”⁹

Then, in December 2012, the SOA published a research report called “Actuarial Modeling Controls: A Survey of Actuarial Modeling Controls in the Context of a Model-Based Valuation Framework.” In this report, there were several key findings. Of those, of significant note was, “Governance frameworks should be set forth in order to ensure the sustainability and repeatability of the modeling process by visibly demonstrating structure and oversight.”¹⁰ Also, “Companies that have established an independent, centralized model steward function, and appropriately empowered the steward, generally have more robust and effective controls in the current state, and as such have fewer areas to improve when moving toward controls under an MBV framework.”

The report also included recommendations about key next steps to move toward leading industry practices. These include, but are not limited to, the following:

• Establish formal documentation policy for actuarial modeling processes.
• Regularly review models and the modeling process against the governance policy.
• Develop a corporate culture that values and aligns with the governance policy.

In April 2014, the Casualty Actuarial Society (CAS), CIA and SOA Joint Risk Management Section released a paper titled “Model Validation for Insurance Enterprise Risk and Capital Models.” This paper defines the types of model risk as being conceptual, implementation, input, output and reporting risk. The paper also discusses mitigating techniques for each type of risk and the risk management control cycle, including governance.

Most recently, the guidance has been focused on governance, especially as it relates to principle-based reserves (PBR). In August 2016, the Academy issued a model governance checklist. This nonexhaustive checklist was offered as a resource for practicing life actuaries involved in actuarial model governance. As the checklist states, “Its development was prompted in response to the need for good model governance as addressed in PBR regulation; however, it will also be of value wherever actuarial modeling is performed.”¹¹ The checklist covers 10 categories, including analysis and validation, reporting and governance standards.

In November 2016, the IAA adapted International Standards of Actuarial Practice (ISAP) 1A – Governance of Models. It states several noteworthy items that the actuary should do in selecting, modifying, developing or using models.

• Be satisfied that there is an appropriate model risk management framework in place that addresses identification of model risks.
• Be satisfied that an appropriate model validation has taken place.
• Understand the context in which the model will be used, how model input will be provided, and how the actuary expects the results of the model will be used.¹²

Now and Into the Future

There also has been current and pending regulatory and actuarial guidance that we can expect to be implemented over the next few years.

On the banking side, the third Basel Accord, Basel III, is currently scheduled to be implemented in 2019. This report includes a section on model validation and backtesting and requirements for ongoing and independent validation. The report states: “The Committee’s comprehensive reform package addresses the lessons of the financial crisis. Through its reform package, the Committee also aims to improve risk management and governance as well as strengthen banks’ transparency and disclosures.”¹³

There is a pending ASOP exposure draft for PBR for life products. The draft is dated June 2015 and contains guidance on model validation, including static validation. It also indicates that the actuary should consider dynamic validations, consistency with results from other internal models and change management. And per the most recent draft, it “was revised to add references to chapter VM-G (“Corporate Governance Guidance for Principle-Based Reserves”) of the Valuation Manual to help clarify that compliance is the responsibility of the company.”¹⁴

Guidance on modeling has grown to include governance. The Actuarial Standards Board recently released a modeling ASOP exposure draft. The deadline for comments on the draft passed in October 2016. The draft ASOP includes a section on mitigation of model risk, validation, and appropriate governance and controls.

Most recently, in December 2016, the Academy released a draft model governance practice note. This note covers several areas, including model definition, development, and governance policy and standards. It also includes model process and controls, validation, documentation and PBR model governance considerations.

Putting It All Together

One thing is clear: No matter the regulatory environment in the future, model governance has entered the actuarial profession to stay. We can point to all of the recent guidance issued by actuarial organizations as evidence of that.

Over the past few decades, several key themes important to future model governance have emerged. The following list covers common themes, but is not exhaustive:

1. It is necessary to have a governance structure, including a formal policy, which should include defined responsibilities and audit oversight. Considerations of materiality, coverage and risk of the associated models should be included. The definition of what constitutes a model should be specified, and a model inventory created.
2. Model validation and testing should be performed, including independent review. The validation should include verification of inputs, calculations and outputs.
3. Model documentation and documentation of validation should exist.
4. There should be ongoing validation, including that of changes. A formal model development lifecycle should be implemented.

As described in the CRO Council paper, “Model risk refers to the risk that a model is not providing accurate output, that a model is being used inappropriately, or that the implementation of a model is flawed.”¹⁵

A good model governance policy should reduce these risks and increase the confidence of model users that the model output is reliable. It should demonstrate to senior management that appropriate diligence has been applied to the development, testing and documentation of the model. It should set a standard that model developers and testers can follow consistently for future changes. And most important, if it meets these objectives, a strong model governance policy should, and will, add value to your organization.