An Open Source

Many companies struggle with the decision to adopt open-source versus closed-source systems for modeling. Models are used to price products, project future profits and determine how much capital to hold, providing important financials for financial reporting as well as management decision-making and predictive modeling. An error in a model or the modeling process can lead to huge losses, penalties, loss of reputation and even financial failure.

The banking industry has mature and regulated governance processes around its models. The insurance industry has a renewed impetus to advance a mature model governance framework due to recent awareness and new valuation regulations emphasizing model governance to reduce model risk. Model risk is an important consideration when choosing between open- or closed-source systems. A common belief in the industry is that closed-source systems pose less model risk than open-source systems, and coding flexibility is sacrificed. We believe this notion is flawed. The perceived model risk of open-source systems can be successfully minimized by imposing an appropriate governance framework over the modeling process to mitigate model risk without sacrificing the coding flexibility of an open-source system. The purpose of this article is to provide the reader with the major pros and cons of open versus closed systems to inform on decision-making when choosing between the two systems under a complete model governance framework.

Key Pros and Cons

Figure 1 summarizes the major pros and cons of closed versus open systems. The two systems are distinguished by a few key features: vendor control of code, transparency, cost, flexibility and training. The more closed the system, the less transparent and the higher the cost to make code enhancements, the more planning required to ensure enhancements are ready when needed and the more oversight needed to ensure enhancements reflect specifications. The more open the system, the more transparency of its inner workings, flexibility to make changes and the lower the cost, but the higher the need for security to prevent unintended changes to the code, an often-cited concern. However, an effective model governance framework can mitigate risks of both types of systems. It is important to note open-source codes such as R and Python are now common actuarial tools that also require a model governance framework as much as valuation and profit projection software tools.

Purpose of Model Governance

The purpose of model governance is to reduce model risk. This can be achieved by instituting strict processes and controls around model components such as source code, assumptions, data and results. Any changes to these model components must be subject to management approval. The production environment must limit access to model components to designated persons. It is important to note that both closed- and open-source systems require model governance to reduce model risk.

In a 1996 Goldman Sachs Quantitative Strategies Research Note, which is still highly relevant today, Goldman Sachs defined model risk as “the risk of loss by using a model to make financial decisions” and identified several forms of model risk.¹

They identified seven types of model risk:

1. Inapplicable model
2. Incorrect model
3. Correct model, incorrect solution
4. Correct model, inappropriate use
5. Badly approximated model
6. Software and hardware bugs
7. Unstable data

The research note provides more details about each type of model risk. However, the meaning of each type of risk should be fairly intuitive. The paper also goes into considerable detail enumerating the signs a model may be incorrect. For example, the modeler may not have considered important factors in the design of the model, or the model may be correct only under ideal conditions, which rarely present themselves.

Design and Implementation of a Successful Model Governance Framework

The model governance framework is a structured set of protocols that govern the use of modeling tools and provide guidance for the use of those tools. The initial steps in setting up a framework are to set up the environment, define the scope and establish the role of the model steward. The next three sections provide some initial guidance to define these three areas.

Initial steps in setting up a model governance framework

Setting the Environment

The implementation and preservation of a governance framework requires unwavering corporate commitment and a virtuous risk management culture. A newly formed governance process needs a lot of care and commitment for its processes and rules to be established and followed. As with any new infrastructure, there will be those resistant to change and governance. Therefore, it is important to enlighten all affected parties about the purpose and importance of the new framework. It will also be beneficial to involve all potentially affected parties in the development of the new framework and its processes. People tend to support structures they helped develop and implement. A risk management culture needs to be woven into the larger operating DNA of the department and company at large.

Defining the Scope

The ultimate scope will include all models that functionally impact the company. However, at the onset of implementing a governance process, it would be wise to select a few key models to subject to the new framework. The new framework will invariably require adjustments to address unique processing components and inefficiencies, and to avoid redundancies. Once the governance framework is sufficiently perfected it is ready for additional scope.

A majority of these functional processes include:

• Assumption-setting
• Data transfer
• Model enhancements
• Model validation
• Model corrections
• Archival of models
• Model results usage
• Management approval
• Software upgrades/conversions
• Peer review

Establishing the Function of the Model Steward

The primary responsibility of the model steward is to make sure the instituted model governance framework processes and controls are followed. The steward may not be appreciated at first by the model developers and users, and the model users and developers will have pain points as the rules are enforced. Some of the early pain points experienced may be:

• Protocol prevents making changes on-the-fly
• Perception that management no longer trusts capabilities of users and developers
• Increase in meetings, write-ups and analysis rather than making model code changes
• Too much time spent on documentation

As time goes by and governance processes are fine-tuned with the help of the model steward, developers and users will realize the framework is effective in reducing model risk. The role of the model steward is defined in the next section of this article.

Roles in a Model Governance Framework

There are different roles in a model governance framework focused on code changes. The model governance framework is a stage for a play, and the success of the play depends on the brilliance of the actors in the starring roles identified in this section.

Model Approvers

The model approvers are tasked with the responsibility of approving changes to model code and model assumptions. They are senior managers who own the assumption-setting and reporting processes. All changes to models must be warranted, peer reviewed, documented and analyzed before being presented for approval. It may be optimal to have more than one group of model approvers, depending on their expertise and the nature of the change. For example, it may be necessary to have a group approve changes to model code that is distinct from the group that approves model assumptions.

Model Developers

Model developers are authorized to change model code and assumptions. Their expertise is in the source code and product knowledge. All code changes are completed per coding standards. The impacts to model results are quantified, validated, analyzed, independently tested and peer reviewed, and documented. The model developers are a center of excellence in the company. They work on model changes at the request of sponsors and will often present their work to the model approvers for approval.

Model Users

Model users are those who use the model results for reporting and analysis purposes. They are responsible for valuation, pricing, generally accepted accounting principles (GAAP) reporting, embedded value reporting, forecasting and so on, as well as for the accuracy of the results produced by the models. They work closely with the model developers. Model users submit the majority of the requests for changes to models and are often heavily involved in the peer review of the work products before they are installed to production.

Gatekeeper

The gatekeeper is the guardian of the production models. The gatekeeper works closely with the model steward to make sure that only approved model changes enter the production zone. A predominant aspect of this role is the archiving of older production versions before replacing them with the latest version from the staging zone, and making sure the new versions perform as specified. The gatekeeper ensures the correct model is installed to the production environment.

The primary responsibility of the model steward is to make sure the instituted model governance framework processes and controls are followed.

Model Steward

The model steward stars in the leading role, making sure everyone adheres to the model governance framework. The major responsibilities of the model steward include making sure the model change control process works smoothly by facilitating approver meetings, scheduling and maintaining a list of all model change requests, prioritizing model change requests and working with all parties to ensure an optimal solution is reached for each request. The model steward is expected to have a good appreciation and understanding of the open-source software. Project management skills are a must for this role.

Sponsors of Model Change

The sponsors of model changes are usually senior management responsible for financial reporting areas such as valuation, GAAP or pricing. They work with the model steward to schedule requested changes to models. The sponsors are tasked with making sure their requested model changes are implemented accurately as specified.

The Acts of the Play

Figure 2 depicts a model change control process for making changes to system code under a model governance framework. The process becomes iterative once a change request is submitted to allow for continual peer review until the modeled change is functionally correct per the change request specifications. The act descriptions provide additional high-level narratives for the execution of each step in the change control process. It is expected that organizations create and maintain detailed implementation catalogs for each act.

• Act 1. A model change in the form of an enhancement or an assumption change is identified and enters the model change control process.
• Act 2. The model change is assigned a developer and reviewer by the model steward.
• Act 3. The model developer works on the requested model change and, along with a sponsor, presents the model change for approval.
• Act 4. The approved changes are brought into production by the model steward and gatekeeper.

The Props

Props support the execution of each act and serve to document the process of moving a requested change through the process to completion.

Formal Processes, Rules and Standards

The implementation of each requested model change needs to follow the process illustrated in Figure 2. Once the rules are set in place, skipping a step in the sequence is not allowed. The amount of rigor built into each act is deliberate for defining and instituting minimum standards for the following areas:

• Coding. As per the coding standards in place.
• Peer review. Appropriate rigor based on the nature of the change.
• Analysis. Appropriate rigor based on the complexity of the change.
• Validation. Level of tolerance specific to metric.
• Documentation. Level of documentation based on its purpose.

Separation of Duties

Actors should only play one part in the process to prevent dilution of and conflicts with standards in implementing other roles. Allowing actors to reenter the process at different entry points defies the very purpose of a model governance framework—independent validation of work products. Therefore, roles and responsibilities need to be defined carefully and clearly. They need to be formally documented and followed at all times.

Information Technology

IT is a key component for the success of a model governance framework. All of the defined roles will need to interact with the IT department to make sure:

• Everyone has the correct access and restrictions to directories and networks.
• Renewal of software licenses and software updates work smoothly and on time.
• There are IT professionals dedicated to the framework and software.
• Disaster recovery and backup processes are set in place.
• IT process controls are in place to audit all work performed.

Final Act

The successful implementation of a model governance framework will have these benefits:

• Model developers will have the satisfaction of working on model changes in a controlled environment. They will learn to abide by the framework in place to perform their role effectively, creating excitement and satisfaction for all.
• The model steward will continually improve controls and processes with the advance of actuarial and regulatory practice. With roles clearly defined, all actors will play their parts more effectively and with a greater commitment to the model governance framework.
• The organization will benefit from a significant mitigation of model risk, resulting in an increased level of confidence in the models by senior management and pride of ownership among the modeling community. The mitigation of risk effected by such a framework will eliminate a number of the cons between open- and closed-source systems to permit an on-par evaluation of the capabilities of each, and, hence, the cost-benefits of each to an organization.

The design and implementation of governance frameworks will differ from organization to organization. Governance frameworks should be designed around existing frameworks, culture and personalities unique to the organization. If senior management has the desire and commitment to implement a successful model governance process, they should certainly seek guidance from experienced professionals.