Put to the Test

By now you know that starting July 1, 2018, the Society of Actuaries (SOA) will be adding a significant amount of predictive analytics material to the requirements for Associateship. Committees are at work defining the learning objectives and readings, which should be set by July 1, 2017. Key to this introduction is the development of an assessment that will test a candidate’s ability to construct a model, evaluate it and then communicate how that work provides an answer to a business problem.

While many details of the logistics of the assessment remain to be determined, past SOA assessments provide guidance as to how this may be done. In this article, I review those past assessments and provide an indication of how they likely will be adapted to the new predictive analytics exam.

Intensive Seminar Assessments

As part of the Flexible Education System and Future Education Methods introduced in the 1980s, the SOA introduced two intensive seminars. One was Course 121, Intensive Seminar on Applied Statistical Methods. This seminar was built on the theory of regression and time series, as presented in the Course 120 exam on these subjects. Part of each candidate’s assessment was a project report.

The instructions for this project say: “The analysis will consist of creating one or several explanatory statistical models and using this framework to interpret the data. … The final report is to be geared to a sophisticated client with a limited to moderate amount of statistical expertise.”

For example, one project asked candidates to use a model to report on the determinants of total cost per hospital admission for a group of insured patients.

Course 7 in Applied Modeling Assessment

From 2000–2007, a fellowship requirement was the Course 7 Seminar in Applied Modeling. As with Course 121, several days of instruction were followed by a project and a report. The focus was more on the modeling process than on a particular technique. One specific project asked candidates to build a regression model to understand the drivers of lapsation. Once again, communication is important.

The instructions of this assessment say: “An important thing to keep in mind as you produce the final report is to write for your audience (the CFO). … His background is in accounting, so he is technical but doesn’t have an actuarial background. Also, he has been in management for a while so he is looking to hear from you just what he needs to know.”

Applications of Statistical Techniques Module

Introduced in 2013, this e-Learning module is a requirement for candidates on the General Insurance track. The focus is on the generalized linear model (though other techniques are also covered), with particular emphasis on its use in classification ratemaking and measuring reserve variability. As with all SOA e-Learning modules, candidates work remotely. Instruction is provided in the R programming environment, and the project requires a data set to be analyzed using R commands. The project is completed offline with the report uploaded for grading. As with the previous assessments, communication is important. For one of the tasks, the instructions inform candidates that their “memo is to be written to Kevin, an experienced statistician. … Given Kevin’s background, this memo can use statistical terms without explanation, but still must provide convincing evidence that you have the best model from the available data.”

Assessing Predictive Analytics in 2018

Starting in the fall of 2018, candidates will be required to pass a predictive analytics exam. As was the case with the previously described courses, the most effective way to assess a candidate’s ability is with a project. Each of the three previous project-based assessments had a weakness that will need to be overcome.

Intensive Seminar 121 was available for elective credit. As such, a limited number of seats were made available each year. This limitation made it possible to offer the seminar at a few universities. Computer labs were available with the necessary software loaded (Minitab, in this case). The proctors were the instructors for the course, so expert assistance was available.

Course 7 was a requirement. At maturity, 160 candidates attended a given session, usually at a hotel ballroom. Candidates brought their own computers, with Microsoft Excel being the only analytics software required. Candidates were allowed to leave the ballroom and work elsewhere (on an honor system without help). Again, the proctors were also the instructors, and thus appropriate help was available.

The Applications of Statistical Techniques project is done as a take-home test. This absence of proctoring is deemed appropriate for fellowship modules but may not be for other assessments.

For the new predictive analytics assessment, some of these problems persist. The large number of candidates means it is likely that a large space, such as a hotel ballroom, will be required. However, the SOA Board has asked that this assessment be fully proctored, so candidates will need to stay in the room. They likely will be required to bring their own computers. This leads to three challenges.

1. Ensuring the proper software is loaded and ready to run. The current thinking is that R, Microsoft Word and maybe Microsoft Excel will need to be available. Possibly, a PDF version of the required text and/or help files also will be available.
2. Ensuring other uses of the computer are blocked. This would include access to the internet or any files or programs beyond those in the first challenge.
3. Proctors may not be knowledgeable in the subject matter.

The first two challenges will require special blocking software to be installed on each computer. The SOA is working to find a vendor with a system that meets our needs. Candidates will need to have prepared their computers in advance, and some basic troubleshooting tools and knowledge should be available on-site. For the third challenge, it will be necessary to ensure that proctors are appropriately trained and candidates be made aware that limited assistance will be available.

As with Course 7, many candidates will need to travel a greater distance to an exam center than for a typical SOA exam. While this assessment is at the Associate of the Society of Actuaries (ASA) level, it is expected that most candidates will take it late in their pathway and thus be more likely to have the resources to cover any additional costs this extra travel may incur.

We are excited about this new development in actuarial education. Look for announcements regarding more details of this assessment in mid-2017.

Questions about the new predictive analytics assessment or other aspects of the SOA’s education system can be sent to education@soa.org.