Article
In the lead-up to the new pension system, processes and calculations are being rapidly automated, often under significant time pressure, while the outcomes of these models play a crucial role in decision-making. Therefore, it is essential to focus on model validation to ensure that a model is "fit for its purpose" and remains reliable in the long term.
In preparation for the new pension system, many organizations are undergoing significant changes. One of these changes is the automation of processes and calculations. Ideally, a new solution should be implemented as quickly as possible, as the Future Pensions Act will not wait for the models. Furthermore, crucial decision-making components will rely on the outcomes and advice resulting from these solutions. This can be challenging, as they are developed within a limited timeline. For this reason, it is crucial to pay attention to the reliability of these models, which can be achieved, among other things, through proper validation.
Model validation can be conducted in many different ways and is highly dependent on the model, its use, and the associated (legal) requirements. Moreover, validation can be seen as an ongoing process: when you dive deep enough into a model, there will almost always be attention points that emerge. That being said, to achieve approval during a validation process, finding all minor inaccuracies is less important. It is more important that a model is “fit for its purpose.” “Fit for its purpose” means that the model is judged on its ability to do what it is supposed to do. Ultimately, validation is a continuous process of monitoring the functionality and assumptions of the models through changes and responses to those changes. Therefore, model validation can be split into two phases: delivery and the future.
Delivery
To undergo a proper validation process when delivering the model, we advise thoroughly considering the model requirements and how the project's validation will be set up before starting a project.
In the digital transformation of an existing model or process, verification of results can rely on existing outcomes. In large models with many input possibilities, these could be thousands of different input combinations. This can lead an organization to decide to set up a separate validation model with the aim of reconciling the results of all different combinations. Another approach is to test the model with practical cases or compare a selected set of input possibilities. Additionally, it is important to think carefully about the “extreme usage” (an unexpected combination of settings). For instance, what happens when you enter a negative age, and is this a functionality you want to capture, or do you choose to rely on the users' common sense?
For new models, the preparation typically relies on the theoretical knowledge of the model developers. Depending on the structure and goals of an organization, validation can be conducted using second-line reviews, the four-eyes principle, or validation by an external party. If this new model is set up in a programming environment, it can introduce additional uncertainties, given that there may be limited programming knowledge within the organization. In this case, technical documents, recording interim results, or a translation by an external party can provide additional support.
The Future
Once a model is in use, it should be periodically monitored. We recommend documenting and verifying new functionalities and changes (with associated consequences) according to the same processes as when delivering the model.
Additionally, setting up a continuity plan at the beginning of the project provides support in introducing a future-proof model. This can be seen as a dynamic plan that can change in response to new developments. We advise updating the plan in line with the latest developments.
In Conclusion
Validating new models is an important factor in setting up a new project. The application of new technologies brings new challenges. We therefore invite you to carefully consider the requirements and validation process before starting a project. Consider the model requirements, the control of results, and the continuity of the model.
