On November 19, 2004, the Payment Cards Center and the Wharton School's Financial Institutions Center hosted a "Forum on Validation of Consumer Credit Risk Models." This one-day event brought together experts from industry, academia, and the policy community to discuss challenges in validating credit scoring and loss forecasting models.
Retail lenders, and particularly credit card lenders, use statistical models extensively to guide a wide range of decision processes associated with loan origination, account management, and portfolio performance analysis. The adoption of sophisticated credit scoring models has increased the speed, precision, and efficiency of underwriting decisions and facilitated more granular risk-based pricing and has led to the broadening of consumer access to credit. Banks also use models to forecast aggregate losses, an important factor in defining capital requirements. The new Basel Capital Accord will soon allow banks to satisfy regulatory capital requirements based on internal models, which one observer stressed will essentially “raise the bar” when it comes to related validation processes.
While the many advances in modeling technologies have facilitated the growth of and enhanced the efficiency of the credit card industry, the pervasive use of these tools has increased the importance of validation processes and procedures. Very simply, the validation process is designed to answer the question: How do we determine whether our credit risk models are working as intended?
The Center chose to focus discussion on credit scoring and loss forecasting models, in part, because they are arguably the most important and commonly used statistical tools in retail banking. At the same time, the conference organizers were eager to gain insights into the theoretical linkages between these two processes. While theoretical arguments are often made about the linkages, in practice the industry tends to treat them as distinct tools. Very basically, a credit scoring model predicts how likely it is that an individual borrower will default, a prediction that intuitively is an essential element in developing a forecast of future losses. During the conference, Federal Reserve Bank of Philadelphia Visiting Scholar Nick Souleles, of the Wharton School, argued that while there are substantial challenges in bringing these approaches closer together, there are also real benefits to be gained.
In addition to this general topic, the conference summary, highlights several other key areas of the day's discussion. Among these were the discussions about the questions: What are appropriate metrics for evaluating a model’s performance? What other validation criteria are important? Can model performance be enhanced by incorporating macroeconomic and other market variables? While the intent of the conference was not to resolve these issues, the conference did provide important insights and furnish directions for new research. In a more general sense, participants agreed that model use and validation is a management process where “art” must be considered along with “science” to achieve optimal outcomes.