Improved modeling for pre-provision net revenue requires strengthening the project framework, working to overcome data sterility and testing alternative approaches.
In past cycles of the Federal Reserve’s Comprehensive Capital Analysis and Review (CCAR), banks pushed to improve their modeling for credit losses as a key input for capital management. More recently the Fed is pressuring banks to advance their modeling for pre-provision net revenue (PPNR). Both in public pronouncements and in private memoranda to many CCAR banks, the Fed is expecting progress in two areas:
First, PPNR models are now expected to reach a level of rigor and consistency in statistical approach commensurate with previous advances in loss modeling. Second, banks are being asked to ensure that their advancements in PPNR modeling bring capital management closer to business forecasting and decision-making.
However, banks and their regulators find themselves in a paradox. Even as banks improve their competency in PPNR modeling, our ongoing review of prevailing practices among CCAR banks indicates that many institutions are shifting toward an increasingly similar and rigid approach to satisfy important statistical tests. This exposes banks and the industry to two implementation risks.
One is business line disconnect. In the act of assuring adherence to regulatory statistical requirements, models are typically bleached of management-related variables, causing the business units to fall back on time-worn approaches (which are lacking in statistical sophistication) to make forecasts and business decisions. This outcome is at odds with the Fed’s desire to integrate capital management decisions with business strategy and execution.
Meanwhile, CCAR banks are tending to converge on one specific approach to PPNR modeling. This creates systemic risk that the industry collectively might make material misestimates of PPNR under different scenarios.
To meet these challenges and expedite their PPNR projects for full benefit, progressive banks are focusing on three priorities. One is developing the right modeling project framework, with emphasis on detailed preparation and a strong working team. Another is overcoming data sterility by incorporating management actions into predictive models. Third is embracing a challenger mindset by testing proposed models against alternative approaches — essential to avoid a singular reliance on a “standard industry approach.” Banks at all stages of PPNR development can use these keys to improve the robustness of their models and enhance their ability to drive business decision-making.
The time between receiving regulator feedback and submitting models for internal validation is short, even considering the 90-day extension of the submission deadline for next year’s plan. Most banks are compelled to cram their modeling development into a tight window — often 90 days or less.
Given this quick pace, one of the largest risks to a successful development cycle is identifying a critical gap late, without enough remaining time to thoughtfully redevelop a model. Banks can avoid this trap by developing a strong working team early in the process — before modeling begins.
This team includes model developers, business managers and model validators. Supported by the right governance, a well-prepared development team can:
Eliminate re-work. CCAR models attempt a delicate balance between statistical reasonableness and business intuition, using a relatively limited set of potential drivers. When banks discover problems late in the process, they do not get “extra time” to correct troublesome issues (e.g., finding that the underlying data of macroeconomic variables are wrong, or that the intended uses of the model has changed). Instead they must settle for making less progress than what was committed to their Board and to their regulators.
Set validation expectations. To make the validation process more productive within their required independent framework, model developers need to stay abreast of any “must do” items, including required advances needed to satisfy Fed Memorandums Requiring Attention (MRAs), or other, more stringent orders.
When brought into the process early, business partners often are more cognizant of the statistical requirements and corresponding limitations of the models. Likewise through early involvement, model validators can provide their general parameters and suggest potential problem areas that the developers should rectify before they go through the labor-intensive phase of documenting models for official review.
The best PPNR development frameworks are clarified and agreed upon before model development begins. This is critical, as timelines suffer when goalposts are moved. The working group must:
Define and source the modeling dataset. Data must be well-defined, extensive, and assembled at the best available level of granularity and frequency. Often this requires judgment calls between models built on shorter, more granular datasets vs. longer-term and less segmented data.
Agree on what to model. Model development primarily will be guided by business needs and data availability, but decisions should be made up-front on whether models will be developed at the general product level or the LOB-specific product level (e.g., consumer DDA); and whether the portfolio or cohort components will be tested.
Finalize the statistical techniques and acceptance criteria. There are some non-negotiable statistical tests every model must pass. The importance, desired structure and acceptance criteria of many other tests — notably out-of-sample testing and serial correlation checks — can vary from bank to bank. Also as addressed more fully below, testing alternative modeling approaches with potentially different statistical requirements is an important cross-validation exercise to be encouraged.
Incorporate business feedback early and often. A critical requirement of CCAR models is that they reflect business intuition and are “bought in” by business managers. Best-practice teams solicit hypotheses from business managers before model development begins, and they review results iteratively as models are finalized.
PPNR Modeling: The Basics
What is PPNR?
Pre-provision net revenue (PPNR) measures net revenue from spreads and non-trading fees. It is similar to operating revenue but excludes items covered elsewhere in the Fed’s Comprehensive Capital Analysis and Review (CCAR). These exclusions include credit losses; markets and trading revenue (typically material for only a handful of banks); and losses stemming from operational risk (for instance, legal settlements).
Why has PPNR modeling become a top priority within CCAR?
The Fed’s focus on PPNR represents a logical progression in its campaign to ensure that major banks have a complete understanding of their capital positions, both in base and stressed scenarios.
In the early years of stress testing, the top priority was ensuring that banks understood the potential impact of credit losses on capital. But as banks improved their credit modeling competency, regulators found that the variability revealed by PPNR estimation (balances, related spreads and fee income) exceeded variations in credit losses. PPNR was the logical candidate for Fed emphasis.
Why is PPNR modeling so challenging?
At first blush, PPNR modeling appears similar to recurring exercises for line of business budgeting and forecasting. But building models to CCAR standards introduces many complexities relative to traditional forecasting approaches, often leaving business lines alienated; modeling teams exasperated; and regulators and model validators dissatisfied.
Data requirements. PPNR modeling often requires much more data than banks typically use for planning purposes, both time span and breadth, including:
- Ten years of monthly balance and fee data (or at least enough to capture a full rate cycle).
- Portfolio-level balance histories at the minimum; ideally such information at the account/customer level.
- Records of management actions (e.g., pricing, marketing, distribution changes) for this long historic interval.
- Treatment for mergers and acquisitions, and other disruptive events.
- Modeling requirements. Unlike customary line-of-business planning exercises, PPNR models must be able to withstand model validation and regulatory scrutiny, pressurizing the model development process.
- PPNR models are typically built within a framework of time-series modeling, for example, which is not frequently used elsewhere in the bank. This often leaves the modeling and validation teams treading on unfamiliar ground.
Another issue is the time horizon and types of variables included. PPNR models typically are expected to explain long-term trends, using both macroeconomic and management variables. By contrast, traditional bank internal forecasting considers a shorter time horizon and frequently includes only management levers. This can lead to situations where model developers and LOB executives wind up “talking past each other” in hypothesis development and driver review.
Taken together, these development challenges often skew the resulting PPNR models in one of two directions: Either they satisfy statistical “purity” while becoming divorced from the expectations of the business; or they satisfy business intuition but fail critical statistical tests. Both pitfalls can result in regulatory sanction and create risks in bank capital management.
—Andrew Frisbie and Jonathan West
Overcoming Data Sterility
Most PPNR teams have realized that the uniqueness of their bank’s strategy and go-to-market approach limits the ability to model a powerful statistical relationship between balance/fee movements and purely macroeconomic factors. For this reason, specific management actions have become a vital input into the more predictive models. Key examples include:
Product rate relative to the competition. For rate-based products, this measure tends to be one of the most influential in improving a model’s predictive power.
Distribution coverage. Examples include the branch network, commercial relationship managers and sales staffing. Banks that make major changes in distribution coverage often experience a multi-year lagged influence on overall balance growth. No other measure directly captures this dynamic.
Product introductions. The launch of a product with different or innovative features often introduces a bank to new market segments which may have been historically underserved, resulting in aggressive short-term growth.
By accounting for such factors, banks can materially improve the predictive power of their models. While broad-brush influences on a product portfolio may track with macroeconomic factors, intra-year or intra-quarter movements often are more directly influenced by the specific levers pulled by management.
Once quantified, these levers can be presented as clearly-defined management adjustments to the macroeconomic-based forecast in the final CCAR submission. Otherwise banks must rely on “finger-in-the-wind” estimation when proposing adjustments to strict model outputs. These purely qualitative adjustments are far less defensible and more likely to invite regulatory and validation scrutiny, compared with documented management actions with proven measurable effects.
Most banks use some level of challenger models and analytics for a “sense check” of candidate model results, compared with industry deposit changes in stressed environments. Robust challenger processes do not treat this as a pro forma exercise, but rather as a mindset.
The goal is to insulate the bank from potential downsides, either stemming from deficiencies in an individual model, or from a systemic deficiency stemming from industry over-reliance on one specific model theory, functional form or set of statistical constraints. A stronger challenger discipline allows banks to:
Quantify the impact of alternative models for a given product. One example is evaluating the effect of certain substitute macroeconomic variables. Another is comparing models that are dominated by macroeconomic variables vs. alternatives that document the influence of management actions.
Evaluate trade-offs, including those inherent in different modeling methods. In the statistical realm there are pivotal decisions to be made about the type of regression technique to be used (for instance, ordinary least squares (OLS) vs. partial least squares (PLS)). Also there are tradeoffs in applying differing standards of statistical purity vs. business intuition.
Leverage alternative analyses to enrich the narrative; the management overlays; and the plans for ongoing progress in PPNR modeling.
To establish the right fact base, each candidate model should have alternatives that test competing and related macroeconomic variables, enabling sensitivity analysis. Banks can also consider applying alternative analytic approaches (e.g., portfolio modeling, cohort-vintage modeling, industry and market share modeling) across the overall set of models, permitting tighter side-by-side comparisons that help to identify consistencies and deviations among significant variables and results.
Andrew Frisbie is a Managing Director and Jonathan West is a Principal at Novantas Inc., a management consultancy based in New York City. They can be reached at email@example.com and firstname.lastname@example.org, respectively.