It is easy for non-statisticians to get lost in statistical PPNR (Pre-Provision Net Revenue) modeling debates. But increasingly, executive viewpoints are needed in areas formerly delegated to specialized technicians. For example, many Treasury executives have grown frustrated with the fight over the use of autoregressive error terms (“AR terms”) to address serial correlation in PPNR balance models — in light of recent comments on the academic paper by Wilkins (2013). This Novantas Perspective lays out (“in English” or mostly so) the arguments for and against including AR terms in PPNR balance models and offers guidance on how to develop a perspective on AR terms that is appropriate for your bank.
What Are “AR Terms” Anyway?
Adding AR terms as independent variables in a regression analysis is one of a handful of tools a model developer can use where the past performance of a time series model can impact future predictions. In this case, the AR terms are attempting to control for serial correlation (“If we over-predict balance change in August, we will probably over-predict balance change in September.”) When left untreated, serial correlation undermines models — it can easily lead to a model being unfit for purpose or otherwise failing model validation. Because AR terms can eliminate serial correlation, model developers often favor AR term inclusion (although there are other approaches, e.g., the use of Newey-West or certain other heteroskedasticity-consistent standard errors).
AR terms, when used, are independent variables just like macroeconomic factors: they have coefficients which indicate magnitude and p-values which indicate statistical significance. Their inclusion can strengthen or weaken the magnitude and statistical significance of other independent variables — notably macroeconomic variables.
From a model development standpoint, there can be both technical and practical justifications for the inclusion of AR terms in time series models. Technically, the inclusion of AR terms can resolve a critical assumption underpinning many common modeling techniques: that no serial correlation, or time-dependent bias, exists in the error term. Practically, models with AR terms are dynamic models characterized by a dependent variable with “memory” of its past performance — current values are influenced by past model performance even after exogenous factors have been accounted for. From a business standpoint, this can be necessary when statistically significant factors are not reflected in the model but affect multiple months of balance performance.
In PPNR balance modeling, this often takes the form of some unexpected tailwind or headwind on monthly balance growth; if an otherwise well-specified model under-predicts growth in July, it then becomes more likely that the model will under-predict growth in August. There is some otherwise uncaptured dynamic beginning in July which spills over into August (e.g., unmodeled elements like a one-off incentive to improve front-line sales, positive press over the introduction of a new product feature, or a marketing campaign with unusually long-tailed effects). The inclusion of AR terms allows the modeler to capture these business dynamics (making it practically justified) and eliminate serial correlation (making it technically justified).
AR Terms Clearly Matter
We most frequently see three status quo treatments of AR terms in banks’ PPNR balance modeling frameworks:
- Option 1 — Exclude. Develop models without AR terms (e.g., standard OLS models from Economics 101)
- Option 2 — Mixed. Develop models with AR terms, but do not apply them in model execution (backcast or forecast). (Note: this approach is impure. If allowed, backcast and forecast results should be shown to be within an acceptable range of Option 3 results)
- Option 3 — Include. Develop models with AR terms, and apply them in model execution
The graph below shows a sample model backcast where a near-final model’s AR terms met technical and practical justifications. In this example, the AR terms were meaningfully large. Note how differently the three approaches performed.
These differences can also have a material influence on forecasts. Below is the same model, forecast with and without the AR terms:
The variance among approaches underscores the importance of AR terms in PPNR balance modeling and demonstrates the importance of every CCAR bank having clear guidance for AR term usage in their PPNR modeling frameworks.
What the Academics Say
Coming out of the 2015 development cycle, we heard some concern that the use of lagged dependent variables (LDVs, a broader category that includes AR terms) as independent variables are being met with skepticism, if not scorn, by regulators. The academic study most frequently being referenced is a 2013 paper by Wilkins; to understand the argument, we need to rewind to 1997 and consider intertwined arguments from several sources:
Achen (2000) determined that the inclusion of LDVs can produce incorrect results due to bias in the resulting estimates, even if the LDVs are known to be related to the current period dependent variable. As he noted, where the use of LDVs has strong practical justification, their use should produce a correctly specified model and eliminate bias — not introduce it. However, where there is only weak practical justification, the resulting LDV models are likely to produce significant misspecification errors — with exogenous factors at risk of being dwarfed by the LDVs. Achen concludes that the risks outweigh the value and advises against including LDVs at all. Note that Achen worked in an academic setting with data that was carefully constructed to fit his desired scenario.
Keele & Kelly (2005) countered that Achen’s viewpoint was too narrow, primarily because he used data that exhibited a rarely-seen set of conditions: a dependent variable where inclusion of lagged terms failed to control for serial correlation. They concluded that LDVs can frequently be justified to control for serial correlation and, if error terms are not strongly serially correlated, provide minimum bias estimates under OLS.
Wilkins (2013) contends Achen and Keele & Kelly found bias not because of theoretical misapplication, but because they focused on Achen’s model — which should have included more than one LDV. A model with a single LDV was insufficient, not because LDVs are technically inappropriate, but rather because Achen failed to include the proper number of them. Wilkins also noted that the analysis in all three papers was performed in a controlled data environment and that modeling LDVs with “real” data is considerably more challenging — since it is generally impossible to establish practical justification with certainty.
And prior to this series of papers, the guidance by Wilkins regarding model specification was presaged by Johnston & DiNardo (1997), which commanded model developers to focus on practical soundness, not just technical soundness. While discussing model development in general, Johnston & DiNardo posit questions salient to PPNR balance modeling. Do our goals include:
- Meaningful interpretation of model parameters? (Which variables influence balance and by how much?)
- Forecasting the criteria based on changes in the distribution of predictors? (What are our scenario forecasts?)
- Determining whether the specified model is structurally invariant to changes in the marginal distribution of predictors? (How confident are we in the estimated effects of our independent variables?)
If these are indeed our goals, they contend, then model developers should bias their models to solutions that adhere to well-considered practical soundness — which in PPNR modeling means active business involvement. There is no rote, model-in-a-vacuum, analytic solution that will otherwise force the models to make sense.
We agree wholly with the thesis that PPNR balance models must address technical issues while being conceptually sound.
Adding AR terms can frequently assist in technical soundness by eliminating serial correlation. But AR terms should not be included without practical justification — which must ultimately come from the business. If there is business logic that supports this conceptual justification, and the technical soundness requirement is met, we feel the use of AR terms is an appropriate means for addressing serial correlation.
Furthermore, the science of appropriate treatment of serial correlation is a work in progress, and new techniques are constantly being developed and tested. For example, models tested using Newey-West Standard Errors (part of a broader family of heteroskedasticity- and autocorrelation-consistent standard errors) are effectively evaluated in such a way that a model with serial correlation can still be appropriately specified without identification of any autoregressive terms. This and other alternatives are interesting avenues for further research as the industry continues to evolve its approach to PPNR forecasting.
We welcome your feedback and are happy to continue the conversation about this article or other Treasury and Risk viewpoints. Please reach out to the head of Novantas Global Treasury and Risk, Pete Gilchrist at firstname.lastname@example.org; or the head of Novantas PPNR Modeling and Forecasting, Jonathan “Wes” West at email@example.com.
Achen, Christopher H. (2000). “Why Lagged Dependent Variables Can Supress the Explanatory Power of Other Independent Variables.” Presented at the Annual Meeting of Political Methodology, Los Angeles.
Johnston, Jack, and John DiNardo (1997), “Econometric Methods,” Fourth Edition, New York: McGraw-Hill.
Keele, Luke and Nathan J. Kelly (2005). “Dynamic Models for Dynamic Theories: The Ins and Outs of Lagged Dependent Variables.” Political Analysis, 23, 186-205.
Wilkins, Arjun S. (2013). “To Lag or Not to Lag? Re-evaluating the Use of Lagged Dependent Variables in Regression Analysis,” mimeo of 13 May 2013, Stanford University.