Beyond: Directions to frontier

What’s beyond? I. Advanced stuff

Our textbook is not a pamphlet, you know it’s 750 pages. But even if it is comprehensive, it will obviously miss a great deal of methods, tools and ideas. Here we collect some brief pointers to more advanced ideas that could be useful for an analyst. For books, see our reading recommendations; for papers and posts organized by chapter, see extra reading by chapter.

What’s beyond? II. New ideas, changed views

When we were students, textbooks seemed like the collection of ultimate and absolute truth. What’s in a textbook must be right. But well, not really.

Sure, there are views that are pretty stable. The first derivative of \(\ln(x)\) is \(1/x\). Smoking is bad for health. But some widely held views do change. Back in the eighties, butter was thought to be bad and you were supposed to get margarine. It turned out, while too much butter is not good, margarine is worse overall.

Similarly in statistics, machine learning, and causal inference, we have known OLS is BLUE for a long time. But the fact that we can get negative weights when estimating a time and unit fixed effect model is pretty new. It was discovered around the writing of the manuscript for the first edition.

Some stuff in the textbook is close to the frontier of knowledge. But this frontier might change. So, here we’ll collect a bunch of very short discussions of emerging ideas.

Chapter by chapter

Part I

Chapter 01: Origins of Data

  • We talk about surveys quite a bit in Chapter 01, giving some practical advice as well. But, of course, there is more. In many jobs, private and public, you will be asked to design surveys. A great review is “How to Run Surveys: A Guide to Creating Your Own Identifying Variation and Revealing the Invisible” by Stefanie Stantcheva, Annual Review of Economics (2023), also as an NBER working paper. With ideas, and cautionary tales.

Chapter 02: Preparing Data for Analysis

Chapter 03: Exploratory Data Analysis

  • Data visualization. Here are some great examples to study a series of good decisions: the Carta 2025 deck on venture capital, as reviewed by the Pragmatic Engineer newsletter. Consider: format, focus, title, ink-to-text, use of color strategically. Great content, too.

Chapter 06: Testing Hypotheses

Part II

Chapter 09: Generalizing Results of a Regression

  • Regarding external validity, one way to check robustness is to take out 1% of the data and repeat the exercise. The simple take is to do it many times randomly + many times by the edge of the distribution of key variables. The smart take is suggested by Tamara Broderick, Ryan Giordano, Rachael Meager in “An Automatic Finite-Sample Robustness Metric: When Can Dropping a Little Data Make a Big Difference?” Hard-core statistics. Preprint

Chapter 10: Multiple Linear Regression

  • Interaction term interpretation. It turns out the functional form of \(x\) matters a lot. See the debate and example with \(\ln(x)\) vs \(x\) by Uri Simonsohn at Data Colada. While aimed at political scientists, the core argument shared by the debaters is that mistaking functional form is a key issue.

Part III

Chapter 14: Model Building for Prediction

Chapter 16: Random Forest and Boosting

Chapter 17: Probability Prediction and Classification

  • As machine learning methods developed, one key desire has been interpretability. What are the key predictors? How do features relate to the outcome? A massive literature emerged using SHAP, an algorithm based on the game theoretical idea of Shapley values. The original paper is Lundberg and Lee (2017), A Unified Approach to Interpreting Model Predictions. A good introduction is Christoph Molnar’s Interpretable ML book. It looks a great tool. However, a paper published in early 2024 showed severe limitations of the method. In a PNAS paper Bilodeau, Jaques, Koh, and Kim Impossibility theorems for feature attribution advised against using it.

Part IV

Chapter 19: A Framework for Causal Analysis

  • On DAGs and Potential outcomes, deep discussion for social scientists: Imbens, Guido W. 2020. “Potential Outcome and Directed Acyclic Graph Approaches to Causality: Relevance for Empirical Practice in Economics.” Journal of Economic Literature, 58 (4): 1129-79. LINK to paper. An amazing review that includes Twitter quotes.

  • Beetroot juice is said to be great. Review study Another review. For example, reference to an RCT with beetroot juice – dietary inorganic nitrate acutely reduces blood pressure. Study. Review in medical journal

  • Another issue is vitamin intake, already featured in the book. In 2024, a large scale study was published following 400K individuals in the US over 20 years. They found no correlation between life span and vitamin intake: Loftfield et al (2024), Multivitamin Use and Mortality Risk in 3 Prospective US Cohorts, JAMA Network Open. The issue with beta carotene is actually noted there, too.

Chapter 20: Designing and Analyzing Experiments

  • On A/B testing, some neat ideas in a presentation by Harlan Harris, with code in R

Chapter 23: Methods for Panel Data

  • A very influential paper on difference in differences is Cengiz, Dube, Lindner, Zipperer QJE 2019 on minimum wage estimation. They show that simple TWFE makes comparisons that are practically irrelevant (e.g. well before the intervention), biasing the treatment effect estimation.

Chapter 24: Appropriate Control Groups for Panel Data

  • Event studies. A great review to see more is Miller’s 2023 JEP paper, “An Introductory Guide to Event Study Models”. Fantastically useful summary. “Event study models are great! But behind that attractive interpretive graph, researchers are necessarily making decisions. This raises risks of bias due to systematic (if perhaps unconscious) model selection processes, committed by either the researcher or the journal review process. Despite these risks, these decisions are unavoidable. There is no “button to push” that can automate the necessary judgment calls. For now, best practice should be to increase transparency through bringing clarity about the specification decisions made (and the reasons for those decisions) and to discuss robustness to alternative decisions, along with providing both estimation code and (whenever possible) data for replication.”

  • Fixed effects. Difference in differences. Event studies. These core methods just experienced the largest revival in research for two decades. Since the finish of the 1st edition manuscript in the beginning of 2020, several very important papers were published on potential pitfalls of the two-way fixed effects model. There are several key new results. First, weights matter. For example, \(xt\) fixed effect results may be incorrect in the sense that some observations are taken with negative weight. Second, more attention must be paid to selecting control observations, especially when the intervention is staggered (deployed at different times on different units) and/or heterogeneous in terms of dose.
  • Synthetic controls and difference in differences evolved separately. But recently a marriage of convenience is taking place. In a 2021 AER paper, Arkhangelsky, Athey, Hirshberg, Imbens and Wager, Synthetic Difference-in-Differences, offered a way to think about them together. Also, Susan Athey has a great youtube video presenting it. On synthetic controls, the best place to start is Abadie (2021), Using Synthetic Controls: Feasibility, Data Requirements, and Methodological Aspects, Journal of Economic Literature. See also Abadie and Vives-i-Bastida, Synthetic Controls in Action.

After

Methods books