Excerpt

Quantitative research has traditionally been focused on estimating the parameters by which different variables are related, with an emphasis on establishing causal relationships. It is well known that any study using observational data has to overcome fundamental challenges to its internal validity and precisely elucidate the reasoning and assumptions made about the counterfactual. In this chapter, we address this crucial concern in the context of the new wealth of textual observational data now available for quantitative approaches to legal studies. We also discuss at a high level the opportunities made possible by these new data, as well as the methodological considerations raised by the use of quantitative techniques in a domain heretofore dominated by qualitative approaches.

The Challenge of Causal Inference

Most traditional work in the field of quantitative empirical legal studies is concerned with estimating parameters that describe the relationship between different variables, with a particular focus on establishing a causal relationship. This is most evident for research attempting to measure the effects of certain policies (e.g., Levitt 1997) or interventions in the legal process (Greiner and Wolos Pattanayak 2012; Ho, Sherman, and Wyman 2018).

Yet, much quantitative research in social science also falls in this category, even when the research is not explicitly framed in causal terms. For example, the research investigating differences in the behavior of individual decision-makers (e.g., Sunstein et al. 2006) amounts to estimating the effect of the involvement of individuals with a specific background in a legal case. Causal estimation is difficult because it invariably involves a counterfactual element. In principle, a researcher interested in estimating a causal relationship wants to compare the state of the world after a certain intervention with the state of the world had there not been an intervention. As this is impossible, the main challenge in causal estimation is to find a way to extrapolate how those parts of the world affected by an intervention would look had they been unaffected by it, and the other way around.

Bibliography

Abbott, A. 2001. Time Matters: On Theory and Method. Chicago, IL: University of Chicago Press.

Angrist, J. D., and J.-S. Pischke. 2008. Mostly Harmless Econometrics: An Empiricist’s Companion. Princeton, NJ: Princeton University Press.

———. 2010. “The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics.” Journal of Economic Perspectives 24 (2): 3–30.

Axelrod, R. 2015. Structure of Decision: The Cognitive Maps of Political Elites. Princeton, NJ: Princeton University Press.

Bennett, A. 2008. “Process Tracing: A Bayesian Perspective.” In The Oxford Handbook of Political Methodology, edited by J. M. Box-Steensmeier, H. E. Brady, and D. Collier. Oxford, UK: Oxford University Press.

Brady, H. E., and D. Collier. 2010. Rethinking Social Inquiry: Diverse Tools, Shared Standards. Lanham, MD: Rowman & Littlefield.

Craver, C., and J. Tabery. 2017. “Mechanisms in Science.” In The Stanford Encyclopedia of Philosophy, Spring 2017, edited by E. N. Zalta. Palo Alto, CA: Metaphysics Research Lab, Stanford University.

Greiner, D. J., and C. Wolos Pattanayak. 2012. “Randomized Evaluation in Legal Assistance: What Difference Does Representation (Offer and Actual Use) Make?” Yale Law Journal 121 (8): 2118–2214.

Helbing, D. 2010. Quantitative Sociodynamics: Stochastic Methods and Models of Social Interaction Processes. New York, NY: Springer Science & Business Media.

Ho, D. E., S. Sherman, and P. Wyman. 2018. “Do Checklists Make a Difference? A Natural Experiment from Food Safety Enforcement.” Journal of Empirical Legal Studies 15 (2): 242–277.

Imai, K., L. Keele, and D. Tingley. 2010. “A General Approach to Causal Mediation Analysis.” Psychological Methods 15 (4): 309–334.

Imbens, G. W., and D. B. Rubin. 2015. Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. New York, NY: Cambridge University Press.

Jensen, J., S. Naidu, E. Kaplan, L. Wilse-Samson, D. Gergen, M. Zuckerman, and A. Spirling. 2012. “Political Polarization and the Dynamics of Political Language: Evidence from 130 Years of Partisan Speech.” Brookings Papers on Economic Activity 43 (2): 1–81.

Kleinberg, J., H. Lakkaraju, J. Leskovec, J. Ludwig, and S. Mullainathan. 2018. “Human Decisions and Machine Predictions.” The Quarterly Journal of Economics 133 (1): 237–293.

Kleinberg, J., J. Ludwig, S. Mullainathan, and Z. Obermeyer. 2015. “Prediction Policy Problems.” American Economic Review 105 (5): 491–495.

Klingenstein, S., T. Hitchcock, and S. DeDeo. 2014. “The Civilizing Process in London’s Old Bailey.” Proceedings of the National Academy of Sciences 111 (26): 9419–9424.

Levitt, S. D. 1997. “Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime.” The American Economic Review 87 (3): 270–290.

McFarland, D. A., K. Lewis, and A. Goldberg. 2016. “Sociology in the Era of Big Data: The Ascent of Forensic Social Science.” The American Sociologist 47 (1): 12–35.

Morgan, S. L., and C. Winship. 2014. Counterfactuals and Causal Inference. Cambridge, UK: Cambridge University Press.

Mullainathan, S., and J. Spiess. 2017. “Machine Learning: An Applied Econometric Approach.” Journal of Economic Perspectives 31 (2): 87–106.

Preacher, K. J., and A. F. Hayes. 2008. “Asymptotic and Resampling Strategies for Assessing and Comparing Indirect Effects in Multiple Mediator Models.” Behavior Research Methods 40 (3): 879–891.

Rubinstein, A. 2000. Economics and Language: Five Essays. Cambridge, UK: Cambridge University Press.

Rzhetsky, A., J. G. Foster, I. T. Foster, and J. A. Evans. 2015. “Choosing Experiments to Accelerate Collective Discovery.” Proceedings of the National Academy of Sciences 112 (47): 14569–14574.

Shwed, U., and P. S. Bearman. 2010. “The Temporal Structure of Scientific Consensus Formation.” American Sociological Review 75 (6): 817–840.

Sunstein, C. R., D. Schkade, L. M. Ellman, and A. Sawicki. 2006. Are Judges Political?: An Empirical Analysis of the Federal Judiciary. Washington, DC: Brookings Institution Press.

Vilhena, D. A., J. G. Foster, M. Rosvall, J. D. West, J. Evans, and C. T. Bergstrom. 2014. “Finding Cultural Holes: How Structure and Culture Diverge in Networks of Scholarly Communication.” Sociological Science 1:221–239.

BACK TO Law as Data