Title: Evaluating the Effects of Court-Technology Innovations
Speaker: JJ Prescott
Abstract: Over the last five to ten years, state courts in the U.S. have begun to develop and implement online dispute resolution (ODR) platforms, communication technology that essentially allows individuals to “go to court” online. Commentators have suggested that these systems have the potential to enhance access to justice, fairness in process, accuracy in outcomes, and efficiency in court operation. With the right data, and sometimes the right implementation schedule, it is possible to use traditional program evaluation methods (as well as experiments) to identify the effects of these innovations on a range of important outcomes. In this talk, I will describe the straightforward techniques I have used in projects studying one ODR system in particular (Matterhorn) across a range of case contexts: traffic, warrants, and small claims. The same approaches can be used to study technology’s effects on racial and other disparities. ODR describes a wide-range of online applications, and it is important to recognize the external validity problem in drawing inferences from my Matterhorn-related findings to the likely consequences of ODR applications in general. Nevertheless, the basic empirical methods available to study technology implementation in courts are valid and general—and they ought to be a part of the legal technology toolkit.

Title: Unexpected Challenges on the Road to Quantitative Evaluation of Legal Technology
Speaker: Maura Grossman
Abstract: Professor Grossman will share some of the surprising trials and tribulations she has faced during more than a decade of efforts aimed at the quantitative empirical evaluation of legal technology and services, specifically, in the area of technology-assisted review (“TAR”) in electronic discovery.  Starting with Professor Grossman’s seminal paper, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, published in the Richmond Journal of Law and Technology in 2011, to her 2017 peer-reviewed scientific study, Navigating Imprecision in Relevance Assessments on the Road to Total Recall: Roger and Me, published in the Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval—confirming the 2011 results under more rigorous conditions—and even since then, she has nevertheless encountered skepticism, criticism, and even downright hostility towards quantitative empirical evaluation of legal products and services.  What accounts for this, and what, if anything can be done about it?

Title: Administrative Law and the Future of an Automated State
Speaker: Cary Coglianese
Abstract: In the future, administrative agencies will rely increasingly on digital automation powered by machine learning algorithms. Can U.S. administrative law accommodate such a future? Not only might a highly automated state readily meet longstanding administrative law principles, but the responsible use of machine learning algorithms might perform even better than the status quo in terms of fulfilling administrative law’s core values of expert decision-making and democratic accountability. Algorithmic governance clearly promises more accurate, data-driven decisions. Moreover, due to their mathematical properties, algorithms might well prove to be more faithful agents of democratic institutions. Yet even if an automated state were smarter and more accountable, it might risk being less empathic. Although the degree of empathy in existing human-driven bureaucracies should not be overstated, a large-scale shift to government by algorithm will pose a new challenge for administrative law: ensuring that an automated state is also an empathic one.

Title: Accountability in Computing
Speaker: Joan Feigenbaum
Abstract: Accountability is a widely studied but amorphous concept, used to mean different things across different disciplines and domains of application. In this talk, I will survey some of the ways in which the term “accountability” has been used in computer science, the concepts that play key roles in defining it, and open questions about the design and implementation of accountable systems. I will cover both purely technical considerations, in which users of “accountable systems” are held accountable for following the system policy, and socio-technical considerations, in which designers and developers of “accountable systems” must demonstrate that their systems conform to legal requirements.