Data is not the plural of anecdote: Legal Scholarship Edition

After a long hiatus due to the weekend and Independence Day, I finally found something that piqued my interest to blog about: a debate about the usefulness of legal scholarship in regards to judging and legal practice. Chief Justice Roberts, in the Fourth Circuit Judicial Conference, noted that law review articles are often useless to judges because they focus on arcane philosophical points rather than resolving difficulties in doctrine or exploring important areas of law. In response, law professor Sherrilyn Ifill penned a response, as posted by Danielle Citron at Concurring Opinions, arguing that law review articles do in fact do doctrinal lifting if only judges would read them. For her argument, she points to a 2007 article used in a recent Court of Appeals case regarding 4th Amendment searches and GPS technology. She then goes on to list 4 examples of law review articles dealing with recent doctrinal issues, which she rounds out with the statement:

“Hutchins’ article is hardly an anomaly.  A recent review of articles posted on the Social Services Research Network, revealed a treasure trove of excellent articles that would greatly assist judges in their work.”

Despite these examples however, Professor Ifill fails to make the argument that she thought she has in the conclusion that law review articles are generally useful to judges if they would read them. She makes a total of three mistakes in her argument:

Mistake One: A few data points do not make a general trend. Her argument supposes that these articles are representative of the general population of articles. However, 5 examples are nowhere near the number necessary to even begin making descriptions using statistical methods. In order to get the gravity of the argument, we must first know the entire population of articles. Taking information from the Washington and Lee Law Journal rankings,  they contain 986 U.S.-based English law journals. Already, our five examples are lost adrift a sea of journals as they constitute less than one percent. Some may contend that some journals are better quality than others, but such arguments necessarily delve into what “quality” means, and whether it is related to the concept of “useful to judges.” Taking our numbers a step further, we need to know how many articles each journal publishes. Using George Mason Law Review as a measuring stick, let’s assume that a law review publishes four articles per issue (not including notes as I want to look at professorial output, not student output) and four issues per year (in the end, some journals publish more per issue but less issues, etc. so it balances out). This leaves us with a total of 15,776 articles per year (4*4*986). So, five articles out of 15,776 comes out to 0.00316, definitely not representative of the population of law review articles. In this instance, what little support Professor Ifill presents is definitely insufficient to make her case.

Mistake Two: SSRN != Published != SSRN. (For those who are not familiar, != is “does not equal in Java.”) Four of the examples provided by Professor Ifill are from SSRN, which is a self-listing by professors of their work, including everything from just abstracts to published works. Although I am a huge fan of SSRN (link to my author page is in the About page), SSRN is not sufficient to reach any conclusions about the state of legal scholarship. First, many of the articles listed on SSRN are works in progress or attempts by authors to draw attention to recent scholarship which has yet to be chosen for publication. Second, SSRN does not include every work published in a law review, so in some ways it would tend to undercount some scholarship while overcounting others. Some academics are not on SSRN, while others update their pages sparingly, and some may not list publications on their directly to draw attention to their major works. The double disconnect means that we have trouble drawing the analogy about legal academia from SSRN as it fails to work in either direction.

Mistake Three: Academia as a category is nearly nonsensical. By attempting to argue that the majority of scholarship is useful, Professor Ifill glosses over the individual characteristics and motivations of particular research. Some pieces are not even intended for judicial consumption, and are targeted toward legislators and other policy makers considering a certain course of action. Although I am unfamiliar with the pieces she references, I do have personal experience with a piece recently cited by the 7th Circuit Court of Appeals that was intended for judicial consumption to resolve doctrinal issues. Before I became the analyst for FantasySCOTUS, Josh gave an assignment to all of his potential interns to work on a segment of the Pandora’s Box. From this perspective, I knew that the main reason behind the article was to discuss the impact of potential victory for McDonald v. Chicago, and to chart the course of law after that decision. Like Pandora’s Box, I’m certain that many of the articles referenced by Professor Ifill are the conscious decision of authors to solve a particularly thorny problem, and could be the product of a practically minded author (both Josh and his coauthor Ilya Shapiro are not tenure track professors). The idiosyncratic nature of both articles and authors would tend to increase the threshold amount of data necessary to make the argument that legal academia in general is useful to justices in doctrinal issues.

In conclusion, Professor Ifill failed to make a persuasive case that legal academia is useful. However, I would also argue that not all legal scholarship SHOULD be useful in that sense. We need high ideas, programmatic comparisons, and doctrinal guidance in order to make any system work, and legal academia is no different. Maybe part of my interest is that my first article is mainly an ideas piece about a novel approach to Supreme Court predictions. However, I feel strongly that not enough scholars make the effort to connect their idea pieces to doctrine or legal practice, and that is what judges are picking up on. For example, my first piece is only the beginning of many, and as I move through my career, I am looking for plenty of opportunities to apply the ideas to different situations such as litigation and cert grants. However, an alternative method of making that connect is to engage in blogging and other forms of communication. By taking the time to show how “the effect of Kant on the evidentiary rules of Bulgaria” relates to something occurring in our legal system right now, and might provide an answer to a question plaguing courts and litigants, we advance scholarship into a happy medium of ideas and action.


The Problem of Prognostication today has an article titled “It’s Hard To Make Predictions, Especially About The Future” Basically, the thesis of the article is that “Experts” have a particularly bad track record with making predictions about the future, so much so that a random process (here called a monkey and a dart board) could actually perform better than Expert (meaning that the Expert could be MORE accurate by having a secret dart board monkey). The examples used in the article include 20+ year predictions about the fate of countries, the economy, the environment, etc. All big events that have a huge effect on the entire globe. The author of a book on this subject then uses behaviorial economics to explain why everyone tends to flock to these often erroneous experts.

As I feel that “experts” tend to be puffed up prognosticators who either make heavily hedged non-predictions, or predictions that are extremely difficult to counter or measure, I am inclined to agree with this article. However, a Bloomberg article today about the partisan predictability of the Supreme Court shows that there are areas where we can make predictions, and often do. If we are so bad at predictions, then we should get these small things wrong as well? I think that we can make predictions like Supreme Court outcomes while getting the big things wrong. One major difference between the two is that Supreme Court predictions are about what the outcome of a case will be this term, not a case 20 years from now. The further we get away from the event in time, the less likely we are to get it right as much can happen in between the prediction and the event. Another factor is that the big predictions tend to be about chaotic systems where small changes early on have a significant impact much later. Finally, and this is the major difference between big predictions and predicting the outcome of a case: A Supreme Court case has at most 9 variables (the justices, or 11 if you count the parties), whereas the big predictions have a near limitless number of variables. In this sense, the complex equations of the experts in economics are useless for predictions, but the unquantifiable shiftiness of human nature and bias makes them easier to predict. Finally, Black Swan Theory neatly explains why the end of the world predictions are wrong, while Supreme Court opinions are rather mundane and common in contrast.

However, we can learn more about the predictions from big events by looking at how the predictions of small events work. My work with FantasySCOTUS gives me too major insights into predictive models and markets: the more you embrace and allow uncertainty, the more accurate we are, and that what we call “experts” are not necessarily knowledgeable. In the article I coauthored with Josh Blackman and Adam Aft, we account for uncertainty by finding confidence intervals for each of our predictions, which lets us know how much to discount certain predictions by. By holding back on weak predictions and putting forward strong ones, we’re more or less detected where and when predictions are going to be accurate (although sometimes our users get things significantly wrong). In respect to experts, we tend to find that factors such as credentials (degrees or publications) or work directly in the field are not indicators of strong predictive ability. Sometimes the exceptionally smart non-lawyer does a better job of predicting the outcome of a case than the Constitutional litigator. One possible explanation is that credentials do not select for predictive capacity, so we are trying to fit experts into a role they are not suited for (although some experts are more than willing to oblige).

As for predictive modeling and mathematics, I have to be on guard to not become one of those occultist economist who uses their crystal balls to gaze into the future. Generally speaking, I always stick to basic level descriptive statistics. By limiting myself to these tools, I know better than to try to make the world fit my visions. Instead, I apply statistics as a tool to give expression to the traits of the predictions our many users have made. So far, it has done very well for understanding how people view the court, and how cases are likely to come out. In the old crowd versus experts debate, I don’t see why we can’t crowdsource to find the real experts, not merely those who got a degree and took some classes.

McDonald v. Chicago Re-Redux

Last night, I made a post about the similarities between McDonald v. Chicago and the ACA lawsuits. Earlier today, Randy Barnett, who I mentioned is playing the role of Alan Gura in this drama, posted his thoughts on the recent Sixth Circuit decision. He opens with this paragraph:

“Volokh readers will remember when two widely-respected conservative Court of Appeals judges, Judge Easterbrook and Judge Posner were on a unanimous Seventh Circuit panel denying both the Due Process and Privileges or Immunities challenge to Chicago’s hand-gun ban. One year later, the Due Process challenge was upheld 5–4 in McDonald v. Chicago. My friend and current adversary, Walter Delliger said yesterdaythat the opinion by Judge Jeff Sutton to uphold the individual mandate “is a complete vindication of the constitutionality of the Affordable Care Act.” Not so fast. Sutton’s opinion was no surprise to anyone who was in the courtroom in Cincinnati. Nor would a contrary opinion have been surprising. Sutton was scrupulously critical of both sides that day. Indeed, his opinion shares the “on the one hand” and “on the other hand” character of his questioning. And it also bears some resemblance to Judge Easterbrook’s opinion in McDonald.”

I’m not the only one who sees that the cases are walking down the well-trod yet confusing path of McDonald. Seems like I can now upgrade from navel-gazing to armchair analysis…

McDonald v. Chicago Redux

With the recent Sixth Circuit decision concerning the Affordable Care Act and the resurface of debates held after each decision comes down, I have decided to join in the navel gazing and prognostication by predicting the eventual Supreme Court outcome of the case. For those in the know, I obviously think that the case will end in a controversial 5-4 decision with the standard ideological split much like McDonald v. Chicago did. However, I want to take my analysis a step further, by looking at the various factions who want to overturn the act or limit the Commerce Clause, and guessing the contours of final decision.

First, we look at the major players in this drama as it unfolds:

Alan Gura, who represented the petitioners in McDonald v. Chicago, attempted a novel approach to incorporation by reviving the long dead Privileges or Immunities Clause. In contrast for the ACA cases, Randy Barnett is attempting to press for a novel theory of an activity/inactivity distinction in the Commerce Clause. Both Alan Gura and Randy Barnett are attempting to invigorate Constitutional restrictions by not merely introducing an argument that wins the present case, but also by moving ideas/legal theory into the mainstream.

In an allied contrast, some other participants just want the law overturned, and are not particularly wedded to a certain theory, as long as one wins the current case. In McDonald v. Chicago, this group was represented by the National Rifle Association and Paul Clement. While there was a disagreement about methods and the safety of using certain arguments, the main concern was a particular policy outcome of ensuring gun rights. In the ACA litigations, this group is represented by Ken Cuchinelli, Attorney General of Virginia, whose ultimate goal is to stop ACA. Although things are much better between Barnett and Cuchinelli than between Gura and the NRA, I would not be surprised if things went down the same road if Cuchinelli feels that Barnett’s theory harms the chance of victory.

Of course, there are always some spoilers in every crowd. In McDonald v. Chicago, those spoilers were Ken Blackwell and Ken Klukowski, who were adamantly opposed to idea of reviving the Privileges or Immunities Clause as they were afraid that it would open a Pandora’s Box of new rights and threaten the Constitutional status quo. In the ACA situation, those role is reprised by Orin Kerr, who feels that there is no basis for Barnett’s theory in the Constitution, and that the Act should stand. Of course, it seemed that Professor Kerr and Professor Barnett had practice back during McDonald v. Chicago arguing with each other. At this point in the ACA situation, the two professors are already going head to head before there is a Supreme Court case.

So noting these similarities is just a meaningless exercise in navel gazing, unless they have a substantial impact on the eventual Supreme Court case. One potential remaining question in this situation is which cases will have cert granted, which will in large part determine who gets to argue the case in front of the Justices. If past history is any indication, this could have a large impact on how well the various coalitions opposing the bill operate. Even if relations sour however, the final outcome will be a 5-4 outcome, with 4 Justices (probably lead by Scalia) will vote to overturn ACA, but not use the activity/inactivity distinction, and a concurring justice (probably Thomas) using the distinction to reach the overturning outcome. The four liberal justices will likely write a angry, scathing dissent (Breyer’s pretty good at those). So the final outcome will be a largely disappointing 4-1-4 split with very little guidance for future cases. This is the McDonald outcome, and since this situation is looking like McDonald v. Chicago Redux, the ACA lawsuits will end up in the same position.

Justice Sotomayor is a Conservative?

Adam Liptak, reviewing the Supreme Court’s recently ended term, attempts to promote his “Roberts Court Is Pro-business” theory by looking back at the term, and pointing to the cases where the conservative justices voted en mass for business interests. Liptak provides no legal analysis that the decisions were only possible by way of extreme partisanship, which would have to be more than mere ideological gloss on facts and law to show that the Court was head over heels for corporate interests. Regardless of my disagreement with the premise, a large section stood out to me near the start of the article, as follows:

“The First Amendment dominated the term, with the court ruling for funeral protesters, the makers of violent video gamesdrug marketers and politicians who decline public financing. The American commitment to free expression, the court said, cuts across politics and commerce, requires tolerance of offensive speech and forbids the government from stepping in when powerful voices threaten to dominate public debate.

In cases involving the nation’s largest private employer,Wal-Mart, and its second-largest cellphone company,AT&T Mobility, the court tightened the rules for class actions and made it easier for companies to do away with class actions entirely by using form contracts.

All of the decisions this term were scrutinized for clues about the arc of the Roberts court as it settles into a period of consolidation and awaits a series of colossal cases, notably the challenges to the health care law championed by President Obama. This term was significant, but the next one may include the most important clash between the Supreme Court and a president since the New Deal.

In the just-completed term, the court’s commitment to free speech and its growing business docket sometimes intersected, as in the cases on drug marketing and video games.

“For the conservatives,” said Lee Epstein, a professor of law and political science at Northwestern, “the First Amendment continues to trump other values, especially if they can help business in the process.””

From the way Liptak phrased it, it seems that the partisan Conservatives of the Court are in the pocket of corporate interests when it comes to the First Amendment. Because of the lack of analysis, the casual reader would not know that Snyder v. Phelps was an 8-1 decision, with a dissent by Alito (one of the conservatives) and involved no corporations. Additionally, Sorrell v. IMS was a 6-3 decision, with Sotomayor joining the conservatives in the majority, while Brown v. EMA was a 7-2 decision with Sotomayor and Kagan joining the majority (Kagan may have joined just to insure availability of Mortal Kombat). The last decision mentioned, Free Enterprise Club v. Bennett, was a 5-4 ideological split which would tentatively support Liptak’s thesis. However, in the direct combination of speech and business, only two of the non 5-4 are mentioned.

So to summarize, Liptak and Epstein think that Sotomayor is a conservative (read the linked post for the best in forgotten arguments) for which “the First Amendment continues to trump other values, especially if they can help business in the process.” OR, we could all just not buy into this pro-business shtick.

Tyranny Of The Breyer

Over at Five Books, Justice Stephen J. Breyer takes a turn at listing his five Intellectual Influences in book form. (HT to Todd Zywicki at Volokh Conspiracy for the heads up about the post.)

In the interview, Justice Breyer lists his first book: Democracy In America by Alexis De Tocqueville. After he generally describes it as a masterpiece, he states “Law in America rises from the bottom up, it isn’t decreed from the top down. When we have a new problem, we start with vigorous debate and discussion that can sound like clamour.” So far so good, I agree that is a positive thing, the reason why we have the 1st Amendment to protect it. He then describes the process in the context of privacy:

“Norms about privacy and how privacy relates to free expression are changing. When we want change in an area like that, we start to discuss it – in schools, in associations, in newspapers, in magazine articles. Debate and discussion bubbles up. Some kind of rule is formed, perhaps through an administrative process. We may change it, there may be legislative hearings, our representatives might write a statute and if that isn’t working well, the new rule is tested through the courts to see whether it falls within the boundaries of the constitution.”

Sounds like the normal course of things. After that point, things fall apart with this line: “But in the vast area between the boundaries, democratically elected representatives make decisions, after all sorts of consultation with the people, after all sorts of clamour. Tocqueville encapsulates all that. His work still helps us understand America, 170 years after he wrote it.”

Now, there is no doubt that Justice Breyer believes in the ability of the elected representatives to sort through the various facts, clamour, discussions, and debates. Even in the context of the First Amendment where such events occur, Justice Breyer advocates that the Court defers to the judgement of the Legislatures in general. Two recent examples come to mind, Sorrell v. IMS Health and Brown v. EMA. In his Sorrell dissent, Breyer states that:

“Since ordinary  regulatory programs can affect speech, particularly commercial speech, in myriad ways, to apply a “heightened” First Amendment standard of review  whenever such a program burdens speech would transfer from legislatures to judges the primary power to weigh ends and to choose means, threatening to distort or undermine legitimate legislative objectives.” (Page 5 of the dissent)

And after a discussion of the various research and literature regarding violence and videogames in his Brown v. EMA dissent, Breyer states that:

“Unlike the  majority, I  would find  sufficient grounds in these studies and expert opinions for this Court to defer to an elected legislature’s conclusion that the video games in question are particularly likely to harm children. This Court has always thought it  owed an elected legislature some degree of deference in respect to legislative facts  of this kind,  particularly when they involve technical matters that are beyond  our competence, and even in First Amendment cases.” (Page 16-17 of his dissent)

While such statements and sentiments are normal for Justice Breyer and others who share his judicial philosophy, deference to the legislature shows that Breyer obviously overlooked one of the lasting thoughts expressed by Tocqueville in Democracy in America: the tyranny of the majority. In Book 1, Chapter 15, Tocqueville discusses the major flaw of democracy in that it empowers the majority in such a manner that it will dominate society both politically and morally. In regards to the political aspects, Tocqueville states that:

“Of all political institutions, the legislature is the one that is most easily swayed by the will of the majority. The Americans determined that the members of the legislature should be elected by the people directly, and for a very brief term, in order to subject them, not only to the general convictions, but even to the daily passions, of their constituents. The members of both houses are taken from the same classes in society and nominated in the same manner; so that the movements of the legislative bodies are almost as rapid, and quite as irresistible, as those of a single assembly.

It is to a legislature thus constituted that almost all the authority of the government has been entrusted.”

On the moral aspects:

“IT is in the examination of the exercise of thought in the United States that we clearly perceive how far the power of the majority surpasses all the powers with which we are acquainted in Europe. Thought is an invisible and subtle power that mocks all the efforts of tyranny. At the present time the most absolute monarchs in Europe cannot prevent certain opinions hostile to their authority from circulating in secret through their dominions and even in their courts. It is not so in America; as long as the majority is still undecided, discussion is carried on; but as soon as its decision is irrevocably pronounced, everyone is silent, and the friends as well as the opponents of the measure unite in assenting to its propriety. The reason for this is perfectly clear: no monarch is so absolute as to combine all the powers of society in his own hands and to conquer all opposition, as a majority is able to do, which has the right both of making and of executing the laws.”

In other words, the most destructive thing one can do is rely on majority opinion to determine what actions the legislature should take, which in turn vastly limits the amount of debate while expanding the majority’s power. One would think that listing an Intellectual Influence would meant one actually read the book, but in Breyer’s case, it looks like his jurisprudence reinforces the tyranny of the majority (especially in the context of the First Amendment) rather than counteract it.