Is There a Selection Effect in Administrative Law?
Devan Schafer, PhD candidate, Queen’s Law School
Introduction
In their seminal 1984 article, “The Selection of Disputes for Litigation,” George Priest and Benjamin Klein developed an economic model of civil litigation which led them to conclude that because of selection effects amongst litigants, no inferences could be drawn about the character of the law based merely on success rates at trial.1 While their model has been subject to criticism (principally on the basis that the conclusion that no inferences could be drawn is too strong), even critics concede that their “central insight that litigated cases are a selected, non-random sample of all disputes” marks an important contribution.2
Priest and Klein’s selection model has proven enormously influential in American law and economics scholarship, to the point of dissuading research on outcomes in litigated cases.3 Understandably, most of the research building on Priest and Klein’s work has focused on civil litigation,4 with a few authors extending the insights of the selection model to other areas of law.5 Priest and Klein’s work has received some passing attention from Canadian legal scholars studying civil litigation,6 but it has had no discernible impact on Canadian administrative law scholarship, and little impact on administrative law even in the United States.7
The question this paper seeks to answer is whether and to what extent the selection model might prove useful in the context of administrative law. The argument will be made that while some of the assumptions of the initial selection model are not immediately applicable to administrative law, the insight of the model about the necessity of allowing for litigant choice to play an explanatory role and the corresponding caution about what conclusions can be drawn from applicant success rates needs to be considered in an administrative law context.
This lesson is important because of the way in which success rates, either on initial appeal or judicial review, can be and are used to draw conclusions about the fairness or viability of administrative regimes. If the choices of applicants in selecting cases for review or appeal are ignored, then the explanation for outcomes naturally switches to adjudicators themselves or to institutional design, which in turn can have political consequences. This problem might be seen as especially urgent because government and media discussions of various administrative regimes are often dominated by headlines about appeal success rates.8 In many, if not all of these discussions, the inference which is implicitly being drawn is that the success rate is somehow representative of the larger body of cases.
To take one quick example: a 2017 article reported on the fact that nearly half of denied applicants for Canada Pension Plan (CPP) disability benefits are successful on appeal to the Social Security Tribunal (SST).9 The article quoted an expert who suggested that this was unacceptable, but that a success rate of 10-20% would be understandable.10 No explanation for why the lower rate might be preferable was offered, but it is not difficult to reconstruct the most plausible line of reasoning here. The criticism seems to be premised on the assumption that the success rate on appeal is indicative of an error rate amongst all initial determinations of eligibility, so that something like one half of CPP disability benefit applicants are wrongfully denied. If this inference could be drawn, then that would point to a severely dysfunctional system of initial determinations (one could also imagine how these same statistics could be used to argue that the SST is overly permissive in granting eligibility). In that case, then obviously a much lower success rate on appeal would be preferable, as it would indicate a more accurate system of initial determinations. If there are even partial selection effects in play, however, then we may need to be more cautious about what the success rates on appeal can tell us about the system as a whole, and by extension what a normatively desirable success rate might look like.
If we can expect to see selection effects in administrative law, then a corollary of this is to highlight the necessity of more careful empirical work in the field. There has been no shortage of compelling and thoughtful doctrinal analysis in Canadian administrative law. Academic analyses often centre around a close reading of a handful of cases which are (invariably) taken to illustrate judicial missteps. Highlighting inconsistent or erroneous judicial reasoning is important work, but at the risk of stating the obvious, doctrinal analysis is necessarily limited to analyzing a relatively small number of potential judicial review applications or administrative appeals – namely, those that are filed and pursued to the point where the court or tribunal renders a final decision. If we are interested in how administrative law jurisprudence affects the decisions and behaviour of actors within the larger administrative justice system, however, then doctrinal analysis can only bring us so far. If the effect of the courts’ application of the standard of review analysis, for instance, is to dissuade parties from bringing judicial review applications, or if the law is so uncertain and inconsistently applied so as to encourage speculative applications, then this important information cannot be revealed by even the most careful doctrinal analysis. Priest and Klein made precisely this point, noting that “doctrinal information discloses very little about how legal rules affect the behavior of those subject to them or [about how legal rules] affect the generation of legal disputes themselves.”11 Calls for further empirical work in Canadian administrative law are commonplace, and the animating idea of this paper is that law and economics scholarship may be able to provide guidance as to how structure that research in a more productive manner.
In Part I, I examine the Priest-Klein selection model and some of the criticisms and clarifications of the model which have been offered. In Part II, I address the applicability of their selection model to administrative law, given that the model was developed to explain trial selection in civil litigation. In Part III, I will look at some of the existing empirical work regarding Canadian administrative law and highlight the failure to consider the inference problem highlighted by the selection model. In Part IV, I will examine how consideration of the selection effect might prove useful via a case study regarding controversy over the Ontario Disability Support Program (ODSP) appeal success rates. Part V contains a brief discussion of how the selection effect may help evaluate the success of the Supreme Court’s attempt in Vavilov to enhance the predictability regarding the standard of review.
Part I: The Priest-Klein Selection Model
A. Economic models of litigation
Before describing the model, it needs to be emphasized why the model is useful. There are, after all, competing models of litigation relating to trial selection, and some scholars have taken the criticism of the Priest-Klein model to be so decisive as to render certain parts of their selection model essentially useless.12 It is beyond the scope of this paper to canvas the various different litigation models and to compare their strengths and weaknesses, but a brief comment can be offered regarding why the selection model might be preferable in an administrative law context.
Joel Waldfogel suggests that there are broadly speaking two competing camps regarding litigation selection models: the asymmetric information theories such as that proposed by Lucian Bebchuk,13 and the divergent expectations theories of Priest-Klein and various others.14 While Waldfogel believes that the two approaches can be reconciled,15 others are less sanguine. Advocates of the asymmetric information theories, for instance, have been critical of the divergent expectations theory of Priest-Klein on the grounds that it lacks “proper game-theoretic rigor, because the parties do not take into account the fact that the other side has information about likely trial outcomes and because the parties do not bargain strategically.”16
Without resolving that particular debate, one reason to doubt the applicability of the asymmetric information theories in a judicial review context specifically is that the asymmetric information models are predicated on one of the parties having private information about factual issues that affect the expected outcome at trial.17 This private information allows one party to more accurately estimate the likely outcome of the trial,18 and the asymmetric information models trace out the implications of that assumption on settlement and trial selection.
It is not clear that these models would offer much in Canadian administrative law context, as least in relation to judicial review. The Canadian evidentiary rules relating to judicial review typically preclude the introduction of new evidence that was not before the original decisionmaker, subject to a few minor exceptions.19 While the information asymmetry theories may be instructive in civil litigation and by extension in the context of some first instance tribunal proceedings (where an information asymmetry likely still exists), it is not obvious that these theories are particularly useful where evidentiary rules dictate informational symmetry. None of this is to say that the information asymmetry models are of no use in administrative law, but rather that the Priest-Klein model may have a broader application because it can account for selection effects on judicial review as well as in tribunal proceedings.20
B. The Selection Model
The initial selection model set out by Priest and Klein in 1984 has undergone subsequent clarification and modification by other scholars, but it is necessary to start with the original model.
The selection model is based entirely on economic concerns and, as the name implies, seeks to explain why it is that parties will elect to go to a costly trial rather than settle. The determinants of litigation as against settlement are the expected costs, the information the parties possess about the likelihood of success, and the direct costs of litigation and settlement.21 As Priest and Klein emphasize, the “most important assumption of the model is that potential litigants form rational estimates of the likely decision.”22 For the sake of simplicity, in the original model the issue of damages is ignored – damages are presumed to be stipulated so that there is no disagreement between the parties on that front.23 Additionally, in the simplest form of the model, the parties have symmetrical stakes in the litigation, since all that is at stake for either party is the pecuniary damages award.24
The model revolves around the idea of a “decision standard,” which refers to a consistent standard for resolving legal disputes. Ideally, the decision standard would be the relevant legal standard, but Priest and Klein allow that it may be something more nefarious such as racial prejudice.25 In the model, all filed cases lie along a continuum based on case quality. The continuum is divided at the decision standard, with cases that would be plaintiff victories if tried lying to the right of the decision standard, and those that would result in defendant victories lying to the left of the decision standard.26 Cases which are far to the right of the decision standard are those in which the plaintiff is very likely going to be victorious. Cases which are far to the left of the decision standard would almost certainly result in a defendant victory at trial. Cases which are distributed near the decision standard then are those where it is much less clear which party might be victorious.
Parties form expectations of the likelihood of victory based on the application of the decision standard to the facts of their case. These expectations may diverge because of the parties’ error in estimating case quality, or because of uncertainty about how exactly the decision standard will be applied (the equivalent, in other words, of a vague legal standard).27 Since the model as originally conceived applies to civil litigation, settlement filters out cases where the parties’ expectations of the likelihood of victory are more closely aligned. For example, if the defendant’s behaviour is thought by both parties to fall so far to the right of the decision standard that plaintiff victory at trial is almost assured, then settlement is extremely likely as the parties will agree on the outcome of the litigation and therefore settle and save the costs of going to trial.28 If the parties’ expectations of plaintiff victory vary significantly, then cases will go to trial.29 Thus, while Priest and Klein never use the phrase in the original article, the model has become known as a “divergent expectations” theory of litigation.
In the simplest form of the model, where there is little uncertainty about the decision standard, the cases which go to trial will be clustered around the decision standard, since those are the cases where the expectations of the parties are likely to diverge most significantly precisely because the facts of the case are so close to the decision standard that it is unclear whether the defendant will be found liable.30 Since these are the cases that are most susceptible to being decided either way, then with a certain decision standard, the plaintiff victory rate at trial will be driven towards 50% regardless of how plaintiff or defendant friendly the decision standard actually is.31 Parallel with this will be a decrease in the number of cases tried. Since parties’ ability to estimate the likelihood of plaintiff victory increases with experience of a given legal standard, the move towards 50% plaintiff victory will also involve a decrease in the number of cases taken to trial.32
Priest and Klein acknowledge that the 50% is a limiting case,33 and that the position of the decision standard will have some effect on the rate of plaintiff victories, but maintain that “the proportion of plaintiff victories in litigation will always be closer to 50 percent than that of disputes within the distribution as a whole.”34
C. Explaining Deviations from 50%
Despite Priest and Klein’s own recognition that the 50% success rate was a limit case, the 50% plaintiff success rate hypothesis drew significant attention, generating both theoretical objections35 and empirical36 work demonstrating deviations from this prediction. This led subsequent scholars to attempt to refine the model in order to demonstrate how changing certain assumptions of the original model could account for deviations from the 50% plaintiff win rate.37 Moving away from the relatively parsimonious initial model likely sacrificed its predictive utility, but as Waldfogel and others argued, controlling for other variables allowed the model to retain some of its explanatory power, even if this turned the analysis into “as much art as science.”38 As Kevin Clermont and Theodore Eisenberg explain, the critiques and subsequent revisions to the model suggest that “the case-selection effect is merely a tendency to remove meaning from outcome data, but it may not do so completely.”39 This has been described elsewhere as a “partial selection effect.”40
A number of factors have been identified which might explain deviations from the 50% rule while still allowing for case selection to offer at least a partial explanation for success rates at trial (some of which, in fairness, were identified by Priest and Klein): differential stakes, differential sophistication of the parties, uncertainty regarding the decision standard, a favourable legal standard for one side, high settlement costs relative to litigation costs, high awards, and agency effects have all been posited as explanations for deviations from 50%.41 It is not necessary here to canvas all of the various factors, but examining the possible selection effects of uncertainty and asymmetric stakes serves to illustrate how case selection may retain some explanatory power even without a 50% success rate.
Waldfogel expands on the effect of uncertainty in the Priest-Klein model, suggesting that uncertainty in the decision standard may increase the representativeness of the tried cases, all other things being equal:
If the parties have substantial uncertainty, the trial filter is less restrictive, and the sample of tried cases both is larger and is more nearly a random sample from the population of filed cases. The closer the sample of tried cases is to a random sample of filed cases, the nearer the plaintiff win rate at trial is to the fraction of plaintiff winners in the filed case population, rather than 50 percent.42
Waldfogel concluded from his own study of tort, contract, and intellectual property law win rates that uncertainty could to some extent account for deviations from 50% plaintiff win rates.43
Priest-Klein and others also turned their attention to situations where the parties’ stakes in the litigation are asymmetric, pointing out that something like the potential reputational loss for one party may serve to raise the stakes for that side of the dispute.44 The effect of asymmetric stakes on the formal model is fairly intuitive – in situations where the defendant has more at stake, the model predicts that the likelihood of a case proceeding to trial will decrease as the plaintiff’s likelihood of success increases.45 The result is that “where the stakes are greater to defendants than to plaintiffs, relatively more defendant than plaintiff victories ought to be observed in disputes that are litigated. The results are reversed where the stakes are greater for the plaintiff.”46 Asymmetric stakes has been described as “the most common explanation” for deviations from the 50% hypothesis.47
D. The Contributions of the Selection Model
As with many influential law and economic theories, Priest and Klein’s contribution lies not so much in the specific conclusions of the original model as much as it does in offering an illuminating explanatory framework for how to think about the law. In the case of the selection model, the original model and subsequent modifications offer important insights about how to evaluate litigation rates. As Yoon-Ho Alex Lee and Daniel Klerman point out, while the model is most often associated with the so-called 50% rule, the 1984 article sets out six distinct hypotheses.48 Of primary interest here are two of the six hypotheses. The first is that disputes which proceed to litigation are “neither a random nor a representative sample of the set of all disputes.”49 Following from this is what Lee and Klerman describe as the “no inferences hypothesis”: that because selection effects are so strong, “no inferences can be made about the law or legal decisionmakers from the plaintiff trial win rate.”50
The reason why even a weaker version of this conclusion is so important is that it is otherwise easy to draw conclusions about the state of the law based on the naïve assumption that cases which go to trial offer a representative snapshot of the state of the law. It is easy, in the words of Priest and Klein, to “presume representativeness” of those cases that go to trial and to draw conclusions about the law on this basis.51
There is no shortage of examples of this mistake being made. Waldfogel illustrates how one can fall into this trap by examining discussions of tort reform, noting that critics of American tort law have pointed to plaintiff win rates at trial as evidence of the pro-plaintiff nature of tort law. Waldfogel argues that this approach is misguided as it ignores the possibility that tried cases are unrepresentative of filed cases.52 Priest and Klein rely on an administrative law example in their initial article when they point out that the Supreme Court of the United States has inferred that the disability benefits system is fair because of a near 50% reversal rate of benefit denials.53 We will see in Part III that some Canadian administrative law scholarship has fallen into the trap of presuming representativeness as well.
Opinions differ about the accuracy of other elements of Priest and Klein’s account, but there is little doubt that a key contribution of the selection model is to highlight that in and of themselves, statistics regarding success rates at trial may not be particularly helpful (and can be misleading) in drawing inferences about the wider body of legal disputes.54
Part II: Applicability in the administrative law context
A. Civil litigation versus administrative law
Priest and Klein’s model was developed in the context of civil litigation. It seems fair to question the utility of its application in an administrative law context. The procedural and substantive differences between civil litigation and at least some areas of administrative law might render the model more misleading than helpful in explaining the success rate of applicants. Certain administrative actions are for nonpecuniary damages, making the quantification necessary for formal modelling somewhat difficult. It might also be contended that there is nothing analogous to settlement negotiations in many areas of administrative law, rendering a selection model based on the costs of settlement versus trial of little use. Furthermore, there are remedial differences in the judicial review context. A successful judicial review will in most cases lead to no more than a quashing of the original decision, rather than a damages award. This may temper how we interpret “success” with respect to judicial review. It may also produce a sort of double-selection effect – parties must be seen to have estimates of the probability of victory about both the judicial review itself and any subsequent proceedings on the merits. Administrative decisions which involve issues of policy are more likely to be polycentric than adjudicatory,55 so the simple two-party bargaining of the selection model may have no application in those situations.
Another reason to be wary of the applicability of the Priest-Klein model in administrative law is that it seems to presume relatively sophisticated legal actors. The 50% plaintiff win rate hypothesis rests on parties being able to make reasonably accurate assessments of the decision standard. This is in turn relies on what Priest and Klein describe as the most important assumption of the model: “that potential litigants form rational estimates of the likely decision, whether it is based on applicable legal precedent or judicial or jury bias.”56 In other words, the presumption is that actors are capable of assessing the law and adjudging the strength of their case accordingly. Parallel to the general behavioural critiques of rational choice theory generally57 and the Priest-Klein hypothesis specifically,58 there is reason to doubt that these conditions hold in some areas of administrative law. The work of Simon Halliday and David Cowan regarding the failure of denied housing applicants to pursue even the almost entirely costless option of internal review highlights the danger of this type of presumption in institutional evaluation and design.59 A brief survey of cases involving self-represented applicants for judicial review at the Federal Court provides further evidence of this issue. When applicants base their application on patently inadmissible evidence,60 or on a clear misunderstanding of the jurisdictional issues at play,61 then we have little reason to believe that these cases were selected for review on the basis of anything resembling the rational decision-making assumed by the original model.62
B. Why the selection model may still be useful
These caveats suggest that the Priest and Klein model cannot be imported wholesale into administrative law. It would be too strong to conclude though that the lessons from the selection model have no application in an administrative context. To recur to an earlier point: the strength of the selection model and its subsequent modifications lies in how they force us to think rigorously about what factors might explain which cases go to trial. As Joseph Feller argues, it is possible to consider the broader application of the qualitative features of the model even if the original quantitative predictions are of little use.63 There are several reasons then why the selection model can still be useful in administrative law.
The first reason is that Priest and Klein do not appear to have envisaged their model as being limited to first instance civil litigation. They explicitly extend the model to appellate litigation, with the only difference being that the stakes on appeal may be more likely to be asymmetrical given that a potentially authoritative precedent may be generated on appeal.64 If the lessons of the model can be applied to appellate litigation, there is no obvious reason why it cannot be used in a judicial review context or with respect to administrative appeals.
The second reason relates to the objection that administrative regimes may not practically allow for settlement discussions because they potentially involve zero-sum legal disputes. An example would be a benefits claim – the claim is either granted or it is not, so there does not appear to be any appreciable range of settlement options. The weighing of settlement against trial may therefore be much different than in civil litigation, where there is typically a much wider range of potential compromises.
There are two responses to this. The first is that settlements do regularly take place in administrative law. Vardy Bondy and Maurice Sunkin’s study of judicial review proceedings in England and Wales found that 56% of judicial review cases settled before proceeding to a hearing (and they estimated that a substantial number of potential claims were informally resolved before litigation).65 There is little reason to believe that this situation is unique to England and Wales.
The second response is that it is possible to conceptualize administrative proceedings as containing steps analogous to settlement negotiations. The work of Andrew Hanssen and Joseph Feller is instructive here. Hanssen tested the Priest-Klein model by examining three different data sets in his study of appointed versus elected judges. One of the data sets was appeals of the initial decisions of state public utility commissions. In explaining why the model was still applicable in this context, Hanssen argues that the initial administrative hearing is analogous to a failed negotiation in civil litigation:
Although the model is perhaps most intuitively applied to trial court litigation (this provides Priest and Klein’s initial illustration) it is equally relevant to appeals—a lower court decision is appealed, just as a dispute is litigated, only when the parties differ sufficiently in their expectations as regards the outcome. When the appeal is of a ruling by a state’s public utility commission (as in the first of the three data sets), a preliminary process has already occurred in which the dispute was aired at a public hearing before the commission. This does not make the model any less applicable—such hearings are in many ways analogous to private negotiations between disputing parties: urgency is signaled, the disputed points are presented and discussed, and the opponents attempt to reach a satisfactory compromise. A dissatisfied party has the right to appeal the commission’s ruling to the state court, just as an individual left unsatisfied by private negotiations has the right to sue. An appeal is presumably only launched when disputants disagree as to its likely outcome, just as the model suggests.66
Feller makes a similar point in his application of the selection model to litigation against federal environmental agencies in the United States. Feller argues that an agency decision might be seen as a take-it-or-leave-it settlement offer, where to decide not to appeal or review the decision could be seen as “acceptance” of the offer.67 Feller acknowledges that the analogy is imperfect,68 but there is enough similarity to suggest that the broader lessons of the selection model may still be applicable in an administrative law context. The central lesson regarding the unrepresentativeness of litigated cases has been applied in circumstances as diverse as employment discrimination,69 disability accommodations,70 and criminal cases,71 amongst others. Administrative law does not seem so obviously distinctive that the same insight would have no application there.
Part III: Presumptions of representativeness in Canadian administrative law scholarship
It was important to argue for the applicability of the selection model in administrative law in large part because there is no evidence that Canadian administrative law scholars have considered the implications of it. There is a dearth of empirical work in Canadian administrative law scholarship, but where it does exist, success rates have been invoked with seemingly little thought as to what the limitations of that information might be.72
With the possible exception of immigration law,73 studies of various administrative regimes are sporadic, if they exist at all. Part of this may be explicable by the fact that tribunals and ministries often undertake to publish basic statistics as part of their own annual reports,74 but even within the relatively small body of scholarship that exists, it seems clear that the lessons of the Priest-Klein model have been ignored. A comprehensive survey of existing empirical work is beyond the scope of this paper, but the following examples demonstrate how a failure to account for the potentially unrepresentative nature of success rates can lead to two erroneous conclusions: the first about the utility of judicial review generally, and the second regarding temporal trends in judicial review.
An example of the first error comes from a paper which predates Priest and Klein’s 1984 article, but which exemplifies how judicial review success rates can be used as evidence impugning judicial review as an institution. In a 1974 article subtitled “Judicial Review – Do We Need It?”, W.H. Angus noted that three studies had found success rates of between 32% and 44% on judicial review applications.75 These (to Angus’ mind) low success rates were marshalled as part of a larger broadside against the utility of judicial review. The concern with this type of reasoning is that the selection model suggests that without further analysis these success rates tell us very little of interest, and consequently cannot be seen to either undermine or support the use of judicial review.
A study of judicial reviews of Ontario Labour Relations Board and labour arbitration decisions before and after the Supreme Court’s consolidation of the standards of review in Dunsmuir v New Brunswick76 illustrates the second pitfall of seeking to demonstrate trends without accounting for the selection effect.77 In finding that applicants were increasingly successful post-Dunsmuir in having labour board and labour arbitration decisions overturned, Luba Yurchak concluded that Dunsmuir had not had the desired effect of increasing judicial deference.78 As Feller points out though, studies of the success rates of court challenges to administrative agencies seeking to demonstrate temporal trends are “at best suspect and at worst meaningless” because many of them suffer from the implicit assumption that cases that are presented to the courts for decision “represent an unbiased sample of federal administrative agency actions.”79 Since Yurchak fails to account for any selection effect, the resulting analysis places the responsibility for the shift in success rate entirely on judicial (mis)interpretations of Dunsmuir. That of course may be a factor, but it may also be explicable in part on the basis of a change in litigant behaviour post-Dunsmuir.
Part IV – ODSP Case Study
In the final section of the paper, I want to illustrate why it is that this has more than mere academic importance by looking at how success rates in ODSP appeals have been interpreted and utilized to criticize and potentially reformulate the program. The ODSP system is straightforward: the initial application is made entirely on paper, and an initial determination of eligibility is made by the Disability Adjudication Unit (DAU).80 If the DAU denies the application, then the applicant must request an internal review before they can appeal the decision to the Social Benefits Tribunal (SBT), where the applicant receives a de novo, in-person hearing.81 The SBT can make a finding that the applicant is disabled within the meaning of the statute.82 Notably, in contrast to the CPP, unsuccessful ODSP applicants can reapply and obtain a new determination of disability based on updated medical evidence, or simply by fixing mistakes which cost them on their initial application.
There is a longstanding perception that the SBT is too generous in granting a finding of disability on appeal. A 2004 report by Ontario’s auditor general was critical of the SBT based on what he perceived to be an indefensibly high success rate of 80% by applicants on appeal.83 In response, an advocacy group defended the ODSP, but what is telling is how they responded to the issue of the high appeal success rates:
There is, at least in theory, more than one possible explanation for the inordinately high turnover rate on appeals: the SBT may be doing something wrong, or the DAU may be doing something wrong, or both the SBT and the DAU may each be doing something wrong; or there may be flaws in the legislative design of the eligibility criteria that are driving the disparities of view between the DAU and the SBT.84
The advocacy group, in other words, could only envisage that the high success rate on appeal was due to decision-making errors or flawed eligibility criteria. Nowhere in either the criticism or the response is there any recognition that the success rate on appeal may reflect a partial selection effect. Like Waldfogel’s example of conclusions being drawn about the state of American tort law based on plaintiff win rates, in this case both critics and defenders of the ODSP system have seemingly presumed that the success rates are representative.
This perceived problem has not gone away – recent reporting has the success rate on appeal at around 75%,85 and a more recent auditor general report about Legal Aid in Ontario was critical of the amount spent by community legal clinics on ODSP applications and appeals.86 Ontario’s Minister of Children, Community and Social Services seemingly took this as support for the provincial government’s decision to shift the ODSP definition of disability to align with the narrower federal definition.87
It is plausible though that there may be a selection effect in play here. Recall the earlier analogy made by both Hanssen and Feller between administrative proceedings and settlement negotiations. If we conceptualize the initial denial of disability as effectively an offer by the government, then the applicant has three options: they can either “accept” the offer and abandon their attempt at obtaining the benefit, they can request a hearing before the SBT, or they can decide to reapply (which might be seen as the equivalent of a counteroffer).
Being mindful of the earlier caveat that explaining success rates is as much art as science, one partial explanation for the high success rate might be the asymmetric stakes between the parties. Since ODSP contains a means test to rule out those who are above a certain income and asset level,88 the benefit will typically represent a significant increase in household income for the applicant. An applicant who decides to pursue a hearing does not face any formal costs consequences, but there is an opportunity cost in that they forego pursuing a potentially stronger reapplication. Pursuing a marginal case to a hearing, given the relative financial importance of the benefit, may in many cases be a less attractive option than simply reapplying. From the government perspective, an individual award will represent a relatively small proportion of the relevant government budget, and since SBT decisions have little precedential value,89 the government need not be concerned with a particularly damaging precedent being produced at any individual SBT hearing. The individual ODSP applicant could very plausibly be seen as having much more at stake then than the government.
Since the effect of asymmetrical stakes is to increase the success rate in favour of the party who has more at stake, the 80% success rate on appeal may not be as egregious as its critics contend. We can return here to the Priest-Klein conception of case quality as lying on a continuum, with weaker applicant cases falling far to the left of the decision standard and stronger applicant cases falling far to the right. Applicants with exceptionally weak cases are those that fall far to the left and may for the most part “accept” the “offer” of the initial denial. Individuals with somewhat stronger cases that fall closer to the decision standard may decide to engage in further “negotiations” by reapplying now rather than risking an adverse decision at a hearing and a consequent delay in getting an opportunity to make a stronger application. Individuals with the strongest post-denial cases will be the most likely to proceed to a hearing, and the high success rate might be reflective then in part of the fact that the strongest cases are being selected for hearings because of the asymmetrical stakes between the parties.
The above explanation is admittedly somewhat crude and would ideally be combined with qualitative evidence, but it offers an account of how a partial selection effect might be in play. This type of explanation does not rule out that the decision standard is favourable to applicants, or that there might potentially be erroneous decision-making either at the DAU level or at the SBT. What it does do, perhaps most importantly, is to widen the field of potential explanations by removing the presumption that appealed cases are representative. Once that is acknowledged, then it becomes possible to move away from the “decision-making error” versus “flawed eligibility criteria” dichotomy which the discussion about success rates almost inevitably leads to if the selection effect is ignored.
Part V– Canada (Minister of Citizenship and Immigration) v Vavilov
One final brief comment will be made here regarding how recognition of the selection effect may be useful in evaluating the Supreme Court’s latest reconfiguration of the standard of review analysis laid out in Vavilov.90 One of the Court’s stated goals in establishing a presumption of reasonableness was to “bring greater coherence and predictability” to the law.91 One way of assessing this, of course, will be a more qualitative doctrinal analysis of published decisions. Another way of assessing this though would be to measure if there is any discernible change in the success rate of applicants within a given administrative regime post-Vavilov (it would probably be too coarse-grained an analysis to measure success rates amongst judicial review cases generally). If Vavilov does provide more predictability to the law, then the Priest-Klein model would suggest that the success rate post-Vavilov would eventually move closer to 50%, all other things being equal, as this would signify an increasingly certain decision standard. This analysis would ideally be conducted by individuals familiar with the administrative regime and therefore sensitive to other factors which may affect the findings. A selection effect analysis conducted without this internal understanding of the regime may offer an explanation which obscures more than it explains. The criticisms of the Priest-Klein model also caution against drawing any categorical conclusions on the basis of this type of quantitative analysis, but it would at the very least offer a means of evaluating the impact of Vavilov which could complement a more traditional doctrinal analysis.
Conclusion
For all its predictive failures, the Priest-Klein selection model still marks an invaluable contribution to legal scholarship in highlighting that trial success rates may not be representative of the larger body of legal disputes. Despite its influence in general, it has had little impact in administrative law scholarship, even in the United States, and certainly not in Canada. This may be because it seems at first glance to be largely inapplicable in an administrative law context. There is some truth to this, but the contributions of the model need not be rejected wholesale. The central insight about selection effects leading to non-representativeness amongst decided cases and some of the more qualitative features of the model could be of use in administrative law, in particular in leading to more nuanced and careful public and academic discussions about success rates. As we have seen in the discussion about ODSP success rates, ignoring the possibility of a selection effect may lead to misguided diagnoses with potentially serious policy consequences.
Endnotes
1 George L Priest & Benjamin Klein, “The Selection of Disputes for Litigation” (1984) 13:1 J Legal Stud 1.
2 Daniel Klerman & YHA Lee, “Inferences from Litigated Cases” (2014) 43 J Legal Stud 209 at 214 [Klerman & Lee, “Inferences”].
4 See e.g., Samuel R Gross & Kent D Syverud, “Getting to No: A Study of Settlement Negotiations and the Selection of Cases for Trial” (1991-1992) 90 Mich L Rev 319.
5 See e.g., Peter Siegelman & John J Donohue III, “The Selection of Employment Discrimination Disputes for Litigation: Using Business Cycle Effects to Test the Priest-Klein Hypothesis” (1995) 24 J Legal Stud 427; Margo Schlanger, “Inmate Litigation” (2003) 116:6 Harv L Rev 1555.
6 Hilary Young, “The Canadian Defamation Action: An Empirical Study” (2017) 95 Can Bar Rev 591 at 606-607; Julie Macfarlane, “Why Do People Settle?” (2000-2001) 46 McGill LJ 663 at 668.
7 Joseph M Feller, “Have Judges Gone Wild – Plaintiffs’ Choices and Success Rates in Litigation against Federal Administrative Agencies” (2014) 44:2 Envtl Law 287 at 301.
11 Priest & Klein,
supra note 1.
12 See e.g., David Zaring, “Reasonable Agencies” (2010) 96:1 Va L Rev 135 at 176, n 133.
13 Lucian A Bebchuk, “Litigation and Settlement under Imperfect Information” (1984) 15 RAND J Econ 404.
14 Joel Waldfogel, “Reconciling Asymmetric Information and Divergent Expectations Theories of Litigation” (1998) 41 JL & Econ 451.
16 Klerman & Lee, “Inferences”,
supra note 2.
17 See e.g., Bebchuk,
supra note 13 at 406; Keith N Hylton, “Asymmetric Information and the Selection of Disputes for Litigation” (1993) 22 J Legal Stud 187 at 188-189.
18 Hylton,
supra note 17 at 189.
19 Association of Universities and Colleges of Canada v Canadian Copyright Licensing Agency (Access Copyright), 2012 FCA 22 at paras 14-20.
20 If one finds Waldfogel’s attempt to reconcile the two approaches to be convincing, then there may be no need to choose between the two.
21 Priest & Klein,
supra note 1 at 4.
26 Joel Waldfogel, “The Selection Hypothesis and the Relationship between Trial and Plaintiff Victory” (1995) 103:2 Journal of Political Economy 229 at 232 [Waldfogel, “Selection”].
28 F Andrew Hanssen, “The Effect of Judicial Institutions on Uncertainty and the Rate of Litigation: The Election Versus Appointment of State Judges” (1999) 28 J Leg Stud 205 at 207.
29 Waldfogel, “Selection”,
supra note 26 at 233.
30 Priest & Klein,
supra note 1 at 15-16.
31 Waldfogel, “Selection”,
supra note 26 at 233.
32 Priest & Klein,
supra note 1 at 19.
33 Priest & Klein,
supra note 1 at 20.
35 Steven Shavell, “Any Frequency of Plaintiff Victory at Trial is Possible” (1996) 25 J Legal Stud 493.
36 Daniel P Kessler, Thomas Meites, and Geoffrey P Miller, “Explaining Deviations from the Fifty Percent Rule: A Multimodal Approach to the Selection of Cases for Litigation” (1996) 25 J Leg Stud 233.
38 Kevin M Clermont & Theodore Eisenberg, “Do Case Outcomes Really Reveal Anything About the Legal System? Win Rates and Removal Jurisdiction” (1997-1998) 83:3 Cornell L Rev 581 at 592.
40 Klerman & Lee, “Inferences”,
supra note 2 at 211.
41 Kessler et al,
supra note 36 at 237-248; Clermont & Eisenberg,
supra note 38 at 589-592; Hanssen,
supra note 28.
42 Waldfogel, “Selection”,
supra note 26 at 233.
44 Priest & Klein,
supra note 1 at 24-25.
45 Feller,
supra note 7 at 304.
46 Priest & Klein,
supra note 1 at 25.
47 Clermont & Eisenberg,
supra note 38 at 589.
48 YHA Lee & Daniel Klerman, “The Priest-Klein hypothesis: Proofs and generality” (2016) 48 Int’l Rev L & Econ 59 at 60.
49 Priest & Klein,
supra note 1 at 4.
50 Lee & Klerman, “Inferences”,
supra note 2.
51 Priest & Klein,
supra note 1 at 3.
52 Waldfogel, “Selection”,
supra note 26 at 231.
53 Priest & Klein,
supra note 1 at 4.
54 Clermont & Eisenberg,
supra note 38 at 588.
55 Lon L Fuller, “The Forms and Limits of Adjudication” (1978) 92:2 Harv Law Rev 353.
56 Priest & Klein,
supra note 1 at 4.
57 See e.g., Russell B Korobkin & Thomas S Ulen, “Law and Behavioral Science: Removing the Rationality Assumption from Law and Economics” (2000) 88 Cal L Rev 1051.
58 Linda Babcock
et al, “Biased Judgments of Fairness in Bargaining” (1995) 85:5 The American Economic Review 1337.
59 Simon Halliday & David Cowan,
The Appeal of Internal Review: Law, Administrative Justice and the (non-) Emergence of Disputes, (Oxford, Hart Publishing: 2003).
60 See e.g.,
Greeley v Canada (Attorney General), 2019 FC 1493.
61 See e.g.,
Lee v Canada (Attorney General), 2019 FC 1189.
62 This is without even considering that reported decisions may at times obscure the work of judges and even opposing counsel of effectively “translating” applications into cognizable legal claims.
63 Feller,
supra note 7 at 302.
64 Priest & Klein,
supra note 1 at 52.
65 Varda Bondy & Maurice Sunkin, “The Dynamics of Judicial Review Litigation: The resolution of public law challenges before final hearing” (2009)
The Public Law Project, at 33, online: Public Law Project.
66 Hanssen,
supra note 28 at 210.
67 Feller,
supra note 7 at 306-307; For another application of the Priest-Klein model to administrative action, see Linda Cohen & Matthew Spitzer, “Judicial Deference to Agency Action: A Rational Choice Theory and an Empirical Test” (1995-1996) 69 S Cal L Rev 431 at 460.
68 Feller,
supra note 7 at 307-309.
69 Siegelman & Donohue,
supra note 5.
70 Michael Ashley Stein, “The Law and Economics of Disability Accommodations” (2003-2004) 53 Duke LJ 79 at 163-165.
71 Harold W Elder, “Trials and Settlements in the Criminal Courts: An Empirical Analysis of Dispositions and Sentencing” (1989) 18 J Legal Stud 191 at 192.
72 But see Paul C Weiler, "The Slippery Slope of Judicial Intervention: The Supreme Court and Canadian Labour Relations 1950-1970" (1971) 9:1 Osgoode Hall LJ 1 (recognizing that Supreme Court of Canada labour relation decisions, which were split evenly between pro-union and pro-management, might be unrepresentative at 12).
73 See e.g. Jon B Gould, Colleen Sheppard & Johannes Wheeldon, “A Refugee from Justice? Disparate Treatment in the Federal Court of Canada” (2010) 32:4 Law & Pol’y 454; Sean Rehaag, “Judicial Review of Refugee Determinations: The Luck of the Draw?” (2012) 38:1 Queen’s LJ 1; Sean Rehaag, “Do Women Refugee Judges Really Make a Difference? An Empirical Analysis of Gender and Outcomes in Canadian Refugee Determinations” (2011) 23:2 CJWL 627.
74 See e.g., Canada, Social Security Tribunal,
Achievements Report 2016-2017 (Ottawa: Ministry of Families, Children, and Social Development, 2018); Ontario Labour Relations Board,
Annual Report 2018-2019 (Toronto: Ministry of Labour, 2019); Ontario, Workplace Safety and Appeals Tribunal,
WSIAT Annual Report 2018 (Toronto: Ministry of Labour, 2019).
75 WH Angus, “The Individual and the Bureaucracy: Judicial Review—Do We Need It” (1974) 20:2 McGill LJ 177 at 180-181.
76 Dunsmuir v New Brunswick, 2008 SCC 9.
77 Luba Yurchak, “Judicial Review of Labour Relations Board and Labour Arbitration Decisions in the Post-Dunsmuir Period in Ontario” (2017) 20:2 CLELJ 447.
79 Feller,
supra note 7 at 290.
87 Paling,
supra note 85.
88 Ministry of Children, Community and Social Services,
supra note 80.
89 1508-07565 (Re), 2017 ONSBT 915 at para 15.
90 Canada (Minister of Citizenship and Immigration) v Vavilov, 2019 SCC 65.