How representative is it really? A correspondence on sortition

Kyle Redman
In my work designing and operating deliberative mini-publics (DMPs) around the world, I’ve made some of the claims to representativeness you discuss in your blogpost. I thought it would be good to discuss!

It seems you want to put the brakes on some claims people have been making about just how representative their DMPs have been. Your key argument is that these processes are not representative, due to small sample sizes, non-response biases, and poor intersectional representation.

Let’s start with small sample sizes. One aspect you discuss is just how representative the output of the process can be. Typically, a DMP might have between 30-100 members. But, you say that they require at least 370, if not more, based on the margin of error calculation for the output of the process (how confident are we that the output reflects hypothetical work of the whole population).

This calculation assumes that there is 50% support for a proposal (we are unsure of the level of support so we assume the most marginal position 50-50). However, in my work DMPs rely on a minimum threshold of 80% support for a proposal. This features in many best practice guides including the OECD’s Catching the Deliberative Wave Report, and the United Nation’s Democracy Beyond Elections Handbook, amongst others. This, in combination with stratification of the sample, changes the margin of error calculation.

It’s critical that we consider the representativeness of the outputs of the process.

This is critical because we’re aiming to find common ground between participants and it reduces concerns that highly variable groups due to small sample sizes might dramatically alter the outcome which should draw into question the representativeness of the outputs.

Paolo Spada and Tiago Peixoto
The idea of requiring a supermajority for decision-making in a DMP to compensate for the small size is intriguing and we have not come across the link between decision-making supermajority and the representativeness claims of the process.

This raises three important concerns from our perspective. First, we know that prioritizing consensus might stifle critical dialogue, favoring shallow agreements and status-quo-friendly results. Second, we would need to assess supermajority impact on participant engagement, particularly minority groups or those with low political efficacy. Third, and most importantly, if we interpret the assembly as a treatment, and we expect people to change opinion due to a mix of group composition and deliberation, then to make claims of representativeness we need to repeat the same assembly many times with different samples and take an average, or we need to make the initial group as representative as possible using a larger sample. A DMP is not the same as a survey of static preferences, since the participants have dynamic preferences that can change, and the legitimacy of the assembly cannot be derived from the voting mechanism because the composition of the group changes the way they will vote.

Kyle
In your blog, you also raised the issue of acceptance rates. Overall, invitation acceptance rates average about 5%, which makes claims of representativeness difficult to sustain. I would like to see more research on who isn’t accepting invitations and what we can do to reach them or fairly weigh their representation in DMPs.

There must be a Goldilocks level of representation that is in balance with the deliberative qualities of the process.

To this end, I’ve been encouraging more generous compensation for participants as well as covering additional costs like transport and care needs. Our recent proposal for a Citizens’ Assembly on Housing Affordability in Australia pays people $3,000AUD for eight weekend meetings.

Another key motivation for people accepting invitations is confidence that their involvement will actually impact decision-making. This can only really be achieved through improving the impact track record of DMPs in general. We know that in Ireland, invitation acceptance rates are above 10%, which I think is a good example of this in practice.

Paolo and Tiago
There is no simple solution to addressing acceptance rates and we agree that a better understanding of non-acceptance is needed. Typically, an acceptance rate below 10% warrants scrutiny to comprehend the unique attributes of participants and non-participants. Collaborative efforts by the deliberative democracy community could help fill this gap in our knowledge by studying the reasons for not participating across different contexts.

Cases like the Brexit Citizens’ Assembly 50% and 88% acceptance rates in the Deliberative Poll on federalism in Japan also highlight that higher acceptance rates can be achieved. This possibility is supported by a wealth of transferable strategies investigated in the ever expanding literature on voter mobilization, organizing and engagement.

We agree that impact and meaningfulness are important drivers of engagement, however, a long line of research explains there is not a linear positive correlation between democratic innovations’ institutionalization and impact, nor between frequency of DI’s adoption and impact. So the idea that institutionalization and replication will solve the problem of minimal acceptance rate is a bit too optimistic in our opinion.

Kyle
Yes, I would like to see more discussion around acceptance rates, and wider use of the supermajority threshold to minimize weak claims of representative outputs.

Finally, I wanted to address your points on stratification, intersectionality, and inclusion. When you discuss intersectional stratification, I think you skim over the role of probability in sampling. You say that “the more complex the population is, the larger the sample required to represent it.” I would say it depends on what you mean by “represent it”!

If we use probabilistic sampling to support the claim that the group’s work will reflect the hypothetical work of the wider population – then we do not require perfect intersectionality. This is why sample sizes do not change much beyond a population of 20,000.

If we desire deep intersectionality because of a high threshold for descriptive representativeness in a political sense – then we are making a different claim, which I think is subject to the relevant political community’s criteria for legitimacy.

I don’t think we want intersectional representation to that degree because it would require the participation of such a large number of people that it would be practically impossible – we have to draw a line somewhere. There must be instead a Goldilocks level of representation that is in balance with the deliberative qualities of the process.

Paolo and Tiago
Stratification, using mutually exclusive strata, reduces error margins, aiding sample size reduction while keeping errors constant. All the nice properties of stratification are derived when the strata are mutually exclusive. In other words we also stratify for the intersection of strata (e.g. we stratify for 4 mutually exclusive groups: young male, young female, old male and old female). However, stratification using a basic list of mutually exclusive strata demands over 100 people. The principle of “the more complex the population, the larger the required sample” applies specifically to this mutually exclusive strata example.

In practice, some use non-mutually exclusive strata (e.g., representing proportions of men, women, young, and old, without representing their intersections), a strategy that can be fitted in smaller samples. This approach lacks a clear advantage in reducing estimation error or justifying smaller samples while maintaining error levels. Moreover, it will misrepresent intersections when the sample is too small to allow for mutually exclusive stratification.

Stratification, in any form, cannot justify samples of 70 or fewer individuals. The justification for these small samples needs to be political.

Our overarching view is that stratification, in any form, cannot justify samples of 70 or fewer individuals. The justification for these small samples needs to be political and has to emerge from community priorities and possible alternative strategies. For example, small sampling strategies could be justified when they address a local or historical failure of inclusion of certain groups (e.g. we fail to engage youth and thus we add youth as a priority strata to engage).

Kyle
What this ultimately comes down to is balancing sample size, precision, and practical feasibility, but I think it’s also important that we consider representation as a feature of the entire deliberative process including in expert, stakeholder, advocacy, and community group representations through to wider community engagement.

It’s important that we consider representation as a feature of the entire deliberative process.

Another favorite of mine is ensuring that the assembly has the opportunity to request additional information and speakers of their own choosing. Assemblies regularly seek to include voices that might not be immediately present, as well as rounding out the diversity of information in the room.

Paolo and Tiago
We want to provoke a conversation about using sortition as a legitimate technology to exclude people, avoiding interest group capture, and to concentrate resources to maximize engagement of people who traditionally do not participate.

This is not a call to abandon mini-publics, but rather a call to remove their technocratic solutionist aura.

Ultimately we suggest abandoning the solutionist arguments about generic representativity of all. The sampling strategy of a mini-public is a political decision that needs to be justified politically and cannot be justified in a technocratic ‘neutral’ fashion. This is not a call to abandon mini-publics, but rather a call to remove their technocratic solutionist aura.

In practical terms, this means that we cannot simply assume a technical ‘automatic’ advantage in terms of inclusion and representativeness of DMP engagement strategies over non-sortition-based strategies. Instead, we must evaluate the most suitable approach for each situation and be prepared to politically justify our choice of the most effective tool for the task at hand.

Kyle
Sure, there is a time and place for expensive DMPs (with more participants) but to me, the combination of random selection and stratified sampling does mean that DMPs are more representative than traditional forms of engagement, though they do have weaknesses that need supporting through additional inclusive strategies (which do happen). This means we can capture the public trust benefits of empowering and representative processes whilst ensuring that they’re open and inclusive.

Overall, I think we agree that it is hard to justify strong claims of representativeness. But, by acknowledging the practical constraints of deliberation, resourcing and incentives, we can say that these processes are broadly representative – just not statistically representative.

About the Authors