Mischief managed: the bias in Defra’s game shooting review

Dr Patrick O’Reilly, BASC’s head of social science, casts a critical eye over Defra’s approach to its call for evidence on game shooting.

Defra’s call for evidence on game shooting reminds me of the Marauder’s Map in the Harry Potter universe. This magical document lets its owner get up to no good, after which the phrase “mischief managed” would erase any traces of bad behaviour.

In a similar way, Defra’s Call for Evidence on Gamebird Release, Shooting and Welfare during Rearing in England is a masterly exercise in managing ambiguity.

We are told the evidence on gamebird release is incomplete, and that the purpose of the call is to gather input on both the “positive and negative” impacts of the practice in order to “improve the evidence base”. The exercise is therefore presented as open, balanced and exploratory.

Those claims matter because they shape the way people will look at and engage with this process. If Defra genuinely wants to investigate an incomplete, uncertain and contested evidence base before making policy, the process itself should reflect that.

A key question, therefore, is whether the structure and wording of the call are consistent with claims of balance and robustness.

Conclusions before the evidence

Even a quick look at the call reveals flaws that suggest Defra’s claims of balance and robustness may not be very… well, robust.

From the outset, the description is vague and imprecise. At one point, it is stated that the current regulatory framework is limited and outdated before going on to tell us that evidence is needed to assess it. In other words, the conclusion comes before the evidence meant to substantiate it.

We are also told that evidence will be thoroughly examined, but not told how. There is an implication that the evidence will support policy, but no information is forthcoming as to how that will work, either.

Bias built into the questions

Earlier hints of bias are amplified in the architecture of the questionnaire and the questions themselves.

A fair questionnaire should avoid steering people towards particular answers. This one doesn’t. Almost a third of its 106 questions (31 in total) contain 47 methodological biases between them. That includes questions that assume an effect rather than test for it, leading phrases, emotive language and uneven levels of detail across topics.

Sections on the welfare and environmental impact of gamebird release display extreme bias. A high level of detail is sought concerning negative welfare and environmental effects linked to gamebird release, without equal attention being paid to their substantial environmental benefits (a classic form of questionnaire bias).

Far from balanced, in many places the questionnaire simply re-hashes a hit list of anti-shooting talking points in question form. It cues up the audience to respond in ways which are likely to “confirm” the presence of those things that are mentioned in detail, while almost guaranteeing the under-representation of those things that are not featured in such depth.

Steering the response

The survey steers people’s responses in specific directions throughout. On pathogens, respondents are asked to comment on “highly infectious” avian flu. On rural policing, you are restricted to commenting on raptor killings associated with bird release. There is no mention of the countless positive contributions that those involved in gamebird release make to crime prevention.

The economic and human wellbeing benefits of shooting are entirely ignored. Indeed, the document’s authors go to significant lengths to exclude comment on the economy by stating that Defra is commissioning independent research on it separately, in the form of a Master’s dissertation project.

Not so neutral questions

Focusing on the individual questions alone misses the scale of the methodological issues in the call for evidence. It also contains architectural biases that operate across multiple questions and, to some degree, the entire questionnaire.

They include a binary framing, negative priming and sequencing bias, the latter being where one question ‘sets up’ the response to the next.

Consequently, even questions that appear individually neutral cannot automatically be regarded as neutral in practice. This is because the context in which they appear has already been structured by preceding questions and by the architecture of the call for evidence to favour certain answers.

Why this matters

These are not simply scientific niceties. Good policymaking depends on all the affected stakeholders trusting that decisions are based on credible data, gathered and analysed fairly.

On the basis of Defra’s document, it is hard to say that the test has been met. Like the Marauder’s Map, the call seeks to conceal mischief. Unlike the map, some of the traces are still visible.

The dice have been loaded against our community so blatantly in this call for evidence that support for BASC’s call for high-quality evidence has never been more important.

Click here for our guidance on how to submit your evidence

Patrick O'Reilly

Patrick O’Reilly is BASC’s head of social science. His role includes working to improve recognition of shooting as an important part of cultural life and making the case for its inclusion in all four of the UK’s national living heritage lists.