A critical appraisal of, the "crime" data in Alberta's report on supervised consumptions sites.
By Dr. Jamie Livingston, re-posted from Twitter (Part II) on March 5, 2020, with permission.
A deeper dive into, and critical appraisal of, the "crime" data in Alberta's report on supervised consumptions sites.
“This review used a mixed method research approach” (P2) First off, this review didn’t use a mixed method approach. It used a multi method approach.
“Convenience sampling was employed” (P2) This isn’t inherently bad or unusual, but it carries significant limitations that could undermine the validity of the findings. Especially, when relying heavily on perceived impacts of an intervention on a community.
“we have attempted to compare before and after trends in neighbourhoods of interest with other parts of the community” (P3) The review does not report or perform any statistical analyses with appropriate statistical controls to make such comparisons . The neighbourhood effects or the effects of variables other than the SCS, such observations are scientifically unsound.
“Although the committee heard opinions that the existence of a SCS reduces crime or, at worst, has no impact on drug-related criminal activity in the immediate area, the preponderance of evidence provided by area residents and officials demonstrates that criminal activity near SCS has increased" (P4)
When the data does not support the desired conclusion, it’s called “opinions.” When the data does support the desired conclusion, it’s called “evidence”. This clearly signals biased interpretation.
“SCS, therefore, are assumed to geographically concentrate the street-level drug market and other criminal activities. The Committee finds this to be credible” (P4)
This conclusion is based solely on stakeholders’ subjective perceptions. The committee makes no attempt to evaluate this claim using other data sources, which is actually what a mixed method study does.
“Concerns were also raised that the exemptions to the legal possession of illicit substances at the SCS resulted in a lack of law enforcement in areas adjacent to the sites” (P5)
Again, no attempts were made to evaluate this claim with other sources of data.
“one respondent who did not live near a site noted, 'There are already a lot of homeless people who linger around my neighborhood, thanks to the LRT… crime rates have skyrocketed'" (P8)
Perceptions of crime often don’t reflect the reality of crime. For instance, since the early 1990s, the crime rate in most Western countries has been declining; however, the public perception is that crime rate has not changed or has increased.
“This inflection coincides with the opening of the SCS site. The seizure data, however, are for Calgary as a whole, which suggests that the problem is not unique to the SCS sites but is more widespread” (P13)
Figure 3 is highly problematic since it has no value for evaluating the effects of the SCS. What then is the function of this figure, other than eliciting fear and creating moral panic?
“A primary concern of most people living near a SCS site was a perceived increase in crime” (P14) I’m pretty certain that people are more concerned about actual crime, as objectively measured, rather than perceived crime.
“The pertinent question thus became whether those areas near SCS experienced changes in the amount of crime disproportionate to other areas of the city after the sites were opened” (P14)
Actually, no, the pertinent question should be the extent to which changes in crime can be attributed to the SCS. This is how questions that evaluate interventions are framed. The methods employed in this report don’t allow for causal inferences between the independent (e.g., SCS) and dependent (e.g., crime).
“It should also be noted that crime is measured in calls for service and not actual rates of victimization. At the town hall and various in-person meetings, many individuals indicated that area residents were suffering from reporting fatigue and were increasingly reluctant to report less serious offences to the local authorities” (P14)
Wow. Um. Okay. So, the report doesn’t actually measure crime, but in the next breath concludes about Lethbridge that “immediately evident is the amount of crime increased substantially in the area immediate to the SCS”.
The criminological literature notes major limitations with using ‘police calls for service’ as a measure of crime. Which would have been known if an actual criminologist was involved.
The analysis of the data represented in these figures are particularly terrible. No criminologist would use a one-year change in crime to measure anything, since knowing about the longer-term trend is vitally important. Moreover, eyeballing two bar graphs and making simple comparisons of two communities to draw conclusions is as unsound as it gets.
Regarding Calgary “it was not possible to perform an exact before/after monthly comparison around the opening date for the site. However, t is possible to get a reasonable impression of whether there was any relative increase in calls for service between 2017 and 2018.” (P15)
The review repeatedly sweeps away major methodological limitations and draws unfettered conclusions from flawed data, including this sections that, in the absence of quality data, methodological rigour, or statistical analysis, states that the data “indicates that residents’ concerns are well founded”.
The report notes that “calls for service decreased by about 1.3 per cent in the area immediately adjacent to the sites” (P19). Unlike the other figures which support the authors’ narrative this is then followed by a disclaimer about the quality of this particular data “Edmonton was also a location where area residents indicated that they were refraining from calling police because of a perceived lack of response.”
The city-specific conclusions in Chapter 4 are scandalous. For instance: “During the limited time the Review Committee spent visiting the area around the site, members directly observed several instances of open drug use, with individuals injecting drugs on the sidewalks close to the SCS site" (P25) Haphazardly seeing drug use near an SCS site is not a finding of scientific or evaluative merit. It was also not described in the methods section of the report.
A conclusion of the report indicates that "Evidence suggested a level of 'de-policing' near some sites." First, this wasn't measured. Second, it is contrary to the supposed finding that police calls for service increase in these areas. This is a baffling conclusion.
