|
Evaluating Crime Prevention Programs CRIME VICTIMS UNITED |
In 1996, a Federal law was passed requiring the United States Attorney General to provide to Congress a report on the effectiveness of crime prevention programs that Congress was funding. The University of Maryland School of Criminology was chosen to study the question and write a report. The report, published in 1998, took the form of a "meta-study" - a review of 500 other studies.
Congress wisely insisted that the review "employ rigorous and scientifically recognized standards and methodologies". This was of critical importance for the credibility of the review, and more importantly, to avoid sending policymakers down a blind alley from which it could take decades to emerge.
The most valuable result of the review was the creation of the Maryland Scale of Scientific Method - a standard for judging the validity of any prevention program. In creating this scale, which is similar to criteria used in medical studies, the authors set a high bar for crime prevention evaluations. But that is where the bar should be if we are to draw conclusions with large implications for public safety.
The Maryland scale assigns a rating of 1 (weakest) to 5 (strongest) based on the design of the evaluation of a prevention program's effect. Here is a simplified explanation of the scale.
| Level | Description | Explanation |
| Level 1
Weakest |
"Correlation between a crime prevention program and a measure of crime or crime risk factors at a single point in time." | A program is initiated and a measure of crime (such as the
crime rate) is observed.
|
| Level 2 | "Temporal sequence between the program and the crime or risk outcome clearly observed, or the presence of a comparison group without demonstrated comparability to the treatment group." | There is no control group but changes following treatment in
the treatment group are studied.
OR There is a control group but it is not demonstrated to be comparable to the treatment group. |
| Level 3 | "A comparison between two or more comparable units of analysis, one with and one without the program." | A group is treated with a program. Another untreated group of comparable offenders as a control group. |
| Level 4 | "Comparison between multiple units with and without the program, controlling for other factors, or using comparison units that evidence only minor differences." | A group is treated with a program. Another untreated group
of comparable offenders is identifed and efforts are made to ensure that
the groups are comparable.
This procedure is replicated. |
| Level 5
Strongest |
"Random assignment and analysis of comparable units to program and comparison groups." | A population of offenders is divided into two groups by
random selection. One group receives treatment and the other does not.
This procedure is replicated. |
The report lists four threats to the validity of a program evaluation. Of these, the most prevalent is "selection bias". Selection bias occurs when the members of the treatment group are chosen in such a way as to distinguish them from the control group so that any resulting difference in outcome could be attributed to the difference in the nature of the groups, not to the program. For example, if a program is offered and those who choose to enter it are considered the treatment group while those who choose not to enter it are the control group, this program suffers from self-selection bias. If the treatment group does better than the control group, it is likely because of the difference in natures between those who want to reform and those who do not, not due to the program itself.
Only level 5 programs, those that use random selection to decide who is treated and who is not, are immune to selection bias.
Of the 500 program evaluations studied by the Maryland group, few rated 4 or 5 on the Maryland scale. The report says "Very few operational programs have been evaluated using scientifically recognized standards and methodologies."
Based on "minimally adequate evidence", the authors sorted programs into three groups: those that provisionally appear to work, those that appear promising, and those that do not appear to work. Programs with two or more studies at level 3 or higher were listed as appearing to work. Those with one study at level 3 were listed as promising. The remainder were listed as not working.
Summary of University of Maryland Report on Crime Prevention Programs