Evaluating Safety Effectiveness

This is the last step in safety management process which is presented in chapter 9 of HSM. We have came up to the 6th level of this 6-story building exploring each level of it individually and in relation with other levels of the building. In this step, we evaluate the effectiveness of the safety enhancement treatments proposed on a study area. The evaluation can include an assessment of how the number of crashes or their severity have changed on a transportation unit (roadway, interchange, or intersection) as a result of implementing the treatment(s). One nice thing about this evaluation is that if we have the before- and after-implementation crash statistics we can come up with our own crash modification factors (CMF) for the given treatment. Also, this step helps us understand how well the fund were invested on improving safety. These insights can help us in future safety managements right?

There are various methods for evaluating safety effectiveness and by evaluation we mean coming up with a quantification of the effects in safety i.e. coming up with some numerical yardstick that provides us with a performance measure to compare the before and after conditions. As it is mentioned in the HSM, there are three study designs that can help in evaluating safety effectiveness: Observational before/after, observational cross-sectional, and experimental before/after studies. The difference between an observational and an experimental study is that in an observational study, no experiment had been designed. let's say the state DOT found some sites with problems and tried to implement some treatment to improve the safety and now, we want to evaluate the treatments' effectiveness. We have no choice other than observe the crash data on those sites and see what happened before the implementation and how it was after the implementation and by comparison and doing some statistical tests come up with an evaluation result. On the other hand, in an experimental study we first design the experiment in which we are interested to evaluate the effectiveness of some treatment. In other words, we choose a treatment and then according to the rules in design of experiment, we assign the treatment to random sites and record the results and then do the tests and comparisons. The type of observation in an experimental study is different because we already know what our objective is, where the treatments are implemented and based on what logic they were assigned to sites etc. everything is basically planned beforehand. For more information on experimental study click here. So which one these two types do we think is more common in reality of safety effectiveness studies? Of course the observational studies. Those huge amounts of funds are usually used to improve safety at given hazardous locations determined through safety management process (and sometimes, political reasons). There is not much fund or time for conducting the experimental studies unless some graduate student at a university would like to accomplish one to get their degree and publish some technical papers. Henceforth, HSM focuses on observational studies in the manual. 

One thing that we should keep in mind when selecting a site for conducting an observational study is, though, to not go for the sites with very high crash frequency as that will result in a bias in the findings of the study. This bias is commonly known as selection bias and it also results in a regression-to-the-mean (RTM) bias. These biases has to be paid attention when conducting any safety analysis as they can mask the results and present a false favorable outcome. For more information on the RTM issue click here.

• Observational Before/After Study

          This is the most commonly used method of evaluation. A treated site is considered and the crash and volume statistics of the site for before and after treatment implementation is recorded. For doing such evaluations we better also have some similar sites available wherein no treatments were implemented so that in the end we can compare the statistics from those sites with our evaluation sites to have a better insight on the effects of safety treatments. If such sites with no treatments are not considered in the evaluation process, it would be called a simple or naive before/after evaluation.

• • Empirical Bayes approach

                    Observational before/after studies that consider no non-treatment sites use Empirical Bayes (EB) method which is the most common approach. This approach is also mainly important where the sites for evaluation are chosen based on a project completed by a highway agency such as a DOT where they tried to improve the safety of a site with high crash frequency. The EB method alleviates the unfavorable effects of the selection bias and somehow compensates the RTM bias. The EB method uses the existing safety performance functions (SPF) (reference) to estimate the expected crash frequencies. Detailed information on how this method works is presented in the fundamentals part A of the HSM in chapter 3, Section 9.4, and Appendix 9A. (reference). The EB method is also built into the FHWA Safety Analyst software for the purpose of safety effectiveness evaluation. 

• • Comparison group approach

                    Another way is to consider non-treatment (similar) sites as comparison groups for the safety effectiveness evaluation of treatment sites. The reason for this is to see "what if there was no treatment implemented at the site(s), what would have been the change in crash statistics in that case". Therefore, the natural trend of change in safety condition can be taken into account along with the implementation of the treatment proposed for a dangerous location. 

• Observational Cross-Sectional Study

So what if we don't know when the treatments were implemented, or the required crash, geometric, or traffic data is not available? How are we supposed to do a before/after study? Then we'll go for a cross-sectional study instead. For example, if we want to see what is the safety effect of adding alligator teeth pavement markings and what we have is only some roadway segments with and without this treatment and not detailed information on the implementation dates. 

It is called observational cross-sectional because we observed the data that exist on a cross-section of transportation facilities (which can be roadway segments, intersections, etc.) and the data is called cross-sectional data. So, basically the data is acquired over space during the same time. If it was acquired over time for only one or two facility, then it would have been a longitudinal data, for the way it is use in the observational before/after data. Now if we have the data for many facilities over time we have both types and name it panel data, because panels have two dimensions, both over space and over time.

Statistical modeling techniques are used in this type of study (also called with/without study) to compare the with and without treatment conditions with each other in terms of their effects on crash statistics.Basically the difference in crash statistics between the two sets of locations with and without the treatment is attributed to the effects of that safety treatment of interest. The data in this type of study is obtained for the same time period of both sets of locations. It should be kept in mind that there no appropriate way to account for the RTM bias as a result of selecting the treatment sites. Also, we not completely sure if the difference between the two locations are truly because of the treatment(s) implemented or some unknown cause.

The rest of the chapter 9 of HSM explains the details on the procedures for different evaluation methods and examples of such evaluations. Since, here I was just trying to briefly represent what the safety management process is about, I am going to skip the details and have those methods and approaches presented in somewhere else.