A New Methodology for Evaluating Incident Detection Algorithms

Problem

Automatic incident detection algorithms (AIDs) can operate on data from inductance loop detectors using methods including filtering, pattern recognition, catastrophe theory, neural networks, and genetic algorithms. Determining which AID is best for a given situation is difficult, in part because parameters must be set by the practitioner, and performance is very sensitive to these settings. Standard evaluation of AIDs has focused on the difference between detection rates (DR) and false alarm rates (FAR), but this method is fraught with difficulties. For one, judging what constitutes an incident is a subjective action, and makes fair comparisons between different algorithms nearly impossible. Also, DR-FAR curves treat all incidents with equal importance, whether they are low-impact breakdowns on the shoulder or major accidents causing hours of delay.

Method

Our approach to evaluating AIDs attempts to solve these problems by using costs rather than DR-FAR curves. That is, we estimate costs of delay as well as the costs of implementing the AID (dispatching tow trucks, etc). The estimates are based on assumptions using training data, and can be consistent for all algorithms under evaluation. Severity of incidents and time to response are automatically factored into the analysis, as they are reflected in the cost.

Practitioners are best qualified for making assumptions linking congestion and costs, and our method allows for flexible formulations of cost. For example, the cost function could reflect increasing benefit for responding promptly to injury accidents.

An AID can be "tuned" to be very sensitive, which reduces delay because incidents are quickly detected. However, greater sensitivity also means increased implementation costs, as more interventions are called for. The goal is to strike a balance between costs of delay and costs of implementation.

Our equation converts delay and implementation actions to costs and finds the lowest possible cost of an AID using a given set of data.

Findings

Our method systematically tunes the parameters of an AID and makes it possible to fairly compare different types of AIDs, avoiding several problem inherent in DR-FAR curves. For example, the severity of an incident in terms of congestion and the time it takes to detect it are automatically factored into the analysis. Our flexible framework allows practitioners to tailor the cost functions to the particular problem, and to set AID parameters systematically by finding the lowest cost for a given set of data.