New Report Recommends CSA Program Overhaul: What Now? | Transportfolio
Hot off the presses, the National Academy of Sciences (NAS) released the widely anticipated report on the Compliance Safety and Accountability program. This report was required by Congress as part of the FAST Act signed by President Obama in December 2015.
There had been growing evidence and criticism by carriers, shippers, and brokers that CSA data, and specifically Safety Measurement System (SMS) BASIC percentiles, were misleading and inaccurate when applied to individual carriers. As a result, Congress required the Federal Motor Carrier Safety Administration (FMCSA) to take down the BASIC percentiles from public view. They tasked the NAS with a full review of the SMS methodology.
Here are some of the important conclusions in the report:
Outcome for shippers and brokers: More research required
Outcome for shippers and brokers: More research required | Transportfolio
- Congress asked the NAS panel if the BASIC percentiles should be publicly available.
In the study, the panel recommends another study to look at this more in depth and acknowledges “the disadvantages of public reporting stem from the fact that SMS is not a perfect discriminator between the carriers that need to improve their safety performance and those that do not.”
- The panel highlights the difficulty of using small sample sizes to predict outcomes for small carriers.
They state in the report “there is a particular concern about false negatives and false positives among smaller carriers, which results from not having much data with which to judge them.”
- BASIC percentiles will remain hidden from public view for quite some time.
Yet another study will try to resolve whether BASIC percentiles are good indicators of an individual carrier’s likelihood of being involved in an accident.
- More shippers and brokers may ask for legislation.
This report will likely lead to more calls from shippers and brokers to pass legislation to establish uniform federal standards when qualifying motor carriers. This legislation has been introduced in both the U.S. House (HR 1568) and the Senate (S 1345)
Outcome for carriers and drivers: More visibility needed
Outcome for carriers and drivers: More visibility needed | Transportfolio
- The report validates many carrier complaints of SMS.
Some of which include, poor quality data, old data, and inconsistent data.
- Carriers have long complained about a lack of visibility to methodology.
In other words, they may think corrective actions will improve scores, but when the new scores come out, carriers are surprised at the lack of impact. This reports recommends that FMCSA produce better predictive models for carriers so they know what actions will have direct statistical impacts. Carriers may start adding data scientists to help with this data.
- There are issues regarding relative rankings versus absolute measures.
The report highlights issues related to relative rankings versus absolute measures (should peer groups be used or not?) The report says both should be used.
- The panel recommends a meaningful change in methodology to an Item Response Theory (IRT) model.
This would be tested over the next two years alongside current SMS methodology. In plain language, this means that CSA methodology will continue to evolve over the next few years. The measurement of carriers and drivers adherence to the rules will keep changing.
You will likely see many more articles in the near future regarding the analysis of this report. We will provide updates as new details emerge.