Saturday, April 25, 2015

Trend Analysis: Checking "Facts"


Click here to download a pdf version of this article with pictures.
Click here to download a pdf version of this article without pictures. 
NOTE: Both version are tagged for screen reader software.



Photo: Dave Lundy (Flikr) Used with permission.
Dangerous Practices:
  • Sometimes people with an agenda misuse statistics to support a pre-determined goal.  They may search for statistical facts that support their position, or tear down their opponent's position. 
  • Sometimes people who are insufficiently trained in statistics make mathematical or interpretive mistakes and don't realize it. 
  • Both circumstances are dangerous because the unsuspecting public may believe these claims because they sound like they are based upon facts.  
Photo: Paolo Massa (Flikr)
Used with permission
A frequently misused statistic is the A-B comparison, where differences between two numbers are calculated.  For example:
  • In 2005, 1,062 Black state workers were promoted.  This is the "A" fact.
  • In 2006 the number increased to 1,134.  This is the "B" fact.
(Source of these data here).  





Illustration: Id Lom (Flikr) Used with permission
Someone with an agenda, or someone who is insufficiently trained in statistics might note the increase in 2006 and then dangerously state conclusions they believe (i.e., guess) explain the rate change, such as: 
  • This is evidence that the state is taking steps to correct historical job discrimination.
  • The rate of promoting Black state workers continues to improve.





Illustration: Benjamin Gray (Flikr) Used with permission

 Yes, the A-B relationship does show an increase in 2006; however, does this prove the state is helping Black state workers promote?






Tables here, Source data here
Trend analysis shows us
the claim is misleading!  

 Looking at the 10 year data, and comparing Black state workers to other groups shows: 
  • The rate of increase for Black state workers in 2006 was the lowest of the five groups.
  • In nine of 10 years the promotion rate for Black state employees was lower than any other group. 
This is a good example of why people should be skeptical of interpretations based upon A-B comparisons, and careful to distinguish what the A-B data say and opinions stated to interpret them. 




TAKE HOME LESSONS THAT WILL SERVE YOU WELL AS YOU STRIVE TO DEVELOP SCIENTIFIC THINKING AND SEARCH FOR CREDIBILITY IN DATA AND EXPLANATIONS
  • When you hear "we improved" statements based upon a single year's data, be cautious.
  • When you hear "we improved" statements in isolation--meaning they aren't compared to other similar measures--be cautious.
  • Develop the habit of questioning A-B isolated data.
  • Develop the habit of questioning anyone's conclusions (even mine!).
  • Develop the habit of asking for source data, and then checking to see if it is correctly represented. 
  • Develop the habit of looking at trend data over time.
  • Avoid, whenever possible, seeing any significance to an isolated A-B change. 

Please cite this blog as:  Nelson, Eric L. (2015).  Trend Analysis: Checking "Facts".  Trends In State Work, http://trendsinstatework.blogspot.com/2015_04_25_archive.html

Click here to download a pdf version of this article (tagged for screen readers).

Posted by at