Sunday, May 10, 2015

The Problem With Percentage Statistics

 Click here to download a pdf version that is tagged for screen readers. (Flikr Commons)
People and organizations sometimes calculate percentage statistics in an effort to assess the effectiveness of traditional or innovative approaches to increasing success or efficiency, or reducing problems, and so forth.

NOTE: Percentage statistics are simple numbers such as 2% or 87%.  They range from 0% to 100%.  Percentages are another way of stating a fraction.  For example, the fraction 1/4 can be stated as 25%.  The way to convert a fraction to a percentage is to convert it to a decimal format then multiply it by 100 (e.g., 1/4 = .25, .25 x 100 = 25%).

Frits Ahlefeldt-Laurvig (Flikr Commons)
Although percentage statistics can provide a useful view on a particular phenomenon, the use of percentage statistics brings with it substantial risk of interpretive error, particularly when they are compared to the last measurement of the same phenomenon, e.g., a year ago, AND, they are interpreted with a confident and authoritative guess.

Stated by themselves (i.e., without statistically valid comparison), percentage statistics require a qualitative interpretation--essentially someone's best guess.  As with all guessing, they might be right, and they might be wrong. 

For example, suppose someone said "In 2008 the Department of Audits had a 10.9% rate of employment of workers with disabilities, and one year later they raised the percentage to 28.3%.  As can be seen, the Department of Audits has gotten serious about hiring people with disabilities."  

Before we drill down, let's first ask: What are the statistics, and what is the qualitative guess?

Statistic: 13 employees with disabilities reported in 2008 (10.9%), and 41 in 2009 (28.3%).

Qualitative guess:
  • Audits is serious about employing PWD's.
  • Audits hired a lot of PWD's in 2009.
Now, let'sfact check that guess.  Here is the state's data on the Department of Audits for 2008 and 2009:

Indeed the number of PWD's reported by Audits approximately tripled in a single year, from 2008 to 2009.  Thus, the statistic is confirmed.  What about the qualitative guess?  What other factors might explain the number of PWD's tripling in a single year?

Alternative hypothesis:  Audits didn't hire more PWD's, it merely reclassified some of its employees as disabled.  If someone wears glasses, they can be counted as disabled.  If they start to wear a hearing aid they can be counted as disabled--even if these state workers don't consider themselves to be disabled. 

We have two competing explanations!

Let us look to trend statistics to see if they favor one explanation more than the other.  The trend figure below was created from 10 years of state data for the Department of Audits.  The state's data is presented in two tables a bit further below.  

  • In two years, (04/05 & 08/09) the percentage of PWD's approximately triples.  In one year (12/13) it increases very slightly. 
  • The rest of the time--7 years--the percent of PWD employees decreases. 
  • None of the four main race groups show similar doubling or tripling of representation rates.  (Because you can't reclassify someone's race). 
  • Importantly, notice there are no hiring spikes in 2005 or 2009, which you would expect to see if Audits was hiring people with disabilities.  If they were, there would be increases in most or all races, corresponding to the number of PWD's hired.  Remember, a PWD is counted twice, by disability and race. 
Click here to see more examples of this type of questionable behavior by state organizations. 
  • Audits didn't hire people with disabilities.
  • Audits reclassified a bunch of employees as "disabled." 
  • Audits should not get credit for increasing rates of employment for PWD's, because they didn't do the work. 
  • Audits should use the LEAP program and hire some PWD's who are looking for work. 

This is a good example of why trend charts should be used to fact check A-B claims, or percentage statistic-based claims
In another blog article, coming soon, we will examine many state organizations with the same, strange spikes in the number of employees with disabilities.  Taken together, these suggest the state may be claiming higher rates of employment of PWD's, when actually all they are doing is re-classifying already hired, already working employees from not disabled to disabled.  

We already learned in another blog that the state's rate of hiring people with disabilities is much lower than for the four race groups (click here to see that blog article).  

In another forthcoming blog article we will see that the state's rate of hiring of PWD's under the age of 30 has dropped 68% in the last 10 years.  So, it may be the state has some explaining to do.  There is an appearance of a shell game.

  • When you hear interpretations of percentage statistics, be cautious.
  • Develop the habit of questioning percentage statistic data.
  • Develop the habit of looking at trend data to evaluate claims based upon percentage statistics.
  • Develop the habit of asking to see source data, then checking to see if it is correctly represented.
  • When possible, avoid seeing any significance in isolated percentage statistics. 

Please cite this as: Nelson, Eric L. (2015).  The Problem With Percentage Statistics.  Trends in State Work,