Data Accuracy and Precision Using Graph Extract


We were interested in estimating the accuracy and precision of raw data extraction from graphs that might be achieved using either manual or automated methods . For these evaluations we used Graph Extract Version 3.0 and two publically available datasets from the sources referenced in the figure's titles.

I. The first data set included mean 2004 SAT Verbal scores reported for the 50 U.S. states plus the District of Columbia (n=51). These mean values ranged from 487 to 593, with a grand mean for all states and the district of 533.7.

The data were plotted using Excel, and the chart was copied and saved as a bitmap which was then analyzed by Graph Extract. The original data were not ordered alphabetically by state, nor sorted by scores; rather the data in this figure were plotted directly in the same order as presented in the source document. The size of this plot was 800 x 546 with a standard bit depth of 24. The image was as follows:

Image of the SAT Scores plot

Series #1. For the first analysis, this figure was analyzed manually three times. For each analysis the figure was reloaded and the Setup values were entered new each time since accuracy in locating the marker points on the X and Y axis would be expected to influence the results. Points were clicked normally with no special effort to locate the values with unusual care. Of the three replicate assays (n=153 points in total), in only two occasions did the selected point markers appear visibly displaced. For those two values the points were erased (right-clicked) and clicked again. A typical replot of the calculated results is given here in an emf plot of the third analysis set:

Image of the SAT Scores replot

The % Error between expected and measured Scores in each case was estimated by the formula: ABS((measured-true)/true)*100. The results from these analyses were reasonably consistent. Means for the three successive estimates were 535.2, 535.5 and 535.9 compared to the expected mean of 533.7 indicating that a small but consistent high bias was found, but the overall mean % error remained quite low at less than 0.4% (mean and SD were: 0.329 ± 0.462, 0.374 ± 0.492, and 0.399 ± 0.484 for the three separate assays).

Series #2. For this set we used automated processing. Since we were interested in comparing manual with automated methods, we used the Recall Setup option to assure the same axis markers were used as in the previous manual analysis. There was essentially no difference in the results compared with manual processing with an overall calculated mean of 535.6 and a mean % error of 0.347 ± 0.507. We did not expect to find any between-analysis variance with automated processing and indeed when the assay was repeated a second time the identical results were obtained. However, when we repeated the automated analysis again but set new axis marks, the mean result did differ and was somewhat higher at 536.2 with a mean % error of 0.467 ± 0.490. As expected, the accurate and consistent placement of axis markers is an important factor in obtaining the best possible results.

II. The second data set was based on an example scatter plot from the Statistics Canada website. This set consisted of 241 x-y values widely distributed between X = 0 to 14 and Y = 0 to 12. This was a larger graph of 1138 x 778 pixels, again as a 24-bit map. Because of the size and the previous observation that there was apparently no significant difference between manual and automated processing, this graph was evaluated using automated mode only. The image was: Screenshot of Scatter Plot

A replot from the automated analysis of this graph resulted in the following image:

Screenshot of Scatter Replot

The evaluation data in this case were the 241 points represented by their individual (X*Y) products incorporating any error in both dimensions, and the % error for each value was estimated as before.

Again for this much larger set good results were observed. The target mean ± SD for the 241 original values was 38.94 ± 32.52, with values ranging from 0.4908 to 151.26. For the GraphExtract data, the observed mean and SD was 39.10  ± 32.47, with a range of 0.4960 to 151.24. The mean percent error for the 241 values was 0.877 ±  1.49. Thus the overall estimate from the extracted data was quite close to the expected value, and the mean error among all 241 estimates was less than 1%.

These assays suggest that it is possible to obtain quite good results by extracting raw data points from published plots. However, there are several factors that will influence the quality of the results and the most important is the figure itself. Obviously the two images used here represented excellent cases for processing. Although simple, well-defined figures such as these are not uncommon, neither are complex and poorly legible ones. It is up to the user to determine what is feasible and what is not, but for figures of low quality or which have many overlapping points it simply may not be possible to obtain reliable results by estimating plotted values. Neither human nor computer can separate multiple overplotted values. In such cases, contacting the original authors to request access to the raw data may be the only viable approach. Nevertheless, under good conditions, the data extraction methods used by Graph Extract should be capable of producing quite acceptable results.