Dixon's Q test for outlier identification (2024)

I recently faced the impossible task to identify outliers in adataset with very, very small sample sizes and Dixon’s Q test caught myattention. Honestly, I am not a big fan of this statistical test, butsince Dixon’s Q-test is still quite popular in certain scientific fields(e.g., chemistry) that it is important to understand its principles inorder to draw your own conclusion of the presented research data thatyou might stumble upon in research articles or scientific talks.

• Sections
  • Application
  • Criticism
  • Method
    • 1) Arrange values for observations in ascending order
    • 2) Calculate Q
    • 3) Compare the calculated to the tabulated critical Q-value
    • 4) Example
  • Implementation
    • Building dictionaries for Q-value look-up
    • Implementing a Dixon Q-test Function
    • Assertion Tests
    • Example application
  • Plotting the Data
    • Bar plot of the sample means with standard deviation
    • Boxplot
  • Conclusion

Dixon’s Q test [1] was “invented” as a convenient procedure to quicklyidentify outliers in datasets that only contains a small number ofobservations: typically 3 > n ≤ 10.

[1] R. B. Dean and W. J. Dixon (1951) Simplified Statistics for SmallNumbers ofObservations”. Anal.Chem., 1951, 23 (4), 636–638

Application

Although (at least in my opinion), the removal of outliers is a veryquestionable practice, this test is quite popular in the field ofchemistry to “objectively” detect and reject outliers that are due tosystematic errors by the experimentalist.

Criticism

If we want to use this test to legitimately remove (potential) outliersfrom a dataset, we should keep in mind that

• our data has to be normal distributed,
• and that we are not supposed to use this test more than once thesame data set.

In my opinion, the Dixon Q-test should only be used with great caution,since this simple statistic is based on the assumption that the data isnormal distributed, which can be quite challenging to predict for smallsample sizes (if no prior/additional information is provided). Personally, I would use the Dixon Q-test to only detect outliersand not to remove those, which can help with the identification ofuncertainties in the data set or problems in experimental procedures.Intuitively, this is quite similar to an approach of identifying samplesthat have a large standard deviation.

For example, if I tested ~1000 chemical compounds in some sort ofactivity assay - each compound 5 times, I would mark compounds thatcontain Q-test outliers for re-testing, because there might have beensome problem in the measurement procedure that could have caused thisinconsistency.

Method

1) Arrange values for observations in ascending order

First, we arrange the data for our sample in ascending order(from the lowest to the highest value):

2) Calculate Q

Next, we calculate the experimental Q-value (\(Q_{exp}\)).

Note that in a later paper in 1953, Dixon and Dean [3] revisited thecalculation of the Q-value and reported different equations fordifferent scenarios:

(from: Rorabacher, David B., 1991 [2])

However, according to a statement/observation in a more recent paper(Rorabacher, David B., 1991 [2]): “The \(r_{l0}\) ratio is commonlydesignated as ‘Q’ and is generally considered to be the most convenient,legitimate, statistical test available for the rejection of deviantvalues from a small sample conforming to a Gaussian distribution. (It isequally well suited to larger data sets if only one outlier ispresent.)”

Therefore, I will use \(r_{l0}\) for the following implementation of theDixon Q-test:

where it is assumed that the data is arranged in ascending order:

[2] Rorabacher, David B. (1991) “Statistical Treatment for Rejection ofDeviant Values: Critical Values of Dixon’s‘ Q’ Parameter and RelatedSubrange Ratios at the 95% ConfidenceLevel.” AnalyticalChemistry 63, no. 2 (1991): 139–46.

[3] W.J. Dixon: Processing data for outliersReference”:J. Biometrics 9 (1953) 74-89

3) Compare the calculated to the tabulated critical Q-value

In the next step, we will compare the calculated \(Q_{exp}\) value to the tothe tabulated critical Q-value \(Q_{crit}\) for a chosen confidenceinterval.
If the calculated Q-value for a particular observation is larger thanthe critical Q-value (\(Q_{exp} \ge Q_{crit}\)), this observation isconsidered to be an outlier according to the Q-test.

\(r_{10}\) Critical values for Dixon’s two-tailored Q-Test for 3 differentconfidence levels**

4) Example

Let’s consider the following sample consisting of 5 observations:

0.142, 0.153, 0.135, 0.002, 0.175

• First, we sort it in ascending order: 0.002, 0.135, 0.142, 0.153,0.175
• Next, we calculate the Q-value:

• Now, we look up the critical value for n=5 for a confidence level95% in the Q-table \(= \ge 0.71\)

and we conclude that 0.002 (since 0.7687 > 0.71), that theobservation 0.002 is an outlier at a confidence level of 95% accordingto Dixon’s Q-test.

Implementation

In the next few subsections, I will implement the Q-test in pure Python.I have to apologize for not using packages from the sci-stack (pandas,NumPy, scipy …) this time and thus making the code look lesselegant, but I wrote this code for a non-Python person and promised tomake it work with a standard Python installation.

Building dictionaries for Q-value look-up

We will build a simple set of dictionaries for different confidenceintervals from the tabulated data in David B. Rorabacher’s paper: Rorabacher, David B. (1991) “Statistical Treatment for Rejection ofDeviant Values: Critical Values of Dixon’s‘ Q’ Parameter and RelatedSubrange Ratios at the 95% ConfidenceLevel.” AnalyticalChemistry 63, no. 2 (1991): 139–46.

Then, we can use this dictionary to look up the critical Q-values(dictionary values) for different sample sizes (dictionary keys).

q90 = [0.941, 0.765, 0.642, 0.56, 0.507, 0.468, 0.437, 0.412, 0.392, 0.376, 0.361, 0.349, 0.338, 0.329, 0.32, 0.313, 0.306, 0.3, 0.295, 0.29, 0.285, 0.281, 0.277, 0.273, 0.269, 0.266, 0.263, 0.26 ]q95 = [0.97, 0.829, 0.71, 0.625, 0.568, 0.526, 0.493, 0.466, 0.444, 0.426, 0.41, 0.396, 0.384, 0.374, 0.365, 0.356, 0.349, 0.342, 0.337, 0.331, 0.326, 0.321, 0.317, 0.312, 0.308, 0.305, 0.301, 0.29 ]q99 = [0.994, 0.926, 0.821, 0.74, 0.68, 0.634, 0.598, 0.568, 0.542, 0.522, 0.503, 0.488, 0.475, 0.463, 0.452, 0.442, 0.433, 0.425, 0.418, 0.411, 0.404, 0.399, 0.393, 0.388, 0.384, 0.38, 0.376, 0.372 ]Q90 = {n:q for n,q in zip(range(3,len(q90)+1), q90)}Q95 = {n:q for n,q in zip(range(3,len(q95)+1), q95)}Q99 = {n:q for n,q in zip(range(3,len(q99)+1), q99)}

Implementing a Dixon Q-test Function

Below, I wrote some simple Python code to test one data row for DixonQ-test outliers:

def dixon_test(data, left=True, right=True, q_dict=Q95): """ Keyword arguments: data = A ordered or unordered list of data points (int or float). left = Q-test of minimum value in the ordered list if True. right = Q-test of maximum value in the ordered list if True. q_dict = A dictionary of Q-values for a given confidence level, where the dict. keys are sample sizes N, and the associated values are the corresponding critical Q values. E.g., {3: 0.97, 4: 0.829, 5: 0.71, 6: 0.625, ...} Returns a list of 2 values for the outliers, or None. E.g., for [1,1,1] -> [None, None] for [5,1,1] -> [None, 5] for [5,1,5] -> [1, None] """ assert(left or right), 'At least one of the variables, `left` or `right`, must be True.' assert(len(data) >= 3), 'At least 3 data points are required' assert(len(data) <= max(q_dict.keys())), 'Sample size too large' sdata = sorted(data) Q_mindiff, Q_maxdiff = (0,0), (0,0) if left: Q_min = (sdata[1] - sdata[0]) try: Q_min /= (sdata[-1] - sdata[0]) except ZeroDivisionError: pass Q_mindiff = (Q_min - q_dict[len(data)], sdata[0]) if right: Q_max = abs((sdata[-2] - sdata[-1])) try: Q_max /= abs((sdata[0] - sdata[-1])) except ZeroDivisionError: pass Q_maxdiff = (Q_max - q_dict[len(data)], sdata[-1]) if not Q_mindiff[0] > 0 and not Q_maxdiff[0] > 0: outliers = [None, None] elif Q_mindiff[0] == Q_maxdiff[0]: outliers = [Q_mindiff[1], Q_maxdiff[1]] elif Q_mindiff[0] > Q_maxdiff[0]: outliers = [Q_mindiff[1], None] else: outliers = [None, Q_maxdiff[1]] return outliers

Assertion Tests

Some simple assertion tests to make sure that the Dixon Q-test functionbehaves as expected/desired.

test_data1 = [0.142, 0.153, 0.135, 0.002, 0.175]test_data2 = [0.542, 0.153, 0.135, 0.002, 0.175]assert(dixon_test(test_data1) == [0.002, None]), 'expect [0.002, None]'assert(dixon_test(test_data1, right=False) == [0.002, None]), 'expect [0.002, None]'assert(dixon_test(test_data2) == [None, None]), 'expect [None, None]'assert(dixon_test(test_data2, q_dict=Q90) == [None, 0.542]), 'expect [None, 0.542]'print('ok')

ok

Example application

Below, I want to go through a naive example for our Dixon Q-testfunction using an example CSV file. In “real” application I would prefer NumPy and/or pandas, howeverfor this simple case the in-built Python csv library should suffice.

Below the example CSV file is shown that we are going to read in:

%%writefile ../../data/dixon_test_in.csv,x1,x2,x3,x4,x5id1,0.95,-0.65,0.6,0.82,NaNid2,2.08,NaN,-1.43,0.38,NaNid3,-0.46,NaN,-1.25,-2.62,0.22id4,0.24,1.88,-0.49,-0.73,-0.49id5,-1.65,2.1,-0.09,NaN,0.8id6,-0.44,0.93,0.19,-4.36,-0.88id7,0.36,-0.47,NaN,0.4,2.12id8,1.29,-0.48,-0.6,-0.38,0.27id9,-1.25,-1.35,1.13,1.7,-0.81id10,0.04,1.98,NaN,NaN,NaN

import csvdef csv_to_list(csv_file, delimiter=','): """ Reads in a CSV file and returns the contents as list, where every row is stored as a sublist, and each element in the sublist represents 1 cell in the table. """ with open(csv_file, 'r') as csv_con: reader = csv.reader(csv_con, delimiter=delimiter) return list(reader)def print_csv(csv_content): """ Prints CSV file to standard output.""" print(50*'-') for row in csv_content: row = [str(e) for e in row] print('t'.join(row)) print(50*'-')def convert_cells_to_floats(csv_cont): """ Converts cells to floats if possible (modifies input CSV content list). """ for row in range(len(csv_cont)): for cell in range(len(csv_cont[row])): try: csv_cont[row][cell] = float(csv_cont[row][cell]) except ValueError: pass

csv_cont = csv_to_list('../../data/dixon_test_in.csv')convert_cells_to_floats(csv_cont)print_csv(csv_cont)

-------------------------------------------------- x1 x2 x3 x4 x5id1 0.95 -0.65 0.6 0.82 nanid2 2.08 nan -1.43 0.38 nanid3 -0.46 nan -1.25 -2.62 0.22id4 0.24 1.88 -0.49 -0.73 -0.49id5 -1.65 2.1 -0.09 nan 0.8id6 -0.44 0.93 0.19 -4.36 -0.88id7 0.36 -0.47 nan 0.4 2.12id8 1.29 -0.48 -0.6 -0.38 0.27id9 -1.25 -1.35 1.13 1.7 -0.81id10 0.04 1.98 nan nan nan--------------------------------------------------

Now, let us add a new outlier column and apply the Dixon Q-testfunction to our data set.

import mathcsv_cont[0].append('outlier')for row in csv_cont[1:]: # skips header nan_removed = [i for i in row[1:] if not math.isnan(i)] if len(nan_removed) >= 3: row.append(dixon_test(nan_removed, left=True, right=True, q_dict=Q90)) else: row.append('NaN')

print_csv(csv_cont)```python-------------------------------------------------- x1 x2 x3 x4 x5 outlierid1 0.95 -0.65 0.6 0.82 nan [-0.65, None]id2 2.08 nan -1.43 0.38 nan [None, None]id3 -0.46 nan -1.25 -2.62 0.22 [None, None]id4 0.24 1.88 -0.49 -0.73 -0.49 [None, None]id5 -1.65 2.1 -0.09 nan 0.8 [None, None]id6 -0.44 0.93 0.19 -4.36 -0.88 [-4.36, None]id7 0.36 -0.47 nan 0.4 2.12 [None, None]id8 1.29 -0.48 -0.6 -0.38 0.27 [None, None]id9 -1.25 -1.35 1.13 1.7 -0.81 [None, None]id10 0.04 1.98 nan nan nan NaN--------------------------------------------------

As we can see in the table above, we have 2 potential outliers in ourdata set. Finally, we let us write the results to a new CSV file forfuture reference:

def write_csv(dest, csv_cont): """ Writes a comma-delimited CSV file. """ with open(dest, 'w') as out_file: writer = csv.writer(out_file, delimiter=',') for row in csv_cont: writer.writerow(row)write_csv('../../data/dixon_test_out.csv', csv_cont)

Plotting the Data

To get a visual impression of how our data looks like, let us make somesimple plots.

Bar plot of the sample means with standard deviation

First, let’s create a bar plot with standard deviation, since such barplots with standard deviation or standard error bars are probably themost common plots in any field. Although it is not always appropriate itis certainly the kind of data visualization we are most familiar with -due to the frequent exposure when reading scientific research articles.

import numpy as npfrom matplotlib import pyplot as pltall_means = [np.nanmean(row[1:6]) for row in csv_cont[1:]]all_stddevs = [np.nanstd(row[1:6]) for row in csv_cont[1:]]fig = plt.figure(figsize=(8,6))y_pos = np.arange(len(csv_cont[1:]))y_pos = [x for x in y_pos]plt.yticks(y_pos, [row[0] for row in csv_cont[1:]], fontsize=10)plt.xlabel('measurement x')t = plt.title('Bar plot with standard deviation')plt.grid()plt.barh(y_pos, all_means, xerr=all_stddevs, align='center', alpha=0.4, color='g')plt.show()

As we would expect, basically every sample has a very (relatively) largestandard deviation. However, we can’t derive any detail about a possiblecause for this large deviation: whether it is just due to 1 outlier, orif the whole data widely spread in general.

Boxplot

A more useful plot in my opinion is Tukey’s boxplot [4]. Boxplots are infacts one of my preferred approaches to quickly and visually indicateoutliers in a Gaussian data set. However, also boxplots have to be usedwith real caution and might also not very informative for small samplesizes.

[4] Robert McGill, John W. Tukey and Wayne A. Larsen: “The AmericanStatistician”Vol. 32, No. 1 (Feb., 1978), pp. 12-16

csv_nonan = [[x for x in row[1:6] if not math.isnan(x)] for row in csv_cont[1:]]fig = plt.figure(figsize=(8,6))plt.boxplot(csv_nonan,0,'rs',0)plt.yticks([y+1 for y in y_pos], [row[0] for row in csv_cont[1:]])plt.xlabel('measurement x')t = plt.title('Box plot')plt.show()

The red squares indicate the outliers here. Quite interestingly, bothoutliers for the samples “id6” and “id1” where also picked up in ourprevious Dixon Q-test. However, the outliers in “id4” and “id7” were notindicated as outliers by Dixon’s outlier test.

Conclusion

I really don’t want to draw any conclusion about which approach isright or wrong here, since in my opinion, drawing any conclusion from adata set that is based on such a small number of observations simplyjust doesn’t make sense!

So you may wonder why I wasted your time if you read this article up tothis point? Since Dixon’s Q-test is still quite popular in certainscientific fields (e.g., chemistry) that it is important to understandits principles in order to draw your own conclusion of the presentedresearch data that you might stumble upon in research articles orscientific talks.