Volume 6 Supplement 1
Proceedings of the 2011 International Conference on Bioinformatics and Computational Biology (BIOCOMP'11)
TSG: a new algorithm for binary and multiclass cancer classification and informative genes selection
 Haiyan Wang†^{1, 6},
 Hongyan Zhang†^{2, 3, 4},
 Zhijun Dai^{2, 4},
 Mingshun Chen^{5} and
 Zheming Yuan^{2, 4}Email author
DOI: 10.1186/175587946S1S3
© Yuan; licensee BioMed Central Ltd. 2013
Published: 23 January 2013
Abstract
Background
One of the challenges in classification of cancer tissue samples based on gene expression data is to establish an effective method that can select a parsimonious set of informative genes. The Top Scoring Pair (TSP), kTop Scoring Pairs (kTSP), Support Vector Machines (SVM), and prediction analysis of microarrays (PAM) are four popular classifiers that have comparable performance on multiple cancer datasets. SVM and PAM tend to use a large number of genes and TSP, kTSP always use even number of genes. In addition, the selection of distinct gene pairs in kTSP simply combined the pairs of top ranking genes without considering the fact that the gene set with best discrimination power may not be the combined pairs. The kTSP algorithm also needs the user to specify an upper bound for the number of gene pairs. Here we introduce a computational algorithm to address the problems. The algorithm is named Chisquarestatisticbased Top Scoring Genes (ChiTSG) classifier simplified as TSG.
Results
The TSG classifier starts with the top two genes and sequentially adds additional gene into the candidate gene set to perform informative gene selection. The algorithm automatically reports the total number of informative genes selected with cross validation. We provide the algorithm for both binary and multiclass cancer classification. The algorithm was applied to 9 binary and 10 multiclass gene expression datasets involving human cancers. The TSG classifier outperforms TSP family classifiers by a big margin in most of the 19 datasets. In addition to improved accuracy, our classifier shares all the advantages of the TSP family classifiers including easy interpretation, invariant to monotone transformation, often selects a small number of informative genes allowing followup studies, resistant to sampling variations due to within sample operations.
Conclusions
Redefining the scores for gene set and the classification rules in TSP family classifiers by incorporating the sample size information can lead to better selection of informative genes and classification accuracy. The resulting TSG classifier offers a useful tool for cancer classification based on numerical molecular data.
Background
With the availability of high throughput genomics data, methods for cancer class classification and prediction based on molecular information have been vigorously pursued in recent years. The objective of this study is to find important molecular markers and/or build a classifier such that the classifier with selected markers as the independent variables can accurately classify the diagnostic disease status of a sample using expression data. Popular methods for this problem include Prediction Analysis of Microarrays (PAM, [1]), Top Scoring Pair (TSP, [2]), kTop Scoring Pair (kTSP, [3]), Support Vector Machine (SVM, [4]) etc. There are also many other endeavors such as individualgeneranking by evaluating the discriminating power of classes (see [5, 6] and the references therein), gene filtering through relevance and correlation analyses [7, 8], gene selection for classification based on the Bayes error [9], comparing the distributions of withinclass correlations with betweenclass correlations via KullbackLeibler divergence [10], recursive feature addition with Lagging Prediction Peephole Optimization to choose the final optimal marker set [11], SVM based recursive feature elimination [12, 13], random forests [14] and random subspace search [15, 16], among others.
There are a few challenges associated with such study. One of them is that the number of independent variables (markers) is typically much more than the number of available samples, often referred as curse of dimensionality. To identify possibly nonlinear effects of many variables and their interactions, it is often necessary to estimate a large number of modeling parameters. A direct consequence of the curse of dimensionality is that the total number of parameters that the data can estimate is restricted by the number of the samples. When the total number of parameters greatly exceeds the number of samples, overfitting occurs such that the prediction of the phenotype works well for the learning data but the performance of the classifier applied to independent test samples exhibit poor classification accuracy. The informative marker selection process unfortunately needs to consider modeling with each possible combination of all markers in order to find the globally best marker set, which has the best discriminating power for the different disease categories and may or may not be the primary biological and pathological driving factors underlying disease progression. Hence, an effective practice is to first reduce the dimensionality of the marker space.
The TSP and kTSP classifiers are two simple algorithms that select gene pairs with top scores to build classifiers. They were shown to perform well for binary classification with gene expression data [2, 3]. The gene pairs were selected based on simple pairwise comparisons between two marker expression levels within the same sample. Specifically, let p_{ ij }(C_{ 1 }) be the percentage of training samples in class 1 that the expression of one marker is less than that of the other marker in the same sample and let p_{ ij }(C_{ 2 }) be similarly defined. The score for a gene pair is defined as the estimated difference between the two percentages p_{ ij }(C_{ 1 })  p_{ ij }(C_{ 2 }). Then the gene pair that received the highest score is selected as the marker set for TSP classifier and the top k gene pairs with highest scores are used for the kTSP classifier. Tan et al. [3] extended the two classifiers to multiclass classification through onevsothers, onevsone, and hierarchical classification (HC) schemes. They reported that the HC schemes for TSP and kTSP gave better performance than the other two schemes.
There are advantages and disadvantages with the TSP and kTSP classifiers. Some advantages of the two classifiers are that they are simple to implement and the resulting classifiers are easy to interpret. They are also invariant to monotone transformations as they only depend on relative rankings of gene expressions within the same sample. The overfitting problem is largely avoided due to simple comparisons. In addition, they are different from most algorithms in that comparisons in other algorithms were mostly between expressions from different samples. Comparison of expressions within the same sample in TSP and kTSP helps to eliminate the influence of sampling variability due to different subjects.
A disadvantage is related to how the scores for gene pairs are defined. As the scores were calculated from percentages, the sample size information was not fully utilized in TSP and kTSP. For example, suppose 4 out of 10 samples in class 1 and 6 out of 10 samples in class 2 satisfy the condition: Marker 1 has smaller expression value than marker 2. The score for the pair with markers 1 and 2 is 0.2, which is the absolute difference between the two percentages. In another case, suppose all the counts are multiplied by 10, i.e. 40 out of 100 samples in class 1 and 60 out of 100 samples in class 2 satisfy the condition. Then the score for the marker pair is identical to the previous case. So the additional information with extra sample size is completely ignored in TSP and kTSP classifiers.
The multiclass classifiers HCTSP and HCkTSP are two versions that showed best performance among all TSP family classifiers [3]. They were derived from a scheme that performs sequential binary classification. At the root node, the training samples are partitioned into two classes, the largest class and the composite class. The largest class containing the largest number of samples is treated as a leaf node for final classification of the phenotype. The composite class is then further partitioned similarly as the root partition. This scheme intends to balance the two classes during each binary partition. However, the markers selected at each binary partition with TSP or kTSP are not necessarily the best marker set to separate all the classes since they are selected based on their differentiating ability to separate the largest class from the composite class at the node. In addition, the selection of markers at each partition does not have a mechanism to control the redundancy of the candidate marker set. For example, in prostate cancer LNCaP cells, forkhead transcription factor (FOXO3a) that is the phosphatidylinositol 3kinase (PI3K/Akt) downstream substrate, is a positive regulator for the induction of androgen receptor (AR) gene expression. The blocking of AR functions by AR interfering RNA leads to dramatic LNCaP cell death. Hence the inhibition of the PI3K/Akt pathway may result in the activation of the FOXO3a transcription factor, which may then induce the AR gene expression to protect cells from apoptosis of LNCaP prostate cancer cells. The PI3K/Akt and FOXO3a could both be selected in the marker selection algorithm of HCTSP or HCkTSP. Apparently, they are highly correlated.
In this article, we propose a new algorithm to overcome the above problems of TSP family classifiers. We introduce a new definition of the score for each marker set so that the sample size information is fully utilized. In addition, it is unrealistic to assume that the number of informative genes is always even as in TSP family classifiers. We present a new algorithm that performs sequential search and do not restrict the informative markers to be even numbered. The binary class and multiclass cases are unified into a single framework. The algorithm was applied to 9 binary class and 10 multiclass cancer genomics datasets. The TSG classifier achieved better leaveoneout cross validation accuracy for the binary classification than TSP or kTSP classifiers. For the multiclass problems, our TSG classifier gives comparable performance or outperform TSP family and other popular classifiers with a big margin in independent test accuracy for several cancer datasets. Beyond high accuracy, our new algorithm also has the advantage of giving small number of informative marker set and all the advantages of the TSP family classifiers.
Methods
For generality, we describe the method in terms of markers, which could represent genes, probe sets, or other molecular units whose intensity is measured with high throughput instruments. Consider expression data from P markers and suppose there are N samples. The data can be expressed as a matrix Xof dimension NxP. The (i, j) element x_{ ij } of the matrix gives the expression value of the j^{th} marker in the i^{ th } sample. Let (y_{ 1 },..., y_{ N }) be the class labels for the N samples, where y_{ k } takes one of the values in the set of all possible classes {C_{ 1 },..., C_{ M } }. C_{i} represents the class phenotype that may be cancerous tumor, normal, or cancer subclasses such as different stages of a cancer. Denote x_{ i } = (x_{ i1 },..., x_{ iP }) to be the P expression values from the i^{ th } sample. The P is typically much larger than N, and could be in the neighborhood of exponential order O(e^{ N }) with high density microarrays. The objective is to use Ω = { (x_{ i } , y_{ i }), i = 1,..., N} to select a parsimonious set of informative markers and build a classifier with these markers such that the diagnosis status of a test sample can be accurately classified by modeling the expression data of the selected markers.
Score of a marker set
To consider the differentiating power of a set of markers consisting of k markers, we first define the score of the marker set. A normal sample contains normal protooncogenes that promote cell growth and mitosis and tumor suppressor genes that discourage cell growth. During cancer development, protooncogenes can be mutated by carcinogenic agents to become oncogenes, which produce excessive levels of growth promoting proteins. Cancer results from cumulative mutations of protooncogenes and suppressor genes which together allow the unregulated growth of cells. Hence, cancer development involves uncontrolled cell division resulting from a series (progression) of gene mutations that typically involve two categories of function: promotion of cell division and inactivation of cell cycle suppression. The expression values of excessive growth genes tend to be much higher than the same genes of a normal sample. Similarly, the level of tumor suppressor genes of a cancer sample tends to be much lower than a normal sample. It is relatively rare that a single marker alone could offer sufficient power to differentiate all the classes well in multiclass data. So we consider marker sets with at least two markers keeping in mind that overgrowth of some genes and inactivation of other genes often happen together in cancer cases. Later stages of cancer involve tissue invasiveness, during which malignant cells travel among tissues via the circulatory and/or lymphatic system and grow and thrive in their new locations. Therefore the relative amount of two or multiple markers in a sample could be an indication of the cancer stages.
Score of marker pairs
Frequency counts of samples in each class for marker pairs.
E_{i} < E_{j}  E_{i} ≥ E_{j}  Total  

Class C _{ 1 }  f _{11ij}  f _{21ij}  n _{ 1 } = f _{ 11ij } +f _{ 21ij } 
⋮  ⋮  ⋮  ⋮ 
Class C _{ M }  f _{1Mij}  f _{2Mij}  n _{ M } = f _{ 1Mij } +f _{ 2Mij } 
Total  ${T}_{1}={\Sigma}_{m=1}^{M}{f}_{1mij}$  ${T}_{2}={\Sigma}_{m=1}^{M}{f}_{2mij}$  N 
where n_{ m } and T_{ q } are the row and column totals from the m^{ th } row and q^{ th } column, respectively. If all the counts in Table 1 are large and all cell counts are at least five, a traditional way to declare significance for the pair is to compare the calculated statistic with the chisquareddistribution with M 1 degrees of freedom. However, the significance level for declaring significance of a single test needs to be adjusted for multiple comparisons. There are various directions including familywise error rate control, falsediscovery rate (FDR) control, among others. The familywise error rate control tends to be conservative while the FDR control could lead to high false positive rates. In this work, we do not use the chisquareddistribution and do not decide how many pairs are significant. Instead, we only use the Chisquare statistic in (1) as an indication of how much departure from independence between the class and the chance of observing marker i expression value less than that of marker j. As the departure from independence increases, the chisquared statistic value increases. We select the top pair of markers that yield the highest value of the chisquared statistic. Additional marker selection will follow the algorithm in section 2.2.
Score of a kmarker set
Frequency counts of samples in each class for a set of k markers.
Class  ${E}_{{i}_{1}}<{E}_{{i}_{2}}$  ${E}_{{i}_{1}}\ge {E}_{{i}_{2}}$  ...  ${E}_{{i}_{k1}}<{E}_{{i}_{k}}$  ${E}_{{i}_{k1}}\ge {E}_{{i}_{k}}$  Total 

C _{ 1 }  ${f}_{11{i}_{1}{i}_{2}}$  ${f}_{21{i}_{1}{i}_{2}}$  ...  ${f}_{11{i}_{k1}{i}_{k}}$  ${f}_{21{i}_{k1}{i}_{k}}$  kn _{ 1 } 
⋮  ⋮  ⋮  ⋮  ⋮  ⋮  ⋮ 
C _{ M }  ${f}_{1M{i}_{1}{i}_{2}}$  ${f}_{2M{i}_{1}{i}_{2}}$  ...  ${f}_{1M{i}_{k1}{i}_{k}}$  ${f}_{2M{i}_{k1}{i}_{k}}$  kn _{ M } 
Total  ${T}_{1}={\Sigma}_{m=1}^{M}{f}_{1m{i}_{1}{i}_{2}}$  ${T}_{2}={\Sigma}_{m=1}^{M}{f}_{2m{i}_{1}{i}_{2}}$  ...  ${T}_{\left(\text{k}1\right)\text{k}/21}={\Sigma}_{m=1}^{M}{f}_{1m{i}_{1}{i}_{2}}$  ${T}_{\left(\text{k}1\right)\text{k}/2}={\Sigma}_{m=1}^{M}{f}_{2m{i}_{1}{i}_{2}}$  K N 
Note that the ${{\chi}^{2}}_{{i}_{1}...{i}_{k}}$only differs from ${\Sigma}_{a=1}^{k1}{\Sigma}_{b=a+1}^{k}{{\chi}^{2}}_{{i}_{a}{i}_{b}}$by the division factor k in the first term. So comparison of ${{\chi}^{2}}_{{i}_{1}...{i}_{k}}$ and ${{\chi}^{2}}_{{j}_{1}...{j}_{k}}$ for two sets of kmarkers {i_{ 1 },..., i_{ k }} and {j_{ 1 },..., j_{ k }} is equivalent to comparing ${\Sigma}_{a=1}^{k1}{\Sigma}_{b=a+1}^{k}{{\chi}^{2}}_{{i}_{a}{i}_{b}}$and ${\Sigma}_{a=1}^{k1}{\Sigma}_{b=a+1}^{k}{{\chi}^{2}}_{{j}_{a}{j}_{b}}$. The latter can be calculated easily without much computational cost after the statistics for marker pairs have been computed. The statistics given in equation (2) should be restricted to comparing marker sets with the same number of markers. All the kmarker sets can be ranked according to the magnitude of the Chisquare statistic values. The kmarker set with the highest Chisquare value is the most informative set among all kmarker sets.
Comparing marker sets of different sizes or identical Chisquare statistic
For comparing multiple sets with different numbers of markers, the Chisquare statistics given earlier can not be used because they accumulate different numbers of terms. In such case, we use the leaveoneout cross validation (LOOCV) accuracy within the training data obtained with the procedure below as the objective function.
 1.
Leave out one training sample S_{ l } to be used as the test data and use the rest of the training samples Ω_{tr}\ S_{ l } as training data.
 2.
For class m, (1 ≤ m ≤ M), assign S_{ l } to this class and calculate the Chisquare statistic for the marker set {i_{ 1 },..., i_{ k }}. We obtain M Chisquare statistics {${{\chi}^{2\left(m\right)}}_{{i}_{1}...{i}_{k}}$, m = 1,..., M}. The predicted class $\widehat{m}\left({S}_{l}\right)$for S_{ l } is the class that has the maximum Chisquare statistics, i.e., $\widehat{m}\left({S}_{l}\right)=\text{arg}\phantom{\rule{0.3em}{0ex}}\underset{1\le m\le M}{\text{max}}{{\chi}^{2\left(m\right)}}_{{i}_{1}...{i}_{k}}$.
 3.
Repeat 1 and 2 for all the samples in Ω_{tr} to get the prediction of all samples in Ω_{tr}: $\widehat{m}\left({S}_{l}\right)$, l = 1,..., N_{ tr }.
 4.
The LOOCV accuracy for marker set {i_{ 1 },..., i_{ k }} is LOOCV(i_{ 1 },..., i_{ k }) = the proportion of correctly classified samples in Ω_{tr}.
The comparison of marker sets of different sizes can be based on the LOOCV accuracy within the training data. The marker set with highest LOOCV is more informative than other marker sets. For marker sets with the same number of markers and identical Chisquare statistic value, we also use the LOOCV accuracy as a measure of their differentiating power toward the different cancer classes.
Marker selection algorithm
 1.
Calculate and record the Chisquare statistics for all marker pairs using the training data.
 2.
If the highest value of the Chisquare statistics is achieved by a unique marker pair, select this pair and denote it as TS_{2}. Calculate the LOOCV for this pair of markers and denote it as LOOCV_{2}. Update the remaining marker set ẞ by removing the marker pair selected.
 3.
If there are multiple marker pairs that have identical maximum Chisquare statistic value, calculate the LOOCV accuracy of these marker pairs using the training data. Keep the marker pairs that have the highest LOOCV accuracy. If the highest accuracy is achieved by more than one pairs, denote the different pairs as TS_{2,1}, TS_{2,2}, etc.
 4.
Find the top scoring triplets by adding additional marker to the top scoring pairs. This is done as follows. For each of the top scoring pairs resulting from 2 and 3, find the marker from the list of remaining markers ẞ such that the triplet has the highest Chisquare statistic value. If there are multiple triplets with identical maximum Chisquare value, calculate the LOOCV accuracy of these triplets and record those triplets that yield the highest LOOCV accuracy. Denote the top scoring triplet as TS_{3} if it is unique, and TS_{3,1}, TS_{3,2}, etc otherwise if multiple sets achieved identical accuracy. Record the LOOCV accuracy of the top scoring triplets.
 5.
For k = 4, 5,..., B, find TS_{k} and their corresponding LOOCV accuracy LOOCV_{k}. As k increases, the set TS_{k} tend to be unique.
 6.
Select the smallest kmarker set such that the LOOCV is maximized over all TS_{k}, k = 1,..., B. If the marker set is not unique, randomly select one of them as the final set. Denote the final selected informative marker set as TS_{G}, where G = arg max_{1 ≤ k ≤ B} LOOCV_{k}.
As discussed in section 2.1.2, the comparison of the Chisquare statistics for ${{\chi}^{2}}_{{i}_{1}...{i}_{k}}$can be simplified by comparing the summation of all Chisquare statistics from unique marker pairs ${\Sigma}_{a=1}^{k1}{\Sigma}_{b=a+1}^{k}{{\chi}^{2}}_{{i}_{a}{i}_{b}}$.
Our experience suggests that it is sufficient to set the upper bound B to be 50 if the total number of classes M ≤ 4, 100 if 5 ≤ M ≤ 8, and 150 if M ≥ 9.
Prediction with TSG classifier
To predict the class information for each sample in the test data, we use the selected marker set and calculate the scores of this sample belonging to each class. A large value for a class suggests that putting this sample in that class helps to increase the separation of different classes. The predicted class is set to be the one that has the largest score. In particular, suppose the selected marker set consists of markers m_{ 1 }, m_{ 2 },..., m_{ k }, the training data is Ω_{tr}, and the sample to be predicted is x_{new}. Let${{\chi}^{2}}_{{i}_{1}...{i}_{k}{C}_{i}}$be the Chisquare statistic value when we put the sample in class C_{ i }, i = 1,..., M. There are M Chisquare values. We assign the sample to the class with the largest Chisquare value:
Class of ${x}_{\text{new}}=\text{arg}\underset{i=1,...,M}{\text{max}}{{\chi}^{2}}_{{i}_{1}...{i}_{k}{}_{}{C}_{i}}$.
If multiple classes reach the same maximum Chisquare value, we further calculate the LOOCV accuracy for these classes. The final prediction is based on which class achieves the highest LOOCV accuracy.
Results and discussion
Microarray data and method of comparison
Binary class gene expression datasets
Dataset  Platform  No. of Genes  No. of samples in class I  No. of samples in class II  Source 

Colon  cDNA  2000  40(T)  22(N)  [18] 
Leukemia  Affy  7129  25(AML)  47(ALL)  [19] 
CNS  Affy  7129  25(C)  9(D)  [20] 
DLBCL  Affy  7129  58(D)  19(F)  [21] 
Lung  Affy  12533  150(A)  31(M)  [22] 
Prostate1  Affy  12600  52(T)  50(N)  [23] 
Prostate2  Affy  12625  38(T)  50(N)  [24] 
Prostate3  Affy  12626  24(T)  9(N)  [25] 
GCM  Affy  16063  190(C)  90(N)  [26] 
Multiclass gene expression datasets
Dataset  Platform  No. of Classes  No. of Genes  No. of samples in training  No. of samples in test  Source 

Leukemia1  Affy  3  7129  38  34  [19] 
Lung1  Affy  3  7129  64  32  [27] 
Leukemia2  Affy  3  12582  57  15  [28] 
SRBCT  cDNA  4  2308  63  20  [29] 
Breast  Affy  5  9216  54  30  [30] 
Lung2  Affy  5  12600  136  67  [31] 
DLBCL  cDNA  6  4026  58  30  [17] 
Leukemia3  Affy  7  12558  215  112  [32] 
Cancers  Affy  11  12533  100  74  [33] 
GCM  Affy  14  16063  144  46  [26] 
First, we consider comparison of TSG and kTSP classifiers for binary datasets based on 5fold cross validation. The subjects in each class are randomly partitioned into 5 parts, 4 of which form the training data and the rest of the subjects constitute the test data. The feature selection and modeling were conducted on the training data and prediction for each subject in the test data was given. For the results to be comparable to the TSP family classifiers, we also follow the same comparison methods as in Tan et al. [3]. In particular, we perform LOOCV for binary datasets and perform independent test for multiclass datasets. In the LOOCV, each sample is taken out and the remaining N1 samples are used to train the classifier, which is then used to predict the class label of the leaveout sample. The LOOCV accuracy is the proportion of correctly classified samples. Each of the multiclass datasets is partitioned into training and test data. We follow exactly the same partition scheme as in Tan et al. [3].
Since an objective of this research is to improve the TSP family classifiers, we present the percent of increase in classification accuracy in barplots. The percent of increase for TSG over TSP is defined as (accuracy of TSG  accuracy of TSP)/accuracy of TSP x 100%.
The percent of increase for any two classifiers are similarly defined. As TSP classifier uses two genes, kTSP and TSG use at least two genes, we are particularly interested in comparing the improvement in accuracy for TSG over TSP and kTSP over TSP in binary classifications. Similarly, in the multiclass cases, we are interested in comparing the increase in accuracy for TSG over HCTSP and HCkTSP over HCTSP.
For reference, we also include the classification accuracy of decision trees (DT), Naive Bayes (NB), knearest neighbor (kNN), Support Vector Machines (SVM) and prediction analysis of microarrays (PAM) in our comparison tables when they are available from the literature. These results were reported in Tan et al. [3] for leaveoneout cross validation for binary data and independent test for multiclass data. We include them only for convenience of discussion. DT and PAM have feature selection function while NB, kNN and SVM perform classification using the entire set of features. Since DT, kNN, and NB in general have lower accuracy than the other classifiers, we focus our discussion on other classifiers.
Accuracy for binary cancer data
In this section we present the comparison of the TSG marker selection algorithm with other algorithms using 9 benchmark binary class cancer expression datasets. These data have been analyzed extensively by many authors with wrapper, filtering and ensemble methods.
Comparison of TSG and kTSP classifiers based on 5fold cross validation
Average and standard error of accuracy from 5fold cross validation based on 10 runs.
TSG  kTSP  TSP  

Dataset  Average  Standard error  Average  Standard error  Average  Standard error 
CNS  81.1  1.61  77.0  1.48  67.0  2.48 
Colon  84.8  0.85  87.4  0.31  86.7  1.85 
DLBCL  95.2  0.69  94.6  0.88  92.9  2.25 
GCM  81.7  0.66  81.9  0.41  76.0  1.04 
Leukemia  93.7  0.75  91.8  1.13  89.4  1.56 
Lung  100  0  98.6  0.15  96.3  0.69 
Prostate1  90.2  0.72  91  0.75  89.3  1.27 
Prostate2  74.4  0.6  77.9  0.69  70.4  2.25 
Prostate3  100  0  95.4  0.79  95.1  1.44 
Average  89.0  88.4  84.8 
In five of the nine datasets (CNS, DLBCL, Leukemia, Lung, Prostate3), TSG has slightly better performance than kTSP. In the remaining four datasets, kTSP has slightly better performance. On average, TSG and kTSP have comparable 5fold cross validation performance. The kTSP improvement over TSP is also marginal in this 5fold cross validation setting for most of the datasets.
LOOCV accuracy
Among the LOOCV accuracy reported in the literature, we find that TSP, kTSP, PAM and SVM are often the top performing classifiers.
LOOCV accuracy and the number of genes used in classifiers (in parenthesis) for binary class expression datasets
Method  Colon  Leuk  CNS  DLBCL  Lung  Pros1  Pros2  Pros3  GCM  Aver 

TSG^{†}  93.55 (2)  98.61 (2)  97.06 (2)  98.7 (2)  100 (2)  95.1 (2)  86.36 (10)  100 (2)  87.5 (7)  95.21 
TSP*  91.10 (2)  93.80 (2)  77.90 (2)  98.10 (2)  98.30 (2)  95.10 (2)  67.60 (2)  97.00 (2)  75.40 (2)  88.26 
kTSP*  90.30 (2)  95.83 (18)  97.10** (10)  97.40 (2)  98.90 (10)  91.18 (2)  75.00 (18)  97.00 (2)  85.40 (10)  92.01 
DT*  77.42 (3)  73.61 (2)  67.65 (2)  80.52 (3)  96.13 (3)  87.25 (4)  64.77 (4)  84.85 (1)  77.86 (14)  78.90 
NB*^{‡}  56.45  100  82.35  80.52  97.79  62.75  73.86  90.91  84.29  80.99 
kNN*^{‡}  74.19  84.72  82.35  89.61  98.34  74.51  73.86  93.94  86.79  84.26 
SVM*^{‡}  82.26  98.61  82.35  97.40  99.45  91.18  76.14  100  93.21  91.18 
PAM*  89.52 (15)  94.03 (2296)  82.35 (4)  85.45 (17)  97.90 (9)  90.89 (47)  81.25 (13)  94.24 (701)  82.32 (47)  88.66 
Accuracy of independent test for multiclass cancer data
Accuracy of classifiers and the number of genes used in classifiers (in parenthesis) for the independent test set for multiclass expression datasets
Method  Leuk1  Lung1  Leuk2  SRBCT  Breast  Lung2  DLBCL  Leuk3  Cancers  GCM  Aver 

TSG^{†}  97.06 (6)  81.25 (20)  100 (44)  100 (13)  86.67 (63)  95.52 (60)  93.33 (16)  91.07 (95)  79.73 (81)  67.39 (112)  89.20 
HCTSP*  97.06 (4)  71.88 (4)  80 (4)  95 (6)  66.67 (8)  83.58 (8)  83.33 (10)  77.68 (12)  74.32 (20)  52.17 (26)  78.17 
HCkTSP*  97.06 (36)  78.13 (20)  100 (24)  100 (30)  66.67 (24)  94.03 (28)  83.33 (46)  82.14 (64)  82.43 (128)  67.39 (134)  85.12 
DT*  85.29 (2)  78.13 (4)  80 (2)  75 (3)  73.33 (4)  88.06 (5)  86.67 (5)  75.89 (16)  68.92 (10)  52.17 (18)  76.35 
NB^{‡}*  85.29  81.25  100  60  66.67  88.06  86.67  32.14  79.73  52.17  73.2 
kNN^{‡}*  67.65  75  86.67  30  63.33  88.06  93.33  75.89  64.86  34.78  67.96 
PAM*  97.06 (44)  78.13 (13)  93.33 (62)  95 (285)  93.33 (4822)  100 (614)  90 (3949)  93.75 (3338)  87.84 (2008)  56.52 (1253)  88.5 
1vs1SVM^{‡}*  79.41  87.5  100  100  83.33  97.01  100  84.82  83.78  65.22  88.11 
The accuracy based on independent test samples for TSG, HCkTSP, and SVM relative to the accuracy of PAM is presented in the right panel of Figure 3. For half of the datasets SVM has better performance than PAM and for the other half of the datasets, PAM performs better than SVM in accuracy. So on average, SVM and PAM are comparable. The accuracy of HCkTSP in majority of datasets is below that of PAM. On average, HCkTSP has accuracy slightly lower than PAM. TSG has better accuracy than PAM in five of ten datasets, and TSG is not as accurate as PAM in four datasets. On average, TSG, PAM and SVM have comparable performance with TSG only slightly better. In summary, for multiclass problems, TSG, SVM and PAM are better than HCkTSP in accuracy.
The number of genes used in classifiers
It can be seen from Figure 5 that TSP classifier has the lowest number of genes in binary data classification. PAM used much more number of genes than other classifiers. The classifier that has the second smallest number of genes in binary data classification is TSG, followed by DT that uses the third smallest number of genes. The kTSP classifier in general used more number of genes than TSP, TSG, and DT.
For multiclass data, DT uses the least number of genes followed by HCTSP in the second place. However, as discussed in Section 3.1, DT and HCTSP are not as accurate as SVM, PAM, HCkTSP, and TSG. In 4 out of 10 datasets TSG used more genes than HCkTSP and in 5 datasets TSG used fewer genes than HCkTSP. On average across all 10 datasets, TSG uses 51 genes and HCkTSP uses 53.4 genes. PAM consistently uses a lot more genes than other classifiers except in Lung1 data. In fact, the number of genes used by PAM is in the magnitude of thousands in order to reach comparable accuracy as TSG. Recall that on average TSG, PAM and SVM have comparable accuracy for independent test data and HCkTSP has lower performance. Now combining the accuracy and the number of genes used, TSG outperforms the rest in that it uses smaller number of genes to reach high accuracy. Smaller number of genes makes it feasible to perform followup studies and further experimental verification after the informative genes selection.
Interpretation of the TSG classifier
The TSG classifier has an easy interpretation. Recall that the main rule to classify a sample is based on which class gives the highest Chisquare statistic value when the sample is assigned to that class (see section 2.3). Note that there is a one to one correspondence between the Chisquare statistic values and the degree of departure from independence between the class label and the column variables in Table 2. The bigger the Chisquare statistic, the further the departure is from independence and vice versa. So the class label assignment tries to maximize the dependence between the classes and the selected variables for a given set of training data and a test sample.
The TSG classifier has the same interpretation as the TSP classifier. In particular, suppose the expression values for these two genes are E_{1} and E_{2} for a sample, the prediction of the class label of this sample depends on whether E_{1} < E_{2}. For Leuk, CNS, and pros3 data, TSG achieved accuracy 98.61, 97.06, and 100 for these three datasets respectively using 2 genes. Also using 2 genes, TSP only achieved 93.80, 77.90, 97.00 in LOOCV accuracy for these three datasets. Therefore, TSG finds genes that are even more informative than TSP classifier in these three datasets. When the selected informative genes have more than 2 genes, then the class prediction of a sample depends on all pairwise comparisons of the expression values ${E}_{{i}_{1}},\cdots \phantom{\rule{0.3em}{0ex}},{E}_{{i}_{k}}$ for the selected genes i_{1}, i_{2},..., i_{k} from this sample, i.e., which of the inequalities ${E}_{{i}_{1}}<{E}_{{i}_{2}}$, ${E}_{{i}_{1}}\ge {E}_{{i}_{2}}$,......, ${E}_{{i}_{k1}}<{E}_{{i}_{k}}$, ${E}_{{i}_{k1}}\ge {E}_{{i}_{k}}$ are true. Due to the similar interpretation to the TSP family classifiers, we do not reiterate for TSG and refer the readers to Tan et al. [3] for details.
Conclusions
In this article, we presented the TSG classifier, an improved version of TSP family classifiers for both binary and multiclass cancer classification. The TSP family classifiers only consider even number of genes and the gene selection process is based on the marginal comparison of pairwise expression values without honoring the fact that some of the marginally important genes may have similar effects as others and therefore could be redundant. We solved aforementioned shortcomings of TSP family classifiers by allowing both even and odd number of genes through newly defined score functions and a new selection algorithm. After some genes have been selected, our gene selection process assesses the importance of additional genes by considering the overall contribution of all the genes included in the informative set. As the joint effects of multiple genes are evaluated together, we expect that the final list of genes selected by TSG is more parsimonious than kTSP and HCkTSP classifiers.
The TSG classifier is in a simple unified form for both binary and multiclass cases. This is different from the TSP family classifiers in that three binary to multiclass extension schemes (1vs1, 1vsothers, hierarchical classification) lead to three different classifiers. As reported in Tan et al [3], the hierarchical classification scheme extension HCTSP and HCkTSP perform the best out of the three schemes. Our TSG classifier is in a single form and in general has equal performance or outperforms kTSP, HCTSP and HCkTSP in the 19 datasets in terms of accuracy and number of genes used. We also compared the performance of TSG with PAM and SVM. In binary classification problems, TSG has better LOOCV accuracy than PAM and SVM; in multiclass problems, TSG, PAM, and SVM give comparable accuracy for independent test data. All three classifiers are more accurate than TSP family classifiers. In terms of the number of genes used, TSG clearly uses much less number of genes than PAM and SVM. PAM often selects thousands of genes in its final classifier and SVM uses the entire set of genes.
An obvious advantage of the TSG as well as the TSP family classifiers is that they are based on the simple pairwise comparisons of expression values between genes from the same sample. Such comparison is robust to monotone transformation and eliminates the concern about variations among different patients, platforms, or bias from preprocessing different samples. Therefore, we expect that the results from TSG are more reliable and robust compared to many other methods that pool the data from different samples to filter genes.
Notes
List of abbreviations used
 TSP:

top scoring pair
 kTSP:

k top scoring pairs
 HCTSP:

multiclass extension of TSP with hierarchical classification scheme
 HCkTSP:

multiclass extension of kTSP with hierarchical classification scheme
 SVM:

Support Vector Machine classification
 PAM:

Prediction Analysis of Microarray
 LOOCV:

leaveoneout cross validation.
Declarations
Acknowledgements
The research was supported by a grant from the Science Foundation for Distinguished Young Scholars of Hunan Province, China (No. 10JJ1005) to ZY, the Scientific Research Fund of Hunan Provincial Education Department (No. 11C0654) to HZ, and a grant from the Simons Foundation to HW (#246077).
This article has been published as part of BMC Medical Genomics Volume 6 Supplement 1, 2013: Proceedings of the 2011 International Conference on Bioinformatics and Computational Biology (BIOCOMP'11). The full contents of the supplement are available online at http://www.biomedcentral.com/bmcmedgenomics/supplements/6/S1. Publication of this supplement has been supported by the International Society of Intelligent Biological Medicine.
Authors’ Affiliations
References
 Tibshirani R, Hastie T, Narasimhan B, Chu G: Diagnosis of multiple cancer types by shrunken centroids of gene expression. Proc Natl Acad Sci USA. 2002, 99: 65676572. 10.1073/pnas.082099299.PubMed CentralView ArticlePubMed
 Geman D, d'Avignon C, Naiman D, Winslow R: Classifying Gene Expression Profiles from Pairwise mRNA Comparisons. Statistical Applications in Genetics and Molecular Biology. 2004, doi: 10.2202/15446115.1071
 Tan AC, Naiman DQ, Xu L, Winslow RL, Geman D: Simple decision rules for classifying human cancers from gene expression profiles. Bioinformatics. 2005, 21: 38963904. 10.1093/bioinformatics/bti631.PubMed CentralView ArticlePubMed
 Chang CC, Lin CJ: LIBSVM: a library for support vector machines. ACM Transactions on Intelligent, Systems and Technology. 2011, 2 (27): 127.View Article
 Li T, Zhang C, Ogihara M: A comparative study of feature selection and multiclass classification methods for tissue classification based on gene expression. Bioinformatics. 2004, 20: 24292437. 10.1093/bioinformatics/bth267.View ArticlePubMed
 Yang K, Cai Z, Li J, Lin G: A stable gene selection in microarray data analysis. BMC Bioinformatics. 2006, 7: 22810.1186/147121057228.PubMed CentralView ArticlePubMed
 Ding C, Peng H: Minimum redundancy feature selection from microarray gene expression data. J Bioinform Comput Biol. 2005, 3 (2): 185205. 10.1142/S0219720005001004.View ArticlePubMed
 Ooi CH, Chetty M, Teng SW: Differential prioritization between relevance and redundancy in correlationbased feature selection techniques for multiclass gene expression data. BMC Bioinformatics. 2006, 7: 32010.1186/147121057320.PubMed CentralView ArticlePubMed
 Zhang JG, Deng HW: Gene selection for classification of microarray data based on the bayes error. BMC Bioinformatics. 2007, 8 (1): 3710.1186/14712105837.View Article
 Wei X, Li K: Exploring the within and between class correlation distributions for tumor classification. Proc Natl Acad Sci USA. 2010, 107 (15): 67376742. 10.1073/pnas.0910140107.PubMed CentralView ArticlePubMed
 Liu Q, Sung A, Chen Z, Chen L, Liu J, Qiao M, Wang Z, Huang X, Deng Y: Gene selection and classification for cancer microarray data based on machine learning and similarity measures. BMC Genomics. 2011, 12 (Suppl 5): S110.1186/1471216412S5S1. doi:10.1186/1471216412S5S1PubMed CentralView ArticlePubMed
 Guyon I, Weston J, Barnhill S, Vapnik V: Gene selection for cancer classification using support vector machines. Machine learning. 2002, 46: 389422. 10.1023/A:1012487302797.View Article
 Rakotomamonjy A: Variable selection using svm based criteria. J Mach Learn Res. 2003, 3: 13571370.
 DίazUriarte R, Alvarez de Andrés S: Gene selection and classification of microarray data using random forest. BMC Bioinformatics. 2006, 7: 310.1186/1471210573. doi:10.1186/1471210573View Article
 Ho TK: The random subspace method for constructing decision forests. IEEE Transactions on Pattern Analysis and Machine Intelligence. 1998, 20 (8): 832844. 10.1109/34.709601.View Article
 Li X, Zhao H: Weighted random subspace method for high dimensional data classification. Statistics and its Interface. 2009, 2: 153159.PubMed CentralView ArticlePubMed
 Alizadeh AA, Eisen MB, Davis RE, Ma C, Lossos IS, Rosenwald A, Boldrick JC, Sabet H, Tran T, Yu X, Powell JI, Yang L, Marti GE, Moore T, Hudson J, Lu L, Lewis DB, Tibshirani R, Sherlock G, Chan WC, Greiner TC, Weisenburger DD, Armitage JO, Warnke R, Levy R, Wilson W, Grever MR, Byrd JC, Botstein D, Brown PO, Staudt LM: Distinct types of diffuse large Bcell lymphoma identified by gene expression profiling. Nature. 2000, 403: 503511. 10.1038/35000501.View ArticlePubMed
 Alon U, Barkai N, Notterman DA, Gish K, Ybarra S, Mack D, Levine AJ: Broad patterns of gene expression revealed by clustering analysis of tumor and normal colon tissues probed by oligonucleotide arrays. Proc Natl Acad Sci USA. 1999, 96: 67456750. 10.1073/pnas.96.12.6745.PubMed CentralView ArticlePubMed
 Golub TR, Slonim DK, Tamayo P, Huard C, Gaasenbeek M, Mesirov JP, Coller H, Loh ML, Downing JR, Caligiuri MA, Bloomfield CD, Lander ES: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science. 1999, 286 (5439): 531537. 10.1126/science.286.5439.531.View ArticlePubMed
 Pomeroy SL, Tamayo P, Gaasenbeek M, Sturla LM, Angelo M, McLaughlin ME, Kim JY, Goumnerova LC, Black PM, Lau C, Allen JC, Zagzag D, Olson JM, Curran T, Wetmore C, Biegel JA, Poggio T, Mukherjee S, Rifkin R, Califano A, Stolovitzky G, Louis DN, Mesirov JP, Lander ES, Golub TR: Prediction of central nervous system embryonal tumour outcome based on gene expression. Nature. 2002, 415: 436442. 10.1038/415436a.View ArticlePubMed
 Shipp MA, Ross KN, Tamayo P, Weng AP, Kutok JL, Aguiar RC, Gaasenbeek M, Angelo M, Reich M, Pinkus GS, Ray TS, Koval MA, Last KW, Norton A, Lister TA, Mesirov J, Neuberg DS, Lander ES, Aster JC, Golub TR: Diffuse large Bcell lymphoma outcome prediction by geneexpression profiling and supervised machine learning. Nat Med. 2002, 8: 6874. 10.1038/nm010268.View ArticlePubMed
 Gordon GJ, Jensen RV, Hsiao LL, Gullans SR, Blumenstock JE, Ramaswami S, Richards WG, Sugarbaker DJ, Bueno R: Translation of microarray data into clinically relevant cancer diagnostic tests using gene expression ratios in lung cancer and mesothelioma. Cancer Res. 2002, 62: 49634967.PubMed
 Singh D, Febbo PG, Ross K, Jackson DG, Manola J, Ladd C, Tamayo P, Renshaw AA, D'Amico AV, Richie JP, Lander ES, Loda M, Kantoff PW, Golub TR, Sellers WR: Gene expression correlates of clinical prostate cancer behaviour. Cancer Cell. 2002, 1: 203209. 10.1016/S15356108(02)000302.View ArticlePubMed
 Stuart RO, Wachsman W, Berry CC, WangRodriguez J, Wasserman L, Klacansky I, Masys D, Arden K, Goodison S, McClelland M, Wang Y, Sawyers A, Kalcheva I, Tarin D, Mercola D: In silico dissection of celltypeassociated patterns of gene expression in prostate cancer. Proc Natl Acad Sci USA. 2004, 101: 615620. 10.1073/pnas.2536479100.PubMed CentralView ArticlePubMed
 Welsh JB, Sapinoso LM, Su AI, Kern SG, WangRodriguez J, Moskaluk CA, Frierson HF, Hampton GM: Analysis of Gene Expression Identifies Candidate Markers and Pharmacological Targets in Prostate Cancer. Cancer Res. 2001, 61: 59745978.PubMed
 Ramaswamy S, Tamayo P, Rifkin R, Mukherjee S, Yeang CH, Angelo M, Ladd C, Reich M, Latulippe E, Mesirov JP, Poggio T, Gerald W, Loda M, Lander ES, Golub TR: Multiclass cancer diagnosis using tumor gene expression signatures. Proc Natl Acad Sci USA. 2001, 98: 1514915154. 10.1073/pnas.211566398.PubMed CentralView ArticlePubMed
 Beer DG, Kardia SL, Huang CC, Giordano TJ, Levin AM, Misek DE, Lin L, Chen G, Gharib TG, Thomas DG, Lizyness ML, Kuick R, Hayasaka S, Taylor JM, Iannettoni MD, Orringer MB, Hanash S: Geneexpression profiles predict survival of patients with lung adenocarcinoma. Nat Med. 2002, 8: 816824.PubMed
 Armstrong SA, Staunton JE, Silverman LB, Pieters R, den Boer ML, Minden MD, Sallan SE, Lander ES, Golub TR, Korsmeyer SJ: MLL translocations specify a distinct gene expression profile that distinguishes a unique leukemia. Nat Genet. 2002, 30: 4147. 10.1038/ng765.View ArticlePubMed
 Khan J, Wei JS, Ringnér M, Saal LH, Ladanyi M, Westermann F, Berthold F, Schwab M, Antonescu CR, Peterson C, Meltzer PS: Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat Med. 2001, 7: 673679. 10.1038/89044.PubMed CentralView ArticlePubMed
 Perou CM, Sørlie T, Eisen MB, van de Rijn M, Jeffrey SS, Rees CA, Pollack JR, Ross DT, Johnsen H, Akslen LA, Fluge O, Pergamenschikov A, Williams C, Zhu SX, Lønning PE, BørresenDale AL, Brown PO, Botstein D: Molecular portraits of human breast tumours. Nature. 2000, 406: 747752. 10.1038/35021093.View ArticlePubMed
 Bhattacharjee A, Richards WG, Staunton J, Li C, Monti S, Vasa P, Ladd C, Beheshti J, Bueno R, Gillette M, Loda M, Weber G, Mark EJ, Lander ES, Wong W, Johnson BE, Golub TR, Sugarbaker DJ, Meyerson M: Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proc Natl Acad Sci USA. 2001, 98: 1379013795. 10.1073/pnas.191502998.PubMed CentralView ArticlePubMed
 Yeoh EJ, Ross ME, Shurtleff SA, Williams WK, Patel D, Mahfouz R, Behm FG, Raimondi SC, Relling MV, Patel A, Cheng C, Campana D, Wilkins D, Zhou X, Li J, Liu H, Pui CH, Evans WE, Naeve C, Wong L, Downing JR: Classification, subtype discovery, and prediction of outcome in pediatric acute lymphoblastic leukemia by gene expression profiling. Cancer Cell. 2002, 1: 133143. 10.1016/S15356108(02)000326.View ArticlePubMed
 Su AI, Welsh JB, Sapinoso LM, Kern SG, Dimitrov P, Lapp H, Schultz PG, Powell SM, Moskaluk CA, Frierson HF, Hampton GM: Molecular classification of human carcinomas by use of gene expression signatures. Cancer Res. 2001, 61: 73887393.PubMed
Copyright
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.