Cosmal.ucsd.edu

& 2006 International Society of Nephrology A novel approach for accurate prediction ofspontaneous passage of ureteral stones: Supportvector machines F Dal Moro1,3, A Abate2,3, GRG Lanckriet2,3, G Arandjelovic1, P Gasparella1, P Bassi1, M Mancini1and F Pagano1 1Department of Urology, University of Padova, Padova, Italy and 2Electrical Engineering and Computer Sciences Department, Universityof California at Berkeley, Berkeley, California, USA The objective of this study was to optimally predict the Referring to the statistics on the incidence of kidney stone spontaneous passage of ureteral stones in patients with renal disease in industrialized countries, we understand how colic by applying for the first time support vector machines important it is to correctly analyze this pathology in order (SVM), an instance of kernel methods, for classification. After to predict accurately which patients need what sort of reviewing the results found in the literature, we compared intervention. Everybody agrees that considering the stone size the performances obtained with logistic regression (LR) and is the most important factor for predicting the spontaneous accurately trained artificial neural networks (ANN) to those passage of calculi.1,2 However, this does not seem discrimi- obtained with SVM, that is, the standard SVM, and the linear native enough when calculi are of mid-size dimensions. At programming SVM (LP-SVM); the latter techniques show an this stage, the urologist needs more information in order to improved performance. Moreover, we rank the prediction take a valid clinical decision, but there is no demonstrated factors according to their importance using Fisher scores and result as to which factor should be considered first and what the LP-SVM feature weights. A data set of 1163 patients are the actual interactions between all the factors.3 affected by renal colic has been analyzed and restricted to In literature, the statistical methodologies employed have single out a statistically coherent subset of 402 patients. Nine been the multivariate logistic regression (LR)4 and the clinical factors are used as inputs for the classification artificial neural network (ANN).5 In this work, we propose algorithms, to predict one binary output. The algorithms are to use the recently developed support vector machines cross-validated by training and testing on randomly selected (SVM),6–8 an instance of kernel methods, for classification, as train- and test-set partitions of the data and reporting the well as linear programming SVM (LP-SVM);9 these are in average performance on the test sets. The SVM-based general believed to outperform the ANN.8,10 approaches obtained a sensitivity of 84.5% and a specificity The paper will unfold as follows: along with a critical of 86.9%. The feature ranking based on LP-SVM gives the analysis of the results presented in medical literature – with a highest importance to stone size, stone position and special focus on the ANN – we describe how the statistical symptom duration before check-up. We propose a tests are performed. Critical results follow, and a discussion statistically correct way of employing LR, ANN and SVM for on their significance, both technically and clinically, is the prediction of spontaneous passage of ureteral stones developed. Conclusions mark the state of the art of our in patients with renal colic. SVM outperformed ANN, as well work, and define some future directions of our research.
as LR. This study will soon be translated into a practicalsoftware toolbox for actual clinical usage.
Kidney International (2006) 69, 157–160. doi:10.1038/sj.ki.5000010 Figure 1 plots the achievable true positive (TP) rate (i.e., KEYWORDS: urolithiasis; ureteral calculi; support vector machine; artificial sensitivity) versus true negative rate (TN) (i.e., specificity) for intelligence; statistical methods; neural networks the different learning algorithms. Each of the four plotscorresponds to a different learning algorithm. Each dotwithin a plot corresponds to the average test-set performanceobtained for a certain setting of the algorithm’s ‘hyper-parameters’, that is, parameters that are a priori chosen and Correspondence: FD Moro, Department of Urology, University of Padova are endogenous to the actual training procedure. The choice Medical School, Via Giustiniani, 2, Padova I-35128, Italy. E-mail: fabrizio.
of SVM and LP-SVM reflects the relative importance the training algorithm should give to false positives versus false 3These authors contributed equally to this work.
negatives. For the ANN and LR, these parameters are, Received 7 November 2004; revised 16 May 2005; accepted 8 July 2005 respectively, related to the actual structure of the network or Kidney International (2006) 69, 157–160 F Dal Moro et al.: Accurate prediction of spontaneous passage of ureteral stones to more technical training issues (weights and thresholds, for and finally on the first five in the ranking. For both rankings, similar results were obtained. Using stone size only led to The best results in prediction accuracy were singled out, acceptable results. Using more inputs increased the perfor- picking up a point at the upper-right-most part of each of the mance, whereas using just the five most important inputs was four plots (see arrows); using the old method of multivariate qualitatively equivalent to the results obtained using all LR, the outcome showed 90.3% sensitivity and 69.7% inputs. Therefore, we concluded that the remaining four specificity (Figure 1a). The ANN matched this performance clinical factors introduce spurious information and, in this with 94.9% sensitivity and 62.9% specificity (Figure 1b).
specific setting, can be regarded as redundant.
When using an SVM, 84.5% sensitivity and 86.9% specificitycould be obtained (Figure 1d). LP-SVM presented results that were on the upper rim of the SVM performance (Figure 1c).
Let us first list and highlight the main pitfalls of the results Again, it was possible to associate to each and every point of presented in literature, which have mostly been obtained with this plot a single combination of all the hyper-parameters of the aid of ANN.11 First of all, the used data sets are often of relatively low cardinality, a condition that is more likely to With respect to our second objective, ranking the input provide poor results or unstable prediction algorithms.12,13 factors, Table 1 shows the ranking obtained using Fisher Second, the ANN results in literature are based on using scores and LP-SVM weights, respectively. As both ranking only one hold-out test set and hence so the reported approaches are essentially different, we should not necessarily performance depends heavily on the particular test set that expect the rankings to be similar. However, when inspecting is used. Therefore, training and testing should be performed the results, we saw a rather high overlap within the top five more than once and the test-set performances averaged out, values of both rankings (the factor identified as most to reduce the variance of the performance estimate. Whereas significant being the same and three factors from the top most literature ignores this fact, we applied cross-validation five overlapping in both results). This certainly advocates the and averaged the performance over 30 randomly chosen test robustness and significance of the obtained outcomes.
sets, as mentioned before. Therefore, we performed statisti- Moreover, these rankings were validated by simulations using cally more accurate tests for all our learning algorithms, only the more relevant inputs. More precisely, we set up and ran the training/testing procedure first on the most Third, we strongly question the ANN results concerning prominent input, then on the two most influential inputs the input rankings: it is known that networks with a structurethat is more complex than that of a perceptron (i.e., with oneor more hidden layers), offer no clear connection betweentheir weights and the relative relevance of their inputs.14,5 Also, it is wrong to look at the absolute values of the weights of even a perceptron when the inputs are not normalized.12,15 We resolve the pitfall of ANN not allowing the determination of the relative importance of the clinical factors by using Fisher scores and the LP-SVM approach.
Table 1 | Classes of importance of the spontaneous stone expulsion factors second to different methods Figure 1 | Comparison of the average test-set performances for the four learning algorithms run on normalized data. (a–d) The axes represent specificity and sensitivity. Each dot within a plotcorresponds to the average test-set performance obtained for acertain setting of algorithm hyper-parameters that are endogenous to the actual training procedure. As stated in the literature, ANN slightly improves the results obtained through LR, while the kernel algorithms outperform the other two methods.
Kidney International (2006) 69, 157–160 F Dal Moro et al.: Accurate prediction of spontaneous passage of ureteral stones As for the particular strength of the SVM approach, we cortisone or alpha-blocker agents (i.e., Tamsulosin), prior to first pointed out the broad range of performances that could and/or after the colic episode, were already excluded before: be achieved in the specificity/sensitivity plane (Figure 1), by in fact, the efficacy of these treatments has been proven by varying the SVM hyper-parameter settings. This gave rise to a curve that was similar to a receiver operating characteristic The fact that the stone size is by and large the most curve, although more specialized. A usual receiver operating influential factor explains why the LR (linear) results are not characteristic curve would be obtained from one set of too far from those obtained with the (nonlinear) ANN.
classifier weights, using the known testing-threshold shift. In Nevertheless, the SVM approach is still able to infer deeper this case, each dot corresponds to a different set of classifier relationships between inputs and outputs, resulting in a weights, obtained from SVM training for a specific hyper- better performance, and therefore represents the method of parameter setting. These plots show how flexible the SVM is in terms of specificity/sensitivity trade-off. The ANN and LRoffer a lot less flexibility.
In the case of ANN, we varied several training parameters, This work proposes the application of the SVM to drastically resulting in only a small variation in TP and TN rates, improve the prediction results for intervention on renal colic although enough to still improve on the prediction accuracy obtained in the literature. The new results, which outclass those obtained via LR and the ANN approach, are The points referring to the SVM, being widely spread particularly interesting from a clinical perspective, as they through the TP/TN plot, show how this method can be more maintain the ANN level of sensitivity (i.e., correctly predicting that no intervention is needed) while improving The SVM prediction improves LR and ANN significantly significantly on the specificity (i.e., correctly predicting the along the specificity axis. This, important from a statistical need for an intervention). The authors are willing to translate standpoint, also has a sharp clinical meaning: a wrong these algorithms into a software toolbox, which would then prediction in terms of specificity would result in the patient help physicians on their fieldwork. This is the first time an missing an invasive intervention, which would effectively instance of kernel methods, that is, the SVM, has been be needed. Thus, it is clear that the best prediction of applied with success to such clinical data. Intelligent systems spontaneous stone passage will be one that combines an such as this could markedly reduce costs of therapeutical outstanding sensitivity with a remarkable specificity. The approaches and recoveries for kidney stone disease. Given the SVM approach offers a great variety of predictive sensitivity/ outstanding performance of SVMs, their application in other specificity combinations, depending on the setting of its fields of urology, such as the oncological field, is imminent.
hyper-parameters. If we consider a possible optimal opera-tion point (corresponding to a specific hyper-parameter setting), that is, 84.5% sensitivity and 86.9% specificity, the We gathered and sorted the information collected from 1163 SVM approach shows significantly better results than those patients who were treated for an episode of renal colic in the period from January to December 2003 in the Urology Institute of the Focusing on the problem of input ranking, we notice how Hospital of Padova, Italy. A focused selection of the patients was the results obtained with the Fisher scores make sense from a made on the basis of some important criteria. The patients excluded clinical point of view. In earlier work, the ranking, computed with ANN, gave questionable results.12,15 The classification patients in whom the colic episode was due to renal calculi; obtained with LP-SVM was similar to the first. We compared patients in whom the actual show-up or expulsion of the calculi the results obtained by those two methods by splitting the spontaneous passage factors in three groups of decreasing patients treated with Ca antagonists, cortisone or a-litics in the importance according to the weights we obtained, so that we 3 months previous and/or after the colic episode; could ponder over their clinical value.
patients with anatomic malformations of the excretory tract; Simulations with an increasing number of input features transplanted or mono-kidney patients, under more aggressivetherapy; improved until the ‘heaviest’ five inputs were used, the latter patients with more than one ureteral calculi; leading to results equivalent to those obtained when using all patients in whom the rigorous follow-up at the 3-month check- input factors. This means that the last four inputs do not add up from the episode was not possible; and any further information to the prediction problem and can patients who, after the axcess to emergency unit underwent extracorporeal shock wave litothripsy (ESWL), endourological The hydration and the medical therapy can increase the or surgical procedures for stone removal.
rate of spontaneous stone passage, but they were not taken Out of 1163 patients with pieloureteral colic, 402 were found into consideration as parameters. That is because it is valuable for experiment, as summarized in Figure 2.
common praxis, when hydration is considered, to advise each Furthermore, for the actual statistical tests, we considered patient with renal colic a minimum 2–3 l of water intake a diagnostic criteria for the renal colic such as spontaneous expulsion day. Patients who underwent treatment with Ca antagonists, (as reported by the patient), colic treatments together with ESWL, Kidney International (2006) 69, 157–160 F Dal Moro et al.: Accurate prediction of spontaneous passage of ureteral stones with ANN or a standard SVM. The latter is based on a methodology known as kernel-based learning,7,8 which allows one to come upwith nonlinear versions of many well-known linear statisticalalgorithms. In the case of SVM, the kernel methodology is used to obtain the nonlinear SVM algorithm, derived from a linear maximalmargin classifier. The algorithms were implemented using MA- TLABs and commercial optimization software Moseks.
The second objective, ranking the clinical factors according to their importance, was addressed in two ways. First, by using Fisher Treatment with Ca antagonists,cortison, antagonists scores: these scores are computed as the difference in means of thefactor values, computed for each class (i.e., input), corrected by their variance within each class; these scores therefore analyze theimportance of every input factor independently. Second, we used the explicit LP-SVM feature weights:8 these weights were obtainedfrom the training algorithm, looking at all factors simultaneously and thus taking the dependence between the different inputs into Segura JW, Preminger GM, Assimos DG et al. Ureteral stones clinicalguidelines panel summary report on the management of ureteral calculi.
Figure 2 | Scheme for the selection of patients.
Anagnostu T, Tolley D. Management of ureteric stones. Eur Urol 2004; 45:714–721.
imaging showing ureteral calculi and clinical findings of the Miller OF, Kane CJ. Time to stone passage for observed ureteral calculi: a physician during the colic episode. As already mentioned, all the guide for patient education. J Urol 1999; 162: 688–690.
patients who, after the excess in the emergency unit, underwent Parekattil SJ, White MD, Moran ME, Kogan BA. A computer model to ESWL, endourological or surgical treatment for the stone removal predict the outcome and duration of ureteral or renal calculous passage.
J Urol 2004; 171: 1436–1439.
were excluded. The interval between first renal colic and stone Ramesh AN, Kambhampati C, Monson JR, Drew PJ. Artificial intelligence passage was 6 months. In total, we considered nine clinically in medicine. Ann R Coll Surg Engl 2004; 86: 334–338.
important factors (i.e., ‘inputs’) for each of the 408 patients (i.e., Cristianini N, Schoelkopf B. Support vector machines and kernel methods.
‘data points’). We selected the factors among those referred to as Boser BE, Guyon I, Vapnik V. A Training algorithm for optimal margin most influential in medical literature: age, sex, body mass index, classifiers. Proc Comput Learn Theory 1992: 144–152, ACM Press.
fever, previous urological treatments, previous expulsion of stones, Cristianini N, Shawe-Taylor J. An Introduction to Support Vector Machines.
duration of the symptoms (in hours), dimension and position of the Cambridge University Press: Cambridge; 2000.
stone.18 With each patient is also associated a ‘binary output’ value, Bradley PS, Mangasarian OL, Street WN. Feature selection viamathematical programming. INFORMS J Comput 1998; 10: 209–217.
corresponding to two classes of patients, that is, those ones with Tu JV. Advantages and disadvantages of using artificial neural networks actual spontaneous expulsion of the stone (0) and those needing an vs logistic regression for predicting medical outcomes. J Clin Epidemiol Experiments were performed using the learning algorithms LR, Batuello JT, Gamito EJ, Crawford E et al. Artificial neural network modelfor the assessment of lymph node spread in patients with clinically ANN, SVM and LP-SVM. Performance was evaluated using cross- localized prostate cancer. Urology 2001; 57: 481–485.
validation, a well-known statistical methodology: 50 of the 402 data Cummings JM, Boullier JA, Izenberg SD et al. Prediction of spontaneous points (i.e., patients) were randomly selected and not used for ureteral calculous passage by an artificial neural network. J Urol 2000; training. After training with LR, ANN and SVM and LP-SVM on the Bagli DJ, Agarwal SK, Venkateswaran S et al. Artificial neural networks 352 training data points, the accuracy of the trained classifier was in pediatric urology: prediction of sonographic outcome following tested on the hold-out test set of the 50 data points, by reporting the pyeloplasty. J Urol 1998; 160: 980–983.
percentage of correctly predicted spontaneous expulsions (true Russel S, Norvig P. Artificial Intelligence, A Modern Approach. 2nd edn, negatives) and the percentage of correctly predicted cases needing Prentice–Hall: Englewood Cliffs, NJ.
Leane MM, Cumming I, Corrigan OI. The use of artificial neural networks intervention (true positives). This procedure was repeated 30 times, for the selection of the most appropriate formulation and processing resulting in 30 different random splits in training and test sets.
variables in order to predict the in vitro dissolution of sustained release Finally, the average true positive and true negative rate on the 30 test minitablets. PharmSciTech 2003; 4: E26.
sets was reported. Also, all simulations were performed both on the Porpiglia F, Ghignone G, Fiori C et al. Nifedipine versus Tamsulosin for themanagement of lower ureteral stones. J Urol 2004; 172: 568–571.
original data set that was not normalized, as well as on a data set Dellabella M, Milanese G, Muzzonigro G. Efficacy of Tamsulosin in the with covariates normalized to have zero mean and unit variance.
medical management of juxtavesical ureteral stones. J Urol 2003; 170: LR and LP-SVM9 are linear classification methods: they work best if both classes of data can be separated reasonably well in a Gomha MA, Sheir KZ, Showky S et al. Can we improve the prediction ofstone-free status after extracorporeal shock wave lithotripsy for ureteral linear way, that is, using a hyper-plane. If this is not the case, a stones? A neural network or a statistical model? J Urol 2004; 172: nonlinear separating function is needed. This can be established Kidney International (2006) 69, 157–160

Source: http://cosmal.ucsd.edu/~gert/papers/kidney.pdf