This is essential to maintain the results from each training-test set experiment, which allows us to compare the performance of two machine learning algorithms while maintaining information from the other machine learning algorithms tested

This is essential to maintain the results from each training-test set experiment, which allows us to compare the performance of two machine learning algorithms while maintaining information from the other machine learning algorithms tested. that support vector classification, deep learning and a consensus were generally comparable and not significantly different from each other using five-fold cross validation and using 24 training and test set combinations. This study demonstrates findings in line with our previous studies for various targets that training and testing with multiple datasets does not demonstrate a significant difference between support vector machine and deep neural networks. measurements only (i.e. comprised of a = modifier for the EC50). At this point, a workflow was invoked to merge duplicate compounds (duplicate compound activities averaged) into a single source and the subsequent outputs were combined into the final specific (WC-SP) and nonspecific (WC-NS) datasets (Table 1). Duplicate compounds were identified by subgraph isomorphism, which is used to group structure-activity input rows into groups. In cases where more than one activity is available, the most pessimistic interpretation is taken: if all values are specific, the average is taken (with a calculated error). If any of the compounds are expressed as inequalities, the least specific interpretation is made. If any of the inequalities contradicts other data, the compound is rejected. Table 1. Training and testing dataset information (WC = whole cell, RT = reverse transcriptase, NS = non specific, Lit -= literature, MW = molecular Pirodavir weight) class labels (i.e. compound is active at a target) correctly identified by the model out of the total number of actual class labels correctly identified out of total predicted class. The TP and FP rate performances are measured when we consider a sample with a probability estimate as being true for various intervals between 0 and 1. The AUC can be calculated from this Pirodavir receiver operator characteristic plot; it is interpreted as the ability of the model to separate classes, where 1 denotes perfect separation and 0.5 is random classification. Accuracy is the percentage of correctly identified labels (TP and TN) out of the entire population: metric by first range-scaling all metrics for each model to [0, 1] and then taking the Rabbit Polyclonal to MAP4K6 mean. This allows for a comprehensive overall model robustness comparison for different machine learning algorithms. To assess if the mean was the most appropriate representative for these metrics, several statistical evaluations were done for each set of metrics (i.e. the individual six metrics that make up each rank normalized score) to ensure that they were normally distributed. Four different normal distribution tests (Anderson-Darling, DAgostino & Pearson, Shapiro-Wilk, Kolmogorov-Smirnov tests) were used in Prism (GraphPad Software, San Diego, CA) for each individual set of the 24 external test set validation pairs gave a near-consensus: these populations are normally distributed for every algorithm tested, suggesting that the mean is the appropriate representative. When looking at the distribution of the rank normalized scores per machine learning algorithm, many of these metrics are statistically unlikely to be normally distributed; this suggests that the median, not the mean was the more appropriate representation of each population. This also suggests that nonparametric statistical tests are also more appropriate to compare machine learning algorithms for these data. Under the assumption that a rank normalized score is an acceptable method of comparison, the question of which machine learning algorithm is best can be thought of from two different perspectives: which one wins most often and by how much, or which one performs better on average by comparing the rank normalized score pairwise or independently (assuming that experimental results of Pirodavir every training-test set pair are equivalently important). Unfortunately, this approach only considers the difference between two machine learning algorithms at a time. Since one of the statistical analysis (Mann-Whitney U) used to compare two algorithms requires ranking of all the metrics from those two algorithms and then comparing these ranks, this creates an issue with asking which algorithm is best overall using just rank normalized scores. If one assumes independence of the rank normalized scores, the comparisons of ranking between two different algorithms are independent of the other algorithms tested. We took a related approach.