Subramani Mani^†, William Rodman Shankle^†‡, Michael J. Pazzani^†,

Padhraic Smyth^†, and Malcolm B. Dick^±

University of California at Irvine, Irvine, CA 92697

The Electronic Medical Record (EMR) has potential value when used in conjunction with Knowledge Discovery and Data Mining (KDD) methods. Clinically, KDD methods can be used to produce decision trees, rules, graphs, quality controls, as well as to detect protocol violations and inconsistent patient data. We are applying KDD methods to understand normal brain aging and dementia. In phase I of this project, we have successfully applied KDD methods to a dementia database to identify a screening test¹ that has much higher accuracy than the same test using nationally recommended scoring criteria². In this report, we describe the initial work related to the development of decision rules for diagnosing Alzheimer's Disease (AD) and Vascular Dementia (VD) using KDD methods applied to the EMR of the UC Irvine Dementia Database.

The EMR of the UCI Alzheimer's Disease Research Center (ADRC) uses a Sybase relational database with a JAM graphical front-end that can be accessed remotely from any platform (MAC, PC, or UNIX). It consists of more than 60 data entry screens with the underlying tables developed in third normal form. Each data entry screen has a standardized graphical format which allows direct data entry through mouse or direct typing by all personnel to reduce the incidence of missing data and transcription errors. Features keyed to data entry include immediate error checking for data type, value range, and logical consistency, plus auto-calculation of categorical and summary scores. To avoid confusion between null values and missing data, there is a mandatory field specifying each screen's status (not done, done, failed to comprehend, refused, or too slow to complete). Standardized coding includes the International Classification of Diseases (ICD9), and the National Drug Codes (NDC), which are accessed by entering partial strings of the disease, symptom or drug name. Disorders, symptoms and drugs specific to dementia that are not included in the ICD9 or NDC have been coded and added so as not to conflict with existing or future ICD9 or NDC codes. The structure of the medical assessment screens is generic and follows DeGowin and Degowin's Bedside Diagnostic Examination³. The screens devoted to pertinent positive and negative features of the chief complaint collect data relevant to memory loss and dementia; otherwise, this EMR can be used for any medical problem. The database currently holds more than 2,000 patient-visits (patients are longitudinally followed) and collects more than 1,200 fields per patient-visit. Since both clinical staff and researchers use this database, there are multiple security access levels to protect patient confidentiality. The data used for the present analysis were generated using standard SQL scripts into formats acceptable to the Machine Learning (ML) algorithms.

Table 1 characterizes the 428 mildly demented patients (Clinical Dementia Rating Scale (CDRS) < = 1;⁴) seen at the UCI ADRC, whose diagnoses were either possible or probable AD⁵, possible or probable VD⁶, or other causes. Patients with multiple dementia etiologies were included to render the decision trees and rules more clinically useful as well as to force the KDD methods to search for unique patterns of positive criteria for these diseases. For each patient, we created three binary diagnosis attributes (AD, VD, and Other Causes). For example, a patient with probable VD and possible AD would be coded as having AD and VD but not Other Causes.

Although space limitations preclude a discussion of prior work on machine learning and differential diagnosis, our previous paper addresses this issue¹. In diagnosing AD and VD with KDD methods, we constructed a binary decision tree for AD vs. not-AD (NAD) and a separate binary decision tree for VD vs. not-VD (NVD). This is because the occurrence of these two dementias is statistically independent of each other (the product of their probabilities equals the probability of their co-occurrence, which is roughly 15%). Hence we argue that the criteria for each etiology should be applied independently. We initially considered automated feature selection but later concluded that it is not feasible due to the computational cost involved. (Feature selection from a subset consisting of 140 attributes ran for more than 3 weeks.) We decided to approach the diagnostic problem in several phases. In the first phase, we will examine specific knowledge domains to identify the best attributes within them. In this paper, we restricted the attributes examined to the set of demographics and the total scores from those neuropsychological tests with relatively few missing values. Tests not administered routinely were excluded from the attribute set. In subsequent phases, we will examine other knowledge domains, then evaluate the best attributes from all domains simultaneously. The specific attributes used in the present analysis measured gender, age, education, dementia severity, judgment, abstract reasoning, category fluency, letter fluency, delayed free recall and recognition, simple and complex attention span, visual-constructional abilities, and object naming.

Specific algorithms. We concentrated on decision tree learners, rule learners and the Naive Bayesian classifier. Decision trees and rules generate clear descriptions of how the ML method arrives at a particular classification. The Naive Bayesian classifier was included for comparison purposes. MLC++⁷ (Machine Learning in C++) is a software package developed at Stanford University which implements commonly used machine learning algorithms. It also provides standardized methods of running experiments using these algorithms. C4.5⁸ is a decision tree generator and C4.5rules produce rules of the form, if..then from the decision tree. Naive Bayes⁹ is a classifier based on Bayes Rule. Even though it makes the assumption that the attributes are conditionally independent of each other given the class, it is a robust classifier and serves as a good comparison in terms of accuracy for evaluating other algorithms. CART¹⁰ is a classifier which uses a tree-growing algorithm that minimizes the standard error of the classification accuracy based on a particular tree-growing method applied to a series of training subsamples. We used Caruana and Buntine's implementation of CART (the "IND" package), and ran CART fifty times on randomly selected 2/3 training sets and 1/3 testing sets. For each training set, CART built a classification tree where the size of the tree was chosen based on cross-validation accuracy on the training set. The test accuracy of the chosen tree was then evaluated on the unseen test set.

In classifying AD vs. NAD, the clinical gold standard consists of the CERAD criteria, which when consistently applied, give about a 17% false positive rate in detecting autopsy-confirmed AD¹¹. These criteria use domains in addition to demographics and neuropsychologic testing. Results obtained with C4.5Rules and Naïve Bayes (25% and 22% false positive rate) are encouraging because they use only a small subset of the domain information that goes into the CERAD criteria. False negatives for C4.5Rules and naïve Bayes were 37% and 31% respectively. The ML results warrant a search for additional, preferably inexpensive, attributes that can raise diagnostic accuracy at or above that obtained by CERAD criteria. If achieved without imaging (a $1,000 cost) for a significant proportion of subjects, a substantial cost savings would be obtained while maintaining diagnostic accuracy. The dementia diagnostic work up costs are presently estimated to be about $1,400¹². Since current diagnostic accuracy among general practitioners is about 65%¹³, applying our results would significantly improve diagnostic accuracy of AD vs. NAD in the community.

For VD vs. NVD, the mean accuracies generated from each ML algorithm did not perform better than chance. These results are consistent with the consensus criteria established by the Alzheimer's Disease Diagnosis and Treatment Center (ADDTC) in that they did not include neuropsychological test results in the diagnostic criteria for probable and possible VD. The initial results of the ML algorithms indicate that alternative domains are needed for diagnosis of VD. It should also be noted that accurate diagnosis of VD by humans is at present still difficult due to a lack of consensus about the neuropathological definition of vascular dementia¹⁴.

The decision rules generated by the ML algorithms also proved useful in identifying subtle types of errors in the database. For example, in generating decision rules for dementia screening¹, we found a rule which classified persons as normal if they could no longer perform their job. After reviewing these cases, we discovered that they misunderstood the question regarding job performance and indicated that they no longer could perform their job if they had retired. This error was missed by all other data validation procedures implemented in the database.

The eradication of decision rules which make no clinical sense is critical for the overall success of this project. Pazzani has shown that, over a broad range of experience, clinicians are unlikely to use decision rules if they contain elements that make no sense clinically, even if the rules give highly accurate results¹⁵. To this end, he has developed simple constraints in FOCL¹⁶ which minimize the occurrence of such nonsense rules.

There are several limitations of the present work. The first limitation is sample size. In order to examine more attributes simultaneously we will require data from multiple centers. This is also true if we are to obtain accurate classifications of the less common dementia etiologies, for which no one center will have a sufficient sample. The second limitation, inherent to any clinical sample, is potential bias. Are the patients with AD, VD and other causes representative of their respective populations? The answer to this question can only be established by similar analyses of other centers' data or by randomly selecting patients with these diseases from their respective populations. The other limitation stems from the lack of well-defined and reliable diagnostic criteria for assigning class (diagnostic) labels even after looking at all available data. This is less of a problem for AD, in which application of the CERAD criteria reliably results in greater than 83% accuracy compared to neuropathologically confirmed cases¹¹ (which is typically done post-mortem), than it is for VD. One deficiency of the literature is the lack of reporting on false negative rates for non-AD dementia; studies have focused on false positive rates for AD. Multiple sets of diagnostic criteria exist for VD and the neuropathologic definition of VD is still debated. This introduces a margin of error at the class assignment stage itself, particularly for VD, and makes the learned classifiers liable to bias in the process.