Contribution of Animal Models to Contemporary Understanding of Attention Deficit Hyperactivity Disorder

Attention Deficit Hyperactivity Disorder (ADHD) is a poorly understood neurodevelopmental disorder of multifactorial origin. Animal-based research has been used to investigate ADHD aetiology, pathogenesis and treatment, but the efficacy of this research for patients has not yet been systematically evaluated. However, such evaluation is important, given the resource consumption and ethical concerns incurred by animal use. Accordingly, we used the citation tracking facility within Web of Science to locate original research performed on animal models related to ADHD, prior to 2010. Human medical papers citing those animal studies were carefully analyzed by two independent raters to evaluate the contribution of the animal to the human studies. 211 publications describing relevant animal studies were located. Approximately half (3,342) of their 6,406 citations were by other animal studies. 446 human medical papers cited 121 of these 211 animal studies, a total of 500 times. 254 of these 446 papers were human studies of ADHD. However, only eight animal papers (cited 10 times) were relevant to the hypothesis of the human medical study in question. Three of these eight papers described results from both human and animal studies, but their citations solely referred to the human data. Five animal research papers were relevant to the hypotheses of the applicable human medical papers. Citation analysis indicates that animal research has contributed very little to contemporary understanding of ADHD. To ensure optimal allocation of Research & Development funds targeting this disease the contribution of other research methods should be similarly evaluated.


Citation analysis
The citation analysis was performed between January 2012 and December 2014. PubMed was searched for articles using animal models to investigate ADHD. We searched PubMed using Medical Subject Heading search terms (MeSH terms): "ADHD" AND title/abstract: "animal" OR "rat" OR "mice" OR "mouse" OR "Rattus" OR " Mus" OR "pig" OR "Cavia" OR "Sus" OR "rabbit" OR "Leporidae" OR "Drosophila" OR "primate" OR "monkey" OR "Macaca" OR "macaque" OR "Cebus" OR "dog" OR "Canis" OR "cat" OR "Felis". MeSH terms are a comprehensive list of key terms related to each disorder designed to identify all relevant studies in an area (Uman, 2011). So, searching for ADHD retrieves other nomenclatures for the same disorder such as hyperkinetic disorder or minimal brain dysfunction.
We included journal papers, books, research reports and conference proceedings written in English or Portuguese. We restricted our search to publications prior to December 31, 2010, to allow adequate time for citation of articles. 543 articles were retrieved. Since our goal was to evaluate the impact of original animal research papers, we used PubMed filters to exclude review articles ("review", "systematic review", "meta-analysis", "bibliography") as well as opinion articles ("biography", "autobiography", "comment", "editorial", "interview").
The remaining 211 papers (see supplementary file at http:// dx.doi.org/10.14573/altex.1507311s) were subjected to a subsequent citation analysis using the cited reference search facility within Web of Science. For each animal study, we recorded the total number of times it was cited, and allocated each citation to one or more of seven categories (animal research papers, human papers, review articles, editorials, in vitro papers, in silico papers and non-invasive animal papers). Whenever it was not possible to define the category of the citing paper (due to language barriers or absence of the abstract), the paper was allocated as "not available". If more than one category could be assigned to a paper (e.g., animal research and human paper), then that paper was allocated to multiple categories.
Using Pearson's Chi-square goodness-of-fit test for distributions, we investigated whether there was a significant difference between the number of citations of the animal articles by human papers and by animal research papers. The Chi-square goodness-of-fit test is used to test whether a sample of observations has approximately the same frequency distribution as a specified probability distribution. This test is especially useful for assessing the distribution of discrete and categorical variables (Freund et al., 2010).
To evaluate the number of citations that the animal papers received we built density plots, i.e., relative frequency divided by bin width, using the statistical software R. A density plot is a graphical method for examining how well an empirically derived density function fits a theoretical density function for a specified probability distribution (Cox, 2005). In our data the papers cited more frequently received citation frequencies that were increasingly distant from each other, apparently following a geometric progression. Hence, it was more suitable to use that family environment and exposure to harmful environmental substances play a role (Ni and Gau, 2014;Han et al., 2015;Neugebauer et al., 2015).
Even though the number of studies aiming to improve the comprehension of the etiology, pathogenesis, and evolution and ultimately cure of this disorder has increased in recent years, there is still a scarcity of relevant knowledge and an urgent need for more effective studies. This need is strengthened by recent studies that suggest that ADHD's prevalence might be increasing worldwide. For example, an American survey ascertained that from 1998-2000 through 2007-2009 the prevalence of ADHD in the US increased among children aged 5-17 years from 6.9% to 9.0% (Akinbami et al., 2011). Due to resource and financial constraints it is important to assess which research methods are the most promising in this field.
Since the mid-20 th century animal research has been a very widely used biomedical research methodology. Furthermore, even though functional investigation methods of the brain are the leading technology in contemporary brain disorder research (Marcucci and Vandresen, 2006;Labate et al., 2013), the emergence and development of transgenic animal models has also led to an exponential growth of animal use in neuroscience research, including in ADHD (Porter et al., 2015). Within ADHD, animals are used to model ADHD-related behaviors and traits (Yen et al., 2013), to seek understanding of ADHD's biochemical pathways (Yen et al., 2013;Huang et al., 2015) as well as responses to putative drugs (Dudley et al., 2013) and other therapies (Ouchi et al., 2013).
However, the benefits of animal models have always been simply assumed. To date the contribution of animal models of ADHD have not been subjected to significant critical scrutiny within peer-reviewed literature. And yet their use is substantially consumptive of research resources and animals lives. To prevent the poor design and reporting of many animal experiments, tools for assessing methodological quality and experimental designs have emerged (Hooijmans et al., 2010;Kilkenny et al., 2012). These tools represent an important step forward towards evidence-based research as well as the achievement of Reduction and Refinement principles. However, they fail to guarantee that the first R (Replacement) is appropriately achieved, i.e., they do not prevent the use of animals in experiments that could be performed by non-animal means.
A systematic evaluation of the contribution of animal models to specific human disorders might prevent the use of animals in studies aiming for a better understanding of those disorders. To conduct such evaluation, we performed a citation analysis and a systematic qualitative analysis of citing publications. Assuming that the studies cited by authors guide and influence their work (Burright et al., 2005), citation analysis provides a partial measure of the impact of cited studies. Previous citation analyses in other fields have demonstrated poor contributions of animal studies to human medical papers (Hackam and Redelmeier, 2006;Knight, 2007). To our knowledge however, such a systematic qualitative analysis of citations has not yet been conducted in the ADHD field. The number of published animal studies on ADHD was small enough to allow us to perform a citation analysis on all published papers.
-Relevant for Methods: When the human paper used the same methodology as the animal paper, with the exception of species. The above categories were defined prospectively and the same criteria were used by both raters.
Animal papers cited in clinical and treatment trials (human categories 1 and 2) were analyzed separately since we also wanted to determine if the animal data had translated to the human situation, i.e., when an animal study was used as a reference for the human trial, the raters independently investigated whether the animal results were in agreement with the human results.
Whenever there was a disagreement between the raters either in determining the category of the human medical paper or in determining the relevance of the animal paper, a consensus was reached after detailed discussion.

Citation analysis
The 211 original animal studies focused on ADHD that were published before the end of 2010 and identified by PubMed search (see supplementary file at http://dx.doi.org/10.14573/ altex.1507311s) were cited 6,406 times by December 2014. However, 43% of these animal studies were never cited in papers describing human studies.
As shown in Figure 1, animal studies were mainly cited by other animal research papers (3,342), followed by review articles (2,010), human studies (500), in vitro papers (168), noninvasive animal papers (100), in silico papers (46) and editorials (14). Nine animal papers were cited in papers that included both animal research and human studies. 226 citing papers were logarithmic intervals. Owing to the occurrence of zero citations within human medical papers and the impossibility to use logarithm zero, we used 0.5 as the logarithm for the "No citations" cluster.

Systematic qualitative analysis of citations
The total citations of animal studies by medical papers on humans (500) were encompassed in 446 articles on humans. Of the latter, 254 were papers on ADHD, and 192 were papers on other topics. 10 human ADHD papers were excluded from the subsequent qualitative analysis due to being either written in a language other than Portuguese or English, or because the papers were unavailable.
The remaining 244 papers on human ADHD were analyzed by two independent raters to evaluate the contribution of each animal research paper cited to the respective human study, as well as the goal of the latter.
To determine the foci of the human studies both raters allocated the human papers to one or more of the following categories defined prospectively: 1. Clinical trials: Papers aiming to test a new drug targeting ADHD. 2. Treatment trials: Papers aiming to study the effect of an existing drug in a new population. This category includes papers on drug-drug interaction and the use of a known drug for a new purpose. 3. Genetics: Papers aiming to explore specific genes, gene sequences or patterns that may be involved in the etiology of ADHD. 4. Psychology: Papers aiming to explore psychological variables that may be involved in the etiology of ADHD, including personality or cognitive traits and behavioral patterns. 5. Epidemiology: Papers aiming to understand natural or social environmental factors that might contribute to the etiology of ADHD. 6. Neurology: Papers that used fMRI, PET scans or other neurological examinations to study brain areas involved in ADHD. 7. Comorbidities: Papers aiming to identify and explore the interactions between ADHD and other disorders. 8. Biochemistry: Papers aiming to describe the biochemical changes that occur in ADHD. 9. Physiology: Papers aiming to describe physiological changes in ADHD. Concerning the relevance of the animal papers cited, the two independent raters classified each animal study as being: -Redundant: When the animal study was only mentioned amongst other studies as an example. When there were multiple studies used as an example of one or more points, the raters were instructed to only rate the study as redundant if there were older or human studies stating exactly the same points. -Minor Relevance: When the animal study was cited in the discussion or introduction providing information not directly related to the hypothesis. -Relevant to the Hypothesis: When the animal study was cited in the introduction, providing information relevant for the hypothesis explored in the human medical paper.

Fig. 1: Number of citations of animal papers on ADHD by category of citing papers
unavailable for categorization due to being unavailable to us or written in a language other than English or Portuguese. Pearson's Chi-square test suggested that, by conventional criteria, the difference between the number of citations by animal research papers and by human studies was statistically significant (Chi-square = 2102.28; p < 0.0001). Figure 2 shows that below value 2 5 (< 32 citations) the density plots were similar, meaning that a published animal paper focused on ADHD had a similar probability of being cited anywhere from one to 31 times. However, the likelihood of such a paper being cited 34 times or more descended abruptly. Figure  3 shows a more linear descending curve, evidencing that an animal paper on ADHD was likely to be cited very few times or not at all by human medical papers. The number of citations by human medical papers above value 2 3 (cited 16 times or more) was residual.

Systematic qualitative analysis of citations
Of the 244 papers focused on human ADHD that cited animal studies, 81 were on genetics, 58 on treatment trials and on neurology each, 45 on psychology, 38 on comorbidity studies, 28 on biochemistry, 7 on epidemiology, 3 on clinical trials, and 2 on physiology. No pattern was identified between the categories of the human studies and the relevance of the animal papers cited. Figure 4 presents a frequency histogram of the relevance categories of the animal papers cited in human papers in all categories except the clinical and treatment trials. The vast majority of citations of the animal papers was redundant or had minor relevance for the human paper. No animal paper was relevant for the methods and only eight papers (cited 10 times) were relevant for the hypothesis explored in the human paper.

Fig. 2: Density vs number of citations by all papers
Each point represents the average number of citations within each interval. The intervals are defined by the power of two (e.g., the interval 2 3 includes articles that received from 8 citations to 15 citations). The use of [ ) means that 2 k is included in the interval and 2 k+1 is excluded from the interval.

Fig. 3: Density vs number of citations by human papers
Each point represents the average number of citations within each interval. The intervals are defined by the power of two (e.g., the interval 2 3 includes articles cited from 8 citations to 15). The use of [ ) means that 2 k is included in the interval and 2 k+1 is excluded from the interval.

Fig. 4: Relevance of the animal papers cited by human papers on ADHD for the citing paper
narrowed that number further, since only eight animal papers (3.68%) seemed to be relevant to the hypothesis of a human medical study (Fig. 4). Only human data reported in three of these was actually relevant to the hypothesis. In sum, amongst the 57% of animal studies that were cited by human medical papers, the ones that may have significantly contributed to medical advances could be narrowed down to five articles, i.e., 2.3% of the overall total.
Those five articles were all published between the years 1999 and 2010 and all used genetically modified mice or rats as the animal model. However, this may simply have been a reflection of the animal species most used within the larger population of animal studies examined. These results suggest that more recent articles may be more effective than older ones. Only one gathered data from mice and a non-human primate model (rhesus monkeys), contradicting claims that the use of non-human primates is crucial for our understanding and treatment of the attention functions compromised in ADHD (e.g., Roelfsema and Treue, 2014).
Three of the five studies aimed to explore the mechanisms by which psychostimulants or other drugs act. One study aimed for a better understanding of dopaminergic pathways and the other study aimed to understand the effects of a knockout gene on visual-spatial abilities.
The animal studies appeared to influence mainly subsequent animal studies. This data emphasizes one of the major obstacles within contemporary scientific research: the segregation between research fields. If we exclude review papers and editorials, we can observe that the proportion of animal studies cited by original papers within other fields is considerably lower than the citations by other animal papers (Fig. 1). With respect to citation rates, there is a startling gap between animal and human studies.
In addition to animal research, the contribution of other research fields to the understanding, prevention and treatment of ADHD needs to be evaluated. Even though there are numerous reviews of candidate animal models for ADHD (Arime et al., 2011;Leo and Gainetdinov, 2013), to our knowledge, there are no reviews of the contribution of other methods, e.g., in silico models.
The use of animal models in biomedical research consumes considerable research resources and raises serious ethical questions. These resources are then unavailable to other research methods or strategies for advancing healthcare. Hence, it is essential to ensure their efficiency and effectiveness.
Some studies have used citation analysis or systematic reviews to examine the contribution of animal models to other health disorders (Hackam and Redelmeier, 2006;Knight, 2007) and some of these studies (Pound et al., 2004;Knight, 2007), have implied that the citations of animal studies by human medical papers are often of little relevance for the human paper that was citing them. Even weaknesses of citation analysis identified by several researchers (Brooks, 1985;Garfield, 1998;Bornmann and Daniel, 2008) are fully addressed with a subsequent systematic qualitative analysis of citations.
Hence, our results suggest that animal studies rarely contributed significantly to contemporary understanding of ADHD.
Of the eight animal papers considered relevant for the hypothesis, three were papers describing both animal research and human studies. Within these three papers, only the human studies were relevant for the citation in question. Therefore, five (2.3%) of the 211 animal studies focused on ADHD contributed to the hypothesis of a later human ADHD study.
The three clinical trials that cited animal papers did not use these animal studies for the hypothesis, methods or results. Therefore, investigation of translational research was not applicable.
Of the 58 treatment trials, four used animal papers for the hypothesis. The results in three out of four animal papers were in agreement with the results of the respective treatment trials.

Discussion
To our knowledge, this paper provides the first systematic study of the contribution of animal-based research to contemporary understanding of ADHD.
We acknowledge that this study had several limitations: Firstly, due to resource constraints we were unable to search a greater number of search engines (e.g., Web of Science, CAB Abstracts, Scopus) to increase the likelihood that we retrieved all animal papers investigating ADHD. We were similarly unable to examine the reference lists of retrieved papers in the hope of locating additional relevant papers. This means that some relevant publications may not have been located. Additional relevant studies may also exist in so-called "grey literature" such as unpublished reports of various kinds. However, it is reasonable to expect that most experiments that made a significant contribution to human healthcare advancements would have been published in a biomedical journal, and further, that most such journals would have been indexed in PubMed. Accordingly, we expect that our results are conservative, compared to the overall results that would have been achieved had it been possible to examine every single publication relevant to our research question.
Secondly, we used MeSH term search for ADHD, which means that all papers investigating this disorder should have been retrieved. However, we acknowledge that a minority of papers focused on this disorder may not have been labeled within PubMed standard MeSH terms for ADHD (e.g., due to labelling errors) and so may not have been located by our search.
Finally, we recognize that there is a level of difficulty in objectively determining the relevance of a cited paper to the paper citing it. Even though we have tried to avoid bias by using two raters, the initial assessment was sometimes divergent between the raters, requiring further discussion to reach a consensus. Hence we acknowledge that different raters using the same criteria might have rated some papers differently. However, we believe these would comprise only a small minority.
The citation analysis showed that 43% of the 211 animal studies were never cited by subsequent human studies and less than 8% of the total number of citations of the animal studies was by human medical papers. The systematic qualitative analysis