Scientific literature needs discipline – an example from a killer whale life expectancy study

If you let a puppy piddle on the carpet without discipline, it will keep doing it. It will grow into a big dog that destroys your carpeting and rugs and makes your whole house stink.

So it is with scientific literature.

We all know bad papers are out there. When you read them, you’re left scratching your head and wondering, “How on earth did these pass peer-review?” Worse still, there are “ugly” science articles, where the scientific method goes by the wayside and data are cherry-picked, misinterpreted or manipulated to justify a political or ideological agenda or to undermine science that interferes with that agenda.

Robeck et al. (2015) is a recent example of ugly science. The authors (three of whom work for SeaWorld, one for the Minnesota Zoo) compared survivorship and longevity of orcas in captivity versus the wild. They stated that “the estimated ages assigned to [free-ranging killer whales]…were inaccurate” and “Our analysis supports a proposed longevity of between 60 and 70 years for females and 50 and 60 years for males…substantially less than the longevity of 80-90 years for females and 60-70 years for males…previously suggested” (p. 1066, emphasis added). This paper also dismissed decades of life history table analyses that have long been accepted within the scientific community, stating that they were of “limited value” (p. 1059).

With these conclusions, Robeck et al. effectively tossed 43 years of research by a number of respected orca biologists under the bus.

In addition, they calculated that captive-born orcas have a mean life expectancy of 47.7 years. Yet the oldest ever captive-born orca is celebrating her 28^th birthday in September this year. All other captive-born orcas either died before they reached 28 or are younger than 28 now. How did the authors manage to calculate a mean life expectancy for a group of whales whose maximum life span to date is nowhere near that mark? Because the authors did something a previous paper (DeMaster and Drevenak 1988) had specifically counseled against doing – using an annual survivorship rate (ASR) derived from data collected over multiple years from a mixed-sex and -age group to calculate a mean life expectancy for that group.

DeMaster and Drevenak (1988) emphasized that using ASR to calculate mean life expectancy would yield a valid result only if ASR remains constant over time and across age classes, and is the same for both sexes. Given how rare this is in any animal (in fact, it pretty much never happens, because very young animals often have higher mortality rates than prime-of-life animals, males and females typically have different survival rates and life expectancies, and, of course, really old animals are much more likely to die than young ones), they recommended against using ASR to calculate life expectancy. Despite this, Robeck et al. actually cited DeMaster and Drevenak in support of their life expectancy calculation using ASR! They also reported that ASR for captive-born orcas varies across age classes and sexes and has improved over time, thus by their own efforts clearly rendering this method to calculate mean life expectancy invalid. So a mean life expectancy of 47.7 years has not just “limited value” as a captive orca life history parameter, but none.

This paper is a perfect example of industry science, heavily biased in favor of a political agenda. A rosier picture than warranted regarding captive orca longevity was deliberately painted by these authors, to the extent that they even somehow failed to report that the free-ranging populations to which they compared their captive sample are listed as endangered and threatened. In short, Robeck et al., while reporting their results as evidence that captive orcas are doing well, actually demonstrated that at best captive survivorship is similar to that of populations struggling with food shortages and habitat degradation.

This paper somehow got through peer review, even though apparently no authors of previous papers related to the topic were involved in that review. It was clearly ripe for rebuttal, given its invalid methodologies and its undermining of long-established empirical research on this species.

The puppy piddled on the floor big time in this case, but discipline has been at best weak and slow to come from the scientific community.

One rebuttal has been published, focused on one issue (none of those listed above) mentioned in the discussion of the original paper. Another rebuttal is in the works, but has yet to be submitted. But the vast majority of scientists with the expertise to address the deep flaws in Robeck et al.’s analysis chose not to respond when this paper was brought to their attention, not just to the flawed analyses, but to the sweeping dismissal of their own work.

Why?

Sometimes this failure to respond to bad science is simply because the relevant experts are too busy. They may feel they need to prioritize their time to paid projects. Others might feel that many rebuttals increase the original paper’s citation rate, making it look more accepted than it is. Still others might consider it unimportant because, hey! Hardly anyone reads scientific papers anyway. If we ignore it, it will just go away. But all too often it’s because the scientific community is leery of getting involved in controversy.

The worst is when researchers just don’t care. Owners who are negligent and ignore their piddling puppies end up with adult dogs who aren’t housebroken and do a lot of damage, all because they were ignored just when they needed the most attention.

When scientists know of flawed or outright wrong papers and do nothing about it, they condone two serious consequences. One, they allow the scientific process to fail. Rebuttals are the process’s way of correcting the “mistake” of allowing the publication of a flawed analysis. Two, they ignore the fact that such papers become the “best available science,” and can be used in court cases, in regulatory actions – in short, they can become the basis for policy.

Several professional scientific societies call on their members to correct bad science when they see it; for example, the professional ethics guidelines of the Ecological Society of America recommend that members

“strive to accurately represent ecological understanding and knowledge and to avoid and discourage dissemination of erroneous, biased, or exaggerated statements about ecology.”

The Society for Marine Mammalogy has a similar statement in its professional ethics guidelines, which is particularly pertinent to the case study here. These guidelines call on members to

“work to accurately represent knowledge of marine mammal science and to avoid and discourage, and if necessary correct, the dissemination of erroneous, biased, or exaggerated statements” (emphasis added).

Puppies need their owners to bestow loving discipline to grow into well-behaved dogs. The peer-review process needs scientists to pay attention and respond when flawed papers are published. Peer review is how we keep the scientific process honest, but it is not infallible. If researchers don’t rebut papers that cry out for rebuttal, they are doing nothing to prevent those ugly stains on the carpet of their profession.

by Naomi Rose & Chris Parsons

Dr. Naomi Rose is the marine mammal scientist for the Animal Welfare Institute in Washington, DC. Dr. Rose oversees marine mammal issues and programs at AWI, including the protection of marine mammals in the wild and in captive situations. She has been instrumental in campaigns opposing the capture and captivity of marine mammals for public display and has been a key player in the international debate on the issue. She is actively involved in several campaigns and coalitions addressing problems associated with cetacean live capture, trade, and captivity, both in the U.S. and abroad. Dr. Rose has been member of the International Whaling Commission (IWC) Scientific Committee since 2000, where she participates in the subcommittees addressing environmental concerns and whale watching. She has appeared and been quoted in numerous news media, including television and radio. She has authored or co-authored over 30 scientific papers and authored numerous articles for animal protection publications, as well as chapters in several books. She lectures annually at three universities and speaks at and participates in various conferences, workshops, meetings, and task forces at the international, national and state level. She has testified before the U.S. Congress four times. Dr. Rose received a Ph.D. in biology from the University of California at Santa Cruz in 1992, where her dissertation examined the social dynamics of wild orcas. She has worked in the marine mammal advocacy field for over 20 years.