Daubert test

Junk science

Junk law

Testing theories

Peer review

Disputed theories

Fit the facts of the case

Court appointed experts

Return to Junk Science page

The Daubert test comes out of the Daubert decision (Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579). In Texas we pronounce it "Daw-bert," instead of in the French manner. It set the rules by which the federal courts were to evaluate expert testimony, especially scientific or technical expert testimony. The basic test is:

    1. The judge should let the jury hear the expert only if the scientific or technical knowledge is such that "the reasoning or methodology underlying the testimony is scientifically valid, and ... can be properly applied to the facts in issue" (Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579, p. 8).
    2. A "key question ... will be whether [the scientific technique or theory] can be (and has been) tested" (Ibid.).
    3. "Whether the theory or technique has been subjected to peer review and publication" (Ibid., p.9).
    4. "The court should consider the known or potential rate of error" (Ibid.).
    5. "Widespread acceptance can be an important factor. ... [A known technique without support] may properly be viewed with skepticism" (Ibid., p. 9).

The background of Daubert is in lawsuits where plaintiffs were trying to prove personal injury from a medication. The medication, Bendectin, was taken by some women during pregnancy. Some of these women had babies with birth defects. The lawyers for the women want a lot of money from the maker of the drug based on children with birth defects.

The U.S. Supreme Court in Daubert did not rule on whether or not the plaintiffs' experts were offering junk science or valid opinions. They just ruled on the test that the Ninth Circuit Court of Appeals had used. The old was wrong; they wrote a new test.

Junk Science. The Daubert test was framed in order to deal with what is often called "junk science." Junk science is not really science at all. It is fiction. It is often not even good science fiction.

Junk science uses theories and techniques that have been made up for lawsuits, or for a single particular lawsuit. This also includes theories and methods that have been published and reviewed, but have not been published with the same usage as they are used in the lawsuit. The most common published-but-still-junk items are experimental methods and theories which have not been published as valid only as possibilities for further research.

There is no way, for example, to study whether or not a 12-month old infant has been harmed or terrorized by an adult. One junk scientist used the Wisconsin General Testing Apparatus (WGTA) in a lawsuit. The WGTA was a widely used experimental method mainly used with primates and undergraduates, it had been used with babies. There had been thousands of peer reviewed published studies that used the WGTA. However, it had never been used to assess fear by an infant of an adult. This scientist did a study just so he could offer an opinion in a lawsuit.

Generally, Daubert holds that the expert has to present opinions to the court or jury, the "trier of fact," that are based on published research that has appeared in journals after peer review.

"Peer review" is the process where the editorial decision about whether or not the study results should be published is made by independent experts in the field of science. So, to get published in a peer reviewed journal, there have to be some other scientists who think the study has some merit.

In the Kennedy Heights case, the plaintiffs' experts said that even though their theories had not been published yet, they were planning to publish them after the trial. They also suggested that the fact that they had each reviewed each others' opinions was the same as "peer review." These ideas may have been made-up by the plaintiffs' jury consultant.

Junk Law. These were two excellent pieces of "junk law." By "junk law" I mean when elements from the actual ruling are used, or misused, to argue the opposite from the intention of the writer of the opinion. The peer review, of course, referred to in Daubert is the review of journal editors who do not know who the author is; it should be "blind review," not the review of other members of the same expert team.

The idea that their made-up theories and models would be published after the lawsuit was over was exactly against what the Court seemed to say in Daubert,

It is true that open debate is an essential part of both legal and scientific analyses. Yet there are important differences between the quest for truth in the courtroom and the quest for truth in the laboratory. ... The scientific project is advanced by broad and wide-ranging consideration of a multitude of hypotheses, for those that are incorrect will eventually be shown to be so, and that in itself is an advance. Conjectures that are probably wrong are of little use, however, in the project of reaching a quick, final, and binding legal judgment ... about a particular set of events in the past. (pp. 10-11)

Testing Theories. While making up theories is good theoretical science, what makes a theory be good are two things. The most important thing is how well it can explain all of the tests that have been made in the past in the problem area. Usually scientists talk about "past data" rather than "past tests." "Tests" is a technological or legal idea, not a scientific one. When some decision has to be made, you want a valid test to guide you in making the decision. This is true when a physician has to make a decision about what disease a patient has. This is true when the trier of fact has to make a decision about the cause of some damages.

Scientists often say, "Data are data." This means that how and why the data were collected does not make any difference. If the theory is any good, it will explain all of the data within acceptable errors of measurement. Lawyers and physicians want a definitive measurement, not lots and lots of different data. One accurate measurement clearly recognized as accurate and relevant to the decision is what they want.

If there is an area of science with no difficult questions, then how is the scientist going to discover something new? Scientists like to study areas where there is doubt and uncertainty.

The other criterion for a "good theory" is the amount of further research and theory building it stimulates. Very few scientific psychologists think that Freud's theories were ever very good in explaining the available data. Whenever Freud, and Freudians, were shown data that disagreed, they accused the scientist of repressing the truth. Yet, Freud's theories are good because they have stimulated tons and tons of research.

Peer Review. As we said above, "peer review" is the process of making decisions by independent experts in the field of study. Peer review can be used to select which paintings will be shown in a gallery. Most good scholarly journals use peer review to select what shall be included in the journal. Most research grants from the National Science Foundation or the National Institutes of Health are awarded based on multiple peer reviews. In most areas of scholarship, you reviewers have to be peers of the applicant to understand and evaluate what is presented.

This means to get published in a peer reviewed journal, there have to be some other experts who thinks your study has some merit. The rejection letters are often rude and always hurt your feelings.

Peer review cuts two ways. If the researchers do have a new scientific insight, say a theory that will become fact in 10 to 20 years, then they will have difficulty getting it in peer reviewed publications. Another researcher who is keeping on in the old, less accurate, ways will get positive peer review. The solution is to start a new journal, one where you get to choose the peers to do the reviewing. Or, write a web site like this one.

Disputed Theories. This is another aspect of the Daubert rule. For an expert to present an opinion based in a disputed scientific or scholarly principle, the disputed theory has to at least have enough support to be a real dispute in the field of the expert. This is really the same as requiring peer review. If you can get enough people to accept a hair brained theory, then you can start a peer reviewed Internet journal.

The Court in Daubert actually wrote about this from the other side, "Widespread acceptance can be an important factor. ... [A known technique without support] may properly be viewed with skepticism" (Ibid., p. 9). For a real dispute to exist, there have to be people supporting both sides. If you are the only scientist with this idea, you are an odd ball. If you make up a theory that, say, 50 other scientists will agree has some merit, then you are a nationally recognized figure. If you make up a theory that everyone, except for a few odd balls, accepts as having some merit then you might win the Nobel prize.

If a theory does not even have 5 to 10 active, publishing scientists who say that there is something good in the theory, then it cannot help a trier of fact make any decision. The trier of fact can make up their own hair brained theories without the help of an expert.

The Facts of the Case. The published basis for the experts' opinions also has to relate to the facts of a particular lawsuit if it is going to meet Daubert. The logical analysis that connects the published knowledge to the particular facts has to be specified and made clear using acceptable methods for the experts' areas.

This is the logical analysis made by the Court of the phrase in the Federal Rules of Evidence, Rule 702, "assist the trier of fact to understand the evidence or to determine a fact in issue" (Italics added). They said, "Rule 702's 'helpfulness' standard requires a valid scientific connection to the pertinent inquiry as a precondition to admissibility" (Daubert, p.8).

One favorite example is where animal studies have show a chemical causes cancer. There are plenty of chemical that cause cancer in some animals but not in other animals. There are chemical that on grain will kill a rat dead, but a horse can eat a box full and not be harmed.

If the trial is about cancer caused in small white mice, then the animal studies would address an issue in the case. But do they address cancer in people?

Court Appointed Experts. When judges have little to no scientific knowledge or skill in a given technical area, then the judges should appoint their own experts, their fees paid by both plaintiffs and defense, to assist them, and to assist the jury if the case comes to trial. Juries will believe the experts appointed by the judge if the experts disagree.

This is in part why otherwise good, professional scientists will agree to testify to junk science for high fees. If the courts and judges do not even care enough to understand the case they are trying, why should the scientists bother?

It may be that judges have made too strong commitments to the adversarial approach to dispute resolution. To appoint experts for the Court to find the truth is not finding the truth by adversarial means. There are lots of situations where the adversarial method does not seem a good fit. The only generally accepted area in law where rank adversarial fighting is not supposed to be used is in child custody; here the "best interest" of the child is supposed to prevail rather than the best fighter.

The problem with alternate dispute resolution approaches is that most parties in major lawsuits, and in child custody cases, do not want to "settle the dispute," they want to "win." If they want to settle the dispute, they will go to binding arbitration and have a scientist be one of the arbitrators.

Error Rates. These are so ease to get wrong. With well researched assays, the error rate can be expressed in terms of "plus or minus" so much. The usual rule is to have a plus-or-minus of a size that would include 95% of occasions between minus 1 plus-or-minus and plus 1 plus-or-minus. But this is in the middle of the range the assay can measure.

At the lower end, there is the "detection limit." This is expressed as the lowest number that we can be sure the assay will find at least 95% of the time. At detection limit of "10" means that if the true value is 10, the assay will find it at least 95% of the time. If the true value is "8," the assay will not be able to find it 95% of the time, but will find it less of the time.

Theoretically there might exist an assay that could have a zero detection limit. But in actual technology there is always going to be some limit. With mature assays, the detection limit is generally well below the point where what you find is insignificant in human affairs.

For example, a well made steel ruler, with out lenses or microscopes, has a lower detection limit of about 2/10,000th of a meter. This is 1/5 of a millimeter. You might be able to use the edge of the ruler as a straight edge, and do an experiment to show that one thing was longer than another, but you would not be using the ruler as an assay of distance. For measuring most things in daily life a detection limit of 0.2 mm is plenty good. A micrometer with a good lens might take the detection limit down to 0.01 mm, but you don't need that to tell what size shoes to buy.

 

Return to top of page

 

Return to Kennedy Heights main page, or to the Junk Science page, or use your <back> key.