Opinion Forum

Experts Aren’t Always Impartial or Very Expert

December 9th, 2011

I am an Expert!

On December 6th in Williams v. Illinois, the U.S. Supreme Court heard oral argument on expert witness issues. As noted at SCOTUS Blog,

Plain English Issue: Whether a court violates a criminal defendant’s rights under the Confrontation Clause [Sixth Amendment] by allowing an expert witness to testify about the results of DNA testing conducted by another analyst who has not appeared as a witness at the trial. (Insert added)

The opinions of expert witnesses, as distinguished from facts within their personal knowledge, can be presented to a jury because experts are considered to have uncommon knowledge in obscure fields where laymen lack the qualifications to tread unassisted. The expert testimony in Williams had been offered, not to establish the reliability of the underlying tests, but to provide the expert’s conclusions based on test results. A problem is that tests sometimes get screwed up and, with only the conclusions drawn from them and no opportunity for counsel to cross examine those who performed the tests, conclusions drawn by the ultimate link in the chain may be no less flawed than the unexamined tests themselves; some test flaws impair the reliability of test results and some don’t. Even though instructed that an expert’s testimony cannot be considered as to the reliability of independently made tests on which he based his conclusions, juries can easily accept his conclusions, and hence implicitly the test results, because the expert accepted them. In those circumstances, there can be little opportunity for a jury to decide otherwise.

The relevant criminal law is complex and the Supreme Court could go either way in Williams v. Illinois. However, that is not what this article is about. Rather, it is about the often unquestioning acceptance of the opinions of our betters, the “experts.” Their opinions are often sought in judicial as well as non-judicial contexts. The media, the government and others often cite their opinions with no examination of the underlying data beyond summaries of whatever the expert may have concluded. An expert having relied upon data, the general assumption is that the data are therefore no less reliable than the expert himself. In a non-judicial context, there is usually no opportunity to cross examine the expert as to the bases of his reliance, his own methodology or anything else. The trend became quite apparent in the field of “climate science,” where only a few perverse “anti-science climate skeptics” were said to disagree with those who established and then perpetuated the mann-made global warming dogma; that dogma long prevailed and, while a bit tarnished, remains powerful. “Garbage in, garbage out” has largely been ignored.

Aside from working for about four years some forty years ago as a young prosecutor, then defense counsel and later special court martial military judge, my experience in criminal proceedings is nil. In later years, my trial experience was in hearings before administrative law judges at the Federal Communications Commission (FCC) where the rules of evidence are less rigorous than in criminal proceedings. Two proceedings involving expert witnesses come to mind as suggesting that reliance on “experts” who rely on work done by others should be tempered.

One proceeding involved an application for a permit requested by a television station in St. Joseph, Missouri to increase its radiated power and to relocate its antenna to a two thousand foot high tower at the middle of a triangle drawn from Topeka to Kansas City to St. Joseph. An at best marginally improved signal to the immediate St. Joseph area would have resulted, along with dramatically increased penetration of the more lucrative Kansas City and Topeka television markets. I represented a television station opposed to grant of the permit. Proceedings are rarely litigated on the basis of what the proponents and opponents actually want and fear; generally, such things are irrelevant and only matters concerning the “public interest,” as defined by the FCC, are considered.

The principal issue before the FCC was whether construction as proposed would deprive a few thousand residents of a small crescent shaped area of their only television signal of Grade B or better quality because they received no other cognizable service and because, after the proposed move to the new taller tower and despite the power increase, it was acknowledged that the calculated Grade B contour of the requesting station would no longer reach them. The principal issue became whether, without regard to engineering calculations, residents of the crescent actually received television service.

I offered expert testimony by a witness with long experience in the television ratings business, which relies heavily on statistically valid data analysis. He had prepared a questionnaire to be used for a telephone survey, the conduct of which he then supervised, of area residents picked by selecting the xth (e.g., the fifteenth) residential listing on every nth (e.g., third) page of the pertinent local telephone directories and then verifying residence in the crescent by finding the given address on a map. If the selected phone number was outside the crescent, the next name in the column was chosen, similarly verified and so on. Based on the results of the survey, he testified that hardly any (he provided numbers but I don’t remember what they were) television station was received there other than the Joplin station. While not challenging my expert’s methodology, the other side offered the expert testimony of a professor of marketing at a mid-western college as supporting a different conclusion. A college professor in a statistics-related field, his qualifications as an expert were also accepted.

The professor’s testimony relied on what he characterized as a “random sample” of residents within the crescent and on confidence limits calculated on the assumption that the sample had, in fact, been drawn randomly. He had designed a questionnaire and had recruited graduate students to conduct house-to-house interviews. He had instructed them to record answers to a question about which television stations each selected household received and deemed “acceptable for viewing.” There were three problems, at least two of which ultimately destroyed the purported randomness and hence the reliability of the survey to the point that the results could not be considered.

First, the students had been given no instructions on how to select, randomly, houses within the crescent (e.g., every fifth house on every third block); that’s considered very important by statisticians. Without such instructions, the students were free to select whichever houses they might wish, based on such subjective factors as they wished: barking dog — don’t interview there; tall roof top antenna — interview there because the desired results (many stations may be received) seem most likely; long driveway — too much trouble; ramshackle house or group of houses — the residents might be unfriendly or just have rabbit ears and not provide the desired results so don’t bother. This problem, alone, could have invalidate the survey.

Second, the students had been given no instruction on whether, or if so in what circumstances, to note on the survey forms any unsolicited comments. Many such comments were apparently offered and some students recorded a total of about ninety of them. There was no way to determine how many did not record how many comments. On one survey form, where station X was recorded as “acceptable for viewing,” a comment on the back of the form was “reception sucks.” Another comment was “only when the weather is good.” Many such comments were facially inconsistent with the responses recorded on the fronts of the forms. Third, the professor had neither seen the comments himself nor tabulated the data from the returned questionnaires but had delegated the task to graduate students. He had then relied upon those tabulations in preparing his (written) direct testimony.

On cross examination, the professor acknowledged that he had never previously seen any of the comments. I spent many pleasant hours cross examining him about facially obvious conflicts between the recorded survey responses and the comments noted on the backs of the forms. He was unprepared and had no answers. Because of those comments, and because of the inherently non-random nature of the survey, the judge threw it and the professor’s testimony out. It was very sloppy work by an “expert.” Had I not been able to cross examine him, his conclusions would have been accepted at face value.

Another pertinent example involved the only applicant for a permit to construct an FM radio station in a small community with no existing radio service (a “white area”). Under the technical calculations generally used by the FCC, the small community received no cognizable radio service from any station anywhere and hence the applicant (my client) was entitled to a substantial preference. However, under Technical Note 101 in the FCC’s rules, it might have been possible for those opposing the preference to show that some radio service was in reality available there due to the refraction of distant AM radio signals passing over a sharp mountain ridge. Opponents presented an expert in the field of radio engineering who testified that, based on measurements taken by his firm, a cognizable signal reached the small community. He had the requisite academic degree and had been employed in the field as a radio engineer for several years.

On cross examination, he acknowledged that he had received the requisite data from those who had done field measurements, had fed the data into a computer as his boss had directed and then printed out the computer-generated analysis which formed the sole basis of his direct testimony. He acknowledged that he had not otherwise analyzed the data or the topographical characteristics of the area before or after entering the data in the computer. He was unaware of what factors the computer software relied upon in reaching its conclusions. He was the only expert witness offered. Following cross examination, the judge struck his “expert” testimony and conclusions from the record and did not consider them in arriving at his final decision.

Experts are necessary because people without expertise in technical fields generally lack the competence to draw reasonable conclusions in those fields. Having very little knowledge of radio engineering, I spent about one whole day prior to each of the cross examinations being tutored by my own experts on flaws to look for. Many experts are competent and are well prepared to be questioned, vigorously, about the data underlying, and the bases for, their conclusions; if they aren’t or that can’t be done, little if any reliance should be placed on their conclusions.

The Earth and the Sun In some cases, the data upon which experts rely are not even made available for analysis by other experts who are “non-believers” and sometimes it later turns out that the data themselves were, in fact, unreliable. Perhaps the most recent and notable occurrences have been in the politically and ideologically charged area of “climate science,” where

There are three threads in particular in the leaked documents which have sent a shock wave through informed observers across the world. Perhaps the most obvious, as lucidly put together by Willis Eschenbach (see McIntyre’s blog Climate Audit and Anthony Watt’s blog Watts Up With That), is the highly disturbing series of emails which show how Dr Jones and his colleagues have for years been discussing the devious tactics whereby they could avoid releasing their data to outsiders under freedom of information laws.

They have come up with every possible excuse for concealing the background data on which their findings and temperature records were based.

This in itself has become a major scandal, not least Dr Jones’s refusal to release the basic data from which the CRU derives its hugely influential temperature record, which culminated last summer in his startling claim that much of the data from all over the world had simply got “lost”. Most incriminating of all are the emails in which scientists are advised to delete large chunks of data, which, when this is done after receipt of a freedom of information request, is a criminal offence.

This does not mean that we should never rely on experts; they are necessary in our world of exponentially expanding technological complexity. I could no more divine the reliability of specific DNA tests or the reliance to be placed on specific DNA analyses without an opportunity to hear questions answered by those who conducted the tests and made the analyses than I could independently measure the distance from my home to Mars. However, to spread a patina of infallibility over such experts, with no questions asked about the factual and other bases for their conclusions, leads far down the steep and slippery slopes of technocracy and toward lazy beliefs in whatever dogmas our high priests and their supporters may, for whatever reasons, currently select for promotion.

(This article was also posted at Dan Miller’s Blog.)

Articles written by Dan Miller
Tags: accuracy, climate change, data, expert, reliability, testimony, witness
Categories: Law, Politics, Science | Comments (3) | Home

3 Responses to “Experts Aren’t Always Impartial or Very Expert”

Tom Carter | December 9th, 2011 at 9:04 pm

Interesting. I just read John Grisham’s latest novel, The Litigators. (Us folks with Kindles can get it in a minute or so.) It deals mostly with personal injury and mass tort lawyers, and he makes some good points about experts.

The first point is that if you pay them enough money, you can find experts who will testify on either side of any issue. That’s one more symptom of a corrupt legal system (and corrupt “experts”). The second is that even legitimate experts (who will still go either way, depending on the money) can be torn down by an intelligent attorney who has a grip on the data.

He also does a good job of differentiating between ethical lawyers who pursue legitimate personal injury cases that benefit people who deserve it and shysters who chase ambulances and play the mass tort game.

I’ve always thought that scientists (and some kinds of engineers) were fair and ethical, for the most part. Collect the data, study it, and publish findings. The data is available to others who can replicate the findings, which is one of the basic principles of science. With everything that has happened in climate science, I’ve changed my mind. Seems even these paragons of virtue can be tempted by money, ideology, and fame. Too bad.

Dan Miller | December 10th, 2011 at 8:21 am

Tom, you note

The first point is that if you pay them enough money, you can find experts who will testify on either side of any issue. That’s one more symptom of a corrupt legal system (and corrupt “experts”).

It’s not necessarily a matter of corruption. Often, it’s possible to find and to advance valid arguments on ancillary matters relevant to the principal issue and they can make the answer to the principal issue neither a clear yes nor a clear no.

In the television case, a competent expert for the other side might have been able to show, with a statistically acceptable survey, that at least some households in the crescent received various stations beyond their Grade B contours. My expert designed an appropriate survey, had an employee take the survey and we prepared an exhibit based on it. That exhibit was exchanged with the other side (as is required) prior to the hearing. Then, he discovered to his horror that the employee had not followed his directions as to sample selection and that the survey was not statistically valid. I declined to offer it in evidence and, had I offered it, my expert would have declined to support it in his testimony.

After the exhibit referenced in the article was rejected, the other side tried to offer mine in its place. The judge did not permit that. I don’t know who would have sponsored the exhibit had the judge permitted them to go forward with it.

As to the radio case, it is possible that, under Technical Note 101, some radio service to the “white area” could have been shown. Again, that would have required a more competent expert more familiar with Technical Note 101. In neither case did either the experts or the attorneys pay enough attention to what they were about. As we often hear about President Obama, incompetence does not necessarily suggest venality.

Tom Carter | December 10th, 2011 at 12:23 pm

I wouldn’t suggest that all experts who testify in trials, some of whom do it pretty much as their prime employment, are either incompetent or corrupt. But some are one or the other or both. If you drag enough money through the campus trailer park, some professor will jump on it, eager to say whatever you want him to say. (Note this advertisement.)

Kind of comparable to the way judges interpret the law. Neither Clarence Thomas nor Ruth Bader Ginsburg is corrupt, I’m sure. But you can usually predict what side they will come down on in any case, based solely on their personal ideological beliefs. Same with experts who always testify for the defense in certain kinds of cases and others who always testify for the plaintiff. At bottom, this means that the law is not based on fundamental truth but is just a matter of opinion.