My previous post discussed how mistaken thinking about research evidence in nutrition has led to a misleading or even false understanding of how we should approach and maintain our health. A broader question than the argument-from-uncertainty fallacy arises: what is it in the nature of scientific investigation that makes these kinds of errors possible? The short answer is that there is no free market in science that would facilitate the quick awareness of both innovation and error. What we have today is government domination of research that deflects attention away from concern for the constituents the science is intended to serve, for example, doctors and their patients in medicine, to compliance with the rules of grantsmanship and peer review the ultimate aim of which is publication for tenure and promotion in the cloistered academic world.*
Though data fabrication and other frauds do occur, the most important cause of flawed research is the “what do we have to do to get the money and then the publication” attitude. This expedient in the protected environment of academia leads to conventional, uninspired, and sometimes substandard, politically biased, and irrelevant work, much of it amounting to useless minutia. I don’t mean to imply that there is a willful disregard for truth or facts on the part of researchers, though again that sometimes does occur, but that the system in which academics compete for money and accolades predisposes them to pursue conventional lines of inquiry.
“The problem in science,” states the chief medical officer of the American Cancer Society, “is that the way you get ahead is by staying within narrow parameters and doing what other people are doing. No one wants to fund wild new ideas.” Except profit-making private companies like Revlon. And peer review does not encourage originality or novelty. As British medical researcher David Horrobin pointed out in 1995, the acceptance of eighteen medical innovations was either delayed or thwarted by the process. The system is also notorious for not publishing findings that are negative, that is, studies that do not produce significant findings, on the alleged assumption that what works is more important and interesting than what does not.
In my 2007 posts (1, 2) on peer review I argued that fear of the new governs and that the process is no guarantee of objectivity. This last has been demonstrated by studies retrospectively analyzing the results of published work. How so? “Miscalculation, poor study design or self-serving data analysis,” in the words of Wall Street Journal science writer Robert Lee Hotz. The pressures of publication cause sloppiness and cherry picking of the data—data mining or data massaging, as researchers call it, to find something publishable. (Selection bias was the term used in last month’s post.) All of this then brings about a recurrent lack of replication of the original study. Failure to replicate puts the science back to square one. Often it is the private, profit-making companies, using the double-blind research technique, who cannot replicate the work of the academics, because academics frequently fail to double-blind their experiments.
Horrobin’s solution to the screening and gatekeeping process of peer review was to divide the research money equally among researchers, then let the researchers be guided by their own interests in choosing what to study. This would remove the rule-bound, paper pushing requirements of the bureaucracy and free scientists to do what they do best, namely research. Horrobin pointed out that research money was distributed in precisely this way prior to 1960 in Great Britain and resulted in more medical innovations then than since. Of course, the grant money Horrobin was talking about was from the government. Remove government involvement in research and let more retained earnings of profit-making businesses be made available and you will see more responsiveness to the market, that is, to the consumers of the research.
Bureaucrats—anyone who works for or is excessively regulated by the government, including college professors who are most of the researchers I am talking about—answer only to the requirements of the bureaucracy. Profit-making businesses, on the other hand, have a huge incentive to innovate, to increase their profits; if they make mistakes, they pay dearly, and not just in lost sales or return on investment. The bureaucrat’s response to anyone who complains about the conventionality or mistakes of present-day research is to say, “I just follow the rules. We have a process to follow if the rules need to be changed. It may take time, but that’s the way it is.”
Entrepreneurial innovators and innovative researchers don’t hesitate to go outside the conventional box, which means they don’t follow rules. Their attitude is, “What can we do to push back the frontiers of business or science?” To do that, they break rules.
*Two-thirds of the roughly $58 billion spent each year on research in universities is financed by the federal, state, and local governments, 59% by the US government alone.
This blog comments on business, education, philosophy, psychology, and economics, among other topics, based on my understanding of Ayn Rand’s philosophy, Ludwig von Mises’ economics, and Edith Packer's psychology. Epistemology and psychology are my special interests. Note that I assume ethical egoism and laissez-faire capitalism are morally and economically unassailable. My interest is in applying, not defending, them.
Showing posts with label David Horrobin. Show all posts
Showing posts with label David Horrobin. Show all posts
Friday, January 20, 2012
Wednesday, October 24, 2007
The Ethics and Epistemology of Peer Review
In a previous post, I argued that academic peer review is a gatekeeping process brought about by the post-World War II growth of government involvement in research and scholarship. Though it may control quality in a narrow, conventional sense, one significant consequence of this process is the suppression of innovation. The present post takes a look at the underlying ethics and epistemology of peer review.
Medical researcher David Horrobin, whom I quoted in the previous post, says that critics of peer review “are almost always dismissed in pejorative terms such as ‘maverick,’ ‘failure,’ and ‘driven by failure.’” Lest those epithets be ascribed to me, I hasten to say that I have had some success in the process and that I am not denigrating anyone who uses it to advance his or her career. The process nonetheless does have serious flaws.
Most significant of its flaws is the view that peer review must be blind in order to maintain objectivity, that is, to prevent bias from entering the process. However, as the British Medical Journal, which has not used blind peer review since 1999, points out, “A court with an unidentified judge makes us think immediately of totalitarian states and the world of Franz Kafka.” Objectivity is the fallacy-free perception and communication of what the object of cognition is, and bias means that some other factor, such as irrelevant preconceived notions, whether formed by emotion or by reason, interferes with this perception and communication. Lack of objectivity stems from a failure to perceive reality accurately.
Neither blind nor open peer review can guarantee this accuracy. Indeed, anonymity removes the need for care and responsibility when commenting on someone else’s work. How many ill-mannered or ill-thought-out remarks would be made about submitted papers if reviewers knew that the papers’ authors will know their names and how to contact them? Being allowed to hide behind anonymity is an invitation to scurrilous behavior. This is why the objectivity of legal systems in free societies demands that witnesses, whether supporters, accusers, or expert testifiers, be identified. Contrary to the conventional wisdom of peer review, objectivity requires at minimum that the process be open.
Objectivity, at root, is an epistemological concept and the failure to perceive and communicate accurately is a function of how one uses one’s mind in the processes of perceiving and communicating. Neither anonymity nor openness will improve this. The most important requirement of objectivity while reviewing someone else’s work is a constant awareness of one’s preconceived notions. The most significant one to watch out for is “This is not how I would have written the paper; it should therefore be changed to . . .”
As one journal editor said, no doubt with some exaggeration, all of his reviewers of so-called empirical papers recommend rejection and those of theoretical papers insist that the papers be “recreated in the reviewers’ own images.” And another editor complained that reviewers have turned into wannabe co-authors, requiring extensive revisions and writing comments that are sometimes as long or longer than the original articles. Clearly, decentering, to use Piaget’s term, meaning the ability to consider other points of view or to appreciate the perspectives of others, is needed by some, perhaps many, reviewers.
Once it has been established that a paper meets a journal’s editorial guidelines and philosophy, that is, that the topic of the paper is appropriate for the journal, then it is the author’s objective that should guide evaluation. Decentering in reviewing, or editing or criticism, means accepting the premises of the author and recommending improvements in execution. The reviewer’s personal preferences on the topic, including agreement or disagreement with the author’s basic premises, should be set aside. The author’s paper is the reality to be adhered to in the reviewing process; interference from irrelevant, previously formed emotional associations and intellectual beliefs destroys the objectivity of the process.
A reviewer, of course, may strongly disagree with the editorial guidelines and philosophy of a journal or with the objective of a paper, but then such a reviewer should either decline to be a reviewer or come to terms with the principle of objectivity. Much suppression of innovation in the peer review process probably stems from the failure of reviewers to distinguish their personal philosophies and preferences from those of the authors they are reviewing. When reviewer and author disagree, the reviewer either demands conformity or recommends rejection.
The issue of objectivity in reviewing (or editing or criticizing) is similar to the so-called problem of taste in art. Is this work of art bad art or is my reaction to it just my taste? Artists have an aim for their art and their execution of that aim makes it either good or bad art. Whether one likes a particular work of art, though, depends on many other factors, including emotional associations and intellectual beliefs. Therefore, as Ayn Rand points out, it is not a contradiction to say “This is a good work of art, but I don’t like it,” and vice versa. The same can be said in reviewing scholarly work, namely, “I don’t like or agree with this paper, but it is well done.”
The reviewer, editor, or critic who can make this last statement is one who exhibits objectivity. When looked at from the standpoint of epistemology, whether the process is blind or open is beside the point.
Medical researcher David Horrobin, whom I quoted in the previous post, says that critics of peer review “are almost always dismissed in pejorative terms such as ‘maverick,’ ‘failure,’ and ‘driven by failure.’” Lest those epithets be ascribed to me, I hasten to say that I have had some success in the process and that I am not denigrating anyone who uses it to advance his or her career. The process nonetheless does have serious flaws.
Most significant of its flaws is the view that peer review must be blind in order to maintain objectivity, that is, to prevent bias from entering the process. However, as the British Medical Journal, which has not used blind peer review since 1999, points out, “A court with an unidentified judge makes us think immediately of totalitarian states and the world of Franz Kafka.” Objectivity is the fallacy-free perception and communication of what the object of cognition is, and bias means that some other factor, such as irrelevant preconceived notions, whether formed by emotion or by reason, interferes with this perception and communication. Lack of objectivity stems from a failure to perceive reality accurately.
Neither blind nor open peer review can guarantee this accuracy. Indeed, anonymity removes the need for care and responsibility when commenting on someone else’s work. How many ill-mannered or ill-thought-out remarks would be made about submitted papers if reviewers knew that the papers’ authors will know their names and how to contact them? Being allowed to hide behind anonymity is an invitation to scurrilous behavior. This is why the objectivity of legal systems in free societies demands that witnesses, whether supporters, accusers, or expert testifiers, be identified. Contrary to the conventional wisdom of peer review, objectivity requires at minimum that the process be open.
Objectivity, at root, is an epistemological concept and the failure to perceive and communicate accurately is a function of how one uses one’s mind in the processes of perceiving and communicating. Neither anonymity nor openness will improve this. The most important requirement of objectivity while reviewing someone else’s work is a constant awareness of one’s preconceived notions. The most significant one to watch out for is “This is not how I would have written the paper; it should therefore be changed to . . .”
As one journal editor said, no doubt with some exaggeration, all of his reviewers of so-called empirical papers recommend rejection and those of theoretical papers insist that the papers be “recreated in the reviewers’ own images.” And another editor complained that reviewers have turned into wannabe co-authors, requiring extensive revisions and writing comments that are sometimes as long or longer than the original articles. Clearly, decentering, to use Piaget’s term, meaning the ability to consider other points of view or to appreciate the perspectives of others, is needed by some, perhaps many, reviewers.
Once it has been established that a paper meets a journal’s editorial guidelines and philosophy, that is, that the topic of the paper is appropriate for the journal, then it is the author’s objective that should guide evaluation. Decentering in reviewing, or editing or criticism, means accepting the premises of the author and recommending improvements in execution. The reviewer’s personal preferences on the topic, including agreement or disagreement with the author’s basic premises, should be set aside. The author’s paper is the reality to be adhered to in the reviewing process; interference from irrelevant, previously formed emotional associations and intellectual beliefs destroys the objectivity of the process.
A reviewer, of course, may strongly disagree with the editorial guidelines and philosophy of a journal or with the objective of a paper, but then such a reviewer should either decline to be a reviewer or come to terms with the principle of objectivity. Much suppression of innovation in the peer review process probably stems from the failure of reviewers to distinguish their personal philosophies and preferences from those of the authors they are reviewing. When reviewer and author disagree, the reviewer either demands conformity or recommends rejection.
The issue of objectivity in reviewing (or editing or criticizing) is similar to the so-called problem of taste in art. Is this work of art bad art or is my reaction to it just my taste? Artists have an aim for their art and their execution of that aim makes it either good or bad art. Whether one likes a particular work of art, though, depends on many other factors, including emotional associations and intellectual beliefs. Therefore, as Ayn Rand points out, it is not a contradiction to say “This is a good work of art, but I don’t like it,” and vice versa. The same can be said in reviewing scholarly work, namely, “I don’t like or agree with this paper, but it is well done.”
The reviewer, editor, or critic who can make this last statement is one who exhibits objectivity. When looked at from the standpoint of epistemology, whether the process is blind or open is beside the point.
Labels:
Ayn Rand
,
David Horrobin
,
epistemology
,
ethics
,
objectivity
,
peer review
,
Piaget
Wednesday, April 11, 2007
Drop Errors and the Trouble with Peer Review
In product development there are two kinds of errors. A “go” error occurs when the green light is given to a product that eventually fails. The Edsel, a $250 million write-off by the Ford Motor Company in 1959, is one example. The “drop” error occurs when an idea that could have been highly profitable is eliminated from further consideration. How do we know that the idea could have been profitable? In a free market dropped ideas have the habit of being picked up by someone else. Chester Carlson’s invention was dropped by such notables as General Electric, 3M, Kodak, RCA, and IBM, but picked up by the small Haloid Company. In 1961 Haloid changed its name to Xerox. Even go errors in a free market often get corrected; just a few years after the Edsel fiasco, Ford rolled out a better idea called the Mustang.
Peer review is the process by which millions of dollars of government money are handed out to researchers in medicine and the physical sciences; the process by which recognition, promotion, and tenure are determined for professors, especially those in the “softer” sciences who do not need or use grants for their research; and one of the criteria—numbers of peer-reviewed journal articles, for example—used to determine accreditation for universities.
Peer review, a “blind” process in which the names of author and evaluator are concealed from each other, requires two or three so-called peers to read a paper or proposal to judge the quality of actual or proposed research before acceptance. As such, peer review is a product development process that protects only against go errors. It is at best quality control that insures accuracy and reliability of research done. At worst it holds back innovation through drop errors. Since there is no free market in scholarly research—today’s government-university-science complex is a severely hampered market—dropped ideas either never get a hearing or take many more years than they otherwise should to surface.
Medical researcher and long-time critic of peer review, David Horrobin, argued that the peer-review process, which developed in its current form largely as a screening device after World War II, has perhaps improved the accuracy and reliability of conventional research published in medicine, but it has done so at the price of innovation. Prior to World War II, unknown researchers could submit papers to journals only with the endorsement of a published author. The editor would then decide whether or not to publish. Peer review was ad hoc and not common. It was the growth of government involvement in education and, especially, the government’s lavishing of money on research that called for the blind-review screening process.
In a paper titled “The Philosophical Basis of Peer Review and the Suppression of Innovation” Horrobin urged that more unconventional and innovative research be encouraged by journal editors. When a reviewer questioned the need for such a statement, Horrobin produced eighteen incidences of medical innovations rejected by the peer-review process. In 1995 Horrobin’s paper was cited by the US Supreme Court as support for the argument that some “well-grounded but innovative theories” may not be published in peer-reviewed outlets.
Horrobin’s solution to divvying up grant money was to give funds equally to all researchers and let each work on whatever his or her interests indicated. Prior to 1960, said Horrobin, this interest-as-guide process was essentially how funding was distributed in the UK and more innovation in medicine resulted in those years than in the years since 1960. Horrobin approved of government involvement in and funding of research, but the analogy to free markets in his solution—bottom-up, self-interested choice by researchers—versus central planning—top-down, “expert” direction by peer reviewers—cannot be escaped.
Never mind that Socrates and Galileo were badly treated by their peer reviewers or that frauds and hoaxes sometimes dodge the quality controllers or, further, that you may want to cite Ayn Rand and Ludwig von Mises but can’t figure out how to get past your peer-review gatekeepers, the real problem of peer review is the severely hampered market in scholarly research. What would a truly free market in scholarly research be like?
First, publishers of journals and scholarly books would have to earn a profit from their buyers and not live off the donations of their authors or other benefactors. Some university presses, for example, are now publishing what are called “supported books,” which means someone, usually the author’s department, must contribute one or two thousand dollars to the publication of the author’s book. And at least one commercial press requires authors to do their own copy edit and provide camera-ready typeset text; this can add up to two thousand dollars or more that authors must fund. Twenty-five dollars per page, charged to authors or their departments, has long been the going rate for published papers in some fields. (In some quarters today this method of getting into print would be called subsidy or vanity publishing.)
In addition, the so-called nonprofits, which finance a portion of today’s research and journals, are in fact creatures of the tax system and must, despite their descriptive name, show an excess of donations over expenses lest their organization become some philanthropist’s very expensive hobby. Under laissez-faire, in the absence of tax write-offs and the guilt and ignorance of economics that wealthy business people tend to exhibit, there probably would be far fewer such organizations than exist today.
Second, there would be no government money to dangle in front of researchers and no government-owned or -regulated universities filled with bureaucratized product lines (curricula designed by committee), bureaucratized sales reps (the professors), or bureaucratized performance evaluations (those mounds of paper, which include lists of published research, that must be produced for promotion, tenure, and most every other consideration). All of this distorts the market and probably encourages the overabundance of pretentious minutia that fills today’s overabundance of academic journals.
Under laissez-faire, only the market would decide who produces what and who gets what in scholarly output. Indeed, the market for this research might not differ much from the product development market in automobiles. Private, profit-making firms, both traditional businesses and universities, would finance the work and effectively and efficiently produce market-satisfying results. Portions of the results might be published in profit-making journals and books, much of it perhaps not.
Yes, there might be some Edsels created by this free-market development process and there might still be some delayed acceptances of Xeroxes, but there also might be a lot more Mustangs! Absent the government-encouraged gatekeepers and other hurdles that must be jumped in order to get into the market, researchers who cannot find an outlet will be free to start their own journals, publishing companies, businesses, or even universities. The hampered market today, which includes the “golden handcuffs” of tenure, makes this quite difficult.
See related posts on the Mises blog: 1, 2.
Peer review is the process by which millions of dollars of government money are handed out to researchers in medicine and the physical sciences; the process by which recognition, promotion, and tenure are determined for professors, especially those in the “softer” sciences who do not need or use grants for their research; and one of the criteria—numbers of peer-reviewed journal articles, for example—used to determine accreditation for universities.
Peer review, a “blind” process in which the names of author and evaluator are concealed from each other, requires two or three so-called peers to read a paper or proposal to judge the quality of actual or proposed research before acceptance. As such, peer review is a product development process that protects only against go errors. It is at best quality control that insures accuracy and reliability of research done. At worst it holds back innovation through drop errors. Since there is no free market in scholarly research—today’s government-university-science complex is a severely hampered market—dropped ideas either never get a hearing or take many more years than they otherwise should to surface.
Medical researcher and long-time critic of peer review, David Horrobin, argued that the peer-review process, which developed in its current form largely as a screening device after World War II, has perhaps improved the accuracy and reliability of conventional research published in medicine, but it has done so at the price of innovation. Prior to World War II, unknown researchers could submit papers to journals only with the endorsement of a published author. The editor would then decide whether or not to publish. Peer review was ad hoc and not common. It was the growth of government involvement in education and, especially, the government’s lavishing of money on research that called for the blind-review screening process.
In a paper titled “The Philosophical Basis of Peer Review and the Suppression of Innovation” Horrobin urged that more unconventional and innovative research be encouraged by journal editors. When a reviewer questioned the need for such a statement, Horrobin produced eighteen incidences of medical innovations rejected by the peer-review process. In 1995 Horrobin’s paper was cited by the US Supreme Court as support for the argument that some “well-grounded but innovative theories” may not be published in peer-reviewed outlets.
Horrobin’s solution to divvying up grant money was to give funds equally to all researchers and let each work on whatever his or her interests indicated. Prior to 1960, said Horrobin, this interest-as-guide process was essentially how funding was distributed in the UK and more innovation in medicine resulted in those years than in the years since 1960. Horrobin approved of government involvement in and funding of research, but the analogy to free markets in his solution—bottom-up, self-interested choice by researchers—versus central planning—top-down, “expert” direction by peer reviewers—cannot be escaped.
Never mind that Socrates and Galileo were badly treated by their peer reviewers or that frauds and hoaxes sometimes dodge the quality controllers or, further, that you may want to cite Ayn Rand and Ludwig von Mises but can’t figure out how to get past your peer-review gatekeepers, the real problem of peer review is the severely hampered market in scholarly research. What would a truly free market in scholarly research be like?
First, publishers of journals and scholarly books would have to earn a profit from their buyers and not live off the donations of their authors or other benefactors. Some university presses, for example, are now publishing what are called “supported books,” which means someone, usually the author’s department, must contribute one or two thousand dollars to the publication of the author’s book. And at least one commercial press requires authors to do their own copy edit and provide camera-ready typeset text; this can add up to two thousand dollars or more that authors must fund. Twenty-five dollars per page, charged to authors or their departments, has long been the going rate for published papers in some fields. (In some quarters today this method of getting into print would be called subsidy or vanity publishing.)
In addition, the so-called nonprofits, which finance a portion of today’s research and journals, are in fact creatures of the tax system and must, despite their descriptive name, show an excess of donations over expenses lest their organization become some philanthropist’s very expensive hobby. Under laissez-faire, in the absence of tax write-offs and the guilt and ignorance of economics that wealthy business people tend to exhibit, there probably would be far fewer such organizations than exist today.
Second, there would be no government money to dangle in front of researchers and no government-owned or -regulated universities filled with bureaucratized product lines (curricula designed by committee), bureaucratized sales reps (the professors), or bureaucratized performance evaluations (those mounds of paper, which include lists of published research, that must be produced for promotion, tenure, and most every other consideration). All of this distorts the market and probably encourages the overabundance of pretentious minutia that fills today’s overabundance of academic journals.
Under laissez-faire, only the market would decide who produces what and who gets what in scholarly output. Indeed, the market for this research might not differ much from the product development market in automobiles. Private, profit-making firms, both traditional businesses and universities, would finance the work and effectively and efficiently produce market-satisfying results. Portions of the results might be published in profit-making journals and books, much of it perhaps not.
Yes, there might be some Edsels created by this free-market development process and there might still be some delayed acceptances of Xeroxes, but there also might be a lot more Mustangs! Absent the government-encouraged gatekeepers and other hurdles that must be jumped in order to get into the market, researchers who cannot find an outlet will be free to start their own journals, publishing companies, businesses, or even universities. The hampered market today, which includes the “golden handcuffs” of tenure, makes this quite difficult.
See related posts on the Mises blog: 1, 2.
Labels:
bureaucracy
,
David Horrobin
,
drop error
,
free market
,
peer review
Subscribe to:
Posts
(
Atom
)

Jerry Kirkpatrick's Blog by Jerry Kirkpatrick is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.