Science Tribune - Article - November 1997
Do the editors of peer-reviewed biomedical journals promote the advancement of science and medicine ?
E-mail : firstname.lastname@example.org
In the delightful musical city of Prague, during a gloriously bright autumn week, over 300 people interested in medical publication were confined in a hotel conference room discussing the subtleties of peer review (International Congress on Biomedical Peer Review and Global Communications, September 1997). Despite the criticisms that can be waged against peer review, many scientists and doctors consider it a pillar - like the statues of the stooping giants by Matyas Braun that flank the doorway of the Clam Callas Palace in Prague - that upholds the integrity and worth of biomedical publications. To endow a pillar with the importance of an entire foundation is, however, foolish even if the unavowed and unconscious aim of the program organizers may have been to divert attention from some of the less tangible issues of publication. The congress' overemphasis on peer review tended to obscure the importance of other aspects of global communications in the sharing of medical knowledge and expertise.
Evaluating peer review
At times, I was even under the impression that I had landed in a congress from another discipline. Results of randomized controlled studies of peer review processes were presented in the same style, with the same logic and methods, as the results of clinical trials of treatments. Should the methodology of clinical trials so obsess medical journal editors that they can only reason within this limited framework ?
Peer review is a social practice where the interplay among the participants may be closer to a situation comedy than to standard two-by-two comparisons of treatment outcomes. Like any social structure, it has its rules and regulations and also underlying political, economic and behavioural motivations that govern its "raison d'être" and survival. If we must use a clinical analogy, then the behaviour of peer reviewers may be closer to that of the doctors who meet out the treatments than to that of the patients whose response is sought.
But, in randomized controlled trials, doctors are often under the heel of sponsors. Are peer-reviewers dominated by editors ? Consequently, are medical editors themselves the most suitable people to analyse their own traditions and judge the intrinsic quality - as opposed to efficacy - of their practices ? Are the criteria and methods of clinical trialists the most appropriate to analyze complex situations and interpersonal relationships ? Should modern clinicians - and the new breed of biostatisticians they attract - despise the knowledge of those whose profession is to survey and describe hierarchical and lateral power relationships, in other words, social scientists ? Are such scientists, in their opinion, scientists in name only ? However, proud an expert may be of his/her own methodology, it is unwise not to listen to the experience of others.
Confusing the aim and means
Every activity, even biomedical publishing and peer review, has its run-of-the-mill tasks which is not to say that they are not important. Analyzing variables such as turnabout time, methods of choosing reviewers from databases, blinding or masking of reviewers (a), etc.. is essential to ensure that the cogs are well oiled. Defining the elements that form a coherent manuscript (e.g. detecting incorrect information, misplaced information, inconsistensies, omissions) and trying to improve the piece by, for instance, requesting structured abstracts (b) are also necessary. But, if all this is just so that author and editor turn out the perfectly structured, well-written, intelligently referenced but essentially irrelevant paper, why bother ! The real question is who - the reviewer, editor, or some other person - is capable of recognising originality and promoting it. Or, are we, perhaps, in a world where the name of the game is just asking fashionable questions and performing standard exercices.
Satisfying the market
The majority of biomedical journals are hardly benevolent institutions for the advancement of science and medicine. Most are either the official organs of professional associations and/or the sales products of publishing companies out to make money, as any self-respecting firm should do. Am I implying that these companies interfere with the choice of work that is published ? No, I am not, even if such a situation may arise from time to time when, for instance, publishers insist upon changing editors-in-chief and their editorial boards in order to promote a refurbished journal with a smarter look. What I am implying is that there is a natural instinct for an editor worth his mettle to create and publish a brand of journal that will please and sell well. If the public wants chat-shows and debates, it will provide correspondence columns; if it wants documentaries, it will provide reviews; if it wants news, it will provide congress reports; if it wants thrillers, it will provide press releases; if it wants opinions, it will provide editorials, and so on. In his own field, the average doctor is no different from the average television viewer.
Editor versus peer reviewer
What is the role of the peer reviewer, who is asked to judge the research, in this enterprise ? In many instances, his/her opinion, even if sought, is not acted upon, the final, often irrevocable, decision being taken by desk editors alone or in committee.
Now there is a crucial point we heard relatively little about at the congress, namely, editorial policy. Journals reject manuscripts outright without sending them out for peer review because they are not within the scope of the journal or they are just too bad. Are these the only reasons ? The editorial decisions of some journals only match 50% of reviewer recommendations. Why ? The easy, no doubt honest, excuse is that there is not enough space for everyone. But why is there space for some and not for others ? On what basis is this decision taken by the gatekeepers ?
- the impact of the discipline ? But who says that your prostates are more important than our ovaries !
- the reputation of the team ? But won't we end up hearing from the same teams (or their clones) all the time !
- the news value ? But why should other media (press, radio, TV) decide what is newsworthy in the medical publishing world ?
- the clout of the sponsors ? I hope not !
- the needs of the reader ? This would be laudable indeed if he/she has not yet been programmed to ask the politically correct questions. In the publication of modern medicine and science, there is a sort of logical continuity where each paper builds up on the preceding one (this is how most introductions to papers are written). What if whole series of studies are moving toward something of a dead-end ? What if the editor is adding value to the valueless ?
The vicious circle : research projects and published research
Any medical editor will rightly point out that he/she does not decide - directly - which research is done in the institutions, which clinical trials are performed in hospitals, etc... The results submitted for publication are those that arise from the decisions of institutional research policies. But hey ! Editorial board members, peer reviewers and protocol committee members are look-alikes. Meet one, meet them all ! And on what are the appointments and reputations of these senior members of the medical arena based ? But, on their academic achievements and on their publications, preferably in the highest impact journals, of course (c). So even though journal editors have no direct authority on the type of research that is performed to-day, their indirect influence is far from negligible. And can any specialist or expert acting as peer-reviewer claim to attain such a high degree of objectivity that he/she has no conflict of scientific interest (d) ? Let's not forget that some of the most novel research in many fields has been published, virtually unnoticed, in very minor journals.
Who can break this vicious circle that is creating a new conventional science and medicine ? There is room in some journals - and much appreciated it is - for cries in the desert, but a scientist labelled a heretic becomes a heretic for life when, instead, he should be metamorphosing mainstream science until the arrival of new heretics (1) (2) (3). It takes time for ideas to mature, for technologies that help 'prove' hypotheses to be developed, but are there not ways, in a world of increasing standardisation (e), of accelerating the process and letting educated readers make up their own minds ?
Competing for the reader's time
A reader tends to retain only what he needs and what interests him. There is thus a strong case for providing him with the greatest choice of subject matter possible together with a set of tools for finding what he wants or likes and for measuring up his a priori opinions, even prejudices, against those of others. Does he just need the expert opinion of Professor so-and-so on the results of the latest trial, mega updated reviews, and questionable meta-analyses of trials that might not be truly comparable because the "right" parameters were not measured to answer the "same" question ? Doesn't he also want to know whether the trial was actually worth performing and why ? Something about hypothesis finding and not only hypothesis testing ? And when faced with the problems of his own patients, mightn't he feel happier if he had access to the descriptive views of colleagues rather the quantitative measures of experts. Qualitative criteria are not to be ignored just because too many people believe that they are tainted by too much subjectivity. Any practitioner wanting to take an appropriate course of action would, no doubt, welcome information from studies performed in areas with similar catchments, product availibility, environment, economic contingencies, etc.. as on his home ground (4). Doctors play at home and not away !
Will the Internet broaden the scope of biomedical journals ?
Richard Smith, the editor-in-chief of the British Medical Journal, told the audience at Prague that, in his opinion, the most interesting facet of many pieces of research work was the exchange of views between peer-reviewers and authors rather than the final published paper. Is there then a case for sharing these views with the public, for instance, via the Internet ? A study performed by the Medical Journal of Australia illustrated the bonuses in implicating a virtual college not only of peer-reviewers but also of readers in the critical appraisal of a manuscript on the Internet. Clearly, a wider, more open discussion can be fruitful.
There are, however, criticisms against more open debate that do have their justification. How does one sift the wheat from the chaff and eliminate "junk" information ? Does the practitioner have the precious time to weed through the discussions to find what he is looking for or what just might interest him ? Is there not a need for the professional judgement of experts in whom practitioners have faith ?
My own view is that there is a need and room for both. The meek will always follow and feel reassured by the guidance of expert opinion. Why not let them have it ! But the others - the curious, the highly spirited, those with a touch of genius or madness - will surely welcome the absence of a straight-jacket and the ability to discover for themselves. It is true that, for the moment, the search tools for finding and analyzing targetted information are still rudimentary. Internet search engines are just a beginning. But electronic technology moves fast and new tools can be used not only to handle the run-of-the-mill tasks of peer-review and publishing but as intelligent aids. Data mining and text mining (5) based on neural networks and genetic algorithms can help find and process information and may turn out to be the keys that will unlock the doors of knowledge for an increasingly large proportion of scientists and doctors.
(a) The definitions of 'blinding' and 'masking' may vary. By blinding, I mean concealing authors' identities from reviewers, by unmasking, revealing the reviewer's identity to other reviewers. According to presentations at the congress, these procedures have little, if any, impact on the quality of the reviews.
(b) Structured abstracts do not necessarily improve the reporting of results; omissions and inconsistencies between text and abstract are still encountered. It would seem that just not enough authors read the instructions to authors with the aim of putting them into practice.
(c) In many small specialist journals, the readers are the authors are the peer-reviewers and editors. The same population holds all three functions. Although this can have advantages, it tends to exclude outsiders who might inject new blood into a discipline and provide an interface with other disciplines.
(d) The directions mainstream science and medicine are taking may be less influenced by conflicts of financial interest than of scientific interest. The whole process of biomedical publication in fact rests on the integrity and objectivity of the authors.
(e) Global analysis of the publication profiles of the 48 most prolific nations in 18 scientific disciplines has shown that nations are spreading their research effort across disciplines in increasingly similar ways (unpublished).
1. Sangalli A. They burn heretics, don't they ? New Scientist, p.47, April 6, 1996.
2. Kuhn TS. The structure of scientific revolutions. 1962 (2nd ed enlarged, Chicago, 1970)
3. Fleck L. Genesis and development of a scientific fact (TJ Trenn & RK Merton, eds) Univ. Chicago Press, 1979 (originally published as Entstehung und Entwicklung einer wissenschaftlichen Tatsache: Einführung in die Lehre vom Denkstil und Denkkollektiv, Benno Schwabe, Basel, 1935.
4. News article. Is this the end of research as we know it ? Br Med J, Vol. 315, Aug 16, 1997.
5. LeBret C. Have you heard about data mining ? http://www.tribunes.com/tribune/art97/lebe.htm