By Lambert Strether of Corrente
As it turns out, I have narrow but deep expertise relevant to the discussion of the New England Journal of Medicine (NEJM) paper on the Pfizer vaccine study, “Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine” (“Safety and Efficacy”), so I thought I’d weigh in, albeit a little late, on the protocols and “exclusion criteria” discussion.
In a past life, several careers ago, I consulted to various large firms, among them medical publishers, on complex document structures; my deliverable would be a formal definition of such structures, plus documentation. For example, and without getting into the weeds on syntax, the structure of all posts at Naked Capitalism includes editorial elements like: a headline (one, required), author (at least one, with an optional bio), and then a long string of running text elements (paragraphs, images, tables, lists, and so on) in any order and any amount greater than one. There are also elements that occur within running text, like emphasis (italic), or links (with their URLs). Run your eye up and down this page, and you will see what I’m talking about. (No, this is not an exhaustive list, and there are plenty of decisions to me made about representation; but let’s not go into those weeds.) I loved that career, because it allowed me to combine my skills as a humanities major with exacting technical work.
Publishers liked having formal definitions of their document structures because (again leaving out syntax) it was possible to have a digital editorial system that converted a single document to several formats without losing information. The NEJM, in fact, faces that very requirement. If you look at the article in question, you will see that it begins with the article type (“ORIGINAL ARTICLE”), then a headline (“Safety and Efficacy…”), then the authors, then two buckets, one the article, and the other figures/media. The article begins with an Abstract, which begins with Background, Methods, and so forth. The NEJM delivers documents structured in this wise in at least three media: (1) online, (2) PDF, both of which we common readers can read, and (3) in a printed version, which we cannot. (There may also be versions for tablets, library readers, even CD; I don’t know, but you can see the advantage of having a single master document composed according to a formal structure and then converted, as opposed to three or more different documents, all of which must be kept in synch with each other.) Of course, the publisher makes an implicit bargain with the reader that all the versions are identical or, if different, are labeled as such[1]. Or not, as we shall see!
Readers (many of them) like documents that are consistently structured (the formalism is, of course, hidden from them) because structure makes it easy to find what they want. Readers expect to be able to flip to the end of a book to find the index, for example, and so if they encountered a book where the index was in the middle, they would justly conclude either that the publisher’s printer had committed a terrible error, or that the author was some sort of pomo buffoon bent on japery. Readers who value their time, like doctors — I well remember a doctor in my study group, who would exclaim “I could be treating patients!” whenever he felt time was being wasted — appreciate structure even more; if they want to know the methods, they go to the Methods section; if the results, the Results section. If editorial matter does not appear where it is supposed to appear, such readers can and will justly conclude the matter is missing, because that is the bargain the publisher made with them through the pattern and practice of structuring the document consistently, issue after issue after issue.
Having set the scene at tedious length — I really did love that work — we now come to IM Doc, who was — I contend — betrayed as a reader by NEJM’s failure as a publisher to uphold its end of the bargain it made with him when it butchered its document structure and its own editorial standards at a critical juncture[2].
IM Doc, in “An Internal Medicine Doctor and His Peers Read the Pfizer Vaccine Study and See Red Flags [Updated]” wrote:
First, a critical issue for any clinician is “exclusion criteria”…. From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria.
The “exclusion criteria” are to be found through another document element called a Protocol (hat tip Rick in Oregon) about which more in a moment.
They certainly are never mentioned…. And now we know there were exclusion criteria, not because of anything Pfizer, the investigators, or the NEJM did but because of stunning news out of the UK. UPDATE: I will address this at greater length, but an alert reader did find the study protocol, which were not referenced in any way that any of the nine members in my review group could find, nor were they mentioned in the text of paper or editorial, as one would expect for a medication intended for the public at large. I apologize for the oversight, but this information was not easy to find from the article, not mentioned or linked to from the text of the article, the text of the editorial, in the “Figures/Media,” or in a supplemental document.
I need to disentangle this a bit, because I think IM Doc is generously taking on a bit more responsibility than he or needed to.
First, we need to consider what is meant by “the article.” As we know, there are at least three versions of “Safety and Efficacy”: The online version, the PDF version, and the printed version. Which one was IM Doc reading? Certainly the PDF version (assuming the printed version had not yet arrived in the mail[3]). We know that from IM Doc’s text, because he tells us so himself: He cites to “page 5, in Table 1,” to “tables on pages 6 and 7,” and “to tables on page 7.” The online version has no pages. We also know that because of the social setting in which “Safety and Efficacy” was read. Reader KLG described this setting (IM Doc’s “journal group”) in Yves’ follow-up piece as follows:
IM Doc [was part of a] like-minded journal club (a common mechanism for all biomedical scientists and many clinicians to keep up with current developments)
And IM Doc describes the meeting:
we had an ad hoc meeting of our Journal Club to discuss the NEJM article
It is extremely difficult for me to imagine that each member of the Journal Club, whether on Zoom or in person, was reading the online version on a device (one person an Android phone, another on an iPhone, the fourth and fifth on two iPads of different sizes, the sixth on a laptop, and so on). It’s far more likely they were reading the PDF, and most likely printed out, because that way, when one reader says “flip to Table 1 on page 5,” all the readers can easily do that simultaneously (rather than swiping, tapping, scrolling, and so on).
So now we come to the first extremely simple reason why IM Doc and his study group, working from the PDF, could not find the Protocol which contained the exclusion criteria.
They could not find the Protocol element in the PDF because it was not there.
NEJM’s document structure treats protocols as separate, external documents[4], and puts a link to them, called Protocol, within an element called Supplementary Material[5], which is placed at or near the end of the article in the online version only. Here is the online version, which contains the Supplementary Material:
And here is the PDF version, which does not:
Once again, the Supplemental Material, which includes the Protocol with its inclusion criteria, is critical. As IM Doc writes:
First, a critical issue for any clinician is “exclusion criteria”. This refers in general to groups of subjects that were not allowed into the trial prima facie. Common examples would include over 70, patients on chemotherapy and other immunosuppressed patients, children, diabetics, etc.. This issue is important because I do not want to give my patient this vaccine (available apparently next week) to any patient that is in an excluded group. Those patients really ought to wait until more information is available – FOR THEIR OWN SAFETY. And not to mention, exclusion criteria exist because the subjects in them are usually considered more vulnerable to mayhem than average subjects. From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria. They certainly are never mentioned.
Wnat possible excuse is there for leaving the Supplemental Material out of the PDF? Saving a few bytes?[5]
There is a second, slightly less simple reason why IM Doc and his study group, working from the PDF, could not find the Protocol which contained the exclusion criteria. This concerns how NEJM formats references to the Protocol element in the running text. Here is how references in running text to elements in Supplementary Materials should be formatted, according, at least, to the Journal of the American Medical Associations style guide:
NISO (the National Information Standards Organization) agrees:
So, one would expect references in running text to the Protocol to appear along the lines of “see the Protocol in Supplementary Materials, to flag the “critical part of the evidence” for the reader — that’s one reason we have capitals — and to guide them to its location. That is not what NEJM did. Here is one example:
There’s no formatting in the PDF to indicate what “the protocol” is, and no indication of where it is to be found. All the other mentions of “protocol” are like that. So no wonder — in the trackless wilderness of NEJM’s house style — the Protocol was difficult to find.
Summarizing:
(1) NEJM’s editorial practice removes the Supplementary Material element from the PDF version, and hence critical material like the Protocol element that contains the link to the external protocol document, which contains the “exclusion criteria” sought by IM Doc. It is entirely reasonable for readers to expect both online and PDF versions to be identical with respect to critical material.
(2) NEJM’s editorial practice permits the copy desk to format text references to the Protocol element in lowercase, with no indication of the element in which it is to be found. It is entirely reasonable for readers to expect references to critical, named elements to be formatted according to JAMA and NISO standards, wihc at a minimum require initial caps.
(3) The net result of NEJM’s editorial practice was that a study group of nine practicing, busy, and stressed physicians were unable to find critical material affecting treatment because (a) the material was not there to be found and (b) there was no signal to show where it was to be found. NEJM should know its readers better.
(4) NEJM should fix its editorial practice. It’s broken.
NOTES
[1] If matters were otherwise, readers would be put in the impossible and absurd position of having to read all the versions of any given paper and check each for consistency. As it turns out, this is exactly the situation the NEJM has put its readers in. (This reminds me very much of Obama organizers who would say “Go look at the website!” when questioned on details of policy. No.)
[2] The misadventures that came to happen with an article that was one of the most important NEJM published this year, and perhaps for many years, would be a matter for internal inquiry at NEJM. Since the editor-in-chief of NEJM was one of the authors of the “EDITORIAL” that accompanied “Safety and Efficacy,” such an inquiry would presumably be easy to initiate.
[3] The printed version and the PDF version should be identical in any case, because generally the PDF is what the publisher sends to the printer, albeit with the addition of crop marks, etc.
[4] Treating the protocol as an external document makes sense, because the protocol is often quite long, and will be formatted in the style of whoever is submitted the article — in this case, Pfizer — and not in the journal’s house style.
[5] There is another element, confusingly also called Supplemental Appendix and placed at the end of the article. A link to the Supplemental Appendix seems to have made a brief appearance somewhere near the top of the screen for the online version, and then vanished mysteriously, as described by both Yves and reader KLG here. It did not and would not have contained the Protocol element, which goes in the Supplemental Material section.
[6] PDFs can also be made clickable, so not only could the Protocol element have been included within in, it could have linked to the protocol.
62 comments
Comments are closed.
i wish there was some efficient and easy way for a group of interested people to collaborate on looking at the data…
Yes, I agree. All these related posts are collaborated and written by people who have more expertise than I do. Yet, I have been able to benefit from their discussion and disagreements. I really appreciate this. It has helped me to be better informed.
Thank you, that’s clarifying.
+1.
Very helpful on structure. One wonders about the validity, let alone utility and honesty, of the substance.
> One wonders about the validity, let alone utility and honesty, of the substance.
My rule of thumb* is that an article is bad, the institution that produced it is presumptively bad as well. The horrid copy editing that the New York Times is a symptom of dysfunction, and sure enough, we’ve just had another scandal about a reporter who made shit up. And that’s before we get to the 1619 Project.**
NOTE * Dr. Samuel Johnson called this the pack-thread test:
The packthread test passes the test of independent invention, since I stumbled on what I called the “random opening test.” I open a book to a random page somewhere in the middle (i.e.. not the Preface at the beginning or the apparatus, if any, at the end). Then I read a paragraph or two. If those paragraphs are bad, I don’t read any further. I haven’t read many bad books.
NOTE ** Yves’ typos don’t count, because that’s down to understaffing relative to the quality and quantity of work demanded relative to the funding available. That is certainly not the case at The World’s Greatest Newspaper™ or at the NEJM, a flagship publication of the America health care business or industry.
“I open a book to a random page somewhere in the middle…”
I was taught this technique as “the page 99 test.” Works every time (except for short books).
The test of independent invention FTW! (Of course, if that test were widespread enough, sociopathic managers would be optimizing page 99 only…)
Lambert, I don’t know if you know this, but as my old blog Norwegianity transitioned from mostly profane anti-war rants to links aggregation, I copied your formatting. Then tweaked it, abused it and in general used it as therapy because it was such a great template for communicating on multiple levels.
I didn’t know it was you specifically who designed the format for the morning and afternoon links aggregating, but it’s a model of simplicity and clarity. I’ve always admired it and now that I know who to thank, thank you.
My business writing service was more than just resumes but the more I had to deal with very poorly designed hiring forms, the more I came to appreciate clarity in formatting.
But my real tribute to you is that you use these skills for good, not evil. Not joking, the more I got into formatting, the more I used it to trick the reader’s eye, drawing their attention to positive information while distracting them from less favorable text. Resumes are ad copy and I did everything in my power to sell my clients.
As a blogger every once in a while I’d do clickbait, teasing my readers with accompanying text that made it hard not to want to find out more. And then they’d click and find something else entirely because I am a very untrustworthy person and found that amusing. And, hopefully, instructive (don’t trust me or anyone else). You sir do not do clickbait or mess with your readers. Your excerpts are honest and upon clicking I invariably find that your pull quotes speak to the heart of the linked article. When they’re more of a sidebar, you indicate as much.
You aren’t just an honest communicator, you’re driven by a need to convey the truth however deeply buried it may be as clearly and unjudgmentally as possible. Any subsequent editorial comments carry more weight as a direct consequence. Even your snark seems understated and more than fair to all concerned.
So kudos, thanks and just plain @#$! wow. You’re the best at what you do and while your readers appreciate your work, I often think they don’t understand how really well you do it.
Your last sentence, Mark, is astute. Someone makes braised beef cheeks, smocks a dress, uses a floor sander – and we think, well, how hard can that be. Turns out to be harder than we think. We know links aren’t tossed on the floor to arrange themselves, but putting them in buckets is a little tricky. So, thanks for pointing it out, and thank you, Lambert.
Lambert is responsible for the more complex WC layout. NC’s links and other features pre-date Lambert joining the site.
> NC’s links and other features pre-date Lambert joining the site.
They are perfect as they are. Otherwise, I would have tried to change them.
Hey, you don’t need to go that far! But I would have welcomed suggestions.
I picked an old Blogger template, Simple, and we reproduced that in WP when we went to WP. Various subsequent WP people have suggested changes, most of which were rejected (and Lambert was very much part of that process after he came on board) but some were value-added, like our header bar with About and Policies.
Then I redirect my compliments other than the parts about how Lambert uses your template. Thank you Yves.
Because it’s so much more than just an outlining process. It’s the most organized and easy to process part of my day, even when the news makes me mad.
Accessibility is critical, and the ease with which readers get news from you has made all of us better informed. (Also less satisfied with the world but I’m not into shooting the messengers.)
You’re a better team than I realized. And, now that I think a little harder about it, a transparent, straightforward presentation is in fact naked (not tarted up like “real” news sites).
Thanks!
Another big reason the site is easy to process is few ads. No intertext ads (we’ve approved them only for mobile where people are used to swiping and then we have only a few). No popups. No video autoplay.
Thank you.
Interrupting a reader’s train of thought trying to comprehend an article with an ad one cares not about (thanks not Google), is earning micro-pennies that does not create understanding of said article better.
Penny-wise, comprehension foolish.
> You sir do not do clickbait or mess with your readers. Your excerpts are honest and upon clicking I invariably find that your pull quotes speak to the heart of the linked article. When they’re more of a sidebar, you indicate as much.
[lambert blushes modestly]
I appreciate Lambert Strether’s pull quotes, his summarizing, his laconic, sometimes word- or phrase-coining commentary, his bucketing, just the way that Strether’s much appreciated methods coordinate with those of the other much-appreciated, similarly talented reporters at Naked Capitalism, not to mention the valuable news content itself.
Naked Capitalism is my “front page” news source, because it took years to find a handful of blogs like Naked Capitalism’s, Duncan Black’s, and Dean Baker’s, that respect readers as much.
> Duncan Black
Atrios was my blogfather, back in the day. Respect for readers is indeed one of his character traits (although I never thought of it that way). He also has a genius for concision.
You, he, and if you’ve ever listened to him talking at a union meeting, Dean Baker. When news twitter got useful, and people were live tweeting speeches, I noticed even Greg Sargent is good at the impossible combination of concise and comprehensive. So rare. Thanks again.
Very nicely researched and presented, Lambert. Format is important in scientific documents. It’s all part of the reader’s level of trust in the information s/he’s seeing (or not seeing).
> It’s all part of the reader’s level of trust
Exactly. Funny how often that issue crops up….
NEJM should know its readers better.
I assume that is irony, as I am sure they know their readers very well. Every website gets metrics, page views, etc. And they (NEJM) also know how many paper copies are delivered. So, important information omitted i any version is at least known/knowable, if not planned.
Once again, thank you, Lambert, for the yellow waders and the deep dive. Could it possibly be that we are being lied to? Isn’t there something in criminal law about method, motive, and opportunity?
Actually, I’m not sure your assumption is correct. NEJM sells subscriptions. Its advertisers want MD eyeballs. KLG said he downloaded the paper through his uni. I suspect most doctors get to NEJM through an employer subscription, so there’s a log-in. I would assume NEJM does care very much about those views, but not so much those of readers who access the public site…..even though that could include MDs from home.
In other words, I’m not sure what their mix of subscription revenues v. ad revenues is. They may get a decently high % from subscriptions.
> I am sure they know their readers very well
One aspect of knowing your readers well is knowing how and what they read. I agree that whichever MBA infestation is managing NEJM looks at some dashboard with metrics, but if IM Doc’s nine-person study group — confirmed by KLG in his network — can’t find critical information affecting patient treatment plans (hence, outcomes) in the midst of a pandemic, then they don’t know their readers at all.
The next step in this chain of logic is to surmise that the NEJM “editorial board” has stopped thinking primarily about the reading experience of it’s core demographic, much less the dissemination of usable information, and concentrates, like all good MBA’s should, on viewership metrics as that influences readership of secondary sources, such as ads and ‘slanted’ editorials.
We live in interesting, if dissolute, times.
Stay safe all!
PS. Yesterday, while standing in line at the Post Office to mail off some Christmas packages, I struck up a conversation with a late twenties looking man. He had heard of this website and was somewhat surprised that this graying geezer had also. Can there be an age component in the readership of the Dreaded NC? Then, at the parting, he wishes me a Happy Solstice. There is hope for us yet.
Or they do and intend to deceive them.
The presence of the Editor as an author of the subject article supports this thesis.
Occam’s razor, I suppose, suggests incompetence per Yves below, but these things always tend to tilt the same direction. I guess I’m getting more paranoid through experience.
I suppose that this is a case of instead of the New England Journal of Medicine trying to defend the indefensible, have decided to take away the indefensible part so that there is nothing to defend. The Lancet would be impressed I am sure.
Never attribute to malice that which can be explained by incompetence.
This was formerly a guiding principle of mine, and it led me astray. Pretence of incompetence is, of course, a standard recourse of persons caught out in malicious behavior. And a malicious nature predisposes a person to certain sorts of incompetence. Incompetent people are a menace in any case, absent concrete evidence of goodwill they are best presumed malicious.
Have you ever heard a person who somehow always manages to say something hurtful, defended with excuses like “He just doesn’t get it?” I say consistency is a hallmark of competence, and often, in fact almost always, the only meaningful indicator of intent.
That’s not incompetence. That’s character, an entirely different matter. Passive aggressives are the worst.
Lambert is talking about institutional incompetence, an entirely different category. Often the result, as some have said, of rule by MBA, particularly firing people in low level positions that the top brass deems to be unimportant but have critical skills and/or institutional memory. Like the NYT firing its copy editors.
I have had 2 days of non-stop time sinks resulting from incompetence, starting with it taking 2 hours to find out what my mother owed in property taxes and make a guess as to how to pay. Aside from the remarkable difficult in finding her record (didn’t come up in a name or property address search, although when I did find it eventually, both were spelled correctly), the site didn’t say who the payee on a check should be or where to send it (I used the payee name from a prior year in her check stubs) and the “pay online” feature was unusable (even to use a credit card, you had to have an existing account; no way to create a new one). And no one answers the phone in the Tax Dept either, and the relevant extensions don’t have VM boxes set up.
I called the assistant to our City Manager, who is on the ball. She didn’t have a # in the county tax dept and said her husband goes to the Jefferson County Tax Collectors office IN PERSON to make payment.
The competence I had in mind in that example was the ability to recognize what is hurtful. Passive aggressives are indeed the worst, but a passive aggressive who can’t figure out how to hurt you – phew!
I actually had institutional phenomena in mind though, and your experience sounds like it might even exemplify it. Whether it’s rule by MBA, or Confucian scholar, or whoever’s down at the office in Jefferson County, enforced norms of incompetence are likely to be key in bureaucratic power structures. Subordination through a routine of humiliating complicity. Malicious stuff.
It’s self-reinforcing of course, and absent exterior correction the required malfeasance tends to escalate. You can end up with the institutional equivalent of dementia, or florid psychosis, “$600 is a lotta money” “Russians!!!” and suchlike.
I hope you don’t have to go anywhere near a government office for the forseeable future. Come to think of it, maybe there’s no one left to answer the phones?
thing is, why is it that this incompetence is always PROFITABLE?
I’m not sure it always is. The Democrats make a lot more sense to me if I reckon the misery of the undeserving is its own reward.
Thanks for this, it explains so much. I’m always in a low grade war with my employer over their insistence on brevity and box ticking in my reports and my obsession with clarity and structure. I recall once being told to not write introductions to my reports ‘to save time’. I was so astonished I didn’t even argue back (I just passively aggressively refused to co-operate).
It does seem beyond incompetence that a journal could produce a document like this which is nearly impossible for professionals to navigate. But given how the academic publishing world is going, it would not surprise me if this was the result of simple carelessness and cost cutting, rather than some sort of deliberate attempt to mislead. Although in my experience of ploughing through technical reports, poor structure is more often than not quite deliberate, it is the simplest way to deceive without lying.
> poor structure is more often than not quite deliberate, it is the simplest way to deceive without lying.
You remind me of the Edward Tufte’s famous idea that the bullet-pointed slides of the Challenger disaster reflected the structure of the institution that produced the slides (akin to Conway’s Law). All the happy talk was in the big bullets at the top, and all the caveats and problems were in the little bullets at the bottom (too lazy to find the article, sorry).
If so, at NEJM we might expect to find certain levels of indirection and equivocation regarding empowering readers to find data to back up claims. (This would be somewhere in between “malice” and “incompetence”). I don’t have time to do a worked up example to see whether Tufte’s dynamic holds at NEJM, but its certainly a suggestive thought.
Its extremely difficult to interrogate technical documents without years of experience of reading specific types. In a past life I had to plough though road traffic projections and analyses, usually for anticipating the traffic impacts of construction schemes. On the surface, these are very simple – the maths is not complicated and the general principles are easy enough for anyone to understand who has ever driven a car. But there were fiendishly multiple ways that an apparently simple analysis could be created to back up whatever argument the engineer wanted (usually: ‘don’t worry, this won’t cause a traffic jam, we promise’).
I wasted many an hour trying to work out if something was wrong when it didn’t seem to make much sense. Even well trained engineers struggled – but any consultant with enough years of experience had the heuristics to immediately see if the assessment was reasonable or not, often without reading anything more than a few headline figures. As one example, using as a baseline traffic during a known quiet period (say, at the end of summer vacation times) was a strong indicator that they were trying to twist the figures to make things look favourable. In the end, I gave up struggling with a calculator and found that applying a few simple rules was far more effective at identifying problems than trying to re-run the analysis. But it takes a lot of experience of any given type of technical document to work out those rules.
> Its extremely difficult to interrogate technical documents without years of experience of reading specific types.
You are correct, and this is one of the skills that citizen scientists — who do not have to be professionals — develop when they get involved in permitting battles.
I agree with some caveats about citizen scientists. I’ve come across many talented amateurs who have identified issues with bad developments/proposals, but I also think that there is a ‘cascade’ factor when it comes to knowledge about a topic. In my experience, there is no such thing as a true ‘expert’, there are specialists, generalists, knowledgable laypersons, etc., all of whom should have an input. On many occasions I’ve found that the ‘specialists’ were wrong, primarily because of the absence of a wider perspective (this is one reason I think why we are in such a mess over Covid). But of course, as I’m something of a professional generalist myself, I’m hardly unbiased.
Just to give an example, a couple of years back I was in dispute involving possible water contamination of a salmonid river via groundwater from a waste processing facility. There were inputs from fishermen, ecologists, freshwater specialists, chemists, hydrogeologists and engineers working for the company that could have been responsible, and an academic specialist in hydrogeology (who happened to have been the main teacher of the course that the engineers who carried out the assessments had attended). What was particularly fascinating is that all their inputs were ‘accurate’ from their particular perspectives, but there was little to no consensus on what the answer to the question should be. For what its worth, the final decision went in favour of the assessment of the academic specialist, over and above those of the main consultants.
I think all this would make more sense if you consider the history of scientific publishing. No doubt you are right that improvements can be made to formats. But these days there are several versions of a paper in different formats with different information content and the reason for this is that they serve different purposes. To scientists all this is just obvious because we are used to it and trained, but I can see why it confuses others including IM Doc who I assume is of a certain vintage, such that they were not trained in the presence of modern publishing formats.
So in the 19th and early 20th centuries scientific publishing was conducted mainly by letters, which would contain some musing, the odd figure, either hand drawn or maybe even a photo or two. As science became more complex journals arrived which would publish papers in longer formats with lots of complex multipart figures. All these were bound into hardback tomes which cost money so they were densely type set and had strict limits on word counts and figure numbers. Furthermore the cost of publishing by was borne by the authors and related to the length of the paper. Nowadays the printed versions are being phased out, but the PDF versions of scientific articles now found online is in exactly this format, and is produced expressly in order to be printed and read on paper and for this reason remains as brief as possible, to keep printing costs low. This is why to protocol is absent.
The advent of the internet removed some of the cost barriers on lengths and also enabled different content types. So people started to add extra information that they would never have been allowed to put in a paper before. This could be experimental data that is not strictly necessary to understand the narrative, or data in different formats. Scientists also started to understand that there were opportunities to increase rigour by demanding raw data, or data in formats which made it easier for others to use to either verify the analysis or use for other purposes. This meant that online versions started to deviate more from the printed version. But the articles were still formatted such that the information is arranged as if to be printed so that the article doesn’t have to be produced twice in to different formats, but with the secondary information in a Supplementary Data or protocols section. Older journals that used to be printed but are now online like NEJM have this hybrid format.
There are often multiple versions of a scientific paper now, preprint versions (before peer review), pre-publication versions (accepted but not type set and formatted in the journal style) and fully online journals where the supplementary data is integrated in different ways into the main article. All these serve different purposes (for instance speed) and contain different information.
This is not to defend NEJM if they are not following their own procedures….
> This is not to defend NEJM if they are not following their own procedures….
I don’t care if they are following their own procedures or not. It seems likely to me that they got sloppy with copy editing on capitalization the reference to the Protocol element, but the removal of the Supplementary Material from the PDF deprived two separate groups of medical practitioners of critical information that would directly affect patient care. In other NEJM articles I looked at, that seems to be the regular practice. If this is their considered editorial policy, it’s somewhere between folly and insanity. The procedures are wrong and should be changed.
> This could be experimental data that is not strictly necessary to understand the narrative, or data in different formats. Scientists also started to understand that there were opportunities to increase rigour by demanding raw data, or data in formats which made it easier for others to use to either verify the analysis or use for other purposes. This meant that online versions started to deviate more from the printed version. But the articles were still formatted such that the information is arranged as if to be printed so that the article doesn’t have to be produced twice in to different formats, but with the secondary information in a Supplementary Data or protocols section.
Yes, that is what I explained at some length in the piece. The issue here is that both JAMA and NISO point out that Supplementary Material can be “critical.” That demands that references to elements in Supplementary Material be properly formatted a la JAMA and NISO (as NEJM did not do) and that if Supplemental Material is not present, then the reader be told where to find it (as NEJM did not do). How hard would it be, for example, to generate, in the PDF version, a standing element with boilerplate text said something like “Supplementary Material will be found in the online version of this document [with URL]”, if the editors foolishly feel that would be better than simply including it. Instead, they simply omitted it without explanation.
> To scientists all this is just obvious because we are used to it and trained, but I can see why it confuses others including IM Doc who I assume is of a certain vintage
So you’re saying (a) that doctors with MDs are not “trained,” and (b) that “vintage” doctors — and, more importantly, their patients — should be deprived of critical information by medical publishers? I think you might want to rethink this.
> There are often multiple versions of a scientific paper now, preprint versions (before peer review), pre-publication versions (accepted but not type set and formatted in the journal style) and fully online journals where the supplementary data is integrated in different ways into the main article.
Yes, if you have been reading Links attentively — perhaps you are new here — you will have noticed that I indicate the publication status (e.g., preprint) of an article in a parenthetical before the source when I quote from it. However, I think you are equivocating on the word “version.” Nobody is saying that all versions of a given article should be the same no matter their stage in editorial, diachronic time; that would be absurd. However, when the article emerges from the editorial workflow as a finished entity, all versions of it, no matter the medium, should be synchronously identical or the differences between versions should be documented for the reader. One should not simply omit critical information, which is what NEJM did. That’s a sloppy and bad editorial practice and/or design, and should be changed.
UPDATE > I think all this would make more sense
In what way does the post not “make sense”?
the RAW DATA has NOT been made available! word is they are looking to release it to a select few at some point
This claim about “scientists” versus MDs is rubbish.
First, reader KLG, who IS A SCIENTIST (30 year of basic research, and a full prof) couldn’t find the Supplemental Materials, because as of when he accessed the article (early, when it was on the front page of teh NEJM site). NEJM had a prominent (in header area) link to a doc called “Supplemental Appendix”). Lambert skipped over the versioning part of the mess, but you can read the details here.
In other words, you assume scientists could find it. They had trouble too. You may have found it after they moved the article off the first page and made the Supplemental Materials dropdown prominent. See here for details: https://www.nakedcapitalism.com/2020/12/further-discussion-of-the-red-flags-in-the-pfizer-vaccine-paper-in-the-new-england-journal-of-medicine.html
Second, you insultingly assume IM Doc is a rube. He and eight colleagues regularly read medical journal articles.
Third, NEJM’s ad materials make clear that their target audience is MDs. Not scientists. Yet per you, NEJM is catering to a non-priority readership?
Plus, as a publisher of PDF magazines, there is no reason not to include hotlinks to the supplementary documents in the PDF. In this manner, the PDF could be opened in an e-reader and with one click access those documents online. We do this for our magazine readers, who can print out the PDF version of the magazine or read it on computer, tablet, or smartphone. In the e-readers, every ad is hotlinked to the internet and all internet references are so linked.
> Plus, as a publisher of PDF magazines, there is no reason not to include hotlinks to the supplementary documents in the PDF
Yep
The study states this:
“We assessed the safety and efficacy of two 30-μg doses of BNT162b2, administered intramuscularly 21 days apart, as compared with placebo. Adults 16 years of age or older who were healthy or had stable chronic medical conditions, including but not limited to human immunodeficiency virus (HIV), hepatitis B virus, or hepatitis C virus infection, were eligible for participation in the trial. Key exclusion criteria included a medical history of Covid-19, treatment with immunosuppressive therapy, or diagnosis with an immunocompromising condition”.
Which looks to me like it is clearly stating at least some exclusion criteria: treatment with immunosuppressive therapy or an immunocompromising condition That would make IM Doc’s statement:
“From my reading of this paper, and the accompanying editorial, one would assume there were no exclusion criteria. They certainly are never mentioned’.
Completely wrong.
> Key exclusion criteria included
Come off it. You’re saying that weasel-wording like “included” substitutes for supplying the actual location of the full exclusion criteria.
Somebody’s completely wrong here, fella, and it’s not me.
+1. Noting the existence of important published material from the same study that’s not included in the main text, but failing to provide a cross reference anywhere in the main text – independent of platform – to the not included important material…. big fail. Scavenger hunts are fine for Halloween Parties, not for medical journals or science journals or tech journals, imo.
He wrote “no exclusion criteria” mentioned when “immunosuppressive therapy or an immunocompromising condition” were mentioned.
And below he asking what to tell his patients on “immunosuppressive therapy” when you can gather even from the original article that it would be untested.
Hence the concern.
I take care of patients. Many of whom are on all kinds of meds with all kinds of medical problems.
Many of whom read all about the anaphylactic reactions occurring with this vaccine – and noting that Pfizer, the FDA et al made such a poor show in properly explaining this to the public that they are looking to their family or primary care doctors to help explain this to them. Many of whom are wondering if the same thing that befell the anaphylaxis patients will befall them because of XYZ med or medical condition. It is a truly rational question, and is coming at me all the time because of the extreme curiosity about all things related to COVID vaccine in this emergency.
Again, unlike many other research trials I have ever looked through, this one does a very poor job of explaining whether immunosuppressive therapy is an exclusion criteria because of possible issues that occurred in Phase I trials – or if they just cut it out all together. This is true of many other medical problems on that list. I am sorry that you are unable to see how the former would be much more problematic for safety concerns in a patient’s mind – and the latter not so much. But again, it has fallen to me and thousands of other PCPs like me all over this country and world to discuss this with patients. It is happening all day every day. And Pfizer/FDA have given us very limited tools to help people understand. It is causing significant hesitation in a lot of patients, even health care workers (https://www.beckershospitalreview.com/workforce/staff-at-hospitals-in-dc-texas-turn-down-covid-19-vaccine.html). If the goal was to vaccinate everyone, they have sure started us off with a very troubled gameplan. This is just one example.
I think this point needs to be made. Once eventually found, the exclusion criteria were reported in the protocol in a very incoherent manner.. As to your point above, you can easily see that there were way more exclusion criteria than just the simple things mentioned above in your comment. And because I have been reading these things for decades, I knew that was going to be the case.
Again, once found, here is the mess of the manner they were presented – this is a word for word cut from the original document:
•Other medical or psychiatric condition including recent (within the past year) or active suicidal ideation/behavior or laboratory abnormality that may increase the risk of study participation or, in the investigator’s judgment, make the participant inappropriate for the study.
•Phases 1 and 2 only: Known infection with human immunodeficiency virus (HIV), hepatitis C virus (HCV), or hepatitis B virus (HBV).
•History of severe adverse reaction associated with a vaccine and/or severe allergic reaction (eg, anaphylaxis) to any component of the study intervention(s).
•Receipt of medications intended to prevent COVID 19.
•Previous clinical (based on COVID-19 symptoms/signs alone, if a SARS-CoV-2 NAAT result was not available) or microbiological (based on COVID-19 symptoms/signs and a positive SARS-CoV-2 NAAT result) diagnosis of COVID 19.
•Phase 1 only: Individuals at high risk for severe COVID-19, including those with any of the following risk factors: ◦Hypertension
◦Diabetes mellitus
◦Chronic pulmonary disease
◦Asthma
◦Current vaping or smoking
◦History of chronic smoking within the prior year
◦BMI >30 kg/m2
◦Anticipating the need for immunosuppressive treatment within the next 6 months
•Phase 1 only: Individuals currently working in occupations with high risk of exposure to SARS-CoV-2 (eg, healthcare worker, emergency response personnel).
•Immunocompromised individuals with known or suspected immunodeficiency, as determined by history and/or laboratory/physical examination.
•Phase 1 only: Individuals with a history of autoimmune disease or an active autoimmune disease requiring therapeutic intervention.
•Bleeding diathesis or condition associated with prolonged bleeding that would, in the opinion of the investigator, contraindicate intramuscular injection.
•Women who are pregnant or breastfeeding.
•Previous vaccination with any coronavirus vaccine.
•Individuals who receive treatment with immunosuppressive therapy, including cytotoxic agents or systemic corticosteroids, eg, for cancer or an autoimmune disease, or planned receipt throughout the study.
•Phase 1 only: Regular receipt of inhaled/nebulized corticosteroids.
•Receipt of blood/plasma products or immunoglobulin, from 60 days before study intervention administration or planned receipt throughout the study.
•Participation in other studies involving study intervention within 28 days prior to study entry and/or during study participation.
•Previous participation in other studies involving study intervention containing lipid nanoparticles.
•Phase 1 only: Positive serological test for SARS-CoV-2 IgM and/or IgG antibodies at the screening visit.
•Phase 1 only: Any screening hematology and/or blood chemistry laboratory value that meets the definition of a ≥ Grade 1 abnormality.
•Phase 1 only: Positive test for HIV, hepatitis B surface antigen (HBsAg), hepatitis B core antibodies (HBc Abs), or hepatitis C virus antibodies (HCV Abs) at the screening visit.
•Phase 1 only: SARS-CoV-2 NAAT-positive nasal swab within 24 hours before receipt of study intervention.
•Investigator site staff or Pfizer employees directly involved in the conduct of the study, site staff otherwise supervised by the investigator, and their respective family members.
I have never seen such a mess in my entire life as a physician – and on such a critically important issue. I have never seen haphazard dumping of Phase 1 exclusion together with Phase 3. It is one thing to do this to a boutique oncology drug – it is in another realm to do this to something we are going to be giving to everyone.
Please tell me – what do I tell my patients on immunologic agents like ENBREL about their safety profile for this vaccine? It has already come up twice this week. It is very clear they were excluded from phase 3 – but WHY? – did something adverse happen to these people in PHASE 1 or PHASE 2 – it certainly isn’t clearly mentioned. Did the investigators just cut them out from the start – again – does not seem to be very clear. These types of issues are usually clearly laid out in a much more functional table than this mess. What about pregnant patients? Did something concerning happen in the PHASE 1 trials – I may have overlooked in this mess – but I cannot see they were excluded from Phase 1 trials? Usually, they would be, and a simple note about off the bat exclusion would have been perfect. But really not obvious in this mess. Again, not totally clear exactly why they were excluded.
I have mentioned this before, but again, there is a part of me really wanting to know why exclusion criteria were even allowed in a study with an emergency drug meant for every human being on earth. It just does not make a lot of sense.
So you don’t like how they wrote it?
Most of those exclusions seem actually to be for Phase 1 which will have nothing whatsoever to do with what you tell your patients. Phase 1 exclusions would always be very extensive because it is mainly trying to get a rough idea of the safety of the vaccine on a very small number of people before trying it on larger groups. So the objective is to not throw a bunch of complications into a determination of basic safety on a small group. It is also to determine dosage ranges that will NOT cause severe side effects. As a doctor you should know this. I would be surprised if Phase 1 had much more than a 100 volunteers.
I know more about the Moderna vaccine and it included 250 micrograms doses in Phase 1 and had severe reactions from several people. It also determined that the 100 microgram dose produced sufficient antibodies without severe reactions so the other phases used 100 microgram doses.
Please answer the question –
What do I tell my patients on Enbrel this week about their safety fears with this vaccine? – And I would add that one of them is a retired chairman of a department in an Ivy League medical school who herself could not figure this out on her own. She had seen her rheumatologist on Zoom the day before me. They could not figure it out together. That is why she brought it up with me. Look at the above presentation and tell me.
And, no, I do not like how they wrote it. That is the entire point. They are not all just Phase 1 exclusions. They are indeed written in a very opaque way. They are all thrown together instead of neatly separated. I have 20 patients a day when we are not having COVID on top of that with 6-10 in the hospital. I am profoundly busy right now. I long for the days when information like this was presented where a busy clinician could advise his patient with confidence without having to do a deep dive in data sets for minutes to hours (assuming it could be found easily) and having to make suppositions on top of that. Again, my patient, a retired dept chair in medicine, and her rheumatologist could not figure this out either.
I think this slide to sloppiness has been going on for some time. I have seen it slowly happening. It is just now becoming apparent to many of us with a degree of urgency because we as clinicians and our patients themselves are looking at this material carefully and much more than usual because of this emergency.
> So you don’t like how they wrote it?
This response is so silly and off-point I’m going to just let it sit, festering in the sunlight of the NC comments section.
Yes, these Dunning Kruger effect case studies are wonderfully blind to how they manage to look foolish and arrogant at the same time.
Looks kinda malicious to me.
> Once eventually found, the exclusion criteria were reported in the protocol in a very incoherent manner..
Oh good. Once you find the protocol, NEJM butchered it even worse. (Note that the Protocol is supplied by Pfizer — that is why the document should be in Supplementary Materials! — but should be reviewed by editors. So NEJM doesn’t get off the hook here.)
Good article. Thank you, Lambert.
It reminds me why, after taking a few courses in technical writing, why I fell in love with LaTex and DocBook and the general priciples of SGML. The templates for good tech writing are part of the packages and the templates help to ensure that you don’t leave out “important stuff”, no matter who your intended audience is.
This speaks of a shabby job by the NEJM for an article that could well be amongst the most read and cited this year. Whether this tells a story of editorial degradation or if it is somehow intentional is anything but clear. I didn’t read the paper in full but it is easy to think that at some point in the introduction, results, methods or discussion sections, there should be some mention to the exclusion criteria in the trial, possibly with a short explanation and a reference to the protocol in the pdf version.
> at some point in the introduction, results, methods or discussion sections, there should be some mention to the exclusion criteria in the trial,
Yes, a proper reference a la JAMA/NISO is the absolute minimum. Better would be a properly functioning Protocol element in a Supplementary Materials section in the PDF. Those together constitute a baseline. It would be best, and surely best for the practitioners who are the putative NEJM readership, to have the critical parts of the protocol excerpted and discussed. Faith-based reading (“Now faith is the substance of things hoped for, the evidence of things not seen,” Heb 11:1) is really not the ideal mode of professional-to-professional technical communication.
Setting aside the issue of editorial ‘slant,’ to this untutored reader, the omission of an obvious and simple link to the Exclusion Criteria is the important item. As IM Doc stated, the Exclusion Criteria are crucial information for a practicing physician. They literally determine a part of the application decision process. As such, wouldn’t the obfuscation of this item constitute criminal negligence? After all, this is basically a technical journal for physicians. The information transmitted is often of the “life or death” kind.
Whether the omissions are the results of negligence or malice, the responsible editors at the NEJM should be “terminated with extreme prejudice.”
Thank you all to Naked Capitalism for giving us information, in a layman readable form, that is of serious importance to “life and limb.”
I have little experience in reading scientific or journal papers, but I often reference them and download material for my archives, which I occasionally have time and reason to refer back to. I don’t even have a printer hooked up now, but will read the PDFs, which should have all the information, in one place. If I wanted to print something out, I see that my reader, Foxit, and I assume other readers, can print out everything, one page, individually selected pages, the entire document.
I have heard busy scientists complain that articles are so broken up with so many auxiliary files that it can be difficult to follow them and find all the material, wasting much time. Printing on paper may get expensive, and so is time, but electrons and disk storage are cheap. My impression is that these sorts of problems are from sloppiness and bad judgement, and that if material is worth publishing then it should be worth doing properly, with all readers in mind.