Below the Radar Transcript
Communicating Scientific Uncertainty About COVID-19 — with Alice Fleerackers
Speakers: Melissa Roach, Alice Fleerackers
[theme music]
Melissa Roach 0:07
Hi there, I'm Melissa Roach and you're listening to Below the Radar. Below the Radar is a knowledge democracy podcast recorded on the territories of the Musqueam, Squamish and Tsleil-Waututh peoples. For this episode, I speak to Alice Fleerackers, a freelance writer and researcher specializing in online science communication. Currently, she's a researcher at ScholCommLab, Research Officer at Art the Science, and a Science in Society Editor at Science Borealis. She's also a PhD student at SFU, where she's exploring how uncertain health science is communicated online. I hope you enjoy our conversation with Alice.
[theme music fades]
Melissa Roach 0:46
Thank you so much for agreeing to speak to me. I've been excited about this for a while to talk about the study. But I also wanted to start off Alice, by saying good when you told me about your particular mashup of disciplines and studies that you're doing for your doctoral studies, I had no idea you could do that. That's incredible. You want to speak a little bit about what it means to be doing an interdisciplinary PhD?
Alice Fleerackers 1:17
Yeah, thanks. I mean, for the record, I had no idea you could do this either, until I ended up doing it. And the way I've been describing it to people who are not in academia is like a “choose your own adventure” PhD. So you really get to, yeah, pick and choose the areas of research that are interesting to you, and that are the most relevant to the kind of research that you want to be doing or the question that you want to be answering. And so the research I'm doing, I guess, spans scholarly communication and scholarly publishing. I'm based at the scholarly communications lab at SFU, which does a lot of things about how academic research is shared and circulated, usually among academics. But in addition to that, I kind of have this science communication focus, like how is research then being communicated and shared beyond the scholarly sphere and into the public sphere? And so you know, journalism is a very obvious way that that happens, but also social media, even events like science cafes, or science art exhibitions. There's a pretty wild collection of things that could be considered science communication. And then in addition to that, I really have a focus on health as a science topic or area that I think is, just has very clear and obvious benefits for the public to know about. And so the kind of rationale for doing research looking at how that evidence is being communicated is just so much stronger for a topic like health. Where, you know, having that information can really improve people's lives. So that's the hodgepodge as it is right now. But as I'm discovering every year, my focus evolves a little more based on what's going on in the world and in my head. So stay tuned. The adventure could change. [laughs]
Melissa Roach 3:11
Yeah, I'm a huge fan of the ScholCommLab. And I'm excited to talk about, I mean, speaking of health, the reason that we decided to speak today, which is the study that you recently were the lead author on about the uptake of preprint research in media communicating COVID 19 Health Science, is that the right way to put it? [both laugh]
Alice Fleerackers 3:35
Pretty good.
Melissa Roach 3:37
I also wanted to just mention your co authors, too, because I think co authors don't get enough airtime for the work they put into things.
Alice Fleerackers 3:45
Yeah, definitely. I'm so glad you asked because a lot of attention has been directed at me in the coverage we've got so far. And it was such a collaborative effort and such an interdisciplinary effort. So on our team, we have Michelle Riedlinger, who is a science communication scholar in Brisbane, Australia. And we have Laura Moorhead, who is a journalism scholar in San Francisco, Rukhsana Ahmed, who focuses on health communications specifically, and is based in New York. And then we have Juan Pablo Alperin, who's also at SFU, in the ScholCommLab, and he's really bringing that expertise about the scholarly side of things and sort of the nature of preprints as a scholarly output. And yeah, I think that the study would look so different if we didn't have all these very different realms of expertise and interest coming together and shaping the way that we framed the questions and the way that we interpreted what we found.
Melissa Roach 4:43
Yeah, it's such a diverse team and then shout out to Juan, I'm a fan of his too. [laughs] I was lucky to be taught by him once upon a time. But yeah, diving into the study and just setting the stage for or why you, like, took up these questions. I've been thinking about, you know, this slow like rusty, well sometimes rusty machine of academic publishing, and how long the public would have had to wait for any information about Coronavirus, COVID-19, without turning to the un-peer-reviewed research. I mean, there's the flip side of that, too. So I'm wondering, in your mind, how do you conceive of the risks and the benefits of turning to preprint research for, like health media, or news media covering health?
Alice Fleerackers 5:38
Yeah, I mean, I think that COVID-19 is a really special context. And that's an important thing to keep in mind when talking about this study is, you know, we were in a public health crisis, we are in a public health crisis. But the timeframe when the study took place, which was the first four months after COVID-19 was declared a pandemic. That was so early on in that crisis. And as you mentioned, you know, the flow and rusty machine of, academic publishing sometimes has a hard time getting research out there fast enough that it is still useful in a situation that's evolving as quickly as COVID-19 has been. And I do want to acknowledge that a lot of scientific journals and peer reviewed outlets did put in a lot of effort to be as fast as possible because of this special context. So we saw, you know, a huge speeding up of science and science publishing across the board with peer review timelines being cut significantly, to try and get research out as fast as possible when it related to COVID-19. But still, despite that, you know, any kind of peer review does take time, and posting a preprint usually takes a day or so. And so I think that scientists who were doing COVID research, were really leveraging preprints during those first four months, and the media kind of had no choice but to cover at least some of those, because the public needed information. You know, that's the kind of the best available evidence at the time for many COVID specific topics. So yeah, related to your question of the benefits, that, for me is the really clear and obvious one, you know, getting research up there fast can be so important, especially when it's in a context like COVID-19, where new evidence about you know, prevention strategies could save lives, you know. Even if it's just something as simple as how long to wash your hands for, that really has the potential to change the nature of how the virus evolves, and how many people end up suffering from it.
Alice Fleerackers 7:42
But of course, as you hinted at, there are risks, right. And I think that is why historically journalists have been reluctant to report on preliminary research and specifically on preprints, is it could turn out that those research results don't hold up in the long term. That another study comes along and ends up finding that the data was flawed or not being able to replicate the findings, that the preprint eventually does go through peer review and it turns out, there were major flaws with the way the data was collected or processed. And in fact it obviously, really hurt people. If the findings were suggesting a dangerous or detrimental activity, like, you know, Trump's promotion of hydroxychloroquine. I can't pronounce this drug hydroxychloroquine? I mean, we also saw, you know, there were a couple preprints, promoting the use of tobacco, because there was potentially a link between tobacco use and prevention of COVID-19. And those were widely covered by the media. And there was even a study in China, I believe that found that people reported smoking more after reading this research. So that's a very obvious example of how a preprint could encourage, you know, negative behaviors. And those studies ended up being highly flawed. So the people who increase their smoking were just increasing their health risk, with actually this false belief that it was going to protect them from the virus when in fact, that evidence was completely flawed. So that's the obvious risk, is misinformation and potentially encouraging people to make bad decisions. And then I think another risk that's also important to consider is this larger question about public trust. I know a lot of people, for example, were frustrated by the back and forth sort of reporting about masks in Canada. You know, when you tell people one thing, and then a month or so later, tell them something else. That means that people can start to lose trust in you as a communicator, in public health officials, in journalism or maybe even science in general.
Melissa Roach 9:50
Yeah, that was the thing that was kind of lurking beneath the surface for me is like the erosion of that trust. Maybe not necessarily would be directed towards the news media that reported it this way, then that way, but in the science and the health researchers themselves. Yeah, it's a huge responsibility on the part of communicators. Right?
Alice Fleerackers 10:12
It is. But I think it's also important to acknowledge that this happens with peer reviewed research too, right? Like the whole, one day coffee is good for you, one day coffee is bad for you. You're so used to seeing these kinds of stories. And I do think that they are a reason that people lose trust or become frustrated with reporting and with health research in general. Like, it's very confusing as a reader to see science covered in that way. And so it doesn't have to be a preprint for that negative impact to follow. It just has to be inconsistent reporting or reporting, that doesn't put findings into context.
Melissa Roach 10:50
Yeah, that's super important to keep in mind. [laughs] So I mean, there's so many interesting things that came out of the data, as I was reading through it, the headline is probably that, what was it? Over 40% of the articles you sampled, neglected or omitted the framing that communicates that the science was not peer reviewed, or unverified. Were you surprised by the numbers that came out of that?
Alice Fleerackers 11:24
You know, it's interesting, because the fact that a majority of studies, you know, I mean, it was just over 50% that did actually include some indication that they were quoting unreviewed research. To me, that was already surprising, because there's a lot of literature showing that kind of tendency to gloss over uncertainties in media coverage, especially about health. So there have been a few studies about how the media portray other forms of kind of preliminary or unreviewed work like conference proceedings or, you know, sort of editorials and other kinds of scholarly outputs that aren't as, quote, unquote, rigorous as a peer reviewed article. And those overall have defined something closer to 20% of news stories portraying the uncertain or unreviewed nature of the research, or preliminary nature. And so in some ways, this is actually what we found was pretty positive 57% of stories did include at least some indication, but at the same time, I was surprised, especially by the fact that stories from outlets that I consider very high quality and rigorous, like New York Times, were quite inconsistent in the degree that they did this. I think the New York Times was one of the outlets where less than half of their stories included a clear indication that this was an unreviewed or unverified study. So that was a big surprise to me.
Melissa Roach 12:46
Yeah, it's really interesting. And speaking of those framing devices, I was wondering, were there any that you found to be more effective at communicating the uncertainty than others? I'm not sure if that was within the scope for you, if that came up at all, but I was wondering about that.
Alice Fleerackers 13:04
Yeah, I mean, we looked at a few different ways of conveying this. And one was a statement that the research had been peer reviewed. So saying something like an unreviewed study or some research that has not yet been peer reviewed. And another one was saying that these findings have not yet been verified by outside experts or by future studies. Another one was saying that the research was sort of preliminary or early in some way. And then the last one was really clearly marking this as a preprint study, like using the word preprint in some way. We didn't look at how public audiences respond to those terms. So we don't know that one is more effective in terms of how people might understand that research, or the sort of their trust perceptions or any of those responses as audience members. But we can kind of say that, I think before this very few people knew what a preprint was. And providing that label without including a definition of what a preprint is, is not that useful. Similarly, one of the most common sort of devices or strategies that you noticed journalists using was the this research has not yet been reviewed, or gone through peer review, which is great if your audience knows what peer review is, but depending on the outlet, they may not always be familiar with what a peer review process actually is. So I think that my personal preference is maybe something more along the lines of "this has not yet been verified by future studies", or a comment on the quality of the data, which we didn't really look at, but preliminary might be an indication of that if it's sort of an early study.
Melissa Roach 13:06
Yeah. For me, this solidified the power of altmetrics.
Alice Fleerackers 14:55
I mean, it was fascinating to trace all the different kinds of places that were using preprints And that's really what Altmetric, the company that we use to get this data is great at, you know, they, they look at and track all the different places that scholarly outputs are being covered. Whether that's blogs, or social media or sort of established news outlets. And it was very diverse. I mean, we only looked at 15 In this paper. You know, because we wanted to get our research out in time for it to still be useful. But even within those 15, there's a huge diversity of outlets, like super medical niche publications, really kind of high end, you know, traditional legacy publications, like New York Times and The Guardian. But then all these, you know, web first aggregators like Yahoo News, and Daily Hunt, or Medium, you know, there's, we saw a lot of Medium outlets that were covering COVID research. So that to me, was also kind of an interesting and maybe surprising takeaway at this is, you know, preprints were once something that almost no journalist would cover and suddenly became this thing that journalists from all walks of life were covering.
Melissa Roach 16:07
Yeah, I had a few other questions. Maybe it would get us into the weeds and we'd be here for too long. I wanted to ask you about a kind of paradigm shift on, you know, appearing credible or authoritative around hedging and communicating uncertainty. And I think it came up that some outlets were using that, like uncertainty to kind of seem more credible or be perceived as more credible. Am I right?
Alice Fleerackers 16:39
Oh, I think you're referring to this use of the like “refers to the study as research.” In many of those news stories or media stories that were describing preprints without mentioning their uncertainty, they often would refer to the preprint of a study or research or, you know, scientific evidence has been found. And for me, that's really interesting, because there's no communication of the uncertain or unverified nature of the research. But there is this sort of, you know, framing it as a research study. And as we know, scientists are among the most trusted sources of information in society. So, you know, we have lots of survey data showing that and I think, in many countries, trust in science actually went up during Coronavirus. So putting something into your media story, into your news reporting that's supposedly backed up by science or by research or by a study. It does kind of grant whatever that claim is some credibility that it wouldn't have if it was maybe portrayed as a preprint. Or if it was just sort of suggested as a statement that the journalists wanted to make, or that, you know, one of the people interviewed wanted to make. So I think that's an interesting consideration, like, what is the effect of framing evidence as a science or as scientific research? Versus as a preprint or as not even mentioning the source of the evidence?
Melissa Roach 18:11
Yeah, that tendency towards, you know, declarative, maybe sensational headlines, going back to like, just wondering if the financial model of journalism right now doesn't really encourage the nuance of embracing the uncertainty?
Alice Fleerackers 18:33
Yeah, I mean, it's really hard online. You know, I don't read every article I see online, I try to when it's something I think is important. But there's really interesting research from digital journalism studies that has found that there's a huge range of kind of online news habits, and some of them include reading, but a lot of them include things like news skimming, or new snacking is a lovely term I've heard, just pulling little bits and pieces from all over. And I think that journalists today have this really important challenge, which is providing the information in a way that is accurate and isn't misleading. But then also keeping people engaged and meeting people where they are in terms of the habits that they have, and the ways that they engage with news, which, unfortunately, is not the same way that we were engaging with these stories, like we were reading them with a fine tooth comb. That's very unusual on the web. And so, yeah, I think journalism is really in a challenging place right now in terms of balancing those two norms and providing thorough and accurate information. But then also not, you know, losing audiences and keeping people engaged.
Melissa Roach 19:50
We touched on it a little bit before but I also wanted to ask you more about, you know, the political consequences. You're talking about, you know, Trump's affinity for this drug right now. But one thing that's been on my mind is wondering if communicating the uncertainty or not communicating the uncertainty and having it be contradicted, you know, wear a mask inside, don't wear a mask inside the grocery store, the potential for kind of emboldening those more harmful views like anti maskers, or anti vaxxers, or, you know, climate change deniers. [laughs] Yeah, I'm just wondering if you had any thoughts about it, especially in light of like, since publication, the vaccine wasn't available when you published this research, right. So that's added a whole other dimension to reporting on COVID-19.
Alice Fleerackers 20:46
Yeah, definitely. And I think that a lot of people have expressed fears that preprint servers could really enable people who have, you know, a mission to promote disinformation, which is different from misinformation, it's different from, you know, misinterpreting or misrepresenting something accidentally. It's like an active desire. Some people call it even mal-information. To put something misleading out there for your own purposes. And that can happen on a preprint server. But luckily, many preprint servers have started putting in some sort of filtering system before they decide to post or not post a preprint. You know, there's no peer review process. But there is some check if is this actually something that appears to be a scientific study? You know, and is this promoting propaganda? Like, is it obviously promoting a nefarious purpose? And if the answer is yes to the nefarious purpose, it's not going to get posted on the preprint server. So I think that's an important thing to realize, at least for many of the sort of most popular preprint servers, I know that that is something that is going on.
Melissa Roach 21:57
Yeah, those servers are a mystery to me. So that's really good to know.
Alice Fleerackers 22:01
Yeah, and I think that there's also good work going on to kind of improve the nature of preprint service as well. So ASAPbio, which is an organization that's dedicated to kind of using and improving the use of preprints in the life sciences, has a project that's specifically encouraging people to give feedback on how they could improve the use of preprints and preprint servers. And I think that's very much tied to what's been going on with COVID-19. So there are ongoing efforts to improve the way that preprints are being used. And, you know, during the pandemic, preprint servers even started posting little warning signs on top of their preprints that said, you know, this is not peer reviewed evidence, it should not be reported on as an established fact, should not be treated like established evidence of medical practitioners, just to really to prevent kind of some of these risks that we talked about, or these negative consequences from happening. But going back to this, you know, potential for anti-maskers or anti-vaxxers are, you know, people to use preprints in a nefarious way, like, that is a real danger that people could, you know, seek out preprints and cherry pick data, misconstrue or misinterpret study findings, you know, pull results out and put them in a new context that serves their personal needs or their agendas. But again, this is something that could happen with any research study, which we've seen again and again with vaccines, right. So yeah, something to be aware of, but again, something to be aware of with any science, unfortunately.
Melissa Roach 23:36
Yeah. Building on that, what do you, you know, as far as what to be aware of, what do you think are the learnings for people from each different quarter like for scientists who are releasing their preprints, or for media and for readers and consumers of media? What are you hoping that people will take away?
Alice Fleerackers 23:55
Yeah, I mean, I think a really big takeaway is that scholarly publishing is changing and that journalism is changing. And that has big implications for people in all of those quarters. For scientists, I think a really important takeaway is that anything you put out publicly can become a news story. And that's not just preprints, that's tweets, that's blog posts. It's, you know, anything you do as a public facing person is fair game. It's not fair. And I've heard journalists say this to me, in our interviews, in conversations, like if you put something out there, and it's interesting, and I see it as newsworthy, it's not fair to expect me not to cover it. And you know, there's no reason that a journalist can't. So I think that's a really key thing to kind of remember as a scientist, like, there's major benefits for using preprints. But you should make sure that you stand behind your findings enough that if a journalist does decide to cover them, you're not going to be horrified. And I think, you know, maybe extending that to if a journalist asks you for an interview about your preprint you know, be prepared to say yes and to explain any limitations very clearly. And then for journalists, I think a key takeaway is that not everything produced by scientists is of the same quality of evidence. So whether it's a preprint, or a peer reviewed research study, you know, do your background checks, seek out outside opinions, really try to determine whether this research is even worth covering or not. And then if it is, what kind of limitations you need to make clear. And then finally, for news consumers, this is maybe the hardest one. But I guess for me, I think it's important to realize that science is not a perfect process. And new findings can be amazing, they can help us make better informed decisions, but they're never going to be established facts. Doubt is just an inherent part of any new science. So it's always important to kind of read critically to take new information with a little bit of a grain of salt, especially if it's coming from a source that you're not familiar with, or that you don't totally trust. So, you know, fact check anything that seems really suspicious. Think before you share, that's a really important one, you know, try at least to read it before you decide it's a good idea to retweet. Rely on sources that you trust. And this is a hard one, but consider the body of evidence, not just you know, these one off studies about, you know, coffee, or wine, as you mentioned, and you know, government websites and public health resources can be super useful there for getting a sense of the state of evidence.
Melissa Roach 26:33
Yeah, read beyond the one article, the one study. [laughs]
Alice Fleerackers 26:37
Definitely. And like, if you can learn a little bit about how science works, and how to evaluate quality of evidence is just so helpful. I think it's becoming more and more helpful every day is, you know, we're going to experience more health emergencies, I'm sure, you know, and learning how to evaluate just some basic things like what's an acceptable sample size? You know, what's a conflict of interest? And what does it mean for how I should interpret the results? Like, learning a few of those bases can be really helpful if you're at all worried about the kind of information that you're basing your decisions off of as a reader.
Melissa Roach 27:14
And because you're also a communicator, and you know, a practitioner of science communication, has this affected how you've approached your own work since completing the study?
Alice Fleerackers 27:27
Yeah, definitely. I mean, I find myself wondering a lot about how I would have covered something if I was writing for one of these outlets. And I think in many cases, I might have just followed the same strategy that these journalists were doing, you know, it's really tempting to cut details and spare people's attention spans from sort of what's sometimes seen as the boring backstory behind research. But this research has just really made me aware of the importance of providing context and of being transparent and honest. I think that we saw some outlets publishing these big stories about the nature of science and like how peer review works, and what a preprint is, and, you know, these topics that you would have never seen anybody writing about five years ago, or even maybe one year ago, because it would have been considered too boring. Where like, you know, a background story that maybe most people are not going to want to read about. But right now science is so much a part of all of our lives. And I think there's a real opportunity to dig into a little bit of that context and find ways to turn the context into a story. So for me as a communicator, that's something that I'm going to pay more attention to, you know, what details am I leaving out? And is there a way to weave those details that is both accurate and engaging in a compelling story? But tall order, right? [both laugh]
Melissa Roach 28:51
Just a wee bit. Thank you so much, Alice for joining us on Below the Radar. It's always a pleasure to talk to you but I am really grateful that you took some time to share about this with me.
Alice Fleerackers 29:04
Yeah, thanks so much for featuring our work and for reading our study so critically, like a great news reader. [both laugh]
Melissa Roach 29:13
Thanks Alice.
[theme music]
Melissa Roach 29:20
Below the Radar is a knowledge democracy podcast created by SFU's Vancity Office of Community Engagement. Thanks for listening to our conversation with Alice Fleerackers. You can find a link to her study in the notes of this episode, along with some fabulous resources she's recommended for both researchers and communicators. Thank you for tuning in and we'll see you next time on Below the Radar.
[theme music fades]