Learning to be a researcher

Last January, when I set out to write a thesis, I was initially interested in media effects, framing, and disparities (from a social justice perspective). I had studied media advocacy in mass media seminar, but I knew that media advocacy was more of an intentional tactic public health folks use to change community norms and less an organic decision that journalists make. I wanted to study the small, everyday decisions journalists have to consider when writing stories and what effects those decisions have on audiences. This desire took me to my main theory of study: framing. I dug through the available framing literature and found it was filled with interesting research questions and studies.

Many of the framing studies I found in the literature included content analyses. I steered away from this design for my own thesis because I knew I wanted to take a more active role in the process. I also viewed the coding part of content analyses as tedious and liked the control experiments give to researchers. I was drawn to the experience of learning how to manipulate something, then controlling for variation, collecting data from human subjects, and analyzing it. I am, however, very grateful to the researchers who took the time to look at how health articles were framed and to those researchers who took the time to interview health journalists about their internal processes. I see now that the content analyses and qualitative interviews are important for developing research questions and learning about next steps. The best researcher would probably employ a combination of these approaches and use both qualitative and quantitative research methods to answer their research questions.

It took months for me to understand how framing theory should guide my literature review and my research questions. I also had a hard time conceptualizing the constructs and how the post-test questions combine into a (hopefully) reliable measure. In fact, l didn’t understand how to create a measure until after I ran my experiment. I thought blame could ask about individual and societal blame, and it was only after I was creating my summary scores that I realized individual and societal blame were separate measures. I worked to remedy this by making different combinations of questions into indices. Unfortunately, I had to use individual questions for two of my measures: societal blame and societal responsibility.

The first experiment I proposed had three independent variables and was far too complex for a first-time experimenter. Brian Houston, who graciously agreed to serve as my external committee member and quantitative methodologist, helped me simplify the design. He challenged me to identify just two independent variables and assured me the process would be complex enough. A few weeks of mental grappling ensued. It was hard to commit. I enjoyed thinking about all the options but settling on a health topic, a frame and a disparity felt so narrowing. The grand ideas were dissolving, and I was left with just a few specifics. It felt like the findings would not be useful enough. But making these decisions also meant getting closer to gathering data. So, I pushed forward. After speaking with my mentors (Katherine Reed, Jeanne Abbott and Kim Walsh-Childers), I chose diabetes as my health topic. It is a pervasive condition with serious physical and economic costs for individuals and society. I also got closer to designing the experiment, with much assistance from Brian. We settled on a 2x2 factorial design between-subjects experiment. My manipulations would be frame: individual and societal and disparity: presence and absence. I chose economics for my disparity. Race disparities are common in the literature, but they are also polarizing. I decided to circumvent race and talk instead about something directly affected by racial disparities: money. The inclusion of a disparity would give me space to talk about the economic cost and economic struggles of a central character.

Once the design was settled, I met with my committee to present the proposal. The meeting itself was helpful. I was nervous, and too stressed from working too many nights and not getting time away between semesters. I was unable to take a step back and effectively edit, so the suggestions from my committee were very welcome. Brian wasn’t able to come because I didn’t send a reminder in time, and I hadn't thought about using an Outlook calendar invitation. I was disappointed but learned a valuable lesson about how to reserve a busy professor's time.

This summer involved a lot of logistics and email exchanges. I waded into Qualtrics and learned how to set up the articles and then randomize them. At times, I was frustrated that I didn’t have more quantitative mentors who could help me. I reached out to a prominent health communication scholar, Glen Cameron, who is based at Missouri, but alas, he didn't have time to help. I pushed forward, knowing that mistakes are part of the process. Guides are crucial, but learning how to be a researcher also means pushing through unfamiliar territory. So I pushed. I worked on the articles. I fought with the technology. I checked off boxes of the many forms required of a master's thesis. By mid-June, with my move to DC looming, I put the research on hold and focused on packing up my life in Columbia. I was nostalgic and proud of all I'd learned. Journalism school was a fantastic choice, and I felt lucky to have called the J-school my home for the past two years.

Once the move to DC was semi-complete, I turned on the heat again. I put in an IRB application and started researching Amazon Mechanical Turk. I read a lot of discussion threads and how-tos for MTurk and got a handle on that world. I also heard back from Betty Jo at Mizzou's IRB office. I got the green light to run my experiment and returned to Qualtrics to edit my articles and the post-test questionnaire. I called the IT department to ask about Qualtrics access and was relieved to learn I had a 30-day grace period. I ran my post-test questionnaire by one of the statisticians at the National Cancer Institute. He graciously assisted. I could tell he was nervous about getting wrapped into a master’s level project, so I again turned inward. I could only get so much advice before I had to just dive in.

I reviewed everything four times and hit run on the MTurk experiment around 5 p.m. on a Thursday in August. The data came flooding in nearly instantly. By Friday morning, I had collected 200 responses. I reviewed them, paying close attention to time spent. The average time on the experiment was about 11 minutes. I rejected people who seemed to complete it too quickly. This turned out to only be one person. I put the experiment back onto the market and collected that last response by Friday afternoon. I approved everyone. Then I waited.

And waited. A week passed. I had a large spreadsheet worth of data from 200 people. I didn’t really know what to do next. I knew I needed to clean the data and then analyze it. But I hadn’t used SPSS since the winter of 2014. About a week later, a coworker asked me how the data analysis was going. It wasn’t. We set an appointment to review data cleaning and analysis for the next day, and I started to dig in.

I found the data cleaning process meditative. It involves creating new columns when needed, running demographic analyses on participants and reviewing the manipulation checks to make sure they worked. After that process, I was able to fix my methodology. I reorganized that section and filled in the holes where the descriptive statistics needed to be. 

My next phase was digging into the data itself. I read tutorials on analysis of variance (ANOVA) and reviewed 2x2 factorial design. I also asked another coworker how she would look for main effects and interactions. I ran tests and stared at the charts. Then I slowly began working through the findings to find meaning in the numbers. 

The results showed that my manipulations (frame and disparity) had a significant impact on readers and that one manipulation was not dependent upon the other. Framing and the inclusion of a disparity had an impact on how readers assigned blame and responsibility to the individual and the government. And the influence was in the direction I had hypothesized. Participants who read an individually oriented article were more likely to assign blame to the individual, while the societally focused article found the opposite. The same was true for responsibility and government solutions: people who read individually oriented articles were more likely to put responsibility on the individual than society or the government and vice versa.

I spent several evenings and weekends staring at data and trying to figure out what it meant and how it could apply to journalists. At first, I struggled to understand the difference between a main effect and an interaction and how to represent my findings in a paper. I am very grateful to Jennifer Taber and Chan Thai at the National Cancer Institute for taking time out of their busy research schedules to mentor me through my roadblocks. I have a much stronger grasp of social science research now that I have run an experiment of my own.

Even though I have a limited scope of real-world journalism, I still feel confident with my discussion and my conclusion. The everyday health journalist most likely uses a combination of the frames and source types I used in my thesis. My manipulations were very pronounced so that I could see the effects clearly. An actual article would probably resemble a mixture of the four articles I produced for this experiment. A newsroom would certainly create something much less "experiment acceptable." But that is the real world. So, I am only able to extrapolate so far. I think it is worth noting though that framing and the inclusion of an economic disparity did have a significant impact on where readers of articles on diabetes placed blame, responsibility and government responsibility. It is a reality I will remember when I have the opportunity to start reporting again (fingers crossed!).