Writing the Lit Review for Research Methods

Research. Olin Warner (completed by Herbert Ad...

I recently finished a pretty big, for me, literature review that totaled about 17 pages, including the title page and two pages of references. Here are some scattered thoughts and lessons learned, at my customarily hideous length:

  • I saw the wisdom of The Scholarly Cassidy's advice to begin the search haphazardly. I spent much early time floundering but tried various keywords that eventually led me to articles of interest. Have to get used to the feeling of confusion at beginning and make friends with it.
  • As with most of the work at SILS, what I did wasn't really hard so much as it was time consuming. The keys are starting early (a lesson I'm always re-learning) and letting the work marinate. Because I'm deeply into self-justification, I am obliged to tell you that I started late because I was finishing up a different assignment and dealing with my full-time job, of course, so my research was tucked into the margins of my daily schedule (i.e., at night before bedtime) or relegated to weekends.
  • I remembered advice to break the writing into three fairly equal time-sized chunks: a third searching, a third compiling and sifting, and a third writing. I altered that to make the writing take only one day, but this division let me know when to stop active searching and when to start writing. Although I did occasional follow-up searches, the bulk of my active searching had stopped days before I started writing.
  • I adapted Cal Newton's Newport's Excel-based research database. I added a worksheet to track the lists of keywords I searched against. I kept a list of all of my sources in the main tab, with their citation (if it was easy to get), a URL to the abstract or document, the year it was published, its abstract, a theme or category to which the article belonged (such as "Community Attachment" or "Personal Networks") and a link to a PDF of the full-text article I'd downloaded to my hard drive. I pretty quickly compiled about 125 sources (plus some duplicates). I started scanning for quotations, but discerned that precise quoting wasn't called for (though page references to specific ideas were). I didn't need quotes so much as synthesis. That said, I still had way too many quotes -- the old reporter habits of tucking the evidence into the story die hard.
  • I used the spreadsheet to scan the abstracts and judge immediately whether an article had relevance to me. (I kept reminding myself this was a short paper, not written to last 20 years.) Instead of deleting those rows, I colored the citation cell red. If I liked the abstract, I assigned a theme or category (and duplicated the row if the article fit into more than one category). This got me familiar with the breadth of my article grabs. Then I sorted on the Year Published column (earliest at the top), and auto-filtered by theme. I could then see this haphazard list snap into place: all the articles for the themes sorted from earliest to most recent, and the progression of thought visible in the abstracts. I'd already decided I only needed about 3-4 themes for this paper, so this process helped me identify weak themes (only one or two articles) and combine similar themes for later processing.
  • When it was clear that I had too many articles for a category (about 25 for the Sense of Community theme, for example), I reduced the number to 3-5, which forced me to generate selection criteria and think about how they would fit into the story I was telling. I then printed out only these articles and read them more closely since they would form the spine of the lit review.
  • I spent most of the days leading up to my writing in working this spreadsheet, finding new sources until I reached saturation (the same titles or authors cropping up), and in thinking about the story -- or as some may call it, "building an argument." Same thing, really. Set up the foundation with the themes you'll come back to, remind the reader of them as you go into the middle introducing new concepts, and by the end, you twine and braid the concepts, draw analogies, point out disagreement or overlap, and so on. As always, I found that these connections leapt out at me as I was writing or during my editing. They weren't there to start with.
  • I took a day of vacation to do the actual writing, and the day went smoothly, without much stress. (Had there been an emergency that had taken me away from my home office, though, I would have been doomed.) I suppose, though, that I had a secret weapon, which is that I've been writing in one form or another since 1984. Most of the lit review writing advice I researched struck me as assuming you don't have much writing experience. I, however, know my writing process pretty well. I figured that if I soaked myself in the literature, and could come up with a logical storyline, then the writing would take care of itself. And I'm relieved to say that is, indeed, what happened (to my satisfaction, anyway).
  • One section I left out of the first draft was the conclusion. I felt it was better to wait and do that after I had let the paper cool down and I had put in my edits. Having spent the day intensely with my chosen material, I was able to write a more coherent conclusion that reflected connections that developed during the first-draft writing.
  • Still, I was up till, oh, the wee small hours. After I finished my draft, I took a two-hour break to do a workout, eat, watch a TV show, and practice my banjo. I edited a hardcopy printout, made notes to myself, and then typed in the edits and my conclusion. Ensuring the paper adhered to the APA style guide (and formatting my citations accordingly) actually was more time-consuming or felt like it.
  • The night of the day I finished the assignment, still tired but unable to sleep, I started reading my assignments for the next week. Taking time to pause and rest was probably as much celebration as I could emotionally afford. The best thing to do, I'm learning, is to have another project or task to pour that nervous energy into. (And this has implications for the night of graduation day, whenever that will arrive.)
  • I realized afterward my brain can make the connections between ideas all on its own without me having to force them, and that's rather a relief to discover. If I've stated the problem correctly, I'm interested in the question, and I'm not in a hurry, then all goes well. I don't have to be an expert, but I can be a sense-maker.
  • Always interesting to reflect that any piece of writing is the tip of an iceberg hiding the hours and pages of thinking and drafts. Would be interesting to study the ratio of material/effort expended for a paper to the final page count, so you could calculate that a page of manuscript will require 12-24 hours of effort, or something like that. I imagine someone's already done that.
Reblog this post [with Zemanta]

Two projects, two fuzzy ideas, two lit review processes

The 696 independent study is starting out as a literature review of risk in institutional repositories -- where it's perceived to lie, and, what's interesting to me, who makes the actual decisions? The OAIS model defines the functions of an archival process but leaves the specifics of implementation to each institution. So, for various managerial functions within an archive (archival storage and data management, for example), those functions could be carried out by one person or teams of people. It depends on resources and staffing. Carolyn has advised me to contextualize the risks within the OAIS model and within institutional repositories, which provides me with a good basis from which to select my sources and also (we hope) prevent me from flying off in all sorts of different directions (such as defining risk, decision making algorithms, how risk is managed in other contexts, and so on). I've collected a mass of documents and web pages that I now need to sort through, skim/read, and decide what the current picture of the situation is like. She reminded me at today's meeting that the goal is not to solve a problem, just to describe the situation.

She liked my abstract and suggested headings/subheadings, so she's assured that I seem to be moving in the right direction. The precise path I'm still working out, but the direction is fine.

For the 780 Research Methods course, we received very good comments and annotations on our Problem Statements, which were intended to help us think through the research problem we're proposing, start looking for some literature to support it, and define the research questions that will drive our projects. The key here is to ask the right questions and make sure they're right-sized, so to speak.

As I was writing my statement, I could feel the question and underlying assumptions change under my fingers. That's OK, that's part of the process. (And the value of deadlines, it must be said, is that they focus one's mind powerfully. Damn them.) The professor started out liking my topic and then seemed to veer toward, well, maybe what you're really asking is this. And I have to agree with her.

So, I need to work on that section some more.

Upcoming is a literature review that has to include at least 8 pieces, at least 4 of which need to be empirical studies. Based on my 696 and problem statement experiences, I can tell that I'll need to review/download about 25-40 items to find references that inform what I want to do. The trick here is being sure in my mind what it is I want to do.

I spent this afternoon at the library and found 4 books on community networks that I hope will have either good info I can use or leads on studies. Generally, once you've found a good article or lit review on the topic, that's the mother lode that can lead to more and better items.

Must keep in mind, though, that the finished piece is due in about 10 days, which isn't much time, given the day job, doing our taxes, getting my car worked on, and other obligations. I've reluctantly realized that I'll never get a whole day to just sit and do this work, so I will have to find a way to fit what I have to do into the interstices of my day. Next weekend, though, will need to be devoted to the writing up of whatever I've found so I can discover whether what I've got will support my research ideas.

Reblog this post [with Zemanta]

Research Journal for my 780 class

Cover of "7 Up"
Cover of 7 Up

Since our 780 Research Methods class doesn’t have a Blackboard site for the class, I’ll post my various links and thoughts to the blog, tagged with “780.”


I wonder if Michael Apted’s wonderful Up series of documentary interviews would be an example of a kinda sorta longitudinal study or panel study? When a new film comes out every 7 years with updates on these people, it’s always fascinating to see where life has -- or hasn’t -- taken them. Instead of gathering statistics about a large group of people, there's something very satisfying about getting to know a small group of people very well.


We’ve been talking about experiments, planning a study, theories, types of studies, etc. One of our last readings was about where one gets ideas for theories. This reminded me of Seth Roberts, a Berkeley researcher in psychology, who frequently touts self-experimentation as a way to generate research ideas. This is one of his more famous papers. He maintains an active and entertaining blog.

What I admire about Seth Roberts is his abundant idea-generation and his zeal for measurement and record-keeping. His goal is to experiment on himself first, then if his data indicates that there are possibly interesting results, then he proceeds with more methodical testing and inquiry, possibly leading to more formalized studies (or not).

When I’ve been thinking about possible studies I might like to try, I remember this quote from one of his blog posts:

SR: Tell me something you've learned about research design.

BW: When I was a graduate student [at the Stanford Business School], I would jog on the school track. One day on the track I met a professor who had recently gotten tenure. He had only published three articles (maybe he had 700 in the pipeline), so his getting tenure surprised me. I asked him: What's the secret? What was so great about those three papers? His answer was two words: "Cool data." Ever since then I've tried to collect cool data. Not attitude surveys, which are really common in my area. Cool data is not always the easiest data to collect but it is data that gets buzz, that people talk about.

Thinking about what “cool data” might mean in a digital curation or archival or info-science context can be tough. I think the social networks are certainly perceived as cool and you can do cool stuff with them, certainly, but I’m not that curious about them. I feel like, were I to study one of them, I’d just be chasing a parade that’s got a five-mile headstart. Better to find my own parade. :)

Curiosity is probably what drives me. Certainly, one of the itches that a researcher must scratch is his or her own personal obsession with some nagging question or detail that no one has really addressed or answered to their satisfaction. (The same way most writers have to write their own poems, stories, and plays, because no one else is publishing what they want to read.)

Check out his numerous posts tagged scientific method (though he’s more usually critical of scientists’ behavior than the method itself) and self-experimentation for more.


Another great Seth Roberts post that got my attention was this one on appreciative thinking, especially as it relates to reading journal articles. I see what he describes in the classes I attend, where we read a paper that’s 1 year, 5 years, or 10 years old, and it’s rather thoroughly shredded during the ensuing discussion for any number of reasons (and I've been guilty of trashing articles, myself).

Instead of this negative critical thinking, I like his suggested questions to ask instead, especially the simplicity of his fifth question: “What’s interesting or enjoyable about it?” Even if I find the writing of an article stilted or atrocious, I think it should be possible to at least admire a piece’s energy, its intent, its point of view, its ability to stir thoughts in me, etc. Saying something constructive is not about becoming a positive-thinking ninny; it's about seeing more sides of the issue than only one.

Even for a piece (Mabry's "Reference Interview as Partnership") that didn’t really touch me, I appreciated that this was the author's distillation of a career’s worth of lessons that she wanted to impart. In my summary of the piece, I said I could see it being used to start a conversation about one’s own personal manifesto for serving at a reference desk. We’re not often asked to reflect on our larger purpose or philosophy when it comes to our jobs, or even our career, so I saw the Mabry piece as a terrific starting point for such a conversation.


Speaking of writing up experiments so they’re repeatable -- how often does repeating an earlier experiment really happen?

Reblog this post [with Zemanta]