Notes From a Fellow: Launching Projects, A Two-Way Street

In my last column, I wrote about the service-oriented mindset that can be valuable in approaching policy and program work (“OES Collaborations Echo Federal Priorities,” May/June Observer). Academic researchers tend to start a project by thinking about what hypothesis they want to test or what process they want to understand. Being service-oriented means that if you’re working with a government office concerned with low benefits takeup, medication overprescription, inefficient energy use, or confusion about a form, you’d start instead by thinking about which theories or interventions might be useful to address this challenge.  

But being service-oriented is not the same thing as saying yes to every project. There are some government agency challenges that behavioral scientists are not well suited to address. The Office of Evaluation Sciences (OES), where I am a fellow, has a list of considerations called “project building blocks,” and we generally take on a new project only when each of these building blocks is present.  

The first building block for a new collaboration at OES is that the outcome of interest has to be tied to the actions that individuals take. These actions look different in different situations; they can range from a farmer contacting a county loan office, to an individual enrolling in the Federal Health Insurance Marketplace, to a woman attending a maternal health appointment. Outcomes that can’t be achieved by changing individuals’ actions aren’t a good fit for the types of research insights OES can apply.  

OES also has a strong preference for randomly assigning an intervention (e.g., different versions of a letter), so the opportunity to apply randomization is a second project building block.  

Third, evaluation projects need to have adequate samples. Large samples allow for more precise estimates of effects, meaning that even null findings can be used to inform program design (and aren’t so noisy that they can’t rule out  the possibility of large effects).  

One thing to consider here is that the key number for the sample is typically the unit where randomization occurs. Think of a state labor department with 20 offices serving a total of 100,000 job seekers. If the goal is to randomly assign offices to use a new procedure or proceed with business as usual, then the sample size is effectively 20 instead of 100,000, likely too small to be informative. Careful power analyses can inform decisions about whether the available population is large enough to make a project meaningful.  

A fourth building block is something I’ve written about before: the availability of administrative data that capture the outcome of interest.  

As an aside, OES has recently taken on more quasi-experimental (nonrandomized) evaluations and even some small-sample qualitative research that doesn’t rely on administrative data. These sorts of projects are undertaken to meet specific objectives, like the desire to better understand the challenges that local governments faced in quickly designing and launching small business grants and loans during COVID-19.  

I’ll mention one final project building block: OES takes on projects only if there is a dedicated agency collaborator willing to work alongside OES and share results across government agencies. We think of this person as a project champion.  

I have to admit that having an agency collaborator is a “building block” that I would have largely taken for granted a year ago. However, my OES colleagues have convinced me that in some ways, this is the single most important ingredient for making sure a project will succeed. This person often plays a key role in getting results acted on—ensuring that the evidence is utilized instead of ignored. A priority goal for the current team is tracking and improving the utilization of evidence that OES produces, which is another reason a dedicated collaborator is important.  

Of course, many researchers who work in universities and labs also want to know that their work is being utilized. It’s probably easier to track “impact” just in terms of citation counts for articles, but many universities also explicitly value research that has an impact on nonacademic audiences. My own employer, the London School of Economics and Political Science, has a “knowledge exchange and impact” unit that helps faculty and staff build the relationships that enable this kind of impact—and identify metrics to track it.  

I’m hopeful that this description of OES’s project building blocks makes it a little easier to understand what does and doesn’t make for a promising project in the government setting. I imagine that other government-based teams have their own sets of considerations, but I’d be surprised if they didn’t overlap quite a bit with what’s important to OES.

Feedback on this article? Email [email protected] or scroll down to comment.