The Method to Our Madness

Can’t afford a randomized controlled trial? You’re not alone, and a pretest-posttest design may get you further than you think. In this episode, Stacey Merola and Bill Pate dig into pre-post designs: where they fall short, how to make them stronger, and why they still matter for nonprofits pilot-testing a program or going after funding to scale.
They start with the threats to validity that can undermine a simple before-and-after comparison: regression to the mean, maturation, test sensitization, rater drift and attrition. Then they turn to practical ways to shore up the design. They cover retrospective pre-post tests, including new research on how survey layout affects what people remember, and Bill’s marine biology program where self-ratings went down after training. They also cover gain scores and propensity score matching for Department of Education projects, delayed post-tests to measure whether effects last, and non-equivalent dependent variables as a low-cost stand-in for a comparison group.
Finally, they get into the analysis: when to use a paired t-test, when to switch to the Wilcoxon signed-rank test, and how careful documentation can help your evidence stand up to funders.
In this episode:
Why pre-post designs still have a place in program evaluation
Common threats to validity, and how to spot them
Getting the most out of retrospective pre-post surveys
Gain scores, propensity score matching and What Works Clearinghouse standards
Delayed post-tests and interrupted time series
Paired t-tests vs. Wilcoxon signed-rank tests
References mentioned:Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference.Hwalek, M., Pierce, J., & Straub, V. (2024). Designing a questionnaire with retrospective pre-post items: Format matters. Evaluation and Program Planning, Volume 103. https://doi.org/10.1016/j.evalprogplan.2024.102411
Episode notes can be found here.
Podcast music was written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino. 

Content note: This episode includes a discussion of sexual assault and gender-based violence. It also contains some explicit language. 
How do you study something as sensitive as sexual assault, and do it ethically? In Episode 23, Stacey and William talk with Dr. Alison Cares, an associate professor of sociology at the University of Central Florida who researches gender-based violence on college campuses. Alison explains how she recruits survivors through surveys, flyers and Reddit threads. She describes how informed consent and empowerment models protect participants, and why IRBs and grant reviewers sometimes underestimate how resilient survivors are. She also explains why campus climate surveys can produce very different numbers, like Cornell’s widely reported 35%, depending on how assault is defined and over what period. The conversation then turns to prevention: what bystander intervention can and can’t do, why community-level change matters, and why teaching consent has to start early.
Mentioned in this episode:
Sexual Citizens: Sex, Power, and Assault on Campus by Jennifer S. Hirsch and Shamus Khan (W. W. Norton, 2021)
A summary of this episode can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

What can a research librarian find that a search engine, or an AI tool, can't? In this episode, Stacey Merola and William Pate talk with Matthew Von Hendy, a research librarian and the founder of Green Heron Information Services, about where library science and evaluation meet.
Matthew worked at science and technology libraries in the DC area, including EPA, NASA and the National Academies, before he started consulting for evaluators in 2012. He tells stories from that work, like tracking down an obscure federal survey question about limited-English speakers and finding hard-to-locate survey items on fish consumption. He also explains what makes a literature search work: the reference interview, structured protocols like PRISMA, and the persistence it takes to find gray literature such as dissertations, government reports and patents.
The conversation also covers:
Why people have predicted that technology would replace librarians for decades, from microfiche to the web to AI, and why it hasn't happened
What AI can and can't do for research, including the risk of made-up citations and the trouble that's causing in courtrooms
The real cost of doing research yourself versus hiring an expert, including the person who spent 35 hours avoiding a $35 article
How a single networking dinner led to lasting collaborations among independent evaluators
How a firefighting-foam research project turned into a popular PFAS newsletter and made Matthew "the PFAS dude"
This episode is for evaluators, researchers and anyone who has ever wondered whether their search missed something important.
Learn more about Matthew's work at www.greenheroninformation.com.
A summary of the episode can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

Can AI really handle coding qualitative data, or is it just a shortcut that costs you in the end? In this episode, Stacey and Bill sit down with Dr. Asher Beckwitt, a researcher with 26 years of experience across 5,000 qualitative and mixed-methods studies, to unpack where AI genuinely helps with qualitative data analysis, and where it falls short. Asher walks through choosing compliant, privacy-safe AI tools, why human verification of AI outputs is non-negotiable, and how to document AI use for transparency and IRB compliance (including the often-overlooked step of fully deleting data from AI systems). They also break down the differences between grounded theory, thematic analysis, and phenomenology, and where AI fits (or doesn't) in each.  A must-listen for researchers weighing AI tools against methodological rigor in their qualitative research projects. 
This episode also touches a fish consumption study Asher led for a tribal nation using community-based participatory research methods. This was a separate, non-AI project worth a listen on its own.
Dr. Asher Beckwitt can be reached at asherbeckwitt@gmail.com or www.asherconsult.com.
A summary of this episode can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

In this episode, Stacey and Bill talk with Joan Wagner about how systems engineering and industrial engineering (IE) work together. The former includes defining and verifying requirements, while the later involves optimizing processes and efficiency.
Highlights include:
The "Systems Engineering V" model and the importance of setting clear project boundaries
Why Lean and Kaizen work best when the people doing the work drive the improvements
A story about building stakeholder buy-in with a resistant shop-floor manager
Costly planning failures, including a machine built too tall for its doorway
Where Six Sigma works well, and where it gets misapplied
The ongoing search for better systems-engineering documentation tools
Joan's outreach work with ISE Day, bringing IE education to K–12 students
Episode notes can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

Focusing on policy recommendations as part of the evaluation is critical because health data without policy recommendations grounded in data creates a dangerous blind spot for policymakers, often leading to well-intentioned regulations that fail on the ground. 
 
While quantitative metrics can pinpoint a drop in Medicaid enrollment, for example, they cannot capture the confusing administrative hurdles, conflicting agency rules, or frontline operational friction causing those trends. By integrating targeted focus groups and interviews into the data findings, evaluators can diagnose the actual root causes of systemic failures which can then provide opportunities for strong policy recommendations and policy frameworks. 
 
In this episode, we're joined by Nalini Pande, Managing Director at Sappho Health Strategies, to talk about the unique challenges of evaluating healthcare programs and translating results into policy. Drawing on her deep background in Medicaid, Medicare, and ACA policy, Nalini makes the case for weaving policy considerations into an evaluation from day one, rather than adding them on at the end.
 
A summary of this episode can be found here. 
 
The music for this podcast was written and produced by Stacey Merola. The artwork was created by Zoe Targino. 
 

What do mixed methods interrupted time series designs, business development, and moose have in common? More than you might think.
In this episode of The Method to Our Madness, we explore the power of mixed methods interrupted time series designs and how they help researchers and evaluators understand change over time. We also share insights from the world of business development, including the challenges and opportunities of building partnerships, pursuing new work, and growing a business. Along the way, we take an unexpected detour into the world of moose, because sometimes the most memorable conversations happen when data, strategy, and wildlife collide.
Whether you're interested in research methods, professional growth, or simply enjoy hearing smart people wander off on a tangent, this episode offers practical insights, candid reflections, and a few moose stories you didn't know you needed.
Podcast music written and produced by Stacey Merola. Podcast artwork was created by Zoe Targino. 
 

What if your logic model is too simple to capture how your program actually works? Hosts Dr. Stacey Merola, and Bill Pate sit down with Dr. Bernadette Wright to explore knowledge mapping. A knowledge map is a visual approach that uses circles and arrows to show the interconnected web of causes and effects behind real-world outcomes, instead of forcing everything into a rigid input-output framework.
Dr. Wright walks through how knowledge maps are built (often with 20-25+ interconnected concepts), why multiple reviewers matter, and how the process helps teams surface feedback loops, catch oversimplified solutions, and get aligned on what's actually driving results. The conversation also covers popular mapping tools, how the approach compares to traditional logic models, and where generative AI might fit into the future of concept mapping.
A must-listen for evaluators, researchers, and program leaders who want a clearer picture of complexity.
An example of a knowledge map can be found here. Source: Strategic Knowledge Mapping: The Co-Creation of Useful Knowledge
A summary of the episode can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

Dr. Stacey Merola and Bill Pate sit down with Dr. Toks Fashola, research professor at American University, to unpack what it really means to do culturally responsive program evaluation. Dr. Fashola breaks down how spurious variables can quietly become confounding ones by using everyday examples like parking tickets in Baltimore and medication adherence in diabetes care and explains why understanding a community's history and context is essential to getting evaluation right. Drawing on her own experience with an after-school program in Baltimore, she makes the case for treating stakeholders as co-designers rather than subjects, and for asking whether disappointing results point to a flawed program rather than a flawed population. The conversation touches on the work of Jean Piaget and Maria Montessori as early models of culturally responsive practice and closes with a bigger idea: that done well, evaluation isn't just a technical exercise — it's a path toward social justice and mutual learning for evaluators and communities alike.
Additional notes from this episode can be found here. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino. 

What if the most tedious parts of your research workflow could shrink from months to minutes? In this episode, hosts Stacey Merola and William Pate sit down with Ron Wilson, Assistant Graduate Program Director for the GIS program at the University of Maryland Baltimore County, to explore how AI is reshaping the way researchers and educators work. Ron shares how AI went from helping him stress-test research ideas to slashing coding time, generating synthetic datasets, and building classroom visualizations in 20 minutes flat. But it's not all hype — Ron also gets candid about responsible AI use, the importance of prompt engineering, and why treating AI as a tool rather than a revolution is the key to getting real results. Whether you're a data professional, educator, or just AI-curious, this episode is packed with practical insight.
Links to additional information
For a summary of this episode, check out the notes here. 
Learn more about the UMBC GIS graduate program here, including their GIS Master of Professional Studies which is a two-year (18 credits) program designed for early-to-mid-career professionals seeking a more foundational knowledge of GI that goes beyond learning the basics of technology or application of analytical tools. 
For more information about this topic, check out the UMBC Mic’d Up Podcast where they recently released an episode on GIS and AI. 
The podcast music was written and produced by Stacey Merola. The podcast artwork was created by Zoe Targino.