Standards Future Series

The Future of Impact Reporting: Blair Dimock and Ben McNamee at Ontario Trillium Foundation

Posted By SAMETRICA Customer Success on Dec 12, 2018 7:34:29 AM

Blair Dimock is the VP of Partnerships and Knowledge at Ontario Trillium Foundation

Ben McNamee is the Director of Measurement and Business Intelligence.

Tell us about you. What motivates you? What challenges have you been focused on solving?

Blair:  I joined Trillium about 12.5 years ago. We've been on a journey down the continuum of moving from measuring only simple things like activities and outputs, to trying to become an impact grantor with the ability to measure and report on the impact and results in different ways. 

Ben:  I joined OTF just over three years ago now, as OTF was just beginning that the new strategy of measuring outcomes and measuring impact, and sort of moving away from that activities and output focus. So my background is as an applied economist and so that skill set fit for the shift.

I view this work really as almost an ethical imperative to look at the actual outcomes of all the dollars invested in the social sector - The hundreds of millions, billions of dollars invested - And not much knowledge as to what is actually changing as a result of these investments. And so the state of impact valuation to be able to answer those questions and not simply answering those questions but then doing the next step of learning from that and acting on that as well. 

"I view this work really as almost an ethical imperative"

Tell us about your experience with impact reporting? What role do you see OTF playing in the impact reporting landscape? How will OTF’s role work with others in the space?

Blair: In 2011, we spent our billionth dollar as a grant-making foundation. I can remember that day when I saw that number come across my desk. I walked into my CEO’s office and said “Hey do you realize we've just invested our billionth dollar? What a great story this could be for the Province of Ontario and the Foundation!” And the look I got back from my CEO at the time was like a deer caught in the headlights because we had no way of creating a narrative around what difference that billion dollars had made in the Province of Ontario. We could cite in great detail where the money has been spent for every single grant that had been made. But we had no way of really talking about what difference that billion dollars had made in the lives of people living in the Province. So we decided to pivot, to reinvent our strategy with a very clear view in mind. We wanted to be able to tell a very different story after we spent the next billion dollars which would've been another 10 years out. And that was actually a challenge that the next CEO put to us and said “OK figure it out. How are we going to demonstrate and tell the story of what difference a billion dollars makes over the next 10 years in communities in Ontario?” That became the focal point for developing a whole new approach both to what we look for when we're asking applicants to fill out an application form to what we're looking for in terms of how the results that are to be reported and defined, the kind of impact that we’re looking at. We’re being much more intentional about the ways in which we are using data and storytelling differently to actually shift the narrative.

"In 2011, we spent our billionth dollar as a grant-making foundation. I can remember that day... I walked into my CEO’s office and said “Hey do you realize we've just invested our billionth dollar? What a great story this could be for the Province of Ontario and the Foundation!” And the look I got back from my CEO at the time was like a deer caught in the headlights."

Ben: My introduction to impact, and measuring impact is through the Social Return on Investment framework and lens. And somewhat through my academic work in applied economics. The level of rigor required of academic work and social return on investment framework is so high, and really is such a barrier for so many people to access this kind of work. But our context at OTF of being a grantor for the whole province from north to south, east to west, small world communities downtown Toronto etc. requires that the approach we take is something I can fit and make work for all of our beneficiaries.

OTF is in this position of trying to straddle the trade-off between really good rigor and making sure we have confidence in the impact measurement and the impact of our stakeholders and grantees. But, also the recognition of meeting our stakeholders and grantees where they’re at, and what works for them and their context and their capacities. And so, I think there's a situation where the field of impact reporting has gone so far beyond and become really sophisticated in some areas. And in some ways, the role we can play in this field is to help everyone catch up and be that bridge between that the forefront of the field and where most of the sector is. 

In some ways what you’re talking about is similar to string theory in physics. Tying together the very small objects with large events like black holes. Being able to bridge the small with the large into one cohesive picture is a challenge. It also sounds like Ontario Trillium Foundation is uniquely challenged, in Canada at least, by scale. What do you think success at scale looks like? In some cases, a grantee may not even know how to define an outcome. 

Blair: There are two challenges around the scale. The first is it’s not just about the scale, but it's about the breadth of the scale. So high volume is different than simply a lot of money. A high number of grants, very diverse array in the portfolio. There's a huge challenge. How do you measure and report on impact in a meaningful way for grants that might be $10,000 to do a small project in one year? Versus a multi-year and larger dollar investment?  

The solution has been to make sure that this is not a one-size-fits-all approach. We have to be very tailored to the extent we can. I think the other challenge for us is striking the right balance between standardization and flexibility for the grantees. Without a certain degree of a standardization, the other challenge of scale can't be addressed, which is how you aggregate results across a multitude of grants. That's been one of our most interesting challenges. Not just measuring impact, reporting on it, learning from it, but how do we do it in a way that allows us to change the narrative from reporting on one grant at a time? What's happening as a result of a portfolio of grants? It's a challenge - One person’s standardization is another person's reductionism.

"One person’s standardization is another person's reductionism."

There's a series of dance steps that you’re constantly doing to try to strike a workable balance.

Ben: That challenge is also such a big opportunity – The distinction of size versus breadth is important we have a huge breadth. We give around 700 grants a year, all in one Province.

Blair: Across 23 different grant results.

Ben: Sounds about right, ha. Maybe 24. And it’s that breadth that allows us to be that bridge between that local context, the grantees, where they’re at, and the need for rigor. I don't know of anyone else who has potential sample 40 different organizations working on getting kids physically active, measuring in the same way. I mean, the potential of a dataset like that to influence the field, to share what’s working in different contexts, to improve our internal decision-making, to share it with the sector, to co-create with the sector using the evidence that comes with this. It's an incredible opportunity. The huge sign of success would be once we start having that feedback loop. 

Blair: The other measure of success for us is that more and more organizations, and people working in the field, are actually engaged in more meaningful impact measurement and reporting, and using that information. We always knew there was a huge appetite in the sector for being better at capturing results and knowing what was working and what wasn't. Sharing that kind of thing, to have an appetite to do something and to be actually able to do it. My experience over the last decade has been really encouraging. There's always resistance to new things. There is always a bit of fear of the unknown. There is always a bit of healthy questioning of that next latest, science-y, or academic approach to doing this work, but at the end of the day, I really think that there's been a growing shared value proposition around getting better at measuring impact and learning from it. 

And to me, that's the ultimate success. We’re not all the way there yet, but I've been encouraged over the last number of years about how I think that that trajectory is increasing in slope.

When you look at the field, what are the signs of encouragement that you’re seeing?

Blair: I'm intrigued by things. Like, as part of our new measurement protocol we introduced a series of standardized pre-and post-survey instruments so that we could capture a qualitative change in certain of the grant results that we are funding within our Grow Stream. We did extensive research to identify the most appropriate and effective and user-friendly survey tools. And predictably there was resistance to the notion of grantees being told what surveys they had to use. Questioning of “why this survey”, questioning of  “why this question in this survey” and questioning of “why these words in this question of the survey” and a lot of pushback about this standardization. But what really encourages me is that while some folks have resisted the idea of the of applying a standardized instrument that they didn't design themselves, others of been really, really helpful.

I tell this the story of this group in Ottawa that works with developmentally challenged adults and they were trying to work with one of the survey instruments that we told them they needed to use. And it clearly was not user-friendly for that clientele. But rather than just putting the walls up and resisting, they actually worked with us to enhance it because there was a recognition that “Yeah, we're trying to do the right thing here. We're trying to actually figure out a way to measure and report on what difference is this work making so that we can strengthen the value proposition for why there should be more funding going in to this kind of work, rather than less”.

Do you see challenges to that ‘increasing slope’?

Ben: Blair’s explanation of some of the challenges of the surveys is bang on. We haven't seen this in OTF yet, but I really see this [challenge] as something that is on the horizon… I mean it is data-heavy work for the most part. It is quantitative and data heavy. And there’s a realization in society in the last year so, of the implications of previous big data projects. Facebook and Twitter and all these things, and the negative implications that can come with that. I see a coming collision in terms of, what does it mean to be a data-heavy organization in this new world? What is ownership and privacy? There is some talk of that. We’ve seen stuff with Sidewalk Labs and then all these peoples are leaving that committee due to the lack of perceived oversight. I see this coming conflict.

And then on the other big piece right now, that's will never go away – There’s always a capacity issue. Capacity and resourcing. They go hand-in-hand. Nonprofits have limited resources. They need to do programming, accounting, HR. They need to do all these things, and then they also need to do this new thing called impact measurement.

We need to ask, “How can we align our impact in our measurement work and requirements so that we're not adding on multiple levels of reporting and measurement work on our grantees?”, so that we can sort of say “do this and that will fulfill requirements for all six of us”. So that's always an ongoing issue. 

Blair: There is increasing talk in the funding community, at least, about how to address such a diverse and large marketplace in terms of the organizations we fund. It's obviously not realistic to expect every single organization that we fund to build up the internal capacity to do the kind of rigorous impact measurement work that we'd love to see. So is there a way to support an intermediary infrastructure of some kind for those organizations that won't ever have the capacity to do the work themselves, and what might that look like? Could we ever imagine the equivalent of a Canadian Institute for Health Information, for example, for the social sector? Hard to figure out exactly how that would work, but that would be the kind of dream state that we would talk about and say “OK how do we enable and support any organization that wants to understand its impact better and share that with others?” How do we actually make that happen without necessarily placing the full burden on that individual organization?”

Yeah, it’s interesting –The field of impact measurement is creating a new way to tell stories. You can kind of create what the roles will look like and figure out how the team will work, because we can borrow from accounting, and we can borrow from technology, but actually it's its own field. That also requires a new set of jobs and titles.

Blair: There’s a great analogy around accounting here.  No one expects small nonprofits to have accountants on staff, but they should have the base knowledge to know how to approach an accountant, to know what the accounts mean. To know where to put things in a software package. It’s the same thing with this, you should have the base knowledge to know what is an outcome, what is an impact, and so on. It's not reasonable to expect everyone to be experts in doing this work, but they should be able to make use of information.

Even at OTF for example, if Ben were to describe the array of talents and skills on his team here at OTF today, it's absolutely remarkable how different it is from what we were talking about, even six or seven years ago.

For this to work successfully for our sector we're going to have to find ways to track down and recruit that needed talent into the sector. I think that will be an ongoing challenge, mostly because of a huge, huge shift from what previous generations have brought in terms of talent in the sector.

Jokingly, you know, tongue in cheek, we kind of say that we used to hire social workers and now we hire data scientists. And those are two very different ways of coming at the work. Both are extremely valuable, right, and our sector would be at a loss without one or the other but the balance is going to shift over time, I'm convinced, and that's what's needed. And I think that will that will be a big driver of change. If we can attract those new types of talent into the sector. I think with some of the new technologies, the new methodologies, the that are becoming more income more mainstream, that'll help a lot.

"Tongue in cheek, we kind of say that we used to higher social workers and now we hire data scientists."

So last question - What do you see as the future of impact reporting. Let's go ahead twenty-five years. What do you think this will look? What will the space look like? And if you have an idea, what role would you want OTF to have to play in that?

Ben: I tend to utopia in my future dreams, thankfully. It’s better than the opposite. But where we have sort of the beneficiary outcome-centered in the sector and impact measurement is something the sector does to improve outcomes. That’s the impetus. It's not about measuring, storytelling, data collection, privacy - All these things -  money, resourcing, capacity, all of that is in service of better outcomes for the people that we serve. And that there's this sort of agreed upon and sort of common understanding of the need to do this and that that it's done at scale. And done as a partnership between all of the players. Between the tech people, funders, nonprofits, intermediary groups, academics and government, and so on, that there is this ecosystem all pushing in the same direction.

We [OTF] are one of the drivers of this right now. That’s what we're trying to push towards and we would be a sort of a key part of that. Being that bridge between all of the pieces of the ecosystem, providing the that a lot of that data insights, and some learnings about doing at scale.

I also see consumers being a consumer of impact reporting and management. I mean, as an as an example, let’s say I'm in crisis. I need a food bank. Where's a good food bank? What are the differences? What fits for me? What would be the best outcomes for me? What food bank should I go to?

Blair: Yeah, I think 25 years from now we will have created a really open, active field of inquiry, and use of data, and so on through shared resources that we won't actually be able to predict what the next big use case is going to be. I guess the analogy would be - No one would have predicted that the actual success of the smartphone was gonna be based on the app developers, not the hardware, not the software of the phones themselves. So I would love to see the day when - And this is actually what lies behind a lot of our efforts to make our data more open and user-friendly – 25 from now, what's the next great app that's coming out of the impact environment? That we would never have been able to predict or guess, because we wouldn't even know where it would come from. And it might be coming from a beneficiary, and it in might be coming from a funder, or whoever, but it’s just become part of the day to day the ecosystem.

Ben: I’m realizing our final dream worlds were very sort of tech-utopic.

Blair: Yeah very much so.

Ben: You know there are cautions to that as well. 

Blair: Yeah. 

Ben: I mean the thing that keeps on sort of running through my head various days and weeks at OTF is that we've also signed on to the philanthropic sector’s call to action around Truth & Reconciliation in Canada. I think a lot about the intersection between impact measurement work and Reconciliation work. Especially as a government agency. Someone working with our indigenous partners, that is a whole other set of histories and contexts and past traumas that come into play. So, there's a piece there around, really, how can it in the in those twenty-five years and we use this tool to really foster and rebuild partnership with indigenous communities at a nation a nation kind of approach.

That takes up a huge piece of real estate in my head. I don’t have any answers, but I know it's something that that needs to be a part of any solution.

Do you have any thoughts around how that might be able to work? 

Ben: I don’t know if it’s a ‘how’. There are principles in place on ownership, data ownership, and valuation principles and so on that we can learn from. But from a more fundamental grounding in different ways of knowing. And different ways of getting to the truth and story and reality. Impact reporting can be very quantitative. It can be very uni-dimensional and very tech-heavy, and so there's a big opportunity there to what does this look like an indigenous context. I mean, we have pretty good indigenous evaluation approaches. What is the indigenous impact reporting approach?

Blair: I think we always have to remind ourselves that even though most of us have come out of a very western kind science-y tradition, right? We have to remind ourselves that these methodologies and approaches are not a value-neutral. They’re value-laden. The challenge we’ve got is, it is a priority for us around our relationship with indigenous communities, but I think even more broadly, in such a culturally diverse context that we work in, we always have to be challenging ourselves – What values are embedded in the methods and technologies that we’re using to do the work? And always have that as a lens to check ourselves to make sure that this is something that’s valuable and meaningful for everybody, not just a slice of the community.

"We have to remind ourselves that these methodologies and approaches are not a value-neutral. They’re value-laden...In such a culturally diverse context that we work in, we always have to be challenging ourselves – What values are embedded in the methods and technologies that we’re using to do the work? "

Ben: That’s where the approach is key. To not approach it as a series of individual projects, but how can we as a collective, as a group, as a sector. How can we work together, really truly together, to move this agenda forward? Bringing in all those different perspectives, those different values, having the process, using tools checked by people and pressure-tested and validated by people from all different contexts. That’s another huge benefit of that. Having that validation and pressure test will help us see where our tools and processes fall down and build from there.

And also allow for multiple perspectives. Because someone could take the data and say “Well, this is what you took from it, and now I am applying a different kind of analysis, and this is the answer that I’m getting”. It allows for a multiplicity of truth that drives towards a scientific consensus where appropriate.

Blair: Particular to our moment in time right now. I think there’s an added imperative around some of this work to make sure that we can be certain about some things that we know to be true. While there are always going to be multiple interpretations of reality and truth and so on, that we do have at least some common foundation of the things that we know to be true and accurate and so on. Because all of that, is really, very much under threat today. And, I think that’s just another lens, like the Reconciliation lens, that we need to keep reminding ourselves of. There’s an important element of this work that is about finding things that we know to be true, and that we can use as a foundation for moving forward and doing things better than we did before. And, not everything is just up to anybody’s interpretation. Otherwise, we’re just going to be floundering. 

There’s a fine and important line between, “What are the facts?” And “what are the interpretations of the facts based on expertise and analysis?”. I think that the first bucket is definitely getting lost. The rigidity needs to come in there – We have to call a fact a fact. There’s nothing else to be said about it. But there’s also an exciting dynamism in that second bucket by saying “Okay, now we can have a discussion about those facts while respecting that they exist.”

As told to Anshula Chowdhury exclusively for SAMETRICA. Transcript edited and condensed for clarity. 


Post a comment

Related Posts

Read More
Standards Future Series

The Future of Impact Reporting: Blair Dimock and Ben McNamee at Ontario Trillium Foundation

Read More
Press releases Standards

SAMETRICA software accredited by Social Value International

Read More
Standards Future Series

The Future of Impact Reporting: Professor Mark Fox at the University of Toronto talks the trajectory of individuals, the need for privacy legislation, and why he focuses on cities

Want to read more industry leading posts?

View All Posts