Thu, March 27, 14:00 - 16:00
2008
This interactive session will explore various approaches to assessing impact and leveraging these efforts to attract resources and drive organisational learning. Participate with panellists in answering key questions including: Who are the real audiences for impact assessment efforts? Who really cares and is it worth the bother? What happens in practice when social entrepreneurs are systematic about measuring impact? How does this lead to organisational learning, increased impact and innovation, or to greater resource mobilisation?
Videos
Mirror, Mirror On The Wall, What’s The Impact Of This All
This interactive session recorded at the 2008 Skoll World Forum explored various approaches to assessing impact and leveraging these efforts to attract resources and drive organisational learning.
Key questions included:
Who are the real audiences for impact assessment efforts?
Who really cares and is it worth the bother?
What happens in practice when social entrepreneurs are systematic about measuring impact?
How does this lead to organisational learning, increased impact and innovation, or to greater resource mobilisation?
Good afternoon. This morning, I was at this session about, speaking about empathy. I wonder how many of you here can empathize with a woman who in summer has to spend four to six hours to haul water for the family or with that little girl child who goes with her to carry two liters of water because that also matters, but that keeps her away from school.
How many of you can empathize with a women who has to get up before the break of dawn and under the cover of darkness has to find a stealthy place where she can defecate. And then, even if she has diarrhea she has to manage to keep it within herself so that there has to be, so that the under the darkness of the night, she can do the same thing.
I'm speaking about what is happening in most developing countries in the twenty-first century. Is this needed? We speak about the Internet and the huge strides we have made? Does a woman has to spend, I mean, forty percent of her time hauling water? Do we need to keep girl children and other children from school because they have to also assist their mothers?
unfortunately, this the truth. This is the truth about my country. many of the countries in Asia and probably in Africa. Four percent Of the rural how source in India have toilets one person have water on tap in the village, one tap in the village. All the others do not have access to save water.
80% of all mobility this is in rural area is caused because of poor quality water. And when we see as to the reason, it is our abysmal Human attitude the disposal of human waste, human waste in raw form find six way back to drinking and x bathing water and you have all sorts of diseases and that accounts for eighty percent of all diseases.
This is not my statistics, it is W.H.O. UNICEF statistics. So in theory, and more than that in practice, eighty percent of the diseases can be eliminated in rural areas in countries like China, India, Countries of Africa, Countries in South East Asia, maybe Countries in South America. If good quality water is delivered in the village, but for this we need sanitation and sanitation on hundred percent basis in an any habitation because even in you cover ninety percent or ninety-five percent or ninety-nine percent of a habitation with sanitation, but this one family shits all over the place, sorry for the language but, what do you have?
you have that recycling of that one family shit all over, I mean in the drinking and bathing water, you are not really improving the health status, so we need to have a 100 percent coverage of sanitation. Another problem we came up in India in this area where I work Orissa is that. Women and men bathe in the same bathing tank or pond women being circumspect as to how much they expose themselves had insufficient cleaning of themselves.
They got fungal and other gynecological problems, which they could not afford to treat and once got, usually kept forever or if got treated, what infected. So if a family would have a toilet, a good toilet, and a bathing room, and Water supply, to use that, and, then the lives of women especially would go would be better?
Girl children would be able to go to school. All the children would be in school. Mothers would have much more dying for their children. They were not being lazy because they have never been lazy. They would engage themselves into other productive means and have a little more leisure for themselves.
So, this is our program. Cover it 100% of habitations with a toilet, a shower and protect water supply, which would be, would be through gravity or some, or would come into a water tower and then Would we distribute to on 3 tabs to every family. One in the toilet, one in the shower and one in the kitchen.
no women after this would walk more then 5 steps to get her water. Well So, what are the impacts that we want to see? Well we have a mechanism right from the very beginning that even before the, we start the program we begin to see what was the existing condition. Every family how much could] they spent on health and curing of deceases especially diarrhea incidents Jaundice.
gastroenteritis and other, cholera, typhoid. Every family, how much they have spent. This may not mean we'll get very good But it takes about two years to implement this program, where people bring all the local materials and externally only the Cement Steal and whatever constructional is given some times no is the dangerous guy.
So people would, and build for themselves the toilet and bathing room and then initially make an endowment which they bring themselves, the interest from this endowment would see that the social cost or external materials are available to future families who come into this village and all without a single exception implemented program.
They pay for it. They maintain it and it's on a hundred percent level. So from the very beginning, we begin to see what was the disease pattern before the two years we implement the program. What is a disease, how much mobility, mortality in the village, and then after safe water and sanitation comes in, and hygiene education, how many children go to school, how many girl children go to school, how many water born diseases every month it is monitored the that man comes to the particular district or the region.
If there is a flashpoint it is immediately acted upon, it comes back to our headquarters Gram Vikas village office, so we monitor it. There are other things we say. We are not building toilets, and water supply. We are building the dignity of the people. We have not yet put a price on dignity, but I think we have to put a price on social inclusion where 100% of the people, especially in India, it's very important; it's cost-driven, that all come under the same platform and implement the program.
We are to put a price on dignity; we need people--I mean we are the brawn, we need brains to put value to that--but just on diseases and the savings that these people make, it is 1:10. For every dollar spent on water and sanitation, safe water and sanitation, we get $9 in return in one year. What greater investments can we make?
What greater impact can we make? What greater empathy can you all have with my sisters and brothers in these developing countries? And we are looking at this impact. Thank you. Thank you very much.
Jeru, and as I've said to the panel owners, "You can sit here, or go to the podium, whichever you're more comfortable with."
Sitting and talking is fine.
The informally, if that's okay with all of you?
And that is fine.
Then why do they stand?
Because you are To get a stretch. Okay, sorry. I'll just talk about and why we think impact assessment is necessary. A flatoon is a name which actually came from children, and a flatoon is a fireball which goes all around the world telling children about their rights, their responsibilities talking to them about how to saving, not just money, but the environment, everything around them; how to plan and budget and also teaching them how to run micro the prices if they want to go for class picnic instead of x chickens and decided that the eggs from the chickens can help them go for a picnic.
Or they've created key chains and things, and they've sold it and they Made money and some of them help their friends Saflatoon is basic teaching children in the age group of 6-14 about them money, rights and responsibilities. Now it sounds really interesting to all of you, but what most of us tell ourselves that's rubbish.
And that's where we come in and that's what in fact is important to us. Most people tell us no way can children handle money, no way are children going be able to do micro-enterprises or planning and budgeting. That doesn't work for children, so for us, impact assessment becomes actually a real a sense to look at proving our concept, proving our belief in children that if you invest in children, you'll actually be making every child, an entrepreneur, social entrepreneur, just a caring human being, and probably helping get them out of the cycle of poverty.
So we've looked at impact assessment and why this session and this topic really interests us. What is it that your trying to measure? How do you go about measuring it, what are the obstacles to measurement, and what do you need to convince policy makers to take the sleep on faith, the sleep of faith because ultimately there are a billion children in the world, who if they had social and finical education, would definitely feel more empowered.
and would probably be able to get out of the cycle of poverty. So that's what we are looking at and those are our challenges. And that's what I'm hoping to look out for in the session and get. And also that is what we are talking about so keeping to the three minutes, which you had said and moving over to the right.
Thank you. Thank you, Brian, over to you. thank you.
Autopsy is to cadaver as evaluation is to most non-profit programs. They were expensive, rare, invasive, backward looking, and often too late to help. There's a big difference though, non-profits nerve die or fail. We have a cultural aversion to looking at failure. What we've been trying to do at Acumen Fund is think of ways to develop measures of a business or enterprise's pulse, its blood pressure, its temperature.
Simple, easily implementable things that will allow you to course correct in course of implementation. I think Jeff said the other night in the opening plenary that the social enterprise space is lacking one thing, time and I would add to that, in addition to time, we're lacking evidence. And it's our hope to continue to lead discussion in the sector about what kind of evidence we can collect from each of the enterprises to prove the case that social enterprise is in fact a better mouse trap.
Let me spend a couple of minutes talking about how we think about impact assessment from an investors prospective. Acumen Fund is a social investor. We've invested about thirty-five million dollars in health, water, housing, and energy companies for-profits and non-profits mostly in South Asia and East Africa.
We act as a venture capitalist at a very early stage of a company's growth, and we try and work with them over the life cycle of our investment, often five to seven years to help them solve a range of operational and business challenges. When we make the investment, we start the metrics or impact assessment question by asking ourselves two questions.
One does the intervention matter, and what evidence do we have that it does, and so we'll often look a the range of scientific literature about whether or not anti-malaria bed nets help reduce morbidity and mortality from malaria when properly implemented. And then we'll ask ourselves the question, who's the best charity out there doing this work, and could our donor achieve more output?
Careful not to talk about outcomes or impact but more output more bed nets distributed per dollar of philanthropy than Save the Children or PSI, and we hold ourselves to the standard of trying to make sure that we are more effective per dollar of output. It's not always the case, we have failures, and we talk about them, mostly internally.
But the social capital marketplace does exist, it's just very inefficient and fragmented. Once we've decided that there's an investment that makes sense, and that there's evidence that the intervention is valuable, we turn to the entrepreneur and help them think through what matrix will matter for their business.
We have a mantra that we do not want to measure anything that does not help the business succeed. We also do not want to measure anything that the business cannot generate from its existing information systems. If the business can not measure things that are important for it's business, from it's existing business systems, then we'll help.
We haven't gotten in many insistence's, but there are times times when we are able to find a few key metrics that will help the businesses identify things that are really important levers. Let me give you a quick example of an ambulance company in Mumbai that we've invested in. 1298 is the first private sector ambulance company in India and it is trying to scale to serve all of the Mumbai metropolitan one region with seventy ambulances.
Right now there are about a couple hundred ambulances in Mumbai. Most of the serve as hearses for all those cadavers. and there's autopsy's. There motto is to in fact create an emergency response system. They're really trying to build an industry and they've got a cross subsidy model. They want to have ambulance access for all as their motto.
And that depends largely on their ability to serve both high-end customers who are going to the private hospitals as well as low-end customers who can't afford a ride to the government hospital. When we talk to them about what metrics matter. We think about revenues, end costs, and that income, those are the obvious ones.
Numbers of ambulance, that's also obviously they are trying to go from 10 to 70 in the next couple of years numbers of call per day per ambulance actually is the key number to where not that there going to make it and whether they'll be profitable. Number of free rides is a percentage of total rides, is the other number.
If they hit the two numbers on number of rides per day and percentage of free rides around twenty-five or thirty percent. Then we think that the model can scale and it can have the kind of social impact that we intended when we made the investment. Those aren't necessarily easy things to track, but we've made it out job to track them on a quarterly basis against plan and when we have the quarterly numbers coming We don't just sort of say, "Great,t hey're succeeding or not." We sit down with the management team and ask them, what's working?
What's not working? Why? What can we do to help? It's not punitive, but it's collaborative to try and identify solutions that will help the business grow. We've been working with some of our colleagues at Google, the Skoll Foundation has been supporting us and a number of our peers in the sector to think about whether or not there's a way for us to take the information management tool that we as an investor have been using for portfolio management, to create a platform for others in the sector, so that when we think about the work at Rimvydas and how much they're able to do with water and sanitation per dollar of investment.
We can compare that to some of our water and sanitation investments, and facilitate the learning across organizations and we're hoping that over the course of the next couple of years we can begin to aggregate and benchmark information and I would go back to what Roger said at the opening; not everything that is important is going to be measured but if you aren't measuring some of these things, you won't be able to learn, adapt and grow the business.
I'll close with a quote from Oliver Wendell Homes which talks about the kind of insight we would like to drive from our own measurement system at Acumen Fund as well as what we'd like to see the field arrive at. I wouldn't give a fig for the simplicity this side of complexity. But it would give my life for the simplicity on the far side of complexity.
And what were hoping to do is identify a way to measure that recognizes the nuance and difficulties of the businesses that we support, but abstracts it to a few simple things that they can actually implement and that will help them make better decisions and approve the lives of the millions of people that they aim to serve.
Thank you.
Thank you Brian and on to you David.
Thank you, good afternoon everyone. Sometimes I feel, Brian, like I'm a bit of one of those complicating factors that thorn in your side as I'm tending to bring in more complicating issues. Very briefly, in the short time that I have I'd like to not discuss some things and take them for granted. One is the kind of complexity and difficulty that it is in trying to measure real impact Issues around attribution, counteractions, organizational versus public domain information and how data moves out and becomes available and many many other factors.
We've heard bits and pieces of those from the cases that Joe shared with us. I'm also not going to talk about some of what I think are the critical trends in the world that are bringing us closer to solutions of some of these problems. The rise and growth of more professional management in the social change domain under stepping up and taking impact measurement in very constructive and learning ways and I think Brian has given us just a great example.
An apple of one thunder that's doing that. Maturing methodologies of evaluation and pluralism in different approaches, mixing and matching different approaches. measurement and evaluation that are being used better, and new communication technologies which help us. These are all big topics and we don't have time to get into them I just want to throw them out as things I'm not going to discuss.
And would rather jump straight to the collisions of our work in the last few years, which has been focused on really trying to answer the question "What do we really want to measure and communicate in order to understand what sustains social change and sustainable development?" And the starting point in working with lots of organizations and looking across at literature and doing lots of net working over the last few years is that both monitoring and evaluation system generally are week.
I'm sure there are exception abounding in this room. But there's a growing awareness of that, a really openistic kind of fixing the problem. We did a survey last year, an online survey, and 85 percent of the NGO's that answered the survey around the world, said they really were seeking to improve their monitoring and evaluation.
And I think the kind of turnout we've had today as another example of the interest in this space. Another big take away from us, and I think it flows right out of what you've heard so far this morning is that you kind of, as we put it, you need a whole system solution that you can't look at measurement only from the point of view of one of those actors in the system who are using the data.
So you don't just look at it from the point of view of donors or management, but also of boards of organizations and, most importantly from our perspective, of those who are meant to be benefiting from the work and how are they involved in actually determining the metrics. And we use the, we ask the question, who counts?
And we meet it in both senses of the term: who's doing the counting, and who matters.
And finally, that the kind of critical overall objective behind all of this has to be learning at points of every different stakeholder that's involved. And our theory of change at Keystone is that society solved the important complex problems when they learned how to solve them.
So we went on from these kinds of takeaways to try to develop a kind of theory of public reporting for social and environmental outcomes that honored those thoughts and others. And in doing that, we're really trying to bear in mind that, that kind of, whatever gets measured gets used. And that that's the critical metric for all evaluation efforts.
If you are going to evaluate the evaluation, the bottom line is, is it being used and how well is it being used? So along the way I've been collecting metrics and looking at indicators, and kind of trying to say what's the best metric that anybody's got that I've found. My current favorite metric comes from Alcoholics Anonymous in the United States.
I don't know how many of you are familiar with this, but it's a really powerful metric, which is days of sobriety. And they measure it at the level at the individual addict, but also at the level of chapters and regions. And it's measured quite carefully. And what's I think particularly powerful about that metric is that it begins with the person who's meant to benefit.
It's something that's generated by that individual in a context of the organization's work and then flows up from there.
Which brings us to our core work. Which has come to be about in all the different things that are happening, and all the different things that are important around measuring and communicating social change We've decided to focus in on this issue of constituency feedback. And making sure that organizations have ready tools and ways to capture feedback and systematically incorporate it into their work and into their public reporting.
So we just want to close by telling you an example of a current pilot project that we're engaged with in the United States with the Alliance for Children and Families, which is a network of three hundred and fifty children and family service organizations. That do both service delivery and advocacy work for poor families in America.
And of the Alliance brought together a group of its members with a consortium of interested funders to talk through the possibility of doing a pilot project where we developed a common questionnaire that would be administered to their primary constituents. In order to get feedback, both about how the constituents felt about that organization but that would also enable us to give some comparitive reference and a kind of ranking with respect to the others in the group.
The value of the comparative dimension is potentially quite high because it allows an organization to see not like if we're using a seven point Likert scale on the questions, they, you might think initially that a 5.8 is a pretty good answer, a pretty good score, and that we're okay on this dimension or this cluster of questions, how responsive we are to our constituents.
But then you see that, actually, 5.8 is only fifty-two percent in the cohort learning conversation in the peer group, in the cohort. So you can start to exchange experience around objective data about what seems to be working for you and how could we try that technique and maybe that would work for us.
So you use it not in a sense as a kind of competitive table, who's on top, but rather as the basis for learning and exchange. So, finally then, we are saying that kind of key missing piece in a lot of evaluation work, which is surrounding systematic or around constituency feedback, that when you take it seriously, you can get evidence of your impact - not always direct evidence, but evidence of progress.
You can certainly get insights into your performance. You increase the voice and stake of your constituents in your organization, which is absolutely critical. This is the "I've been so for 15 days." You all may be thinking at this point, "This guy hasn't been sober in a week." And also, when put into the public domain, this data really establishes your accountability to externals because you're showing the way in which you're listening and responding to your primary beneficiaries.
Thank you.
Thank you very much, David. Last, but certainly not least, Faye.
So in...can you hear me? In 1989, the Centers for Disease Control made it The first grants to community based organizations to do HIV prevention work. And I was a young, kid of my twenties at the time. And I had just joined a non-profit consulting firm. And my first assignment was to be the project director, the Principal investigator on the evaluation for a community based HIV prevention program.
I was so excited. I went out to the organization, a good organization in San Francisco, that was very excited to have this big grant from the Centers of Disease Control. and doing some prevention work with high-risk young people in San Francisco, run-away kids and high-risk young people. And at the project launch meeting, I met with a project director who was an incredible visionary, a guy by the name of Danny Keenan we had a Meeting and talking about evaluation.
After the meeting we had a submeeting and I said, "Danny, you know, I am here to help you. I want to figure out what the best methods are." on how we can really learn from this. And he turned to me and he said, "Faye, I don't give a shit about evaluation. You do what you need to do and stay out of my way because I'm here to save lives." I was 25 and very eager and I was quite deflated.
I'm gonna...I only have four minutes left, so I'm gonna spare you the rest of the story. But I tell you that story because the lesson that Danny taught me in that moment was that evaluation always had to add value; that my work needed to be as important and contribute to his work; that the measurement needed to add value to those who are working to save lives.
And that's...if you take nothing else away from what I'm going to say, that is sort of the most important principle, that the measurement always needs to add value.
Danny and I went on to have a great relationship, and he ended up using lots of data involving in the data collection, and we did really good measurement, and he used that data to help spawn the development of really great psycho-social and medical clinic for young people that went on to do even further great work, which we further evaluated and continued to learn from.
So, it had to add value, and it did and I always remember Danny's lesson in my everyday work. I want to make five points in next three and a half minutes. One is, I think good measurement starts with good strategy. Good strategy starts with framing a problem clearly. Framing the problem, that's point number one.
Tell you a quick story about the value of that. In the Gates Foundation in our work, in our local work in the pacific northwest we have three different program areas that will help build development in the U.S. program. In the U.S. program we have our local Local giving which is in the pacific northwest and in that giving program we have a strategic program called, 'Sound Families' and its a program devoted to addressing the issues related to family homelessness.
We had a seven year initiative from 2000 to 2007 that was focused on family homelessness in the three county region. And we had a goal, a measurable goal, fifteen hundred units that we were gonna build. Bricks and mortar strategy. We were gonna build 1500 units that combined housing with services, so it was sort of the state of the art thinking was that's you needed to do to really address the issue of homelessness was combine housing and services, and help homeless folks transition off of the streets, give them a period of time to stabilize and move into homes.
Seven years later, we did a formative evaluation. That was before my time. We had hit that mark, we had hit 1500 units and 66 percent of the the families were stable, their incomes had gone up a little bit, and some of the kids were doing a lot better. We had lots of good data to speak to what had happened.
But we actually didn't solve the problem of family homelessness in the Pacific northwest. And in fact, during that time, the family, the problem of family homelessness got worse. And the problem of services got worse. So we never actually set out to solve the problem we set out to do what? Build 1500 units.
And we did that. So actually, it was a successful initiative. But now, was we began to this past year work on looking at "What is our strategy go to be of family homelessness?" We're now re-framing the problem to not be about actually we didn't actually frame the problem last time. We just said we're gonna build 1500 units.
And now, we're framing the problem to say we actually are gonna try and eliminate family homelessness. We're gonna, I'm not gonna go through the whole strategy, but it's a prevention method, a rapid rehousing method, and a whole set of investments in infrastructure We're going to learn a lot, we may fail its a risky strategy.
The point is that we're framing the problem differently and all of our measurement is now related to that framed problem. And what can we learn about how to actually prevent family homelessness that can be useful, both in the Puget Sound region, but also Nationwide, the second point, I'm going to try to pick up the pace, is the benefit of knowing must be worth the cost of asking, we were talking about this in preparing our talk about how expensive it is to do measurement.
Impact measurement in the traditional sense, of the term with counter factual, is very expensive to do. Sometimes measurement can take on a life of its own. And so, one of the things that we're beginning to experiment with at the foundation is, really being very clear, again related to strategy, what are our learning questions and try to craft a learning agenda for every single strategy that we have.
What are learning questions? Why do we have these questions? What decisions the informer. If we had the answers, would we do anything differently?. When do we need the answers to these questions by, and with what degree of certainty? And are there data sources available to actually answer these questions?
And what are the best methods for using for answering these questions? And making sure that all of those things calibrate because often, you've got measurement going one way and program and investment going another way. But this is true in the foundation world and before I came to the Foundation World a year and a half ago I was on the Practice World, it was true there too, that measurement often goes one way and practice goes another, and so, really making sure that you calibrate your data collection, your measurement, to what it is you're actually trying to do, is really important.
A quick story on that. I was just in India a couple, about a month ago visiting our foundation's program in what we call our Avahan program, HIV prevention program, working with sex workers, men who have sex with men, and truckers have sex with both of them and I was visiting with the female sex workers and It was amazing because they were talking about their work doing peer education on the streets with other sex workers, and they were talking with such sophistication about their measurement they there's a slide that I'm not going to show because we're not doing power point now.
But this is one of my favorite pictures from my trip to India of this female sex worker who is a peer educator, who is pointing to a slide that says monitoring and evaluation. And she's using all the data that they're collecting related to the work that she's doing on the streets do outreach with other sex workers doing education, doing distributing condoms and, because she's illiterate but very smart and very sophisticated.
She's. They're. They have developed a whole coding system that is figurative so they use pictures to describe what it is they are doing and they use different colored to put on these charts that describe what it is their doing and what kind of contacts their doing with which women and which street corners and then they collate that data once a month.
And they look it and they use it to course correct, on the ground, as they're doing their work, and then that data gets added up to look at the district, and then it's looked at across all the districts on a regular basis as well. So it's a really good example of, "How do you use data and aggregate it from the grounds on up to a whole huge portion of the country.
It not only makes for good value in measurement. it also increases the quality of the data because it's actually being used. Point three in the spirit of and the rest of the contents of this conference around culture in context id say how you measure is really important how you do your measurements.
So the example in India in Avahan with the sex workers is a really good case in point. How their doing measurement is consistent with their cultural context. And and it's useful and meaningful and doable. We find a similar kind of lesson everywhere. When you're working with high school kids, you've got to be really careful how you ask them those questions Those high school kids know exactly how to answer.
The fourth point I want to make is time, beware of many dimensions of time, and time can be of the essence, and a quick story again on the Abahan program when Bill and Melinda went to India a number of years back, they were very moved of what was happening. There with the female sex workers and wanted us to do something as a foundation to address the HIV epidemic grow and what we thought of at the time as a fast-growing epidemic in India.
They said, "We want to do something fast. There's a real sense of urgency." So we went And and built our India office and had a great sense of urgency, and the gentleman who was running that program knew that he needed to do measurement. So he convened the best minds in evaluation at the time and said, "I want you to help me figure out how I should do measurement".
And they talked and talked and talked, and they spent a year talking about the best ways to do measurement. And in the meantime, our office was filled with a sense of urgency to get going on the program and so we did get going on the program. And in the process lost the opportunity to get a true baseline.
And so, time is of the essence, that was a really important lesson for us. And we're making up for it, now using other methods, but it was just a very important lesson. My last point is just to remember to question assumptions, don't focus on proof as much as learning and always be curious.
Thank you. my inclination now, was to actually to go back to Joe in Duru and say we've just heard from three folks who are doing lots of the leading thinking on, on measurement of impact. What are some of your reactions to what you You've heard about the theory and the practices they've, as they've described it.
That makes sense to you in your organizations. What about that, what would work best what they've said would be the biggest challenge. Jaru?
I think what actually for me what resonates is what said last. Is this whole thing of when do you start with the base line, what do you do because we were really discussing that and you want to get things just right. And, at the same time,you know you have to do for the fun side so for me this whole time.
Something which I personally am going to take back. At the same time which is being said even by Brad and David, one of. things which, from what all three of them are saying and also what school keeps talking about and I think hasn't been answered is, we talk about bringing about systems change. And yet I didn't hear from any of the three and even I don't know the answer.
So it would be nice to discuss this. What is the the impact that we need to measure. Or what is it that we need to measure which can help us really bring about systems change? so if I look at our main theory is children have economic rights. They have social rights and that's what we want to see, people's mindset to start thinking about that.
I have no measurement tools to start looking at that and my biggest worry is instead of bringing about that system's change I'm start will be looking about that activity based indicators and that's where my fear and my attention is. And, I haven't really heard that answer in this and I don't know, I'm a bit caught.
Sorry. Sorry for asking a fantastic question. Don't be. Don't be. No. No. I mean what Brian, David a, what, I mean I think is proposing one of the trickiest questions out there, Well. your thoughts on that. David?
I'll take a first cut. yeah. Thank you. on what i may add as we built our methodology we were very focused on trying to look not just at the organization as the unit of change, but if you will, at the system. So we ask the kind of first question who needs to be involved in measuring in order to get a picture of system change.
And then we look to practical operation cycle planning, doing, reporting, with that in mind. One of the simple things we suggest that organizations do, Jeroo, and it used to this. Develop a story of what success looks like in a sense of what the indicators of change together with different actors in the system so that you get a composite starts to piece together the system as a whole indicators from different perspectives and so in a sense the role that the system changer is playing is one of a conversation across the system about what success would look like, what system change would be.
From that you can work back and start to put activities and other indicators output kind of indicators together that start to lead you toward that.
Brian do you have a thought? I somehow think that Oliver Wendall Homes would like the question too, right yeah, I don't know what to say, because it goes right to code.
yep yep.
I think part of it defines that depends on how broadly you define the system over which you want to have change. Our theory of change at Acumen Fund is that market-based solutions can in fact complement a development in charitable or purely private sector approaches in that the one two nine eight example that I gave before will be successful if they create a market for emergency services professional emergency services staffed with doctors on ambulances that can get to people who call a free number within eight to fifteen minutes.
That doesn't happen right now in India, and if in fact they can do it in Mumbai and prove the case and other governments and pick it up and figure out a way to do it. Another one two nine eight is doing it or not. They have changed the system for emergency medicine. That is does access to emergency medicine end poverty?
I don't think so. So the question is how we think about defining the system is around market based approaches if the marketplace responds times then a system in fact has been created. But I do think that is a tricky question around interventions that have loftier aspirations in some of the problems are global in scope we have to be thinking about how the systems intercontect.
yep. Did you have a thought, Page?
Yep yep yep. Well I do, I have three. One is that you have to have a strategy that's targeting system change. So, and I would caution you to define what you mean by system change. I think we use that, those, we use Such loaded terminology that it's important to just unpack that. What exactly do you mean when you say that?
Because when you go into the measurement world but the more specific you can be the better. So that's the first point is that do you have a strategy that you would targeting what you would consider is system change. The second is there are two things I heard in your question; one is, what kind of measurement is gonna help us to achieve system change that's sort of measurement as intervention, and then there's, what kind of measurement can we use to demonstrate that we've created system change.
those are related but I think also distinct and so I think again just unpacking what is it that you mean because measurement is an intervention and so when we say, you know, we want to measure things and see which models work which approaches work so we can influence a governmental system or bring in markets to influence markets that's using measurement to advocate for change and then there's are indicators that we can be tracking about systems, about health status or poverty status, that actually with the way government works or the way markets are working that actually speak to system change.
In the example of housing, the homeless families, that is a system change strategy and there's lots of measures in there that look at how are we going to impact the system of how resources are allocated, which actors are in place, but being very specific about what we mean is really key.
Yeah, I I would just chime in there and say yes I would agree entirely with that and I think typically the visionaries who imagine a systems change actually have a pretty complicated picture in their mind of, here's why, for instance yours on dignity. What a lot of change the feeling of dignity. I suspect that you have.
There is a whole bunch of things that factor into your vision about that and the same with that sort of self of a children and so I think what Fay talks about is not anywhere close to It's ability, in fact I think it's right in the heads of the, of the visionary, but it's often, in my experience, it's left, left too implicit, but you can say this cause of this, cause of my theory is this causes this which causes this which causes this and you cannot measure these intermediate steps.
What do you think? What struck you as you listened to the three talk about the issues in measurement and impact.
measurement and impact. One or two times I found the word market and that sort of disturbed me a little bit because often the market as seen as the fantasy to end all ills. But, when you look at developing countries there are there certain, there are certain sections of people who are beyond the market.
And unless you have societal systems Items which catered to these people in a special way, and which may be beyond the market. You are not never going to bring any social, my feeling, it's a very strong statement, but I doubt very much if it can bring social justice social equity, gender equity, into the society, that we, I mean into this society and the society we one for the future.
So, while market, I mean, is the answer to very many problems, I think like in India. About twenty percent of the people, I mean India is shinning, we've got, I mean, nine, ten percent growth all that is good that maybe ten to twenty percent of the people at work going down the drain. Because the, the safety basket which was there is slowly disappearing, and its not people should we not devise as a as a society that cares for everyone, as an inclusive society, systems whereby we can also take these people?
How will you measure those? Measuring those in
clusive processes where the people who cannot shout for themselves, who cannot even ask for it, how do we bring in that particular section as a caring society is something that bothers me.
What are your thoughts, Brian on, you'd like to bring market Oriented solutions, what a acumen fund but I suspect you have the same view of, or similar view of, the limits. yeah, I know, I don't disagree with what Joe said. We do think that the market is a very good listening device. If you design with the user The consumer or the individual in mind and they in fact express their preference through purchase if they can.
Then you're going to design. products for them. I think our experience in development is that too many things have been designed on people's behalf in Geneva or Washington or Seattle. And, if they are designed on the ground with their preferences in mind, the solution may in fact be sustainable. the markets, we don't think they are the panacea.
There is a need for charity. I don't think that the markets in Asia or the US have done much to narrow the equity gap, and so those are important issues. But, when you look at the amount of development assistance that is being channeled into south Asia, Africa the lack of accountability, the lack of feedback mechanisms the market might be one complement to some of those flows of funding that would shine a light onto solving problems into a new way.
I hear a real piece of the market that you care about is the user and how the Reacts to services, offers, what have you. Yeah.
Yeah.
Any other thoughts on on Joe's comments from David or Fay. Faith?
Well I think everything that he said is important, I mean that I think what you just said is important. I think hearing from constituents is critical when you are crafting your strategy, as your measuring results, interpreting results. We're trying, we're grappling now with how do we put in place feedback loops so that we can hear from who we consider the ultimate beneficiaries of the work.
Trying to experiment with different feedback loops so that we hear from small holder farmers in Africa, from high school students in America, from the different folks who are are from the female sex workers in India. So, who are our targets so that we both hear from our grantees and our intermediaries.
But also from the end user in a way that is meaningful. So i think it's really important. I don't think we've cracked that code but it certainly a value.
As I understand, David. I'll ask a question, if you want to answer a different, or had a different point, go ahead, but my understanding is your working on mechanisms to hear that user voice more clearly, if I'm understanding keystone properly.
Yeah, actually the point I was going to make fits well with that. I really wanted to just say that the word we haven't heard yet, though it's been very, very close to the surface and what, in the cases that Joe and Jarew have said, is power. And oftentimes in the kind of measurement conversation we tend to leave it out; it's a kind of a. It's part of our thinking, but it isn't as explicit as it needs to be, and we won't really answer into a full learning conversation with our critics, if you will, until we really embrace the kind of classic power question.
So, one way it comes up here is to look at the power of organized voice, in the communities that you're working, among children among high school students. And where, how does that voice, not just how is it expressed in our metrics. But how's it organized and how do we support that organization and self actualization in that sense and so that can become part of the way in which we measure as it goes to framing the problem.
My new theory of change, its a really important part of the story.
I can't help but asking the following question based on the Again, the things the various panelist have said. There is a notion that some things are measurable and some thing are not or are less so, something they are more measurable something are less so, in the case of your, your wonderful organization it's easier to measure the decrease in disease.
Incidents of disease, and days of from disease. That's harder to measure the impact on dignity. In the sense of well being. How hard should we push if that is the biggest pay off? How hard should we push on measuring those things and could be satisfied to measure some things, and leave others in the realm of unmeasured.
Do you have a thought? already I am sorry you it.
I dont think we should leave them unmeasured. Because then it can very well say, well, this is something we taught and we still think it is correct. But, that would the anecdotal or incidental experiences, on off are that an organization will go through. Okay. In this village, this was a response of a particular woman to a bureaucratic coming and asking for a bride?
how does she respond? or the people how they came when they was exploitation of a money lender, and, or a liquor merchant, in that particular village How do they respond to that and how did they eliminate these people? So we can always find They're more important I think for the implementing organization than the funding agencies.
It think if
to show that we are on track That we mean what we say, we are getting where we want to get. This is important. So measure them perhaps in different But measure them none the less. What do you think on you measure, should you measure the impact you have on these kids feeling of self-worth, self-efficacy.
on organized we've been struggling with it, just thinking but I think we are measuring it anecdotally but i think if you want collective change and if you really want to change Mindset. You actually need to push yourself harder to try to find some criteria. So I actually, the first likely with joy, I think, it's important to have criteria and to beyond anecdote.
But it is very, very, very difficult to do that and I know that it's a thing to struggle. Especially for us because we are a network, so trying to get our partners to do it makes much more difficult.
Yes, yep, so you don't have direct access yourself. Hey did you have a thought, thought on that, that question? Are there some things you shouldn't measure and some things you should or?
Well it goes back to what we've been talking about. You should only measure it if you are going to use it. And some things are, they just are harder to measure. Even when you do measure them, it does not mean the measures are any good. So self esteem and those measures are classic examples, when you start taking these standard scales and applying them in different cultures.
If you don't use standard scales, then you have a problem with what exactly are we measuring, and how we know actually that this is a good measure of what it is we're trying to do. But then if you use standard scales, you have a problem of cultural translation of those scales, and you also have a problem, cultural, also in terms of age, we're working with a lot of young people here, there is a lot of controversy on things like self esteem scales.
And actually, is it a good thing for young people to have high self esteem or does it actually mean that they feel more invulnerable than they really should?
So, once you open up those doors to those kinds of issues, it just gets tricky. And then you think about how are we going to use it? And are we just going to question the methods if we don't like the results, or are we actually going to use the data to do something differently? It really goes back to what we've been talking about of how are we actually going to use use it the data.
So it might go to Joe's point which says that's more useful for the agency in question than the funder because they're, closer, maybe, more culturally aware of how to interpret the results.
Maybe. I think it's the same issue, actually, I'm not sure it's that different, because if you get back results that you don't like, or that don't make sense to you, are you going to try measuring it differently? Or will you do things that are different? And I'm not sure it's that, I mean, I think there are differences in terms of what funders look for, maybe and what programs and practitioners are looking for.
In that case I still think it is a question of how much can we trust the measures and how useful are they to us.
Before I go to the audience for questions I would love to know if anybody on the panel, based on the comments, has a question for anybody else on the panel? Do you have one?
I'm anxious to hear from the audience.
All right?
Yeah.
Yeah, yeah. All right. So please, now we have mikes somewhere, presumably. Do we have mikes? Do we, yeah, we do have mikes. Okay. There I have a hand, and then I'll go here, and then here, then here. How's that?
Hi, my name's Onarosa and I'm with the World Bank's developing market place. We are a program that funds concept innovative concept level ideas. My question is going to be about the communication between the funder and the fundee while designing M;E structures. Specifically, when I sit down with the 30 projects that I supervise and I want... and I'm trying to design the M;E structures with them to track from the very early... when we start funding the idea, and the project right when the project starts through the very end, I sense that the organization is sort of defensive of their intervention and they want to give me as little information as possible.
Whereas I'm trying to communicate to them that this is not only a tool for me to understand where the money is going, which is public money in our case, but that this information is really useful for the organization itself. I want to hear from both sides of the house in terms of the organizations as well as the donors what is the best way to structure that framework without... while communicating it to them very effectively that you are not out there to get them or you know, prove that they are failing, but that This is a useful piece of information that needs to embedded in the program itself.
Thank you.
Who's got a thought? Right yea. Sure. Great Andrew.
Experience having done it with other entrepreneurs and past that you can point to, is the first thing I would say. If you can actually show that you've added value that's one thing. The second thing is terms in your term sheets. That require information rights of the investor is another helpful way of just saying look this is grown up capital and you're going to show us your financial statements and we're going to use them to help you.
Alignment of interest is a third thing. For our, our investments. We make sort of quasi- commercial returns if the companies are successful, and if they aren't then we don't achieve our returns and our model is not as effective as we thought it would be. But, you know, it's important to shift the dynamic, I mean, many large funding institutions, public institutions, have a reputation for asking for reports that are not helpful in a business for asking for reports the day before the secretary or the directorate is going to give a speech and, you know, scrambling for results at the last minute.
We all do it. We still do it at Acumen Fund. But making sure that the conversation around metrics is a long-term shared partnership, not a "give me the information tomorrow so I can justify my program for further funding." And that takes a long time of shifting the sort of behavior patterns. But again, showing that you've actually added value in other cases and being able to build it into the economics of the relationship are essential.
And if...there are some people we do not get information from, and we just have given up the fight. And they're capable of managing their business on their own, and that's fine. And we will not sacrifice the relationship on the basis of getting information. But most people now as a matter of course expect it, and it's not...it's no longer a big issue.
Actually, our problem is even more difficult, because we want the information and we don't provide any funds. So all we have is a concept and material and we're saying we want all this information. So everyone can tells us to go and get lost. No that's the truth. They don't tell you that do they?
Why should we do it? But, this is pretty cheeky what we are doing in some of these. What we try to do is instead of trying to say it's a burden, though even I think sometimes it's a pain, to be honest, what we try to do is we've sort of created consultative process around it to say that this is not information that we need, but this is part of all of our learnings and then sharing our publications.
So one of the first things when we're two and a half years old, but within a year and a half we had a publication which sort of shared all that information; so the partners will giving them a deal, sold the publication and they said "Well OK, this makes a bit of sense," because they immediately saw that the data was being used for something And wasn't going somewhere in fifty piles.
Second, which I've done in another organization, is started using the data for them and teaching them how today's funds, so then again they're seeing the use of the data which is coming in, but the third and the most important is seeing the data on how we can change our program strategy. Every person in the field wants to deliver a quality program by and large.
And if you can make them see that the data that you're getting, the information that you're asking is going to qualitatively improve the program. It has for us. Then people will definitely start giving the details slowly and steadily and I think that's what we have tried to do and we have nowhere close to a 100% success but our partners are seeing the need to do that sort of thing.
And we have a fantastic panel of experts so they can take on the partners. we'll go onto, we'll go onto the next one. Orange sweater over here and right behind, beside the orange sweater then right here and then right up there and then I'll do a whole new other one. Oh . We've got another one, we'll do that first.
I have the mic so, You got the, I guess I should probably ask my question.
You have, you have the power, so much for your plan, you have the power. It's all yours. Thank you very much for a very insightful panel. I am Tim Shy [sp?] from International Bridges to Justice and I think you have all spoken about the benefit that the organization can derive from impact measurement.
But, I'm wondering, sort of understanding that were all working in a sort of very resource constrained environment, whats happening in the social capital marketplace to create sort of extra incentives, for entrepreneurs to invest in impact measurement. And what can foundations do, or what can the holders of capital do to really, sort of, create this partnership.
Who's got thoughts on that? David.
This is a very quick response. I mean, I think the first and most important thing to say is that the funders should pay for it and adequately. So, that it's not at the expense of program activities, but the resources are there. Now that requires a conversation about proper costing and benefits and all the rest, but it's an important starting point.
Fay?
Well I would agree with that, but I guess I would challenge the statement that "what's the incentive for social entrepreneurs to measure stuff," because I thought that was actually part of the definition of what it meant to be entrepreneurial is that Is that, you know, are measuring things along the way and course correcting and using data informed practice, and that's what I think of actually as the hallmarks of entrepreneurship is that you're bringing the best of business practices to the sector.
Hopefully bringing the best of non profit practices retaining those and having the best of both worlds, so I would actually say that absolutely measurement is not cost neutral need to do it. But I think of it as part of the ethic of social entrepreneurship.
Yeah, I guess I would think as a funder, as we are in part, I would be slightly nervous about an organization that didn't feel that didn't feel that they would learn from measurement from that's worth any other thoughts here, or shall we go on to orange jacket now.
I'm Catherine Hansen with A-Learn, which is a non profit, educational nonprofit focused on increasing the underrepresented groups going to college several of you mentioned was self-reporting. And one of the things that I'd like you to address to help our thinking on this issue is working with kids who do get into college.
We might know that they got there. We don't know whether they finished. We don't know when they dropped out. We don't know in many cases whether they have remedial courses and how many. So tracking a population that by that definition is mobile and where you don't have a connection like a scholarship to sort of force them to self report and the accuracy of self-reporting.
So I'd like comments on that.
I can start since we invest in that area.
Faye? Yeah. Yeah.
It's a big problem. So we don't have the answer. Like most of the areas that we invest in or many of us do, there isn't good data infrastructure. And so, you know, when you start to think about what happens as these kids graduate from high school, go into community colleges in most cases, just like we don't really have good data across our...and consistent data and a national repository of consistent data for our high schools and what happens in our high schools.
We don't have it in community colleges or college systems either, so we really have to build it. I mean, that's really...the ultimate answer is that. And actually we're making some investments in that area, sort of data infrastructure in community colleges and trying to actually build a infrastructure for that.
But it's very, very hard to track things where there are no data and there's...unless there are incentives built in for young people to come back and report. Which is probably the best way to do it is to provide some incentives, not necessarily for the social entrepreneurs, but for the young people to come back, because we know that actually incentives work for self reporting.
Any other thoughts on that? Brain, did you have a...?
Well, as we think about how we might trust the quality of the data that our peers would put into a shared system that is largely self reported, on the financial side it's easy enough to get audited Financials to correct whatever is self-reported on a quarterly basis, but on a social output basis how many were sold, or how many ambulance rides were taken.
That's a huge question. We've been thinking about whether there are ways of doing, sort of either peer reviews or independent evaluations that are well short of the social audit, but when you think again, when you are counting outputs, the outputs are typically derived from your operating numbers, which are typically sitting on top of your financials and it's not, it's hard to game the systems when it ties back to the economics.
But that is a big question. We haven't figured out how to solve it. What we did in my other organization, Child Helpline, is we had all the helplines to self-report their organizational processes. How they interacted with the children, all of that. What we found is again if you take the basic principal of trust.
Say you are not forcing people to do any anything. But look at it as a self regulation mechanism. People took these processes very seriously. Since how you decide to communicate to the people and make them feel the ownership that this is going to benefit them. And that's what we found. So initially there will be a lot of resistance to reporting, because honestly, if I ask myself, I hate any sort of reporting too.
Who likes school exams? Lord, I don't think anyone here liked exams did we?
Anybody liked exams? No?
No.
You're right, you're right.
Exactly. So if we don't like exams, why are we asking people to have reporting? You know, it's that whole mental connotation which we have. So, why dont you try to say hay you know maybe you do an assignment which will benefit you, because you are going to read so much more, prefer assignments not too many, but you prefer assignments then you sort of move on.
So it's creating the mindset and I think even us having had lots of networks, main thing is creating the mindset for a positive frame of mind, either for impact assessment or self-regulation, and people will automatically start doing it because then you see the benefit for yourself; and that's what, at least I've found, has worked the best.
So we geton this one.
Yeah, go.
This is the whole point about, how do you set the incentives, really, to get people to come back and give you data down the road? It's an example of a larger problem in getting feedback from folks who are the primary constituents. Because, sometimes people don't actually, they may not know why you're asking the question, or they may want to answer in a way that maximizes their advantage.
So they're wondering how to do that. So, they referred to trying to get feed back from farmers, poor small holder farmers in Africa. When you come into that setting and you ask an assessment type question, they're thinking, for the most part, "Okay, how do I answer this to get the best benefit for me?" They're not set in a natural way to answer the question to help the program.
So, the solution to all this is to go right back to the beginning . So at the planning stage to set expectations and to set in the context so that people understand to three, four years from now. We're, we, this program, that you're part of, will depend on how, how you can form us on what happened to you.
So I guess the simple answer to what I am saying in a long winded way is you have to start at the very beginning, at the planning, which is why we argue its so important to involve the primary constituents in actually setting the indicators at the very beginning of what success looks like.
So right beside in the gray jacket, then I'm gonna here, and then back there. Then I'll start a new group.
Hi. Can you hear me? My name is Courtney Bickard. I'm the Balkans Regional Director for SNV, then Netherlands Development Organization. We are a capacity building organization, so we aim to work with our clients to build their own capacity to bring about change in their own societies. So we like to say when change actually occurs that we can't take credit for that change actually occurring.
And I think that's a little bit too convenient and I'd like to move beyond the outputs and processes that measure the number of people trained, assessments that they said "they actually loved working with us" etc. To be able to actually test our assumptions about if we, because we have a shared vision with our clients about what change they're to bring about.
Do you have any suggestions about a sort of intermediary level of metrics where you can start to test your assumptions around if we build the capacity of X institutions, that can lead to Y change in the societies. Anybody wanna take, take a shot at that? Right I would ask what what thee organization you are building the capacity of are doing and then I would say what do they do it time zero and what they do it time plus one, after the intervention of the training, the capacity building, I would begin, we would ask the questions of whether or not There are things that they are currently counting that you could keep counting, where there are things that they are not counting that they should be counting.
And this systems questions about whether or not it really matters. But I would look how they define success. Work backwards from that, and see if there are intermediate variables that you can begin to track. That's what we do with each of our investments, and it's very strait forward and simple, but if you don't know what you are doing last year you can't compare how you're doing this year.
Thoughts, thoughts? David? A growing kind of literature and experience in field around measuring capacity building outcomes, and I can point you toward that, but I am sure you are familiar with a lot of it but you need to have an clear counter theory or logic causal logic in what you're doing, and you would arrive ideally together with your partners.
So, they would agree with you what success looks like to them two, three years out if support from you. You need a initial snapshot of where your at at the beginning, and one of the things I like to look for, is the extent to which two, three four years out people are actually using things that they picked up from your course, and if you do that systematically, track them systematically through surveys across a lot of different training.
You start to see which training seem to be sticking the most and seemed to be most useful over time. It'll help you change your training, and focus in on the things that seem to be most useful. And by the way, you can also, and this is where it gets really exciting I think, use it not as the feedback loop for how good was our training, but rather designing new products.
So you can add, when you're asking people, kind of one of the things that you learned in our training that you are exposed in this training four years ago or you are still doing are relevant to you, you can ask a simple set of question surround what. What do you think would be useful? who would you like to hire?
What skills do you want to bring into your organization now that could come out of our programs tomorrow? In that rein-courses the relationship over time. Because, your alumni are your best designers of your, of the kind of new products.
Or we we flip up to there, and then I'll go to here. Just so we have back and forth. You'll go next. Yes.
Hi there. My name is Audrey Sally. I'm from Rianta capital which has an adventure philanthropy initiative in India, and I've just been thinking as was listening to the panel speak that is from the investor perspective, an investor will set out the values and objectives that they want to see sort of manifest in their portfolio and then you get into the field and you start actually looking at projects and maybe starting to make investments, and you realize that there is this whole host of other indicators and measurements, just idiosyncratic factors that you hadn't thought about.
So if your looking at a portfolio of the projects that you want to have invested in, you find yourself you find yourself not, having to compare not just apples and oranges, but mangoes and kiwis and tomatoes. I mean, it's very difficult to actually Figure out at what point on the continuum of the standard measurements like number of bore wells or crop dams versus all the idiosyncratic types of measurements that come out, like which women in which self-help groups end up getting helped.
So that's my first question, first part of it, is just where on that continuum is it good to be, do you want to continue to try to capture as much specific data as possible, or is it really good to simplify as much as you can? And then, the second part is, there's a group of us in London, coming from different funds and foundations and other venture philanthropy groups, that kind of have these informal jam sessions once a month, trying to talk about these things, and the first one we had was about metrics and measurement and needless to say we didn't really get anywhere so far, but it seems like we all have this dream of coming up with this fantastic dynamic template using ICTna web tools, you know, trying to come up with a way of having us all share this information, especially the idiosyncratic elements because only in aggregate, only if the sample set is huge do they start to make sense and so I'd also like to ask the panel, Is there any utility in that, do you think, and is it worth the effort?
what do you think, Brian, you gotta to that?
So, on the first one, I'll answer quickly that I think this is a practice we need to get better and better at doing it with each investment, and we track far too many indicators for each of our investments. And do not have enough consistent metrics that we think matter across them. But there are a few things, like revenues, jobs, number of people served that we want to try and make sure are the standard basis.
Then for any given intervention, whether it's health clinics, there are a few things that matter. Number of customers, revenues per square foot in the clinic, you know, trained medical professionals, percentage of staff, they're all, a number of things that if you actually really know how to run a clinic, you'd ask as the natural metrics, so anyone who really wants to be in this business should learn what those are, and that includes both investors as well as operators.
And for then for every there's there's special metrics, every one's special, so if there are things that really matter that are unique to the local context, that's fine. Continue to measure them, but don't expect to be able to use those in the context of benchmarking I would love to talk with you offline about the second point around building a shared information platform that allows for both consistent measures across different ventures as well as customization, which is what we've done with our portfolio data management system that we've built with Google.
I think it'd be a broadly enough of interest.
Sure.
maybe just spend a couple minutes of what you've done thus far and are planning to do on. Sure.
On that. Yeah.
So, the Acumen Fund was founded with the ethic of metrics transparency accountability at its core, and about two or three years ago, we recognized as we opened up our, our, our first offices in India, Pakistan, Kenya where we now have more of our portfolio staff then in New York we needed something other than the email, Microsoft excel thing.
You know? All saved on the server, to enable consistent performance tracking across a global portfolio, and so we asked ourselves, well geez, there's got to be some tool out there. Adventure capitalists have tools, but really VC's care about financial returns they don't care about job creation or social impact of the companies they invest in, so there isn't anything that includes both financial as well as operating in social metrics that you can get off the shelf.
We were looking for a soup-to-nuts solutions from the first time we meet the entrepreneur to the end of the life cycle of the relationship, but guess what sales force has already built the worlds best customer relationship management system, and they give it for free to non-profits. So, we said terrific, why don't we use that to the point where me make an investment?
But there's still this question of how we manage the portfolio, collect store the information add qualitative reporting to the quantitative reporting, measure against targets on a quarterly basis So, there was nothing there. And we have a support from Google institutionally, and there was this vague offer of help from their engineers, and we're co-located in their building in New York.
And we appealed to a few engineers who spend their day working on search infrastructure and now spend their nights helping us design a system that allows us to track our portfolio. We have been ruthlessly self-interested in building this system. I want it to help me and my colleagues do our job better.
There are lots of discussions in the sector about, you know, grand multi-party collaborations that sort of collapse under their own weight. And so, all I wanted to do was make my job better.
Over the course of building this tool, using one of our staff members and volunteer engineers from Google, we recognize that we think we built the best inbreed software solution for people who do social investment. There aren't too many people doing social investment, so this isn't some, you know, go-raise -capital investible buisness idea, but its something that might in fact be worth sharing.
So Google, as it was setting up its own portfolio management processes asked, "Can we use the tool?" And we said, "Sure." And so we've been adapting the tool for their use. And then Skoll found out about it and they said, "Well, can we use the tool?" And we said "Sure", so we just ran a beta test of the tool to see if, in fact, others would want to use it.
And the conclusion was, sure, there are lots of people who want to use this tool. We're not a software company. Google doesn't sell software services, so we are now at this inflection point to figuring out what we do with this asset. If we get enough critical mass using the tool, and we've had 15 beta testers and about 50 people taking a look, including groups like Root Capital and E+Co and some of our peers in the sector in the States.
IFC's looked at it, Skoll, the DOB foundation. You might be able create a data infrastructure that has standard metrics. There are a whole host of problems around creating what we call the social venture one. Around self-reporting, data quality, definitions, GAAP as tightly defined as generally accepted accounting principles are, there's still a lot of wiggle room, and if you try and translate that to, you know, generally accepted social performance indicators, it's a total mess, and we've learned through experience that you have to be very careful at how you define, you know, number of clinics open.
Is that the number of clinics you have open at the end of 2007, or was that the number of clinics you opened in 2007? And if you don't know what you're actually entering, then all the data's a complete mess.
So we're at the point right now of trying to figure out whether this a tool that would be valuable for the sector more generally and Google is committing either engineering resources, or cash, to help us figure out what it would take. And then we need to figure out who, ultimately, would own this asset because Acumen Fund is, again, not a software business and can't be both judge and jury in a tool that the sector might use.
So we have to find a third-party home. We talked to GuideStar about this this morning. There's a whole bunch of other folks who might be partners in this. We'd be happy, again, to give you more details after this session.
Perfect, Drew, did you have a thought on this one?
I just have this story.
Good. I like stories.
So, basically we like, in fact, measurement, and we said, you know, we have to capture everything. So I really am enthusiastic. Team came up with almost fourteen pages and more than two hundred something impact indicators when we started looking at it, and then when we To our partners, "Are you mad or are you stupid?" And I'm saying this because sometimes you get carried way, in trying to measure impact from a distance that everyone was important, because like you, we wanted to capture each and Every small detail, for each and every target we were working with.
And it was really impossible. So, on that, we really had to shrink to saying what other mains.. Seventeen. What are things we can measure globally? What are the things we can measure locally? So if we only went from going all over the world, and having a nice dream imagination to being extremely focus to the seven things or the five things which link to our program.
And then our effectiveness, and I think that's important that we shouldn't get too carried away also that sometimes the measurement is more important though. you make it more important than actually the program, because then it will fit. Perfect, alright, oh, yes, yes, please go ahead. just going back to purpose and why are you measuring and getting very clear on how you are going to use the data.
So, I'm being fairly specific. Because specificity is measurements friend. The more specific and clear you can be be, the better data you will get and the more useful the data will be. So, I would just caution about that. I just, I'm feeling a little something in my stomach, so I want to just actually say something the term 'impact'.
Sure. Because we use the term 'impact' to mean a lot of different things colloquially. But in the measurement field, impact evaluation, actually, has a particular meaning. And so, so It's fine to use it in conversation. You can do whatever you want, but just as we're entering the I just think it's important to be very clear, again, about what is the measurement that we're doing and why, and that to Brian's point earlier, when you're measuring against milestones.
That's one thing when you're measuring outputs, that is, the things that are happening along the way. That's another thing. When you're measuring outcomes, what are the changes we're observing. That's another thing. 'Impact' actually has a technical meaning in the evaluation world, which is the World Bank definition, but it is also the the traditional definition of impact assessment is: you know, what is the difference that this intervention or set of interventions is making.
It is about attribution. And so, that involves some kind of counter factual, in saying, if this intervention didn't exist, we wouldn't have this change. Or, because of this intervention, we have this change. Now, I'm not a true believer, thinking that you need to always measure impact in that way, with counter factuals.
But it is important to know the difference. So that when you're out there talking about what are we measuring and how are we measuring progress, versus is this model causing this change, because my investors want to actually know that. That's just a different order of magnitude in terms of the rigor and the certainty and and the responsibility in terms of dotting your i's and crossing your t's.
So it serves a different purpose. So, it's not that one is necessarily better than another. There it goes to purpose. Just to be very clear as to what you are measuring and why.
Yep, nope, it makes good sense. Patience Chuting.
Thank You and I'm actually on the board of a social innovation and investment group and so I'm, sort of, really coming from the, it's like, point of governance, I suppose. Well, thank you very much for your very erudite and very specific sort of advice to everyone. But I just want to perhaps get more into the general nature of things.
And I thought that it's important to talk about culture and Perhaps even language, and it is quite interesting the thing about markets for example, and the different, you know, the definition we have of markets, because some people can take offense at markets, and, but but of course we know that if you actually do want to make a change and want to help with finances, a social entrepreneur.
There is also a relationship. And so, I'm just asking about intuition with these things, you know, instead of this very specific in measurement, and also, I would like to know how many for example of you, specially those who are founders who actually go into the field and what kind of relationships that you actually have with the people fund and perhaps people are coming from two sides there, so perhaps if I can get the answers from people who fund it as well.
So, what is intuition role of that in your work and the other is is getting close to what's going on in the field. Intuition!
I love intuition. I do most of my stuff intuitively. And then I get rapped on my knuckles and I'm told by my team, "that's not enough! Let's get scientific, and let's have things put into, you know but I think, honestly, everyone must have read the book "Blink," and you do make more positions in the first few seconds, but having said that intuition is very important, but you also so we need to be looking at what is causing the change, what are the factors that you want to change.
So there has to be some amount of scientific evidence to back changes Back systems to look at what you want to do, so there has to be a mix of both. You cannot really rely on only intuition, and this is coming from a person whose basic initial premise is to do everything intuition. If you don't have that you can not systematically create... institutional systems change.
It's then personality lead, individual lead, as compared to something which is long term. So I think that would be my thing, and that applies across all cultures, everything.
Brian, do you have a thought?
No I agree with that. I was going to actually disagree earlier with Fay, in terms of the point you made about enterprise, social enterprise, and data being baked into it. Actually, most of the entrepreneurs I know are highly intuitive, but intuition in my experience, doesn't scale very well. As Jurie just said.
And there are cognitive biases and if you think about all the successful business people that are risk takers and think by their think on their feet. There are many failed business entrepreneurs who were risk takers who think on their feet, and so the characteristics of both successful entrepreneurs and failed entrepreneurs, and there's a lot of literature about cognitive psychology of how you make decisions under uncertainty, but I think intuition is critically importance.
And for us, part of the next step of that is pattern recognition, but recognizing the pattern involves writing things down and seeing them, whether it's pictures or whether it's data or whether it's other feedback or input, but understanding what those patterns are is critical. For us, the relationships are essential in the field.
We are a relationship business, and we now have 15 people on our portfolio team in India, Pakistan, India or Kenya, and five in New York. When I joined Acumen Fund four years ago, it was four people New York and none,in Indiapaks Center, Kenya. And so the relationships matter. We are the boards where we take equity positions.
We sometimes ask for board observers rights for companies where we loan money. We really take a very active relationship. We've got Acumen Fund fellows who are a highly selective group of folks seven a year who spend a year on the ground with invests, essentially working for them in Seconment on their top management challenges.
So we have a very hands-on relationship, and believe that only the learning will happen by rolling up our sleeves and working side by side with the entrepreneur. Yeah, if I can just chime in on intuition. Intuition is very much a catch-all word that is used for a lot of things, but a piece of what people generally refer to as intuition is, I think, critical, and there's an argument to made that the kind of thinking that leads to new ideas It's what a lot of people refer to as intuition, it's called abductive logic,which is the inference to the best explanation.
And that's what the best entrepreneurs do. They say, like Joe says, I look at this villages, and my best explanation, I can't prove it, but my best explanation for why what's happening is happening is the following. And I'm going to intervene in the following way. I don't have proof that I'll reduce sickness and increase dignity but I think it will.
And then you go and do it. And, sometimes it fails, but other times it succeeds, and then and then you can, as you get experience with it, you can use more traditional analytical methodologies for evaluating progress, and the like. leap. It's an abductive leap of logic. What a guy named Charles Sanford Peirce called inference to the best explanation.
And he argues that that's the only way new ideas come into being. And the major tool-box of logic for us. Supposedly educated people are inductive and deductive logic, neither of which has ever produced a new idea in history. That's his argument. intuition is important, but not, to Brian's point, not sufficient.
And Gerry's point. critical but not sufficient to grow organizations. Does that make some sense? me there is a stage even before intuition that is gut reaction. I mean, if there's an unjust situation, like I said in the beginning, that a woman spends 6 hours drawing water for the family. It's a gut reaction this is not needed in today's world.
We have the technology, we have the means, to eliminate this unnecessary. So it's a gut reaction to something, an injustice that is taking place. Then comes an intuitionintution Oh God, I think I can do this this that is an intuition but then you muddle over that intuition, you sleep over it, you discuss it, you have a great conversation with as many people, especially people who will not agree with you and then you come to a plan as to how to put your gut reaction, your enthusiasm, into a program activity which you think has a ghost of a chance to succeed, and then I go on.
So for me, three stages.
Excellent One last question right at the back, and then I'll do a little bit of a summary. I'm Maria Barelias, and I come from Hand in Hand. Thank you very much for a very interesting discussion that sets off a lot of thoughts. We are a charity that works with entrepreneurial self-help solutions in south India.
in Afghanistan, in South Africa, now going to Brazil and China. And we are very interested both in intuition and metrics, and We don't think there is any contradiction between the two. They support each other very well. But, we have a problem with our donors. Not the private donors, not the corporate donors, but governments.
Governments are very interested in measuring, but they love to measure methods. We love to measure goals. So we have a problem in educating governmental donors and government-related donors in doing the right type of metrics. They can spend 50 pages of application on what type of trainers, which addresses, where do they come from.
We want to know talk what's happened to the women who got training, did it help them? They want to spend hours on training people in special courses. If you are going to apply for grants from the DFID in Britain you have to go to special course just to fill in the papers, so how can we as charities educate the donors to focus on the right things?
That's my question.
Oh, it's a huge challenge No, I could take a shot at this is near and dear to my heart because it is the score joke of government regulation practice which is which is input focused rather then output focused. And if you want I've written a couple of things on that. But I think it's a long tough battle because that is the mindset, set up a bunch of inputs and I'll give you the Canadian example that I rail against.
We want Canadians to watch Canadian TV, Canadian-produced TV for some odd reason, so we regulate that fifty percent of prime time television programming has to be Canadian produced. That Canadian programming gets ten percent of the viewership, so by regulating the input, number of hours of prime time, we get a pathetic output, and if we simply regulated the output, told the Canadian broadcasters you have to produce twelve percent viewership.
We would get, and by the way, we get fifty percent, because they get no viewership, they spend no money on the shows, right because they are not going to make any money on them, so we get incredibly bad shows that nobody wants to watch. If you just said you have to get twelve percent viewership, I don't care how you do it, they'd make a few great shows and we'd have great exportable shows and more viewership.
So, it's a scourge and I think it just has to be battled on every street corner.
If you think about measurement, this is very crude but. Having three purposes: of accountability, informing practice and informing the field. Government tends to be more about accountability. And just makes it hard because it's not about learning. It's not necessarily generative. It's not all the things that we're all talking about measurements should be.
It's a good fight to fight. I'm not sure that I have the answer. There was a microphone already in somebody's hand, and so i just have to do that, so this will be the last question, i promise.
It's actually not a question, so it's probably good the end anyway.
I only want questions. If it's not a question, i don't want it.
I'd just like to inform people about the work that's been going on in [xx] over the last ten xx which i think would actually be valuable to people, so if you would mind. Ok, be quick. Basically the work started initially looking at impact assessment, realized that impact assessment was not I mean not useful, not feeding to practice, and there is a lot of work then on trying to have practitioner friendly impact assessments.
So a lot of practitioner based tools. This was funded by USAID program. It then moved on to really thinking about not impact as an end result but the process by which you get to impact. And really a movement away from saying we should be expecting each organization to be assessed In their impact, but thinking about what is the process by which an organization manages its performance towards that impact.
So very much we've moved towards social performance management. And the measurement comes as something internally useful to organizations that then can be externally validated, externally but really the focus is on what are the steps to get there rather than just measuring the output. As I said, this is something people have been working on this for the last ten years in micro financing There's a lot of resources available.
If anyone wants to talk to me after this, I have a lot of resources. There's also the Seagap Micro Finance Gateway has a resource center called Social Performance, and there's a huge amount of resources there, and I would encourage people who are getting in to at least look at that 10 years of experience.
Terrific, thank you. Listen, I'm going to try and summarize what I heard panel what I've missed. But, I just scroll things down as I listen. And I think that I've heard that there are three important "P's." First is positives as in positive sum. But I heard a strong view that measurement and metrics have to add value, because it will cost so they have to add value.
We need to measure things that will actually help the business invest and be better and if our measurements don't help the business better they will probably be zero sum or negative sum, and then we've got to make sure that the businesses in question use the data and the number one use of the data is learning getting better, and that's the only way, at least as I hear it, the only way to make impact measurement, a positive sum gain.
Second "P" I guess would be precision, with this notion that we have to have the logic kind of worked out. Measurement can only follow a clear strategy. If you don't have a clear strategy we're not gonna be measuring the right thing. I mean, to have to have the problem framed properly and that we have to be able to unpack the logic and understand the logical chain so that we can measure intermediate outputs, as well as the end output.
And we can't have that unless we have certain precision in the thinking and the logic. Recognizing, of course, that the first leap maybe a gut feeling, intuition. But then that has to be followed up with evermore precision there.
The last one is pluralist. I came away feeling that we really have to take a pluralist outlook on most of these things. Some of the the measurements are going to be investee-defined because it is about their business. But, some are going to be investor-defined because that is what they've got to have to be able to get their funders to be continuing to be willing to fund it.
They have to be pluralist and sensible measures somethings and other things we won't be able to measure or be able to measure in, in different ways. If we think we can only measure the same things in all the same ways, we're going to get crummy measurement systems. We have to be pluralist in that there are multiple constituencies that we'll have to think about measuring on behalf of not just one.
Yeah, and you have to understand what the users care about, but also what the donors care about, etc. So, I heard it, I heard those three messages, positive some, precision in the thinking logic, and, and pluralist. Did I miss something important, panel members. No? That works for you Drew? Okay, it works for Drew so, it works for me.
Thank you very much for being here. And take care.
How many of you can empathize with a women who has to get up before the break of dawn and under the cover of darkness has to find a stealthy place where she can defecate. And then, even if she has diarrhea she has to manage to keep it within herself so that there has to be, so that the under the darkness of the night, she can do the same thing.
I'm speaking about what is happening in most developing countries in the twenty-first century. Is this needed? We speak about the Internet and the huge strides we have made? Does a woman has to spend, I mean, forty percent of her time hauling water? Do we need to keep girl children and other children from school because they have to also assist their mothers?
unfortunately, this the truth. This is the truth about my country. many of the countries in Asia and probably in Africa. Four percent Of the rural how source in India have toilets one person have water on tap in the village, one tap in the village. All the others do not have access to save water.
80% of all mobility this is in rural area is caused because of poor quality water. And when we see as to the reason, it is our abysmal Human attitude the disposal of human waste, human waste in raw form find six way back to drinking and x bathing water and you have all sorts of diseases and that accounts for eighty percent of all diseases.
This is not my statistics, it is W.H.O. UNICEF statistics. So in theory, and more than that in practice, eighty percent of the diseases can be eliminated in rural areas in countries like China, India, Countries of Africa, Countries in South East Asia, maybe Countries in South America. If good quality water is delivered in the village, but for this we need sanitation and sanitation on hundred percent basis in an any habitation because even in you cover ninety percent or ninety-five percent or ninety-nine percent of a habitation with sanitation, but this one family shits all over the place, sorry for the language but, what do you have?
you have that recycling of that one family shit all over, I mean in the drinking and bathing water, you are not really improving the health status, so we need to have a 100 percent coverage of sanitation. Another problem we came up in India in this area where I work Orissa is that. Women and men bathe in the same bathing tank or pond women being circumspect as to how much they expose themselves had insufficient cleaning of themselves.
They got fungal and other gynecological problems, which they could not afford to treat and once got, usually kept forever or if got treated, what infected. So if a family would have a toilet, a good toilet, and a bathing room, and Water supply, to use that, and, then the lives of women especially would go would be better?
Girl children would be able to go to school. All the children would be in school. Mothers would have much more dying for their children. They were not being lazy because they have never been lazy. They would engage themselves into other productive means and have a little more leisure for themselves.
So, this is our program. Cover it 100% of habitations with a toilet, a shower and protect water supply, which would be, would be through gravity or some, or would come into a water tower and then Would we distribute to on 3 tabs to every family. One in the toilet, one in the shower and one in the kitchen.
no women after this would walk more then 5 steps to get her water. Well So, what are the impacts that we want to see? Well we have a mechanism right from the very beginning that even before the, we start the program we begin to see what was the existing condition. Every family how much could] they spent on health and curing of deceases especially diarrhea incidents Jaundice.
gastroenteritis and other, cholera, typhoid. Every family, how much they have spent. This may not mean we'll get very good But it takes about two years to implement this program, where people bring all the local materials and externally only the Cement Steal and whatever constructional is given some times no is the dangerous guy.
So people would, and build for themselves the toilet and bathing room and then initially make an endowment which they bring themselves, the interest from this endowment would see that the social cost or external materials are available to future families who come into this village and all without a single exception implemented program.
They pay for it. They maintain it and it's on a hundred percent level. So from the very beginning, we begin to see what was the disease pattern before the two years we implement the program. What is a disease, how much mobility, mortality in the village, and then after safe water and sanitation comes in, and hygiene education, how many children go to school, how many girl children go to school, how many water born diseases every month it is monitored the that man comes to the particular district or the region.
If there is a flashpoint it is immediately acted upon, it comes back to our headquarters Gram Vikas village office, so we monitor it. There are other things we say. We are not building toilets, and water supply. We are building the dignity of the people. We have not yet put a price on dignity, but I think we have to put a price on social inclusion where 100% of the people, especially in India, it's very important; it's cost-driven, that all come under the same platform and implement the program.
We are to put a price on dignity; we need people--I mean we are the brawn, we need brains to put value to that--but just on diseases and the savings that these people make, it is 1:10. For every dollar spent on water and sanitation, safe water and sanitation, we get $9 in return in one year. What greater investments can we make?
What greater impact can we make? What greater empathy can you all have with my sisters and brothers in these developing countries? And we are looking at this impact. Thank you. Thank you very much.
Jeru, and as I've said to the panel owners, "You can sit here, or go to the podium, whichever you're more comfortable with."
Sitting and talking is fine.
The informally, if that's okay with all of you?
And that is fine.
Then why do they stand?
Because you are To get a stretch. Okay, sorry. I'll just talk about and why we think impact assessment is necessary. A flatoon is a name which actually came from children, and a flatoon is a fireball which goes all around the world telling children about their rights, their responsibilities talking to them about how to saving, not just money, but the environment, everything around them; how to plan and budget and also teaching them how to run micro the prices if they want to go for class picnic instead of x chickens and decided that the eggs from the chickens can help them go for a picnic.
Or they've created key chains and things, and they've sold it and they Made money and some of them help their friends Saflatoon is basic teaching children in the age group of 6-14 about them money, rights and responsibilities. Now it sounds really interesting to all of you, but what most of us tell ourselves that's rubbish.
And that's where we come in and that's what in fact is important to us. Most people tell us no way can children handle money, no way are children going be able to do micro-enterprises or planning and budgeting. That doesn't work for children, so for us, impact assessment becomes actually a real a sense to look at proving our concept, proving our belief in children that if you invest in children, you'll actually be making every child, an entrepreneur, social entrepreneur, just a caring human being, and probably helping get them out of the cycle of poverty.
So we've looked at impact assessment and why this session and this topic really interests us. What is it that your trying to measure? How do you go about measuring it, what are the obstacles to measurement, and what do you need to convince policy makers to take the sleep on faith, the sleep of faith because ultimately there are a billion children in the world, who if they had social and finical education, would definitely feel more empowered.
and would probably be able to get out of the cycle of poverty. So that's what we are looking at and those are our challenges. And that's what I'm hoping to look out for in the session and get. And also that is what we are talking about so keeping to the three minutes, which you had said and moving over to the right.
Thank you. Thank you, Brian, over to you. thank you.
Autopsy is to cadaver as evaluation is to most non-profit programs. They were expensive, rare, invasive, backward looking, and often too late to help. There's a big difference though, non-profits nerve die or fail. We have a cultural aversion to looking at failure. What we've been trying to do at Acumen Fund is think of ways to develop measures of a business or enterprise's pulse, its blood pressure, its temperature.
Simple, easily implementable things that will allow you to course correct in course of implementation. I think Jeff said the other night in the opening plenary that the social enterprise space is lacking one thing, time and I would add to that, in addition to time, we're lacking evidence. And it's our hope to continue to lead discussion in the sector about what kind of evidence we can collect from each of the enterprises to prove the case that social enterprise is in fact a better mouse trap.
Let me spend a couple of minutes talking about how we think about impact assessment from an investors prospective. Acumen Fund is a social investor. We've invested about thirty-five million dollars in health, water, housing, and energy companies for-profits and non-profits mostly in South Asia and East Africa.
We act as a venture capitalist at a very early stage of a company's growth, and we try and work with them over the life cycle of our investment, often five to seven years to help them solve a range of operational and business challenges. When we make the investment, we start the metrics or impact assessment question by asking ourselves two questions.
One does the intervention matter, and what evidence do we have that it does, and so we'll often look a the range of scientific literature about whether or not anti-malaria bed nets help reduce morbidity and mortality from malaria when properly implemented. And then we'll ask ourselves the question, who's the best charity out there doing this work, and could our donor achieve more output?
Careful not to talk about outcomes or impact but more output more bed nets distributed per dollar of philanthropy than Save the Children or PSI, and we hold ourselves to the standard of trying to make sure that we are more effective per dollar of output. It's not always the case, we have failures, and we talk about them, mostly internally.
But the social capital marketplace does exist, it's just very inefficient and fragmented. Once we've decided that there's an investment that makes sense, and that there's evidence that the intervention is valuable, we turn to the entrepreneur and help them think through what matrix will matter for their business.
We have a mantra that we do not want to measure anything that does not help the business succeed. We also do not want to measure anything that the business cannot generate from its existing information systems. If the business can not measure things that are important for it's business, from it's existing business systems, then we'll help.
We haven't gotten in many insistence's, but there are times times when we are able to find a few key metrics that will help the businesses identify things that are really important levers. Let me give you a quick example of an ambulance company in Mumbai that we've invested in. 1298 is the first private sector ambulance company in India and it is trying to scale to serve all of the Mumbai metropolitan one region with seventy ambulances.
Right now there are about a couple hundred ambulances in Mumbai. Most of the serve as hearses for all those cadavers. and there's autopsy's. There motto is to in fact create an emergency response system. They're really trying to build an industry and they've got a cross subsidy model. They want to have ambulance access for all as their motto.
And that depends largely on their ability to serve both high-end customers who are going to the private hospitals as well as low-end customers who can't afford a ride to the government hospital. When we talk to them about what metrics matter. We think about revenues, end costs, and that income, those are the obvious ones.
Numbers of ambulance, that's also obviously they are trying to go from 10 to 70 in the next couple of years numbers of call per day per ambulance actually is the key number to where not that there going to make it and whether they'll be profitable. Number of free rides is a percentage of total rides, is the other number.
If they hit the two numbers on number of rides per day and percentage of free rides around twenty-five or thirty percent. Then we think that the model can scale and it can have the kind of social impact that we intended when we made the investment. Those aren't necessarily easy things to track, but we've made it out job to track them on a quarterly basis against plan and when we have the quarterly numbers coming We don't just sort of say, "Great,t hey're succeeding or not." We sit down with the management team and ask them, what's working?
What's not working? Why? What can we do to help? It's not punitive, but it's collaborative to try and identify solutions that will help the business grow. We've been working with some of our colleagues at Google, the Skoll Foundation has been supporting us and a number of our peers in the sector to think about whether or not there's a way for us to take the information management tool that we as an investor have been using for portfolio management, to create a platform for others in the sector, so that when we think about the work at Rimvydas and how much they're able to do with water and sanitation per dollar of investment.
We can compare that to some of our water and sanitation investments, and facilitate the learning across organizations and we're hoping that over the course of the next couple of years we can begin to aggregate and benchmark information and I would go back to what Roger said at the opening; not everything that is important is going to be measured but if you aren't measuring some of these things, you won't be able to learn, adapt and grow the business.
I'll close with a quote from Oliver Wendell Homes which talks about the kind of insight we would like to drive from our own measurement system at Acumen Fund as well as what we'd like to see the field arrive at. I wouldn't give a fig for the simplicity this side of complexity. But it would give my life for the simplicity on the far side of complexity.
And what were hoping to do is identify a way to measure that recognizes the nuance and difficulties of the businesses that we support, but abstracts it to a few simple things that they can actually implement and that will help them make better decisions and approve the lives of the millions of people that they aim to serve.
Thank you.
Thank you Brian and on to you David.
Thank you, good afternoon everyone. Sometimes I feel, Brian, like I'm a bit of one of those complicating factors that thorn in your side as I'm tending to bring in more complicating issues. Very briefly, in the short time that I have I'd like to not discuss some things and take them for granted. One is the kind of complexity and difficulty that it is in trying to measure real impact Issues around attribution, counteractions, organizational versus public domain information and how data moves out and becomes available and many many other factors.
We've heard bits and pieces of those from the cases that Joe shared with us. I'm also not going to talk about some of what I think are the critical trends in the world that are bringing us closer to solutions of some of these problems. The rise and growth of more professional management in the social change domain under stepping up and taking impact measurement in very constructive and learning ways and I think Brian has given us just a great example.
An apple of one thunder that's doing that. Maturing methodologies of evaluation and pluralism in different approaches, mixing and matching different approaches. measurement and evaluation that are being used better, and new communication technologies which help us. These are all big topics and we don't have time to get into them I just want to throw them out as things I'm not going to discuss.
And would rather jump straight to the collisions of our work in the last few years, which has been focused on really trying to answer the question "What do we really want to measure and communicate in order to understand what sustains social change and sustainable development?" And the starting point in working with lots of organizations and looking across at literature and doing lots of net working over the last few years is that both monitoring and evaluation system generally are week.
I'm sure there are exception abounding in this room. But there's a growing awareness of that, a really openistic kind of fixing the problem. We did a survey last year, an online survey, and 85 percent of the NGO's that answered the survey around the world, said they really were seeking to improve their monitoring and evaluation.
And I think the kind of turnout we've had today as another example of the interest in this space. Another big take away from us, and I think it flows right out of what you've heard so far this morning is that you kind of, as we put it, you need a whole system solution that you can't look at measurement only from the point of view of one of those actors in the system who are using the data.
So you don't just look at it from the point of view of donors or management, but also of boards of organizations and, most importantly from our perspective, of those who are meant to be benefiting from the work and how are they involved in actually determining the metrics. And we use the, we ask the question, who counts?
And we meet it in both senses of the term: who's doing the counting, and who matters.
And finally, that the kind of critical overall objective behind all of this has to be learning at points of every different stakeholder that's involved. And our theory of change at Keystone is that society solved the important complex problems when they learned how to solve them.
So we went on from these kinds of takeaways to try to develop a kind of theory of public reporting for social and environmental outcomes that honored those thoughts and others. And in doing that, we're really trying to bear in mind that, that kind of, whatever gets measured gets used. And that that's the critical metric for all evaluation efforts.
If you are going to evaluate the evaluation, the bottom line is, is it being used and how well is it being used? So along the way I've been collecting metrics and looking at indicators, and kind of trying to say what's the best metric that anybody's got that I've found. My current favorite metric comes from Alcoholics Anonymous in the United States.
I don't know how many of you are familiar with this, but it's a really powerful metric, which is days of sobriety. And they measure it at the level at the individual addict, but also at the level of chapters and regions. And it's measured quite carefully. And what's I think particularly powerful about that metric is that it begins with the person who's meant to benefit.
It's something that's generated by that individual in a context of the organization's work and then flows up from there.
Which brings us to our core work. Which has come to be about in all the different things that are happening, and all the different things that are important around measuring and communicating social change We've decided to focus in on this issue of constituency feedback. And making sure that organizations have ready tools and ways to capture feedback and systematically incorporate it into their work and into their public reporting.
So we just want to close by telling you an example of a current pilot project that we're engaged with in the United States with the Alliance for Children and Families, which is a network of three hundred and fifty children and family service organizations. That do both service delivery and advocacy work for poor families in America.
And of the Alliance brought together a group of its members with a consortium of interested funders to talk through the possibility of doing a pilot project where we developed a common questionnaire that would be administered to their primary constituents. In order to get feedback, both about how the constituents felt about that organization but that would also enable us to give some comparitive reference and a kind of ranking with respect to the others in the group.
The value of the comparative dimension is potentially quite high because it allows an organization to see not like if we're using a seven point Likert scale on the questions, they, you might think initially that a 5.8 is a pretty good answer, a pretty good score, and that we're okay on this dimension or this cluster of questions, how responsive we are to our constituents.
But then you see that, actually, 5.8 is only fifty-two percent in the cohort learning conversation in the peer group, in the cohort. So you can start to exchange experience around objective data about what seems to be working for you and how could we try that technique and maybe that would work for us.
So you use it not in a sense as a kind of competitive table, who's on top, but rather as the basis for learning and exchange. So, finally then, we are saying that kind of key missing piece in a lot of evaluation work, which is surrounding systematic or around constituency feedback, that when you take it seriously, you can get evidence of your impact - not always direct evidence, but evidence of progress.
You can certainly get insights into your performance. You increase the voice and stake of your constituents in your organization, which is absolutely critical. This is the "I've been so for 15 days." You all may be thinking at this point, "This guy hasn't been sober in a week." And also, when put into the public domain, this data really establishes your accountability to externals because you're showing the way in which you're listening and responding to your primary beneficiaries.
Thank you.
Thank you very much, David. Last, but certainly not least, Faye.
So in...can you hear me? In 1989, the Centers for Disease Control made it The first grants to community based organizations to do HIV prevention work. And I was a young, kid of my twenties at the time. And I had just joined a non-profit consulting firm. And my first assignment was to be the project director, the Principal investigator on the evaluation for a community based HIV prevention program.
I was so excited. I went out to the organization, a good organization in San Francisco, that was very excited to have this big grant from the Centers of Disease Control. and doing some prevention work with high-risk young people in San Francisco, run-away kids and high-risk young people. And at the project launch meeting, I met with a project director who was an incredible visionary, a guy by the name of Danny Keenan we had a Meeting and talking about evaluation.
After the meeting we had a submeeting and I said, "Danny, you know, I am here to help you. I want to figure out what the best methods are." on how we can really learn from this. And he turned to me and he said, "Faye, I don't give a shit about evaluation. You do what you need to do and stay out of my way because I'm here to save lives." I was 25 and very eager and I was quite deflated.
I'm gonna...I only have four minutes left, so I'm gonna spare you the rest of the story. But I tell you that story because the lesson that Danny taught me in that moment was that evaluation always had to add value; that my work needed to be as important and contribute to his work; that the measurement needed to add value to those who are working to save lives.
And that's...if you take nothing else away from what I'm going to say, that is sort of the most important principle, that the measurement always needs to add value.
Danny and I went on to have a great relationship, and he ended up using lots of data involving in the data collection, and we did really good measurement, and he used that data to help spawn the development of really great psycho-social and medical clinic for young people that went on to do even further great work, which we further evaluated and continued to learn from.
So, it had to add value, and it did and I always remember Danny's lesson in my everyday work. I want to make five points in next three and a half minutes. One is, I think good measurement starts with good strategy. Good strategy starts with framing a problem clearly. Framing the problem, that's point number one.
Tell you a quick story about the value of that. In the Gates Foundation in our work, in our local work in the pacific northwest we have three different program areas that will help build development in the U.S. program. In the U.S. program we have our local Local giving which is in the pacific northwest and in that giving program we have a strategic program called, 'Sound Families' and its a program devoted to addressing the issues related to family homelessness.
We had a seven year initiative from 2000 to 2007 that was focused on family homelessness in the three county region. And we had a goal, a measurable goal, fifteen hundred units that we were gonna build. Bricks and mortar strategy. We were gonna build 1500 units that combined housing with services, so it was sort of the state of the art thinking was that's you needed to do to really address the issue of homelessness was combine housing and services, and help homeless folks transition off of the streets, give them a period of time to stabilize and move into homes.
Seven years later, we did a formative evaluation. That was before my time. We had hit that mark, we had hit 1500 units and 66 percent of the the families were stable, their incomes had gone up a little bit, and some of the kids were doing a lot better. We had lots of good data to speak to what had happened.
But we actually didn't solve the problem of family homelessness in the Pacific northwest. And in fact, during that time, the family, the problem of family homelessness got worse. And the problem of services got worse. So we never actually set out to solve the problem we set out to do what? Build 1500 units.
And we did that. So actually, it was a successful initiative. But now, was we began to this past year work on looking at "What is our strategy go to be of family homelessness?" We're now re-framing the problem to not be about actually we didn't actually frame the problem last time. We just said we're gonna build 1500 units.
And now, we're framing the problem to say we actually are gonna try and eliminate family homelessness. We're gonna, I'm not gonna go through the whole strategy, but it's a prevention method, a rapid rehousing method, and a whole set of investments in infrastructure We're going to learn a lot, we may fail its a risky strategy.
The point is that we're framing the problem differently and all of our measurement is now related to that framed problem. And what can we learn about how to actually prevent family homelessness that can be useful, both in the Puget Sound region, but also Nationwide, the second point, I'm going to try to pick up the pace, is the benefit of knowing must be worth the cost of asking, we were talking about this in preparing our talk about how expensive it is to do measurement.
Impact measurement in the traditional sense, of the term with counter factual, is very expensive to do. Sometimes measurement can take on a life of its own. And so, one of the things that we're beginning to experiment with at the foundation is, really being very clear, again related to strategy, what are our learning questions and try to craft a learning agenda for every single strategy that we have.
What are learning questions? Why do we have these questions? What decisions the informer. If we had the answers, would we do anything differently?. When do we need the answers to these questions by, and with what degree of certainty? And are there data sources available to actually answer these questions?
And what are the best methods for using for answering these questions? And making sure that all of those things calibrate because often, you've got measurement going one way and program and investment going another way. But this is true in the foundation world and before I came to the Foundation World a year and a half ago I was on the Practice World, it was true there too, that measurement often goes one way and practice goes another, and so, really making sure that you calibrate your data collection, your measurement, to what it is you're actually trying to do, is really important.
A quick story on that. I was just in India a couple, about a month ago visiting our foundation's program in what we call our Avahan program, HIV prevention program, working with sex workers, men who have sex with men, and truckers have sex with both of them and I was visiting with the female sex workers and It was amazing because they were talking about their work doing peer education on the streets with other sex workers, and they were talking with such sophistication about their measurement they there's a slide that I'm not going to show because we're not doing power point now.
But this is one of my favorite pictures from my trip to India of this female sex worker who is a peer educator, who is pointing to a slide that says monitoring and evaluation. And she's using all the data that they're collecting related to the work that she's doing on the streets do outreach with other sex workers doing education, doing distributing condoms and, because she's illiterate but very smart and very sophisticated.
She's. They're. They have developed a whole coding system that is figurative so they use pictures to describe what it is they are doing and they use different colored to put on these charts that describe what it is their doing and what kind of contacts their doing with which women and which street corners and then they collate that data once a month.
And they look it and they use it to course correct, on the ground, as they're doing their work, and then that data gets added up to look at the district, and then it's looked at across all the districts on a regular basis as well. So it's a really good example of, "How do you use data and aggregate it from the grounds on up to a whole huge portion of the country.
It not only makes for good value in measurement. it also increases the quality of the data because it's actually being used. Point three in the spirit of and the rest of the contents of this conference around culture in context id say how you measure is really important how you do your measurements.
So the example in India in Avahan with the sex workers is a really good case in point. How their doing measurement is consistent with their cultural context. And and it's useful and meaningful and doable. We find a similar kind of lesson everywhere. When you're working with high school kids, you've got to be really careful how you ask them those questions Those high school kids know exactly how to answer.
The fourth point I want to make is time, beware of many dimensions of time, and time can be of the essence, and a quick story again on the Abahan program when Bill and Melinda went to India a number of years back, they were very moved of what was happening. There with the female sex workers and wanted us to do something as a foundation to address the HIV epidemic grow and what we thought of at the time as a fast-growing epidemic in India.
They said, "We want to do something fast. There's a real sense of urgency." So we went And and built our India office and had a great sense of urgency, and the gentleman who was running that program knew that he needed to do measurement. So he convened the best minds in evaluation at the time and said, "I want you to help me figure out how I should do measurement".
And they talked and talked and talked, and they spent a year talking about the best ways to do measurement. And in the meantime, our office was filled with a sense of urgency to get going on the program and so we did get going on the program. And in the process lost the opportunity to get a true baseline.
And so, time is of the essence, that was a really important lesson for us. And we're making up for it, now using other methods, but it was just a very important lesson. My last point is just to remember to question assumptions, don't focus on proof as much as learning and always be curious.
Thank you. my inclination now, was to actually to go back to Joe in Duru and say we've just heard from three folks who are doing lots of the leading thinking on, on measurement of impact. What are some of your reactions to what you You've heard about the theory and the practices they've, as they've described it.
That makes sense to you in your organizations. What about that, what would work best what they've said would be the biggest challenge. Jaru?
I think what actually for me what resonates is what said last. Is this whole thing of when do you start with the base line, what do you do because we were really discussing that and you want to get things just right. And, at the same time,you know you have to do for the fun side so for me this whole time.
Something which I personally am going to take back. At the same time which is being said even by Brad and David, one of. things which, from what all three of them are saying and also what school keeps talking about and I think hasn't been answered is, we talk about bringing about systems change. And yet I didn't hear from any of the three and even I don't know the answer.
So it would be nice to discuss this. What is the the impact that we need to measure. Or what is it that we need to measure which can help us really bring about systems change? so if I look at our main theory is children have economic rights. They have social rights and that's what we want to see, people's mindset to start thinking about that.
I have no measurement tools to start looking at that and my biggest worry is instead of bringing about that system's change I'm start will be looking about that activity based indicators and that's where my fear and my attention is. And, I haven't really heard that answer in this and I don't know, I'm a bit caught.
Sorry. Sorry for asking a fantastic question. Don't be. Don't be. No. No. I mean what Brian, David a, what, I mean I think is proposing one of the trickiest questions out there, Well. your thoughts on that. David?
I'll take a first cut. yeah. Thank you. on what i may add as we built our methodology we were very focused on trying to look not just at the organization as the unit of change, but if you will, at the system. So we ask the kind of first question who needs to be involved in measuring in order to get a picture of system change.
And then we look to practical operation cycle planning, doing, reporting, with that in mind. One of the simple things we suggest that organizations do, Jeroo, and it used to this. Develop a story of what success looks like in a sense of what the indicators of change together with different actors in the system so that you get a composite starts to piece together the system as a whole indicators from different perspectives and so in a sense the role that the system changer is playing is one of a conversation across the system about what success would look like, what system change would be.
From that you can work back and start to put activities and other indicators output kind of indicators together that start to lead you toward that.
Brian do you have a thought? I somehow think that Oliver Wendall Homes would like the question too, right yeah, I don't know what to say, because it goes right to code.
yep yep.
I think part of it defines that depends on how broadly you define the system over which you want to have change. Our theory of change at Acumen Fund is that market-based solutions can in fact complement a development in charitable or purely private sector approaches in that the one two nine eight example that I gave before will be successful if they create a market for emergency services professional emergency services staffed with doctors on ambulances that can get to people who call a free number within eight to fifteen minutes.
That doesn't happen right now in India, and if in fact they can do it in Mumbai and prove the case and other governments and pick it up and figure out a way to do it. Another one two nine eight is doing it or not. They have changed the system for emergency medicine. That is does access to emergency medicine end poverty?
I don't think so. So the question is how we think about defining the system is around market based approaches if the marketplace responds times then a system in fact has been created. But I do think that is a tricky question around interventions that have loftier aspirations in some of the problems are global in scope we have to be thinking about how the systems intercontect.
yep. Did you have a thought, Page?
Yep yep yep. Well I do, I have three. One is that you have to have a strategy that's targeting system change. So, and I would caution you to define what you mean by system change. I think we use that, those, we use Such loaded terminology that it's important to just unpack that. What exactly do you mean when you say that?
Because when you go into the measurement world but the more specific you can be the better. So that's the first point is that do you have a strategy that you would targeting what you would consider is system change. The second is there are two things I heard in your question; one is, what kind of measurement is gonna help us to achieve system change that's sort of measurement as intervention, and then there's, what kind of measurement can we use to demonstrate that we've created system change.
those are related but I think also distinct and so I think again just unpacking what is it that you mean because measurement is an intervention and so when we say, you know, we want to measure things and see which models work which approaches work so we can influence a governmental system or bring in markets to influence markets that's using measurement to advocate for change and then there's are indicators that we can be tracking about systems, about health status or poverty status, that actually with the way government works or the way markets are working that actually speak to system change.
In the example of housing, the homeless families, that is a system change strategy and there's lots of measures in there that look at how are we going to impact the system of how resources are allocated, which actors are in place, but being very specific about what we mean is really key.
Yeah, I I would just chime in there and say yes I would agree entirely with that and I think typically the visionaries who imagine a systems change actually have a pretty complicated picture in their mind of, here's why, for instance yours on dignity. What a lot of change the feeling of dignity. I suspect that you have.
There is a whole bunch of things that factor into your vision about that and the same with that sort of self of a children and so I think what Fay talks about is not anywhere close to It's ability, in fact I think it's right in the heads of the, of the visionary, but it's often, in my experience, it's left, left too implicit, but you can say this cause of this, cause of my theory is this causes this which causes this which causes this and you cannot measure these intermediate steps.
What do you think? What struck you as you listened to the three talk about the issues in measurement and impact.
measurement and impact. One or two times I found the word market and that sort of disturbed me a little bit because often the market as seen as the fantasy to end all ills. But, when you look at developing countries there are there certain, there are certain sections of people who are beyond the market.
And unless you have societal systems Items which catered to these people in a special way, and which may be beyond the market. You are not never going to bring any social, my feeling, it's a very strong statement, but I doubt very much if it can bring social justice social equity, gender equity, into the society, that we, I mean into this society and the society we one for the future.
So, while market, I mean, is the answer to very many problems, I think like in India. About twenty percent of the people, I mean India is shinning, we've got, I mean, nine, ten percent growth all that is good that maybe ten to twenty percent of the people at work going down the drain. Because the, the safety basket which was there is slowly disappearing, and its not people should we not devise as a as a society that cares for everyone, as an inclusive society, systems whereby we can also take these people?
How will you measure those? Measuring those in
clusive processes where the people who cannot shout for themselves, who cannot even ask for it, how do we bring in that particular section as a caring society is something that bothers me.
What are your thoughts, Brian on, you'd like to bring market Oriented solutions, what a acumen fund but I suspect you have the same view of, or similar view of, the limits. yeah, I know, I don't disagree with what Joe said. We do think that the market is a very good listening device. If you design with the user The consumer or the individual in mind and they in fact express their preference through purchase if they can.
Then you're going to design. products for them. I think our experience in development is that too many things have been designed on people's behalf in Geneva or Washington or Seattle. And, if they are designed on the ground with their preferences in mind, the solution may in fact be sustainable. the markets, we don't think they are the panacea.
There is a need for charity. I don't think that the markets in Asia or the US have done much to narrow the equity gap, and so those are important issues. But, when you look at the amount of development assistance that is being channeled into south Asia, Africa the lack of accountability, the lack of feedback mechanisms the market might be one complement to some of those flows of funding that would shine a light onto solving problems into a new way.
I hear a real piece of the market that you care about is the user and how the Reacts to services, offers, what have you. Yeah.
Yeah.
Any other thoughts on on Joe's comments from David or Fay. Faith?
Well I think everything that he said is important, I mean that I think what you just said is important. I think hearing from constituents is critical when you are crafting your strategy, as your measuring results, interpreting results. We're trying, we're grappling now with how do we put in place feedback loops so that we can hear from who we consider the ultimate beneficiaries of the work.
Trying to experiment with different feedback loops so that we hear from small holder farmers in Africa, from high school students in America, from the different folks who are are from the female sex workers in India. So, who are our targets so that we both hear from our grantees and our intermediaries.
But also from the end user in a way that is meaningful. So i think it's really important. I don't think we've cracked that code but it certainly a value.
As I understand, David. I'll ask a question, if you want to answer a different, or had a different point, go ahead, but my understanding is your working on mechanisms to hear that user voice more clearly, if I'm understanding keystone properly.
Yeah, actually the point I was going to make fits well with that. I really wanted to just say that the word we haven't heard yet, though it's been very, very close to the surface and what, in the cases that Joe and Jarew have said, is power. And oftentimes in the kind of measurement conversation we tend to leave it out; it's a kind of a. It's part of our thinking, but it isn't as explicit as it needs to be, and we won't really answer into a full learning conversation with our critics, if you will, until we really embrace the kind of classic power question.
So, one way it comes up here is to look at the power of organized voice, in the communities that you're working, among children among high school students. And where, how does that voice, not just how is it expressed in our metrics. But how's it organized and how do we support that organization and self actualization in that sense and so that can become part of the way in which we measure as it goes to framing the problem.
My new theory of change, its a really important part of the story.
I can't help but asking the following question based on the Again, the things the various panelist have said. There is a notion that some things are measurable and some thing are not or are less so, something they are more measurable something are less so, in the case of your, your wonderful organization it's easier to measure the decrease in disease.
Incidents of disease, and days of from disease. That's harder to measure the impact on dignity. In the sense of well being. How hard should we push if that is the biggest pay off? How hard should we push on measuring those things and could be satisfied to measure some things, and leave others in the realm of unmeasured.
Do you have a thought? already I am sorry you it.
I dont think we should leave them unmeasured. Because then it can very well say, well, this is something we taught and we still think it is correct. But, that would the anecdotal or incidental experiences, on off are that an organization will go through. Okay. In this village, this was a response of a particular woman to a bureaucratic coming and asking for a bride?
how does she respond? or the people how they came when they was exploitation of a money lender, and, or a liquor merchant, in that particular village How do they respond to that and how did they eliminate these people? So we can always find They're more important I think for the implementing organization than the funding agencies.
It think if
to show that we are on track That we mean what we say, we are getting where we want to get. This is important. So measure them perhaps in different But measure them none the less. What do you think on you measure, should you measure the impact you have on these kids feeling of self-worth, self-efficacy.
on organized we've been struggling with it, just thinking but I think we are measuring it anecdotally but i think if you want collective change and if you really want to change Mindset. You actually need to push yourself harder to try to find some criteria. So I actually, the first likely with joy, I think, it's important to have criteria and to beyond anecdote.
But it is very, very, very difficult to do that and I know that it's a thing to struggle. Especially for us because we are a network, so trying to get our partners to do it makes much more difficult.
Yes, yep, so you don't have direct access yourself. Hey did you have a thought, thought on that, that question? Are there some things you shouldn't measure and some things you should or?
Well it goes back to what we've been talking about. You should only measure it if you are going to use it. And some things are, they just are harder to measure. Even when you do measure them, it does not mean the measures are any good. So self esteem and those measures are classic examples, when you start taking these standard scales and applying them in different cultures.
If you don't use standard scales, then you have a problem with what exactly are we measuring, and how we know actually that this is a good measure of what it is we're trying to do. But then if you use standard scales, you have a problem of cultural translation of those scales, and you also have a problem, cultural, also in terms of age, we're working with a lot of young people here, there is a lot of controversy on things like self esteem scales.
And actually, is it a good thing for young people to have high self esteem or does it actually mean that they feel more invulnerable than they really should?
So, once you open up those doors to those kinds of issues, it just gets tricky. And then you think about how are we going to use it? And are we just going to question the methods if we don't like the results, or are we actually going to use the data to do something differently? It really goes back to what we've been talking about of how are we actually going to use use it the data.
So it might go to Joe's point which says that's more useful for the agency in question than the funder because they're, closer, maybe, more culturally aware of how to interpret the results.
Maybe. I think it's the same issue, actually, I'm not sure it's that different, because if you get back results that you don't like, or that don't make sense to you, are you going to try measuring it differently? Or will you do things that are different? And I'm not sure it's that, I mean, I think there are differences in terms of what funders look for, maybe and what programs and practitioners are looking for.
In that case I still think it is a question of how much can we trust the measures and how useful are they to us.
Before I go to the audience for questions I would love to know if anybody on the panel, based on the comments, has a question for anybody else on the panel? Do you have one?
I'm anxious to hear from the audience.
All right?
Yeah.
Yeah, yeah. All right. So please, now we have mikes somewhere, presumably. Do we have mikes? Do we, yeah, we do have mikes. Okay. There I have a hand, and then I'll go here, and then here, then here. How's that?
Hi, my name's Onarosa and I'm with the World Bank's developing market place. We are a program that funds concept innovative concept level ideas. My question is going to be about the communication between the funder and the fundee while designing M;E structures. Specifically, when I sit down with the 30 projects that I supervise and I want... and I'm trying to design the M;E structures with them to track from the very early... when we start funding the idea, and the project right when the project starts through the very end, I sense that the organization is sort of defensive of their intervention and they want to give me as little information as possible.
Whereas I'm trying to communicate to them that this is not only a tool for me to understand where the money is going, which is public money in our case, but that this information is really useful for the organization itself. I want to hear from both sides of the house in terms of the organizations as well as the donors what is the best way to structure that framework without... while communicating it to them very effectively that you are not out there to get them or you know, prove that they are failing, but that This is a useful piece of information that needs to embedded in the program itself.
Thank you.
Who's got a thought? Right yea. Sure. Great Andrew.
Experience having done it with other entrepreneurs and past that you can point to, is the first thing I would say. If you can actually show that you've added value that's one thing. The second thing is terms in your term sheets. That require information rights of the investor is another helpful way of just saying look this is grown up capital and you're going to show us your financial statements and we're going to use them to help you.
Alignment of interest is a third thing. For our, our investments. We make sort of quasi- commercial returns if the companies are successful, and if they aren't then we don't achieve our returns and our model is not as effective as we thought it would be. But, you know, it's important to shift the dynamic, I mean, many large funding institutions, public institutions, have a reputation for asking for reports that are not helpful in a business for asking for reports the day before the secretary or the directorate is going to give a speech and, you know, scrambling for results at the last minute.
We all do it. We still do it at Acumen Fund. But making sure that the conversation around metrics is a long-term shared partnership, not a "give me the information tomorrow so I can justify my program for further funding." And that takes a long time of shifting the sort of behavior patterns. But again, showing that you've actually added value in other cases and being able to build it into the economics of the relationship are essential.
And if...there are some people we do not get information from, and we just have given up the fight. And they're capable of managing their business on their own, and that's fine. And we will not sacrifice the relationship on the basis of getting information. But most people now as a matter of course expect it, and it's not...it's no longer a big issue.
Actually, our problem is even more difficult, because we want the information and we don't provide any funds. So all we have is a concept and material and we're saying we want all this information. So everyone can tells us to go and get lost. No that's the truth. They don't tell you that do they?
Why should we do it? But, this is pretty cheeky what we are doing in some of these. What we try to do is instead of trying to say it's a burden, though even I think sometimes it's a pain, to be honest, what we try to do is we've sort of created consultative process around it to say that this is not information that we need, but this is part of all of our learnings and then sharing our publications.
So one of the first things when we're two and a half years old, but within a year and a half we had a publication which sort of shared all that information; so the partners will giving them a deal, sold the publication and they said "Well OK, this makes a bit of sense," because they immediately saw that the data was being used for something And wasn't going somewhere in fifty piles.
Second, which I've done in another organization, is started using the data for them and teaching them how today's funds, so then again they're seeing the use of the data which is coming in, but the third and the most important is seeing the data on how we can change our program strategy. Every person in the field wants to deliver a quality program by and large.
And if you can make them see that the data that you're getting, the information that you're asking is going to qualitatively improve the program. It has for us. Then people will definitely start giving the details slowly and steadily and I think that's what we have tried to do and we have nowhere close to a 100% success but our partners are seeing the need to do that sort of thing.
And we have a fantastic panel of experts so they can take on the partners. we'll go onto, we'll go onto the next one. Orange sweater over here and right behind, beside the orange sweater then right here and then right up there and then I'll do a whole new other one. Oh . We've got another one, we'll do that first.
I have the mic so, You got the, I guess I should probably ask my question.
You have, you have the power, so much for your plan, you have the power. It's all yours. Thank you very much for a very insightful panel. I am Tim Shy [sp?] from International Bridges to Justice and I think you have all spoken about the benefit that the organization can derive from impact measurement.
But, I'm wondering, sort of understanding that were all working in a sort of very resource constrained environment, whats happening in the social capital marketplace to create sort of extra incentives, for entrepreneurs to invest in impact measurement. And what can foundations do, or what can the holders of capital do to really, sort of, create this partnership.
Who's got thoughts on that? David.
This is a very quick response. I mean, I think the first and most important thing to say is that the funders should pay for it and adequately. So, that it's not at the expense of program activities, but the resources are there. Now that requires a conversation about proper costing and benefits and all the rest, but it's an important starting point.
Fay?
Well I would agree with that, but I guess I would challenge the statement that "what's the incentive for social entrepreneurs to measure stuff," because I thought that was actually part of the definition of what it meant to be entrepreneurial is that Is that, you know, are measuring things along the way and course correcting and using data informed practice, and that's what I think of actually as the hallmarks of entrepreneurship is that you're bringing the best of business practices to the sector.
Hopefully bringing the best of non profit practices retaining those and having the best of both worlds, so I would actually say that absolutely measurement is not cost neutral need to do it. But I think of it as part of the ethic of social entrepreneurship.
Yeah, I guess I would think as a funder, as we are in part, I would be slightly nervous about an organization that didn't feel that didn't feel that they would learn from measurement from that's worth any other thoughts here, or shall we go on to orange jacket now.
I'm Catherine Hansen with A-Learn, which is a non profit, educational nonprofit focused on increasing the underrepresented groups going to college several of you mentioned was self-reporting. And one of the things that I'd like you to address to help our thinking on this issue is working with kids who do get into college.
We might know that they got there. We don't know whether they finished. We don't know when they dropped out. We don't know in many cases whether they have remedial courses and how many. So tracking a population that by that definition is mobile and where you don't have a connection like a scholarship to sort of force them to self report and the accuracy of self-reporting.
So I'd like comments on that.
I can start since we invest in that area.
Faye? Yeah. Yeah.
It's a big problem. So we don't have the answer. Like most of the areas that we invest in or many of us do, there isn't good data infrastructure. And so, you know, when you start to think about what happens as these kids graduate from high school, go into community colleges in most cases, just like we don't really have good data across our...and consistent data and a national repository of consistent data for our high schools and what happens in our high schools.
We don't have it in community colleges or college systems either, so we really have to build it. I mean, that's really...the ultimate answer is that. And actually we're making some investments in that area, sort of data infrastructure in community colleges and trying to actually build a infrastructure for that.
But it's very, very hard to track things where there are no data and there's...unless there are incentives built in for young people to come back and report. Which is probably the best way to do it is to provide some incentives, not necessarily for the social entrepreneurs, but for the young people to come back, because we know that actually incentives work for self reporting.
Any other thoughts on that? Brain, did you have a...?
Well, as we think about how we might trust the quality of the data that our peers would put into a shared system that is largely self reported, on the financial side it's easy enough to get audited Financials to correct whatever is self-reported on a quarterly basis, but on a social output basis how many were sold, or how many ambulance rides were taken.
That's a huge question. We've been thinking about whether there are ways of doing, sort of either peer reviews or independent evaluations that are well short of the social audit, but when you think again, when you are counting outputs, the outputs are typically derived from your operating numbers, which are typically sitting on top of your financials and it's not, it's hard to game the systems when it ties back to the economics.
But that is a big question. We haven't figured out how to solve it. What we did in my other organization, Child Helpline, is we had all the helplines to self-report their organizational processes. How they interacted with the children, all of that. What we found is again if you take the basic principal of trust.
Say you are not forcing people to do any anything. But look at it as a self regulation mechanism. People took these processes very seriously. Since how you decide to communicate to the people and make them feel the ownership that this is going to benefit them. And that's what we found. So initially there will be a lot of resistance to reporting, because honestly, if I ask myself, I hate any sort of reporting too.
Who likes school exams? Lord, I don't think anyone here liked exams did we?
Anybody liked exams? No?
No.
You're right, you're right.
Exactly. So if we don't like exams, why are we asking people to have reporting? You know, it's that whole mental connotation which we have. So, why dont you try to say hay you know maybe you do an assignment which will benefit you, because you are going to read so much more, prefer assignments not too many, but you prefer assignments then you sort of move on.
So it's creating the mindset and I think even us having had lots of networks, main thing is creating the mindset for a positive frame of mind, either for impact assessment or self-regulation, and people will automatically start doing it because then you see the benefit for yourself; and that's what, at least I've found, has worked the best.
So we geton this one.
Yeah, go.
This is the whole point about, how do you set the incentives, really, to get people to come back and give you data down the road? It's an example of a larger problem in getting feedback from folks who are the primary constituents. Because, sometimes people don't actually, they may not know why you're asking the question, or they may want to answer in a way that maximizes their advantage.
So they're wondering how to do that. So, they referred to trying to get feed back from farmers, poor small holder farmers in Africa. When you come into that setting and you ask an assessment type question, they're thinking, for the most part, "Okay, how do I answer this to get the best benefit for me?" They're not set in a natural way to answer the question to help the program.
So, the solution to all this is to go right back to the beginning . So at the planning stage to set expectations and to set in the context so that people understand to three, four years from now. We're, we, this program, that you're part of, will depend on how, how you can form us on what happened to you.
So I guess the simple answer to what I am saying in a long winded way is you have to start at the very beginning, at the planning, which is why we argue its so important to involve the primary constituents in actually setting the indicators at the very beginning of what success looks like.
So right beside in the gray jacket, then I'm gonna here, and then back there. Then I'll start a new group.
Hi. Can you hear me? My name is Courtney Bickard. I'm the Balkans Regional Director for SNV, then Netherlands Development Organization. We are a capacity building organization, so we aim to work with our clients to build their own capacity to bring about change in their own societies. So we like to say when change actually occurs that we can't take credit for that change actually occurring.
And I think that's a little bit too convenient and I'd like to move beyond the outputs and processes that measure the number of people trained, assessments that they said "they actually loved working with us" etc. To be able to actually test our assumptions about if we, because we have a shared vision with our clients about what change they're to bring about.
Do you have any suggestions about a sort of intermediary level of metrics where you can start to test your assumptions around if we build the capacity of X institutions, that can lead to Y change in the societies. Anybody wanna take, take a shot at that? Right I would ask what what thee organization you are building the capacity of are doing and then I would say what do they do it time zero and what they do it time plus one, after the intervention of the training, the capacity building, I would begin, we would ask the questions of whether or not There are things that they are currently counting that you could keep counting, where there are things that they are not counting that they should be counting.
And this systems questions about whether or not it really matters. But I would look how they define success. Work backwards from that, and see if there are intermediate variables that you can begin to track. That's what we do with each of our investments, and it's very strait forward and simple, but if you don't know what you are doing last year you can't compare how you're doing this year.
Thoughts, thoughts? David? A growing kind of literature and experience in field around measuring capacity building outcomes, and I can point you toward that, but I am sure you are familiar with a lot of it but you need to have an clear counter theory or logic causal logic in what you're doing, and you would arrive ideally together with your partners.
So, they would agree with you what success looks like to them two, three years out if support from you. You need a initial snapshot of where your at at the beginning, and one of the things I like to look for, is the extent to which two, three four years out people are actually using things that they picked up from your course, and if you do that systematically, track them systematically through surveys across a lot of different training.
You start to see which training seem to be sticking the most and seemed to be most useful over time. It'll help you change your training, and focus in on the things that seem to be most useful. And by the way, you can also, and this is where it gets really exciting I think, use it not as the feedback loop for how good was our training, but rather designing new products.
So you can add, when you're asking people, kind of one of the things that you learned in our training that you are exposed in this training four years ago or you are still doing are relevant to you, you can ask a simple set of question surround what. What do you think would be useful? who would you like to hire?
What skills do you want to bring into your organization now that could come out of our programs tomorrow? In that rein-courses the relationship over time. Because, your alumni are your best designers of your, of the kind of new products.
Or we we flip up to there, and then I'll go to here. Just so we have back and forth. You'll go next. Yes.
Hi there. My name is Audrey Sally. I'm from Rianta capital which has an adventure philanthropy initiative in India, and I've just been thinking as was listening to the panel speak that is from the investor perspective, an investor will set out the values and objectives that they want to see sort of manifest in their portfolio and then you get into the field and you start actually looking at projects and maybe starting to make investments, and you realize that there is this whole host of other indicators and measurements, just idiosyncratic factors that you hadn't thought about.
So if your looking at a portfolio of the projects that you want to have invested in, you find yourself you find yourself not, having to compare not just apples and oranges, but mangoes and kiwis and tomatoes. I mean, it's very difficult to actually Figure out at what point on the continuum of the standard measurements like number of bore wells or crop dams versus all the idiosyncratic types of measurements that come out, like which women in which self-help groups end up getting helped.
So that's my first question, first part of it, is just where on that continuum is it good to be, do you want to continue to try to capture as much specific data as possible, or is it really good to simplify as much as you can? And then, the second part is, there's a group of us in London, coming from different funds and foundations and other venture philanthropy groups, that kind of have these informal jam sessions once a month, trying to talk about these things, and the first one we had was about metrics and measurement and needless to say we didn't really get anywhere so far, but it seems like we all have this dream of coming up with this fantastic dynamic template using ICTna web tools, you know, trying to come up with a way of having us all share this information, especially the idiosyncratic elements because only in aggregate, only if the sample set is huge do they start to make sense and so I'd also like to ask the panel, Is there any utility in that, do you think, and is it worth the effort?
what do you think, Brian, you gotta to that?
So, on the first one, I'll answer quickly that I think this is a practice we need to get better and better at doing it with each investment, and we track far too many indicators for each of our investments. And do not have enough consistent metrics that we think matter across them. But there are a few things, like revenues, jobs, number of people served that we want to try and make sure are the standard basis.
Then for any given intervention, whether it's health clinics, there are a few things that matter. Number of customers, revenues per square foot in the clinic, you know, trained medical professionals, percentage of staff, they're all, a number of things that if you actually really know how to run a clinic, you'd ask as the natural metrics, so anyone who really wants to be in this business should learn what those are, and that includes both investors as well as operators.
And for then for every there's there's special metrics, every one's special, so if there are things that really matter that are unique to the local context, that's fine. Continue to measure them, but don't expect to be able to use those in the context of benchmarking I would love to talk with you offline about the second point around building a shared information platform that allows for both consistent measures across different ventures as well as customization, which is what we've done with our portfolio data management system that we've built with Google.
I think it'd be a broadly enough of interest.
Sure.
maybe just spend a couple minutes of what you've done thus far and are planning to do on. Sure.
On that. Yeah.
So, the Acumen Fund was founded with the ethic of metrics transparency accountability at its core, and about two or three years ago, we recognized as we opened up our, our, our first offices in India, Pakistan, Kenya where we now have more of our portfolio staff then in New York we needed something other than the email, Microsoft excel thing.
You know? All saved on the server, to enable consistent performance tracking across a global portfolio, and so we asked ourselves, well geez, there's got to be some tool out there. Adventure capitalists have tools, but really VC's care about financial returns they don't care about job creation or social impact of the companies they invest in, so there isn't anything that includes both financial as well as operating in social metrics that you can get off the shelf.
We were looking for a soup-to-nuts solutions from the first time we meet the entrepreneur to the end of the life cycle of the relationship, but guess what sales force has already built the worlds best customer relationship management system, and they give it for free to non-profits. So, we said terrific, why don't we use that to the point where me make an investment?
But there's still this question of how we manage the portfolio, collect store the information add qualitative reporting to the quantitative reporting, measure against targets on a quarterly basis So, there was nothing there. And we have a support from Google institutionally, and there was this vague offer of help from their engineers, and we're co-located in their building in New York.
And we appealed to a few engineers who spend their day working on search infrastructure and now spend their nights helping us design a system that allows us to track our portfolio. We have been ruthlessly self-interested in building this system. I want it to help me and my colleagues do our job better.
There are lots of discussions in the sector about, you know, grand multi-party collaborations that sort of collapse under their own weight. And so, all I wanted to do was make my job better.
Over the course of building this tool, using one of our staff members and volunteer engineers from Google, we recognize that we think we built the best inbreed software solution for people who do social investment. There aren't too many people doing social investment, so this isn't some, you know, go-raise -capital investible buisness idea, but its something that might in fact be worth sharing.
So Google, as it was setting up its own portfolio management processes asked, "Can we use the tool?" And we said, "Sure." And so we've been adapting the tool for their use. And then Skoll found out about it and they said, "Well, can we use the tool?" And we said "Sure", so we just ran a beta test of the tool to see if, in fact, others would want to use it.
And the conclusion was, sure, there are lots of people who want to use this tool. We're not a software company. Google doesn't sell software services, so we are now at this inflection point to figuring out what we do with this asset. If we get enough critical mass using the tool, and we've had 15 beta testers and about 50 people taking a look, including groups like Root Capital and E+Co and some of our peers in the sector in the States.
IFC's looked at it, Skoll, the DOB foundation. You might be able create a data infrastructure that has standard metrics. There are a whole host of problems around creating what we call the social venture one. Around self-reporting, data quality, definitions, GAAP as tightly defined as generally accepted accounting principles are, there's still a lot of wiggle room, and if you try and translate that to, you know, generally accepted social performance indicators, it's a total mess, and we've learned through experience that you have to be very careful at how you define, you know, number of clinics open.
Is that the number of clinics you have open at the end of 2007, or was that the number of clinics you opened in 2007? And if you don't know what you're actually entering, then all the data's a complete mess.
So we're at the point right now of trying to figure out whether this a tool that would be valuable for the sector more generally and Google is committing either engineering resources, or cash, to help us figure out what it would take. And then we need to figure out who, ultimately, would own this asset because Acumen Fund is, again, not a software business and can't be both judge and jury in a tool that the sector might use.
So we have to find a third-party home. We talked to GuideStar about this this morning. There's a whole bunch of other folks who might be partners in this. We'd be happy, again, to give you more details after this session.
Perfect, Drew, did you have a thought on this one?
I just have this story.
Good. I like stories.
So, basically we like, in fact, measurement, and we said, you know, we have to capture everything. So I really am enthusiastic. Team came up with almost fourteen pages and more than two hundred something impact indicators when we started looking at it, and then when we To our partners, "Are you mad or are you stupid?" And I'm saying this because sometimes you get carried way, in trying to measure impact from a distance that everyone was important, because like you, we wanted to capture each and Every small detail, for each and every target we were working with.
And it was really impossible. So, on that, we really had to shrink to saying what other mains.. Seventeen. What are things we can measure globally? What are the things we can measure locally? So if we only went from going all over the world, and having a nice dream imagination to being extremely focus to the seven things or the five things which link to our program.
And then our effectiveness, and I think that's important that we shouldn't get too carried away also that sometimes the measurement is more important though. you make it more important than actually the program, because then it will fit. Perfect, alright, oh, yes, yes, please go ahead. just going back to purpose and why are you measuring and getting very clear on how you are going to use the data.
So, I'm being fairly specific. Because specificity is measurements friend. The more specific and clear you can be be, the better data you will get and the more useful the data will be. So, I would just caution about that. I just, I'm feeling a little something in my stomach, so I want to just actually say something the term 'impact'.
Sure. Because we use the term 'impact' to mean a lot of different things colloquially. But in the measurement field, impact evaluation, actually, has a particular meaning. And so, so It's fine to use it in conversation. You can do whatever you want, but just as we're entering the I just think it's important to be very clear, again, about what is the measurement that we're doing and why, and that to Brian's point earlier, when you're measuring against milestones.
That's one thing when you're measuring outputs, that is, the things that are happening along the way. That's another thing. When you're measuring outcomes, what are the changes we're observing. That's another thing. 'Impact' actually has a technical meaning in the evaluation world, which is the World Bank definition, but it is also the the traditional definition of impact assessment is: you know, what is the difference that this intervention or set of interventions is making.
It is about attribution. And so, that involves some kind of counter factual, in saying, if this intervention didn't exist, we wouldn't have this change. Or, because of this intervention, we have this change. Now, I'm not a true believer, thinking that you need to always measure impact in that way, with counter factuals.
But it is important to know the difference. So that when you're out there talking about what are we measuring and how are we measuring progress, versus is this model causing this change, because my investors want to actually know that. That's just a different order of magnitude in terms of the rigor and the certainty and and the responsibility in terms of dotting your i's and crossing your t's.
So it serves a different purpose. So, it's not that one is necessarily better than another. There it goes to purpose. Just to be very clear as to what you are measuring and why.
Yep, nope, it makes good sense. Patience Chuting.
Thank You and I'm actually on the board of a social innovation and investment group and so I'm, sort of, really coming from the, it's like, point of governance, I suppose. Well, thank you very much for your very erudite and very specific sort of advice to everyone. But I just want to perhaps get more into the general nature of things.
And I thought that it's important to talk about culture and Perhaps even language, and it is quite interesting the thing about markets for example, and the different, you know, the definition we have of markets, because some people can take offense at markets, and, but but of course we know that if you actually do want to make a change and want to help with finances, a social entrepreneur.
There is also a relationship. And so, I'm just asking about intuition with these things, you know, instead of this very specific in measurement, and also, I would like to know how many for example of you, specially those who are founders who actually go into the field and what kind of relationships that you actually have with the people fund and perhaps people are coming from two sides there, so perhaps if I can get the answers from people who fund it as well.
So, what is intuition role of that in your work and the other is is getting close to what's going on in the field. Intuition!
I love intuition. I do most of my stuff intuitively. And then I get rapped on my knuckles and I'm told by my team, "that's not enough! Let's get scientific, and let's have things put into, you know but I think, honestly, everyone must have read the book "Blink," and you do make more positions in the first few seconds, but having said that intuition is very important, but you also so we need to be looking at what is causing the change, what are the factors that you want to change.
So there has to be some amount of scientific evidence to back changes Back systems to look at what you want to do, so there has to be a mix of both. You cannot really rely on only intuition, and this is coming from a person whose basic initial premise is to do everything intuition. If you don't have that you can not systematically create... institutional systems change.
It's then personality lead, individual lead, as compared to something which is long term. So I think that would be my thing, and that applies across all cultures, everything.
Brian, do you have a thought?
No I agree with that. I was going to actually disagree earlier with Fay, in terms of the point you made about enterprise, social enterprise, and data being baked into it. Actually, most of the entrepreneurs I know are highly intuitive, but intuition in my experience, doesn't scale very well. As Jurie just said.
And there are cognitive biases and if you think about all the successful business people that are risk takers and think by their think on their feet. There are many failed business entrepreneurs who were risk takers who think on their feet, and so the characteristics of both successful entrepreneurs and failed entrepreneurs, and there's a lot of literature about cognitive psychology of how you make decisions under uncertainty, but I think intuition is critically importance.
And for us, part of the next step of that is pattern recognition, but recognizing the pattern involves writing things down and seeing them, whether it's pictures or whether it's data or whether it's other feedback or input, but understanding what those patterns are is critical. For us, the relationships are essential in the field.
We are a relationship business, and we now have 15 people on our portfolio team in India, Pakistan, India or Kenya, and five in New York. When I joined Acumen Fund four years ago, it was four people New York and none,in Indiapaks Center, Kenya. And so the relationships matter. We are the boards where we take equity positions.
We sometimes ask for board observers rights for companies where we loan money. We really take a very active relationship. We've got Acumen Fund fellows who are a highly selective group of folks seven a year who spend a year on the ground with invests, essentially working for them in Seconment on their top management challenges.
So we have a very hands-on relationship, and believe that only the learning will happen by rolling up our sleeves and working side by side with the entrepreneur. Yeah, if I can just chime in on intuition. Intuition is very much a catch-all word that is used for a lot of things, but a piece of what people generally refer to as intuition is, I think, critical, and there's an argument to made that the kind of thinking that leads to new ideas It's what a lot of people refer to as intuition, it's called abductive logic,which is the inference to the best explanation.
And that's what the best entrepreneurs do. They say, like Joe says, I look at this villages, and my best explanation, I can't prove it, but my best explanation for why what's happening is happening is the following. And I'm going to intervene in the following way. I don't have proof that I'll reduce sickness and increase dignity but I think it will.
And then you go and do it. And, sometimes it fails, but other times it succeeds, and then and then you can, as you get experience with it, you can use more traditional analytical methodologies for evaluating progress, and the like. leap. It's an abductive leap of logic. What a guy named Charles Sanford Peirce called inference to the best explanation.
And he argues that that's the only way new ideas come into being. And the major tool-box of logic for us. Supposedly educated people are inductive and deductive logic, neither of which has ever produced a new idea in history. That's his argument. intuition is important, but not, to Brian's point, not sufficient.
And Gerry's point. critical but not sufficient to grow organizations. Does that make some sense? me there is a stage even before intuition that is gut reaction. I mean, if there's an unjust situation, like I said in the beginning, that a woman spends 6 hours drawing water for the family. It's a gut reaction this is not needed in today's world.
We have the technology, we have the means, to eliminate this unnecessary. So it's a gut reaction to something, an injustice that is taking place. Then comes an intuitionintution Oh God, I think I can do this this that is an intuition but then you muddle over that intuition, you sleep over it, you discuss it, you have a great conversation with as many people, especially people who will not agree with you and then you come to a plan as to how to put your gut reaction, your enthusiasm, into a program activity which you think has a ghost of a chance to succeed, and then I go on.
So for me, three stages.
Excellent One last question right at the back, and then I'll do a little bit of a summary. I'm Maria Barelias, and I come from Hand in Hand. Thank you very much for a very interesting discussion that sets off a lot of thoughts. We are a charity that works with entrepreneurial self-help solutions in south India.
in Afghanistan, in South Africa, now going to Brazil and China. And we are very interested both in intuition and metrics, and We don't think there is any contradiction between the two. They support each other very well. But, we have a problem with our donors. Not the private donors, not the corporate donors, but governments.
Governments are very interested in measuring, but they love to measure methods. We love to measure goals. So we have a problem in educating governmental donors and government-related donors in doing the right type of metrics. They can spend 50 pages of application on what type of trainers, which addresses, where do they come from.
We want to know talk what's happened to the women who got training, did it help them? They want to spend hours on training people in special courses. If you are going to apply for grants from the DFID in Britain you have to go to special course just to fill in the papers, so how can we as charities educate the donors to focus on the right things?
That's my question.
Oh, it's a huge challenge No, I could take a shot at this is near and dear to my heart because it is the score joke of government regulation practice which is which is input focused rather then output focused. And if you want I've written a couple of things on that. But I think it's a long tough battle because that is the mindset, set up a bunch of inputs and I'll give you the Canadian example that I rail against.
We want Canadians to watch Canadian TV, Canadian-produced TV for some odd reason, so we regulate that fifty percent of prime time television programming has to be Canadian produced. That Canadian programming gets ten percent of the viewership, so by regulating the input, number of hours of prime time, we get a pathetic output, and if we simply regulated the output, told the Canadian broadcasters you have to produce twelve percent viewership.
We would get, and by the way, we get fifty percent, because they get no viewership, they spend no money on the shows, right because they are not going to make any money on them, so we get incredibly bad shows that nobody wants to watch. If you just said you have to get twelve percent viewership, I don't care how you do it, they'd make a few great shows and we'd have great exportable shows and more viewership.
So, it's a scourge and I think it just has to be battled on every street corner.
If you think about measurement, this is very crude but. Having three purposes: of accountability, informing practice and informing the field. Government tends to be more about accountability. And just makes it hard because it's not about learning. It's not necessarily generative. It's not all the things that we're all talking about measurements should be.
It's a good fight to fight. I'm not sure that I have the answer. There was a microphone already in somebody's hand, and so i just have to do that, so this will be the last question, i promise.
It's actually not a question, so it's probably good the end anyway.
I only want questions. If it's not a question, i don't want it.
I'd just like to inform people about the work that's been going on in [xx] over the last ten xx which i think would actually be valuable to people, so if you would mind. Ok, be quick. Basically the work started initially looking at impact assessment, realized that impact assessment was not I mean not useful, not feeding to practice, and there is a lot of work then on trying to have practitioner friendly impact assessments.
So a lot of practitioner based tools. This was funded by USAID program. It then moved on to really thinking about not impact as an end result but the process by which you get to impact. And really a movement away from saying we should be expecting each organization to be assessed In their impact, but thinking about what is the process by which an organization manages its performance towards that impact.
So very much we've moved towards social performance management. And the measurement comes as something internally useful to organizations that then can be externally validated, externally but really the focus is on what are the steps to get there rather than just measuring the output. As I said, this is something people have been working on this for the last ten years in micro financing There's a lot of resources available.
If anyone wants to talk to me after this, I have a lot of resources. There's also the Seagap Micro Finance Gateway has a resource center called Social Performance, and there's a huge amount of resources there, and I would encourage people who are getting in to at least look at that 10 years of experience.
Terrific, thank you. Listen, I'm going to try and summarize what I heard panel what I've missed. But, I just scroll things down as I listen. And I think that I've heard that there are three important "P's." First is positives as in positive sum. But I heard a strong view that measurement and metrics have to add value, because it will cost so they have to add value.
We need to measure things that will actually help the business invest and be better and if our measurements don't help the business better they will probably be zero sum or negative sum, and then we've got to make sure that the businesses in question use the data and the number one use of the data is learning getting better, and that's the only way, at least as I hear it, the only way to make impact measurement, a positive sum gain.
Second "P" I guess would be precision, with this notion that we have to have the logic kind of worked out. Measurement can only follow a clear strategy. If you don't have a clear strategy we're not gonna be measuring the right thing. I mean, to have to have the problem framed properly and that we have to be able to unpack the logic and understand the logical chain so that we can measure intermediate outputs, as well as the end output.
And we can't have that unless we have certain precision in the thinking and the logic. Recognizing, of course, that the first leap maybe a gut feeling, intuition. But then that has to be followed up with evermore precision there.
The last one is pluralist. I came away feeling that we really have to take a pluralist outlook on most of these things. Some of the the measurements are going to be investee-defined because it is about their business. But, some are going to be investor-defined because that is what they've got to have to be able to get their funders to be continuing to be willing to fund it.
They have to be pluralist and sensible measures somethings and other things we won't be able to measure or be able to measure in, in different ways. If we think we can only measure the same things in all the same ways, we're going to get crummy measurement systems. We have to be pluralist in that there are multiple constituencies that we'll have to think about measuring on behalf of not just one.
Yeah, and you have to understand what the users care about, but also what the donors care about, etc. So, I heard it, I heard those three messages, positive some, precision in the thinking logic, and, and pluralist. Did I miss something important, panel members. No? That works for you Drew? Okay, it works for Drew so, it works for me.
Thank you very much for being here. And take care.
