WEBVTT 1 00:00:02.850 --> 00:00:06.533 Hello and welcome to the video lecture on evaluation 2 00:00:06.533 --> 00:00:11.533 where you will learn about how to use many of the methods 3 00:00:12.510 --> 00:00:15.150 and tools that we're learning in this class 4 00:00:15.150 --> 00:00:18.294 for the specific purpose of evaluating 5 00:00:18.294 --> 00:00:23.294 something like a program to see how it's doing. 6 00:00:24.068 --> 00:00:28.443 Is it worthwhile? How could it be done better, et cetera. 7 00:00:32.910 --> 00:00:34.320 By way of agenda, 8 00:00:34.320 --> 00:00:38.460 I'm going to talk a little bit about evaluation 9 00:00:38.460 --> 00:00:42.450 and two types and four main topics. 10 00:00:42.450 --> 00:00:47.020 And then, give the social context of why, 11 00:00:47.020 --> 00:00:51.690 what you learn and report in an evaluation 12 00:00:51.690 --> 00:00:54.480 may not be used in ways that you think is right, 13 00:00:54.480 --> 00:00:57.540 as well as the use of social indicators, 14 00:00:57.540 --> 00:01:00.600 which are sort of data that can tell us something 15 00:01:00.600 --> 00:01:05.600 about the wellbeing or the status of a community. 16 00:01:06.990 --> 00:01:11.990 And that can also form another set of information or data 17 00:01:14.880 --> 00:01:19.880 which can be used in evaluation or in research in general. 18 00:01:21.960 --> 00:01:26.960 So, this is a research methods class, 19 00:01:30.384 --> 00:01:33.450 but evaluation isn't so much a set of methods 20 00:01:33.450 --> 00:01:38.450 as it's a purpose to understanding 21 00:01:38.610 --> 00:01:42.480 that the benefit of the impact of the effectiveness 22 00:01:42.480 --> 00:01:46.110 of some sort of intervention or program. 23 00:01:46.110 --> 00:01:48.570 There's two main types. 24 00:01:48.570 --> 00:01:52.980 First, summative is overall, what's its merit, 25 00:01:52.980 --> 00:01:57.390 what was its impact in many cases, should it be done or not? 26 00:01:57.390 --> 00:01:59.760 Was it good? Was it worthwhile? 27 00:01:59.760 --> 00:02:04.760 And formative tells how it can be done better. 28 00:02:04.890 --> 00:02:07.260 What changes could be made 29 00:02:07.260 --> 00:02:10.413 in this program that would make it more effective. 30 00:02:11.691 --> 00:02:15.780 So, here are four common topics. 31 00:02:15.780 --> 00:02:18.630 Note that they are not mutually exclusive 32 00:02:18.630 --> 00:02:21.773 and I will talk a bit about each one. 33 00:02:21.773 --> 00:02:24.573 So, first, needs assessment. 34 00:02:26.880 --> 00:02:30.720 And I would also put asset mapping hint to this, 35 00:02:30.720 --> 00:02:35.250 which tells that the beginning state of things, 36 00:02:35.250 --> 00:02:38.430 defining what are the problems, 37 00:02:38.430 --> 00:02:40.803 what are the assets that are in play. 38 00:02:42.330 --> 00:02:47.330 This usually should be done with a community partner. 39 00:02:51.870 --> 00:02:54.360 It's a good way to build relationships 40 00:02:54.360 --> 00:02:57.510 and understand their perspective on things. 41 00:02:57.510 --> 00:03:00.240 And this tends to be formative 42 00:03:00.240 --> 00:03:03.633 in that it says what should be done and how. 43 00:03:05.017 --> 00:03:08.280 Cost benefit analysis looks at, 44 00:03:08.280 --> 00:03:11.163 what are the costs and what are the benefits? 45 00:03:12.629 --> 00:03:17.629 And do the benefits outweigh what it costs? 46 00:03:17.970 --> 00:03:19.410 The cost might be time, 47 00:03:19.410 --> 00:03:22.080 it might be money, maybe be opportunity costs. 48 00:03:22.080 --> 00:03:26.640 And these are often summative. 49 00:03:26.640 --> 00:03:31.173 What's the bang for the buck? Does it pay for itself? 50 00:03:33.420 --> 00:03:38.420 Monitoring provides a flow of information. 51 00:03:38.460 --> 00:03:42.390 What's happening as we're doing this, 52 00:03:42.390 --> 00:03:45.690 not just at the end, but as we do it. 53 00:03:45.690 --> 00:03:48.210 Well, what are we learning here? 54 00:03:48.210 --> 00:03:52.320 This tends to be formative, 55 00:03:52.320 --> 00:03:55.743 what works, and what needs to be improved? 56 00:03:58.350 --> 00:04:02.515 Monitoring can be both the implementation. 57 00:04:02.515 --> 00:04:05.610 Is it happening as it was intended? 58 00:04:05.610 --> 00:04:10.610 And if not, why not? As well as the process. 59 00:04:10.830 --> 00:04:12.780 Are all voices being heard? 60 00:04:12.780 --> 00:04:15.513 Do all key stakeholders have a place? 61 00:04:17.370 --> 00:04:19.443 Are their needs being met? 62 00:04:22.530 --> 00:04:26.910 And last, program assessment. 63 00:04:26.910 --> 00:04:30.870 Does the intervention provide the results? 64 00:04:30.870 --> 00:04:33.810 Did a job training result in employment? 65 00:04:33.810 --> 00:04:36.497 Did a membership drive bring members? 66 00:04:36.497 --> 00:04:39.810 What is the impact? What is the outcome? 67 00:04:39.810 --> 00:04:44.097 And this can be either summative or formative. 68 00:04:51.240 --> 00:04:55.050 Here are a set of social indicators that are often used. 69 00:04:55.050 --> 00:04:57.480 So, this was a well-known study 70 00:04:57.480 --> 00:05:02.480 from the 1940s where, "As You Sow", 71 00:05:02.550 --> 00:05:07.550 where a researcher looked at two communities in California, 72 00:05:15.330 --> 00:05:20.330 two farm, agricultural farmer-based rural towns. 73 00:05:21.102 --> 00:05:26.102 One that had a large number of smaller farms 74 00:05:26.280 --> 00:05:29.130 and one that had a smaller number of larger farms 75 00:05:29.130 --> 00:05:32.310 that mostly had employees. 76 00:05:32.310 --> 00:05:35.941 And looked at a number of these social indicators, 77 00:05:35.941 --> 00:05:40.560 things like all these things that you see here 78 00:05:40.560 --> 00:05:44.610 and looked at how do these two measure? 79 00:05:44.610 --> 00:05:48.240 And not surprisingly, 80 00:05:48.240 --> 00:05:53.240 found that the town with a larger number 81 00:05:53.310 --> 00:05:58.310 of owner-operated smaller farms 82 00:05:58.506 --> 00:06:01.650 had scored much better in these 83 00:06:01.650 --> 00:06:05.943 than the town with just a few large farms. 84 00:06:08.820 --> 00:06:10.920 One of the key indicators 85 00:06:15.030 --> 00:06:20.030 of wellbeing here is income inequality. 86 00:06:21.600 --> 00:06:24.453 And here, you see draft, 87 00:06:25.830 --> 00:06:30.780 the degree of income inequality 88 00:06:30.780 --> 00:06:34.140 and an index of social indicators. 89 00:06:34.140 --> 00:06:39.140 And I think you can see how things stand. 90 00:06:39.480 --> 00:06:40.822 This is a few years ago, 91 00:06:40.822 --> 00:06:45.822 but gives you an idea of how things like this can be used. 92 00:06:47.850 --> 00:06:52.020 Note too that just because that you might give 93 00:06:52.020 --> 00:06:56.700 what you think are amazing results and they're not used. 94 00:06:56.700 --> 00:07:01.700 First, the researcher has more control 95 00:07:03.930 --> 00:07:07.530 for the most part of the parts of the thing 96 00:07:07.530 --> 00:07:11.190 that they're examining than the evaluator does. 97 00:07:11.190 --> 00:07:16.190 That many times, life is more uncontrollable 98 00:07:20.862 --> 00:07:24.033 than just an experiment or a survey. 99 00:07:26.550 --> 00:07:31.550 You cannot often assign treatment and control groups. 100 00:07:33.360 --> 00:07:36.960 The treatment might change in midstream 101 00:07:36.960 --> 00:07:40.170 due to lack of funding, due to social changes. 102 00:07:40.170 --> 00:07:42.508 And in many cases, the results 103 00:07:42.508 --> 00:07:45.750 that you might think you have clear results, 104 00:07:45.750 --> 00:07:49.230 but they're not used, 'cause they're not understood, 105 00:07:49.230 --> 00:07:52.537 because they are a challenge to authority 106 00:07:56.610 --> 00:08:01.610 that the vested interests might block and oppose it. 107 00:08:02.820 --> 00:08:06.810 So, even more than research, 108 00:08:06.810 --> 00:08:09.420 it may not be that just because you could come up 109 00:08:09.420 --> 00:08:14.420 with great findings in an evaluation, they may not be used. 110 00:08:15.660 --> 00:08:18.780 And this is what we did. 111 00:08:18.780 --> 00:08:23.780 Here are the key takeaways and a recap. Thank you.