Позитивные изменения, Том 3 №1, 2023. Positive changes. Volume 3, Issue 1 (2023). Редакция журнала «Позитивные изменения»

were able to agree on the basic concepts that an evaluator must be familiar with, that is why certification is possible there. It is unlikely to happen in the U. S. because the field is too diverse there.

      Let’s talk about evaluation methodology. Have there been any changes in this area?

      Of course. Artificial intelligence (AI), big data and its sources have emerged. The pandemic brought forth great changes: online services have enabled conducting interviews and focus groups remotely. Satellite imagery has been used in new ways.

      For example, thanks to satellites, it is no longer necessary to go somewhere in the field to check whether new agricultural equipment is being used there. If we want to know how many children attend a school in a developing country, we can use satellite imagery again.

      Remote data collection, AI, big data, the Internet, and new data visualization techniques allow for better communication. It has become easier to distribute evaluation results thanks to the development of the Internet and social media, to find people to interview, to follow a person on social networks. All of these technological advances are making their way into the field of evaluation.

      So many changes and so many possibilities. Does every evaluator need to be familiar with all these new products? What are the most important qualities and requirements for a good evaluator, and can we talk about the “gold standard” of an evaluator?

      The way I see it, we have defined the “gold standard” as the ability to evaluate a program in the proper way, which meets the needs of that specific program. Methodologically, there is no “gold standard.” In any case, I do not believe that randomized controlled trials are better than other methodologies, because the program needs are the key, and because such phenomena as the pandemic, climate change, global political instability, growing numbers of refugees, famine and drought, the speed of change in the world around us, create the need for evaluation approaches that can adapt quickly to new conditions, similar to those caused by the pandemic.

      Programs need to change, and so does evaluation. There is no single method that would represent the “gold standard.” The main challenge and difficulty is adaptability and appropriateness. You need to understand what a particular program needs at a particular point in time.

      If an evaluation involves, for example, the use of advanced sociological tools, which I am not familiar with as an evaluator, I would not be able to conduct an evaluation at the proper level, would I?

      Absolutely. This is partly why it is so important to have professional associations. Today, thanks to them, to social media, to networking, you can always find someone to help you learn these methods. Many evaluations these days are done collectively, so no single evaluator would have all the necessary techniques and skills. However, if you can find the right people through an association, the Internet, or social media, you can find evaluators with all those skills.

      Would you say that most clients today require the use of advanced sociological analysis methods?

      They need to be able to get feedback from the beneficiaries, if that’s what you mean by sociological analysis. Programs should have good opportunities to get high-quality, real-world data from program participants, so they can find out what is really going on. Perhaps the most important and most common thing is that evaluators get direct, independent and real feedback from the target audiences involved in the program – how well the program is working, whether they are getting the help they need, and what can be improved.

      You need to be a good interviewer, to know how to do surveys, use social networks, and get data directly from people.

      You are a professional sociologist. In your experience, is it common for evaluators to have a background in sociology or psychology, a professional mastery of SPSS and similar programs, or is it really not necessary?

      Fortunately, there is additional training available. Much of my sociological education was strictly academic, and I had to learn evaluation by doing. When I started, there were no courses, no seminars. I had to take interviews and surveys as sociological methods and adapt them to evaluation. Psychologists had to do the same with tests, and economists with cost-benefit analysis. Working in evaluation requires the ability to adapt these academic research methodologies. This is a key skill.

      Today evaluation is not only about techniques, but also about interpersonal relationships. Evaluators must be good listeners, able to work with people from different cultures, be able to make connections, communicate well, solve problems and manage conflicts.

      Let’s talk about the so-called “soft skills.” Are they really important to the evaluator?

      That’s a good question. The American Evaluation Association has established a set of competencies that are essential to an evaluator. Of course, these include methodological competence, project management, knowledge of professional and ethical standards.

      And then there are interpersonal competencies. Evaluators must be good listeners, able to work with people from different cultures. They must treat people with respect. Be able to make connections, communicate well, solve problems and manage conflicts.

      One of my recent books focuses on the relevant skills for working with stakeholders at various levels. This is an important change, as evaluation today is not only about techniques, but also about interpersonal relationships.

      In some of your publications, you mentioned the concept of evaluative thinking and suggested sharing this type of thinking with the project team, with the client. Doesn’t this threaten the existence of the evaluators as a profession? Wouldn’t evaluation become just a management function?

      I don’t consider this a danger. You described the situation very well, but evaluative thinking in management only means that you have better communication because you speak the same language. People understand the concepts of logical frameworks, theories of change, SMART goals, different types of evaluation for different purposes, but in order to do the actual evaluation, you still need a fresh perspective, you still need people who can look at things differently.

      When people do have an evaluative mindset, they begin to appreciate having a specialist who helps them look at things through the eyes of an outsider. It is something I do often. As an evaluator, I help managers, employees and even shareholders understand how evaluators think, how they define criteria and interpret data, how they draw conclusions – but they would still need an evaluator to make their job easier, to have another pair of eyes to look at things from a different perspective. Helping to think in an evaluative way is about improving communication and understanding of what’s going on.

      So you don’t think that evaluation will become just a part of management, because the evaluator has other functions besides allowing you to understand what’s really going on.

      Yes, that’s right. Evaluators also give an additional point of view. Even where there are internal evaluators in the management, their job is to do the evaluation and make sure there’s good data and make sure people take the time to interpret it. I believe the evaluator function will remain, but the better management understands evaluation, the better communication and use of evaluation results will be.

      In your 2009 article, “The Future of Program Evaluation,” you wrote that the “gold standard” of evaluation methodologies would be hotly debated over the next few decades. Can we say that the answer to this question has changed significantly since then? Or do we still have the same leader – randomized controlled trials?

      The debate continues, but it’s becoming increasingly clear that in a rapidly changing world, we need fast-turnaround techniques that produce real data in real time. Randomized controlled trials are really not very useful in a complex, dynamic, rapidly changing world. They work best in a stable situation where there are clear fixed interventions.

      In a rapidly changing world, we need fast-turnaround techniques that produce real data in real time and provide them just as quickly to people.

      If you have to work on topics like pandemics, climate change, or civil unrest when things are changing rapidly, randomized