“Be data driven” is one of eight long-term strategic goals of SAT-7, a network of Christian satellite TV channels in the Middle East and North Africa. With the support of Kindermissionswerk ‘Die Sternsinger’, CAMECO helped organise a joint online workshop from the 2nd to the 14th of December 2020, during which media impact evaluation experts from the Zurich University of Applied Sciences (ZHAW) guided the SAT-7 development and programming staff through a special method to achieve this goal.
“The Result Chain-oriented Impact Evaluation” developed by the ZHAW is a method of measuring impact by investigating the way through which programmes achieve their goals. The method not only helps determine if a programme has achieved its final objectives, but it also focusses on process tracking, thus allowing evaluations to be part of a real learning experience.
With its own studios in Lebanon, Egypt, Turkey the UK and production teams in Tunisia and Algeria, SAT-7 produces content in Arabic, Persian and Turkish, focusing on programmes for Christians in the region but also on formats related to developmental issues and children’s education. In the past, SAT-7 had embarked on regular quantitative audience surveys by IPSOS (a multinational market research company), documenting a considerable regular viewership of more than 30 million regular viewers in the region. With its specialised programming for children being particularly successful, the organisation was keen to go a step further and find out more about the quality of the programmes’ impact.
For impact evaluation of this kind, it is necessary to develop and precisely delineate the hypothesised result chains, factoring in all the steps TV viewers are expected to take (thinking, discussing, comparing, reviewing, trying out) towards the goals (changes in attitude and behaviour, and knowledge gain). Once these result chains are identified, they can be tested with audiences through qualitative and quantitative methods to determine whether or not they have taken place.
Due to the COVID-19 pandemic, the workshop was organised in six half-day online sessions with 15 to 22 participants and was limited only to the method’s introductory steps to identify the result chains. For four TV programme formats, result chains were developed jointly by the research and development department and the production staff. The design of the qualitative method, the development of questionnaires and the audience surveys to validate them, were not part of the workshop. However, the identification of the result chains is the most important step in this method given that they build the foundation for the subsequent steps and relate the survey to the programme makers’ specific questions as to whether their assumptions really work and how.
The introductory phase resulted in the SAT-7 staff acknowledging the benefit they garnered from the discussion involving their programmes’ result chains, which helped increase their awareness and made them reflect on their own theory of change. During the debriefing session, one participant mentioned that the workshop helped her refocus on making a change for the viewers instead of only showcasing change in the programmes.
That the workshop facilitated a better understanding between the R&D and programming departments was highly appreciated by the SAT-7 staff. The two departments were aided by the workshop to find a common vision to achieve their objectives despite their disparate routines, logic and constraints.
During the identification of the result chains, the participants discovered that alternative approaches, such as addressing parents who might be watching a programme alongside their children, might prove more effective in achieving some of the desired results.
SAT-7 has expressed its interest to continue the collaboration with CAMECO and ZHAW to finalise the result chains and put them to the test as soon as the current situation makes it possible. The channel is also keen on further strengthening the capacity within its staff to carry out this kind of impact evaluation.
January 2021 / MU
