Experiences Beyond the Aura Programme’s Research One (R1) Learning Intervention

The implementation of the AURA progamme at Strathmore has become a reality. By September, 2015, the participants had already engaged in various online activities such as: the diagnostic and pre-online learning session undertaken in preparation for face to face learning (The face to face learning took place at Strathmore from 30 September to 2nd October 2015.)  What follows are reflections on the approaches, processes and methodologies encountered in the learning interventions.

Blended learning approaches

The implementation of the AURA programme has involved a blended learning approach complementing face to face interactions with online experience. The main benefit of this blended learning approach is that the participants have the possibility of internalizing, and practicing, specific skills from the interactions between themselves and the facilitators. Non-mediated human interaction has a special element that cannot be bridged effectively by technology hence combining face to face with the online learning. This approach could be one of the key factors contributing to the effectiveness of the AURA programme in transforming participants’ research practices.

Benefits of a diagnostic process to customize content

In the process of the face to face delivery, it emerged that there was a difficulty in maintaining the consistent participation by senior researchers, who were of the opinion that the sessions did not address their specific capacity building concerns but were more focused on the needs of young scholars.

An attempt had been made to ensure that the content would appeal to a mixed audience, but it was difficult in practice to achieve this objective without a deliberate content re-design prior to running the intervention. This highlights the benefit of the diagnostic process in customizing content and demonstrates that poor participation in this process may have contributed to the problem of the undifferentiated content arising in the first place.

Using an experiential learning facilitation methodology

The experiential learning facilitation methodology that was employed by the AURA team is the enduring strength of the face to face intervention. This methodology is learner-centred and successfully engaged the faculty, who themselves are teachers in their own disciplines and can be highly critical of traditional teacher-centred approaches.. The assessment of learning at the end each day, using the reflective journal, is a best practice tool that helped participants internalize and document their personal learning. The R1 face to face learning is a valuable precursor to, and has raised expectations for, the forthcoming teaching intervention which will be rolled out in 2016.

The other appealing, and motivating, aspect of the AURA programme learning interventions is the practical assistance the participants received through detailed feedback on their assignments from enthusiastic professors in the AURA network. An opportunity presented by the GKEN4Africa 5th International Multidisciplinary Conference 2015  for two participants from Strathmore to showcase their “work in progress” – this opportunity has further fuelled the excitement of other participants.

Future considerations

Looking to the future, it is anticipated that individualised attention to the participants will be critical in achieving the anticipated programme results, such as publications. This is a strong learning point that Strathmore University would like to carry into its internal capacity building programmes for teaching and research.

Stephen Ng’ang’a and Cavin Opiyo are based at Strathmore University in Nairobi, Kenya.

When E-learning is Not Like a Duck to Water

Photo:  creisi,  Pixabay

Creative Commons CC0 Public Domain License

The advancements of information and communication technologies have enhanced teaching and learning practices across the globe including Africa. Through the African Universities' Research Approaches (AURA) programme, Muhimbili University of Health and Allied Sciences (MUHAS) is also now using these technologies to deliver its teaching and research intervention training programmes to faculty members and students. However, the main question remains whether MUHAS researchers and students are prepared to fully embrace these technologies to enhance their teaching and learning and research practices amidst the multiple challenges they face?

Research One (R1) Learning Intervention

Through the AURA programme from September to October 2015, MUHAS rolled out AURA's Research One (R1) learning intervention.  This involved a pre-online session, a face to face session and a post-online session.  Research One (R1) was followed by an online regional event on the “Future of research in the 21st Century” organized from 2-4th November 2015.  

A high participant attrition rate was observed during the online training sessions mainly because participants were instructed to join the training individually in their offices. Most participants did not know how to use the e-learning platform due to lack of skills and also due to their mind-set towards online e-learning systems.

This observation may not be so surprising as many African universities, including MUHAS, face a number of challenges related to use of e-learning platforms including: frequent power outages,  the lack of the online learning culture, the mind-set of teacher-centred  (versus student-centred) on the part of many of the senior faculty, as well as lack of ICT skills, unreliable internet services and many other challenges. 

The low level of attendance of participants to the online training sessions at MUHAS calls for a need to build the capacity of faculty members and to cultivate a culture of online learning  within our institutions. However, as short term measure, this experience forced the MUHAS project coordinators and ALIRT teams to find a temporary solution to avert this situation. This short term measure included facilitating the online learning sessions collectively in a single room until participants were comfortable with attending online training individually in their offices.  

This solution however had cost implications in that MUHAS would need to have some equipment to conduct online learning to researchers collectively - headphones with speakers for each participant, - in order for them to fully participate in the AURA programme online activities.

Conclusion

The key lesson learnt from this experience is that before one adopts new technologies and strategies,   you will need to prepare and assess end users in terms of  their  competence or attitudes. Mitigate these deficiencies early on to ensure achievement of project objectives.

Dr. Doreen Mloka is a Medical Microbiologist/Molecular biologist. She is a Medical Education Fellow and the Director of Continuing Education and Professional Development at Muhimbili University of Health and Allied Sciences (MUHAS), Tanzania and co-coordinates the African Universities' Research Approaches (AURA) programme at Muhimbili University of Health and Allied Sciences (MUHAS).

Professor Lwoga holds a PhD in information studies from the University of KwaZulu Natal, South Africa. She teaches and supervises both undergraduate and postgraduate students. She has facilitated a number of workshops and short courses.  She has published widely and has presented over 30 research papers in both international and local conferences.  Professor Lwoga currently co-coordinates the African Universities' Research Approaches (AURA) programme at Muhimbili University of Health and Allied Sciences (MUHAS), Tanzania, together with an additional four projects working with international partners in Sweden, South Africa and USA.

Experience in Enlisting Participants

The findings of the AURA background study in southern African universities, "Building Research Capacity: EnablingCritical Thinking Through Information Literacy in Higher Education in Africa" (Hepworth, Mark; Duvigneau, S. IDS. 2012), documented the realities of a burdened faculty with heavy teaching loads and little practical time to undertake any research activities. The findings of the study are applicable at Strathmore University, especially in bringing staff into the African Universities' Research Approaches (AURA)programme, as it can be difficult to get commitments from staff who already feel over-burdened.  As the project coordinators (PCs) for AURA at Strathmore, we had two options for enlisting participants:  a formal process and a targeted approach.

Formal process: pros and cons

In the formal process, the management direct the faculty to participate in the programme and are required to brief the appointing authority on their progress. The advantage of the formal process is that the recruitment of participants is simplified and there are usually a good number of participants in the specific programme. However, the downside of this formal approach is that it does not tap into the intrinsic motivation of the participants, hence the personal commitment of the participants may be lacking. Occasionally, the organizers of the programme events may have to keep appealing to authority for participant mobilisation.

Targeted approach:  pros and cons

In the targeted approach, faculty are offered the opportunity to participate in the programme on their own volition. This is done by arranging open fora where faculty are inducted into the programme and those interested then sign up. The advantage of this approach is that those who sign up are usually self-motivated and are more intensively engaged in the project. The downside is that the number of participants may be lower than anticipated.

Targeted approach at Strathmore

We opted for the targeted approach having carefully considered the deep level of commitment in time and involvement from the faculty demanded by the AURA programme.

The targeted approach requires a continuous level of personal engagement with the individual faculty to retain enthusiasm among the participants and to grow the number of participants to a critical level. Personal engagement implies: managing the different levels challenges that create barriers to effective participation of busy faculty. For example, scheduling events properly, effective time management, clear actionable communications amongst other factors.  In effect, as the PCs for the AURA programme at Strathmore, this targeted approach required both of us to take on the role of champions of AURA to encourage and foster the personal engagement and enthusiasm of faculty and to enlist participants.  For us, this approach was the better of the two options as it meant free to respond to the programme needs as required without involving an authority unnecessarily.

Lessons learned

The lessons from the Strathmore approach to enlisting participants as outlined above highlights the benefit of the role of champions who can keep the interests of participants alive without necessarily appealing to an authority.  The role of champions is a critical role because it is a key way of engaging individuals’ interest once an intervention has passed in the on-going programme and for building a critical mass of support within an institution.

Stephen Ng’ang’a and Cavin Opiyo are based at Strathmore University in Nairobi, Kenya.

Managing Internal Bureaucracy to Facilitate Project Inception

Strathmore University, like all corporate entities, is structured in a manner to facilitate the achievement of its objectives. Usually, and over time, institutions develop a culture informed by experience and new knowledge. In the case of external engagements, the partnerships in Strathmore University have grown rapidly and this growth has come with new management procedures and regulations both to govern and to ensure value addition. These procedures can sometimes slow down project conceptualisation and inception.

In the case of the African Universities' Research Approaches (AURA) programme, this process was further complicated because the programme has a design that is co-creative, hence aspects of programme design are left open to be developed in collaboration with participating partners. This more open approach in the programme design made it challenging to sell as an idea to university management, and potential beneficiaries, who were not so familiar with a co-creative approach.  This led them to perceive the co-creative approach as ‘lack of clarity’ in the design at the onset which was likely to pose a greater risk.

Establishing internal collaboration

The first step in taking the AURA collaboration forward internally was to establish, in June 2015, a bridge by having the Strathmore Learning Teaching Services Department (LTS) and the Research Office as the key drivers of the programme.

LTS has a mandate on faculty development and this was a key pillar in the AURA application process. The Research Office has a mandate on research management and therefore plays a key role in contributing to the achievement of the strategic goals of the University in the area of Research.

The AURA programme provided an opportunity to establish an internal collaborative venture between the two departments and, in effect, to situate the programme at the confluence of teaching and research objectives.

The concept of the ALIRT team is a collaborative one, bringing on board the academia as well as critical service departments of the Library and IT. This collaborative platform between Strathmore project coordinators and the ALIRT team added credibility to the inception efforts for implementing the AURA programme. The visit by the AURA team for the institutional assessment also provided an opportunity for advocacy with the high level university management team.

Conclusion

The university management and the participants in the AURA programme developed high expectations from a collaboration that has the prospect to break new grounds in teaching and research at Strathmore. The potential benefits of the AURA programme are reflected in these internal collaborations as well as through cordial external engagement with the consortium partners for the programme.

Stephen Ng’ang’a and Cavin Opiyo are based at Strathmore University in Nairobi, Kenya.

AURA e-Newsletter Issue 1, January 2016

The African Universities' Research Approaches (AURA) Programme is pleased to announce a monthly e-Newsletter. Catch up on  all the latest information and updates concerning our work. Also get more information on how to access all our online learning resources.



What to expect in Issue 1:

• News on the AURA Research One (R1) Intervention - the first of the research capacity interventions (focusing on research philosophy, epistemology and research orientations) which 
  was took place at Strathmore University and Muhimbili University of Health and Allied Sciences (MUHAS) during September to November 2015.
• Information about AURA Research Two (R2) Intervention - the second of the research courses; this one focuses on researcher-led (analytical) research methods.
• An update on the Researchers of the future, Regional Learning Event which took place online in November 2015.
• Highlights of the Global Knowledge Exchange Network (GKEN) December 2015, where AURA was represented in Ethiopia.

Also the e-newsletter will give you more information about AURA's learning management system where you can find a variety of resources including lesson plans, videos, presentations and handouts.

Download a printable version

Sign up to receive the AURA monthly e-newsletter.

Teaching and Learning: Which way to go in Africa?

Teaching and Learning in Africa: Is there a connection with poverty eradication?

There has been a silent revolution in teaching and learning methods over recent years throughout the world. While the more developed countries have been quicker to adopt new methods and new technologies in teaching and learning, from apprenticeship to digital methods, it has been more of a roller-coaster experience for other parts of the world.  Large populations in the developing world, particularly in Africa, for example, continue to invest and to use traditional teaching methodologies that are passive and do not permit unhindered critical thinking so are likely to miss out on new methods and new technologies in teaching and learning.

While this is, perhaps, common knowledge, the many problems that afflict the developing world have never really been extensively interrogated, or possible recommendations to redirect donor or partner resources into efforts that would promote these new learning methodologies. New knowledge may then translate into an ability to address common maladies, like poverty and disease, in a cheaper and a more sustainable manner.

GKEN4AFRICA 5th International Multidisciplinary Conference and Workshop, Addis Ababa, December 2015

The recent Global Knowledge Exchange Network (GKEN) workshop held in the Addis Ababa, Ethiopia, in December 2015 had  a stimulating theme that interrogated knowledge and innovation. A multidisciplinary galaxy of academics from around the world, who are originally African, gave a good exposition of issues which through new methods of learning and research may drive a paradigm shift in this continent.

The African Universities' Research Approaches (AURA) programme, through a rigorous scientific process, is determined to explore these new possibilities of enhancing teaching and learning and increasing capacity in research.

These are definitely the game changers in our New Africa.  Watch this space!

Dr. Kahiga Titus Muhu specializes in Clinical Pharmacy.  He is a Lecturer at Kenyatta University,School of Medicine, Nairobi, Kenya.

"Many cooks may not spoil the broth but may enhance it": A multidisciplinary approach to education

Photo:  Vocalcoach, Pixabay

Creative Commons CC0 Public Domain License

Adopting and modifying the best practices of  teaching and learning may be the key to fostering innovation in teaching and learning.  The concept of borrowing innovative experiential and interactive teaching and learning methods from unrelated disciplines may be a novel strategy to ensure that graduates acquire the competencies they require.

This novel approach was this year's Global Knowledge Exchange Network (GKEN) 5th Conference theme. The conference created a platform for professionals from a multitude of unrelated disciplines to learn and share experiences of  best practices of teaching and  learning that foster development of  competencies in graduates.  Educators got the opportunity to learn from each other's discipline in such a way that it facilitated the establishment of collaborative multi-disciplinary networks and groups dedicated to enhance teaching across unrelated disciplines.

As educators, we view this approach as an inevitable extension of the global trend of fostering collaborative multidisciplinary research from seemly unrelated disciplines. As educators we have forgotten that good research comes from acquiring good research skills.  Adopting the GKEN model will not only increase  the  number of educational methods available to us, but also the opportunity to develop better educational methods through interdisciplinary collaboration. 

In our view that is the future of educational innovation, in ensuring we have the best teaching methods to impart the professional competencies required.

Professor Lwoga holds a PhD in information studies from the University of KwaZulu Natal, South Africa. She teaches and supervises both undergraduate and postgraduate students. She has facilitated a number of workshops and short courses.  She has published widely and has presented over 30 research papers in both international and local conferences.  Professor Lwoga currently coordinates the African Universities' Research Approaches (AURA) programme at MuhimbiliUniversity of Health and Allied Sciences (MUHAS), Tanzania, together with an additional four projects working with international partners in Sweden, South Africa and USA.

Dr. Doreen Mloka is a Medical Microbiologist/Molecular biologist. She is a Medical Education Fellow and the Director of Continuing Education and Professional Development at Muhimbili University of Health and Allied Sciences (MUHAS), Tanzania.

AURA developing a framework for strengthening a research culture

The African Universities' Research Approaches (AURA) programme completed its first research capacity intervention, earlier this year in Nairobi and Dar Es Salaam working with African universities to enable faculty to do more research and develop researchers of the future.

Photo:  StockSnap-894430, Pixabay

Creative Commons CC0 Public Domain License

How the study of people's information behaviour informs the programme

The AURA programme's strategy draws on the study of people's information behaviour. In particular the assumption that information behaviour is intrinsically related to how we navigate and at the same time co-construct our experience. This shared experience, although open to interpretation, is reflected in our thoughts and informed and communicated through our actions including speech, writing, the creation and use of images, and with an array of technologies. The 'experience' that is the focus of the AURA programme is our learning experience, in particular that learning experience associated with conducting an inquiry into a particular topic or problem, i.e. conducting research and becoming informed.

As a consequence the project draws on theories of people's information behaviour that explores and documents the factors that influence and drive information behaviour, some generic some context specific. The sub-domain, information literacy, i.e. concentrating on the cognitive, emotional and behavioural capabilities relating to being informed; where people identify their information needs and appreciate and effectively utilise their socio-technical environment that can enable them to become informed or may be used to manage or communicate their 'research' i.e. the product of 'finding out' and learning.

Drawing on current approaches within higher education and research

The AURA programme also draws on a knowledge of current approaches to become informed within, on the whole, higher education and research environments.  These include an overview of the ontological, epistemological orientations and approaches that are taken. These include the broad ontological distinction between Cartesian and non-Cartesian and broad epistemological viewpoints within the post positivism or interpretivist paradigms such as social constructivism or critical realism or phenomenology.

Furthermore the distinction between citizen-led, highly participative or researcher-led forms of research (which are associated with distinct epistemological orientations) are highlighted.

The theoretical is however grounded in relation to the practice of conducting research and the process of developing research questions that will lead to funding from donors and research sponsors as well as getting published in respected publications plus ensuring that research is communicated effectively and has impact.  Hence, mapping the researchers network and identifying stakeholders (other academics, policy makers, publishers, international organisations etc.) and also mapping the knowledge, information and data landscape is seen as fundamental. The academic's research is therefore seen as integral to the wider socio-economic-technological context. Thus the research has to be contextualised and justified in terms of potential impact on society as well as to be clear about the academic contribution the research makes. In fact successful research is likely to stem from this contextualisation process. Without this contextualisation process, research is less likely to succeed or be seen to contribute.

Using pedagogical theory and practice to enable learners to engage with interventions

Another key component of the AURA programme is pedagogy. Pedagogy is the theory and practice of enabling learning and education. It is therefore intrinsic in that it enables us to implement interventions that are based on strong pedagogic theory and will enable learners to engage with the subject i.e. research and how to go about research. The current teaching of research, although there are examples of excellent practice, in many cases the learners do not engage and benefit little from research methods courses. Plus they tend to be patchy in their coverage. In the past the teaching of research has tended to take a teacher-centred approach rather than participative and experiential. It also tends have taken on board to a limited extent recent developments, for example, the importance placed on participative research or the blurring between quantitative and qualitative data or the increased emphasis on holistic/systemic approaches in addition to the more traditional analytical approaches. These changes are related to an increased appreciation of the complexity of many problems that we face and that need to be tackled from a multi-disciplinary, mixed methodological perspective.

Furthermore drawing on current knowledge about pedagogy enables the modelling of methods and techniques that can be applied in the universities and the learning stemming from AURA to be cascaded and institutionalised. It also enables current practices to be adopted such as reflective practice or the use of social media or blended learning which draws on different modalities of delivery and learning.

Concluding reflections

Strengthening research capacity within an institution is however challenging. There are organisational factors that impinge on the project. These include their: history, culture, goals, organisation and infrastructure. One key challenge is the large numbers of students and the shortage of staff. A host of different players operate in this context: faculty/researchers, students, the research office, the library, continuing professional development, the graduate school, deputy vice chancellors of research and teaching, the ICT providers. Each play a role and have an interest in certain aspects of the 'problem' i.e. strengthening the research culture and the capacity to do research. As a consequence AURA has involved representatives from these groups in each African university. They form the core group of partners and co-developers.

 This blog post was originally shared by Professor Mark Hepworth on LinkedIn.

Dr Mark Hepworth is Professor in People’s Information Behaviour, at the Centre for Information Management at Loughborough University, Loughborough, UK. He champions access to information for underrepresented groups in society, is passionate about research philosophy and methodology and about strengthening people’s capacity to conduct research in educational, workplace and community based contexts. He specialises in participative, qualitative research.

Read more at Mark Hepworth's Blog and on Twitter: @kampalamark.

Critically Reflecting in Practice

Photo source: Google images

Introduction

In this blog post I consider what critical reflection in learning practice entails and how it can foster learning, why it is useful and how it can be applied. Finally I explore its meaning in my own practice as learning practitioner.

What is critical reflection?

Critical reflection enables a learner to analyse what has been learned and how learning fosters self-development. It is in light of these two analysis aspects that an importance is placed on critical reflection in professional development of teaching practices. Cranton (1996) defines critical reflection as the “process by which adult learners engage in and identify the assumptions governing their actions, locate the historical and cultural origins of the assumptions, question the meaning of the assumptions, and develop alternative ways of acting”. Brookfield (1995) adds that part of the critical reflective process is to challenge the dominant social, political, cultural, or professional ways of acting. Through the process of critical reflection, adult learners come to interpret and create new meaning and actions from their experiences and are able to create a learning strategy for life-long learning.

How can it be fostered in the classroom?

Brookfield (1988) identified 4 activities central to becoming critically reflective: Assumption analysis describes the activity a learner engages in to bring to awareness beliefs, values, cultural practices, and social structures regulating behaviour and to assess their impact on daily activities. Assumptions structure our way of seeing reality, govern our behavior, and describe how relationships should be ordered. Contextual awareness is achieved when adult learners come to realise that their assumptions are socially and personally created in a specific historical and cultural context. Imaginative speculation provides an opportunity for learners to challenge prevailing ways of knowing and acting by imagining alternative ways of thinking about phenomena Cranton (1996). The outcome of assumption analysis, contextual awareness, and imaginative speculation is the fourth activity referred to as Reflective skepticism the questioning of any universal truth claims or unexamined patterns of interaction. In other words, critical reflection enables us to locate ourselves within a broader social context; to understand our values, beliefs, and biases and to assess our learning so that our learning informs our practice.

Good reasons for incorporating reflection into your own practice

The value of reflecting on practice recognises the importance of taking action on the basis of assumptions that are unexamined. Critical reflection is one particular aspect of the larger process of reflection. To understand critical reflection properly we need first to know something about the reflective process in general. Brookfield (1995) highlights that critical reflection is important as, "It helps us take informed actions that are based on assumptions that have been carefully and critically investigated. It helps us develop a rationale for practice not only grounds our actions, but also our sense of who we are as educators in an examined reality. It rounds us emotionally when we neglect to clarify and question our assumptions, and when we fail to research our students, we have the sense that the world is governed by chaos. It increases democratic trust through learning whether independence of thought is really valued, or whether everything depends on pleasing the teacher. They learn either that success depends on beating someone to the prize using whatever advantage they can, or on working collectively".

How can critical reflection be applied?

The African Universities Research Approaches (AURA) programme, fosters shared learning across partner institutions. Participants on a learning programme are encouraged to reflect, ask questions and draw on concepts that can help to understand learning in their own practice. Participants are introduced to an experiential learning model developed by Kolb, 1984, to identify, investigate, reflect on and report on a learning dimension of their work. The ORID questioning process which is based on the Kolb’s experiential learning cycle allows participants to apply simple steps that supports their thinking in a critically reflective manner. The steps / framework outlined can assist the reflective writing process. 

• (Analysing) Analyse how it made you feel? “I feel…” 
• (Evaluating) What did you conclude from this experience? “I concluded…” 
• (Creating) What will you do differently now? “I will do… in the future”. 

Through describing a critical incident arising from the practice learning environment the participant is able to make sense of what has been shared. During an AURA learning intervention, learners engage in a reflective practice within group and facilitated online discussions pre- and post- the face-to-face intervention and encouraged to continue reflective writing when away from the learning environment. Reflective writing enables learners to pursue the critical reflections on a deeper level and confront the challenge of explaining their research ideas. During the online and face-to-face interventions reflections underpinned their understanding of theory and course content and to link experience and knowledge.

Why is critical reflection important to an educator?

Critical reflection blends learning through experience with theoretical and technical learning to form new knowledge constructions and new behaviours or insights. Reflection is necessary to develop the skills to become lifelong learners. Through my observer role and own view point I feel I am in a good position to develop a learning strategy from my viewpoint / identity as a learning practitioner. Through the AURA programme I am able to observe a case study of my own ‘critical incident’. This may not always be the case. If tackling a learning dimension in which I have little experience, more research would need to be done, for example finding out about others experience as a learner / and or learning practitioner in relation to the learning dimension in similar contexts. My own approach as a learning practitioner are informed in part by experience (self or others) and in part by the relevant and prevailing theoretical and distinct conceptual perspectives.

References:                                                    

Brookfield, S. 1995. What it means to be a Critically Reflective Teacher in “Becoming a Critically Reflective Teacher,” San Francisco Jossey-Bass.

Brookfield, S.D. 1990. Using critical incidents to explore learners’ assumptions. In pages 177-193 of J. Mezirow (Ed).

Brookfield, S. 1988. Developing Critically Reflective Practitioners: A Rationale for Training Educators of Adults. Training Educators of Adults: The Theory and Practice of Graduate Adult Education. S. Brookfield, Editor. New York: Routledge.

Cranton, P. 1996. Professional Development as Transformative Learning: New Perspectives for Teachers of Adults. San Francisco: Jossey Bass.

Kolb, D. A. 1984. Experiential Learning Experience as the Source of Learning and Development. Prentice Hall Inc. Englewood Cliff, New Jersey. USA. Retrieved from the AURA programme course material.

How to be critical when reflecting on your teaching (2015). Retrieved on 23 November http://www.open.edu/openlearn/education/learning-teach-becoming-reflective-practitioner/content-section-2.1

Learning through Reflection (2015). Retrieved on 23 November http://www.nwlink.com/~donclark/hrd/development/reflection.html

Myrtle Adams-Gardner is the Training Quality Coordinator representing the South for the African Universities’ Research Approaches programme. She is experienced in mentoring and coaching, pedagogies and assessments of learning. She has been involved in the development of capacity development programmes promoting teaching and learning capabilities in Sub-Saharan Africa.

Challenges of monitoring and evaluating: attributing causality

Photo:  DarkoStojanovic, Pixabay
Creative Commons CC0 Public Domain License

In popular culture the laboratory is a place with largely negative connotations: the word conjures up images of beakers, test tubes and white coats. Yet, as a social scientist I look upon it with a slight sense of envy. This is because the laboratory represents a degree of experimental control rarely possible in the social sciences.  As a monitoring and evaluation specialist on the African Universities Research (AURA) programme, however, ours is a study of human behaviour in its natural surroundings. These natural surroundings happen to be diverse: the first year of the programme focuses on multiple stakeholders at several universities spread across Eastern Africa. This familiar terrain of bureaucracy and institutional politics ensures our focus is far removed from the controlled environment of the laboratory. My argument is that the further your research moves from the laboratory the harder it becomes to exercise control, and therefore the harder it becomes to attribute causal relations between variables. This is a problem given the prevalence of linearity assuming quantitative indicators in donor reporting, particularly in logframes.  So, given this degree of complexity in programmes such as AURA, it’s sensible to adopt mixed methods approaches. This is the first in a series of blogs I will be writing to make this case using examples from my experiences as an evaluator on the AURA programme. 

Measuring short-term impact and long-term value
AURA is a capacity development programme aimed at shaping behavioural change. This will be achieved, in part, through the AURA suite of capacity development courses. One of the challenges of the evaluation effort lies in showing that these courses have been effective. This is done at different levels for different attributes. So let’s first consider an example in which the objective is to measure improvements in the research skills of our workshop participants. Firstly, this requires a benchmark, something to compare progress against. So, before participating in any sort of workshop, the participant takes a combination of test and self-assessment questions that give us an idea of their current skills set. The scores on these questions are then compared to scores obtained from a similar exercise immediately after the workshop. The difference between them is what we report in our logframe; this is the increase in skills measured, and typically it is expressed as a percentage. How well this method works depends on how sophisticated the test questions are and also how well it’s supported by data obtained from other, typically qualitative, methods. At its best this builds a strong case for attributing an increase in skills to the intervention. But this snapshot covers only a short space of time: the immediate pre and post intervention periods. Yet, the real value of the AURA programme will depend on how these skills develop over a longer time period and what’s done with them. This will determine the impact the programme has had; and for good reason this is increasingly what donors are focusing on.

Correlation does not equal causation
Let’s carry on with our example. Our participants, having completed their workshop go back into their everyday lives at their respective institutions. The easiest and perhaps least rewarding evaluation exercise might be to repeat the same test given to them previously, but taken several months after the workshop to determine whether they’ve sustained their skill set. However, we are now into less secure territory as far as causality goes. What made the first assessment so useful is the very thing that hinders this one: the lapse of time. Earlier we could more confidently attribute the increase in skills to our intervention, because the testing was done immediately before and after it. Now, however, several months have passed and each of our participants have done different things in that time, some of these activities might have advanced these skills further, while others might have hindered them. So even if we have another snapshot of the participants skill set, several months down the line, how do we know whether any shifts are because or in spite of our intervention? Even if we found an upward surge in our participants skill set, how can attribution be safe given the possibility of so many other variables? Perhaps we could correlate the rise of skills with the amount of post-workshop interaction we have had with the participant. This is the point at which I hear the voice of my statistics tutor who reinforced over and over in his class: that correlation does not equal causation. Anyone who’s studied the social sciences has their favourite examples of two independent patterns that are strongly correlated, and my favourite come from military intelligence analyst Tyler Vigen, in his book Spurious Correlations. Here we learn that the per capital consumption of mozzarella cheese is strongly correlated with the number of doctorates awarded in civil engineering; and that the greater the consumption of sour cream the more motorcycle riders killed in non-collision transport accidents. It makes you wonder: are the patterns I’m finding between our interventions and a participants long term progress just as spurious as these examples. 

Towards a mixed methods approach
This needs to be investigated further. The evaluator has a choice to make at this point, to either move towards or away from the laboratory. The move towards is taken by the econometrician, only this time it is a figurative laboratory, and instead of using actual instruments of control we use numerical values in a regression analysis. A regression analysis is a tool used to investigate the relationship between different variables, which in our case might be a relationship between high performance on one of our courses and, say, an increase in earnings. However, there are several other factors that might explain an increase in earnings that are independent of the AURA programme, such as the participant’s prior levels of education, income differentials between different geographical locations and different sectors of the economy.  The aim of the regression equation is to zone in on the relationship between the variables in which we are interested while controlling for those that we are not. This approach, however, relies on being able to ascribe numerical values to your control variables. This is not always possible. To use our example further:  what if the difference in earnings is due not to what we’ve hypothesised or tried to control for, but rather simply down to the participant having the right family connections? It’s less clear how we can place numerical values on things like patronage, and knowing this and conceding the point allows us move away from the laboratory and embrace a mixed methods approach. 

Jagdeep Shokar is Monitoring and Evaluation Advisor for the African Universities’ Research project. He is an M&E specialist on the evaluation of capacity development programmes promoting research and communications capabilities. He has been involved in the monitoring and evaluation of programmes held across South Asia and Sub-Saharan Africa.