Iddo Dror, ILRI’s head of capacity development, reflects on recent workshops and their effectiveness.
Stakeholders involved in Humidtropics platforms from Africa, Asia, and Central America came together for the Humidtropics Capacity Development workshop held between April 29 and May 2, 2014, in Nairobi, Kenya. The workshops covered different approaches to agricultural innovation and were measurably successful in increasing knowledge and changing attitudes.
We’ve blogged before about the workshop here, and you can have a look at the detailed workshop report, watch a short video, and browse some photos from the event.
In this post, I’d like to reflect a bit on the learning that took place during the workshop. I should preface this by saying that the more I attend “traditional” workshops, the more I wonder whether our seemingly natural propensity for holding face-to-face workshops is the best investment of time and money in terms of learning outcomes. I believe we aren’t measuring our “ROI” on this investment rigorously enough, so it is hard to answer this question.
There’s seemingly good logic for this: a typical workshop would cost far less than the cost of running a randomized control trial on different options of delivery. However, take into account the many hundreds (possibly thousands) of such workshops that take place every year across the CGIAR system, and suddenly the picture looks (very) different.
My hypothesis is that a blended learning approach, which would involve as a minimum three core pillars – namely a pre-course e-learning component, followed by (one off or a series of) short face-to-face workshops, and complemented by ongoing mentoring over a 6-18 month period following the training would probably yield a better ROI – by delivering longer lasting impacts and demonstrated higher learning outcomes – sustained changes in knowledge, attitudes and behaviors. I’d love to have a more meaningful discussion around this on occasion. Perhaps this could be a future e-discussion topic for the CGIAR Capacity Development Community of Practice? It would be wonderful to have an evidence-based discussion about this in the not so distant future!
In the interim, when designing the Humidtropics CapDev workshop, we did try to take a quick and affordable stab at tailoring the content to the participants’ needs, and assessing the effectiveness of our training.
How did we go about it? Well, for starters, we set up a training needs assessment (TNA) via a simple survey. You can read more about how different facilitators used this info in shaping / tweaking their respective sections of the course in the full workshop report.
Next, and mainly, we started the workshop with a pre-workshop quiz, and ended the workshop with a nearly identical post-workshop quiz. This provided us with an opportunity to analyze some of the (short-term) learning that took place during the workshop. While there are certainly methodological aspects that can (and should) be looked at for such quick-fire assessments, it did provide us with some interesting aspects.
Have we cracked it? Hardly. Was this a useful exercise? Definitely (at least for me). Will this lead to more debate and exchanges on how we can improve the design and delivery of workshops at the CGIAR? Here’s hoping!
Oh, and in case you’re interested, below are some highlights for each of workshop session / days.
I’d love to know what you think – thanks in advance for letting me know!
Iddo
[submit a comment at the bottom of this post!]
Day 1: Introduction to complex problems and agricultural innovation [presentation]
Agricultural problems are inherently complex. They have many dimensions, many unpre-dictable factors, and many stakeholders with potentially competing interests. Day 1 of the workshop, led by Marc Schut of WUR / IITA, aimed to stimulate new ways of thinking about agricultural problems and presented innovative strategies for tackling these prob-lems. In particular, this session introduced the Rapid Appraisal of Agricultural Innovation Systems (RAAIS) and included a RAAIS mini-workshop.
Post-workshop surveys indicate that the workshop was successful in increasing knowledge and changing attitudes. There was a shift from a technology-oriented definition of innova-tion to a systems-oriented definition of agricultural innovation. The percentage of re-spondents identifying with the latter approach increased from 33% before the workshop to 56% at the end of the workshop. Particularly notable was the participant enthusiasm for the Rapid Appraisal of Agricultural Innovation Systems (RAAIS) session; working with actu-al tools was engaging for everyone.
Participants were also asked about overall constraints in implementing agricultural inno-vation. Most commonly cited were economic, institutional, or political barriers. Biophysi-cal, technological, and socio-cultural constraints were less significant.
Day 2: Deciphering the DNA of innovation platforms [presentation]
Innovation platforms connect different stakeholders together in a network in order to achieve common goals. This workshop covered topics such as the innovation platform lifecycle, the benefits of innovation platforms, and how to monitor innovation platforms. These are new concepts for many individuals and organizations involved in Humidtropics, and many related platforms are in the initial stages. This part of the workshop focused on introducing the concepts and on making resources accessible, allowing stakeholders to approach their tasks with confidence. I led this day, with some inputs and co-facilitation from my ILRI colleague Zelalem Lema.
The pre/post quiz for this day covered topics such as the phases of innovation platforms (IP), benefits of IP, constraints in IP, and the pros and cons of existing versus new IPs. As you can see from the figure below, with the exception of the power and representation issues, all areas saw an improvement. The score on the key roles of researchers in IP, for instance, improved from 36% to 52%. On average, participants’ post-workshop quiz score was 29% better than their pre-workshop score.
This improvement, while clear in aggregate, was not shared equally amongst participants. In fact, about a quarter of participants scored (marginally) lower in the post-workshop quiz! Luckily, the vast majority did score better in the post-quiz, some scoring several times their initial score.
Day 3: Knowledge, learning and making meaning in innovation platforms [presentation]
Led by Julia Ekong from ICRA, this session consolidated the information of the first two days and explored innovation platforms further. Four frameworks for facilitators were introduced as illustrations and to capture platform dynamics.
This session faced some challenges because most of the participants were at the initial stages of establishing or implementing platforms, though some participants had significant experience in the area. The heterogeneous composition of the workshop with regards to participants’ levels of expertise meant it was difficult to evaluate what knowledge was being created (beyond technological solutions). This session could not leverage existing knowledge as much as was anticipated. In the future, this type of material would perhaps be more successful if presented to more homogeneous groups. Some more thought about quantitative metrics for assessing the learning taking place would probably also be useful.
Day 4: Reflexive Monitoring in Action (RMA) [presentation]
This session, led by Marlèn Arkesteijn (Capturing Development) discussed the three dimensions of complex problems—certainty, agreement, and systemic stability—and how reflexive monitoring in action (RMA) can support the learning processes when dealing with complex problems and innovation. The key principles of RMA were introduced, and various aspects of reflexivity were highlighted, such as change of perspective, the re-construction of reality, emergent outcomes, and turning the camera. Working with the timeline was also introduced, and the timeline was used to evaluate the workshop.
The results of the pre/post quiz indicated significantly more RMA confidence after the workshop; whereas in the pre-workshop quiz, ten participants indicated they knew only one RMA principle, that number had fallen to four in the post-workshop evaluation. Post-workshop, fifteen participants indicated on a questionnaire that RMA “really appeals to me;” only one participant felt that RMA “is not what I’m interested in.” Sixteen out of seventeen participants said, “for sure I will use some of the principles and tools.”
ILRI Clippings 
Nice article. Raises a big problem.
I find the escalating number of face-to-face international meetings organized by CGIAR initiatives, centres, research programs, projects and staff in its recent ‘reform years’ to be frankly scandalous. ‘Value for money’ is neither proven nor attempted to be proven in most of these meetings, most of which are ‘workshops’ where little real work is achieved. Even ‘value for learning’ is modest, as Iddo’s attempt to assess this in his workshop shows.
I’m sure the networking opportunities at these many meetings are large, and even important. I’m also sure much of this could be accomplished by disciplined (regular, well-planned and high-quality) communications via webex, skype, email, telephone . . . . Where can we see a breakdown of the costs of all these get-togethers versus the outputs? Does anyone in the CG system imagine that staff from successful private-sector companies or NGO groups working for developing countries meet as often as they do? Have frequent flier award miles superseded frequent communications in denoting scientific status?
I’m happy to see that Iddo and his colleagues at least attempted to assess the value in this Humidtropics meeting of theirs, but the tools appear to have been crude at best. I do think face-to-face meetings are important: so important that we ought to reduce their number in the CGIAR drastically and ensure that the meetings we do hold are professionally facilitated and EVERY SINGLE PERSON attending those meetings is held accountable for planning, participating, sharing and reporting on them well and thoroughly. Not to do this, and instead to continue to spend donor agency monies on endless international talkfests in the world’s major hotels, would appear to me to be undisciplined networking of the most casual (read ‘unproductive’ in terms of our poverty-alleviating mission) kind.
We might make bigger inroads to reducing world poverty by funding global get-togethers of our poor clients (food producers, sellers and consumers), who tend to not waste words. Or space. Or money.
[End of rant. With thanks to my colleague Iddo for spurring it.]
Excellent comments Susan, with which I fully concur. I have experienced life in the CGIAR, universities and the private sector and I would have to say that the propensity for meetings in the CGIAR, and their duration, is unsurpassed. Take the former ICW/AGM, for example; about 300 participants in the late 90’s in Washington, to a peak of more than 1,000 for Marrakech in 2005. I have sat on R&D boards that transact their business in a day, at most. Contrast this with the 5+ days of scientific tourism the centres offer to their boards of governance; one wonders about the effectiveness of these boards, especially when one looks at the examples of failed governance in recent years. Be that as it may, the suggestions you propose for more effective, and accountable, use of meetings are very valid and should be followed up.
Yes, agree with you that there is no assessment on ROI of the traditional workshops. In some cases, people attend the events just for collecting per-diem and do not have any interest or decision making and participatory skills. Some attendants have been assigned or ordered to join the workshop by their superior and have no any clue with what to be presented and discussed in the workshop. A lot of time, efforts and money have been spent for meetings and workshops, but less outcomes on the ground. Where is the value of money? I think learning by doing, I.e learning with doing is more cost efficient.
Interesting post. Face to face workshops are important, but the real learning takes place later, if/when delegates put into practice what they experienced during training. We recently did a follow up of 123 participants who’d attended seed conservation training courses from 2002 – 2013. Of the 87 respondents, 97% had used the skills and knowledge gained to improve procedures in their organisation, and provided detailed examples. 77% had trained others. We are now looking to develop more of a blended learning approach, with post-course mentoring/support provided by fellow participants as well as facilitators.
I’m not sure a generalised question like the one posted in this blog fully captures the nuances of how people learn, and thus what they know. There are many aspects to how people learn. Some people are visual learners, some learn more through-hands on activities and perhaps others prefer ongoing mentorship. Other aspects of face-to-face workshops is whether there is a language barrier for people who attend the workshops, cultural norms and whether people feel empowered to participate. Hence, it’s not necessarily the workshop concept that is the problem per se – but the way it is organised, structured and facilitated. I’m not advocating for or against workshops, but merely stating that there are additional learning factors in relation to workshops, that need to be considered before determining return on investment. Intuitively, the suggested model of pre-training, worskhopping and active post-workshop mentoring make sense as a way to maximise learning but require a large commitment of time and resources – that may not always be available. Glad to see however that workshop evaluations are being performed in order to inform future initiatives.