One of our programmes of work disrupted by the COVID 19 crisis is Measuring what Matters. While day to day activity on this programme has had to pause, we can share some key messages and learning so far.
Last year SCSN worked with Evaluation Support Scotland (ESS) to bring together practitioners from across the sector to develop an evaluation framework to help those working in the field to have a better understanding about their outcomes and how to measure what matters in their work to prevent unintentional harm. The tool helps to shift the balance from measuring only what can be counted (such as number of people, number of events) to measuring outcomes for people. The framework, in draft format, is hosted on in the Building Safer Communities Hub.
Since then, SCSN and ESS have offered practitioners across sector the opportunity to test the framework. Eleven projects are part of the testing phase, which involves accessing bespoke support from ESS to apply the framework to their work, creating a simple evaluation plan, putting their plans into action and then sharing their learning through case studies and blogs.
Participating projects are:
- Home safety visits, Fife CAREs
- NHS Highland older people falls prevention
- Safer North Ayrshire partnership
- Home safety visits, Perth and Kinross Home Safety Partnership
- The Risk Factory, Edinburgh
- Port Glasgow new parents’ initiative, SFRS and Inverclyde Council
- RoSPA
- Podiatry service, South Lanarkshire unintentional injuries group
- Road safety for young drivers, SFRS Dumfries
- Fire and home safety services, Wheatley Group
- Staying active, Red Cross
The Risk Factory, a purpose-built safety education centre, are keen to test the framework because they are increasingly being asked to report on outcomes, rather than activities alone. Caroline Wilson, who runs the Risk Factory, highlights that evaluation is a challenge as they are visited by 8000 children each year and have limited contact with children when they return to school. ESS worked with Caroline to identify outcomes and indicators relevant to their work. The Risk Factory outcomes include helping children better understand risks and what to do in emergency situations.
Caroline also adapted some of their existing methods, including a feedback form for teachers, giving teachers a greater role in observing children’s learning. Caroline has gathered feedback from children on a sticky wall and post-it notes as well as using a quiz during the debrief to capture the children’s learning about risks.
Children told us they learnt:
- Electricity can jump
- Not to charge your phone or tablet overnight
- About the 105 (electrical network) number
- Instead of going back into a house that’s on fire to save someone who’s stuck, call the fire brigade
- Not to wear headphones when on your bike
- That a train travels a good distance before stopping
“The sticky wall and post it notes worked well the first few times we tried them. We plan to use a combination of these methods and see which ones work best. Time pressure is our biggest challenge; building a slot into the programme for evaluation at the end has helped us, even if it’s not always possible!”
RoSPA have also carried out some early testing of the Measuring what matters framework and found that “The process was straightforward and we found the framework served as a useful map and guide to keeping the whole process of evaluation on track. RoSPA will use again in the future and recommend it to any organisation working within the injury prevention sector” (Carlene McAvoy, Community Safety Development Manager). Click here for RoSPA’s case study.
Once these activities have resumed, we anticipate that we will be able to share further examples of their learning.
The evaluation framework is available for any practitioner working in unintentional harm/community safety. You can access it here.
If you would like more information on how to use the framework please contact Hannah Dickson/Nicola Swan.