Policy Buddy with Engine AI

Briefly describe the initiative/ project/service; please include your aims and objectives

We know that Policy and Procedures hold valuable information, but we also know that they are difficult for the workforce to read in their entirety and digest fully, especially in a heavily regulated sector like Children and Young People’s Services. 

We worked with Engine AI to develop a bespoke Large Language Model (LLM) which looks, feels and has all of the functionality of Chat GPT4 but the key difference is that rather than looking at the internet for it’s answers the Policy Buddy only looks at national legislation and North Yorkshire’s policies and procedures. 

The Policy Buddy is designed for the Children’s Services workforce and is available through browser and mobile phone. It has an over 90% accessibility score and the user is able to interact with it by type or by voice. 

We wanted to have a bespoke LLM to benefit all areas of our service. This means if you are a new member of staff, you can ask it a hundred questions without worrying it will get tired of answering you. If you are an experienced employee or middle manager dealing with a niche query you can get instant advice and guidance or if you’re a senior manager with multiple areas of responsibility you can quickly get answers to really detailed questions. 

Above all else we wanted to be able to provide more bespoke services to our staff and to the children and families who use our services. The policy buddy will interact with you in hundreds of languages, if English isn’t your first language. It will also produce bespoke, accessible guides for children and their families about our services and the meetings we hold, complicated processes and where to reach out for support. 

The Policy Buddy also has the ability to analyse the questions asked of it and provide key themes meaning the organisation can focus on delivering training that is truly responsive to the needs of the workforce. 

The most impressive thing is that because it only looks at national legislation and our policies and procedures it is accurate, and it achieves all of this in seconds! 

We hypothesised that our employees, when faced with a question they were unsure about, would ask a colleague. Failing that, they might ask a manager and if all else fails they would seek out the policy or procedure that answered their question, finding the link to the policies and procedures, navigating a document library and then trying to find the right policy to satisfy their query or question. 

This hypothesis worried us though because it meant that asking a colleague would rely on them knowing the answer, would use both of their time and neither would know if a policy had been updated. Equally, we thought that new members of staff might not always ask multiple questions – instead considering if they’d asked their quota for the day and instead might not know an answer through a lack of confidence. Through the use of staff surveys we evidenced that this hypothesis was factual. 

As people, we have a tendency to find the document that we think answers our question and then stop looking, meaning we could miss out on complimentary or supplementary information. 

Our goals was to enable our practitioners to make the best decisions for the children and families we work with. To do this, we needed to create the right document library, comprehensive enough to know everything about Children’s Services but not so full that it risks getting confused and giving misinformation. Then to test the Policy Buddy to see if it was valuable to the workforce in all of the ways described above. We also wanted to create the right technology environment for the Policy Buddy, measure the costs and the effectiveness and once all of these were successful, to roll the Policy Buddy out to the workforce for a full test and evaluation. 

All of this work naturally meant upskilling our workforce in the art of “prompting” and taking them with us on a transformation journey. We knew that some would pick it up and run with it straight away, whilst others would need more support and guidance. 

Many organisations are experimenting with different types of AI but the Policy Buddy is an “outside the box” concept which really interested us. 

We also wanted to share our journey with the sector to encourage collaboration and learning in a new and developing area. 

What are the key achievements? 

The Policy Buddy has been a staggering success. We were able to navigate the transformation process quickly, meaning that we could keep momentum. 

We initially agreed to test it with 50 employees for 12 weeks, including over the summer holidays which we knew would impact us due to staff availability. By week 8, we knew that it was so good that we needed to roll it out to the entire workforce for a comprehensive one-year test which is what we’ve done. 

Taking only week 8 of the trial, we analysed results from 20 of the employees who used the tool. A conservative time saving estimate was that it saved them 37.7 hours collectively, but this figure could have been as high as 165 hours if we had been confident that without the policy buddy they would have gone to search for the answers in the policies and procedures. From our user testing, though, we noted that not everybody would do this. Most people said they would ask their colleagues or managers but not all would progress to going to search for answers in policy and procedure. 

• – Using the tool represents an overall time saving of 80% for the following “top tasks”
• – Looking up information 
• – Creating documents and summaries 
• – Seeking guidance and best practices 
• – Clarifying eligibility and entitlements 
• –  Requesting translations 

We have had some technical challenges hosting the Policy Buddy in our own tenancy but navigated this in the interim with Engine hosting the policy buddy. 

The most significant impact though, has been on our staff. They have given us incredible feedback about having access to the Policy Buddy. One member of staff told us “I thought I would hate it (new technology) but I love it” whilst another told us it had changed their working life……in only 8 weeks! 

Demonstrating the tool to an experienced social work team manager, she saw what it could do, stood up from her chair and jumped up and down before summoning her team to come and look at it. 

The Policy Buddy demonstrates that transformation doesn’t have to be huge to have a huge impact. 

Analysing the queries that are inputted into the Policy Buddy we can see that the top use is Looking up Information – definitions and explanations, legislative and regulatory information and procedures and processes. This is followed by creating documents and summaries, seeking guidance and best practices, clarifying eligibility and entitlements and requesting translations. 

As a result of having the Policy Buddy we have been able to upskill our staff in the art of prompting. We found that initially people searched for key words in the way they would when using a search engine, but after follow up analysis we found that the prompts were becoming more comprehensive and conversational as we taught our workforce to prompt in order to get the best results. This is not only helpful for using the Policy Buddy but will become an increasingly important skill as they engage more with AI in future tools. 

In a follow up survey 100% of users said that Policy Buddy had helped them and their practice and 100% reported that the Policy Buddy had changed the way they accessed policies and procedures. 100% also said that they would want to continue being able to use the Policy Buddy. 

Although not a direct success, an indication of how successful the Policy Buddy has been the requests from other parts of the organisation for their own Policy Buddy’s. There is significant demand for Health and Adult Services Policy Buddy and HR Buddy to be developed. What this indicates is that people are talking about it and sharing it and that others are seeing the potential benefits to their parts of the service. 

Throughout the process we have shared our journey. We have hosted multiple webinars to demonstrate the tool to the sector and had interest from across the UK and even from New Zealand! We are keen to share our work in the hope that it helps other Local Authorities in their journey into AI. 

How Innovative is your initiative? 

There has been an explosion of AI in recent years and whilst many organisations are coming to terms with which bits of AI are useful this is a really practical example of how it can be used. 

The cost is minimal (circa £6k per year) but the impact and future potential is huge. 

We have had to navigate challenges to get to the point of rolling it out, including the technical challenges of hosting it in our tenancy, navigating the procurement process, building a document library containing the right documents and helping our staff to enrol and adopt the Policy Buddy but having a dedicated development lead has helped with this, along with significant support from our Transformation service, IT support and colleagues in data governance and data security. 

Whilst we’ve heard of Local Authorities experimenting with chat bots and LLM’s we haven’t seen or heard of any similar projects to the Policy Buddy. We feel it represents a really great mix of technical capability mixed with practice rich knowledge to form a solution that is robust and effective alongside being invaluable to Social Work practice and easy to adopt, even for a workforce that isn’t comprehensively digitally skilled. 

Another way our approach has been innovative has been the speed at which we’ve been able to scope, test and scale the Policy Buddy. Utilising our transformation approach and our Emerging Technologies board we were able to navigate the technical requirements of implementing the Policy Buddy. From initial discussion to initial testing, we achieved this within weeks. From there we got from initial testing to workforce testing within 2 months. This agile approach to trying things and building on the things that show promise or potential, we’ve been able to shape the direction of our journey without getting in to over complicated routes such as full procurement of products. 

Our focus has been about the functionality rather than the product. As good as it is, there will be others and because it’s not tied into a core element of our ecosystem this means changing would be straightforward but our workforce will retain the skills involved in prompting and interacting with LLM’s. 

What are the key learning points? 

Our key learning points have been that innovation and transformation don’t have to be huge projects aiming for significant change. They can be small changes which have big effects. 

We have also learnt that there are some challenges and complexities involved with working with smaller technology suppliers, especially in new and emerging technologies. Our work with Engine has been positive because we have both been learning from the journey. Them, from working with a local authority, rather than their traditional customer base of schools and multi academy trusts, and us navigating the robust safety and security protocols of a local authority. 

This mutual commitment to finding solutions and sharing our learning has been key to the Policy Buddy’s success. 

We have also learnt a lot about our workforce. We knew, from previous work, that our workforce had a diverse range of digital maturity which makes development and adoption of technologies a really important consideration throughout. We learnt that making a product accessible and instantly valuable made it’s adoption much easier. Instead of us marketing and “selling” the product to our employees, they were sharing it with each other and asking us for access to it or asking how to get the most out of it. 

Lastly, working with Large Language Models – the key is in the quality of the prompt. Initially many people used it in a similar way to Google. They might say “statutory visit timescales” and end up with a sub-standard answer. Teaching people to prompt better has led to much better answers, resulting in more user satisfaction. This is a skill our workforce will need to keep developing as technology and AI become a bigger part of our working lives. 

Additional Comments 

This work has been a collaboration between North Yorkshire Council and Engine AI. Their business creates RAG AI models for schools and Multi-Academy Trusts but this work together was focussed on taking the principles of a RAG AI model and building it specifically for Children’s Services.