From February until June this year, fourteen institutions ran a pilot of Jisc’s digital capabilities discovery tool (beta) with members of staff. If you are interested in joining an open pilot of a new version of the discovery tool, you can sign up on behalf of your institution here.
A quick recap on the discovery tool (beta) pilot
The discovery tool allows individuals – working under their own initiative – to reflect on their digital confidence. Through a series of questions they explore different areas of the Digital Capability Framework. Through their answers they are made aware of digital skills they already have and new ones they might try. 
Feedback includes a digital capability profile, a summary of suggested actions to build confidence further, and links to more detailed resources.
In this pilot version, users were asked to provide their own feedback on the experience at different points in time: immediately after completing the survey, after receiving their results, and two weeks later.
Organisations receive summary data about the number of staff in different areas completing the discovery tool, and broadly about their capability scores (remembering that these are self-assessed and so reflect confidence rather than competence). Individual staff are never identified, but pilot sites have been giving their staff opportunities to share and talk through their results, and to identify further development needs they have or interests they want to develop. We have also been helping organisations to take a broad approach to developing the digital confidence and capability of their staff, with resources such as the digital capability profiles and organisational guide. Many pilot organisations attended our community of practice launch and shared their experiences there. 
We have also been following up with contacts in the pilot institutions and evaluating their experience with the discovery tool.
Findings from the pilot evaluation
Overall we collected 907 responses from a pop-up question completed immediately after the reflective questions, and 265 responses to a more reflective survey completed at the end of the process (about half were completed after a two week delay). Feedback from the survey was very positive, with over 70% choosing to ‘agree’ or ‘strongly agree’ with all six positive statements suggested to them. The closing summary of their personal digital capabilities was rated the most positive element overall. From qualitative feedback, the three benefits most often cited by users were:
- Analysis of personal skills (‘strengths and weaknesses’)
- Better understanding of digital capability as a concept or group of concepts
- Targeted feedback/resources for development
However, there is considerable room for improvement in some areas of the user experience, especially the interface through which the questions and feedback are delivered. Many users found the discovery experience ‘somewhat useful’ but had suggestions for improvement. Only 36% followed up the resources suggested to them at the end of the process, although these users found the resources to be useful (90%) and relevant to their personal needs (80%). Around a third of users said they were planning to do something new or different as a result of the discovery experience.
Organisational users were positive about the way the discovery tool had raised awareness of digital capability, encouraged conversation, and help staff to understand the range of capabilities they might need in their digital practice.
- The great benefit for us in being involved in the pilot is that it has increased awareness of the need to engage with these capabilities.
- Using the diagnostic and the other resources helps grow the understanding around the topic
- Provided opportunities for digital discussions which can then lead on to more engagement
- Using the pilot as a ‘fun’ way into digital capabilities and getting the conversations going with key influencers
However, organisational contacts were less convinced that the process had allowed them to identify staff strengths and weaknesses clearly enough. Some wanted to be able to customise the questions and feedback to suit their own systems, while others wanted a finer-grained discrimination of staff in different roles. There are clearly some difficult trade-offs between generic and specific digital skills, which need to be managed in any tool of this kind. There are also tensions between personal engagement – a tool that is self-directed and rewarding to use – and organisational development – a tool that can help identify the development needs of staff. These are questions we have shared and discussed with the pilots.
Some unanticipated benefits were identified, such as:
- Saves us time and effort rather than having to develop our own materials
- Offers the credibility of Jisc’s backing and support
- Signposts high-quality resources (there were only positive comments about these)
There were a wealth of useful suggestions from the pilot organisations – both through the evaluation process and through engagement in webinars and the community of practice. These are summarised in the following section as recommendations, many of which we are following up already, and all of which will be put in place as the pilot continues.
In July we also tested a specific set of questions for FE & Skills Teaching, which received a positive response from our pilot institutions. Feedback from this evaluation is informing the development of teacher-specific questions and feedback for the open pilot version.
Recommendations for further development
We can see that there is a demand for a discovery tool, but that we need to build and improve on the first pilot if it is to be really beneficial. We are therefore running a second, open pilot with a completely new platform and further development of the content and concept. You can sign up for the open pilot here.
- Clarify who the discovery tool is aimed at and how it can be of value: the tensions between personal self-reflection/self-development and organisational planning need to be worked out. This being addressed in how we explain and contextualise the tool in the future.
- Modern design interface with better chunking of content, visual design, navigation etc. This has already been addressed with the decision to use Potential.ly as the platform for delivery.
- Develop a version for students. This was an overwhelmingly popular recommendation and is being pursued in the open pilot.
- Work on an institutional data dashboard with advice on how to use the discovery tool for organisational development.
- Provide a coherent framework of guidance and resources for institutions, including quick links to the capabilities framework and other background material found to be valuable.
We are also reviewing and updating the questions, feedback and resource links in the light of the detailed comments we have received. if you are one of the people who has been interviewed, surveyed or asked for your opinion, we’re very grateful. Some of the changes we are proposing – though not all will be available immediately – are these:
- offer different, role-specific versions – to include teaching staff in HE and FE, possibly also research staff and professional staff in student-facing and other roles.
- focus on ‘the capabilities I need for work’ with less emphasis on personal and social uses of digital media
- revise and fully user test the feedback, considering more finely-grained feedback depending on responses to specific questions
- provide feedback more progressively
Some of these changes will already be available when you sign up for the open pilot: others will be made available over the coming months. Please consider joining us on the discovery journey and having a say in its future.
Next steps
Sign up for the open pilot (more details available from September)
Hear more from six of the fourteen pilot institutions about their use of the discovery tool
Join the Jisc digital capabilities JiscMail list and via the list, get involved with the community of practice (next meeting 30 November 2017 – details to follow)
Review some of the digital capability resources available on the project page.