Today, we took part a webinar about the project’s evaluation work.

One important part for us was the idea that Jay referred to, that “evaluation has to be supported by some kind of theory of change”. This is something we’re still grappling with. Our account of change has to look at the kind of day-to-day work that goes on in the institution, because we want to improve the day-to-day experiences of students. (In this sense, we certainly feel comfortable with what Jay described about the evolutionary processes aiming at “an iterative deepening of understanding”; I think we’d agree we’re less clear about structural measures of accountability.) What our early work has shown us is that structures don’t determine these experiences, because students are creative, adaptive and resourceful. So we can’t rely on claims about changing structures; we have to be able to talk about changes to what people do, too. We’re drawing on sociomaterial accounts of change to help us with this. We expect these to be able to help us provide convincing accounts of the ways that the project has helped to reconfigure structures, resources and practices – but we’re aware that we’ll need to provide convincing accounts of this.

The accountability side of things is harder to talk about – but not because it isn’t happening. Rightly, Jay challenged us to answer the “so what?” question: does the work contribute to ‘big picture’ outcomes; is the work evaluative; will the answers be direct, explicit and actionable?

This is a fair challenge. Over the last few months, we’ve been discussing how our close analysis of practices helps us to do these things. Here’s some ideas that Lesley and I have been exchanging that shows our thinking; they’re all tentative and provisional, but are an indication of our general direction.

Building on this, and our baseline work:

1. We can make our students more ‘digitally literate’ if we can identify areas where they struggle to coordinate people/places/things and ‘black box’ them. An example of where this would work well would be single sign-on authentication, but library access to electronic journals via library sign-on is still a success story in this respect: students don’t need to manage personal authentication for a dozen different journals. Sociomaterially, we coordinate networks that students can readily co-opt to their ends. It adds an environmental, infrastructure element that we can tackle at an institutional level.
2. We can explain how some students demonstrate their digital literacy in the ways that they can overcome shortcomings in the coordinated environment we provide them with. If they can’t work through our networks but can work around it – by co-opting other people, places, things – they prove themselves digitally literate and suggest areas where we need to black box more effectively. (Students’ decision to use printers in another institution rather than here might be one example of this.)
3. Combining the above, ‘Black boxing’ may create risks: if a system is disaggregated it might be possible to work with some and work around others. For example, setting the printers at the IOE so that you can’t edit it to double-sided printing black-boxes in a way that might be helpful for the least able students, but is actively unhelpful for those who want more control over printer settings. We should think about the design of boxes in such a way that they can be opened gracefully. Email forwarding might be another example. Student frustration over lack of control might be a signal for black boxes that we should make more amenable to opening.

On the basis of this, we’ve started using words like “resilience” when talking about students’ digital literacy, and thinking about how we can help make students’ digital literacy more “resilient” by helping them to think about strategies they could use to reconfigure networks when things go wrong.