Proposing a competency framework for generative AI use

Judging from popular discourse, one might assume that a quick course in prompt engineering and awareness of hallucinations is all it takes to become a proficient user of Generative AI tools. But it is hard to reconcile the blog posts reporting on the near-superintelligent performance of generative AI tools with my own experience working with Generative AI tools. I am an enthusiast. I know that I can do more as a result of these tools but they certainly haven’t transformed my productivity. I look on with envy as some of my colleagues with computer science and software engineering backgrounds scale up and innovate using Generative AI tools and wonder (not for the first time) whether I should have studied computer science.

It’s an important question - what human skills, knowledge and mindsets do we need to harness the benefits of generative AI tools? And I shouldn’t only talk about the benefits. Effective use of Generative AI tools also involves avoiding risks, and these encompass more than Generative AI hallucinations.

Luckily, it is my job to investigate interesting questions like these. And my colleague Einat Grimberg has led the way in recruiting and engaging with expert users of generative AI tools to get some answers. We (but mostly Einat) interviewed knowledge workers from a range of sectors and occupations with the goal of investigating whether there was a set of core skills, knowledge or mindsets that underpinned effective use of generative AI.

Rather than simply asking that question, our approach was to get experts to identify examples where generative AI tools were used effectively and ineffectively and then delve into these examples to reveal what skills knowledge and mindsets had contributed to those instances where generative AI tools were used well.

What we learnt was that effective use of Generative AI is actually underpinned by several, fairly traditional human competencies, such as communication skills, domain-specific expertise, metacognition and diligence. Furthermore, although AI literacy is often mentioned as the foundational knowledge for working with AI, our experts did not suggest that AI literacy was required. Rather, they believed that working with Generative AI tools required an understanding of how Generative AI works- its strengths and weaknesses, and the different types of Generative AI that are available. In other words, the expansion of AI tools is creating more specialised AI literacy needs and users of Generative AI need Generative AI literacy.

In my own awkward way, I’ve also been trying to generate some click-bait by declaring that prompt engineering is not a skill. The thing is, when we probed experts in detail, it became clear that good prompting is itself the product of skills. It relies on traditional communication skills like perspective taking - the ability to consider what the Generative AI tool doesn’t know and how to convey it in a way that aligns with the affordances of Generative AI tools. It also relies on old-fashioned domain expertise which allows you to use the terminology and provide the relevant information from which you get ‘good’ Generative AI output. Domain expertise is also how we identify when and how the Generative AI output could be improved. So we’re arguing that good prompt engineering is an indicator of proficient Generative AI use - along with other factors, like making responsible and informed choices about when and how to use generative AI tools.

If you take the time to read our article (found HERE) you will discover that many of the Generative AI competencies that emerged from our research have been suggested by other researchers. What I believe is important about our work is that it provides a comprehensive framework, outlining the range of skills, knowledge and mindsets that seem to underlie proficient use of Generative AI tools.

Here’s a quick preview:

But DO please read, review and consider citing our preprint. This is a proposed framework and it certainly needs testing through further research. Our hope is that it serves to reconfirm the ongoing importance of the many different facerts of human intelligence in an era of Generative AI-enabled work.

Claire