Computers are useless. They only give you answers. (Pablo Picasso)
This blog is a review of the report “Optimizing Engagement, Research, Evaluation and Learning in Public Diplomacy”.[i] The report addresses the perennial question of how to measure the effectiveness and impact of public diplomacy by looking at “how approaches to Research, Evaluation and Learning (REL) are executed by major players in the Public Diplomacy (PD) sector around the globe.” (p.8).
The report was commissioned by the US Advisory Commission on Public Diplomacy and it is suggested that the State Department takes on a convening role “to leverage the PD community’s collaborative spirit and appetite for knowledge sharing”. The report suggests that this could ensure that the US enjoys a “long-term competitive advantage in the influence space” by making sure that it stays up to date with the most up to date tools and approaches around the world.
The report could, in fact, be seen as a necessary first step towards this aspiration. Leaving aside whether the State Department would in practice be able to generate advantage for the US in this way, it certainly reflects the fact that thinking about the measurement of impact is definitely a global phenomenon. The report draws on 28 detailed case studies in 17 countries, including the UK, Brazil, China, Turkey and Russia. Definitely not only the usual suspects.
Given this global reach, it is somewhat disappointing that while the report was noted by some in the PD sector (eg USC Center on Public Diplomacy), it does not feature on the search results of the major browsers. This is perhaps not surprising – it is, after all a report to Government, but the issues raised are more widely relevant.
The issues are clearly identified, and recommendations are addressed to the policy and research community under 3 headings: strategy, practice and data gathering. They are all important. I discuss some highlights here:
There is a need for a model of influence that goes beyond communications variables only.
Absolutely. We need to be able to measure more than just “likes”, “hits” or “pokes”. These are measures of popularity or recognition, so-called “vanity metrics”. We need to be able to at least be able to understand who is influencing discussion and views on particular topics, or about particular entities like countries. We also need to understand the digital space better – a need that goes well beyond the practice of public diplomacy into academia, think tanks and policy makers who may not think of themselves as public diplomats, but who are involved in transnational discussions, debates and decision making.
The report recommends that the State Department creates a central hub for REL.
While that probably makes sense for them, there is also a need for an independent (of government or corporate interests) hub dedicated to REL in all aspects of the flows of influence across the globe. Such a hub, if founded on ethical principles, could be of enormous benefit in countering misleading narratives about influence, stimulate innovative research and make it available to a global community, and help develop a better global understanding.
There are opportunities here both for Governments and for the private sector.
True – but a big challenge is how to do that within a trusted ethical framework and without excluding civil society, or those who lack resources.
The report calls for investment in capacity building research teams in tech-enabled and digital research approaches and methods.
Again, absolutely. As the instigator of the UK-based DIGI project identified in the report, I can only echo that this is an urgent need. The pace of change is such that we need more and better knowledge, and an impact and knowledge sharing model for dissemination and engagement.
As the world is very fast-moving, cross-platform analysis and issue-mapping would provide a basis for data gathering.
This may sound techy, but it is essential. We need to have a structured approach to which issues are important and why we need data about them. As big data analysis and AI are only as good as the data that feeds them, this matters.
A “typology-based” approach to engagement in different contexts based on key criteria could be significant.
As the report points out, this could help identify, for example, optimal conditions for engagement, levels of support for policies or positions, cultural conditions, language skills and historic issues that shape perceptions. All this is highly relevant to strategy development and implementation. Today, we simply do not know with a sufficient level of assurance, what the impact of our Public Diplomacy, Cultural Diplomacy or Cultural Relations activities are. We do not know if they generate value for money. We have knowledge derived from experience, but that is often locked into the heads and memories of individuals. In other words, we could manage our risks better.
There is a need for a model of how change happens, with metrics.
Absolutely. No need to say more here – that would be a whole other discussion.
Visual data are dominant on social media and in popular culture – need methods to analyse it.
This is clearly true. While at Edinburgh University, I was involved for a short while in a project that looked at the combined impact of visual and written material on Twitter. Photos of middle-aged white men standing in front of flags simply do not do it. More work needed!
This blog is a bit of a list, but I believe that this report should be more widely studied, and built on. I hope that this starts a new conversation about digital influence, what it is, how we research, measure and talk about it.
[i] Report by M&C Saatchi for the Advisory Commission on Public Diplomacy (ACPD) at the US Department of State, April 2018 https://www.state.gov/pdcommission/reports/281372.htm