Measuring What Matters: Insights on DPI Interoperability from Global Experts

Key Takeaways:

• Standards-Interoperability Relationship is Contextual: As systems mature and ecosystems need to connect, explicit standards become essential for maintaining interoperability at scale. Effective DPI measurement approaches must work to

assess different maturity stages and prioritise real-world outcomes over rigid compliance metrics.

• Testing as Research: Creating empirical tests that directly measure system compatibility provides more usable metrics than theoretical assessments. Real-world compatibility tests offer a way to transform abstract interoperability standards into measurable outcomes that reflect actual user experiences.
• Compliance with Purpose: Interoperability should be measured against its ability to deliver tangible value rather than technical compliance. Measurement frameworks should prioritize service continuity, reduced transaction costs, and cross-border functionality over standards adherence.
• Interoperability is Multidimensional: Effective measurement must consider technical, syntactic, semantic, and legal dimensions of interoperability. Each dimension presents unique challenges and requires different assessment approaches to capture both capabilities and limitations.

Overview:

Interoperability presents a fundamental governance challenge for digital public infrastructure, one that extends beyond technical specifications to encompass political economy, institutional design, and public value creation. In our second community of practice session “Measuring DPI Interoperability” we aimed to unpack some of the key features and tensions informing measurement practices that describe the interoperability of DPI projects. The session brought together four expert practitioners whose diverse insights challenged us to reconsider some of the prevailing perspectives around interoperability: Robin Berjon (IPFS Foundation), Siobhan Green (Fenix Digital), Daniel Abadie (CDPI) Ville Sirviö (Nordic Institute for Interoperability Solutions).

“Interoperability for Interoperability’s sake, there’s no point to it” - Siobhan Green

Our discussion quickly established the importance of maintaining a clear understanding of the purpose and values that underlie interoperability as a core feature of open DPI projects. As Siobhan Green noted, "Interoperability for interoperability's sake, there's no point to it." This sentiment was echoed by Daniel Abadie, who challenged us to articulate "what's the real value of interoperability? It's about service continuity, reducing time, portability, cross-border, cross-sector." The experts converged around the understanding that before meaningful measurement can occur, we must first clarify what we're trying to accomplish through interoperable systems. As David Eaves summarised,  "Standards only matter insofar as they're facilitating the creation of value...there is a real danger for us with the map and for other projects to kind of fetishize the interoperability nature". This conversation was further enriched by perspectives from across our community, with members raising critical questions about achieving semantic and legal interoperability across different jurisdictional levels national, sub-national, and domain-specific agencies. To this end, it is necessary to establish clear definitions, purposes, and success criteria as prerequisites for developing metrics that truly capture public value creation.

Re-centring the discourse on interoperability around public values offers a way to guide the development and implementation of measurement practices. Rather than simply tracking technical compliance with predetermined standards, we must evaluate whether interoperable systems genuinely deliver public value across diverse contexts and populations.

Robin Berjon: Lessons from Web Platform Testing

Robin Berjon shared knowledge from his experience developing web platform interoperability tools which offer an enlightening practitioner perspective on measurement frameworks. As Berjon explained, the web ecosystem of the early 2010s presented similar governance challenges that parallel those facing DPI today.

Berjon's insight challenges conventional approaches to interoperability governance. Rather than treating standards as primary and testing as secondary, his team inverted this relationship, establishing testing as the foundational mechanism through which interoperability is achieved and measured. "If you're doing (interoperability) after the specifications have been done, you're doing it too late," he emphasised, highlighting the importance of pre-defined measurement practices.

“Tests are the research” - Robin Berjon

Berjon's most provocative assertion, "the tests are the research", reframes how we understand interoperability assessment. Rather than theorising abstract frameworks, this approach generates empirical evidence about what works in practice. Extending this insight onto the space of DPI measurement, this perspective suggests that effective DPI measurement requires creating institutional mechanisms that continuously test interoperability in real-world conditions, rather than merely documenting technical specifications or policy intentions.

Siobhan Green: The Dimensions of Interoperability

Siobhan Green drew on her fieldwork on gender-based violence data systems to provide an illustration of semantic interoperability challenges. She explained that despite technical connectivity, systems across government departments operated with different classifications and definitions, rendering meaningful data integration a persistent challenge. As she explained, "One of the barriers for interoperability was that the Police Commission didn't feel that they had the legal right to share data with the Ministry of Gender," Green explained, demonstrating how legal frameworks often prevent data sharing even when technical capabilities exist. Examples such as these demonstrate the importance of understanding interoperability as both a technical and governance practice operating at the intersection of legal structures, institutional incentives, and technical systems. Such failures of interoperability highlight how institutional contexts and disciplinary boundaries shape what data means and how it can be interpreted across domains.

Perhaps most revealing was Green's insight into the economic drivers of interoperability: "The real reason why we can't make these two systems talk to each other is no one's paying us to do it". This candid observation cuts to the heart of public sector innovation challenges, where budget constraints and misaligned incentives often prevent interoperability despite technical feasibility. Green made the claim that to achieve effective measurement

we must first identify the purpose interoperability serves in specific contexts, then track the economic, legal, and governance barriers to that purpose.

Daniel Abadie: Valuing Real-World Interoperability

In his presentation, Daniel Abadie offered a practice-focused perspective, using cross-border examples from Latin America to illustrate what successful interoperability looks like in real-world contexts. "This is what we're trying to beat," he said, referring to an example of paper documentation. "It's portable, it's verifiable by people, it's been accepted everywhere in the world, it's cheap, and everyone trusts it".

By establishing this baseline, Abadie pushed us to evaluate digital solutions against tangible everyday standards that citizens already understand and trust. His case studies demonstrated how interoperability enables concrete value creation: the integration of digital identity systems between Uruguay and Brazil allowed mate tea exporters to create digital certificates without crossing borders twice, while Mercado Libre's payment integration across Argentina, Brazil, and Uruguay enabled 61 million users to make cross-border payments through their local accounts.

“Paper credentials, that is what we’re trying to beat” - Daniel Abadie

These examples highlight the importance of including the downstream economic and social impacts of interoperability as they are felt by end-users. Stressing this point, Abadie challenged the premise of measurement itself by asking, "Why do we need to measure DPI interoperability?" before arguing that measurement should focus on outcomes: "What's the real value of interoperability? It's about service continuity, reducing time, portability, cross-border, cross-sector". Abadie’s perspectives encouraged the session participants to consider ways to shift the focus of measurement practices from technical implementation to social and economic impacts.

Ville Sirviö: Standards Evolution in X-Road

Ville Sirviö's leadership of the Nordic Institute for Interoperability Solutions - the organization behind the DPI project X-Road - provided unique insights into how interoperability standards evolve through institutional learning and adaptation. With X-Road now serving as critical digital infrastructure for 25 countries and over 540 million end users, Sirviö's experience illustrates the complex governance challenges of maintaining and evolving public digital infrastructure at scale.

In his presentation, Sirviö highlighted the complexity of interoperability in federated systems: "We are not talking about system-to-system interoperability, but we are looking into the future with federations of data ecosystems." This evolution creates new measurement challenges, as interoperability must be assessed not just between individual systems but between entire ecosystems with different governance models.

This perspective reveals a crucial insight for measurement: interoperability requirements evolve as systems mature. As Sirviö explained, "For the ones who are part of the early wave of adopters, interoperability will come first without a lot of attention on standards," while mature ecosystems require explicit standards to maintain connectivity at scale. This developmental trajectory suggests that effective measurement approaches must be adaptive to the maturity of DPI projects - recognising that the same metrics cannot be uniformly applied to systems at different stages of institutional development.

Standards vs. Interoperability: Rethinking Governance Frameworks

The relationship between standards and interoperability emerged as a central tension in the  discussion between panellists during the Q&A section. Robin Berjon articulated a provocative inversion of conventional wisdom, arguing that "you don't have a standard if you don't have interop...if you don't have interop, it's just a piece of paper." This perspective positions interoperability as the empirical validation of standards rather than their outcome - suggesting that effective governance must prioritise demonstrable connectivity over formal documentation. By reframing technical governance in this way Berjon questioned the tendency of public institutions to focus on publishing standards without ensuring their practical implementation.

Ville Sirviö offered a nuanced counterpoint, suggesting that "we cannot have interoperability without (at least de facto) standards." He emphasised that standards provide a foundation for interoperability, especially when federating multiple data ecosystems. These positions represent different points of emphasis around the state of infrastructure maturity rather than contradictory claims. As subsequent discussion between the panellists revealed, they each apply at different stages of institutional development.  This discussion highlighted the fact that as systems mature and interconnect more broadly, explicit standards become necessary coordination mechanisms for maintaining interoperability at scale.

Why Measure Interoperability?

Daniel Abadie's question—"Why do we need to measure DPI interoperability?" - prompted reflection on the purpose of measurement itself. Participants identified several compelling reasons why measuring the interoperability of DPI projects is important:

1. Demonstrating value: Measurement helps quantify the benefits of interoperability investments, particularly important for public sector systems that must justify expenditures.
2. Identifying barriers: Effective metrics reveal where interoperability is breaking down, whether due to technical, semantic, or legal issues.
3. Creating incentives: As Siobhan Green noted, "The role of measurement is a way for governments and ministries to argue for more interoperable systems and move away from bespoke siloed systems."
4. Comparing alternatives: Measurement allows comparison between automated interoperability and manual workarounds, highlighting inefficiencies.
5. Learning from history: Participants noted lessons from measuring open data's value, emphasising the importance of capturing economic and social benefits, not just compliance metrics.

Next Steps:

The session revealed that measuring DPI interoperability requires looking beyond technical specifications to capture real-world outcomes and value creation. Participants emphasised that effective measurement must assess whether interoperability serves its intended purpose - whether enabling cross-border service delivery, reducing administrative burdens, or creating new economic opportunities.

The insights from this session will inform ongoing work to develop more sophisticated interoperability metrics for the DPI Map, balancing technical assessment with real-world impact measures. Rather than creating a rigid framework, participants advocated for adaptable approaches that recognise interoperability's contextual nature and its role in service delivery.

As our Community of Practice continues to evolve, we invite practitioners, researchers, and policymakers to join us in refining these frameworks and developing practical measurement approaches.

Get Involved: To suggest topics for future sessions follow the link here.

DPI Map Updates: To receive updates about future sessions or contribute to this community, please subscribe to our newsletter at dpimap.org/measurement-community.

Upcoming Session: Register to join us for Session 3 on June 2nd (more details to follow).