How to evaluate emerging knowledge infrastructures

Work on knowledge infrastructures of the past decade has shown that infrastructures are central to many crucial types of knowledge about climate change, human migration or biodiversity. They are also important in supporting diversity and pluralism in knowledge production. But how can knowledge infrastructures best be evaluated?

Changing infrastructural and evaluation landscape

Science involves many kinds of evaluation, peer-review being the most ubiquitous form. Most contemporary approaches to evaluation focus on techniques of assessment, often through standardized comparison, whether to establish benchmarks or metrics, to develop rankings, or to provide evidence of outcomes. However, finding common ground for comparison between knowledge infrastructures–whether they serve scientific projects, function as facilities such as observatories, or more virtual settings– is notoriously difficult because each entity is unique by design. In addition, when forms of diversity and heterogeneity become important aspects of knowledge infrastructure (Goldstein and Nost 2022), comparison seems an even less promising avenue. An important element in thinking about evaluation is therefore to examine how it is the result of both assessment practices and the values embedded in them.

Need and form of new approaches to evaluation

In this workshop, we use the case of knowledge infrastructures that seek to support innovative forms of knowledge production —more often than not, unique resources—to explore the possibilities of evaluation beyond comparison.

This aspiration is in line with recently expressed wishes from infrastructure developers (Tasovac et al. 2023) and international bodies (https://plus.google.com/+UNESCO 2020) to reflect on and align infrastructure and assessment practices.

In sum, building on the scholarship on KI so far, which shows the entwinement of values and knowledge via infrastructures, we want the workshop to be the occasion to extend the analysis and reflection on the importance of values not only for design and epistemic consequences but also for modes of assessment. The workshop is therefore especially timely because it will map out the transitions of knowledge infrastructures towards increasingly important values–openness, sustainability, epistemic diversity, democratisation and participation. It will also explore which forms of evaluation are best aligned to these values and how they might be incorporated into evaluation practices.