September 18, 2024

Launched in May 2023, Data & Society’s Algorithmic Impact Methods Lab (AIMLab) takes an interdisciplinary, collaborative, and deliberative approach to designing and building a cohesive algorithmic impact assessment (AIA) methodology grounded in the public interest. This effort aligns with our broader aim to map and conduct ongoing experiments in evaluating algorithmic systems, and to document practices of identifying, prioritizing, measuring, and managing the consequences of deploying these systems.  

Our goal is to generate, pilot, and disseminate algorithmic impact assessment methods that emphasize the rights and needs of impacted communities. AIMLab serves as a space to facilitate change in how algorithmic systems are built to mitigate their potential harms, orient stakeholders to consider alternatives that can strengthen resilience in access to and delivery of services, and create forms of redress — ultimately building footholds for public interest accountability for the impacts algorithmic systems have on individuals and communities. 

Of course, this work does not happen in a vacuum. Over the first year of AIMLab’s work, policy and AI safety conversations tended to have an outsized focus on technical, rather than sociotechnical, evaluations of AI systems. There were mass layoffs in the tech industry and a backlash against DEI efforts, responsible AI, and sustainability initiatives. At the same time, we saw AI regulation calling for stakeholder engagement and impact assessments, and broad cross-sector conversations on fostering participation and operationalizing risk management frameworks. This created space for our community-focused, qualitative work. Against this backdrop, our focus has not only been on the various methods that might produce evaluation frameworks for AI, but also the social conditions under which such frameworks become actionable. We grapple with understanding and doing AIAs within organizations and the power relations among researchers, actors within companies, and community members that mutually shape the impact assessment process. 

METHODS FROM THE FIELD 

We actively engage in pilot projects across government, nonprofit, and corporate research sites. Some of these projects unfold over many months and require long-term trust-building, while others are one-off events. We document lessons learned from our research sites and conduct pilot studies to observe how organizations are grappling with AIAs and their associated pain points. 

For example, we are investigating how climate in particular has become a place where measurement practices and standards collide with organizational cultures and social justice concerns within the tech industry. Accordingly, we have developed a set of projects focused on interrogating how decarbonization metrics are produced and used to make decisions about product development within companies. We juxtapose these decisions with potential downstream environmental and social impacts on communities. 

In all of our work, we uniquely focus on adapting Participatory Action Research (PAR) methods to develop viable, actionable methods for understanding the context and interests of impacted communities on their own terms. These methods bring together community advocates, technical auditors, social scientists, and tech workers — with their respective interests and expertise — to identify, deliberate, assess, and reflect on how algorithmic systems interface with their environments and identify potential for algorithmic impacts and harms. We further aim to incorporate the expertise of legal scholars, agency administrators, and policy experts to ensure that methods for impact assessment are compatible with existing (and proposed) legal and administrative frameworks to secure the public interest. 

We are especially focused on qualitative approaches to impact assessment; this stems from our team’s disciplinary backgrounds in science and technology studies, cultural anthropology, and information science. AIMLab also builds on prior foundational research from Data & Society’s AI on the Ground program. We have prior experience in and published research on algorithmic impact assessments, sociotechnical analysis, participatory methods, red-teaming practices, auditing, AI accountability, and the creation and implementation of standards for responsible AI. We have been researching the risks as well as the social and environmental impacts of algorithmic systems, including generative AI, in diverse domains ranging from creative labor and social media to financial and government services. Our core competence is in collaboratively experimenting with methodologies to identify and promote social/organizational changes, and to design and develop tools and regulations that work together to make algorithmic systems more equitable.

In exploring the diverse strategies used to evaluate algorithmic impacts, we also pay attention to the rapidly evolving and intellectually vibrant landscape of algorithmic audits. Algorithmic impact assessments share a family resemblance with ongoing developments in practices of algorithmic auditing, which have also grown to become much broader in scope from evaluations of system performance over the years. By comparison, impact assessments tend to be associated more closely with regulatory processes. They are organized to ask: what can built systems do to the people who live with them, and to communities and environments in which they operate? The focus is, thus, more explicitly on interactions between systems and their environments, identifying potential ways in which such interactions can produce harm, and developing strategies to mitigate the possibility of such harmful interactions.   

We are not consultants. Instead, we collaboratively experiment with our partners at our research sites to develop methods that provide an empirical foundation for deliberations over social/organizational changes, tools, and regulatory interventions to manage the interface between algorithmic systems and social contexts in which they operate. There may be times in which an impact assessment is published or shared in open fora by other means, but given the uncertainties in knowing the full gamut of consequences that a system may produce, we do not endorse or certify safety of a system. 

WHAT WE’RE LEARNING 

We are writing a report documenting our approach to producing community-driven evaluation frameworks for algorithmic impact assessments. In addition to short write-ups on individual engagements, this modular report will detail overarching pain points and research insights based on our fieldwork across sites, including our analysis of (1) the political economy of impact assessments; (2) concerns around how AIAs are budgeted within organizations and how community participants are compensated for their time; (3) tech hiccups, or the disconnect between what AI is imagined to do and its actual functionality; (4) navigating time constraints, including the loss of institutional memory when precarious staff turn over or product managers are pressured to release products without time for proper impact assessment. The report will offer a collection of ethnographic vignettes and self-reflections from the AIMLab research team. 

Through our research partnerships, we have begun to notice patterns across sectors and case studies. There are tensions inherent to doing this work: Is our participant observation in these processes a form of harm reduction, even as we risk inadvertently rubber stamping harmful systems? How does our ability to speak as outsiders open up different possibilities? In some cases, we have been told by research partners that our presence allows them to say no, to slow down assessment timelines, or to make the case for abolition. 

Because these pilots have taken long-term relationship building not only with the research sites where we are conducting pilots but also with potentially impacted communities, we have shifted from thinking about this work as about “impact assessment” to thinking about it as a form of impact engagement. 

This process provokes high-stakes questions about what it means to engage communities and incorporate their insights into algorithmic governance. What is considered AI expertise, by whom, and who is brought in under what circumstances? What are the benefits to communities who are part of the process? For example, in one engagement, city residents wanted to participate in some educational workshops around the uses of AI, not merely to evaluate a system on the City’s behalf.  

These are some of the emergent themes our future work will address:

Structural limitations. Impact engagements are a matter of logistics; they are shaped by budgets, managerial priorities, time constraints, staff turnover, and precarity. How do you build meaningful relationships and trust over time? A lack of resources within organizations means that there is also a lack of continuity or institutional knowledge and a general inability to see something through various stages of engagement and development. 

In one case,  a product dramatically shifted twenty minutes before our first participatory workshop, leaving us scrambling to arrange the parameters of the engagement. In others, staff we were working with lost their jobs or were rushed because of contracts that were soon to end. 

Relationship dynamics. Balancing positive relationships with developers/deployers and providing accurate (and often critical) feedback is a craft. We need to build trust with the internal staff at particular research sites and are sometimes in positions where we need to tell them things they don’t want to hear. We also need to build trust with community-based organizations and other stakeholders who we are bringing into the process. All of this takes time and we have found that almost all of our pilots have taken far longer than we anticipated, with projects unfolding over many months on uneven timelines full of stops and starts. 

Values alignment. Another salient aspect of this work involves navigating aligned goals and priorities between our team and partner organizations, which are also our research sites. In order to determine if a project was overly risky or worthwhile, based on our affinity or lack of affinity with particular research site partners, we created a partnership assessment template. The template served as a conduit for having tough conversations about decision-making, power dynamics, and our capacities as external nonprofit researchers. 

Safety and privacy. In a number of case studies, participants had concerns about ensuring safety and privacy in their interactions with algorithmic systems, including chatbot interactions with sensitive information or with computer vision technologies that might surveil them in their everyday activities. There is also a gap between what a technology is imagined to do and its actual affordances. Sometimes communities might wonder why is AI being used in the first place — is an automated system the best use of resources? 

Accountability. As a team, we are navigating the challenge of observing internal processes and facilitating community engagements without having direct influence, and often without having a final say in outcomes. One of our goals is to strengthen connections between developers or deployers and impacted communities to help create more space for accountability, contestation, and refusal. If we hold one or two workshops with community-based organizations, even if the community’s recommendations are taken up by the technical team, that should not be the end of the story. We want to ensure that communities are part of the process in the long term and are able to make decisions about technologies that will affect their lives. In short, the nature of the contribution that communities can make is directly related to the processes employed to solicit their contribution, which brings us back to our first observation that impact engagements are a matter of logistics.  

We will expand on these findings in a forthcoming toolkit and will be publishing findings from individual pilots in the coming months.