Algorithmic Impact Methods Lab

As pre-deployment algorithmic impact assessments (AIA) become a legal and policy requirement in many jurisdictions, our work set out to support staff with resources to conduct these assessments. While regulatory frameworks for AIA are emerging in different forms, each reflects a shared recognition that systems must be evaluated for potential harms before they are deployed.

This toolkit is the result of 18 months of piloting and experimentation at Data & Society’s Algorithmic Impact Methods Lab (AIMLab). It provides a practical framework for conducting AIA using a strategy anchored in centering community voices.

Our participatory approach asks the developers and deployers of AI systems to connect directly with the people most affected, anticipating harms and surfacing concerns before systems are put in place. The materials are also adaptable for community organizations seeking to evaluate systems from the ground up.

The toolkit walks users through the steps of a participatory AIA: identifying stakeholders, designing engagement, and translating community input into actionable insights. It helps uncover false assumptions in system design, answer community questions and concerns, and surface key risks and blind spots to mitigate before launch. It also supports organizations to better understand local needs and explore alternative solutions.

It is intended for use by people evaluating any AI system before deployment, like IT staff in the public sector, user research teams in tech companies, and community engagement or tech teams in non-profits. It complements other pre-deployment AI assessments, such as budget, system performance, organizational readiness, and environmental impacts.

Below are editable, ready-to-use templates, scripts, consent forms, session plans, slides, email templates, and other resources to guide this process.

This starter guide gives an overview of the algorithmic impact assessment process and how our method engages impacted communities.

This outreach plan helps identify which community-based organizations you should reach out to as part of your impact assessment process.

Use this template to compose informative welcoming emails to community-based organizations identified in the outreach plan.

Engagement Session Templates

The following sessions can be co-hosted with partner organizations. Session one provides background on AI, with a choice of a lecture or interactive format. Session two focuses on impact assessment and eliciting community hopes, expectations, and concerns.

Session One

An AI 101 session to help participants gain a shared understanding of AI in order to inform future discussions about the system you are assessing. Run this session with each group identified in the outreach plan. This can also be used standalone. Note: all links go to Google docs or slides.

90 minute lecture + interactive option: AI 101

• – Run of show
• – Informed consent form also available in Spanish
• –AI 101 script also available in Spanish
• – AI 101 slides also available in Spanish

45 minute interactive option: ChatGPT Skillshare

• – Run of show
• – Informed consent form also available in Spanish
• – Skillshare script also available in Spanish
• – Skillshare slides also available in Spanish

Session Two

A content framework to elicit feedback from impacted community groups on the system being assessed. Run this session with each group identified in the outreach plan; it can be run multiple times for complex projects as they evolve.

– Slides also available in Spanish

– Script also available in Spanish

– Run of show also available in Spanish

– Informed consent form also available in Spanish

AIA Template

A fillable template to report on the impact engagement process and the necessary content to create a transparent report. Once you’ve completed the process, we recommend posting your AIA publicly and sharing back with participating communities.

– Condensed AIA template

– Full AIA report template

Acknowledgements

Thank you to Micah Epstein for their help on toolkit design. Thank you to Siera Dissmore for helping the toolkit get over the finish line. Thank you to Patrick Davison, Kiara Childs, Eryn Loeb, and Alice Marwick for all of their helpful developmental editing and suggestions. Thanks to Leila Doty, Chelsea Palacio, Serena Oduro, Quinn Anex-Ries, and Madeliene Dwyer for taking the time to review and give comments on our drafts. Thank you to Chris Redwood and Alessa Erawan for our web layout and to Tess Demir for our web design. All errors and omissions are our own.