Safety Out of Sync: AI Policy, Codes of Conduct, and the Future of Incident Response
Sorry for some delays in writing. I have been out in nature again, but also working on a few things; One of those things is this: exploring the evolution of policy related to safe participation, and whether a new generation of AI safety tooling and related policies meet (or do not meet) the established ecosystem standards for effective incident response.
As the outcome of community research and designing standards for incident response, I wrote a blog post many years ago to describe milestones of response as well as a summary of what it looks like to weave safety into organizational processes - some of which is outdated, but I think generally holds solid as a set of human milestones.
I've also taught (and still teach) engineering and community teams how to create and implement moderation plans and advised a United Nations round table on metrics for safety in digital spaces.
First focusing on policy, I looked at some of the open source AI policies we've tracked in the CHAOSS moderation repo to see how guidlines around human behavior, and impact were being documented - if Codes of Conduct were evolving, or (as I suspected) the AI policies are becoming either a replacement, or extension of it.
Of 30 listed, I examined 14 in depth. 9 of the 14 had not substantively updated their code of conduct in more than two years (and most for over 5 years) at the time they created an AI policy. Its just a small group, but does support my hypothesis that policy around safe collaboration is expanding beyond the Code of Conduct itself.
For example, (Ghostty) relies on its AI policy alone to state contributor norms (no Code of Conduct that I could find). Another project showing intention to unify these policies - Zulip updated its Code of Conduct in the same commit as its AI policy ; Another, scikit-learn placed its AI policy inside the Code of Conduct itself, and Mastodon adopted Contributor Covenant 3.0 three weeks after publishing its AI policy. Contributing.md remains another location you'll find behavioral policy (human and AI).
Whether we think about it or not AI policies, are centrally are about the human experience of collaboration: avoiding overwhelm for maintainers, restricting harmful content and so on. That policy location evolves - is neither good nor bad; what matters is whether the wellbeing of contributors stays at the center and that it is enforceable, as Codes of Conduct has evidenced. Having multiple policies certainly is a change in how we need to think about incident response.
From a safety tooling perspective the ecosystem is advancing to combat volume, and complexity with use of classification, rule-based automation, detection and workflows that involve humans with collections like ROOST and with the availability of many open models in Hugging Face.
I asked myself: "how are organizations thinking about unifying all of these policies into one effort, how would contributors know without having to shift through a series of documents, that change from project to project? If people experience harm, or distrust do they know how to resolve that in ways that help them feel trust the process, and to feel safe?". These are versions of questions we've asked before, but with new urgency, and complexity to consider.
To answer that for myself have started to build a prototype that will test two things: a) whether all policy related to human interactions can be unified, through automation or standard (or both), and flagged when it falls behind ecosystem standards; and b) whether safety tooling plus human moderation can build on established incident response processes, helping moderators manage situations that put humans at risk, especially people in the protected groups well documented in Contributor Covenant 3.0 and elsewhere. I'll also bring in my own experience with stakeholder escalations, and I hope to share early results soon.
Sounds like a lot, but I feel that centering humans is the most important thing we can do right now, and its my favourite thing to work on. Hope to share a bit of a prototype in the next couple of weeks.