A Mirror That Talks Back

Centering open communities in the AI feedback loop

"[The new digital mirrors] show only where the data say that we have already been, never where we might venture together for the first time." The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking by Shannon Vallor

"Much like Narcissus mistaking his reflection for another being, we risk confusing AI's statistical predictions with genuine intelligence. AI does not think; it merely projects complex reflections cast by our recorded thoughts, judgments, desires, needs, perceptions, expectations, and imaginings" — The AI Mirror: How to Reclaim Our Humanity in an Age of Machine Thinking by Shannon Vallor

I find myself in spaces where people are on one of two very polar opposites when it comes to AI + Open Source right now. The first is those who are captivated by the potential, the opportunity and improvements to their work and efforts thanks to the assistance of AI. The other: angered, frightened, entirely opposed or feel victimized and overwhelmed by its presence in spaces designed for humans.

I suppose, I, like many, have been on a journey of discovery for where AI fits into my life and work. It certainly speeds up mundane tasks - it's incredible to help my son build a website using a coding agent; essentially wiping away the learning curve of my generation. I also worry deeply about his future with environmental impacts, the 'rich getting richer', AI 'note takers' storing my data god-knows-where, the exploitation of open communities' labour - the loss of creativity and agency.

In looking more closely at the topic of AI and Ethics, I found Shannon Vallor's book The AI Mirror which I have quoted above, and it was really the first framing that made sense to me as both a challenge and opportunity. AI has been built on, among other content, vast amounts of open source code, writing, open educational resources, open science, open data - all representing the work and labour of countless humans, over decades.

The future depends on a continuation of democratization of technology, which is not where AI is taking us right now - and certainly there is no indication that 'open source' licenses will change that (sorry). By that I mean that no matter how 'open' the stack says it is, I cannot change the output, I cannot (easily, or sometimes at all) trace a license, I do not know how to influence future output - in fact I am more likely (like Cal.com closing source) to hide my writing, and make my open repositories private to protest the unpaid, unattributed, non-consenting method of training models right now.

"[We are] on the brink of surrendering the urgent task of engineering ourselves and our societies anew to mindless tools without hope or vision, that only predict what the historical data say we will probably do next."

In this moment, even where we see shockingly good models to help with critical topics like OSS security (like Mythos), it seems important to remember AI is only a mirror of the past. To make the future something of value, for humans (and not billionaires), there have to be measurable ways for open communities to help shape not only what models are trained on (increasing the value exchange), but also how creators can inspect, improve and CHALLENGE the output.

Lets face it open communities are not the image of perfection, we've been working on reducing bias, including language, perspective and combatting toxic behavior for a decade - the loss of that progress and potential is significant to address.

This is not a new concept. Models already get tuned through human feedback (RLHF), and increasingly through AI feedback - RLAIF, Reinforcement Learning from AI Feedback - which moves the loop further away from the people whose work made the training possible in the first place. And the "humans" in RLHF are not who you might think: they are mostly contracted data labelers (often gig workers, often paid very little) and in-house staff at the labs. Not the communities and creators whose work was used.

What is missing is a way for open communities to do this work themselves, in the open, with mechanisms that are visible and measurable. Vallor again:

"The merely rhetorical 'we' must be forged into a real one, with at least some capacity for a human will... To construct that shared will, we require many tools, new and old, from the humane and technical arts alike, and AI technologies belong among them."

That is what we are building over at the CHAOSS AI Alignment Working Group. Two metrics, in early form, anyone can commit or contribute!:

• Community Consent and Approval - whether a community was asked, what it agreed to, and where the line is.
• Community Feedback and Effectiveness - whether the feedback a community gives makes it back into the model, and whether anything in the model actually changes as a result. A lot of this framing we hope can lead to new innovation for good.

If AI is only a mirror of the past, these are an attempt to make sure the future it reflects has more of us in it.

Further reading

optimistic framing (open communities focus)

• Adafruit, The open source world can write its own rules for AI… and nobody has to ask permission (Feb 2026)
• Anthropic + Collective Intelligence Project, Collective Constitutional AI: public input shaping a model's behavior
• Anthropic, Claude's new constitution (Jan 2026): external community critique invited
• Hugging Face, Argilla / Data is Better Together (DIBT): community-built preference datasets

The critical framing (open communities focus)

• Jeff Geerling, AI is destroying Open Source, and it's not even good yet (2026)
• Quippd, AI Code is Hollowing Out Open Source, and Maintainers are Looking the Other Way (Apr 2026)
• Audrey Watters, Broken Record, 2nd Breakfast (May 2026)
• Axios, AI agents are flooding open-source maintainers with security reports (Mar 2026)
• InfoQ, AI 'Vibe Coding' Threatens Open Source as Maintainers Face Crisis (Feb 2026)
• The New Stack, Curl's Daniel Stenberg: AI is DDoSing Open Source
• The Register, Gentoo Linux bans code contributions written with AI

RLHF / RLAIF: the technique and the people behind it

• Lee et al., RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback: the canonical RLAIF paper
• Anthropic, Constitutional AI: Harmlessness from AI Feedback: Anthropic's RLAIF approach
• TIME, OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic (Jan 2023)
• Mary Gray and Siddharth Suri, Ghost Work: How to Stop Silicon Valley from Building a New Global Underclass (2019)
• Karen Hao, Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI (2025): names the gap between communities and the people inside the loop
• Kate Crawford, Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence (2021)
• Pistilli & Trevelin (Hugging Face), Can AI be Consentful?: directly engages whether consent is even tractable in current pipelines

As promised a collage of my tulip garden (yes I need to weed - this weekend!)