How Can Open Communities Shape Representation in a Model?

-It’s no surprise, then, that the cultural energy around open vs closed source has shifted from software to AI models. - https://lsvp.com/stories/the-engine-that-builds-the-engine/ - Nnamdi Iregbulem

For twenty-something years, if you wanted to influence the open source software you depended on, the answer was

• Create or contribute to an existing issue
• Come up with your idea and submit it as patch or pull request
• Create or join a working group
• Show up in the related thread
• If all else fork (but don't be hostile about it)

We have proven ourselves, imperfect but intentional in building collaborative systems with governance to support our mutual success. Of course not all PRs are accepted, but review that a voice was heard, and opinion considered - by other humans.

With AI increasingly a part of open source tools, workflows and interactions - it is also increasingly opaque how one might change an LLM's behavior - especially as an open community - you are not aligned with or consulted as part of the training process. I mean 'alignment' in the human sense, not ML mathematical sense:

In the field of artificial intelligence (AI), alignment aims to steer AI systems toward a person's or group's intended goals, preferences, or ethical principles. An AI system is considered aligned if it advances the intended objectives. A misaligned AI system pursues unintended objectives.

Wikipedia: AI alignment

You can see that lack of representation in the (mostly) reactive approaches to AI turning up in open communities, tooling and spaces:

• Closing source (like Cal.com recently did, citing AI-driven security threats)
• Moving platforms to avoid AI (GitHub to Codeberg where no AI is present)
• Creating complex policies for AI use and contribution in open source
• Shutting down programs like bug bounties
• Some form of pleading into the void
• Codes of Conduct updated to require consent for AI note takers (etc)

None of these outcomes appear change model behaviors, or how people are using them. I say 'appear' because maybe? How would we know?

Nnamdi Iregbulem's recent Lightspeed essay, The Engine That Builds the Engine, was helpful framing for me - as I attempt to create a hypothesis around what might work.

The model is now the core of the software-building engine - the thing that converts intent into code. It has inherited the "preciousness" that code used to hold. It's expensive to build and hard to replicate.

This makes sense - the fight for democratized technology has migrated to the model and there's no on-ramp for the communities whose artifacts they are built on.

Asking for a "Pull Request" on models isn't the right framing, we know that

No branches, no diffs, no merge targets. Models aren't repositories. That's correct, and also beside the point.

Open source didn't invent pull requests because the mechanism was technically inevitable. It built them because communities needed a legible, contestable way to be represented in the things they depended on. The PR was one form of representation that happened to run on git.

For models, we DO have the substrate (RLHF, fine-tuning, evals, red-teaming, model cards) but its not clear to me as an outside, how communities might participate.

In machine learning, reinforcement learning from human feedback (RLHF) is a technique to align an intelligent agent with human preferences. It involves training a reward model to represent preferences, which can then be used to train other models through reinforcement learning.