AI Governance Is Becoming Part of Software Engineering

Most companies did not start using artificial intelligence by first creating an AI governance framework. Most company leaders probably don’t even know when their company started using AI. They started because an employee drafted an email in ChatGPT, an engineer connected to a model API, a team adopted a coding assistant, or somebody discovered that a workflow that used to take several hours could suddenly be completed much faster.

That is probably the normal sequence. Technologies usually become useful for individuals before organizations consider formal implementation, much less develop rules around them. The problem is that AI is moving quickly and looks different across departments, while carrying the same fundamental risks of potential instability and privacy/confidentiality breaches. From individual productivity tools to internal processes, customer interactions, and operational systems, AI is affecting the systems companies actually depend on. Once that happens, responsible AI use becomes less about sharing best practices and more about stability, compliance, and governance.

The phrase AI governance often makes the subject sound more abstract than it needs to be. In practice, an AI governance framework should answer a relatively concrete set of questions. Where are we using AI? What data can those systems access? What actions can it take? Who is responsible for its output? How do we test it? What happens when they fail? How do we know when the underlying model or system has changed?

In our area of specialty, software engineering, these are not new questions. Good engineering organizations already think about permissions, testing, observability, security, change management, and accountability. AI introduces some new characteristics, particularly because model behavior can be probabilistic and difficult to predict exactly, but much of the governance problem is familiar.

The first mistake is thinking about AI governance entirely in terms of the model. Most companies don’t train frontier models themselves. They are building systems around models created by somebody else. A customer support application, for example, might retrieve information from an internal knowledge base, send context to a language model, generate a response, and then present it to an employee or send it directly to a customer.

The AI model is only one part of that system. The level of risk depends on what the retrieval layer can access, what information the model receives, what tools the model can call, whether the system can take actions, how permissions work, what gets logged, and whether a person reviews the result before anything consequential happens.

Responsible AI therefore has to apply to the whole system, not simply the model. In many ways it might look more like governance around a staff member than a traditional piece of software.

A practical governance program should begin by knowing where AI is actually being used. This sounds obvious, but adoption often happens from the bottom up. Engineers use coding assistants, marketing teams generate content, operations teams automate workflows, customer support teams summarize conversations, and product teams begin adding AI features. An organization can easily accumulate dozens of AI use cases without ever making a deliberate decision to have an AI strategy or policy.

A comprehensive inventory of AI use gives the company a way to distinguish between fundamentally different kinds of risk. Those audits should include checks on what access models have to various company databases, shared drives, or email archives. An employee asking a model to rewrite an internal paragraph is not the same as a system changing production infrastructure, and could require less governance, but that model also shouldn’t have access to a shared drive with confidential customer information. Generating a first draft of marketing copy is not the same as handling sensitive customer information or recommending a consequential financial decision. Good governance should recognize those differences rather than applying the same process to every use of AI.

The useful principle is that controls should increase with consequence.

Low-risk applications may need little more than approved tools and basic rules around data handling. Systems interacting with sensitive data, production environments, money, security controls, or customer-facing decisions deserve stronger safeguards. Those safeguards might include restricted permissions, human approval, testing in isolated environments, audit logs, monitoring, output validation, clear ownership, and the ability to quickly disable or roll back the system.

Again, none of these ideas are particularly exotic. Software organizations already use least-privilege access, code review, staging environments, observability, deployment controls, and incident-response procedures because software can fail. AI systems should inherit the same habits. What might be new to many parts of an organization is applying those same processes to marketing, sales, or finance teams. 

Human oversight also needs to be treated as a real system design question rather than a slogan. Saying that a process has a “human in the loop” is not very meaningful if that person is expected to approve thousands of outputs they cannot reasonably evaluate. Effective human oversight depends on who is reviewing the result, what information they have, what they are expected to check, when escalation happens, and whether the system can act before the review occurs.

In some applications, a human should remain directly in the execution path. In others, automated systems may operate independently while people monitor aggregate performance and investigate exceptions. The correct design depends on what happens when the system is wrong.

Overcorrection is also a risk. An AI governance framework that requires a committee to approve every employee experiment will probably produce bureaucracy rather than safety. People will either stop experimenting or simply work around the process, avoiding the safeguards altogether.

Good governance should make responsible adoption easier. Companies can establish approved tools, define what types of data may be used, create standard implementation patterns, specify when review is required, and provide a clear path for unusual or higher-risk applications. The objective is not to eliminate experimentation but to create boundaries within which experimentation can happen safely.

This matters even more as AI increasingly participates in software development itself. Gun.io’s work has included AI applications, Retrieval-Augmented Generation (RAG) systems, AI-assisted development practices, codebase professionalization, and engineering governance around AI-generated software. Those systems ultimately live alongside the same architecture, infrastructure, backend systems, integrations, security requirements, and engineering practices that govern other production software.

Over time, the distinction between AI governance and engineering governance may therefore become less meaningful. If an AI system can write code, access data, call APIs, communicate with customers, spend money, or change production systems, governing that AI means governing the software environment it operates in.

That requires policies, but policies are only the beginning. The real work happens in architecture, permissions, testing, monitoring, documentation, ownership, and the judgment of the engineers building and operating the system.

Responsible AI does not need to begin with an answer to whether artificial intelligence will eventually transform civilization. For most companies, the immediate task is much simpler: understand what the system can do, control what it can access, measure whether it works, and know what happens when it does not.

An AI governance framework is increasingly just the discipline required to do that well.

In other words, it is becoming part of software engineering.