I’ve spent the last few months writing about the risks of giving AI agents too much latitude without structure: the skill atrophy problem, the security blind spots engineering teams need to care about, and why you probably need an AI working charter before you touch any of this in production.
That was the diagnosis, but this is the solution my team and I actually built.
Over the last three months we developed a five-layer agent orchestration framework and then hardened it against a 17-prompt test suite before calling it our team standard. The suite found real failure modes, but more on that below. We developed this for our GitHub Copilot setup, but this framework is tool agnostic and can be applied to Claude Code, Codex, etc.
The architecture
Layer 1 is the constitution: a system prompt always in context, with hard rules agents can’t violate, tech stack constraints, and an explicit routing table. If it’s not here, no agent knows it exists and there is no guarantee of what training data it falls back on by default for its output.
Layer 2 is more surgical. Instruction files auto-inject based on file path patterns: you define a glob in the frontmatter, and when an agent touches a file in that path, the right rules load automatically. No invocation, no prompt engineering, no hoping the model remembers. We scope ours by domain: one file per major code zone, one for the API layer, one for the test suite. Edit a file in one zone and that zone’s rules appear. Edit a different zone and different rules appear. The agent gets exactly the context it needs for targeted scoping and nothing it doesn’t.
Layer 3 is four specialized agents with restricted tool access: planning-architect (read-only, never touches production code), code-upgrade-refactor, test-qa-engineer, and dependency-manager.
Layer 4 is where I’ve seen most teams stop, but stopping here means your framework is still confined in scope to the application repo. That’s certainly useful, but it’s not the whole picture. Custom skills are what break that ceiling. They activate semantically when the agent pattern-matches on your request, and they’re the bridge between your orchestration framework and the external systems your team actually lives in every day.
The workflows that are the biggest time sinks aren’t usually the ones where you’re writing code: they’re the ones where you’re gathering context before you can write code. Tracing a request through logs. Pulling a ticket and following the Epic chain. Cross-referencing both to figure out why production is behaving differently than the spec said it should. All of that is manual, all of it is slow, and it’s exactly what skills automate really well.
We’ve built two so far (and are actively working on more): jira-digest fires on any ticket key pattern, follows the Epic → Story → linked-ticket chain, and strips the noise out of the JIRA API response. grafana wraps a TypeScript script querying our Loki log infrastructure: UUID-based request tracing, keyword searches, microsecond-precision timestamps, all from a natural language ask.
And they chain. Ask “why is request X returning a 500?” and Grafana activates, runs the log search, and if a JIRA ticket ID appears in the output, jira-digest fires to pull context on the related ticket. Two external systems, no dashboard switching, one question. That workflow used to take 30-60+ minutes of tab juggling. Now it’s a prompt. Multiply that across a team and across a week, and this architecture moves from being a developer experience improvement towards being a real force multiplier.
Layer 5 is /remember: a slash command that captures lessons from the current session and writes them into the right instruction file, committed to the repo. Institutional knowledge that doesn’t live in someone’s head or a Confluence page nobody reads.
What the test suite actually caught
The 17 prompt test suite revealed concrete failures. A single instruction file covering both frontend zones caused contamination: the wrong zone’s rules would load on the wrong files. Splitting into separate files with tight applyTo globs fixed it at the filesystem level, because the wrong file literally cannot load. /remember had scope leakage, writing to user-level memory instead of workspace-level. The mandatory planning gate was getting bypassed on vague requests until we tightened the language and made the file-count threshold explicit. Each failure had a direct fix, and the prompt testing suite is what surfaced them before they became accepted habits.
Most teams just prompt, blindly trust the tools and hope the model keeps up, but this results in low quality output, needless iterations of rework, muddying context bloat and recklessly inefficient token usage. We decided to define the rules instead and the difference has been night and day.
Before this, every session felt like re-teaching the AI what it had just learned. Now it behaves consistently across developers, across sessions, and across the complexity of a multi-framework codebase, because the guardrails are infrastructure, not suggestions.