A Real-World Case Study in Secure AI-Assisted Development
The integration of Large Language Models (LLMs) and AI agents into the software development lifecycle (SDLC) is no longer theoretical. We now have concrete, publicly documented examples of complex features being shipped with significant AI assistance. One such case, a detailed account by Mitchell Hashimoto of building an auto-update UI for the Ghostty terminal emulator, provides an invaluable blueprint for establishing a secure and effective AI-assisted development workflow. This analysis unpacks the process from an AI security and red teaming perspective, highlighting critical controls and methodologies that can be adopted by engineering and security teams.
The project involved shipping a non-trivial production feature written in Swift. The development process was meticulously documented across 16 distinct coding sessions over two days, spanning approximately 8 hours of active development time. The total token cost for the AI assistance, provided by the Amp Code agent, was a mere $15.98. The availability of full, shareable transcripts for each session offers an unprecedented opportunity for auditing and process analysis—a practice all security-conscious teams should demand from their AI tooling.
The “Anti-Slop Session”: A Critical Security Gate
Perhaps the most critical security control demonstrated in this workflow is the concept of a dedicated “cleanup step,” referred to as the “anti-slop session.” This is a mandatory phase where the developer steps in to manually review, refactor, and integrate the AI-generated code. From a security standpoint, this is not merely about code quality; it is a fundamental security gate.
- Mitigating Insecure-by-Default Code: LLMs can generate code that is functionally correct but lacks appropriate security hardening. This manual review forces a human expert to scrutinize the code for vulnerabilities, logic flaws, and insecure defaults before it’s merged into the codebase.
- Forcing Code Comprehension: As stated in the account, “To cleanup effectively you have to have a pretty good understanding of the code, so this forces me to not blindly accept AI-written code.” This is the core principle. Blind acceptance of AI output is a significant risk. This step ensures the developer fully understands the attack surface and failure modes of the new code.
- Improving Future Agent Performance: A direct security benefit arises from this process. Cleaner, better-documented, and more logically sound code serves as higher-quality context for subsequent interactions with an AI agent. This reduces the likelihood of the agent introducing errors or vulnerabilities in future sessions based on misunderstood or “sloppy” existing code.
Strategic Human Intervention: Guiding the Agent to Secure Patterns
Effective AI-assisted development is not a passive process. The developer must act as an architect and security champion, actively guiding the AI agent away from potentially weak patterns and towards more robust, secure designs. A key example from this project was the decision to manually refactor a core data structure.
Initially, the view model was a struct containing multiple optional properties. This is a common pattern but can lead to complex state management, null reference bugs, and an increased logical attack surface. The developer intervened:
I spent some time manually restructured the view model. This involved switching to a tagged union rather than the struct with a bunch of optionals. I renamed some types, moved stuff around. I knew from experience that this small bit of manual work in the middle would set the agents up for success…
This is a proactive security measure. By enforcing a stricter, more explicit data structure like a tagged union, the developer eliminates entire classes of potential bugs and simplifies state validation. This strategic intervention demonstrates the symbiotic relationship required: the AI handles generative tasks, while the human expert enforces architectural integrity and security principles.
Agent Self-Auditing: A First-Pass Red Teaming Prompt
A novel and powerful technique employed was using the AI agent to perform a final review of its own work. At the conclusion of a session, the developer would prompt the agent to look for omissions and potential improvements without writing new code. This acts as a form of automated, first-pass red teaming.
Example Self-Auditing Prompt
A powerful refactoring prompt used during a cleanup session was:
Turn each @macos/Sources/Features/Update/UpdatePopoverView.swift case into a dedicated fileprivate Swift view that takes the typed value as its parameter so that we can remove the guards.
Following such sessions, a concluding prompt was used to probe for weaknesses:
Are there any other improvements you can see to be made with the @macos/Sources/Features/Update feature? Don't write any code. Consult the oracle. Consider parts of the code that can also get more unit tests added.
The directive to “Consult the oracle”—an Amp-specific feature to engage a more powerful and expensive model for analysis—is particularly interesting. This is equivalent to escalating a code review to a senior principal engineer. From a security perspective, this prompt specifically asks the agent to identify areas that could benefit from more unit tests. Inadequate test coverage is a common precursor to security regressions, and using the AI to pinpoint these weak spots is an efficient way to direct testing and verification efforts.
Conclusion: A Framework for Secure Agentic Workflows
The argument over whether AI makes development “faster” often misses the more critical point of how it changes the nature of the work. This case study suggests the primary benefit is not just raw speed, but a form of parallelization: “the AI can work for me while I step away to do other things.”
For AI and LLM security professionals, this real-world example provides an actionable framework for integrating AI agents securely:
- Mandate Auditability: Insist on tools that provide complete, shareable transcripts of all developer-agent interactions. This is your audit trail.
- Enforce a Security Gate: Institute a mandatory manual review and refactoring stage—an “anti-slop session”—for all AI-generated code. This is a non-negotiable security control.
- Promote Proactive Guidance: Train developers to act as security architects, intervening to enforce secure design patterns rather than passively accepting the agent’s first output.
- Leverage Agent Self-Auditing: Implement end-of-session prompts that ask the agent to critique its own work, specifically probing for gaps in test coverage, error handling, and overall robustness.
By adopting these principles, organizations can harness the productivity gains of AI-assisted development while actively managing the inherent risks, creating a development lifecycle that is both accelerated and secure.