Skip to Content

Building in Production: How I Stopped a $2,500 Bleed Through Observation

I walked in expecting to run a standard IT audit and uncovered a silent $2,500 monthly operational bleed—fixed not by a slide deck, but by observing the physical floor.

Table of contents

No titles in the blog post

Recently, I began working with Colorado's Drug Testing Agency (CDTA). I walked in expecting to conduct a standard technology audit, but what I discovered was a much deeper operational bottleneck causing a silent $2,500 monthly bleed. I didn't walk in with a pre-packaged slide deck on day one. Instead, I approached the floor as a Systems Architect and simply observed how data moved through their physical space—or rather, how it was getting trapped there.

The Anatomy of the Bleed

The problem wasn't malicious intent, and it certainly wasn't lazy employees. The staff was working incredibly hard. The issue was a fundamental failure of the system to adapt to the agency's rapid growth. Every single day, hundreds of probation vouchers were being processed. These vouchers represented critical revenue, but they were being handled entirely through a fragile physical paper trail, symptomatic of the anatomy of chaos in paper workflows.

When you rely on paper, human operators are forced to act as manual data bridges. They were constantly copying information from forms into legacy databases, double-checking handwriting, and trying to reconcile mismatched records. The operational friction was immense. Not only was this incredibly slow, but it introduced severe vulnerabilities regarding data privacy and HIPAA compliance. Every manual touchpoint was a point of potential failure. Clients had no access to their /my/intakes historical chain-of-custody, forcing them to call the front desk repeatedly.


​

I quickly learned a harsh truth: the moment sensitive data is committed strictly to analog paper, you lose the ability to query, track, and protect it effectively. You lose your data sovereignty.


Trial, Error, and the Sovereignty Pivot

My initial thought was straightforward: just build a custom web form and have the clients fill it out on an iPad at the /toxicology self-service portal. It seemed like the easiest fix. But as I continued to explore the agency's specific constraints, I realized that approach was fundamentally flawed. We were dealing with highly sensitive medical and legal information. If I just spun up a generic web app, I wouldn't be solving the problem; I would just be shifting the vulnerability from a physical filing cabinet to a vulnerable cloud database, committing the sin of the cheap IT nightmare.

I realized we needed absolute data sovereignty. Instead of bringing the sensitive data to a third-party model or app, we needed to bring the processing model directly to the isolated data. We needed a zero-knowledge architecture where the information could flow securely from the client's intake directly into a heavily encrypted backend without any human interception or third-party exposure.


The Challenge of Live Environments

The most daunting aspect of this project wasn't the code; it was the reality that CDTA operates continuously. I couldn't ask them to pause their business for three months while I built a new system. I had to build the new aircraft while the old one was still in flight.


Building the Aircraft in Freefall

Executing a massive architectural shift in a live production environment requires intense transparency and a lot of trial and error. I couldn't just flip a switch. I started by building a shadow digital replica of their exact paper workflow. For weeks, I simply watched the data flow through the old system and mirrored it in the new, isolated Odoo Community edition environment I was constructing on AWS Fargate.

I made mistakes along the way. Some of the initial data mapping protocols I wrote dropped packets or misaligned patient IDs. Because I was running a shadow system, these failures were invisible to the actual agency operations. It gave me the freedom to experiment, fail, and iterate rapidly without risking their daily workflow.

What I Learned from the Chaos

Eventually, the digital shadow system proved to be 100% deterministic and accurate. Only then did we slowly begin transitioning the physical workflows to the digital counterparts. The $2,500 monthly bleed—which was primarily caused by lost physical vouchers and reconciliation errors—stopped almost overnight. We achieved true enterprise compliance.

This project taught me that you don't need to be a genius to fix enterprise problems. You just need to be relentlessly curious. You need to follow the paper trail, identify where the data is trapped, and experiment with secure, sovereign architectures until the flow is restored.

Archive

Comments

Sign in to leave a comment
Plugging the $2,500 Voucher Leak: A Lesson in Operational Vulnerabilities
Test volume was high, yet revenue kept falling short. Tracing the physical lifecycle of a single paper probation voucher exposed a $2,500 monthly billing leak.