Shipping Secure: Should the AI get the final say?

On Tuesday I wrote about lawsuits over data sitting in systems where it wasn’t clear who could look at it, who could share it, and who was checking. Those cases are allegations and none has been decided, but the questions they raise are worth carrying into your own build. Tuesday’s post

I’m building a product that will hold other people’s business records. I’m not a security expert. These days you can have something working in an afternoon. What I didn’t expect was how quickly the real question shows up after that:

What is this thing allowed to do on its own?

Here’s what I mean. Say your app has an AI that can read a voice note, guess which job it belongs to, and flag a likely duplicate. That’s useful. But the same wiring could, if you let it, decide who gets access, delete a record, or share something with another company. A wrong guess in the first case is an annoyance. A wrong guess in the second is a data exposure.

Here’s the moment it clicked for me. Handling people’s information is one job. Keeping it away from the people who shouldn’t see it, and from accidental leaks, is a different job, and it’s the one that matters most. If I don’t keep that information safe, that’s on me, and it could get me in real trouble. That’s when “does it work?” stopped being the only question I was asking.

Three questions I think are worth asking before you ship anything with AI in it:

1. What can it suggest, and what can it actually do? Write two lists. They should be different.
2. What happens when it’s wrong, or when it’s down? If the answer is “the app stops working” or “it does the risky thing anyway,” that’s a design problem.
3. Who is accountable for the consequential decisions? A named person, not “the system.”

Tomorrow I’ll share the rule I wrote down for my own build and how I’m testing that it holds.

This is general information, not legal advice.