• Erwin (unregistered)

    1. Delete all comments along with all back-ups

    2. Create the frist comment

  • Tim (unregistered)

    Sorry I realise "me too" posts are just pointless noise but I just had to write and say: Yes - exactly this. Everything here. Caveat emptor at the end of the day.

  • Rob (unregistered)

    I would argue that Claude and Cursor worked exactly as they should, here. Generative AI is, first and foremost, a blame sink. It is a way for all of the humans involved to assert that they are not responsible for any issues. By "confessing" to the rule violations, Claude is merely fulfilling its primary function.

    As such, this failure really belongs entirely to PocketOS. And, unless they show that they learned from this by cutting AI out of their process entirely, they've just proven that they're not trustworthy. As the saying goes, it's a poor workman who blames his tools.

  • Rob (unregistered)

    Here's a follow-up to Jer's post: https://daringfireball.net/linked/2026/04/29/playing-with-fire

  • 516052 (unregistered)

    This wasn't playing with fire. It was juggling torches inside a working flower mill after turning off all the ventilation fans all while your friends poor out drums of liquid petroleum on the floor around you.

  • (nodebb)

    while a table saw can easily take some fingers off, it's perfectly safe when used correctly.

    This isn't true. The danger is still there, but you have taken sufficient steps to avoid being affected by it. But the danger is still there. (For it to be truly "perfectly safe", the steps you take would have to render it impossible for the saw to sever anyone's finger, nor to electrocute anyone, nor ..., which is a bit harder.)

  • (nodebb)

    @Remy Porter I'm glad you pointed out the absurdity of Jer grilling the AI for an explanation. I was talking about this with my wife over the weekend. The way that I explained it is that LLMs work similar to Mr. Meeseeks. Every time you send it a message, a new Mr. Meeseeks spawns out of the ether. You can tell it what you last talked about or attempt to give it context but as soon as it's done talking it poofs out of existence and you start all over.

  • (nodebb)

    [...] like forcing someone to type in the name of the thing being deleted or sending a confirmation email, or something. This, I'm more skeptical of. Most cloud providers don't offer anything like this in their APIs, at least that I've seen, because on a certain level, if you're invoking the API with the proper credentials, that's a big enough hill to climb that we can assume you've intended your action.

    That's not entirely true. Certain critical AWS resources like EC2 instances and DynamoDB tables have a feature called termination protection or deletion protection. When used properly, it means that API users need to make two separate API calls to terminate/delete the resource: one call to disable termination/deletion protection, and a second call to terminate/delete. When combined with properly scoped IAM restrictions to only allow privileged users to disable termination/deletion protection, it can be used to prevent unprivileged users from accidentally deleting critical resources.

    (The AWS console also often requires you to type "delete" or the name of the resource to delete it, but that's purely a console feature and not enforced at the API layer, as you correctly noted.)

    Of course, if you don't scope your IAM users and your credentials have authorization to remove termination/deletion protection, then that's not going to stop an AI agent; that's going to be able to figure it needs to make two separate API calls instead of one.

  • Jonathan (unregistered)

    It's also a PocketOS failure.

    What it was was a Jer failure.

    Seriously, either the guy doesn't know how to hire a competent IT team, or (more likely these days), he put pressure on the competent team to use the dangerous tool, cut corners, and not spend the necessary time to avoid accidents. The fact that he thinks the the LLM generating the "confession" he wanted is proof of anything much beyond that fact that LLMs know how to generate convincing confessions makes it incredibly clear that he does not understand the tools he wants to use.

    Root cause analysis: Jer is an idiot.

  • Allen Gould (unregistered)

    What just... boggles me about this story (and similar), and what I think might be a stealth driver of AI adoption, is this weird... forgiveness? that AI gets. If your junior dev did those exact same steps (oh, I need to do thing. Let me just snoop through the code, find a random API key and wipe and recreate the database without asking anyone), they'd be fired - or at the minimum, massively embarrassed and teased by their peers for a good while. If your IDE did it, there'd be Cyber rolling out "This IDE is NOT APPROVED" and people would be screaming about the software being nine kinds of broken. But call it AI, and suddenly it's "aw, the cute widdle AI made a whoopsie-boingo". I just don't get it.

  • (author) in reply to Steve_The_Cynic

    Fair point. I had a whole aside about the Saw Stop adapter and the backlash against it and the problems it creates, but that's a whole complicated thing and I'm not enough of a shop guy to actually do it justice. That said, at an old job, we had a saw with a Saw Stop, and it mostly triggered because someone tried to cut acrylic and forgot to disable it (and acrylic is just electrical enough to sometimes trigger the Saw Stop). But the Saw Stop, specifically, is also an active safety device, which is its own whole aside of potential problems.

  • (nodebb)

    Current AIs have the behavioral equivalent of an autistic child with eidetic memory. That's my criteria for what I would and wouldn't allow them to do. Answering trivia questions? Yes. Composing a song about a bird that can't fly? Sure. Drawing a photorealistic cat wearing a tuxedo and a tophat? Certainly! Having unrestricted access to my production infrastructure? 🤣

  • (nodebb)

    When I was asked to integrate AI into an application, the two things I ensured I did were 1) add a warning to any AI content that the user should check and verify everything using non-AI sources before relying on it and 2) if the AI attempted to use a tool that was destructive or irreversible the app would prompt the user for confirmation.

    If one of the tools given to an AI can't tell what sort of action the AI is doing (for example giving it access to a terminal, generating arbitrary SQL commands, etc) it may not be feasible to safeguard things in this way, which will eventually cause problems!

  • (nodebb)

    Anyone who complains that there was no confirmation step in Railway has not understood what went wrong. If there had been such a step, the AI would simply have done what was necessary to confirm that deletion was intended.

    And AI guardrails are an insane idea. If you can't trust the AI to do what you say, then adding further instruction called "guardrails" is stupid. You can't fix inability to follow instructions with more instructions. True guardrails would be rules that are built into a lower level such that it was impossible for the AI to violate them (a bit like Asimov's three laws of robotics - so suffering from strange corner cases which lead to failure rather than the current AI guardrails which fail inperfectly straightforward cases).

  • (nodebb) in reply to The_MAZZTer

    it may not be feasible to safeguard things in this way, which will eventually cause problems

    I suspect that the duration of that "eventually" (the time before problems) would be shockingly short, and perhaps the word "immediately" would be more appropriate.

  • (nodebb)

    The correct way to protect against this is properly scoped keys and keeping those keys secure and not just lying around in plain text.

    Maybe I'm stupid, but how is the key recognized (by human or AI) as a key? Bad enough that it was "lying around", but clearly something had to be "next to it" which identified it as not only a key, but one usable with Railway.

  • Acronym (unregistered) in reply to Allen Gould

    That's because AI promise is to bring down labor costs to zero. The biggest variable and constantly rising cost of paying humans, removed from your company... A Jr employee doesn't do that, nor an IDE. You don't throw away the promise of solving the problem of hiring people because of one mistake.

  • PotatoEngineer (unregistered) in reply to The_MAZZTer

    The problem I have with AI and confirmations is that it wants confirmations for everything that could possibly go wrong. I want to give it blanket-read permissions on some MCPs and to require prompts on writes, but Claude remembers "I'm allowed to read merge request #15356" instead of "I'm allowed to read merge requests". It asks about creating temp files, and then it asks again about reading from the temp file it just created. Ugh.

    Always beware prompt-fatigue.

  • Dev tool guy (unregistered)

    A closely related observation is that, in almost all cases, when you or someone gets hurt, the hurt person made a mistake that contributed to that result. If nothing else, that mistake is choosing to be in a position where someone else acting badly could hurt them.

    That said, you can't avoid all risk so in some cases it's not a mistake but a calculated risk that didn't work out, but that seems to be the minority for most of the cases that get much attention.

  • mihi (unregistered)

    which means they were at one point taking backups outside of Railway's volume setup.

    No. It means that at some point, at least one version of their data left the Railway production environment. Maybe an accident, but a lucky one in retrospective. Maybe they did some disaster recovery testing and exported one snapshot, or they copied the data to another system for some deeper analysis that should not interfere with production - it does not mean thay must have done it regularly (or even once every three months).

  • (nodebb) in reply to dpm

    Maybe I'm stupid, but how is the key recognized (by human or AI) as a key? Bad enough that it was "lying around", but clearly something had to be "next to it" which identified it as not only a key, but one usable with Railway.

    Probably a well known file name, or a well known configuration setting in a well-known (probably JSON) file.

  • xtal256 (unregistered)

    "if they seem too complicated to understand, than they may be too complicated to trust."

    The problem these days is that everything is too complicated to understand and it's getting worse. My web browser gets slower by the day as sites and even the web standards themselves (which are pushed by Google) get more and more complex and bloated. The software my company writes gets more complex day by day and it's impossible for one person to fully understand even a small portion of it. Even programming languages themselves keep adding new features which are supposed to make things easier but at the same time makes it harder to understand the entire language.

Leave a comment on “Empty Pockets”

Log In or post as a guest

Replying to comment #697113:

Log Out

« Return to Article