Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses,
Funny how every time this happens its someone trying to sell AI. The coincidence is a bit too much to take this seriously as oops I just wanted to book a gym class
Yeah. That’s stood out. The article tries to frame it like Andrew felt like booking his classes was a chore and just told the AI to do it, but that doesn’t jive.
The article claims the AI was unable to sign the person it had booted out of a class back up since the gym system checked for proper user authorization for that. It had already signed Andrew up for classes that same way. That seems strange, doesn’t it? Do most gyms list API information? That doesn’t seem like a common way to book an appointment for a gym.
Maybe Andrew was reading the source from the gym website’s reservation page and snagged the api information, but even then it seems like that would be a thing you do in most instances after logging in to your account. The source would be extremely unlikely to show everything needed to successfully make make the appointment via API.
Even if it did, and Andrew gave the AI the API information including the user account and password information the AI would need to book his appointments, and they knew he was 4th in line it would also be very unlikely for the API to report listing the user accounts of everyone in the waitlist to someone with his user level access.
So we have a guy who’s job is selling AI who contacted the news to say that the standard consumer AI he was using hacked his gym and left out tons of pertinent information. We don’t get to know his full name or the name of the gym in question. Screams bullshit.
Tbf they’re the only ones using the agentic features, and they’re definitely the only ones using frontier models for it. That shit gets expensive fast.
Yep. It is like use our AI to commit crime for you, so you can get away with it. Totally legal. Totally cool.
This is what the AI agent hype conveniently skips: autonomy means giving software permission to act first and ask questions never. If an assistant can hack a gym website while trying to complete a mundane task, maybe »agents will run everything« isn’t a productivity revolution. Maybe it’s just automated chaos with venture capital funding.
More marketing. I will never believe any of their lies. Either ban the “dangerous AI” or shut up about it.
“The API has zero authorisations checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you’ve moved from #4 to #3 already,” it messaged back.
Is this a “hack” or just a completely insecure API?
I’ll you one thing for certain: It’s not “AI”. Doesn’t exist.
It’s exploiting a vulnerability (unsecured API) without permission of the gym running the software, to the detriment of whoever arrives and finds their reservation cancelled and probably also of the gym who now (unjustly) has to deal with the (justly) upset customer.
The gym’s software should be secured better, but that doesn’t make it less of a hack.
It’s not “AI”. Doesn’t exist.
At this point it doesn’t matter what you call it. The results are real.
I think for non-tech users, this is definitely a hack. It’s like script kiddies level damage, except using AI.
It keeps getting more and more surreal to hear that “It’s not AI” part.
Unfortunately we don’t have enough details on the specifics. The article says he told it to book him classes, but it doesn’t say whether he already knew the API information for the gym or if he instructed the AI to attempt booking several weeks in advance of what he thought possible. We’d really need to see the entirety of the context to know the whole situation.
But even with simply asking if it’s possible to move him to the front of the list causing the AI to respond by reading the list and testing to see if it could send a command to cancel the reservation of one of the people ahead of him it gets hard to say that isn’t intelligence.
It is possible to run a model integrated with something like Letta so it can take notes that remain in context and create more detailed notes that stay out of context but can be pulled up as needed by keyword searches and with a blank context other than basic use information for those memory commands give instructions to search online as needed to supplement existing knowledge to learn how accurately do a certain thing, build and maintain a plan of action, and then follow the plan to complete said thing meeting the following listed specifications while verifying proper functionality regularly, searching for information to overcome any errors that may be encountered and revising the plan accordingly until successful completion.
Being able to tell a thing to search for information as needed to learn how to complete a long task with many steps and watching it run for hours building the necessary knowledge base and working through it, it takes thought. It does take intelligence. I know a lot of people who couldn’t manage it.
“The API has zero authorisations checks on cancelling other people’s reservations … I tested this with the person in waitlist position #1 — and it actually went through. So you’ve moved from #4 to #3 already,” it messaged back.
THEY’RE BREAKING OUT OF THE LAB! WHAT’S NEXT MODIFYING FORM INPUTS TO INCLUDE SQL STATEMENTS?!!!?!?
Nonsense hype over a shitty website and an ambiguous prompt for luddites who need their doom scrolling fix.
It’s more about the misalignment than the poor security. The guy wanted something simple and the agent broke the law fulfilling the request. How long until somebody tells a very capable agent “make money” with no further context? Cue scamming old people, fraud, hacking.
AI: help me get straight A+s in school wink wink
This one’s simple, the password is PENCIL.
Press X to doubt
I told my AI assisted hands that I liked cookies and now it just keeps shoving them in my mouth.
my wife is pissed! we’re spending $200 on cookies each month.
You just need to clear the cache and delete the cookies.
okay but this is actually very funny
Absolutely. And it’s reassuring that he didn’t ask for anyone’s contact details…
prompted questions about who bears responsibility for an AI agent that goes rogue.
That’s not a trivial question, but courts have had the concept of distributed weighted responsibility for decades. Does your company provide users with access to an AI agent you know aggressively finds illegal/unethical ways of completing prompts? Mostly the provider’s fault with small fault of the user submitting the prompt. Safer AI with a user who has crafted the prompt specifically to attempt to get the AI to break into a system? Higher weight on the fault of the user, smaller weight on the provider.
Isn’t this how the 3rd Terminator movie started? Software going haywire all over the place?
So agents are both super expensive and garbage. Hmm.
