- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
Frist (unless there is a duplicate thread created to "help" Paula.
And
Ummmm,that is one serious :WFT: I'm lost for words...speechless...... (which may may some people happy :smile: )
Admin
:giggity:.
Also, this looks like something I did not too long ago...
Filed under: Never again...
Admin
I mean, Joep was basically like an intern, right?
Admin
Who the fuck is named "Joep"?
Admin
https://en.wikipedia.org/wiki/Joep_Lange
EDIT: The article is about Dutch people and the above guy is Dutch... so...
Admin
Also, wtf is up with text selection?
Admin
I can live with that. WHO THE FUCK IS DENNIS?
Seriously, those are the only times Dennis is mentioned. Gah!
Admin
Dennis is Joep's actual name?
Admin
No, it's his nickname because he's a menace.
Admin
I'm also confused! Where did Dennis go? Was he fronting for Jeop at the time and we all just play along until Jeop comes back?
Admin
Dennis was the first name I made up for the character who became Joep, but I decided I should actually use a Dutch name, since he is Dutch. I apparently did not do as complete a find/replace as I believed.
Admin
So the TWTF is that the DBAs don't make backups at a useful frequency, or are we to believe that they sign up more than 1000 new customers per day. If they can 'loose' 1000+ customers between backups I can only imagine how many transactions with their other customers get lost.
Next even if we accept that its often impossible to prevent business office folks form getting access to run ad-hoc queries, why on earth do you grant their account anything more than SELECT on the application schema? Joep should never have been able to change data from SQL he does not need that for reporting. If he need to transform data he should be selecting it into tempdb and altering his personal copy there.
Admin
Our head of IT once did a rm -r (remove recursive) on the live Unix box, but he didn't tell it not to follow symbolic links, so it wiped out the live database.
Not where I'm working now, I hasten to add.
Admin
rmdoesn't follow symlinks. It just deletes the links, but not what they are pointing to.Admin
Common Dutch male surname. Really Dutch, like in you'll get laughed at in Belgium for having it.
Admin
You never, EVER, give a developer or a "power user" direct access to a production database, especially not in write mode. They are prone to the "what ifs" that end in data loss and "I didn't know I was touching live data".
As for losing 1000 customers: sounds like someone was trying to be "smart" by doing infrequent level 0 backups. Definitely penny wise but pound foolish.
Admin
Worse: the database must have been in "Simple" recovery mode - otherwise, they could have taken a transaction log backup, and then done a point-in-time restore to just before the deletion.
Admin
TRWTF here is Anna, surely?!?
What the hell was she thinking? Giving a loaded gun to a monkey?
I know, I'm going to solve all my users disk space problems by just telling them all the root password. Just clean up after yourselves, people. No worries. Fuckin' A
Admin
I did essentially that with a customer of ours. He was already comfortable with the DB webinterface. When he spent hours to reset a field to 0 for a thousand records, I told him there was a command for that. I also told him it's easy to mess it up.
So far we've had only one panic and luckily it was easy to solve. He does know how to pull a DB backup and I think he conscientiously gets one right before messing with the DB.
Power to the people.
Admin
Even I who is not a DB expert (far from it) always create a user which only have the right to SELECT. This is what I use in ahum phpmyadmin to check some thing.
Admin
Unimportant. **What's the frequency, Kenneth?**strong text
Admin
Did you do a Ctrl-I instead of a Ctrl-H or something?
Admin
That's a sensible course of action if your DB is always evolving and resetting it to a backup even a few minutes old would lose important transactions.
For our case that would be overkill. Accidental edits (which are very unlikely after all, we don't tend to confuse SELECT and DELETE) could be remedied by restoring from backup.
Admin
Why would you even want autocommit to default to being on?
Admin
TRWTF (beyond providing an account that can do more than SELECT) is that they did not have any foreign key relationships set up or at least they had cascade delete on, otherwise SSMS would have surely declined to delete anything.
Admin
Maybe you should call Anna and see if she can set up a better find/replace/drop database command for you.
Admin
The real WTF here is Anna. She could have set up another user in the database that only had SELECT rights, and give that to Joep. Handing over user details that could do damage, that's her fault there, not Joeps, who obviously had no idea that MSSQL Server would only run the statement selected out of a whole SQL statement (that's a minor WTF in itself, MSSQL Server should at least be smart enough to identify a whole statement up until the closing semicolon and warn about only executing selected statements).
Admin
Years ago I had a situation with a client where I was recommending, then requiring, then begging for a backup system replacement. Old backups went from running sometimes to not running at all. My recommendations were shelved for nearly a year because "budget". Then it happened: a critical VM had a snapshot that hadn't been deleted/consolidated in 9 months...and burped. 9 months of snapshot data, lost. Many users were unhappy to discover they lost 9 months of work when the VM was brought back up.
(Yes, the VM/SAN issue was the other TRWTF - consultants responsible for those weren't monitoring it. Oops.)
Fortunately, we had a secondary solution - DFS replication to an off-site location. Upon discovering what had happened, I immediately disabled DFS, then made arrangements to copy the replicated data to a removable drive and drove over to the main site to set up a restore (that alone took several days) - AFTER storage issues on the SAN that caused the VM to burp in the first place were rectified.
About the time I had things settled down, the CEO pulled me aside and asked if the new backup solution would have worked better. I told him: "Yes, the system would have been restored in about 1 hour instead of 1 week, with only the morning's work lost instead of the last few days" (because WAN DFS lag). I got approval for the new system that afternoon.
Admin
At least Anna didn't end up moving to a position in Josepon, Ohio.
Admin
Also...
We'll just ignore this part of the article, right?
Because you'd think after a year of using a utility and not messing things up, you're still supposed to have alarm bells ringing in your mind about such things. Always. :trolleybus:
Good question.
Admin
Hah, it could be even worse than that. My friend Sophianna worked at a company as a web designer, and was dragooned into doubling as the system admin and tech support when I left the firm (this was the second dot.bomb where this had happened to her, BTW - apparently, I was so unnecessary that no one thought they needed to find someone new to replace me). She spent nine months trying to convince the CEO/President/Principal Owner that they really needed some way of backing up the main development drive, especially when about three months in it started getting glitchy from stiction. The MGT refused time and again, stating that a second external drive, a backup tape drive, or even a CD-R would cost too much.
Guess who got fired for failing to requisition a backup drive when it died, taking the entire company's code base and data with it beyond reasonable recovery?
Admin
Funnily enough, Dennis is more common than Joep (Dennis: 36402, Joep: 7184).
FTFY ;-)
Admin
Marie?
Those wacky Dutch.
Admin
Could have been prevented in MySQL/MariaDB with the option "--i-am-a-dummy" : https://dev.mysql.com/doc/refman/5.7/en/mysql-command-options.html#option_mysql_safe-updates
Admin
Defaulting to AutoCommit on is how it works with Java's JDBC????
Admin
I'd expect such stupidity with Discourse, but not a professional environment.
Admin
He is president of the AIDS society.
Admin
TRWTF is a language where probably 100 people in the world every day do exactly this, because it's easier to delete all records than specific ones.
Admin
https://www.youtube.com/watch?v=ptTTe8p4aAw
Admin
Admin
Sounds like a refactoring tool failure
Admin
I would love to see the notes from the design process that came up with this syntax. Also, the notes from the first call from a dba or user who discovered this.
Bet it was an undercover IDMS salesman
Admin
Reminds me of a shop that started locking down Windows. They ended up with developers who couldn't delete shortcuts off their desktops, but had full delete access to all databases...
Admin
I'd be more likely to go for "Willem" as a typical Dutch name.
Admin
The second I read 'and disabled auto-commit' I knew we were in a major problem. I thought to myself 'surely Ana at least did this on a reporting server or a separate one, so there's no way this story will end with this guy dropping or truncating tables'. So much for optimism!
On another note, it pains me to see non tech companies go through these growing pains. They lost THOUSANDS of customers?? How many customers did they enter the last few minutes? Where were the transaction log backups? No log shipping? No delayed secondary? Ok, where was last night's backup?? Surely they didn't add thousands of customers within the prior night to the next day...
Edit: Also, wtf??? She gave him a DBO or SA user?? She's lucky that's all he did. She didn't tell the DBA's that she's sharing her admin/owner credentials to the prod DB server with a freaking business analyst? He never locked the DB causing time outs? What happened to PCI or security requirements, it seems like this would apply to this company. Why did the DBA's give her such high rights on Prod? Doubtful that she needed it. Fails, fails everywhere! Why are there so many fails on DB stories?
Double Edit: Also also, no Foreign Keys on a 'customers' table?? Why even use a RDBMS then if even your customers table doesn't have any FKs on it? That's usually one with tons of them. Perhaps it was legacy code or other requirements needed RDBMS transactional support but from what I read, I doubt it.
Admin
No, Oracle requires a explicit commit, SQL Server programmers love to say 'SQL Server doesn't assume anything' but they should finish with "except commit". Default behavior for no commit commands in SQL Server is to commit anyways. Not really a driver issue since the code will have to enforce transaction isolation/rollback.
Admin
+1 internets for you friend. Prod databases too I'm sure, correct?
Admin
We'll just ignore this part of the article, right?
Maybe I'm reading too much into it, but that seems like she allowed him to use a tool that, after her configuration was complete, allowed him to delete live data.
She danced with the devil, and the devil loves to dance dirty ;)
Admin
SQL developer might not, but sqlplus definitely starts in autocommit mode by default.
Admin
TRWTF is that the user account she gave Joep even HAD Insert, Update, and Delete permissions.
it should only have had select!