- Feature Articles
- CodeSOD
- Error'd
-
Forums
-
Other Articles
- Random Article
- Alex's Soapbox
- Announcements
- Best of…
- Best of Email
- Best of the Sidebar
- Bring Your Own Code
- Coded Smorgasbord
- Mandatory Fun Day
- Off Topic
- Representative Line
- News Roundup
- Editor's Soapbox
- Software on the Rocks
- Souvenir Potpourri
- Sponsor Post
- Tales from the Interview
- The Daily WTF: Live
- Virtudyne
Admin
:interrobang:
Admin
I coulda sworn I had a comment on this topic…
Admin
Paula created a duplicate.
Admin
Sounds like a quite nice architecture to me, compared to all the WTFs I have seen and read about.
Why didn't they just change the column to a BIGINT instead of truncating the table?
Admin
My thoughts exactly. Also, it seems like a bit of an overreaction to quit your job just because a single column is int instead of bigint...
Admin
My feeling too: some architect weasel words, some truth, some dumb implementation, but no WTF. Only the HOS (Hourly O Shit). Then ending might have been added for dramatic effect, though. No WTF story is complete without the hunt for a new job. Where do all these people go, I wonder.
Admin
Yeah, I don't get why they didn't just do that. But then again I've worked in places that use char instead of varchar and were too afraid to just change it.
Admin
And was later served with a lawsuit for assaulting a coworker.
Admin
Yeah, I'm not sure I'd quit my job for that. It doesn't seem to bad to either change the column to a bigint, or to periodically truncate the table (or delete old data and reset the ID counter when it wraps if their DB engine supports that). That seems like a minor change to me.
The management seem to have been behind Danny as well, and didn't seem to be blaming him - so why quit? Maybe that's TRWTF?
Admin
To write articles here would be my guess. Either that or to go around Spain looking for anybody who even checked in anything in any code base they've worked on.
Admin
What? :raising_hand:
Admin
Just goes to show, Hell has weather just like any other place. Sometimes it's a bit colder, and all the denizens will say 'Wow, it's a cold day in hell today'.
Too bad for Roger's monolith.
Admin
http://thedailywtf.com/articles/The-Professionals
Admin
2147483647 monitoring records ought to be enough for anybody.
Admin
http://what.thedailywtf.com/t/the-monolith/47593/6?u=eldelshell
Admin
Roger: "Listen you morons, my design is scalab..."
Admin
Why would you do that?
Admin
Yeah, they should append to a log.
Admin
Analytics.
Admin
Admin
But why would you keep that live as part of the application? Analytics is meta-processing, and can be shuffled off to a dedicated server if necessary since it's quite likely to involve unpleasant things like ad hoc queries…
Admin
I didn't build their fucking system, I don't know. I was just answering your dumbass question.
Admin
You can't do it off line, or by the time you do, the tinfoil hat brigade will have slipped through your fingers.
Admin
If you're worried about that, just use a 16-bit integer for your ID column. Like that there'll be almost nothing happening until you've had a chance to analyse it!
Admin
At which point I could answer the question with....
Frog tongue juice.
I don't know, I didn't build their system.
Admin
But I want to track those guys and offer them ads for the same brand as they make their hats out of. Geez, that's the whole freakin' reason for analytics!
Admin
That's how the Very Big Corporation of America discovered the true Meaning of Life: That people are not wearing enough hats.
Admin
And that's where @abarker comes in
Admin
Admin
Yay ambiguity!
Filed Under: inb4 lojban
Admin
Each one just moves into the empty position created at the end of the previous WTF. It's like Hilbert's Grand IT Department, only with no room service.
Admin
It depends on what you want to log.
Many logging frameworks allow you to send log entries to a DBMS.
We send simple log entries to our DB; there is an ID for the log entry, an ID for the user (also special IDs for servers) that caused the action, an enum ID for the action, the type of log entry (error, action, etc.), the date and time, and the type of main and secondary objects the entry related to (customer, order, etc).
Then the log entry in an actual text log also contains the ID from the DB.
So now a person can search the DB for a specific log entry for a given type of entry, in a given date/time range for a given "user" and then correlate that with the entry in the log file. Eventually we will write an app (or find one) that can do the searching from a user friendly app. This is mostly for the help desk, QA and dev. to work with.
Admin
Because it would take an hour or so for the alter table to run and the web site would be down for that time and unlike the initial problem Management would know exactly who to blame. I've worked in places where taking the website off line for a hour for some annual maintenance was considered a grater evil than letting the database crash once a year and spending half a day trying to fix it.
Admin
http://drawception.com/pub/panels/2012/6-7/scnmwSOc5g-6.png
Admin
I think you're too optimistic. How many of those 2 billion rows were still in the table? The ALTER TABLE could take a lot longer than one hour. And if it dies because the transaction log fills up while attempting that, how many hours will go by before the transaction rollback finishes and you're right back where you started? I don't think I've ever worked for management that would be happy with that.
Rename the original table and create a new one with a BIGINT primary key, starting at 2.2 billion. Set up a job to copy records from the original table into the new one in small batches, limiting the rows/minute to something that won't stress the system. Stop it when it has copied enough recent history to make everyone comfortable (last week, month?). Then drop the original table. Alternatively, skip the copying and keep the original table around for a while. Drop it when it is no longer relevant.
Admin
So I'm only an Ahole DBA and not a jerk face developer, but there might have been a few options to keep the old records and keep the system live. Creating the table with a different name, putting a view on top it excluding the ID column, and then renaming the two tables could work live depending on the architecture. Then they would keep their old data and still be able to query it.
They could just rename the old table in a script right after creating a new one with BIGINT as the ID column while keepign everything live. Yes BigInt will take up more space that's true, but they don't seem too terribly concerned about space and there's good ways of dealing with that.
Truncate is just so.... brutal and non reversible live esp. if you aren't in a emergency to get usable DB space. Truncate makes me sad.
EDIT: Caveman ANGRY! Caveman truncate table with big club. Caveman now happy he took care of problem. Ooo ooo ! That's my view when someone immediately in a panic suggests to truncate a table in an emergency. Not this particular story.
Admin
As others have said, logs in an RDBMS are really nice for QA/quick problem resolution (well, as quick as the queries run), etc. Just don't expect 'em to scale to a large number of requests per time period.
For an app whose traffic averages less than a request per second, I've got just such an infrastructure in place. When we start serving 100x that number of requests, we'll figure out something else.
For another app that I've worked on, peak traffic is around 70 requests per second. I probably wouldn't recommend this architecture for them. Throw logs into a database? Sure...just not a relational one. Cassandra maybe?
Admin
I have a hard drive named "The Monolith". It was an external drive, and came in a sleek, black Lacie case.
https://www.lacie.com/imgstore/product_intro/hd_hddesignbyneilpouton_triple.jpg
Admin
shhhh... don't say the "No" word or all the i-relational ones will jump at you.
Admin
But, but BAD ARCHITECT! He designed this WRONG! :crying_cat_face: :crying_cat_face: :crying_cat_face:
And he's on a beach :crying_cat_face: :crying_cat_face: :crying_cat_face:
Admin
NoSQL?
is good at solving one class of problem, is utter crap at solving most of the problems people ask it to solve.
Admin
It's just the latest
Free LunchSilver Bullet.Admin
How long has The Architect been on sabbatical? Long enough for the volume of requests to increase 100x, apparently.
Admin
yep.
I love NodeJS... i use it all the time myself for personal projects, it's a fun language and the design makes it easy to fall into a pit of success for certain classes of problems.... but it's no silver bullet.
and neither is NoSQL. @todo uses nosql for storage and i'm still not sure i havent made a serious error there...
Admin
It depends what your definition of large is. Using proper data warehousing techniques you could get nice high concurrent queries against a table with billions upon billions of rows. You'd want to utilize techniques such as indexed views, compression, partitioning, and highly tuned queries/indexes. There's a lot of considerations with these NoSQL solutions but they are very exciting IMO, I would just suggest not worrying so much about size but the entire app lifecycle personally.
Admin
I'll bite. Personally I'd store the logs in a RDBMS in a few scenarios: 1- It's highly joinable. Do I have other tables that need to join this data? If so, are they reasonably wide rows where having deduplicated copies of them make sense? Let's not forget the 'magic sauce' of many NoSQL solutions for 'web scale' is that they keep it in memory. Keeping highly redundant data in memory could be a bad design.
2- Does it need to be transaction controls and ACID compliance? Maybe the app has government compliance or PCI tier 1 requirements/etc. etc. where you have to enforce this level of compliance on your logging data. Perhaps you can get this with many NoSQL solutions but this is one area I'd look at.
3- You have sufficient in house experience and investment in hardware/software/support and you don't in the other areas nor are you really motivated to get it.
Meh those are the ones off the top of my head. Maybe someone will find them useful.
Admin
Admin
TRWTF is Roger being able to retire on his Sabbatical.
Admin
The site was down anyway while they were trying to reach the architect on his seagull-infested island.
Admin
The real wtf here is of course tcp/db.