The Daily WTF: Curious Perversions in Information Technology

2009-07-24 Reply Admin

dkf:
You really want that? Even the ones that charge by the hour? How the heck would you fit that much information on the screen and get the user to understand anything other than “Shit! There's lots of hotels in New York”.

Of course, I might not. But with pagination there is often no way to express that I want all results at once. Also, pagination does not solve the issue you raise: with pages there a still a lot of hotels in New York except now I can't do CTRL+F "Daily Rate" (or whatever search criteria the developer didn't deem necessary to include). A better design would be to have a message "hey, sure are a lot results .. you can refine your search like this etc"

dkf:
Wrong tool for that job. Why would you do things through a poxy UI when you could do a database query? If you can't do the database query, that's the real problem right there…

That's easy hand waving! In this example the reason is that the user store actually is a composition of an in-build user database (but with a horrible object/binary schema that you wouldn't be able to query) and an LDAP and an other LDAP neither of which I had access to.

Yeah, not the tools for the job but considering that pagination does not solve any UI problems (but rather, and that's the only appropriate use for it, performance problems) and at the same time breaks the age old "copy 'n' paste" interface I think it's never a good idea to include pagination "just because there could be many entries".

Never break Copy and Paste!

2009-07-24 Reply Admin

jasmine2501:
Yeah I usually only decide between one hotel and the next based on the prices at the bar.
These comment threads are always interesting. What I'm thinking after reading these is "this is why programmers don't get invited to meetings"

I find it funny that this article was about someone who was given the algorithm for solving his problem, and decided to complicate it beyond what was given, and the comment thread shows a strong desire to do just that.

Bad decision, my man, bad decision.

Those of us with true hotel-fu make our choice based upon the price (and other sundry conveniences and attractions) of the bar down the road. Once upon a time, in the early days of the 'net, we could look forward to the day when an online reservations system might actually give us this choice. Now, it's all corporate bullshit, and whether or not you should separate DB from BizzTalk from PHP (clue: PHP is in the cage, over there), and which friggin Mercator projections to use.

What you fail to realise is that programmers are perfectly sensible people, yet we are raised to a level akin to Boddhisatva.

We argue about all this crap precisely because we don't want to "get invited to meetings."

2009-07-24 Reply Admin

Kazan:
shimon:
A full-blown route calculation is, however, computationally expensive and there may also be substantial license fees involved.
http://www.movable-type.co.uk/scripts/latlong-vincenty.html
With this little function, you can calculate geodesic distances with accuracy within 0.5mm. For free. Still better than unprojected cartesian. Still giving you something comparable no matter how close to a pole you are.

and you're still, in your ignorance of actual user desires, still performing crow-flies calculations.

The User wants driving/walking/bussing directions. That involves using the A* algorithm on a network[see note] graph.

Note: the 'network' in this case is road/sidewalk/bus/subway watever - transit network.

This user doesn't givadam. I want a general idea of distance...I want to know if the hotel is 1 mi or 10 mi from $touristsite...don't care if it's 1.1 or 9.9.

mr

2009-07-24 Reply Admin

shimon:
in a situation like the code described above there are going to have to be some code filters on the list - things that their database may not have the most up to date data on and the only sane way of updating that data is on-demand [and not bother storing it in the DB at all] since it can change in real time and you're not the data source: an external entity is the data source.

I would better cache it in the database once per 30 minutes and not bother.

Pros: it becomes naturally easy to filter the results by the query alone.

Cons: the data might be outdated in those loooong 30 minutes. Oh wait, it might get as well outdated as soon as the availability information travels from the vacancy webservice to the website and from there to the client! So who will give a shit?

Check the most recent information only when the client selects a particular hotel, don't do it for all of them. You can then even update the availability data in the database without waiting for every-30-min-job. The bonus track is that the hottest hotels have the most accurate information.

Personally I hate it when I choose a hotel & the info is unrelated to what was displayed. Wasting time on more than one or two events like that at a site & I tend to try another site. I realize there will be some delay for a number of reasons, but I do expect reasonably fresh data on every initial query, and I do expect some way for a refresh. I might be sitting on a site for a long time...discussing hotels or travel options w/ the SO, offline interruptions, work, whatever.

mr

2009-07-24 Reply Admin

Smyle:
Listening to the bickering here, I've just come up with a great plan for deciding the best way to code a project when I have no idea where to start.
I explain the problem in great detail, decide on the worst possible implementation, and post it to TDWTF. Let the one-upsmanship begin, and steal the best code posted.

Shhh! Don't spoil it for the rest of us

2009-07-24 Reply Admin

Code Dependent:
items[] = loadAllItemsFromDatabase() items[] = filter ( items[] )
This disturbs me. Was there really no way to filter the items during the query? If, based on the desired location as specified by the user, it's possible to filter out the hotels by distance after they're returned, then why can you not simply create a query that has the desired location as a parameter and returns only the appropriate hotels, ordered by distance?

Same here. I'm not convinced that retrieving ALL the results to paginate is a good idea. I also like how the pseudo-code has a hard-coded "10" as well as an "itemsPerPage" variable. But well, it's only pseudo-code, maybe the implementation is actually smarter.

2009-07-24 Reply Admin

Andrew Brehm:
"The obvious, simple solution? Split up the results into multiple pages."
I hate, hate, hate Web sites that do this.

Web browser already allow for pagination since they only show the part of the page that is actually visible in the window.

The only thing pagination does for me, like the "summary" view in The Daily WTF, is create a need to load the page (or the next page) once more.

This is particularly annoying on mobile devices when you wait two minutes for the page to load only to find six summaries of stories rather than six stories or ten hotel listings rather than hundreds that can easily be scrolled through.

Yes, loading one very long page takes longer than loading a short version, but it also means that after it has loaded one is DONE with loading and can start reading.

Especially with hotels pagination only causes the user to decide between ten visible hotels or clicking another link not knowing if the next ten results are better or worse than the current ten results.

So my vote is that the real WTF in this story is the idea that the results should appear on more than one page. (Whereas I think that pages with random numbers of hotels being displayed are at least more entertaining than unnecessary ten-hotel lists.)

Luctus?

Yeah, I agree it tends to be annoying. I like the sites that give you the option. Other sites provide a print article function that puts everything on one page for you.

Otherwise, I expect decent sorting options to allow me to "filter" so to speak what comes up at the first page or two.

bc

2009-07-24 Reply Admin

Bim Job:

...

We argue about all this crap precisely because we don't want to "get invited to meetings."

Mod +1 QOTD

2009-07-25 Reply Admin

The hotel location changes in real time?

Yeah, thats my kind of hotel! (Or maybe its the brandy)

Jake Clarson · 2009-07-26 Reply Admin

Kazan:
Outlaw Programmer:
Any time your webapp features the term "logic handled by the front-end," you're probably doing something wrong. I don't see any reason why their server couldn't have done these things. The new guy should have slapped Bruno before starting the project.

[...] not everyone has the same exact definition of "Front end" and "back end". in much of web development the "back end" = storage/db and "front end" = your PHP, JSP, ASP.NET, whatever code.

Exactly, in large enterprise systems it's not uncommon to have a couple of tiers. The DB is typically at the lowest tier, with some data access tier just above that. Depending on the size of the system, there might a services (business rules) tier just above that and next to that there is view tier. All of these tiers run server side. Perhaps on the same server, but most likely at least the DB is a different server and the other 3 ones run on an application server (e.g. Java EE, .NET, etc). Finally, there's the display tier that consists of the HTML, scripts, etc that run in a user's browser.

It goes without saying that the kind of logic the article mentions should not be done in either the user's display tier or the server's view tier. It's most likely best at place at the services tier.

In general, there is a tension between doing stuff in the DB and doing stuff in the AS. The DB might be a central thing, while you might have several application servers. In terms of load distribution, it might make sense to offload some processing (like sorting) to your AS, even if the DB could theoretically do this. Caching solution like Oracle Coherence often take such an approach. Another rationale is that you might have global business rules in place that are not easily expressed in terms of SQL or where such rules apply to data from many different sources (webservices, files, perhaps user input) and you don't want to duplicate their definition for each source.

Remember that not all applications are of the pattern {"query db", "iterate over result-set and spit out html"}

Jake Clarson · 2009-07-26 Reply Admin

Kazan:
* and yes a DB - even a good RDBMS is a data storage and retrieval system. it is NOT for running programs.

Amen to that!

I've seen quite a lot of systems being build where the engineers thought the DB was basically everything and that anything and everything should be done in SQL. For those people, JSP, PHP, etc are just needed to wrap a little HTML around stuff that's returned from the DB.

For some reason, those systems always seem to fall apart when the system grows in complexity. Now I wonder why that is...

2009-07-26 Reply Admin

andreas:
having implemented such a feature in java - i can tell you that the problem described is not the real wtf. the problem, implemented correctly is not simple and not implemented in one day. user request page 5 of X for hotels surrounding point A which are available. steps to make: 1) determine query center 2) query DB for nearest elements using a spatially-enabled DB 3) take first Z elements where Z is a rough estimation of how many hotels you might need considering step 4. 4)weed out all unavailable hotels using a web service. 5) either sort elements by distance -> apply wgs->utm coordinate calculation or use a webservice for sorting the elements. (not recommended, can take infinite amount of time.) 6) apply windowing function on the resultset to display the exact number of results to the user.
there are so many fallacies here, even my captcha:facilisis

Here, in a nutshell, is what's wrong with Java development.

2009-07-27 Reply Admin

mr_smith:
Kazan:
ps aux:
http://php.about.com/od/mysqlcommands/g/Limit_sql.htm
items = pageNumber * 10;

LIMIT items, 10

It's (should be) faster to store the results in the session if each record is small - no need to have the database scan a non-random-access* collection for matching results, discard the first X each then return the next Y each time you flip the page when you can store your entire result set in a random-access collection.

but that's just my high-falutin university education speaking :P

[edit]

yes i know in general the collection IS random access - if you place no constraints on the data returned. when you have to search for results matching a pattern you effectively loose the random access because you don't have an ordered index listing ONLY entries that match that criteron.

So what if you are using a high falutin framework like ASP.NET and your session identifier is being stored in a cookie and you want to support a user doing multiple searches in different tabs/windows of the same browser?

I'd just use limit in the database unless you run into problems unless testing for the load you have doesn't work out so well.

In ASP.Net? I'd use this little thing called a POSTBACK to store the session id. Cookies are the realm of long term storage/multitab records, or of course (in your case) old school thinking.

2009-07-27 Reply Admin

And yes, I know about sharing of the link between friends and those various arguements against POSTBACK. But for the problem you stated it is a valid solution, and your suggestion of using cookies would have had the same problems.

2009-07-27 Reply Admin

That was a sad ending! I was expecting some inspiring message. I was thinking maybe Greg had taken that long because he had made the pagination system himself from scratch but Bruno didn't realize it and Greg got it working. The end.

2009-07-27 Reply Admin

julmu:
Yeah I hate Google too. http://www.google.com/search?q=wtf has only 38 million results. But if I want to know what's on the last page then I have to go to through millions of pages. If only I could click on Search, go to sleep and then in the morning when the page has loaded, I could simply press End on my keyboard.
The first 10 pages of results are irrelevant anyway. All the good stuff is hidden away on the last page because Google doesn't want us to find it.

luctus indeed

Silly me. I forgot to mention to use common sense. My bad.

If the hotel search gave you millions of results, the WTF is quite different from a pagination-related issue, I suppose.

2009-07-27 Reply Admin

I think that the REAL WTF is that so many people here don't understand the difference between good, and good enough...

2009-07-27 Reply Admin

Kazan:

If you do know the first thing about real world programming stop acting like an idiot and thinking that EVERYTHING can be done in data storage*.

and yes a DB - even a good RDBMS is a data storage and retrieval system. it is NOT for running programs.

Suggestion: get a refund for your Comp Sci "education". You were ripped off.

2009-07-29 Reply Admin

Observer:
ps aux:
http://php.about.com/od/mysqlcommands/g/Limit_sql.htm
items = pageNumber * 10;

LIMIT items, 10

For whatever reason, they had to cull out unavailable hotels after they retrieved a list from the DB. Also, hotels needed to be sorted by distance which could not be a part of the SQL query, so getting 10 results at a time would not work. The system needed ALL the results so it could process the list and THEN paginate.

Define a store procedure that execs the query into a cursor and then culls them according to a "driving distance" parameter...

... having said that, a database might need to interface with an external system to do that type of geographical computation. For instance, Oracle can run Java or contact a LDAP server, but I don't know about other databases.

The thing is, to compute driving distance from point A, you use a standard location field for each record point (.ie. a zip code for instance), each standard location field having an estimated area size. That zip code is distance zero; select those zip codes adjacent to it (these are distance 1), select those zip codes that are adjacent to those at distance 1 and label them distance 2, and so on until you reach the first estimation with an estimated area larger than the driving distance. That's your area of interest.

Then, you return those hotels that are in that area. The calculation for "area of interest" could be performed inside the database (as part of a stored procedure), if the DB has access to such spatial info.

Otherwise, the middle tier (not front-end or back-end or whatever PHP/JSP/xyz developers want to call it), middle tier can do that calculation, and pass to the db (again, via a store proc) an array of zip codes of interest, with the result being the same: get only those hotels that are in the zip codes of interest.

If sort by 'driving distance calculation' could also be done in the db, that'd be great. However, if that's not the case, the middle tier can do so w/o having to ZOMG!!11 SQL * FROM HOTELS!!

2009-07-29 Reply Admin

[quote user="Luis Espinal"] For whatever reason, they had to cull out unavailable hotels after they retrieved a list from the DB. Also, hotels needed to be sorted by distance which could not be a part of the SQL query, so getting 10 results at a time would not work. The system needed ALL the results so it could process the list and THEN paginate.[/quote]

Define a store procedure that execs the query into a cursor and then culls them according to a "driving distance" parameter...

... having said that, a database might need to interface with an external system to do that type of geographical computation. For instance, Oracle can run Java or contact a LDAP server, but I don't know about other databases.

The thing is, to compute driving distance from point A, you use a standard location field for each record point (.ie. a zip code for instance), each standard location field having an estimated area size. That zip code is distance zero; select those zip codes adjacent to it (these are distance 1), select those zip codes that are adjacent to those at distance 1 and label them distance 2, and so on until you reach the first estimation with an estimated area larger than the driving distance. That's your area of interest.

Then, you return those hotels that are in that area. The calculation for "area of interest" could be performed inside the database (as part of a stored procedure), if the DB has access to such spatial info.

Otherwise, the middle tier (not front-end or back-end or whatever PHP/JSP/xyz developers want to call it), middle tier can do that calculation, and pass to the db (again, via a store proc) an array of zip codes of interest, with the result being the same: get only those hotels that are in the zip codes of interest.

If sort by 'driving distance calculation' could also be done in the db, that'd be great. However, if that's not the case, the middle tier can do so w/o having to ZOMG!!11 SQL * FROM HOTELS!![/quote]

Quoting myself to present a caveat: Doing the adjacent zip calculation is a cheap way of doing things, and it would require to have estimated dimensions for each zip code.

There are tools and services out there for that kind of thing, and if a company provides that kind of web search capability, I would imagine they can afford to rent/buy that kind of computational service to meet their business needs.

2009-07-29 Reply Admin

Fast algorithms for these kind of things require preprocessing that commercial databases don't provide.

captcha: vereor -- Truly eeyore?

AndyCanfield · 2009-07-30 Reply Admin

The basic problem is that the kid does not have the algorithm correct, so his code does not work, and he tries to fix the code. I used to do this in my early days. Finally I began to build a list of algorithms which were independent of the implementation code. I think my first was how to read a sequential file and give multi-level subtotals. Not easy in FORTRAN.

Now, of course, I've learned to work out the algorithm first, then implement it.

2009-07-30 Reply Admin

julmu:
Yeah I hate Google too. http://www.google.com/search?q=wtf has only 38 million results. But if I want to know what's on the last page then I have to go to through millions of pages. If only I could click on Search, go to sleep and then in the morning when the page has loaded, I could simply press End on my keyboard.

You'd run out of memory and/or bandwidth limit first I'm afraid.

On a more sensible line of conversation, the last thing I want is for the web application to dump one thousand results to me and have to wait for it to render, then after I do something within one result I have to wait for the whole freaking thing all over again and so on. What the man describes would only ever be useful in lists you can new-tab off of.

2009-07-31 Reply Admin

Kazan:
The User wants driving/walking/bussing directions. That involves using the A* algorithm on a network[see note] graph.

You do realize that these 3 can differ a lot from eachother? Hell, just changing the direction can alter the result a lot. office parking lot -> nearest Burger King (car): 2.0km nearest Burger King -> office parking lot (car): 2.8km office parking lot <-> nearest Burger King (walk): 900m

IOW: You* have to decide on ONE of them, in which case the faster-to-calculate "as the crow flies" becomes viable again.

Using eg Haversine formula will give a good enough result. If anyone cares about a specific way to travel between A and B they will look at a map anyhow. Using street maps etc on the inital search is certainly overkill. Actually, Haversine already is a bit too much for almost anything < 20 km. If you really want to, let the user pick favorites from the search results and then offer a "distance analysis" using all the different graphs etc. I dare say on most sites that feature will have less PI than your "about us" page.

*: Of course you can leave the choice to the user, but most people will not care anyhow. Or lack the knowledge to make an informed decision.

2009-08-06 Reply Admin

Outlaw Programmer:
Here's The Real WTF:
The logic used to calculate distances, for instance, was handled by the front end, as well as real-time checks for vacancies would affect the sorting and remove some results from whatever was returned from the database.

Any time your webapp features the term "logic handled by the front-end," you're probably doing something wrong. I don't see any reason why their server couldn't have done these things. The new guy should have slapped Bruno before starting the project.

Exactly, I was surprised that this wasn't in the earlier comments...

Pagination Consternation

Leave a comment on “Pagination Consternation”