The Daily WTF: Curious Perversions in Information Technology

2014-03-27 Reply Admin

Like a true scripting guy He was Bourne, Bourne to be while, He can sleep so long He's never gonna die.

chubertdev · 2014-03-27 Reply Admin

NamingException:
If we were meant to write infinite loops, there would be a native language construct for them.

...

Java

2014-03-27 Reply Admin

NamingException:
If we were meant to write infinite loops, there would be a native language construct for them.

GOTO

2014-03-27 Reply Admin

Have the script write a PID file and do some basic sanity checking regarding other instances of itself before running.

That, and don't write to fixed files, use random tempfile/pipe names, etc.

2014-03-27 Reply Admin

Chelloveck:
RFox:
Of course if all of these monitor jobs got started when the system booted they'd all land on it about the same time. With cron presumably you'd skew the start times a bit.

Actually, it's probably the other way around. You could set up cron so everything started at a different time, but in practice you typically see all the jobs started with "0 * * * *" or "@hourly" or thrown into /etc/cron.hourly (depending on which flavor of cron you're using). These all start jobs at minute 0 of each hour.

I thought cron.hourly worked like cron.daily - it's started somewhere near the top of the hour/day, but not exactly on it as to spread out the load spike a bit. This might not be true on older versions of cron though.

2014-03-27 Reply Admin

There was even a server where every user was allowed to put their scripts, that was painfully slow to use, partially because at any time you could see a couple hundred instances of 'sleep'

Where? I must install Kaiten!

2014-03-27 Reply Admin

Martin:
This is a fail in it's own class. - Launches unrequired test command

Depending on the shell, it may not - on bash, at least, the "test / [" command is a shell built-in. Besides, on the scale of stuff routinely seen here, launching an unnecessary short-lived process once an hour is hardly worth mentioning...

tchize · 2014-03-27 Reply Admin

One now retired coder made such codes where i work. There was the ftp sending routines, which run in the model of

for each file in directory, send files to the remote ftp, sleep 1 minutes, try again.

Sometimes, the ftp client would hang on some bad ftp server, so a wrote a program in the form

while true check if there are waiting file older than 5 minutes, if so kill the ftp process and restart it. Sleep for 5 minutes.

Sometimes, the watch dog script would hang, so there was a crontab that restarted the watchdog every hour...

2014-03-27 Reply Admin

foo AKA fooo:
On the other hand, you now have the overhead of starting the sleep process (which is not usually a shell builtin) while cron sleeps internally.

You mean it sleeps with its child process? But you said they were eunuchs! Now how will we ever figure out how many sexes there are?

2014-03-27 Reply Admin

Chaarmann:
Instead of "$i = 1", he should have used ":".

$i have many identities.

2014-03-27 Reply Admin

Plus, you could just use /usr/bin/true.

2014-03-27 Reply Admin

Martin:
i=1 while [ $i = 1 ]; do
This is a fail in it's own class.

No need to use a "variable" (it never gets changed, so not very "varible"), 1 = 1 would do the same

Launches unrequired test command

= compares strings, you should use -eq to compare numbers.

No, you failed. First, test is a builtin, so it's not forking. Second, use (( i == 1 )), it's the numeric context. For example:

x=( 9 4 13 2 5 )
y=( apple banana pie )
for (( i = 0; i < 100; i += 1 )); do
    if (( j = i ^ 85, j % 3 )); then
        echo "${y[i % 3]}: x[$(( (j << 1) % 5))] * i = $(( x[(j << 1) % 5] * i ))"
    fi
done

oheso · 2014-03-27 Reply Admin

nitePhyyre:
So what do you do if you have a long running script, and want to re-run it every hour after it is finished?
...

What's the best option here?

No doubt there's a better way to do this (bash is definitely not my forte), but I work with a third-party app. which has a cron job of indeterminate duration. (Based on a variety of factors, it may run a simple process which is over in seconds, or it may run a more lengthy housekeeping process which can take more than an hour.) It's in PHP so it's not like I can check running processes to see if it's completed -- PHP should always be there (or apache, most likely). And the job has no output I can test for completion.

So what I came up with is a script which touches a specific lock, then calls the housekeeping process, then deletes the lock. It probably doesn't work as intended -- calling the process probably spawns a thread and then my shell deletes the lock while the thread is still running. I suppose if I were bothered enough, I could look into keeping track of the threads.

I also have a once-a-day job which backs up the app. in question, a process which can take two or three hours to complete (gigs and gigs of data, not high priority enough to really invest in anything faster). So this again starts by touching the same lock. That way at least I don't have the cron housekeeping job trying to run while the app. is being backed up.

I think the Rolls-Royce solution would be to modify the third-party housekeeping process to log its progress and to delete the lock when it's through (rather than having the calling shell script delete the lock). Easily done as it's all in PHP. But that means I would have to redo this bit each time the third-party app. is upgraded (every few weeks) and test it again. Which, again, if this were important enough I could be arsed to do it ....

2014-03-27 Reply Admin

Clark:
Plus, you could just use /usr/bin/true.

Not on this site. Here's what you need on this site:

# cat >/usr/bin/filenoutfound
echo filenotfound: filenotfound
<Ctrl+D>
# chmod +x /usr/bin/filenotfound

Then you know what you could use.

2014-03-28 Reply Admin

Norman Diamond:
Clark:
Plus, you could just use /usr/bin/true.
Not on this site. Here's what you need on this site:
# cat >/usr/bin/filenoutfound
echo filenotfound: filenotfound
<Ctrl+D>
# chmod +x /usr/bin/filenotfound
Then you know what you could use.

/usr/bin/filenotfound: file not found

2014-03-28 Reply Admin

Sigmund Freud:
foo AKA fooo:
On the other hand, you now have the overhead of starting the sleep process (which is not usually a shell builtin) while cron sleeps internally.
You mean it sleeps with its child process? But you said they were eunuchs! Now how will we ever figure out how many sexes there are?

Of course, there are three sexes: little endian, middle endian, big endian and endianness not found.

AndyCanfield · 2014-03-28 Reply Admin

"amateur unix sysadmin"? I'm not an amateur! I get paid for it. I may not be very good at it, but I get paid. More in the category of "ignorant professional". This comments session has had lots of interesting alternatives to the crontab jobs that I've been depending on. Thanks, guys.

2014-03-28 Reply Admin

Muphry:
Norman Diamond:
Clark:
Plus, you could just use /usr/bin/true.
Not on this site. Here's what you need on this site:
# cat >/usr/bin/filenoutfound
echo filenotfound: filenotfound
<Ctrl+D>
# chmod +x /usr/bin/filenotfound
Then you know what you could use.
/usr/bin/filenotfound: file not found

outch

2014-03-28 Reply Admin

Chris Angelico:
If this is a cron job, then it would eventually bring the server to its knees because it'll start more and more of these processes. (If, instead, you have a cron job that invokes an Upstart job, for instance, you won't have that problem. Or just, yaknow, have a cron job that does the check once and terminates, and gets invoked every hour. But that's way too obvious for a Scripting Expert.) The entire purpose of cron facilities is to relieve you of the need to invoke a ton of infinitely-looping processes like this.

Thank You Captain Obvious.

2014-03-28 Reply Admin

The last thing your script should do is

system('at', $path_to_script, 'now', '+', '1', 'hour')

Do a man at for more info

tgape · 2014-03-28 Reply Admin

Bourne shell (not to be confused with bash, even in "incompatible" mode) implements very few shell builtins. As such, just about every "Bourne shell" command (apart from assignments that do not invoke subshells, flow control (for, while, etc) that do not invoke subshells, and other commands that couldn't be implemented in separate commands (bg, for example) forks and then execs another program. Note that assignments or flow control statements that do invoke subshells still involve forks, and probably execs.

exec is one of the most resource intensive system calls, probably the most resource intensive call that's commonly used. As such, Bourne shell scripts frequently run at least an order of magnitude slower than the same script rewritten in a more modern shell, such as bash or zsh. Those are still shells, but they implement most of the commonly used commands, like sleep, as shell builtins.

Note that there's another effect of having a lot of Bourne shell scripts running that is applicable here: the commands they are executing do not last long. Because of this, they don't show up on top much or on ps often. A Bourne shell script that top claims is only using 2% of the CPU could easily be using 15% or more of the CPU in reality, due to the fact that its kids are doing all the work. Few OSes properly recognize the effect that fork/exec bombing has on drive I/O as well.

I've been tasked with cleaning up the performance of system that was running sluggishly with only about 10% of CPU time showing as used by top. The admin of the system claimed that too many resources were being consumed by sleeping processes, but all I had to do was convert a few Bourne shell scripts to zsh scripts. (Bash would have almost worked as well, but bash was not on the list of approved programs.) Admittedly, this did cause all of the sleep processes to vanish, because sleep is a zsh (and bash) builtin.

The author of the Bourne shell scripts was astonished that in the same effort, I reduced those scripts' run time by over 95%. Later on, one of those scripts was converted to perl, and received another 97% reduction in run time. A further conversion to C did not have a huge difference in run time, because the perl version was already disk bound. But it did drop the CPU time used by the script by about 90% from what the perl version was using. (Note: I wasn't involved in the conversion to C; it was done by the original author who hadn't realized Bourne shell scripts would be less efficient than more modern shell scripts.)

tgape · 2014-03-28 Reply Admin

John:
The last thing your script should do is
system('at', $path_to_script, 'now', '+', '1', 'hour')

Do a man at for more info

Not nearly as helpful as you think, because the sleep is not the problem. Bourne shell itself is the problem. You're like the doctor who, when presented with a patient who complains his arm hurts when he straightens it, tells him to not straighten it. It may address the issue the patient's aware of, but not the underlying problem.

2014-03-28 Reply Admin

This is nothing. Our company (busy being out-outsourced by local govt politics) had to face a problem where a local govt "IT dude" had seen one or our scripts, decided it did stuff X times so just cut 'n' paste X times, entirely missing the whole point of the test within the loop.

Our loop: test forever, if fail bail out with code.

Their loop: do it multiple times even if successful, if it fails repeat until "paste" times completed. Don't even return result.

2014-03-29 Reply Admin

coder:
if you really want to get pedantic, while [ 1 ]; do will work just as well, why even invoke a comparison when all you want is 'true'

No, if you really want to be pedantic, while true is the way to go, and you should use while : if you're playing code golf.

2014-03-31 Reply Admin

tgape:
Bourne shell (not to be confused with bash, even in "incompatible" mode) implements very few shell builtins. As such, just about every "Bourne shell" command (apart from assignments that do not invoke subshells, flow control (for, while, etc) that do not invoke subshells, and other commands that couldn't be implemented in separate commands (bg, for example) forks and then execs another program. Note that assignments or flow control statements that *do* invoke subshells still involve forks, and probably execs.
exec is one of the most resource intensive system calls, probably the most resource intensive call that's commonly used. As such, Bourne shell scripts frequently run at least an order of magnitude slower than the same script rewritten in a more modern shell, such as bash or zsh. Those are still shells, but they implement most of the commonly used commands, like sleep, as shell builtins.

Note that there's another effect of having a lot of Bourne shell scripts running that is applicable here: the commands they are executing do not last long. Because of this, they don't show up on top much or on ps often. A Bourne shell script that top claims is only using 2% of the CPU could easily be using 15% or more of the CPU in reality, due to the fact that its kids are doing all the work. Few OSes properly recognize the effect that fork/exec bombing has on drive I/O as well.

I've been tasked with cleaning up the performance of system that was running sluggishly with only about 10% of CPU time showing as used by top. The admin of the system claimed that too many resources were being consumed by sleeping processes, but all I had to do was convert a few Bourne shell scripts to zsh scripts. (Bash would have almost worked as well, but bash was not on the list of approved programs.) Admittedly, this did cause all of the sleep processes to vanish, because sleep is a zsh (and bash) builtin.

The author of the Bourne shell scripts was astonished that in the same effort, I reduced those scripts' run time by over 95%. Later on, one of those scripts was converted to perl, and received another 97% reduction in run time. A further conversion to C did not have a huge difference in run time, because the perl version was already disk bound. But it did drop the CPU time used by the script by about 90% from what the perl version was using. (Note: I wasn't involved in the conversion to C; it was done by the original author who hadn't realized Bourne shell scripts would be less efficient than more modern shell scripts.)

Very interesting lesson. Glad I've only ever used the Bourne Again SHell and never had to deal with it's older friend.

2014-03-31 Reply Admin

Bobby Tables:
Bourne-again shell. Something commonly known as bash or the "bash shell" (which would actually make it the "Bourne-again shell shell"). Or it could be the Bourne shell (sh) it's based on. At any rate - it's definitely a shell.

It's also a reference to the 1969 Steppenwolf song "Born to be Wild"

2014-04-01 Reply Admin

Scripts doing infinite loops and then sleeping for an hour were often the norm. The reasoning behind these being that "cron used too much server resources!"

No, the reasoning being that cron is not installed everywhere.

For example, Cygwin doesn't install cron by default, and cron is actually a pain in the ass to install on Cygwin because it involves setting up a Windows service to run it. So if you need to run a periodic task in Cygwin on some Windows box, an infinite loop with a sleep in it is perfectly practical to leave running in a terminal window.

tgape · 2014-04-08 Reply Admin

Crom:
Scripts doing infinite loops and then sleeping for an hour were often the norm. The reasoning behind these being that "cron used too much server resources!"
No, the reasoning being that cron is not installed everywhere.

I like how some people can read someone else's explanation for why things were done at a certain place, and, without having actually been a part of the discussion for why things should be done a certain way at that location, feel the need to argue that the person was wrong, and things were done for an entirely different reason...

Crom:
For example, Cygwin

that didn't exist yet when the things had been done.

Now, the story didn't say exactly when the scripts were written, but I've worked in a shop much like the one described, which had been doing things the way they were doing since 1993, before Cygwin even existed. Also, it clearly wasn't that cron didn't exist everywhere, because they'd been a pure unix shop at the time they started the thing, and cron certainly existed there. And, while bash and zsh may not have been in a viable state at that time, ksh certainly was and switching to it gave a significant performance boost (though ksh does not implement as many builtins as bash or zsh - it was much better, which was the point.)

Bourne to be While

Leave a comment on “Bourne to be While”