• Fred4 (unregistered) in reply to jergosh
    jergosh:
    old bloke:
    http://www.ietf.org/rfc/rfc959.txt 3.1.1.1. ASCII TYPE This is the default type and must be accepted by all FTP implementations.

    I believe that the command line ftp in Linux distros is set to binary by default (common sense!).

    The standard says servers must be ASCII by default. Well-written clients will usually send a "BINARY" command immediately after logging in to override this though.

  • MrJohnson (unregistered)

    After installing Vista random file corruption was a real problem on my home computer. Downclocking the CPU solved the problem though.

  • Dan (unregistered)

    WTF...

    -Were you doing NOT using your tape backup strategy?! -Were you doing using FTP to backup stuff!!! -Were you doing using an INTERN to FTP stuff!! -Were you doing using a COMMAND PROMPT FTP CLIENT?? (Or an ftp client that you actually needed to type in 'bin' to switch modes).

  • (cs)

    A real geek would have typed, "corruption never^H^H^H^H^H rarely happens".

  • Mike Edenfield (unregistered) in reply to Darth Vader

    Either he was aware of the fact that it doesn't work that way, or he tried it and found out for himself :) If it were that simple, the answer would just be to download the file through FTP again, in ASCII mode, and it would "put back" the original line feeds anywhere it stripped them out on the way up.

    I suspect the problem was the backups were done on Windows and uploaded to UNIX, so the extra \0x0D in the Windows newline sequence was stripped off and would need to be put back. This idea falls over when you realize that binary files are chock full of standalone \0x0A characters too, and sticking a random \0x0D in front of every one isn't making things any better.

  • (cs) in reply to Confused
    Confused:
    I work for a company with a product that performs FTP, so I am really getting a kick out of these replies...
    "Some of you guys are very good at making it sound like you know what you are talking about. But trust me.... You don't. I think you just want to make yourself sound smart, when in reality you dont know what you are talking about. This is how bad info gets passed around. If you dont know about the topic....Dont make yourself sound like you do. Cuz some Worse Than Failure readers believe anything they hear."

    Man, that thing would apply to just about every thread on this site. Story gets posted. User posts condescending "solution" that wouldn't actually work. People jump on him and offer their own solutions. Repeat.

  • Mike Edenfield (unregistered) in reply to Mattastic

    According to the FTP protocol standard, ASCII is 8-bit NVT-ASCII, the same format defined by telnet. The main problem is that the standard also defines a standard end-of-line sequence and requires the sender to convert from their internal sequence to the protocol-defined sequence during transfer, which actually changes the contents of the file.

  • Milkshake (unregistered) in reply to jimlangrunner
    jimlangrunner:
    derby:
    When I was a kid, we used a keyboard

    keyboards! You were lucky. We had to flip toggle switches all day long if we wanted to interact with our computers. Kids and there damn fancy buffered input.

    toggle switches! Lucky. We had to flip those bits by hand, with magnets. I won't even tell you how much fun it was to read it back out! Kids and their damned mechanical input devices.

    You had hands?!!

  • (cs) in reply to Dark Shikari
    Dark Shikari:
    The real WTF is the fact that the FTP software defaulted to ASCII for file transfer.

    I think most of them guess the transfer type by peeking at the first X bytes and determining the ratio of printable characters to nonprintable characaters. It not would be surprising for table data to look like ASCII.

  • mike5 (unregistered) in reply to Darth Vader
    Darth Vader:
    No, the real WTF is that he used a month old backup instead of running sed -e 's/\r\n/\n/g' on the backed up file. Seriously, WTF was he doing as an admin?

    No, the real WTF are smart-asses who think they know everything but forget that in ASCII Chr(26) means EOF regardless of what comes after that.

  • (cs) in reply to wtfwtf
    wtfwtf:
    The new WTF is the crazy opinions in the articles. Files don't get corrupted? On what planet, in what alternate reality, are bits always stored perfectly, never get flipped, and a program never does anything wrong?? I want to live there. Maybe they even name their wtf sites properly.

    To put it bluntly: I have never worked on a large (say > 50MB) Word document that didn't get corrupted at least ONCE every few weeks. Any by corrupted I mean you work on it one day, save and close, then the next day you try to open it and Word complains that it's not a valid file format.

  • Sharkie (unregistered)

    I call FAKE...

    Had this been an actual corporation, the following would have actually happened:

    1. The sys admin and Ness would have been fired for incompetence.
    2. The interns responsible would have been promoted to middle or upper management.
    3. The ftp process would have been off-shored to write a custom set of scripts that fed the data onto punch-cards that would then later be bin-hex transcribed onto a backup system.
  • Older than thou (unregistered) in reply to Milkshake
    toggle switches! Lucky. We had to flip those bits by hand, with magnets. I won't even tell you how much fun it was to read it back out! Kids and their damned mechanical input devices.
    You had machines?!?!
  • Dan (unregistered) in reply to jimlangrunner

    Magnets! Hah! We had to lay sticks on the ground and write binary code with them. If we didn't check in by sunset, pa would beat us with the sticks and we'd have to start again.

  • Jeff Bell (unregistered)

    In highschool, the only computer had toggle switches and core memory.

    Once the power went out before I could save my program, but by jumping to the interrupt handler I was able to save it after power came up.

  • much older dude (unregistered) in reply to Dan
    Dan:
    Magnets! Hah! We had to lay sticks on the ground and write binary code with them. If we didn't check in by sunset, pa would beat us with the sticks and we'd have to start again.

    When I was a youngun, trees and shrubs hadn't yet developed... there were no sticks. We had to dig troughs (empty=0, full of water=1) to write programs.

    while (true)
  begin day
    do
      hunt food
      throw rock at food
    while not food hit by rock
    do
      chase food 
      if food turns around and chases me
         shit
         run
    while not food dead
    drag food to cave
    beat woman to prepare food
    eat
  end day
  • Spectre (unregistered) in reply to zip
    zip:
    Evo:
    Hmmm, let's say we're missing about 2GB (= 2.000.000.000 bytes (Yes, really)). We could just do: head -c 2000000000 /dev/random and append that to the file. If we're lucky, we have the original file....

    A nice chance of 1:256^2000000000...

    Then again, pipe enough data to a file, run it, and you'll have recreated Windows. What? This isn't an illegal copy of Windows, it's simply the first X bytes from /dev/random !

    What the heck are you trying to say??

    I guess he's trying to say that... Hey, we've got another excuse for a pirated copy of Windows!

  • (cs) in reply to Saladin
    Saladin:
    Confused:
    I work for a company with a product that performs FTP, so I am really getting a kick out of these replies...
    "Some of you guys are very good at making it sound like you know what you are talking about. But trust me.... You don't. I think you just want to make yourself sound smart, when in reality you dont know what you are talking about. This is how bad info gets passed around. If you dont know about the topic....Dont make yourself sound like you do. Cuz some Worse Than Failure readers believe anything they hear."

    Man, that thing would apply to just about every thread on this site. Story gets posted. User posts condescending "solution" that wouldn't actually work. People jump on him and offer their own solutions. Repeat.

    We're going to have to ask you two to take your farking memes elsewhere.

  • Bobbo (unregistered) in reply to seejay
    seejay:
    I'll try to remember that when my 2GB thumb drive wouldn't allow me access to a directory last week

    That's some high quality tense-mixing. Can I borrow your time machine please?

  • (cs) in reply to Asd
    Asd:
    ASCII translation in FTP is the real WTF. One of the funnier cases of "corruption" that I have seen was a developer shutting down Oracle with shutdown abort, and not realizing that was anything unusual. Where did he get they idea that would be OK?
    Shutdown abort does not cause file corruption - it just means that any uncommitted transactions are lost.
  • AdT (unregistered)

    No one wants backup, they only want restore...

  • Crash Magnet (unregistered) in reply to much older dude

    When I was a youngun, trees and shrubs hadn't yet developed... there were no sticks. We had to dig troughs (empty=0, full of water=1) to write programs.

    while (true)
  begin day
    do
      hunt food
      throw rock at food
    while not food hit by rock
    do
      chase food 
      if food turns around and chases me
         shit
         run
    while not food dead
    drag food to cave
    beat woman to prepare food
    eat
  end day
    [/quote]

    I really think you should have put the "food hit by rock" loop inside of the "food dead" loop.

    Otherwise, there is the terrible possiblity you would spend the whole day running from food and never throw another rock?

    Say, what kind of debugger did you guys use in rock and stick days?

    Crash Magnet

  • Loren Pechtel (unregistered)

    I've had a nasty run-in with random corruption before. It was too catastrophic to trace properly, we quickly backed off the system. It appeared to be a network driver ignoring errors on the wire and returning a block of zeroes in place of the data that was supposed to be written.

  • Steve (unregistered)

    Files never get corrupted, eh?

    I guess Skeptical Sally's never had a system crash while saving a file.

    Lucky her.

  • Hanneth (unregistered) in reply to Thief^

    Automate, test. Pffff, if it compiles ship it. What is this testing you speak of?

    Thief^:
    Someone was still using a command-line ftp? How old is this?

    EDIT: More to the point, why wasn't it automated, and why did they never test the backups after changing to the new system?

  • seebs (unregistered) in reply to gerrr
    gerrr:
    I use the ftp command line in dos all the time. I am sure if you have a non-gui version of Linux or Unix it would be a very similar client.

    Well, given that MS just stole the old Berkeley one, yes. Only it would be about 15 years newer, and would default to binary.

  • dkf (unregistered) in reply to Confused
    Confused:
    I hate, loathe and despise ASCII mode. The concept is flawed to begin with.
    Maybe you've never been "fortunate" enough to use FTP with systems that used the EBCDIC charsets? In those cases, you really really REALLY want to have ASCII mode set by default so that your directory listings work, and back when FTP as a protocol came into service such machines were still fairly common. (They were still hanging on in the '90s, for goodness sake!)

    If this was a WTF from any time recently, the Real WTF! is that they're using FTP at all. It's a horrible protocol in rather many ways, and WebDAV (or even just plain HTTP) is much better in a number of important technical ways.

  • joomama (unregistered) in reply to gerrr
    gerrr:
    Thief^:
    Someone was still using a command-line ftp? How old is this?

    I use the ftp command line in dos all the time. I am sure if you have a non-gui version of Linux or Unix it would be a very similar client.

    Yeah I use the commandline ftp client in windows all too often (it's a piece of junk that can't even do simple wildcards). In linux I do have gui ftp clients, but it is much quicker to use a commandline (s)ftp or scp or whatever.

  • (cs)

    Uploading binary as ascii? Wow, that takes me back...

    My dad told me a story once; there was this biomedical student who had to experiment on a dead cat over a period of some 36 hours, and his testing equipment sent the data back to the server. Unfortunately, the server had been configured to treat the incoming data as ASCII.

    Whereupon the student had to wait a long time indeed before he was granted another dead cat.

    Remember: Whenever you forget to specify your data as ASCII or binary, God kills a kitten. Or, well, a cat.

  • George without WB (unregistered) in reply to nobody
    nobody:
    derby:
    When I was a kid, we used a keyboard

    keyboards! You were lucky. We had to flip toggle switches all day long if we wanted to interact with our computers. Kids and there damn fancy buffered input.

    Luxury! We had to magnetize little magnetic cores, and check the magnetization with tiny magnets!

    I'm still amazed by the EM crowd, back in my days computers where just plain and simple mechanical calculating machines

    Pascal :D

    Captcha: pirates (yes back in those days)

  • (cs)

    Secondary RWTF-in-the-comments Word failing to read back a file it's saved isn't 'random file corruption' - it's a bug in Word. Similarly a system crash while writing the file isn't either, tho' arguably it's closer. Random file corruption surely means some failure of the storage medium, not imperfect coding. Thus things like the legendary gamma-ray flip of a bit in RAM, or a flaw in the surface of a disk which was not correctable by the disk firmware would qualify, not some lack of testing on large Word files in Redmond.

  • (cs) in reply to dkf
    dkf:
    Confused:
    I hate, loathe and despise ASCII mode. The concept is flawed to begin with.
    Maybe you've never been "fortunate" enough to use FTP with systems that used the EBCDIC charsets? In those cases, you really really REALLY want to have ASCII mode set by default so that your directory listings work, and back when FTP as a protocol came into service such machines were still fairly common. (They were still hanging on in the '90s, for goodness sake!)

    If this was a WTF from any time recently, the Real WTF! is that they're using FTP at all. It's a horrible protocol in rather many ways, and WebDAV (or even just plain HTTP) is much better in a number of important technical ways.

    One of the best bugs I've seen was when I was working on a program that interfaced with an IBM zSeries mainframe. (Yup, an EBCDIC machine.) We were making a proof-of-concept program for a new management API interface* that sent back strings.

    [* Yes, I do mean to repeat the "interface"; the way you called APIs changed from RPCs to text over sockets.]

    We looked at the data that was coming over the wire, and for a while was very confused. The strings were there, but between them there were three null bytes followed a byte of apparent nonsense, then the next string.

    It turned out what was happening is that the server was sending counted strings, and the four bytes between strings was the length. We initially didn't see this because the length (in binary) was going through the same EBCDIC translation as the string proper!

    I don't know what they ended up doing with that, but what we did was read until the first null byte, then ignore the four bytes until the next string.

  • (cs) in reply to Evo
    Evo:
    Hmmm, let's say we're missing about 2GB (= 2.000.000.000 bytes (Yes, really)).

    Uhmm.... You might want to reread that wiki entry. I don't think it says what you think it says.

  • M.G. (unregistered) in reply to mike5
    mike5:
    Darth Vader:
    No, the real WTF is that he used a month old backup instead of running sed -e 's/\r\n/\n/g' on the backed up file. Seriously, WTF was he doing as an admin?

    No, the real WTF are smart-asses who think they know everything but forget that in ASCII Chr(26) means EOF regardless of what comes after that.

    If that's the real WTF, then what are the smart-asses who think they know everything but forget that not every OS has the same EOF character?

    Furthermore, ASCII does not define character 26 as an EOF character, it is the "substitute character" character. The correct "EOF" character in ASCII is character 04, which means "end of transmission."

    So, ASCII 26 does not mean EOF "regardless of what comes after it" in every case, nor does ASCII 26 really mean EOF by the ASCII standard.

    CAPTHA: ninjas - like how I karate-chopped your post.

  • (cs) in reply to EvanED
    EvanED:
    dkf:
    Confused:
    I hate, loathe and despise ASCII mode. The concept is flawed to begin with.
    Maybe you've never been "fortunate" enough to use FTP with systems that used the EBCDIC charsets? In those cases, you really really REALLY want to have ASCII mode set by default so that your directory listings work, and back when FTP as a protocol came into service such machines were still fairly common. (They were still hanging on in the '90s, for goodness sake!)

    If this was a WTF from any time recently, the Real WTF! is that they're using FTP at all. It's a horrible protocol in rather many ways, and WebDAV (or even just plain HTTP) is much better in a number of important technical ways.

    One of the best bugs I've seen was when I was working on a program that interfaced with an IBM zSeries mainframe. (Yup, an EBCDIC machine.) We were making a proof-of-concept program for a new management API interface* that sent back strings.

    [* Yes, I do mean to repeat the "interface"; the way you called APIs changed from RPCs to text over sockets.]

    We looked at the data that was coming over the wire, and for a while was very confused. The strings were there, but between them there were three null bytes followed a byte of apparent nonsense, then the next string.

    It turned out what was happening is that the server was sending counted strings, and the four bytes between strings was the length. We initially didn't see this because the length (in binary) was going through the same EBCDIC translation as the string proper!

    I don't know what they ended up doing with that, but what we did was read until the first null byte, then ignore the four bytes until the next string.

    That strikes me as one of the better arguments I've heard for preferring C-style strings to Pascal-style strings. Admittedly, if you invoke your character encoding converter properly it should always be possible to avoid converting the integer representing the length, but this type of bug couldn't really happen with a null terminator (assuming that no character set is stupid enough to use 0 as the encoding of a character...)

  • noname (unregistered) in reply to zip
    zip:
    Evo:
    bling:
    Darth Vader:
    No, the real WTF is that he used a month old backup instead of running sed -e 's/\r\n/\n/g' on the backed up file. Seriously, WTF was he doing as an admin?

    Maybe you have a sed recipe for appending all the stuff that was truncated after the 'end-of-file' character was encountered, too?

    Hmmm, let's say we're missing about 2GB (= 2.000.000.000 bytes (Yes, really)). We could just do: head -c 2000000000 /dev/random and append that to the file. If we're lucky, we have the original file....

    A nice chance of 1:256^2000000000...

    Then again, pipe enough data to a file, run it, and you'll have recreated Windows. What? This isn't an illegal copy of Windows, it's simply the first X bytes from /dev/random !

    What the heck are you trying to say??

    I think he's explaining why Vista took so long.

  • og (unregistered) in reply to Crash Magnet

    me see mammoth me move rock me see mammoth me move more rock

    rock rock

    this much mammoth

    (rock on)

  • anonymous (unregistered) in reply to chrismcb
    chrismcb:
    Evo:
    Hmmm, let's say we're missing about 2GB (= 2.000.000.000 bytes (Yes, really)).

    Uhmm.... You might want to reread that wiki entry. I don't think it says what you think it says.

    You might want to reread chrismcb's post. I do not think it says what you think it says.

  • James Schend (unregistered) in reply to dkf
    dkf:
    Confused:
    I hate, loathe and despise ASCII mode. The concept is flawed to begin with.
    Maybe you've never been "fortunate" enough to use FTP with systems that used the EBCDIC charsets? In those cases, you really really REALLY want to have ASCII mode set by default so that your directory listings work, and back when FTP as a protocol came into service such machines were still fairly common. (They were still hanging on in the '90s, for goodness sake!)

    If this was a WTF from any time recently, the Real WTF! is that they're using FTP at all. It's a horrible protocol in rather many ways, and WebDAV (or even just plain HTTP) is much better in a number of important technical ways.

    FTP is, indeed, a terrible protocol. But suns can flare to life, form planets, evolve life, and die during the length of time it takes WebDAV to upload a 10,000 file website. Or at least Apple's implementation of it, which appears to reconnect for every single file...

  • Watson (unregistered) in reply to Anodyne
    Anodyne:
    Oh... no friend, no. Not having BIN on sends the file as 7bit ASCII instead of 8 bits to save bandwidth. I don't think line ending having anything to do with this WTF
    Not with any FTP setup I've ever heard of. Anyway, line ending translations are quite bad enough...

    ASCII/BIN ... CRLF/LF/CR ... defaulting to ASCII .... No doubt there were very good reasons for all of these at one time or another. Remember people: your quick hack may one day end up becoming an ineradicable standard protocol.

  • Akerbos (unregistered) in reply to Watson

    You assume that Word does random file corruption? Well, you hold those MS coders in high regard.

    I think there is a major difference between the feeling of randomness, which might be translated as arbitrariness or indeterministicness or "I do not know all parameters", or real random, which might be an illusion at all.

    I bet that there exists a proper reason for every "random file corruption" - we may not be able to find it, though.

  • (cs) in reply to Crash Magnet

    [quote user="Crash Magnet"]When I was a youngun, trees and shrubs hadn't yet developed... there were no sticks. We had to dig troughs (empty=0, full of water=1) to write programs.

    while (true)
  begin day
    do
      hunt food
      throw rock at food
    while not food hit by rock
    do
      chase food 
      if food turns around and chases me
         shit
         run
    while not food dead
    drag food to cave
    beat woman to prepare food
    eat
  end day
    [/quote]

    I really think you should have put the "food hit by rock" loop inside of the "food dead" loop.

    Otherwise, there is the terrible possiblity you would spend the whole day running from food and never throw another rock?

    Say, what kind of debugger did you guys use in rock and stick days?

    Crash Magnet[/quote]

    What about the event that you die? Do you continue the loop? shouldn't there be a break in there in the event you die? =p

  • VeXocide (unregistered)

    Why ? Why does it always have to go wrong before management decides to buy tapes..

  • (cs) in reply to Darth Vader
    Darth Vader:
    No, the real WTF is that he used a month old backup instead of running sed -e 's/\r\n/\n/g' on the backed up file. Seriously, WTF was he doing as an admin?

    gotcha - you sad muppet, you - How do know which \r\n-sequence was added by the ftp-ing and which one was there in the original data ? Hah, Hah ... ?

    Hmmmmmmmmm ... I thought so.

  • (cs) in reply to operagost
    operagost:
    Asd:
    One of the funnier cases of "corruption" that I have seen was a developer shutting down Oracle with shutdown abort, and not realizing that was anything unusual. Where did he get they idea that would be OK?
    With 7.1 for VMS at a company I used to work with, this was the only way to get Oracle to actually shut down: - Shutdown abort - Startup restrict - then Shutdown

    This must have been okay with Oracle because we had a support contract with them.

    With Oracle db you support Oracle !!!

    But seriously: you have an Oracle maintenance contract so (a) Oracle will ackknowledge your existence and talk to you and (b) to cover your arse and NOT (c) to justify shoddy operations procedures like your above folklore.

  • Tom_fan (unregistered) in reply to Andrey

    There is only one way to test a backup: RECOVERY!

  • * (unregistered) in reply to Anon
    Anon:
    obediah:
    I guess "random" depends on your frame of reference, a random coin flip is simply a function of a few million (or more) variables.

    Ever heard of quantum mechanics?

    Which may only appear to be random because of the tiny tots of energy needed to influence quantum particles.

  • * (unregistered) in reply to Sarcastic Ahole
    Sarcastic Ahole:
    you guys are all dorks
    Yes we are.
  • (cs) in reply to Andrey
    Andrey:
    Forgetting to manually set BIN mode in an ftp client that doesn't do it for you (99% do) is an entirely forgivable mistake. What is inexcusable is the fact even `ls -l` on the local machine and the backup server would've shown that something is seriously messed up. In a critical process such as this, the absolute minimum of testing would've been to compare md5 hashes of the local files and the ones on the remote server to make sure no corruption happened. While I doubt that a place too cheap to buy a few extra tapes would have a test server for test restores, a couple of extra checks in the backup script/instructions would've revealed the problem really quickly.

    Guys, Guys, Guys: the actual technical failure described in the OP is a simple, yet devstating, one which can happen to anyone, especially an inexperienced user/intern. The point is not the failure but the problems in the backup processes that caused the described failure:

    • PHB management with penny pinching attitude (the very root cause).
    • A backup administrator and it's supervisor designing and approving a (presumably daily) manual process based on an old, insecure protocol like FTP.
    • A manual backup process run by an intern which by definition is inexperienced, (possibly) not educated about backups and is expected to make mistakes.
    • The complete lack of testing/verifying the backups.

    I recommed to dismiss the PHB cancelling the original purchase request for new tapes (if there are two PHB's or more involved, just cut their bonuses to recover the extra cost incurred due to the failure), the backup administrator's supervisor and the backup administrator. Don't fire dismiss the intern due to this: interns by definition are expected to make mistakes (as stated above) and you will never get away with a dismissal if the intern sues.

    As to database file corruption, I have seen it again and again: Oracle 8 cluster installation under SGI Irix using database files on raw partitions (during normal operations because Oracle was buggy at one time), Windows Oracle DB installations shot to hell and gone by a UPS failure, Access databases both locally and over the network (no wonder here with M$ code). So there is no call to say that file corruption does not exist like stated in the OP.

    Whew, that was a long post .... sorry.

  • (cs) in reply to James Schend
    James Schend:
    dkf:
    Confused:
    I hate, loathe and despise ASCII mode. The concept is flawed to begin with.
    Maybe you've never been "fortunate" enough to use FTP with systems that used the EBCDIC charsets? In those cases, you really really REALLY want to have ASCII mode set by default so that your directory listings work, and back when FTP as a protocol came into service such machines were still fairly common. (They were still hanging on in the '90s, for goodness sake!)

    If this was a WTF from any time recently, the Real WTF! is that they're using FTP at all. It's a horrible protocol in rather many ways, and WebDAV (or even just plain HTTP) is much better in a number of important technical ways.

    FTP is, indeed, a terrible protocol. But suns can flare to life, form planets, evolve life, and die during the length of time it takes WebDAV to upload a 10,000 file website. Or at least Apple's implementation of it, which appears to reconnect for every single file...

    Muahahahahahahahahaaaaaaaaaaaaaaaaaa .... ROFL

Leave a comment on “The Loony BIN”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article