• Iain Collins (unregistered) in reply to gero
    gero:
    Q. Locking? A. You can't control the watcher

    It doesn't necessarily matter if you can't modify the Watcher (which, incidentally was a condition introduced only halfway through to try to point the candidate in another direction, not a limitation from the outset).

    Not being able to modify the Watcher to use flock()/lockf() (which any app like that should be doing in the first place if it's been written professionally) just means you have to get a mandatory lock, rather than an advisory lock.

  • A Problem At The File System Level (unregistered) in reply to Rhialto
    Rhialto:
    SoonerMatt:
    Marc:
    Rename?
    Yeah I was thinking that too. Rather than make it move a 3 gb file (which could fail in itself), I would start the transaction as a .tmp file then remove the .tmp when it's completed.
    Why does everybody seem to think that moving a file involves copying it? Not even MSWindows does it that stupidly, I think. (Or does it?) Anyway this was apparently about Linux.
    What if the file is 3 gigs in size? The .tmp idea will fail the same way as the temp-directory idea, i.e. the "mv xxx.tmp xxx" command will obviously take a while and the Watcher might already see the file when the rename operation hasn't finished writing the file's data to disk.

    Just kidding, of course. :-)

  • Iain Collins (unregistered) in reply to Dracolith
    Dracolith:
    I mount my filesystems using NFS on UNIX which does not provide proper file locking of any sort. It is common practice to use such filesystems.

    "Proper" file locking is absolutely supported on NFS under UNIX, even Windows supports locking over NFS.

    Most systems just require an appropriate RPC daemon and portmap to be running (I think it's a Service in Windows). This is not that unusual, I've certainly used commercial/3rd party software under UNIX which requires file locking on NFS be enabled if you want to run it from an NFS share.

    It's not "a liberal and unwarranted" assumption, as it's trivial to check if you got the lock (and something you would need to do anyway, even if you were writing to another location and then moving the file) - certainly it's less work than the conditional checking you'd have to do to handle using a temporary file location.

  • Carra (unregistered)

    Heck, my first idea was "just send a signal when a file is done" too. Watcher just listens on port x, downloaders sends a signal to port y. Or a triple handshake.

    Although I did get the "use a temp dir" when he said the watcher could not be changed.

    Can't blame the recruit for assuming it's one of those M$ trick questions where for every solution you give, they dismiss it.

  • (cs)

    What a lot of old crap about locks, atomic moves, filesystem semantics, *nix etc. It's exhausting!

    The simple solution has been suggested a couple of times above, and ignored, i.e. the semaphore file which is sent after the big file. The Watcher only registers the arrival of the semaphore file, and only processes the big file.

    And don't tell me "the Watcher can't be modified." - this is the real world, and it's a piece of code; it can be modified.

  • pincho (unregistered) in reply to asdsdf
    1. File copies/moves across a filesystem are (probably) much quicker than the downloading. The interviewer never said it was a problem with multiple processes accessing a file, just that if the Watcher reached the end of the file too quickly. If a copy/move can copy bytes faster than the Watcher can process the bytes, then it should be fine, no?

    That's what I call sound approach to creating a robust solution free of timing dependencies ;-)

  • alioth (unregistered) in reply to crystal mephistopheles
    I don't think it's fair to say network I/O is "a lot of work". Granted, his temporary directory solution is even simpler

    Basic network I/O is easy, but often you also need security. You're probably going to need (by policy) to have some authentication for any network traffic. While you can use SSL to provide this (which makes it easy from the programming level) you now have to manage certificates.

    It's therefore about two orders of magnitude easier to just use the filesystem which both programs already have access and are protected by the necessary permissions or ACLs, rather than having to add yet more authentication and authorization to make sure these things remain secure - especially if you're encumbered by an ISO27001 security process.

  • Iain Collins (unregistered) in reply to vr602
    vr602:
    What a lot of old crap about locks, atomic moves, filesystem semantics, *nix etc. It's exhausting!
    From the comments it seems like a lot of posters don't lock files (be they temporary or otherwise) even when writing (let alone reading) because it's "old crap" and "exhausting".

    "Locking files is so 1990's - this is Web 2.0 baby!

    We just try to append and hope for the best!"

  • A Problem At The File System Level (unregistered) in reply to gero
    gero:
    We're looooooping here...
    It would seem the reason why we're looping here is because...

    The real WTF is the forum software, of course! If you could view posts by thread, it might make it a lot easier to check whether something was already said before (by the way: was it already mentioned that a file move from the temp directory might not be atomic?), or whether a certain post has already been replied to, at least by letting you easily ignore all the posts on a completely unrelated topic.

  • Inigo Montoya (unregistered) in reply to Gorfblot

    A diamond in a rough is a gem that stands out among a collection of dull rocks.

    What you're talking about is a "rough diamond".

    Inconceivable!

  • smeg (unregistered) in reply to vr602

    This is the real world; inter-department and inter-company politics and bureaucracy will ensure no-one gets access to the piece of code in order to modify it - assuming whoever is responsible actually feels responsible enough to do something

  • (cs) in reply to Iain Collins
    Iain Collins:
    vr602:
    What a lot of old crap about locks, atomic moves, filesystem semantics, *nix etc. It's exhausting!
    From the comments it seems like a lot of posters don't lock files (be they temporary or otherwise) even when writing (let alone reading) because it's "old crap" and "exhausting".

    "Locking files is so 1990's - this is Web 2.0 baby!

    We just try to append and hope for the best!"

    Yes, well I knew someone would miss the point completely.

    Who says I don't lock files? I lock files if I need to; the point with my solution is it doesn't matter if the file is locked or not, it's simply not a factor. The "lock the file" solution is old crap not because locking files is wrong, or because I can't be bothered to do it. but because it's irrelevant. Oh and incidentally, I don't do any web development at all, I'm very much an old-fashioned dinosaur of a programmer with ancient standards and working practices that date back to 1986, so please don't accuse me of being new-style lazy.

  • Philluminati (unregistered)

    I have to admit I failed this like everyone else it seems.

    My first reaction is that the watcher should be invoked by the downloader when it has finished getting the file rather than constantly running.

    If you can't do that I have some other suggestions but certainly one of them wasn't "download them to a temporary directory and move them when they're finished".

    Why would you be able to "chain" an event to the end of the downloader, such as a move, but not chain the watcher launching then?

  • Stephen (unregistered) in reply to Max

    Erm.. Moving a file is always the right answer to this problem. In linux you can even move a file ontop of an existing file and it wont even break processes that have the original open. They can merrily keep using that file descriptor until they close and reopen the file.

    Seriously... anyone who fails to get this right is not worth hiring. They're usually the ones that develop horrifically overcomplex systems for simple problems.

  • Spiun (unregistered) in reply to SomeCoder
    SomeCoder:
    Yeah, solution #2 is just great. Because we all know that every file system command is guaranteed to be atomic, right?

    mv may be CLOSE to atomic but it's definitely not guaranteed to always be atomic. And if we suddenly have to change directories across partitions then it damn well is NOT atomic.

    I think Jeremy H should get a better interview question.

    Well, the best self-advice I can squeeze out of this situation is thinking out loud at an interview. Basically its possible that the temporary directory solution was a candidate solution, but the developer thought about the possibility that it may not be on the same volume so he silently discarded it (and the probability is not that small either).

    So... if you figure your reasoning is good, you can think-out-loud the discarded solutions too, possibly getting bonus points.

  • Dan (unregistered)

    “Oh,” he scoffed, “so, we’re equals now, is that it?”

    ROFL!! I highly doubt it!

  • Arlen (unregistered)

    Nice trackback, too.

  • equex (unregistered)

    I once went for an interview and was told to meet up directly at their clients offices, where he would appear and meet up with me to discuss the next project with them.

    This company makes a well known open source CMS system.

    The guy doesnt show up, and i have to act as a standin with their clients and i sit there and bullshit as good as i could.

    20 minutes later he shows up for the still ongoing meeting.

    Afterwards, i team up with him and we go to their offices where the actual interview would take place.

    The interview consisted of 30 minutes of slides about their biggest customers (NASA, etc etc. NASA cant make their own CMS ? lol) and thats when i started to smell rats.

    After the presentation they go on about how they dont want to hire anyone, they only set up franchises. So they wanted me to start my own company and enter into their franchise so they wouldnt have to hire me.

    I went for a cigarette and left without saying anything.

  • Steve (unregistered)

    Regarding story #2, I've had to deal with kluges like that in the past.

    A larger number of years ago than I can comfortably think about, I worked in an installation where we had a Cray X-MP running something called CTSS (Cray Time Sharing System) and a bunch of VAXes, which acted more or less as front ends to the Cray. We had a film recorder system running on one of the VAXes and the Cray was connected to that sysem by means of something called Hyperchannel, back when that was Hot Stuff.

    The Cray would deposit image files to be recorded on the film recorder for movies.

    As with the problem posed, there was no way of telling when the file was complete (or if it actually was complete, since the Hyperchannel connection would occasionally go wonky).

    It's difficult to describe how ugly and complicated this became.

    It's not a trivial problem. Believe me. I have the scars to prove it.

  • James R. Twine (unregistered) in reply to tgape

    In Windows, moving files across drives (letters/partitions) is done via a copy and then a delete behind the scenes (look it up and/or try it with a large enough file - the article does talk about multi-GB files). As such, the file can appear in the directory listing of the destination before it is fully "there".

    Posix or not, assumption is a big thing, too.

  • James R. Twine (unregistered) in reply to tgape
    tgape:
    James R. Twine:
    Ok - so the majority here believe that mv is an atomic operation if the src and dst are on the same filesystem (or partition?)... Does this hold true for ALL filesystems running under Linux/Unix? It might be that the directory is not running on ext2/ext3... What if it was a mounted FAT32 partition, or RiserFS, or UFS, or a SMB share?
    It holds for every single file on any POSIX OS, regardless of whether that file is on a native filesystem or not. This is because rename(2) is atomic, according to POSIX, so if you want to be POSIX, rename better be atomic. And it's incredibly easy to have rename(2) be sufficiently atomic for this purpose. (Note: even if rename was implemented by doing a link(2) followed by an unlink(2) of the old name, it would *still* be fast enough, unless Watcher is so fragile that it dies if it ever sees a file with multiple links. For it to do that, it'd pretty much have to be coded to do that, however - and it would still be such a tiny window that it's virtually inconceivable that it would happen regularly.)

    Btw, this means Windows, too. POSIX is a big thing.

    (Note: there are many unix OSes which have optional POSIX support. However, their non-POSIX mode is still loosely POSIX, and still does this. It's too necessary to not do it, and it's too useful to write the code to do it and then not always use it.)

    In Windows, moving (not renaming) files across drives (letters/partitions) is done via a copy and then a delete behind the scenes (look it up and/or try it with a large enough file - the article does talk about multi-GB files). As such, the file can (and does) appear in the directory listing of the destination before it is fully "there". Oh, and be sure to try that on different filesystem types, too.

    POSIX or not, assumption is also a big thing.

    Remember, my original point was that these kind of "only one right answer"-type questions often take a lot of assmptions for granted.

  • (cs) in reply to Joel H.
    Joel H.:
    Jeremy H. is a tool. He said that you can't modify the Watcher but you can modify the Downloader?

    I've had tools like this ask me questions where they really want one golden answer. He is not looking at how you approach the problem, rather whether you reach his solution.

    The temporary directory solution is a poor one - it creates a race condition. It assumes that a copy operation from temp to the real dir is going to be an atomic and allows the watcher to pick up half written files. Sure it will crash less - but it will still crash.

    Communication between the downloader and watcher makes sense to me. Sounds like the guy was just getting annoyed that Jeremy is a tool.

    Actually he's very much looking at how the person approached the problem. He presented a scenario and refined it to see how the interviewee would respond to a changing scenario.

    Instead of using the new information (no EOF, can't modify the watcher) to step back and rethink how he would approach the problem he continued to simply modify his solution despite the fact that it was getting increasingly convoluted, complicated and time consuming. The interviewer wanted to see if the employee had an open mind to changing his proposed solution. Instead the employee started with the assumption that the file cannot be moved and decided to just figure out harder ways to do that rather then step back and ask if his initial assumption was correct.

  • (cs) in reply to gero
    gero:
    Joel H.:
    The temporary directory solution is a poor one - it creates a race condition. It assumes that a copy operation from temp to the real dir is going to be an atomic and allows the watcher to pick up half written files. Sure it will crash less - but it will still crash.
    Chris Leather :
    Hang on a minute - surely the watcher program will try to process the file while the downloader program was moving it from the temporary directory (which would take a while due to it's size), which would produce precisely the same result - the Watcher program crashes as it processes an incomplete file.

    We're looooooping here...

    Seriously guys, the real solution is to mount the remote location (there are plenty of FUSE modules for every imaginable protocol and we already know these guys are using Linux from the context), have the downloader just add a symbolic link to the remote file (that should be as atomic as it gets), and then the watcher will act on the file pointed to by the symbolic link.

    Sheesh.

    Yea, mv is way too complicated.

  • (cs) in reply to Casey
    Casey:
    Azeroth:
    There is another simple solution with the downloader/watcher problem - downloader should open the file exclusively while it's being downloaded, this way watcher won't be able to access it until it's closed. This way it's not even required to move anything anywhere.

    Nope, sorry.

    Watcher tries to open file, and gets can not open file error and abends. -904 resource unavailble

    The proper way to do this is to have the downloader trigger the watcher program when done. Has no one heard the term "batch" or "job schedule" before?

    It's a good thing you *nix kiddies haven't tried to reinvent the car, as I suspect it would not have wheels.

    So what about the scenario where the downloader runs on server xyz. It downloads a file from stuff.com and puts it in a shared folder \xyz\files. The watcher runs on server abc which is entirely outside your domain of control, you are not allowed access to this server in any way thus there is no possible way to trigger the watcher.

    Granted there are times when downloading it to a temp directory will not work but they are very very very few and far between. I would much rather use the very simple, very reliable solution and only make things more complex when it is required rather then using the more complicated and less usable solution as my default.

  • (cs) in reply to Cpt
    Cpt:
    And the temporary directory solution will pose the same problem: what if the copy function is not finished yet and the Watcher gets more cpu cycles? The only solution in this case is to modify the protocol (or rather introduce a very crude one) with a separate flag file which contains a value indicating that either the provider is downloading, the provider is ready, the watcher is reading or the watcher is ready. From these 4 states both parties can derive what the other is or has been doing (or not) and whether it is save to do its stuff. If there is a contention for this status file you could add a second locking file (just keep it locked, nothing more) and create a true double semaphore.

    Why is it all you dolts that are suggesting locking/semaphores/IPC and so forth don't seem to understand that none of those will work unless you can modify both the Watcher and the Downloader, and you were expressly told that wasn't possible?

    It isn't really that hard to understand. Let me try one more time:

    You cannot use a solution that requires modifying both sides of the process because you're NOT ALLOWED TO MODIFY BOTH SIDES OF THE PROCESS!

    Does that help you at all? If not, please find another career you (collectively speaking to all people posting solutions that fail to comprehend the above) are more qualified for, like flipping burgers, washing dishes, or digging ditches. Thanks so much for playing.

  • (cs) in reply to Zaippa
    Zaippa:
    Hmm... just tried copying a large file (on windows). It does show the file in the destination directory while copying (in explorer and cmd)... So it looks like it does insert the file entry first.. Never mind :)

    Which is why everyone keeps stressing that you move the file, not copy the file. Please pay attention.

  • (cs) in reply to Earl Colby Pottinger
    Earl Colby Pottinger:
    Real people. Personally, while reading the problem I right away came up with a solution - Have 'Watcher' look for the next file before processing the present file.

    Example, If you expect to see File1, File2, File3 .... FileX then process File1 when File2 appears, process File2 when File3 appears ... etc ... FileX I would process after a reasonable delay or add a dummy End_Of_Files file.

    Really, brainiac? You found a solution that requires modification to the Watcher after you were expressly told that wasn't allowed? Wow! You're an immediate hire! For the new job listing for a janitor, of course; you're obviously not qualified for anything more technical.

    See Security down the hall for your pass. They'll know what areas you're not allowed to access.

  • (cs) in reply to Joel H.
    Joel H.:
    Jeremy H. is a tool. He said that you can't modify the Watcher but you can modify the Downloader?

    I've had tools like this ask me questions where they really want one golden answer. He is not looking at how you approach the problem, rather whether you reach his solution.

    The temporary directory solution is a poor one - it creates a race condition. It assumes that a copy operation from temp to the real dir is going to be an atomic and allows the watcher to pick up half written files. Sure it will crash less - but it will still crash.

    Communication between the downloader and watcher makes sense to me. Sounds like the guy was just getting annoyed that Jeremy is a tool.

    Joel H. is a tool. He doesn't seem to know enough about operating and file systems to understand the difference between a copy and a move operation. Therefore, he came up with yet another moronic description of a problem that doesn't exist to justify why he too would have failed to pass Jeremy's easy interview question.

  • Ray (unregistered) in reply to oppeto

    How about checking timestamps too? If the file is unchanged for a minute and the file size hasn't changed, it's probably done.

    If it's really that big of an issue, I'd probably use lsof or fuser too.

  • (cs) in reply to Mike
    Mike:
    Sorry but that temporary files copying question is a crock of shit. You still have the same problem. If the file is large, and the disks are slow/physically separate from each other, the operating system will copy it piece by piece from the source directory to the target; it's not an atomic operation, so you still can't know when it's fully complete.

    Sorry, but you're wrong. Thanks for playing. Don't bother trying again.

    (For an explanation of why you're wrong, read the other 100+ posts in this thread that tell you.)

  • (cs) in reply to Gorfblot
    Gorfblot:
    Someone You Know:
    I don't think "IE" means what you think it means.

    It's certainly possible. I think it means id est, and loosely translates into "That is to say".

    I used it between two attempts at explaining the metaphor- A literal one, and one where I attempted to show a situation where the description might be more valid.

    What do you think it means?

    You don't use id est ("that is") to introduce an example, you use it to restate the previous sentence more clearly and succinctly. You use exempli gratia ("By grace of example") to introduce examples, always, no exceptions.

  • Mr^B (unregistered)

    I think we've discovered TRWTF:

    "Developers who don't actually listen to the customer and produce what they think the customer wants, rather than what they asked for."

    Hoorah! Tea and Scones all round!

  • Tepsifüles (unregistered) in reply to Philluminati
    Philluminati:
    I have to admit I failed this like everyone else it seems.

    My first reaction is that the watcher should be invoked by the downloader when it has finished getting the file rather than constantly running.

    If you can't do that I have some other suggestions but certainly one of them wasn't "download them to a temporary directory and move them when they're finished".

    Why would you be able to "chain" an event to the end of the downloader, such as a move, but not chain the watcher launching then?

    Because the downloader is invoked by your own cronjob, while the watcher has to be consuming the files in its working directory constantly. Scenario: the watcher converts medical data files from one format to another, users bring in tapes written by devices in the next room without a network connection and just throw them into the directory; every night, a load of such records from another institution arrives via the downloader. So you can control when does the downloader run, what is run immediately before/after, but can't/don't really want to mess with the watcher, which needs to get all the files in its working directory, downloaded or not.

    And to break the loop: chmod FTW. With a slight exaggeration, that's what permissions are made for.

  • (cs) in reply to Tepsifüles
    Tepsifüles:
    And to break the loop: chmod FTW. With a slight exaggeration, that's what permissions are made for.

    Assuming the watcher doesn't crash when it can't read a file due to permission issues.

  • (cs) in reply to Marc
    Marc:
    You could build custom FPGA that intercepts packets on the the network and copies them to flash memory. The hardware can use a serial interface to indicate when the download is complete to a third program, 'The Mounter', which mounts the flash disk to the location the Watcher is expecting.

    The hardware can have a pool of flash memory disk areas, one being written to from the network, one mounted. Each flash memory area would only hold one file at a time.

    Since the Watcher is always running, I'm assuming it uses some sort of event handling system. An operating system hook to the event which indicated the Watcher is done processing and is now watching could be used to tell 'The Mounter' when its time to unmount a flash disk and mount the next one in the queue.

    Gloves.
  • grammernarzee (unregistered) in reply to KenW
    KenW:
    Why is it all you dolts that are suggesting locking/semaphores/IPC and so forth don't seem to understand that none of those will work unless you can modify both the Watcher and the Downloader, and you were expressly told that wasn't possible?

    It isn't really that hard to understand. Let me try one more time:

    You cannot use a solution that requires modifying both sides of the process because you're NOT ALLOWED TO MODIFY BOTH SIDES OF THE PROCESS!

    Does that help you at all? If not, please find another career you (collectively speaking to all people posting solutions that fail to comprehend the above) are more qualified for, like flipping burgers, washing dishes, or digging ditches. Thanks so much for playing.

    1 Stop shouting. 2 Get over yourself. Just because you think your solution is better than everyone else's, that doesn't mean they should be flipping burgers. 3 It's perfectly reasonable to ask why e.g. the Watcher can't be modified. Why not? No really, why not? Because someone says so? Well big flipping burger deal, maybe he's wrong and should be washing dishes. And if it can't be modified then let's write our own - even you could do that, right? Why jump through hoops to comply with a crummy piece of software that doesn't do what you want, when it easily could do? Maybe you'll shout back that I'm not "solving the problem as stated". Well so what, I'm getting things done, not writing overcomplicated solutions for foolish problems, in an attempt to justify my arrogance.

  • Good Greif (unregistered) in reply to KenW
    Cpt:
    Why is it all you dolts that are suggesting locking/semaphores/IPC and so forth don't seem to understand that none of those will work unless you can modify both the Watcher and the Downloader, and you were expressly told that wasn't possible?

    What in the name of all that's holy do you think "mandatory" file locks are, exactly?

    "Thanks so much for playing." ... indeed.

  • Zaippa (unregistered) in reply to KenW
    KenW:
    Zaippa:
    Hmm... just tried copying a large file (on windows). It does show the file in the destination directory while copying (in explorer and cmd)... So it looks like it does insert the file entry first.. Never mind :)

    Which is why everyone keeps stressing that you move the file, not copy the file. Please pay attention.

    Thank you, KenW. Perhaps you should have been paying attention to what i wrote as well. I am well aware that a move would solve the problem, but if you read the post above the post you quoted, you would realize that i was talking about that copying could easily be made atomic as well. (but i guess that would introduce other problems, and is why it isn't done like that).

    Normally i would ignore posts like yours, but you seem like a bitch and i'm in a bitchy mood as well, so there you go :)

  • (cs)

    The correct solution is pretty obvious. You have the watcher and downloader walk into a talent agency. The watcher says to the talent agent, "We have a really amazing act. You should represent us."

    The agent says, "Sorry, I don't represent family acts. They're a little too cute."

    The downloader says, "Sir, if you just see our act, we know you would want to represent us."

    The agent says, "OK. OK. I'll take a look."

    "First I come out, wearing a tuxedo, playing Brahms. Just as the music reaches a crescendo, the downloader in an evening gown runs on stage and undresses me before dancing provocatively on top of the piano.

    Just as I finish playing the song with my cock, the downloader strips and does a backflip off the piano in a split on stage. Once her naked ass hits the floor, my 7 year old daughter and 13 year old son rush on stage juggling flaming lawn darts. the downloader does a handstand and catches the lawn darts in her cunt, she then manages to queef them out, making her the third part of this juggling act.

    The queefs force her to squeeze out a few turds, which I eagerly start smearing on my naked body, which arouses me quickly. Once I'm fully aroused my daughter and son take turns blowing me while the downloader straps on a monstrous dildo and begins reaming each child while i ejaculate in the eyes of my offspring.

    Once I cum, I run into the audience, shit-covered body still sticky with cum and grab my parents and in-laws to involve them into the act. I strip them all nude and instruct them to start a circle jerk while screaming racial slurs. So my father and my father-in-law start screaming, "Fuck the niggers" while mutually masturbating, and my mother and mother-in-law begin diddling one another and chanting, "I hate spics and jews!" Once they reach a geriatric climax, the downloader uses their ejaculate to lube up her fist which she uses to start fisting me.

    As my asshole is violated, I start playing double dutch with my kids, and once they get tangled in the ropes, start a torrid 69. All the sucking and slurping cause my in-laws and parents to get aroused again and they start sodomizing and fisting one another.

    The downloader at this point has completely started dry-heaving, so she vomits all over my ass and my back. I line up each of my family members who take turns licking the chunks of spew off my back and out of my ass.

    By now my children have to defecate so I tell them to shit in each other's favorite orifices. My son, ever the trooper takes a thick, dense shit in his sister's vagina while my daughter shits in my son's nose.

    My young daughter also conveniently starts her menstrual cycle shortly thereafter, and the menses and boy-shit in her cunt make for great lube, as each of my in-laws begin fucking my daughter. My son, blinded in shit, heads back to the piano and does his best Stevie Wonder impression while the downloader runs back into the audience to grab a toddler from the crowd.

    She begins stuffing this child into her vagina, while my parents begin screaming how she's possessed by Satan and start performing a nude exorcism on her. The power of christ compels them to kill the toddler, which also makes it easier to cram into the downloader's lovehole.

    By now, I'm so horny and aroused that I start fucking the dead baby inside the downloader while my young son starts licking my asshole and fingering his paternal grandparents. My in-laws finish abusing my daughter and start wrestling each other, which culminates in a huge powerbomb through the piano bench. The impact shatters my mother-in-law's hips, leaving her crippled.

    The strain of the throw caused my father's bad heart to seize, and he collapses in a heap on the stage. As he gurgles and foams at the mouth, my daughter runs over and begins rubbing her shit covered pussy lips all over my crippled mother-in-law.

    The downloader grabs the wooden shards of the piano bench and begins playing her father's dying body like a xylophone. My son pulls his tongue out of my asshole and begins sucking his dying grandfather's cock.

    I diall 911 and call for the paramedics who revive my father-in-law and then take turns fucking my daughter and eating the menses and shit out of her tight cunt.

    Once he's conscious we all assemble in a large circle holding hands and chanting gibberish before launching into a rousing group impression of 'A Downs Syndrome' perspective on the horrors of the holocaust, 9/11 and the bombing of Pearl Harbor.

    As we're moaning and screaming, my son runs off-stage to get the family dog. The dog runs over to my crippled mother-in-law and begins peeing on her. Once the dog finishes leaving her in a puddle of piss, my daughter stops blowing the paramedics to light the dog on fire.

    The dog yelps and howls before collapsing. My son runs over to fuck the burnt corpse while screaming, "White is right!"as my daughter begins goose-stepping around the stage, squeezing shit out of her cunt and offering Nazi salutes to the audience.

    My father-in-law begins raping my father, claiming that he's doing it for the forgotten Vietnam vets and POWs. My downloader puts my crippled mother-in-law on her shoulders as I put the downloader on my shoulders and we play a game of naked chicken.

    Once my son finishes fucking the dead dog. He takes the pieces of the piano bench and begins crucifying the corpse. Once the dog is hung like jesus, he begins weeping at the foot of the cross, saying, "Why my god have you forsaken me?"

    My daughter mounts the top of the crucifix, using it as a wooden dildo. My parents, my in-laws and the downloader join hands at the center of the stage and start singing "The Hills Are Alive With The Sound Of Music"

    I grab the lawn darts and shove one up everyone's ass before heading back to the piano to finish off the show with a rendition of Freebird."

    For the longest time, the agent just sits in silence. Finally, he manages, "That's a hell of an act. What do you call it?"

    And the watcher says, "The Aristocrats!

  • Franz Kafka (unregistered) in reply to James R. Twine
    James R. Twine:
    tgape:
    James R. Twine:
    Ok - so the majority here believe that mv is an atomic operation if the src and dst are on the same filesystem (or partition?)... Does this hold true for ALL filesystems running under Linux/Unix? It might be that the directory is not running on ext2/ext3... What if it was a mounted FAT32 partition, or RiserFS, or UFS, or a SMB share?
    It holds for every single file on any POSIX OS, regardless of whether that file is on a native filesystem or not. This is because rename(2) is atomic, according to POSIX, so if you want to be POSIX, rename better be atomic. And it's incredibly easy to have rename(2) be sufficiently atomic for this purpose. (Note: even if rename was implemented by doing a link(2) followed by an unlink(2) of the old name, it would *still* be fast enough, unless Watcher is so fragile that it dies if it ever sees a file with multiple links. For it to do that, it'd pretty much have to be coded to do that, however - and it would still be such a tiny window that it's virtually inconceivable that it would happen regularly.)

    Btw, this means Windows, too. POSIX is a big thing.

    (Note: there are many unix OSes which have optional POSIX support. However, their non-POSIX mode is still loosely POSIX, and still does this. It's too necessary to not do it, and it's too useful to write the code to do it and then not always use it.)

    In Windows, moving (not renaming) files across drives (letters/partitions) is done via a copy and then a delete behind the scenes (look it up and/or try it with a large enough file - the article does talk about multi-GB files). As such, the file can (and does) appear in the directory listing of the destination before it is fully "there". Oh, and be sure to try that on different filesystem types, too.

    POSIX or not, assumption is also a big thing.

    Remember, my original point was that these kind of "only one right answer"-type questions often take a lot of assmptions for granted.

    We aren't talking about windows, and we aren't assuming that it goes across partitions - you don't have to deal with that case, so don't.

  • Hans (unregistered) in reply to blabla

    And what if the two directories don't actually reside on the same partition?

  • Hans (unregistered) in reply to Hans

    Ah, I see this point has already been flogged to death

  • Interviewer (unregistered) in reply to Not Worthy

    The program runs on Windows, so there is no user permissions, what do you do now?

  • Hans (unregistered)

    This solution depends entirely on whether the file copy is doing it one file at a time (ie rsync), but you could simply watch the directory and just work on any fire that isn't the newest one. Yeh, so you have to change the watcher script, and you need to add a little logic to deal with the last file (have the watcher monitor if the file has changed size in x seconds/minutes/whatever, or just have the downlaoder script drop an empty file at the very end of its copy).

    I just assume this would be simple but also avoid any potential issues like moving atomicity when your file system is partitioned in different ways.

  • Hans (unregistered) in reply to Interviewer
    Interviewer:
    The program runs on Windows, so there is no user permissions, what do you do now?

    Interviewee tries to answer. Interviewer quickly interjects: "... it's Windows CE specifically and you are about to experience a power outage in a few seconds and the server is on fire. What do you do now? Huh? Hypothetically..."

    Halp!

    :p

  • (cs)

    (And neither is "cp", I tried both in this scenario...)

    OK - so on one of my Linux boxes I created a ~858MB file filled with zeros by doing a cat /dev/zero > test.bin and then doing a ^C after a few seconds.

    That file is located on one LVM filesystem using ext3. The other filesystem is ext2 on a software RAID mirror. Using two terminal sessions I started the copy from the src directory on one session, and kept doing ls -laF in the other session which was in the dest directory.

    The filename appears immediately and its size grows while the mv (or cp) is in progress.

    Result: mv Is Not Atomic.

    Details: uname -a output: Linux LCARS 2.6.25.6-55.fc9.x86_64 #1 SMP Tue Jun 10 16:05:21 EDT 2008 x86_64 x86_64 x86_64 GNU/Linux

    directory listings while mv in progress:

    [root@LCARS raidmirror]# ls -laF
total 70064
drwxr-xr-x 8 root     root         4096 2008-09-25 13:05 ./
drwxr-xr-x 4 root     root         4096 2008-07-21 19:50 ../
drwxrwxrwx 2 root     users        4096 2008-06-20 11:58 BackedUp/
drwxr-xr-x 6 backuppc backuppc     4096 2008-06-23 05:14 BackupPC/
drwx------ 2 root     root        16384 2008-06-20 09:51 lost+found/
drwxrwxrwx 4 nobody   users        4096 2008-09-12 12:00 Mirrored/
drwxrwxrwx 3 root     root         4096 2008-06-22 08:56 Subversion/
drwxrwxrwx 3 root     root         4096 2008-06-22 08:49 temp/
-rw------- 1 root     root     71622656 2008-09-25 13:05 test.bin
    

    [root@LCARS raidmirror]# ls -laF
total 85912
drwxr-xr-x 8 root     root         4096 2008-09-25 13:05 ./
drwxr-xr-x 4 root     root         4096 2008-07-21 19:50 ../
drwxrwxrwx 2 root     users        4096 2008-06-20 11:58 BackedUp/
drwxr-xr-x 6 backuppc backuppc     4096 2008-06-23 05:14 BackupPC/
drwx------ 2 root     root        16384 2008-06-20 09:51 lost+found/
drwxrwxrwx 4 nobody   users        4096 2008-09-12 12:00 Mirrored/
drwxrwxrwx 3 root     root         4096 2008-06-22 08:56 Subversion/
drwxrwxrwx 3 root     root         4096 2008-06-22 08:49 temp/
-rw------- 1 root     root     87838720 2008-09-25 13:05 test.bin
    

    [root@LCARS raidmirror]# ls -laF
total 98764
drwxr-xr-x 8 root     root          4096 2008-09-25 13:05 ./
drwxr-xr-x 4 root     root          4096 2008-07-21 19:50 ../
drwxrwxrwx 2 root     users         4096 2008-06-20 11:58 BackedUp/
drwxr-xr-x 6 backuppc backuppc      4096 2008-06-23 05:14 BackupPC/
drwx------ 2 root     root         16384 2008-06-20 09:51 lost+found/
drwxrwxrwx 4 nobody   users         4096 2008-09-12 12:00 Mirrored/
drwxrwxrwx 3 root     root          4096 2008-06-22 08:56 Subversion/
drwxrwxrwx 3 root     root          4096 2008-06-22 08:49 temp/
-rw------- 1 root     root     100982784 2008-09-25 13:05 test.bin
    If you doubt the results, how about trying a test yourself before posting? Oh, and Windows does something similar.

    Thanks!

  • JPK (unregistered) in reply to SoonerMatt

    Nice reference to Hypothetical Questions

  • (cs) in reply to Franz Kafka
    Franz Kafka:
    We aren't talking about windows, and we aren't assuming that it goes across partitions - you don't have to deal with that case, so don't.
    (OK - as I already proved, the mv can be non-atomic on Linux, too -- at least on mine, so Windows/non-Windows is not a factor.)

    Where in the OP was it indicated what assumptions are and are not valid? Or did you just assume that the mv was not across filesystems/partitions?

    AGAIN...

    "Only one right answer" questions need to include the conditions (assumptions) in the scenario. Otherwise, only the people with limited experience with complex setups will assume that the mv would be atomic because the file would not have to cross filesystems/partitions, or because they just do not know any better.

    More experienced people, that know (for example) that /tmp should be on its own filesystem so that you do not get DoSed out of you own box by something filling up /tmp with garbage, will consider the way the real world works, and may not come up with the "one right answer."

  • JPK (unregistered) in reply to SoonerMatt
    SoonerMatt:
    RBoy:
    AMerrickanGirl:
    He was stuck in traffic, with a cell phone that worked perfectly, since they got through on the first try, but he hadn't bothered to call his office to tell them to make nice to the 9 am interview that he was going to be very late for.

    Ah, but what if his cell phone didn't work?

    Seriously?!? They said the office called him and his cell phone worked perfectly.

    Let's try again...

    Nice reference to Hypothetical Questions

  • (cs) in reply to jtwine
    jtwine:
    Franz Kafka:
    We aren't talking about windows, and we aren't assuming that it goes across partitions - you don't have to deal with that case, so don't.
    (OK - as I already proved, the mv can be non-atomic on Linux, too -- at least on mine, so Windows/non-Windows is not a factor.)

    I already said on page 1 that malicious config was out. Both directories are on the same filesystem, so shut up about non-atomic moves. move the syscall is always atomic. period.

    Where in the OP was it indicated what assumptions are and are not valid? Or did you just assume that the mv was not across filesystems/partitions?

    No, I stated it in my solution.

    More experienced people, that know (for example) that /tmp should be on its own filesystem so that you do not get DoSed out of you own box by something filling up /tmp with garbage, will consider the way the real world works, and may not come up with the "one right answer."

    you're the one with a hardon for /tmp. don't use /tmp.

Leave a comment on “A Problem at the Personal Level & More”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article