• Alan (unregistered)

    “Oh,” he scoffed, “so, we’re equals now, is that it?”

    No. I'm not an asshole. Good day.

  • (cs)
    “Oh, okay,” the candidate replied. He pondered for a full minute and said “so in that case, I would hae the Watcher listen on a TCP/IP port, and have the Downloader tell it when it was done downloading.”

    “That seems like a lot of work,” I said....

    I don't think it's fair to say network I/O is "a lot of work". Granted, his temporary directory solution is even simpler, but most high level languages (.NET, Java, etc.) have fully defined classes you can use to implement this in only a few lines of code, and I would classify this as an acceptable solution.

  • (cs)

    First!

    And if it's not first, I'll just patch CS so it identifies all third comments as first.

  • Max (unregistered)

    Oh, seriously. Each one of these is a job you shouldn't take anyway.

    1. Obvious reasons...
    2. An interviewer who thinks downloads in progress are a problem but file copies in progress are not shows a lack of understanding.
    3. Interviews are two-way -- if the people interviewing you are clueless, the job will suck.
  • (cs) in reply to Max

    Disagree on (3) here. The people interviewing him weren't clueless. This kind of "fizz-buzz" problem is pretty common and it's amazing how many "developers" can't pull off the simplest things.

  • Azeroth (unregistered)

    There is another simple solution with the downloader/watcher problem - downloader should open the file exclusively while it's being downloaded, this way watcher won't be able to access it until it's closed. This way it's not even required to move anything anywhere.

  • andrew (unregistered) in reply to crystal mephistopheles

    This is actually a bit more graceful even. There is no polling involved of the directory, you know exactly when to process the file. This could be 100x better if the mount is NFS'ed. Also, what if you need to change the "copy" framework to something else. I think mv /tmp/foo /end/foo is fine to start but its not robust for large(r) scale or more complicated designs.

  • The Undroid (unregistered)

    Better Dave should reveal his tendencies at the interview, even at the cost of an hour and a half wasted, than hire Shari and gradually work out whatever his problem is with Brown grads.

    Also, how do we know there are no troublesome latencies in the Dowloader's local copy?

  • blabla (unregistered) in reply to Max
    Max:
    2) An interviewer who thinks downloads in progress are a problem but file copies in progress are not shows a lack of understanding.

    Umm... I think the guy said 'move' rather than 'copy', which is at least closer to atomic, if not atomic?

  • ThomasP (unregistered) in reply to Max

    I think you missed the part where they were talking about file moves in Linux, not Windows.

  • Marc (unregistered)

    Rename?

  • (cs)

    Speaking of bad recuiters... I once dealt with a recruiter that kept trying to hook me up with positions I was not qualified for. I only had a couple of years experience as a Developer and on three different occasions scheduled me for interviews for Senior positions, and then would be totally baffled when I would tell him that I bombed the interviews.

    I wonder sometimes whether or not these recruiters get paid by the number of interviews they schedule and not by the number of positions they actually fill.

  • SoonerMatt (unregistered) in reply to Marc
    Marc:
    Rename?
    Yeah I was thinking that too. Rather than make it move a 3 gb file (which could fail in itself), I would start the transaction as a .tmp file then remove the .tmp when it's completed.
  • asdsdf (unregistered) in reply to Max
    1. File copies/moves across a filesystem are (probably) much quicker than the downloading. The interviewer never said it was a problem with multiple processes accessing a file, just that if the Watcher reached the end of the file too quickly. If a copy/move can copy bytes faster than the Watcher can process the bytes, then it should be fine, no?
  • (cs) in reply to Max
    Max:
    2) An interviewer who thinks downloads in progress are a problem but file copies in progress are not shows a lack of understanding.

    How about a lack of reading comprehension? Move, not copy - makes a big difference if the files are moving between folders / directories on the same volume.

  • smasher (unregistered) in reply to Max

    "moving" a file and "copying" a file are not the same. mv is hardly atomic, but if the file appears in the target directory, it's already been written to disk.

  • (cs) in reply to SoonerMatt
    SoonerMatt:
    Marc:
    Rename?
    Yeah I was thinking that too. Rather than make it move a 3 gb file (which could fail in itself), I would start the transaction as a .tmp file then remove the .tmp when it's completed.

    The watcher appears to process whatever files are in that particular folder - simply altering the extension might not be enough.

  • Alan (unregistered)

    If you are qualified for the job - of course the test will be easy. My company does basic testing for every applicant and every new recruit slags of the test - until they see the reject pile.

  • Gorfblot (unregistered)

    A diamond in the rough is something that looks to be of little value, but is actually worth quite a lot once some polishing has been done. IE, You talked with one of the Tier 1 helpdesk people and realized they had some talent and were quick to learn. You get them some further training, move them to a junior development position, and in a short time they become a major contributor to success- That's a diamond in the rough.

  • (cs) in reply to jpaull
    jpaull:
    Speaking of bad recuiters... I once dealt with a recruiter that kept trying to hook me up with positions I was not qualified for. I only had a couple of years experience as a Developer and on three different occasions scheduled me for interviews for Senior positions, and then would be totally baffled when I would tell him that I bombed the interviews.

    I wonder sometimes whether or not these recruiters get paid by the number of interviews they schedule and not by the number of positions they actually fill.

    I've found that sometimes recruiters will hold back some critical piece of info (say, requires relocation across the country) when they tell you how great the job is, so that you'll go to an interview you'd otherwise pass up. This way it looks like they're providing qualified candidates to the company.

    It's a waste of your time, and if I find out about it before leaving the interview, I'll let the interviewer know that the headhunter wasted his time and mine, and perhaps they should use a different agent in the future. I come off honest and the loser headhunter gets what he deserves.

  • (cs) in reply to Max
    Max:
    2) An interviewer who thinks downloads in progress are a problem but file copies in progress are not shows a lack of understanding.

    As noted, atomic renames (and I think moving files to another directory is atomic even under Windows, provided you stay within the same volume, right?) shouldn't have this problem.

    What if you can't modify the Downloader, either? Other than "replace one or both programs with something that can be modified", my first thought is to write a third program that also checks the remote site, and alters file permissions on the downloads-in-progress so the Watcher can't access them until the download is finished.

    Max:
    3) Interviews are two-way -- if the people interviewing you are clueless, the job will suck.

    They were clueful enough to reject unqualified candidates. And it's possible (though I wouldn't bet on it) that the recruiter isn't a WTF either, that the pool of prospects just really is that lousy in that area (on the theory that good prospects get hired quickly while bad ones keep trying).

  • imMute (unregistered) in reply to Max
    Max:
    \ 2) An interviewer who thinks downloads in progress are a problem but file copies in progress are not shows a lack of understanding.

    File copies would still be a problem, but file moves (also known as renames) are extremely quick as long as the src/dst are on the same volume: the FS only moves the inode data. Hell, even windows handles this as well as *nix.

  • (cs) in reply to Alan
    Alan:
    “Oh,” he scoffed, “so, we’re equals now, is that it?”

    No. I'm not an asshole. Good day.

    bows in awe

  • ST (unregistered)

    Thanks for the interview tales, this is one of my favourite sections. Mind you, I'm pretty shocked at how many of the resident professionals are trying to come up with alternative answers for the problem in the second tale. Obviously you use a temp filename (ignored by the watcher) or a temp directory. What kind of mindset comes up with anything else? Especially consdering that the candidate is being prompted by the interviewer that he is looking for a simple solution. I don't think they lost any worthwhile talent by turning down that hire.

  • Aaron (unregistered)

    If its in linux, the lsof command works quite well at telling you a file is still open, without having to modify the kernel or anything.

    lsof | grep filename || start whatever you wanted

  • David Emery (unregistered) in reply to Azeroth

    The "open exclusive" is the -right way*- to do this. If /tmp is located on a different file system/drive than the destination, then the file move operation is not guaranteed atomic. You can have the same problem on the disk copy than you have on the download (although the latency is less, it's certainly not zero.)

    dave

    (* I won't say "the only right way" but I might strongly imply it :-)

  • (cs) in reply to Gorfblot
    Gorfblot:
    A diamond in the rough is something that looks to be of little value, but is actually worth quite a lot once some polishing has been done. IE, You talked with one of the Tier 1 helpdesk people and realized they had some talent and were quick to learn. You get them some further training, move them to a junior development position, and in a short time they become a major contributor to success- That's a diamond in the rough.

    I don't think "IE" means what you think it means.

  • James (unregistered) in reply to David Emery
    David Emery:
    The "open exclusive" is the -right way*- to do this. If /tmp is located on a different file system/drive than the destination, then the file move operation is not guaranteed atomic. You can have the same problem on the disk copy than you have on the download (although the latency is less, it's certainly not zero.)

    dave

    (* I won't say "the only right way" but I might strongly imply it :-)

    a) since you're designing the fix, you can design it so the /tmp is on the same volume as the /final, so move is effectively guaranteed atomic

    b) I was going to suggest having Watcher poll the filesize and do its scan a fixed time after it stops changing. I think that can work, if you're able to be sure that Downloader is using a protocol with a spec'd timeout. Assuming, of course, that Watcher is interested in failed downloads as well as those that finish...

  • wds (unregistered) in reply to imMute
    imMute:
    File copies would still be a problem, but file moves (also known as renames) are extremely quick as long as the src/dst are on the same volume: the FS only moves the inode data. Hell, even windows handles this as well as *nix.
    All this assuming they're on the same partition right? So if they're not, you're in serious trouble. Considering how /tmp is the usual place to drop stuff in, and /tmp is often on another partition I don't see how this is an acceptable solution. Not to mention the problem with having allocated space in /var/program/blah but not in /tmp and thus running out of room to drop your multigig executables.

    See I'd just have used a lockfile.

  • (cs)

    I'd say that expecting move to be atomic is a rather poor assumption. You certainly open yourself up to problems if the volume structure changes. For instance, someone replaces the local directory with a symbolic link to a large scratch disk if the multi-GB downloads are filling the disk (or interfering with other activity on that drive).

    It may be more robust to download into another location & then create a symlink in the original directory.

  • Mike P (unregistered) in reply to ST
    ST:
    Obviously you use a temp filename (ignored by the watcher) or a temp directory. What kind of mindset comes up with anything else?

    Well, my first thought was "Start the Downloader, wait for it to complete, then start the Watcher".

    Presumably the interviewer would have told me I couldn't do this, in which case the temp directory solution would have been my second answer.

    On Linux (mentioned by the interviewer) "mv" is perfectly atomic provided both source and destination are within the same filesystem.

  • (cs) in reply to Someone You Know
    Someone You Know:
    Gorfblot:
    A diamond in the rough is something that looks to be of little value, but is actually worth quite a lot once some polishing has been done. IE, You talked with one of the Tier 1 helpdesk people and realized they had some talent and were quick to learn. You get them some further training, move them to a junior development position, and in a short time they become a major contributor to success- That's a diamond in the rough.

    I don't think "IE" means what you think it means.

    Internet Explorer? If that's the case, I make sure to pronounce it, 'AIEEEEEE!'

  • jwa (unregistered) in reply to wds

    And your whole response is predicated upon the assumption that the term temporary file meant strictly /tmp. Which is not always the case, more likely it was "a file which is temporary" not "a file in the system temporary directory".

  • Charles Duffy (unregistered) in reply to David Emery
    David Emery:
    The "open exclusive" is the -right way*- to do this. If /tmp is located on a different file system/drive than the destination, then the file move operation is not guaranteed atomic. You can have the same problem on the disk copy than you have on the download (although the latency is less, it's certainly not zero.)

    It's a move, not a copy. Moves within the same partition are atomic.

    Moreover, the tempfile-and-move approach (where, yes, you put your temporary file in the same directory or otherwise somewhere known to be on the same partition) is the generally accepted solution to this problem; if you don't know it, there's no way in hell I want you on my systems engineering team.

    Again, the problem the interviewer was looking for is the way every single UNIX developer with any kind of a clue does atomic file updates.

  • (cs) in reply to crystal mephistopheles
    crystal mephistopheles:
    “Oh, okay,” the candidate replied. He pondered for a full minute and said “so in that case, I would hae the Watcher listen on a TCP/IP port, and have the Downloader tell it when it was done downloading.”

    “That seems like a lot of work,” I said....

    I don't think it's fair to say network I/O is "a lot of work". Granted, his temporary directory solution is even simpler, but most high level languages (.NET, Java, etc.) have fully defined classes you can use to implement this in only a few lines of code, and I would classify this as an acceptable solution.

    True, but this requires you to have access to both the downloader and the watcher. In the "ideal" solution you only need access to the downloader.

  • (cs)

    Is it just me, or does Jeremy sound like a bit of a jerk? He simply wouldn't let the guy use any solution that wasn't his pet solution. Seems the simplest solution is to have the Downloader activate the Watcher when it's done.

  • SomeCoder (unregistered)

    Yeah, solution #2 is just great. Because we all know that every file system command is guaranteed to be atomic, right?

    mv may be CLOSE to atomic but it's definitely not guaranteed to always be atomic. And if we suddenly have to change directories across partitions then it damn well is NOT atomic.

    I think Jeremy H should get a better interview question.

  • Paul (unregistered)

    re: File watcher.

    Ignoring the fact that several people have raised that files appear before they've been fully copied or moved!

    The interviewer shot themselves in the foot, halfway through the (reasonable) list of solutions given by the candidate, they said it was not possible to modify the downloader. However, their simple solution given at the end required a modification of the application they said couldn't be downloaded!

  • oppeto (unregistered) in reply to ST
    ST:
    Thanks for the interview tales, this is one of my favourite sections. Mind you, I'm pretty shocked at how many of the resident professionals are trying to come up with alternative answers for the problem in the second tale. Obviously you use a temp filename (ignored by the watcher) or a temp directory. What kind of mindset comes up with anything else? Especially consdering that the candidate is being prompted by the interviewer that he is looking for a simple solution. I don't think they lost any worthwhile talent by turning down that hire.
    Start with the assumption that you can modify the watcher, but not the downloader (exactly like here at my workplace, actually)... check the file size, wait a minute, check again, only process if the size is the same. Not foolproof, but "good enough" and doable in a small shell script.
  • Not Worthy (unregistered) in reply to emurphy
    my first thought is to write a third program that also checks the remote site, and alters file permissions on the downloads-in-progress so the Watcher can't access them until the download is finished.

    THANK YOU! Seriously, the problem states that the Downloader downloads to a particular directory, and the official solution is "just change that directory". Even the official solution violates the terms of the problem.

    Seriously, run the watcher and downloader as different users. If the directory has the right permissions and the right umasks are in place, the watcher cannot even see the files which the downloader downloads until something (maybe a third process, but preferably the downloader) changes the permissions on the file, say to world-readable. That's the right way to do it.

  • (cs) in reply to SomeCoder
    SomeCoder:
    Yeah, solution #2 is just great. Because we all know that every file system command is guaranteed to be atomic, right?

    mv may be CLOSE to atomic but it's definitely not guaranteed to always be atomic. And if we suddenly have to change directories across partitions then it damn well is NOT atomic.

    I think Jeremy H should get a better interview question.

    Actually I believe that if you stick within the same volume it is guaranteed to be atomic. Given that you can modify the downloader it's fairly simple to have the tmp location on the same volume as the final location

  • Branan (unregistered) in reply to Max

    A file move is different from a file copy in Linux. A move involves changing one pointer in the filesystem, no information is actually "moved". So it's not actually a problem.

  • (cs)

    The first story about the interviewer going out to lunch reminded me of a somewhat similar experience.

    I was contacted by a recruiter who asserted that he had a couple of jobs that were perfect for me. But first, he insisted on meeting me in person.

    His office was 40 miles away, but I agreed to be there at 9 am.

    I arrived there at 9 am. He wasn't there yet. They put me in a small windowless room and asked me to wait.

    Twenty minutes later I wandered out to find someone and ask what was going on. They didn't know why Recruiter Guy wasn't in yet, but they sent out his cube mate, another recruiter, to talk to me. Problem was, she didn't have any of my information available to her so it was kind of a waste of time.

    Another twenty minutes went by and someone finally called Recruiter Guy. He was stuck in traffic, with a cell phone that worked perfectly, since they got through on the first try, but he hadn't bothered to call his office to tell them to make nice to the 9 am interview that he was going to be very late for.

    They had the nerve to ask me if I would wait another 40 minutes for him to arrive. I said no thanks and walked out. If he had made the effort to call his office and apologize to me for being late, I would have waited. But he could have cared less, so he wasn't getting any commission off my back.

  • Chris (unregistered)

    I think the problem with the downloader/watcher question is actually in the wording of the question, which serves to misdirect the candidate.

    It is phrased thusly: "Every night, a Downloader program will... save them to a certain directory on disk. A Watcher program monitors this directory"

    The clear implication is that the Watcher watches the download directory. This would probably give the candidate a fairer shot:

    "Every night, a Downloader program will... save them to disk. A Watcher program monitors a certain directory

  • persto (unregistered) in reply to Paul
    Paul:
    re: File watcher.

    Ignoring the fact that several people have raised that files appear before they've been fully copied or moved!

    For moves within a filesystem (renames), this does not happen. That's why it's called "rename", you're taking a file that already exists, and just pointing a new name at it. It is essentially a pointer assignment, "inode *file; ...; dir_ent.target = file".

    The interviewer shot themselves in the foot, halfway through the (reasonable) list of solutions given by the candidate, they said it was not possible to modify the downloader. However, their simple solution given at the end required a modification of the application they said couldn't be downloaded!
    Read again... the said not to modify the *watcher*.
  • mauhiz (unregistered)

    I don't know what I am missing there, why not just wait for the Downloader to finish its job?

    a script would look like this :

    Downloader && Watcher

    I don't think having a daemon poll a directory is a good practice anyways...

  • (cs) in reply to AMerrickanGirl
    AMerrickanGirl:
    The first story about the interviewer going out to lunch reminded me of a somewhat similar experience.

    I was contacted by a recruiter who asserted that he had a couple of jobs that were perfect for me. But first, he insisted on meeting me in person.

    His office was 40 miles away, but I agreed to be there at 9 am.

    I arrived there at 9 am. He wasn't there yet. They put me in a small windowless room and asked me to wait.

    Twenty minutes later I wandered out to find someone and ask what was going on. They didn't know why Recruiter Guy wasn't in yet, but they sent out his cube mate, another recruiter, to talk to me. Problem was, she didn't have any of my information available to her so it was kind of a waste of time.

    Another twenty minutes went by and someone finally called Recruiter Guy. He was stuck in traffic, with a cell phone that worked perfectly, since they got through on the first try, but he hadn't bothered to call his office to tell them to make nice to the 9 am interview that he was going to be very late for.

    They had the nerve to ask me if I would wait another 40 minutes for him to arrive. I said no thanks and walked out. If he had made the effort to call his office and apologize to me for being late, I would have waited. But he could have cared less, so he wasn't getting any commission off my back.

    I worked for a boss who was always late. He hated me from when I told him that being late showed an absolute disrespect for the value of other people's time.

    Funnily enough, he left soon afterwards...

  • Brandon (unregistered) in reply to Alan
    Alan:
    “Oh,” he scoffed, “so, we’re equals now, is that it?”

    No. I'm not an asshole. Good day.

    And then when he starts to come back with some witty remark you interrupt him with, "I said GOOD DAY!"

  • Marc (unregistered)

    You could build custom FPGA that intercepts packets on the the network and copies them to flash memory. The hardware can use a serial interface to indicate when the download is complete to a third program, 'The Mounter', which mounts the flash disk to the location the Watcher is expecting.

    The hardware can have a pool of flash memory disk areas, one being written to from the network, one mounted. Each flash memory area would only hold one file at a time.

    Since the Watcher is always running, I'm assuming it uses some sort of event handling system. An operating system hook to the event which indicated the Watcher is done processing and is now watching could be used to tell 'The Mounter' when its time to unmount a flash disk and mount the next one in the queue.

  • AnonymousInterviewer (unregistered)

    The problem with the bad recruiter is hardly a joke for me. I have faced this on numerous instances where we were looking for candidates. I felt they were resorting to corrupt HR practices, like nepotism. Ofcourse, I have never been able to gather any substantial evidence to file a formal complaint with HR.

    The only time I came close to nailing them down was when I got so frustrated with them that I posted a job on a local bulletin board, totally bypassing them. I got more responses from good local candidates in about 6 hours than I got from them in 2 weeks. Of course, there was no way I could proceed with this because my action (bypassing them) itself was in violation of HR policy.

    This proof was enough for me to quit the company. I don't think this practice has changed, though. BTW, I'm talking about a large corporation, here - a big name in software.

Leave a comment on “A Problem at the Personal Level & More”

Log In or post as a guest

Replying to comment #:

Log Out

« Return to Article