To close out the year, enjoy one of our popular articles from 2014. Happy holidays!

In the bowels of a business unit, a director got a great deal on a third party software package. He bought it, without talking to corporate IT, and then was upset when it couldn’t gracefully integrate with any of the corporate IT assets. Eager to throw good money after bad, the director hired his nephew’s consultancy to build an integration tool to make his new toy work.

A few months later, the users complained about performance, and somehow, fixing this thing became Jeff’s problem. The process was simple enough: slurp enterprise data out of a text file, and pass the data on to the third-party tool. It didn’t take Jeff long to figure out why it performed poorly: 

Private Sub ProcessFile()

    ' prepare to do stuff

    Do Until blnLastTime = True
        
        Set fileReader = fso.OpenTextFile(strFileName)
        
         If fileReader.AtEndOfStream = True Then
            blnLastTime = True
         Else
            strTextLine = fileReader.ReadLine
         End If

        ' actually do stuff
         
        fileReader.Close
        Delete_Line (strFileName)
    Loop
    fileReader.Close

End Sub



Private Sub Delete_Line(strFile)

    Set fileReader = fso.OpenTextFile(strFile)
    
    If fso.FileExists(strFile & "2") Then
        fso.DeleteFile (strFile & "2")
    End If
    
    Set fileWriter = fso.CreateTextFile(strFile & 2)
    
    If fileReader.AtEndOfStream = False Then
        fileReader.ReadLine
    End If
    
    If fileReader.AtEndOfStream = False Then
            strLine = fileReader.ReadAll
            fileWriter.Write (strLine)
    End If
    
    
    fileReader.Close
    fileWriter.Close

    fso.DeleteFile strFile, True
    fso.CopyFile strFile & "2", strFile, True
    fso.DeleteFile strFile & 2, True
    
End Sub

Start by opening a file “foo.txt”. Read a single line from the file. Send it to the third party app. Close the file. Open “foo.txt” again. Open another file, called “foo.txt2”. Read the first line from “foo.txt”, again. Throw that away. Read the remainder of “foo.txt”, and write it to “foo.txt2”. Copy “foo.txt2” back over “foo.txt”. Now, go back to the top of the loop and read a single line from “foo.txt” again.

So, for a 10,000 line file, this would perform 30,000 file open operations, write nearly 50 million lines, delete 20,000 files, and perform 10,000 copy operations. It didn’t take Jeff very long to rewrite this to simply read the file, one line at a time. The runtime dropped from a few hours to less than a minute.

[Advertisement] BuildMaster allows you to create a self-service release management platform that allows different teams to manage their applications. Explore how!