Selectively Importing Records

Ole wrote concerning an importing problem he was having. It seems that the files he needs to import are routinely larger than 65,000 records. Excel, of course, will only handle up to 65,536 rows of data. Ole doesn't need most of the input rows, however, and normally gets rid of them once the records are imported into Excel. Ole is looking for a way to get rid of the unwanted records during the import process, so that he doesn't need to break the input file up into smaller files.

There are a couple of different ways that a solution to this problem can be approached. One solution is to use Access as your first importing step. Access will easily handle more than 65,000 records. You could import the file into Access, filter out the unwanted records, and then export the resulting table as an Excel workbook.

The best solution, however, may be to bypass Excel's import filters entirely. You can easily write an import routine in VBA, and allow it to process the import file. For instance, consider the following macro:

Sub Import()
    Dim sFile As String
    Dim sUnwanted As String
    Dim sDelim As String
    Dim iRow As Integer
    Dim iCol As Integer
    Dim bBadRecord As Boolean
    Dim iTemp As Integer

    sFile = "d:\data.txt"
    sUnwanted = "bad text"
    sDelim = ","

    Open sFile For Input As #1

    iRow = 1
    While Not EOF(1) 'Scan file line by line
        iCol = 1
        Line Input #1, sBuffer

        ' Check to see if should ignore record
        bBadRecord = Instr(sBuffer, sUnwanted)

        If Not bBadRecord Then
            iTemp = Instr(sBuffer, sDelim)
            While iTemp > 0
                With Application.Cells(iRow, iCol)
                    .NumberFormat = "@" 'Text formatting
                    .Value = Left(sBuffer, iTemp-1)
                End With
                iCol = iCol + 1
                sBuffer = Mid(sBuffer, iTemp+1, Len(sBuffer))
                iTemp = Instr(sBuffer, sDelim)
            Wend
            If Len(sBuffer) > 0 Then
                With Application.Cells(iRow, iCol)
                    .NumberFormat = "@" 'Text formatting
                    .Value = sBuffer
                End With
            End If
            iRow = iRow + 1
        End If
    Wend
    Close #1
End Sub

This macro opens a data file and reads each record in the file. It checks the record to make sure it is OK to import, and then pulls the record apart, based on a delimiter, and stuffs the information into the current worksheet. You can change the name of the data file (the sFile variable), the text that indicates a bad record (sUnwanted variable) and the delimiter (sDelim variable).

As an example, let's assume that you have a data file named Customers.txt. This file contains all your customer records, but you don't want to import the records for customers with addresses inside the United States. Further, the records in the data file use a tab character between each field. In this case, you would only need to make the following changes to the variables at the beginning of the macro:

    sFile = "d:\Customers.txt"
    sUnwanted = "United States"
    sDelim = Chr(9)

Once you run the macro, the current worksheet contains just the desired data.