remove duplicate lines in text file

[yinkajewole]

how can i use a macro to remove duplicate lines in my text file such that the last one will remain, for instance
the boy is good
great thinking
great thinking
he lives
the boy is good
what's up
great thinking
it should now be like this
he lives
the boy is good
what's up
great thinking

 

[Domenic]

Since duplicate lines are going to be deleted, it shouldn't matter which instance should remain. Or am I missing something?

 

[MARK858]

Maybe....
Code assumes that your data is in column A and you have a header in A1.

Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub

 

[yinkajewole]

Maybe....
Code assumes that your data is in column A and you have a header in A1.

Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub

the data is not in excel, it's in a text file (.txt)

 

[MARK858]

You do realize this is an Excel site? Import the data into Excel run the code and save as a text file.

 

[yinkajewole]

You do realize this is an Excel site? Import the data into Excel run the code and save as a text file.

or is it possible to write lines in a text file while it will remove its duplicate?

 

[MARK858]

No, you can't remove duplicates while still in a text file, what is so difficult about importing it into Excel and then saving it?

 

[Eric W]

Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub

 

[yinkajewole]

Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub

this is too real to be true... Exactly what I was looking for. Many thanks you all

 

[Eric W]

Glad to help.