Remove duplicate row using PowerShell

Question

I have a text file:

1 2 4 5 6 7 1 3 5 6 7 8 1 2 3 4 5 6 1 2 4 5 6 7 

Here the first and last line are close. I have many double line files. I need to remove all duplicates.

Answer 1

It all seems very complicated. It is as simple as:

 gc $filename | sort | get-unique > $output 

Using actual file names instead of variables:

 gc test.txt| sort | get-unique > unique.txt 

Answer 2

To get unique strings:

 PS > Get-Content test.txt | Select-Object -Unique 1 2 4 5 6 7 1 3 5 6 7 8 1 2 3 4 5 6 

To remove a duplicate

 PS > Get-Content test.txt | group -noelement | ` where {$_.count -eq 1} | select -expand name 1 3 5 6 7 8 1 2 3 4 5 6 

Answer 3

If the order is not important:

 Get-Content test.txt | Sort-Object -Unique | Set-Content test-1.txt 

If order is important:

 $set = @{} Get-Content test.txt | %{ if (!$set.Contains($_)) { $set.Add($_, $null) $_ } } | Set-Content test-2.txt 

Answer 4

Try something like this:

 $a = @{} # declare an arraylist type gc .\mytextfile.txt | % { if (!$a.Contains($_)) { $a.add($_)}} | out-null $a #now contains no duplicate lines 

To set the contents of $ a to mytextfile.txt:

 $a | out-file .\mytextfile.txt 

Answer 5

 $file = "C:\temp\filename.txt" (gc $file | Group-Object | %{$_.group | select -First 1}) | Set-Content $file 

The source file now contains only unique lines

Already published options did not help me for some reason