Delete duplicate lines in a text file

Question

I have a text file that I am trying to remove duplicate lines.

Example text file:

new featuredProduct('', '21640'), 
new featuredProduct('', '24664'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 
new featuredProduct('', '22142'), 

PHP code I tried:

$lines = file('textfile.txt');
$lines = array_unique($lines);
file_put_contents('textfile.txt', implode($lines));

The PHP file is called duplicates.php, and the text file is in the same directory. I would like to leave only:

new featuredProduct('', '21640'), 
new featuredProduct('', '24664'), 
new featuredProduct('', '22142'),  

The file function tries to read the file in the $ lines array and then array_unique () to remove duplicate entries. Then postpone the filtered results in the same file.

Answer 1

The problem is the new line characters at the end of each line. Since you do not have a new line symbol at the end of the last line, it will not be the same as the others.

So just delete them when you read the file, and then add when you save the file again:

$lines = file('test.txt', FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES);
$lines = array_unique($lines);
file_put_contents('test.txt', implode(PHP_EOL, $lines));

yo do: var_dump($lines); file(), :

array(7) {
  [0]=>
  string(36) "new featuredProduct('', '21640'), 
"
  [1]=>
  string(36) "new featuredProduct('', '24664'), 
"
  [2]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [3]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [4]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [5]=>
  string(36) "new featuredProduct('', '22142'), 
"
  [6]=>
  string(34) "new featuredProduct('', '22142'), "
       //^^ See here                            ^ And here
}

Answer 2

, PHP, , Linux/Unix Windows, bash, , . , PHP :

awk '!a[$0]++' input.txt

Answer 3

try it

$string = file_put_contents('textfile.txt');
$splitstr = explode('),', $string );
$str = implode('),',array_unique($splitstr));
var_dump($str);