Filter the data loaded by the MySQL LOAD DATA command

Question

I am using the following command.

LOAD DATA  INFILE 'source.txt'
INTO TABLE tblData
CHARACTER SET utf8
FIELDS TERMINATED BY '\t';

This works fine, but the source.txt I get is a 800 MB file containing only 10% of the lines I need. I cannot filter in a text file before downloading. I can filter my results based on the value of one particular column. Is there a way I can specify this condition in my load statement, so that only the required lines are loaded.

Answer 1

No, at least not with LOAD DATA INFILE.

However, you can create a script that parses your file and inserts only those records that match your criteria

Answer 2

script , LOAD DATA INFILE , .

CREATE TEMPORARY FILE temp LIKE tblData;
LOAD DATA  INFILE 'source.txt'
    INTO TABLE temp
    CHARACTER SET utf8
    FIELDS TERMINATED BY '\t';
INSERT INTO tblData (field1, field2)
    (SELECT field1, field2 FROM temp WHERE some condition);

, , .