All geek questions in one place

How can I split large files?

I have a large CSV file (7.3 GB, 16.3 million lines), how can I split this file into two files?

+6
source share
2 answers

Have you looked at the split command? See man for more details.

This page contains an example of using this command.

Besides

The man -k command is useful for finding unix / linux commands if you are not completely sure what a particular command is. Specify the keyword using the man -k command, and the system will pull the related commands. For instance.

 % man -k split

will give:

 csplit (1) - split a file into sections determined by context lines dirsplit (1) - splits directory into multiple with equal size dpkg-split (1) - Debian package archive split/join tool gpgsplit (1) - Split an OpenPGP message into packets pnmsplit (1) - split a multi-image portable anymap into multiple single-image files ppmtoyuvsplit (1) - convert a portable pixmap into 3 subsampled raw YUV files split (1) - split a file into pieces splitdiff (1) - separate out incremental patches splitfont (1) - extract characters from an ISO-type font. URI::Split (3pm) - Parse and compose URI strings wcstok (3) - split wide-character string into tokens yuvsplittoppm (1) - convert a Y- and a U- and a V-file into a portable pixmap zipsplit (1) - split a zipfile into smaller zipfiles
+16
source

split -d -nl / N filename.csv tempfile.part.

splits a file into N files without line breaks. As mentioned above, the header is not repeated in every file.

+1
source

Source: https://habr.com/ru/post/915605/

More articles:


All Articles