Split Large File into Multiple Files

I was involved in a research project where I needed to provide a result set back to the requester. The file contained 7 million records which is big to send it over via an email. Even if I manage to send it to the requester via FTP or similar tools, the file is huge for anyone to open in a notepad or in a text editor. It is advisable to split a big file into many smaller files that are more manageable to view, edit or analyse.

If data is being pulled from a database, you can limit the number of records in your query while spooling to a file. If this is not an option then I would use Linux SPLIT command to split in a way that is more meaningful to a requester.  I am going to show split command on how to split a large file into smaller files. A file can be split in many ways, we will go over more than one way to slice a big file.

Split Syntax:

split [OPTION]... [FILE[PREFIX]]

Split based on file size [b]: This option (-b) splits a file into many smaller files of size included with option -b

split -b5000000 large_file.zip file_prefix_

This creates small files of size 500000 bytes.

Split based on the number of small files [n]: This option splits into the specified number of files mentioned with option -n50

split -n50 -e large_file.zip small_prefix

Creates 50 files out of large_file.zip.

Split based on the number of lines on each file [l]: This option splits file put the number of lines specified in -l to each file

spit -l500 large_file.zip small_prefix

Creates multiple files with 500 lines of records on each file.

Awesome, you can now split a large file into many small files. I prefer to split mine based on the number of lines per file (-l). There are instances where you may be sick of managing multiple files and wanted to have a big single file. How do you accomplish this? cat is a Linux command which is short for concatenation that allows you to create files, view the content of files, and also let you combine multiple file into a single file. We have xaa, xab, xcc, and xde files that we need to combine into a single file.

cat x* > large.zip
cat xaa xab xcc xde > large.zip

Split and cat are Linux commands to split a file or concats multiple files. These commands have very powerful and I only showed you the most frequently used options. I would encourage you to man these commands and scan all the options these commands offer.  Hey windows fan, how you do you split and cat file, we would like to hear from you as Linux is not an option for everyone.