Splitting large CSV files

10 March 2026 · Terminal · 2 min read

So I once got my hands on a very large CSV file - when I say large, I mean almost 9GB large! Needless to say I had issues opening it. It really didn’t want to work with Excel, VS Code, even my old faithful Sublime Text.

The fix is to split it into manageable chunks from the terminal. The split command does the job:

split -b 100m file_to_split.csv part_
for i in part_*; do mv "$i" "$i.csv"; done

The -b 100m flag splits by size into 100MB chunks, and part_ is a prefix for the new files so the rename loop doesn’t touch the original. (My first version of this post used for i in * - which happily renamed the source file too. Lesson learned.)

Splitting on lines instead

Splitting by size will almost certainly cut a row in half at each boundary, which is no good if you plan to import the chunks into a database. Better to split by line count. Peek at the size-based chunks to gauge a sensible number, then:

split -l 415000 file_to_split.csv part_
for i in part_*; do mv "$i" "$i.csv"; done

-l tells split how many lines each file should hold, so every chunk ends on a complete row.

One header, many files

One gotcha: only the first chunk will contain the CSV header row. If whatever you’re importing with expects headers everywhere, prepend it to the rest:

head -1 file_to_split.csv > header.csv
for f in part_*.csv; do
  cat header.csv "$f" > "with_header_$f"
done

And if you just want to look at a giant CSV without splitting anything, less -S file.csv will happily scroll through gigabytes without breaking a sweat. Wish I’d known that at the time.

← All notes