I do a lot of file manipulation on my system and in one particular batch job I end up with around 16 Gb file. I need to prepare this data into smaller chunks for another process.

I split it into 10k lines per file and numeric index, padded to 5 digits

split -a 5 -d -l 10000 large_input_file /out_path/out.

This way I end up with files named

The problem is that this way indexing always starts with 0.
Is there a way to set it to arbitrary starting index? man reveals nothing ...

  • Why do you want to start counting the chunk filenames from a non-zero index? Knowing the use case would help to find a solution to your problem. – loevborg Jan 1 '11 at 16:52

It's not a big deal to renumber the files afterwards.

split -a 5 -d -l 10000 large_input_file /out_path/tmp.
for x in /outpath/tmp.*; do
  mv -i -- "$x" "${x%/*}/out.$(printf %05d $((10#${x##*.}+1)))";

Note a couple of pitfalls:

  • The naive approach of renaming out.00000 to out.00001 and so on doesn't work, because globbing expands to a lexicographically ordered list, so the first renaming would overwrite the second file. In zsh, you can use the (On) glob qualifier. In other shells, the easiest way is to change the stem of the name as well.
  • A leading 0 in shell arithmetic causes the number to be interpreted in octal. 10#${…} forces a decimal interpretation.
  • printf %05d is a way to maintain the padding to 5 digits.
  • nice, and by replacing +1 with +$start_id I get arbitrary first index ... – nEJC Jan 2 '11 at 15:49
  • btw. what is the meaning of -- in mv command? Can't seam to find it ... probably "no more switches after this point", right? – nEJC Jan 2 '11 at 15:52
  • @nEJC: Yes, that's exactly it. In this particular case, it's not necessary, because the arguments can't begin with -, but it's a good habit to get into (and it's a reflex with me). – Gilles 'SO- stop being evil' Jan 2 '11 at 16:15

I think this option doesn't exist in split. I've this bug (feauture request) for coreutils -- http://savannah.gnu.org/bugs/?22231

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.