I need to split a binary file into two. I was wondering if head and/or tail could be used but then I wondered...is it safe to use redirection, piping etc with binary data? Do new lines get messed about with, or nulls ignored, or backspace or delete do something special? (bash, kubuntu 18.04 LTS)

  • 1
    Take a look at the split command. – egmont Dec 29 '18 at 22:11

Yes it's safe if you pipe it to another process or save it to a file. There is potential "weirdness" if you let binary stdout print to a terminal since it can contain escape sequences (at random) that can temporarily mess up the terminal display.

  • 6
    In which case you can type reset and press enter to fix it. – Baard Kopperud Dec 29 '18 at 16:42
  • 4
    @BaardKopperud I thought I read somewhere about some corner cases where tset/reset wouldn't work – Xen2050 Dec 30 '18 at 0:04
  • 1
    @Xen2050 I don't know. the only case that would happen if some escape sequence changes the keyboard layout/encoding, so that typing reset<enter> does not actually type that sequence of characters as seen by the terminal... – Bakuriu Dec 30 '18 at 10:02
  • 3
    See also Fix terminal after displaying a binary file and Why does the console need sometimes a reset after CTRL+C. As suggested in the first link, stty sane; tput rs1 sequence of commands will do the trick for when there are corner cases of reset not working. Such cases, in addition to mentioned by Bakuriu, could include width of the terminal line/columns or I'm guessing the settings related to serial communication ( baudrate/parity). – Sergiy Kolodyazhnyy Dec 30 '18 at 10:34

The main problem with using commands like head or tail is that they are line-oriented and binary files are not. If they do have newlines in them, they are often not being used to represent the end of a line and if they are, they may be just be part of strings like program messages or data fields.

If the data is structured in any way, then you have to take that into account in choosing split points so you don't break structures in the middle.

If you know the structure of the file, you can use a command such as

dd -if input-file -of output-file ...

with options to only copy so many blocks of data of a specific size starting at a particular (incremented) offset into the file.

It looks like the split command as mentioned by @egmont will automate this process for you, but it appears to be line-oriented by default, so you'll have to specify additional options such as --bytes count to tell it how large each piece of the file should be.

As a side note, if you don't know what's in a file, but suspect it contains at least some meaningful textual data, the strings command is a great way of taking a first look to see what you're dealing with.

strings -n 6 file | less

will find all runs of printable characters at least six characters in length and display them in a pager so they don't fly by on the terminal. Using a number a bit larger than the default of 4 characters helps eliminate tiny snippets of data that just happen to be printable, but are not being used that way in the file.

If you later have to explore the file in more detail with binary editor such as hexedit, you'll have some landmarks that point out where something interesting might be found.

strings has an option -t x that will precede each printed string with its offset into the file in hexadecimal (o for octal/d for decimal) so you know where to find it later. Even very short files are a lot to deal with when you have to look at them character by character.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.