How to push specific bits into a new file?

Question

Is there any way for me to push specific bits into a new file, without creating temporary files and then concatenating them?

For example, I want to create a 10 octets file, where the first three are set to 4, the next two to 7 and the final five to 32.

0000.0100 0000.0100 0000.0100 0000.0111 0000.0111 0010.0000 0100.0000 0010.0000 0010.0000 0010.0000

I need a solution that would allow me to create large files, where specific bits are repeated many times.

Answer 1

This is quite simple in Python. You can start the interactive interpreter by just typing python and do something like this:

>>> open("test.bin", "wb").write("\4\4\4\7\7\32\32\32\32\32")

'wb' means "open the file for writing binary data to it", merely opening the file with w will assume you want to write text to it.

Note these are octal numbers (\32 is 0x1a, or 26)! They can also be hexadecimal (\xff), and you can save some typing by doing something like this:

>>> "\x00" * 4 + "\x09"
'\x00\x00\x00\x00\x09'

The normal operator precedence applies.

For more data, you can use lists of decimal integers - which I find easier to read and type:

>>> d = [1, 2, 3, 4, 5]
>>> ''.join(chr(i) for i in d)
'\x01\x02\x03\x04\x05'

If you prefer using python3, you'd use bytes() like this:

>>> bytes([1, 2, 3, 4, 5])

Of course 'arithmetic' works as you would expect:

>>> [1, 2, 3] * 2 + [9] * 4 + range(4)
[1, 2, 3, 1, 2, 3, 9, 9, 9, 9, 0, 1, 2, 3]
>>> range(10, 0, -3)
[10, 7, 4, 1]

Binary, orcal and hexadecimal literals are written like this:

0b1100 == 0o14 == 12 == 0xc

(Or 014 if you prefer)

If you want to do some complicated stuff, open the file first and then gradually write to it:

>>> f = open("test.bin", "wb")
>>> f.write("\xff" * 100)
>>> for i in range(10):
...     f.write("\xff\xfa\x03")
...
>>> f.close()

And if you want to script it, you can use the -c switch to invoke the code directly from the command line:

python -c "open('test.bin', 'wb').write(''.join(chr(i) for i in [1, 2, 3]))"

Answer 2

In ruby, writing the pattern 10 times:

File.open('bits','w') do |f|
    10.times do
        f << ([0x04]*3+[0x07]*2+[0x20]*5).pack('C*')
    end
end

By writing to the file on each iteration, we only ever hold a single copy of the bit pattern in memory. That way you can write arbitrarily large files by changing the 10.times to 10000.times etc.

So from the command line:

$ ruby -e "File.open('bits','w'){|f|10.times{f<<([0x04]*3+[0x07]*2+[0x20]*5).pack('C*')}}"

Then check the file:

$ xxd -b bits 
0000000: 00000100 00000100 00000100 00000111 00000111 00100000  ..... 
0000006: 00100000 00100000 00100000 00100000 00000100 00000100      ..
000000c: 00000100 00000111 00000111 00100000 00100000 00100000  ...   
0000012: 00100000 00100000 00000100 00000100 00000100 00000111    ....
0000018: 00000111 00100000 00100000 00100000 00100000 00100000  .     
000001e: 00000100 00000100 00000100 00000111 00000111 00100000  ..... 
0000024: 00100000 00100000 00100000 00100000 00000100 00000100      ..
000002a: 00000100 00000111 00000111 00100000 00100000 00100000  ...   
0000030: 00100000 00100000 00000100 00000100 00000100 00000111    ....
0000036: 00000111 00100000 00100000 00100000 00100000 00100000  .     
000003c: 00000100 00000100 00000100 00000111 00000111 00100000  ..... 
0000042: 00100000 00100000 00100000 00100000 00000100 00000100      ..
0000048: 00000100 00000111 00000111 00100000 00100000 00100000  ...   
000004e: 00100000 00100000 00000100 00000100 00000100 00000111    ....
0000054: 00000111 00100000 00100000 00100000 00100000 00100000  .     
000005a: 00000100 00000100 00000100 00000111 00000111 00100000  ..... 
0000060: 00100000 00100000 00100000 00100000                        

See the documentation for Array.pack here.