How to remove all but number from hexdump output?

Question

I have got the base64 output for a huge file, the output is also obviously huge and in a file of its own, I wanted to convert the big number in it to base10 so I have now used hexdump to get it into base16 format... But the output looks something like this:

0077000 0022 00de 0000 0000 de00 0800 0000 0000
0077010 0008 0473 0c16 221f 0200 0000 0001 0100
0077020 0001 0022 00de 0000 0000 de00 0800 0000
0077030 0000 0008 0473 0c16 221f 0200 0000 0001
0077040 0100 0101 0032 00df 0000 0000 df00 0800
0077050 0000 0000 0008 0473 0c16 221f 0200 0000
0077060 0001 0100 0010 0073 006c 002d 006d 006f
0077070 0064 0065 006d 0000 0000 0000 0000 0000
0077080 0000 0000 0000 0000 0000 0000 0000 0000
*
0077800 0022 00df 0000 0000 df00 0800 0000 0000
0077810 0008 0473 0c16 221f 0200 0000 0001 0100
0077820 0001 0022 00df 0000 0000 df00 0800 0000
0077830 0000 0008 0473 0c16 221f 0200 0000 0001
0077840 0100 0101 0078 d690 0008 0800 90d6 961a
0077850 0006 0600 1a96 0473 0c16 221f 0000 0000
0077860 0001 0100 0056 0073 006c 002d 006d 006f
0077870 0064 0065 006d 002d 0064 0061 0065 006d
0077880 006f 006e 005f 0032 002e 0039 002e 0031
0077890 0031 007e 0032 0030 0031 0031 0030 0033
00778a0 0032 0031 002d 0031 0031 005f 0069 0033
00778b0 0038 0036 002e 0064 0065 0062 0000 0000
00778c0 0000 0000 0000 0000 0000 0000 0000 0000
*

I know how to remove all spaces and returns and make it all on one line, but how do I turn the stars into the removed lines they represent and remove the first 8 characters which are not part of the number? Or is there perhaps a way to make hexdump just output the number instead of all the other stuff? The file is obviously too large for me to do it manually so a command to do it automatically would be greatly appreciated. I am running Ubuntu GNOME 15.10 with GNOME 3.18.

Answer 1

You can use xxd to create plain-style hex dump

$ xxd -p /etc/passwd | head -n 3                                                                                      
726f6f743a783a303a303a726f6f743a2f726f6f743a2f62696e2f626173
680a6461656d6f6e3a783a313a313a6461656d6f6e3a2f7573722f736269
6e3a2f7573722f7362696e2f6e6f6c6f67696e0a62696e3a783a323a323a

And output it to file

$ xxd -p /etc/passwd > output.hex

You can restore data later with -r -p options

$ xxd -r -p output.hex  | head -n 3                                                                                   
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin

Couple other methods :

$ hexdump -ve '1/1 "%.2x"' /etc/passwd
$  od -t x1 /etc/passwd | awk '{$1=""; gsub(/\ /,"");print}'

Answer 2

Use hexdump specifying a custom output format:

hexdump -ve '8/1 "%04x " "\n"'

• -v: prints all data (gets rid of the asterisks, which mean repetition of the last line of output)
• -e '8/1 "%04x " "\n"': prints 8 times a single byte in a zero-padded (up to four digits) lowercase heaxadecimal representation followed by a a space and appends a newline until no more data is available.

% </dev/urandom head -c 32 | hexdump -ve '8/1 "%04x " "\n"'
003c 00b0 00bb 00de 003d 0041 0072 00a5
00c2 0078 00b9 009b 00b7 00fc 0076 0030
00b7 005c 00f4 0036 00a0 004e 003b 001b
0028 00cd 006a 0079 0055 0014 000c 007b

---

I'm not sure why you'd want to do that, but apparently you want to print everything on a single line without spaces.

If that's the case:

hexdump -ve '/1 "%04x"'

% </dev/urandom head -c 32 | hexdump -ve '/1 "%04x"'
0046007c00b2003300e0009b00eb00d500bf006900570081008e003e005a002f0024001700ab00000084001e00ad00ab003a004800bf0039005c00aa00030072%
% 

Answer 3

By default hexdump will remove repeated line and put a single * in the place where one or more lines have been removed. By removing the first column with file positions, you will have lost the only indication of how many lines were removed, so it is no longer possible to reconstruct the original file.

Presumably you want a number that uniquely represents the original file, so you would have to start with hexdump -v. As shown in a previous answer, it is possible to give hexdump other arguments to change the output format into the format you were asking for.

However if you for some reason only have the output of hexdump -v and not the original file it would also be possible to remove the first column and white space like this:

cat filename.txt | cut -c9- | tr -dc 0-9a-f

Here cut -c9- will remove the leftmost 8 characters of each line outputting only the 9th character and everything after it. And tr -dc 0-9a-f will keep only characters in the range 0-9 or a-f, so all spaces and newlines are removed.