Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a huge text file that contains IP addresses scrambled all over but not in one unit

for example.

So what the@192@ heck are you doing@168@ in my house @10@.you were not @16@ supposed to be here.

What I want is to make an output file and in that make ipadresses file like following

    1.192.168.10.16
    2.192.223.22.44
    etc..

I have good understanding of regular expressions and can extract specific information from a text file but here I need to combine these that where I got confused.How should one approach problem like that? I am using Ubuntu 12.04.

share|improve this question
    
Please give a more complete example. Are the number that make up the IPs always delimited by @? Can you have an IP spread out across multiple lines? –  terdon Apr 12 at 22:16

3 Answers 3

up vote 6 down vote accepted

The simplest approach I can think of, assuming that the numbers you want are always delimited by @ symbols, is:

$ grep -oP '@\K\d+' file | perl -pe '$. % 4 != 0 && s/\n/./;'
192.168.10.16
192.169.10.16
192.128.10.16
192.162.10.16

This does not number the lines though, so to add them do

$ grep -oP '@\K\d+' file | perl -pe '$. % 4 != 0 && s/\n/./;' | perl -pe 's/^/$.. /'
1. 192.168.10.16
2. 192.169.10.16
3. 192.128.10.16
4. 192.162.10.16

Explanation

  • grep -oP '@\K\d+' file : -o means "print only the matching part of the line" and -P enables Perl Compatible Regular Expressions (PCRE) for grep. This lets us use the \d to match numbers and, most importantly, the \K which means "forget whatever you matched before me". The \K lets me grep for @\K10 and only print the 10 because the @ is before the \K.
  • perl -pe : read the input file line by line, apply the script given by -e to each line and then print that line (-p).
  • '$. % 4 != 0 && s/\n/./; : % is the modulo operator, $. is the current line number of the input file. This code will substitute a newline charatcer (\n) with a . on lines that are not divisible by 4. The result is that since we are feeding it a list of numbers (the output of grep), each group of 4 numbers will be printed on the same line since the \n was converted to a ..
  • perl -pe 's/^/$.. /' : Just add the current line number to the beginning of each line.

Steeldriver suggested a very nice alternative:

grep -oP '@\K\d+' file | xargs -n4 printf '%d.%d.%d.%d\n' | cat -n

Which made me think of this one:

printf '%d.%d.%d.%d\n' $(grep -oP '@\K\d+' file ) | cat -n

If you like, you can do the whole thing in Perl and avoid pipes but I would use the method above. Anyway, always assuming that your numbers are surrounded by @, this will also work:

perl -ne 'push @f,(/@(\d+)@/g); 
          END{
            $k=1;
            for($i=0;$i<=$#f;$i+=4){
                print "$k. " . join(".",@f[$i..($i+3)]) . "\n"; $k++}
            }' file

You can paste that directly into your terminal, just change file for the actual file name. The output looks like this:

1. 192.168.10.16
2. 192.169.10.16
3. 192.128.10.16
4. 192.162.10.16

Explanation

  • perl -ne : read the input file line by line (-n) and apply the script given by -e.

  • push @f,(/@(\d+)@/g); : Save each number surrounded by @ as an element of the @f array.

  • END{} : do this after you have finished processing all lines
  • for($i=0;$i<=$#f;$i+=4){} : iterate through the array. Since IPs have 4 sets of numbers, we read the array in jumps of four.
  • join(".",@f[$i..($i+3)]) : This connects the 4 elements of the array with . for printing.
  • The $k is just to print the numbers in front of the IPs.

share|improve this answer
    
Very nice! I didn't know about \K (was trying to do something similar using perl lookarounds). I did find an alternative for the modulo counting though - using xargs e.g. grep -oP '@\K\d+' file | xargs -n4 printf '%d.%d.%d.%d\n'. One could pipe through cat -n for the numbering. –  steeldriver Apr 13 at 0:00
    
@steeldriver ooh, that printf is clever! Not at my computer at the moment but unless you want to post an answer yourself, I'll modify mine to include those suggestions. Assuming you don't mind of course. –  terdon Apr 13 at 0:05
    
Sure no problem! I wish I could figure out a way to do the counting natively as well... –  steeldriver Apr 13 at 1:14
    
@steeldriver done, thanks. I also thought of an even shorter one: printf '%d.%d.%d.%d\n' $(grep -oP '@\K\d+' file ) | cat -n. –  terdon Apr 13 at 13:50
    
@terdon Please explain a little bit more and easy to understand the following expression '$. % 4 != 0 && s/\n/./; –  khan Apr 14 at 17:53

Using GNU sed

You could also use sed for this purpose.Assuming the numbers in the ipaddress are present in-between @@ symbols.

$ sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' file
192.168.10.16
192.169.10.16
192.128.10.16
192.162.10.16

Below command puts the number in an order before the fetched ip-addresses,

$ sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' file | awk '{ print NR". "$0}' 

Example:

$ echo 'So what the@192@ heck are you doing@168@ in my house @10@.you were not @16@ supposed to be here.' | sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' | awk '{ print NR". "$0}'
1. 192.168.10.16

Please Explain your code a bit?

sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' file | awk '{ print NR". "$0}'
|                                                                   |   |                     |
|                                                                   |   |                     |
|<----------------------First part--------------------------------->|   |<-----Second part--->|   

OP mentioned that Ip addresses are scrambled(spreaded) all over the file with each part of an ipaddress is enclosed within @@ and each line contains four @...@ parts. So he wants to fetch all the numbers present inside @@ line by line and prints it in an ip-address format(xxx.xxx.xxx.xxx).

First Part

sed parses the input file line by line.

Consider the below regex in my code and also the above mentioned example. We have to give the regex that matches the whole line and also it would contain fetching groups in-order to fetch the words according to our criteria, so that the fetched group would be reused through back-reference.

Example line:

So what the@192@ heck are you doing@168@ in my house @10@.you were not @16@ supposed to be here.

Regex:

.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*

.*

It matches any character 0 or more time except newline character.

@\(.*\)@

In sed, ()(fetching groups) this parentheses are used to fetch group of characters or words or numbers. By default sed uses basic regex so we have to escape the parenthesis inorder to make fetching to work.But if you use sed with -r(extended-regex) flag, you don't need escaping.

In our case it's \(.*\) , both open and close brackets are escaped. On sed reading the whole line, it stops the matching and starts to fetch all the characters or numbers or anything after the @ symbol and stops fetching until it finds the next @ symbol. Then it stores the fetched group in a special buffer called (pattern space). so that fetched characters would be used further. Now sed fetches the numbers between the first @@ symbols (i,e.192).

.*

After catching the first group, sed starts to parse the next characters and matches anything 0 or more times.

@\(.*\)@

Fetch the numbers between the second @@ symbols.(i,e 168)

.*

Matches any and then it moves on.

@\(.*\)@

Fetch the numbers between the third @@ part.(i.e 10)

.*

Matches any and then it moves on.

@\(.*\)@

Fetch the numbers between the fourth @@ part.(i.e 16)

.*

There may or may-not be characters present after the fourth @@ symbols. So we have to give this .* in-order to match all the characters after the fourth @@ part.

So sed fetches the exact numbers we want and stored it in a buffer.

Default Format(syntax) of sed,

sed 's/regex/replacement/g' file

code:

sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' file

so sed searches for the match of this regex. Once it finds the match, it will be replaced by the replacement part. And g global flag helps to replace all the occurrences of regex string with the replacement part.(g-global).

In our case, regex will matches the first line and the whole line would be replaced by our fetched groups 1,2,3,4. Then sed does this operation on all the lines which matches our regex. In the replacement part the fetched groups are preceeded by \. And hence it's called back-referencing. If we fail to separate the groups by .(dots),

\1\2\3\4

the output would be,

1921681016

So we have to separate the groups with .(dots). So that it would appears in an ip-address format.

\1.\2.\3.\4

Now the output would be,

192.168.10.16

Second Part

awk '{ print NR". "$0}'

Now the sed output for the first line would be 192.168.10.16. This output was feded as input to the second awk command.

  • Awk's NR(Number of Records) variable stores the line or record number.Note that, at-last value of NR would be the last line number. Like sed , awk parses the input file line by line. So NR value of 1st line would be 1 and the second line would be 2 and so on.

  • In awk's print function, the character would be printed as it is, if it was placed within double quotes. So it prints . after the line number(i,e current NR).

  • $0 prints the whole line as it is.

So the output of whole command would be,

$ sed 's/.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*@\(.*\)@.*/\1.\2.\3.\4/g' file | awk '{ print NR". "$0}'
192.168.10.16
192.169.10.16
192.128.10.16
192.162.10.16

You may also use this command,

sed 's/.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*/\1.\2.\3.\4/g' file

Example:

$ echo 'So what the@192@ heck are you doing@168@ in my house @10@.you were not @16@ supposed to be here.' | sed 's/.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*@\([0-9]\+\)@.*/\1.\2.\3.\4/g'
192.168.10.16
share|improve this answer
    
+1 for beauty and conciseness. –  don.joey May 10 at 8:53
    
Please Explain your code a bit? –  khan May 11 at 21:19
    
If you still have any doubts, don't hesitate to ping me from here. –  Avinash Raj May 12 at 8:46

There may be a fancy way to do this with terminal, but I don't know how. Here's how I would do it using python

Copy this code into a new file, name it whatEverYouWant.py and change the line that says 'input.txt' to be 'yourFileWithIps.txt'

import re

validIpAddressRegex = "^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$";

ips = []
with open('input.txt','r') as f:
    output = f.read()
    ips = re.findall(r'[0-9]+(?:\.[0-9]+){3}', output)

for x in range(1, len(ips) + 1):
    print str(x) + '.' + ips[x-1]

then from terminal navigate to where you saved whatEverYouWant.py and type

python whatEverYouWant.py

and that should output what you want.

Results from my own test

cam@cam-P5E:~/Desktop$ python getips.py
1.192.168.0.1
2.255.255.255.0
3.10.0.0.1
4.192.192.192.192
share|improve this answer
1  
My python-fu is very limited but as far as I can tell, your script is looking for actual IPs, as in 123.456.7.8, but the OP has the component numbers spread out randomly in the file, presumably delineated by @. In any case, your script gives no output on the OP's example file. –  terdon Apr 12 at 22:56
    
I think your regex should incorporate something like (@(\d{1,3})@.*){4} (#NOTE untested), viz. 4 groups of two @'s with inbetween 1 to three digits that you select as subgroups. –  don.joey May 10 at 8:56

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.