Extracting text from file and output the line into a file

Question

After using grep on an html file, I get the following output:

      <div id="v3060000-3062005" class="BLAH...>
      <div id="v50001027-50002018" class="BLAH...>
      <div id="v907200-907202" class="BLAH...>
      <div id="v20024011-20024012" class="BLAH...>

I need to extract the strings of numbers from the lines above and combine them into a URL such as:

http://x.y.z/3060000-3062005,50001027-50002018,907200-907202,20024011-20024012.mp3

May I know how I can do this using a shell script?

Answer 1

Normally, I would advise that you use a proper HTML parser to parse HTML.

However, this data looks pretty straightforward: using a double quote (optionally followed by "v") as the field separator, grab the 2nd field of each line. Then join the pieces with commas

result=$( grep ... file.html | awk -F'"v?' '{print $2}' | paste -sd, )
echo "http://x.y.z/$result.mp3"

Answer 2

Assuming you have the input data from your question stored in a file called data.txt, you can use this command to generate the URL:

grep -Po '\d+-\d+' data.txt | tr '\n' ',' | sed -r 's/(.*),$/http:\/\/x.y.z\/\1.mp3/'

Here is an example run:

$ grep -Po '\d+-\d+' data.txt | tr '\n' ',' | sed -r 's/(.*),$/http:\/\/x.y.z\/\1.mp3/'
http://x.y.z/3060000-3062005,50001027-50002018,907200-907202,20024011-20024012.mp3

What it does:

• grep -Po '\d+-\d+' data.txt extracts all the NUMBER-NUMBER sequences from your data, like 3060000-3062005 and outputs one sequence per line.
• tr '\n' ',' converts all the newline characters of its input to commas, effectively joining the lines together.
• sed -r 's/(.*),$/http:\/\/x.y.z\/\1.mp3/' removes the trailing comma from its input and embeds it into the string http://x.y.z/\1.mp3, replacing \1. Note that all slashes in the string must be escaped with a backslash.

Answer 3

With awk:

... | awk -F'"' '{part=part substr($2,2) ","} \
                  END {sub(",$", "", part); print "http://x.y.z/" part ".mp3"}'

• -F'"' sets the field separator as "

• part=part substr($2,2) ","} gets the second field, and substr($2,2) gets strips off the starting v from string, and the result is saved with adding a trailing , as variable part

• END {sub(",$", "", part); print "http://x.y.z/" part ".mp3"} chunk is run after parsing all records, here we are stripping off the last , from variable part, and printing the desired formatted output

Example:

% cat file.txt
      <div id="v3060000-3062005" class="BLAH...>
      <div id="v50001027-50002018" class="BLAH...>
      <div id="v907200-907202" class="BLAH...>
      <div id="v20024011-20024012" class="BLAH...>

% awk -F'"' '{part=part substr($2,2) ","} END {sub(",$", "", part); print "http://x.y.z/" part ".mp3"}' file.txt
http://x.y.z/3060000-3062005,50001027-50002018,907200-907202,20024011-20024012.mp3

Answer 4

First, you'll need to figure out how to get the text you need to output. Since other answers show how to do this, I won't need to include that information in my answer.

However, I will be addressing how to output that information into a file.

To do this, insert > filename after your command. This will overwrite the file's contents (creating it if it doesn't exist) with the standard output (stdout) of the command used.

An alternative method is to use >> instead of > to append the text to a file instead, rather than overwriting it.