Parsing a HTML page with Bash

Question

I'm trying to write a Bash script that will extract informations from a HTML page (using wget). I know my informations will be between <h*> tags, but is there a nice way to get those ?

To be more precise let's have an example :

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

I'd like to extract "otherHeader2", a.k.a. the second (but it could be anywhere) header afer the header1.

Answer 1

this is a simple python script that will parse your html and put all the values into a list and print it out. You can either write the rest of your script in python, or call this script from python, or plug this short code as a snippet in bash. Check out the examples below.

test.html

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

parse_header.py

#!/usr/bin/env python
import sys, re
print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())

script can be called from bash

cat test.html | parse_header.py

python code can be put into bash script

cat test.html | python -c "import sys, re;print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())"

the last option I believe is not very readable in your case. It makes more sense if you have some simple code where it's not worth putting it in it's own script.

Answer 2

You should never parse html with regular expressions. Always use a proper html or xml parser. For example, using xmlstarlet:

$ xmlstarlet sel -t -v '//h2[2]' html.html 
otherHeader2