1

I'm trying to write a Bash script that will extract informations from a HTML page (using wget). I know my informations will be between <h*> tags, but is there a nice way to get those ?

To be more precise let's have an example :

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

I'd like to extract "otherHeader2", a.k.a. the second (but it could be anywhere) header afer the header1.

2
  • It could be done, but it would be somewhat painful, I think. If it were me, I'd use another scripting language, such as perl, python, or PHP, which can be run like a Bash script by using the proper shebang (hashbang); ie #! /usr/bin/python for a python script. You may even find existing code for one of those, and it might be useful to learn the languages. Oct 5, 2012 at 23:30
  • What about using sed or awk?
    – dearN
    Oct 20, 2012 at 23:35

2 Answers 2

3

You should never parse html with regular expressions. Always use a proper html or xml parser. For example, using xmlstarlet:

$ xmlstarlet sel -t -v '//h2[2]' html.html 
otherHeader2
1

this is a simple python script that will parse your html and put all the values into a list and print it out. You can either write the rest of your script in python, or call this script from python, or plug this short code as a snippet in bash. Check out the examples below.

test.html

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

parse_header.py

#!/usr/bin/env python
import sys, re
print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())

script can be called from bash

cat test.html | parse_header.py

python code can be put into bash script

cat test.html | python -c "import sys, re;print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())"

the last option I believe is not very readable in your case. It makes more sense if you have some simple code where it's not worth putting it in it's own script.

1

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .