1

I'm trying to write a Bash script that will extract informations from a HTML page (using wget). I know my informations will be between <h*> tags, but is there a nice way to get those ?

To be more precise let's have an example :

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

I'd like to extract "otherHeader2", a.k.a. the second (but it could be anywhere) header afer the header1.

  • It could be done, but it would be somewhat painful, I think. If it were me, I'd use another scripting language, such as perl, python, or PHP, which can be run like a Bash script by using the proper shebang (hashbang); ie #! /usr/bin/python for a python script. You may even find existing code for one of those, and it might be useful to learn the languages. – Marty Fried Oct 5 '12 at 23:30
  • What about using sed or awk? – dearN Oct 20 '12 at 23:35
1

this is a simple python script that will parse your html and put all the values into a list and print it out. You can either write the rest of your script in python, or call this script from python, or plug this short code as a snippet in bash. Check out the examples below.

test.html

< h1>header1< /h1>
< h2>header2< /h2>
< h2>otherHeader2< /h2>
< h1>lastHeader1< /h1>
< h2>lastHeader2< /h2>

parse_header.py

#!/usr/bin/env python
import sys, re
print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())

script can be called from bash

cat test.html | parse_header.py

python code can be put into bash script

cat test.html | python -c "import sys, re;print re.findall(r'< h\d>(.*)< /h\d>', sys.stdin.read())"

the last option I believe is not very readable in your case. It makes more sense if you have some simple code where it's not worth putting it in it's own script.

| improve this answer | |
2

You should never parse html with regular expressions. Always use a proper html or xml parser. For example, using xmlstarlet:

$ xmlstarlet sel -t -v '//h2[2]' html.html 
otherHeader2
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.