0

I have a set of .html files within a folder and I'm trying to look through each folder and extract a particular bit of info and store it into a CSV file.

Each file contains a similar line of data:

<h1><span class='noScreen'>Intranet:<br/></span>CLAS1206: Web Infrastructure (2019-2020)</h1><div class='pageContentBars0'><hr class='hide'/>

I'm trying to take the CLAS plus the following 4 numbers (different for each file) and take the class name "Web Infrastructure" (again different for each file) get achieve the following:

CLAS1206, Web Infrastructure(\n)
CLAS1000, Mathematics(\n)
CLAS2000, Science(\n)

etc...

Thus far I've managed to use the sed command to remove everything before CLAS but I have no idea where to go from here. I have the following in my script so far:

#!/bin/bash

grep "<span class='noScreen'>" ./modules/CLAS1206.html | sed 's/^.*\(CLAS\)/\1/' | sed

which gives me the following output on the terminal

CLAS1206: Web Infrastructure (2019-2020)</h1><div class='pageContentBars0'><hr class='hide'/>

Any help on how to further break down this line and get it into the correct format would be appreciated!

Improve this question
1
  • 1
    A substitution with sed and a suitable regular expression should do the bulk of the work. – Gyro Gearloose Oct 1 '20 at 13:11
0

here is what I would work around :

sed '/noScreen/ !d ; s/.*\(CLAS[0-9]\{4\}:\).*\(class=.*\).*/\1\2/ ; s/class=.//; s/.\/.*//'

👨francois@💻zaphod🐙:~/tmp$ cat toto
<h1><span class='noScreen'>Intranet:<br/></span>CLAS1206: Web Infrastructure (2019-2020)</h1><div class='pageContentBars0'><hr class='hide'/>
<h1><span class='noScreen'>Intranet:<br/></span>CLAS1234: Web Infrastructure (2019-2020)</h1><div class='pageContentBars0'><hr class='testline2'/>
<h1><span class='noScreen'>Intranet:<br/></span>CLAS5678: Web Infrastructure (2019-2020)</h1><div class='pageContentBars0'><hr class='testline3'/>
👨francois@💻zaphod🐙:~/tmp$ sed '/noScreen/ !d ; s/.*\(CLAS[0-9]\{4\}:\).*\(class=.*\).*/\1\2/ ; s/class=.//; s/.\/.*//' toto
CLAS1206:hide
CLAS1234:testline2
CLAS5678:testline3
👨francois@💻zaphod🐙:~/tmp$

/noScreen/ !d # grep the lines you need

s/.*\(CLAS[0-9]\{4\}:\).*\(class=.*\).*/\1\2/ # substrings CLASxxxx and class=

s/class=.//; s/.\/.*//' # substring to manage tailing chars

tips : whenever it is possible (maybe all the time), choose only one language or tool for one action (do not use grep+sed+awk or even 2 of them) keep your scripts simple

Improve this answer
2
  • How does "sed '/noScreen/ !d" know which directory to look in? The files are located in a subfolder named "modules" 1 level deeper. Also I'm looking for 'CLAS1206,Web Infrastructure' not the :hide, sorry just trying to wrap my head around all the sed commands used – DamLk Oct 1 '20 at 14:24
  • it uses the file path sed 'commands' filepath/file it is a "grep" nothing else ; if you want to manage many directories & files you have to make a loop over the sed command for i in pathregexp ; do sed ........ $i ; done – francois P Oct 1 '20 at 15:04
0

Here you go, mate :
filenames="1.html 2.html 3.html"; for filename in $filenames; do grep 'pageContentBars0' $filename | sed 's/.*\(CLAS[^<\(]\+\).*/\1/' | sed 's/\:/,/'>>final.csv ; done

I am assuming they all follow the same formatting, hence filtering initially using class pageContentBars0 :
grep 'pageContentBars0' $filename

Here, I am looking for anything starting with CLAS, capture anything until ( or < .
.*\(CLAS[^<\(]\+\)

Ignore the rest of the bits and just replace the entire line with what was captured as the important bits :
.*/\1
Finally, replace : with , :
sed 's/\:/,/'

Improve this answer
4
  • The only issue I have with this method, is that I am searching through ~100 files, I don't really want to specify all ~100names. Is there a way to make filenames="*.html" or something along those lines? – DamLk Oct 1 '20 at 14:58
  • sure mate.. you could do filenames=$(find ./ -type f | grep html). Just make sure you put something specific in the grep and the directory. – MidwayNomad Oct 1 '20 at 15:02
  • That's fine the folder literally only contains .html files that need to be searched through! I managed to get it working somewhat. Just for the last part "Finally, replace : with ," Am I able to replace ": " with ",". My output currently gives me "CLAS1206, Maths ". but I'd like to remove the space between the ", M". So "CLAS1206,Maths" is the output. Do I need to declare the space as its own special character using \? – DamLk Oct 1 '20 at 15:12
  • Replace the last sed with this sed 's/\: /,/' – MidwayNomad Oct 1 '20 at 15:17
0

You can use xmllint for parsing xpath nodes from your html file.

$ xmllint --html --xpath  '//h1/text()' file.html \
  | sed -r 's/(.*): (.*) \(.*\)/\1, \2/' > file.csv
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.