Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a big html file on my desktop that looks like

src="http://images.alaablubnan.com/images/Balls/20.jpg"
alt="http://images.alaablubnan.com/images/Balls/20.jpg"/></a></td><td><a
href="http://images.alaablubnan.com/images/Balls/32.jpg"
target="_blank"><img
src="http://images.alaablubnan.com/images/Balls/32.jpg"
alt="http://images.alaablubnan.com/images/Balls/32.jpg"/></a></td><td><a
href="http://images.alaablubnan.com/images/Balls/30.jpg"
target="_blank"><img
src="http://images.alaablubnan.com/images/Balls/30.jpg"
alt="http://images.alaablubnan.com/images/Balls/30.jpg"/></a></td></tr><tr><td><table><tr><td>webpage/url</td><td>http://www.playlebanon.com/webservices/website/lotto/PopUps/HistoryDetail.aspx?t=1405536730503&FromDraw=1&ToDraw=1213&Draw=0</td></tr></table></td><td>2</td><td>complete
lotto results</td><td>complete lotto results</td><td>2</td><td><a
href="http://www.playlebanon.com/webservices/website/lotto/PopUps/HistoryDetail.

If possible, I want to:

  • get all the .jpg files, remove all the html code (it's 1.jpg, 2.jpg... to 42.jpg)
  • I want to remove the .jpg extenstion
  • I want each row of numbers to have 7 numbers only, then insert new line
share|improve this question
2  
What row of numbers? Please show us your desired output instead of describing it. –  terdon Jul 25 at 9:54
    
@terdon this is an example of what I'm aiming to get imgur.com/xpPBiMo only difference is i want 7 columns in each row instead of 6, and then new line –  Fischer Jul 25 at 11:49
    

2 Answers 2

up vote 10 down vote accepted

This is not actually a particularly good job for sed but here goes:

sed -nr 's#.*/([^"]+).jpg.*#\1#p' file 

The above will get you a list of numbers, one per line:

20
20
32
32
32
30
30
30

Now, it is actually possible to get all these on the same line with 7 numbers per line using sed but it is really not worth the effort. Just use standard *nix tools instead:

$ echo $(sed -nr 's#.*/([^"]+).jpg.*#\1#p' file | tr $'\n' ' ') | fold -sw 21
20 20 32 32 32 30 30 
30

Or, if you want to remove duplicates:

echo $(sed -nr 's#.*/([^"]+).jpg.*#\1#p' file | sort -u | tr $'\n' ' ')
20 30 32

Explanation

The sed command uses a few tricks:

  • -n: don't print any lines by default.
  • -r: enable extended regular expressions, this lets us use ( ) to capture groups without needing to escape the parentheses and + for "one or more".
  • s#from#to# : while the standard substitution operator in sed and other, similar tools, is s/from/to/, you can a non standard delimiter so that you can include / in the pattern. In this case I am using # but you could use something else like s|from|to| as well.
  • s#.*/([^"]+).jpg.*#\1#p : this will match everything from the beginning of the line until a / and then captures the longest stretch of non-" characters until .jpg. This is the filename minus extension. The filename is captured in the parenthesis and the whole line (because of the .* on either side) will be replaced with the captured patter (\1). The p at the end means that it will print the lines where the substitution was successful.

Personally though, I would have done all of this with perl in the first place:

$ perl -e '@k=grep(s/.*\/([^"]+).jpg.*/$1/s,<>); print "@k[0..6]\n@k[7..$#k]\n"' file 
20 20 32 32 32 30 30
30

Or, for a larger file:

$ perl -e '@k=grep(s/.*\/([^"]+).jpg.*/$1/s,<>); for($i=0;$i<=$#k;$i+=7){print "@k[$i..$i+7]\n"}' file 
20 20 32 32 32 30 30 30
30       

Or grep even:

$ echo $(grep -oP '[^/]+(?=.jpg)' file | tr $'\n' ' ' ) | fold -w 21
20 20 32 32 32 30 30 
30

Or, stealing @Olli's clever xargs idea:

$ grep -oP '[^/]+(?=.jpg)' file |  xargs -n7 echo
20 20 32 32 32 30 30
30
share|improve this answer
    
Actually, the fold -w 21 part is not going to work for the OP, since he states that the numbering scheme starts with 1, not 01... –  Class Stacker Jul 25 at 11:48
    
@ClassStacker ah, point. Yes, the fold counts characters, not words. column should be able to do this dammit but I couldn't get it to play nice. xargs -n7 is probably the simplest. –  terdon Jul 25 at 11:50
    
There's also the problem (that I share) that the HTML probably isn't going to be split so that there's only one number per line. Convert the input into a single line and most of the sed functions fail hard because they replace too greedily. That last grep though is the star, just needs a little uniq so you're not pulling in alt="" versions of the same number (IMO, but the OP might want duplicates) –  Oli Jul 25 at 11:52
1  
An old-school way of displaying data with multiple (e.g., seven) entries per line is pr -7 -l1. (pr -7 would give you typical multi-column output, with 1-56 in column 1, 57-112 in column 2, etc. -l1 says to output pages that are one line long, so values 1-7 go on the first line, 8-14 on the second, etc.) –  Scott Jul 25 at 15:29
1  
another technique to print 7 words per line: paste - - - - - - - –  glenn jackman Jul 25 at 15:34

I assume you're trying to scrape some sort of result. In this example there are only three balls and we can extract them by searching for Balls/<one-or-many-digits> and grouping (the \(..\) construct) around the number and then replacing the whole lot with that group (the \1 is a reference to the first group).

$ sed -n 's/.*Balls\/\([0-9]\+\).*/\1/gp' htmlfile | uniq | xargs -n7 echo
20 32 30

sed is going through this line by line. I'm asking it to match and replace everything on the line (which is why we cap each end with .* — "any amount of anything") with whatever it matches in the group. The -n and /p are used together to not print unless the line was a match and the /g means it'll keep matching until it hits the end of the file.

It's a fairly complicated example if you're new to regular expressions.

I'm passing it through uniq because there's a lot of duplication going on there.

And I'm using | xargs -n7 echo on the end to group 7 arguments together and pass them all onto echo. There aren't 7 balls here so it's only showing 3.


It probably slows it down but you can have a slightly more readable expression if you use the -r extended syntax for sed:

sed -nr 's/.*Balls\/([0-9]+).*/\1/gp' htmlfile | ...

Does the same thing, just without some of the confusing looking escaping.
Probably ever-so-slightly slower.

share|improve this answer
1  
isn't the g modifier for making multiple substitutions in the current pattern space (usually the current line)? AFAIK it doesn't mean "keep matching until [it hits] the end of the file" - that will happen anyway, unless you explicitly quit (q or Q). –  steeldriver Jul 25 at 11:18

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.