I have a big html file on my desktop that looks like

lotto results</td><td>complete lotto results</td><td>2</td><td><a

If possible, I want to:

  • get all the .jpg files, remove all the html code (it's 1.jpg, 2.jpg... to 42.jpg)
  • I want to remove the .jpg extenstion
  • I want each row of numbers to have 7 numbers only, then insert new line

This is not actually a particularly good job for sed but here goes:

sed -nr 's#.*/([^"]+).jpg.*#\1#p' file 

The above will get you a list of numbers, one per line:


Now, it is actually possible to get all these on the same line with 7 numbers per line using sed but it is really not worth the effort. Just use standard *nix tools instead:

$ echo $(sed -nr 's#.*/([^"]+).jpg.*#\1#p' file | tr $'\n' ' ') | fold -sw 21
20 20 32 32 32 30 30 

Or, if you want to remove duplicates:

echo $(sed -nr 's#.*/([^"]+).jpg.*#\1#p' file | sort -u | tr $'\n' ' ')
20 30 32


The sed command uses a few tricks:

  • -n: don't print any lines by default.
  • -r: enable extended regular expressions, this lets us use ( ) to capture groups without needing to escape the parentheses and + for "one or more".
  • s#from#to# : while the standard substitution operator in sed and other, similar tools, is s/from/to/, you can a non standard delimiter so that you can include / in the pattern. In this case I am using # but you could use something else like s|from|to| as well.
  • s#.*/([^"]+).jpg.*#\1#p : this will match everything from the beginning of the line until a / and then captures the longest stretch of non-" characters until .jpg. This is the filename minus extension. The filename is captured in the parenthesis and the whole line (because of the .* on either side) will be replaced with the captured patter (\1). The p at the end means that it will print the lines where the substitution was successful.

Personally though, I would have done all of this with perl in the first place:

$ perl -e '@k=grep(s/.*\/([^"]+).jpg.*/$1/s,<>); print "@k[0..6]\n@k[7..$#k]\n"' file 
20 20 32 32 32 30 30

Or, for a larger file:

$ perl -e '@k=grep(s/.*\/([^"]+).jpg.*/$1/s,<>); for($i=0;$i<=$#k;$i+=7){print "@k[$i..$i+7]\n"}' file 
20 20 32 32 32 30 30 30

Or grep even:

$ echo $(grep -oP '[^/]+(?=.jpg)' file | tr $'\n' ' ' ) | fold -w 21
20 20 32 32 32 30 30 

Or, stealing @Olli's clever xargs idea:

$ grep -oP '[^/]+(?=.jpg)' file |  xargs -n7 echo
20 20 32 32 32 30 30
  • Actually, the fold -w 21 part is not going to work for the OP, since he states that the numbering scheme starts with 1, not 01... – Run CMD Jul 25 '14 at 11:48
  • @ClassStacker ah, point. Yes, the fold counts characters, not words. column should be able to do this dammit but I couldn't get it to play nice. xargs -n7 is probably the simplest. – terdon Jul 25 '14 at 11:50
  • There's also the problem (that I share) that the HTML probably isn't going to be split so that there's only one number per line. Convert the input into a single line and most of the sed functions fail hard because they replace too greedily. That last grep though is the star, just needs a little uniq so you're not pulling in alt="" versions of the same number (IMO, but the OP might want duplicates) – Oli Jul 25 '14 at 11:52
  • 2
    An old-school way of displaying data with multiple (e.g., seven) entries per line is pr -7 -l1. (pr -7 would give you typical multi-column output, with 1-56 in column 1, 57-112 in column 2, etc. -l1 says to output pages that are one line long, so values 1-7 go on the first line, 8-14 on the second, etc.) – Scott Jul 25 '14 at 15:29
  • 2
    another technique to print 7 words per line: paste - - - - - - - – glenn jackman Jul 25 '14 at 15:34

I assume you're trying to scrape some sort of result. In this example there are only three balls and we can extract them by searching for Balls/<one-or-many-digits> and grouping (the \(..\) construct) around the number and then replacing the whole lot with that group (the \1 is a reference to the first group).

$ sed -n 's/.*Balls\/\([0-9]\+\).*/\1/gp' htmlfile | uniq | xargs -n7 echo
20 32 30

sed is going through this line by line. I'm asking it to match and replace everything on the line (which is why we cap each end with .* — "any amount of anything") with whatever it matches in the group. The -n and /p are used together to not print unless the line was a match and the /g means it'll keep matching until it hits the end of the file.

It's a fairly complicated example if you're new to regular expressions.

I'm passing it through uniq because there's a lot of duplication going on there.

And I'm using | xargs -n7 echo on the end to group 7 arguments together and pass them all onto echo. There aren't 7 balls here so it's only showing 3.

It probably slows it down but you can have a slightly more readable expression if you use the -r extended syntax for sed:

sed -nr 's/.*Balls\/([0-9]+).*/\1/gp' htmlfile | ...

Does the same thing, just without some of the confusing looking escaping.
Probably ever-so-slightly slower.

  • 1
    isn't the g modifier for making multiple substitutions in the current pattern space (usually the current line)? AFAIK it doesn't mean "keep matching until [it hits] the end of the file" - that will happen anyway, unless you explicitly quit (q or Q). – steeldriver Jul 25 '14 at 11:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.