9

I have some HTML that I am trying to extract links from. Right now the file looks like this.

website.com/path/to/file/234432517.gif" width="620">
website.com/path/to/file/143743e53.gif" width="620">
website.com/path/to/file/123473232.gif" width="620">
website.com/path/to/file/634132317.gif" width="620">
website.com/path/to/file/432432173.gif" width="620">

I am trying to use sed to remove the " width="620"> from all the lines. Here is my sed code:

sudo sed -i "s/\"\swidth\=\"\d+\"\>//g" output

Why is this not working? everything I google leads to some code that looks like this but this does not work for some reason.

3
  • 1
    Another solution to your problem would be to use cut : cut -d'"' -f1 will return the first field separated by ", that is the gif url. Assuming the url is fixed length, cut -c 1-38 would also work, returning the 38 first characters of each line that compose the url.
    – Aaron
    Jul 21 '17 at 15:15
  • 1
  • 9
    @DavidFoerster strictly speaking, this is just ASCII text. The example shown isn't even valid HTML and it is absolutely simple enough that regular expressions can indeed deal with it. While we're all fond of that gem of an answer and it is absolutely true in general, it doesn't mean that you can never parse any XML/HTML-like data with simple tools. Only that it is usually a bad idea unless you're Tom Christiansen.
    – terdon
    Jul 21 '17 at 15:49
24

Because you are using PCRE (Perl Compatible Regular Expressions) syntax and sed doesn't understand that, it uses Basic Regular Expressions (BRE) by default. It knows neither \s nor \d. You are also escaping all sorts of things that don't need to be escaped (neither the \= nor the \> are doing anything useful) while not escaping things that do need to be escaped (+ just means the symbol + in BRE, you need \+ for "one or more".

This should do what you need:

sed 's/" width="[0-9]\+">//g' file

Or, using Extended Regular Expressions:

sed -E 's/"\s*width="[0-9]+">//g' file

Finally, as a general rule you never use sed -i without first testing without the -i to be sure it works or, if you do, at least use -i.bak (-i with any text will do this) to create a backup.

2
  • 2
    Generally speaking, if you have a choice for these things go with Perl as it has one of the most powerful regular expression engines available in a command line tool. Jul 22 '17 at 7:25
  • Oh I did not know there were different regular expression languages. I just filled my data into regexer.com did made up the RegEx and assumed it would work. Thanks. Jul 22 '17 at 12:19
4

Here is my sed solution:

sed -E 's/(.*)" width="[0-9]+">/\1/' filename

And as an alternative to the sed I suggest using grep to extract data from a file:

This would work for you:

grep -o "website.*\.gif" filename

And as terdon suggested, here is a look ahead solution using grep:

grep -Po '.*(?="\swidth="\d*">)' filename

Also cut is a good option in your situation:

cut -f1 -d'"' filename
5
  • 1
    Good Idea. But -P is not needed in that case, and I would use [^\"]* instead of .*\.gif. That would be less specific.
    – pLumo
    Jul 21 '17 at 12:54
  • Yeah, edited... I was testing something which didn't worked ;)
    – Ravexina
    Jul 21 '17 at 12:55
  • 1
    A better approach using grep would be grep -oP '.*(?=" width="\d+">)' file since that i) doesn't assume the presence of any string (like "website" in your example) not mentioned by the OP and ii) uses the same basic idea as the OP so we can be sure it will match their data.
    – terdon
    Jul 21 '17 at 13:00
  • @terdon I'll add a look ahead solution right now ;) thanks.
    – Ravexina
    Jul 21 '17 at 13:01
  • There are hundreds of working solutions with grep and sed. I love it.
    – pLumo
    Jul 21 '17 at 13:02
1

Or for a shorter exchange simply remove everything after gif

sed 's/gif.*/gif/' file

The .* matches any number of any characters, as long as what you want to lose is always after a string that you can locate... and that there are no other instances of it in a line. It would match website.com/path/to/gif/xyz.gif" width..." on the earlier gif, so give undesired results.

0

Since you've written a Perl regular expression, you can just use Perl. You can issue a one-line perl command instead of a sed command. Then you don't have to translate your regular expression into a different dialect, and you don't have to forgo any of Perl's handy features.

  • Instead of sed -i, use perl -pi -e.
  • Instead of sed -i.bak, use perl -pi.bak -e. (You can use any suffix, it needn't be .bak.)

As terdon says, with -i it's best to specify a suffix so it creates a backup file, at least if you haven't tried the same command without -i first. (This is as true with perl as with sed.)

Here's what it looks like with your specific command:

perl -pi.bak -e "s/\"\swidth\=\"\d+\"\>//g" file

If you use single quotes around the search and replacement pattern, you don't have to escape the " characters that appear in it or keep track of the shell's interesting rules for double-quoted text. This will make it easier to write and read. (That's not specific to perl; your sed command could be similarly simplified.) This command is equivalent:

perl -pi.bak -e 's/"\swidth\="\d+"\>//g' file

With either of those commands, the lines in your example are changed to:

website.com/path/to/file/234432517.gif
website.com/path/to/file/143743e53.gif
website.com/path/to/file/123473232.gif
website.com/path/to/file/634132317.gif
website.com/path/to/file/432432173.gif

Further Reading:

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.