Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I have a file which consists of a URL. I'm trying to get the URL from that file using a shell script.

In the file, the URL is like this:

('URL', 'http://url.com');

I tried to use the following:

cat file.php | grep 'URL' | awk '{ print $2 }'

It gives the output as:

'http://url.com');

But I need to get only url.com in a variable inside the shell script. How can I accomplish this?

share|improve this question
add comment

6 Answers

up vote 9 down vote accepted

Something like this?

grep 'URL' file.php | rev | cut -d "'" -f 2 | rev

or

grep 'URL' file.php | cut -d "'" -f 4 | sed s/'http:\/\/'/''/g

To strip out http://.

share|improve this answer
3  
Or: cat file.php | grep 'URL' | cut -d "'" -f 4. –  Eric Carvalho Feb 17 at 15:02
    
I tried the answer by Frantique it gave http://url.com not url.com –  Tarun Feb 17 at 15:04
1  
@Tarun Yes, I just wanted to say that there's no need to double reverse the text. –  Eric Carvalho Feb 17 at 15:12
1  
When you want to match something with / in sed, you should usually use a different delimiter, e.g. sed s@http://@@g. –  Kevin Feb 17 at 21:18
2  
This is very inefficient, though, solution 1 calls 5 processes over 4 pipes, and solution 2 calls 3 processes over 2 pipes including 2 regexes. This can all be done in the Bash shell without any pipes, processes or dependencies. –  AsymLabs Feb 21 at 0:10
show 4 more comments

You can do everything with a simple grep:

grep -oP "http://\K[^']+" file.php 

From man grep:

   -P, --perl-regexp
          Interpret  PATTERN  as  a  Perl  regular  expression  (PCRE, see
          below).  This is highly experimental and grep  -P  may  warn  of
          unimplemented features.
   -o, --only-matching
          Print  only  the  matched  (non-empty) parts of a matching line,
          with each such part on a separate output line.

The trick is to use \K which, in Perl regex, means discard everything matched to the left of the \K. So, the regular expression looks for strings starting with http:// (which is then discarded because of the \K) followed by as many non-' characters as possible. Combined with -o, this means that only the URL will be printed.

You could also do it in Perl directly:

perl -ne "print if s/.*http:\/\/(.+)\'.*/\$1/" file.php\
share|improve this answer
    
very nice answer. +1 from me. –  souravc Feb 17 at 17:39
    
Very nice compact solution. My favourite too. –  AsymLabs Feb 19 at 10:44
add comment

Try this,

awk -F// '{print $2}' file.php | cut -d "'" -f 1
share|improve this answer
    
No didn't worked. –  Tarun Feb 17 at 15:05
    
what is the problem? can you tell me if you are getting it correct echo "define('URL', 'http://url.com');" | awk -F// '{print $2}' | cut -d "'" -f 1 –  souravc Feb 17 at 15:08
    
The problem is that the url.com to a different url as well like abc.com its dynamic and I need to grab this url using shell script. –  Tarun Feb 17 at 15:18
add comment

Revisiting this again, and trying to use nothing but a Bash shell, another one line solution is:

while read url; do url="${url##*/}" && echo "${url%%\'*}"; done < file.in > file.out

Where file.in contains the 'dirty' url list and file.out will contain the 'clean' URL list. There are no external dependencies and there is no need to spawn any new processes or subshells. The original explanation and a more flexible script follows. There is a good summary of the method here, see example 10-10. This is pattern based parameter substitution in Bash.

Expanding on the idea:

src="define('URL', 'http://url.com');"
src="${src##*/}"        # remove the longest string before and including /
echo "${src%%\'*}"      # remove the longest string after and including '

Result:

url.com

No need to call any external programs. Furthermore, the following bash script, get_urls.sh, permits you to read a file directly or from stdin:

#!/usr/bin/env bash

# usage: 
#     ./get_urls.sh 'file.in'
#     grep 'URL' 'file.in' | ./get_urls.sh

# assumptions: 
#     there is not more than one url per line of text.
#     the url of interest is a simple one.

# begin get_urls.sh

# get_url 'string'
function get_url(){
  local src="$1"
  src="${src##*/}"        # remove the longest string before and including /
  echo "${src%%\'*}"      # remove the longest string after and including '
}

# read each line.
while read line
do
  echo "$(get_url "$line")"
done < "${1:-/proc/${$}/fd/0}"

# end get_urls.sh
share|improve this answer
    
Nice, +1. Strictly speaking though, there is a subshell, the while loop happens in a subshell. On the bright side, this works with just about any shell except [t]csh, so it's good for sh, bash, dash, ksh, zsh... –  terdon Feb 25 at 9:39
    
Bash for the win! –  Andrea Corbellini Feb 28 at 18:17
add comment

If all the lines contain a URL:

awk -F"'|http://" '{print $5}' file.php 

If only some lines contain a URL:

awk -F"'|http://" '/^define/ {print $5}' file.php 

Depending on the other lines you may need to change the ^define regex

share|improve this answer
    
It worked just had to add a cut statement the command I used is awk -F"'|http://" '/^define/ {print $5}' file.php | cut -d ")" -f 1 –  Tarun Feb 17 at 15:17
add comment

Simple:

php -r 'include("file.php"); echo URL;'

and if you need to remove the 'http://', then:

php -r 'include("file.php"); echo URL;' | sed 's!^http://\(.*\)!\1!'

So:

myURL=$(php -r 'include("file.php"); echo URL;' | sed 's!^http://\(.*\)!\1!')

If you need a certain part of the URL you need to refine your terminology, a URL is all of the following, sometimes more:

URL := protocol://FQDN[/path][?arguments]

FQDN := [hostname.]domain.tld
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.