3

I have a file of URLs, in the format as shown below:

com.blendtuts/S
°=
com.blengineering.www/:http
±=
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²=
com.blenheimsi
³=
com.blenkov.www/page/media/18/34/376
´=
com.blentwell.www/bookmarks.php/jackroldan/sp
¸=
com.blentwell.www/tags.php/I

The file size is in GigaBytes. Say around 250 GB of the file size.

I was trying to reverse the words in the file and extract only the domains from the text. I tried to make it using Ubuntu OS terminal commands. Let me tell you what I have tried:

First I removed the data after “/” using the following command:

~$ ex -sc '%s/\(\/\).*/\1/ | x' newfile.txt > ddm.txt

And the result as:

com.blendtuts/
 °=
com.blengineering.www/
±=
com.blenheimgang.www/
²=
com.blenheimsi
³=
com.blenkov.www/
´=
com.blentwell.www/
¸=
com.blentwell.www/

Now I reversed the complete text in the file using the solution from : https://stackoverflow.com/questions/40467918/how-to-reverse-the-word-in-ubuntu

And got the following result:

    /blendtuts.com
    °= /www.blengineering.com
    ±= /www.blenheimgang.com
    ²= blenheimsi.com
    ³= /www.blenkov.com
    µ=  /www.blentwell.com
    ¶=  /www.blentwell.com
    •=  /www.blentwell.com

/www.blentwell.com

But still the problem is not solved. I would like to how it is possible to extract URLs and put them into another file using Ubuntu. As you can see above the output what still I have is not the domain, it has a backslash with it.

If there is another solution to such a problem using any other operating system, do let me know. I prefer to go with Ubuntu.

I would like to extract domains out of the file and separate them to another file and that too in a proper format.

If I get the unique domain then it will be an excellent solution to my query. Otherwise, I am using command as:

$ sort filename.txt | uniq > save_to_file.txt

Please try not to give me a solution using awk command, as it does not work on my system.

Sample data:

com.blendschutzrollo.www/d_chefsessel6_Maxx_Chefsessel_mit_Kopfstutze_Chefdrehsessel___Munchen__374
¯= 
com.blendtuts/S
°= 
com.blengineering.www/:http
±= 
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²= 
com.blenheimsi
³= 
com.blenkov.www/page/media/18/34/376
´= 
com.blenoir.www/lat
µ= 
com.blentwell.www/bookmarks.php/bashment%20jack/re
¶= 
com.blentwell.www/bookmarks.php/djcable/rt
·= 
com.blentwell.www/bookmarks.php/jackroldan/sp
¸= 
com.blentwell.www/tags.php/I
¹= 
com.blentwell.www/tags.php/eurot
º= 
com.blentwell.www/tags.php/mitarbeiters
»= 
com.blentwell.www/tags.php/verw
¼= 
com.blenzblog/tag/olympic-w
½= 
com.blepharoplastyusa.www/albany-n
¾= 
1
  • 2
    "Please try not to give me a solution using awk command, as it does not work on my system." I'm pretty sure awk works on ubuntu, at least on my box.
    – user12753
    Nov 9, 2016 at 11:39

2 Answers 2

6

A Perl solution, adapting one of the string reversal solutions:

$ perl -F/ -anle 'print reverse(split("([^.]*)", $F[0])) if /\./' input
www.blendschutzrollo.com
blendtuts.com
www.blengineering.com
www.blenheimgang.com
blenheimsi.com
www.blenkov.com
www.blenoir.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
blenzblog.com
www.blepharoplastyusa.com

The arguments:

  • -F/ -a creates an array F out of each line of input, splitting on /.
  • -nle runs the expression (-e <expr>) on each line of input, without automatically printing (-n), while handling the newline at the end of each line (-l)
  • The line is already split on /, and we only need the part before the first /, so the first element of array F: $F[0]. Then we split it on . and reverse each of those, and print if the line contains ..

Now you can sort -u this.

1
  • Excellent answer.. Thank you very much Muru you have solved my query. Nov 9, 2016 at 4:51
1

What about just get everything between 'com' & 'www'? And print them by 'sed'

sed -ne 's/com\.\(.*\)\.www\(.*\)/www.\1.com/p' infile

This will ignore two lines of your input sample which has no 'www'.

4
  • I don't know why I am getting this error when I use your command--- 'sed: -e expression #1, char 31: unknown option to `s' Nov 9, 2016 at 4:54
  • sorry updated... Nov 9, 2016 at 5:01
  • Still the same error---sed: -e expression #1, char 38: unknown option to `s' Nov 9, 2016 at 5:06
  • What is your sed version? For me it's working fine, maybe try like this sed -ne 's/com\.\(.*\)\.www\(.*\)/www\.\1\.com/p' infile Nov 9, 2016 at 9:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .