3

I have a file of URLs, in the format as shown below:

com.blendtuts/S
°=
com.blengineering.www/:http
±=
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²=
com.blenheimsi
³=
com.blenkov.www/page/media/18/34/376
´=
com.blentwell.www/bookmarks.php/jackroldan/sp
¸=
com.blentwell.www/tags.php/I

The file size is in GigaBytes. Say around 250 GB of the file size.

I was trying to reverse the words in the file and extract only the domains from the text. I tried to make it using Ubuntu OS terminal commands. Let me tell you what I have tried:

First I removed the data after “/” using the following command:

~$ ex -sc '%s/\(\/\).*/\1/ | x' newfile.txt > ddm.txt

And the result as:

com.blendtuts/
 °=
com.blengineering.www/
±=
com.blenheimgang.www/
²=
com.blenheimsi
³=
com.blenkov.www/
´=
com.blentwell.www/
¸=
com.blentwell.www/

Now I reversed the complete text in the file using the solution from : https://stackoverflow.com/questions/40467918/how-to-reverse-the-word-in-ubuntu

And got the following result:

    /blendtuts.com
    °= /www.blengineering.com
    ±= /www.blenheimgang.com
    ²= blenheimsi.com
    ³= /www.blenkov.com
    µ=  /www.blentwell.com
    ¶=  /www.blentwell.com
    •=  /www.blentwell.com

/www.blentwell.com

But still the problem is not solved. I would like to how it is possible to extract URLs and put them into another file using Ubuntu. As you can see above the output what still I have is not the domain, it has a backslash with it.

If there is another solution to such a problem using any other operating system, do let me know. I prefer to go with Ubuntu.

I would like to extract domains out of the file and separate them to another file and that too in a proper format.

If I get the unique domain then it will be an excellent solution to my query. Otherwise, I am using command as:

$ sort filename.txt | uniq > save_to_file.txt

Please try not to give me a solution using awk command, as it does not work on my system.

Sample data:

com.blendschutzrollo.www/d_chefsessel6_Maxx_Chefsessel_mit_Kopfstutze_Chefdrehsessel___Munchen__374
¯= 
com.blendtuts/S
°= 
com.blengineering.www/:http
±= 
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²= 
com.blenheimsi
³= 
com.blenkov.www/page/media/18/34/376
´= 
com.blenoir.www/lat
µ= 
com.blentwell.www/bookmarks.php/bashment%20jack/re
¶= 
com.blentwell.www/bookmarks.php/djcable/rt
·= 
com.blentwell.www/bookmarks.php/jackroldan/sp
¸= 
com.blentwell.www/tags.php/I
¹= 
com.blentwell.www/tags.php/eurot
º= 
com.blentwell.www/tags.php/mitarbeiters
»= 
com.blentwell.www/tags.php/verw
¼= 
com.blenzblog/tag/olympic-w
½= 
com.blepharoplastyusa.www/albany-n
¾= 
  • 2
    "Please try not to give me a solution using awk command, as it does not work on my system." I'm pretty sure awk works on ubuntu, at least on my box. – user12753 Nov 9 '16 at 11:39
6

A Perl solution, adapting one of the string reversal solutions:

$ perl -F/ -anle 'print reverse(split("([^.]*)", $F[0])) if /\./' input
www.blendschutzrollo.com
blendtuts.com
www.blengineering.com
www.blenheimgang.com
blenheimsi.com
www.blenkov.com
www.blenoir.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
www.blentwell.com
blenzblog.com
www.blepharoplastyusa.com

The arguments:

  • -F/ -a creates an array F out of each line of input, splitting on /.
  • -nle runs the expression (-e <expr>) on each line of input, without automatically printing (-n), while handling the newline at the end of each line (-l)
  • The line is already split on /, and we only need the part before the first /, so the first element of array F: $F[0]. Then we split it on . and reverse each of those, and print if the line contains ..

Now you can sort -u this.

  • Excellent answer.. Thank you very much Muru you have solved my query. – Jaffer Wilson Nov 9 '16 at 4:51
1

What about just get everything between 'com' & 'www'? And print them by 'sed'

sed -ne 's/com\.\(.*\)\.www\(.*\)/www.\1.com/p' infile

This will ignore two lines of your input sample which has no 'www'.

  • I don't know why I am getting this error when I use your command--- 'sed: -e expression #1, char 31: unknown option to `s' – Jaffer Wilson Nov 9 '16 at 4:54
  • sorry updated... – αғsнιη Nov 9 '16 at 5:01
  • Still the same error---sed: -e expression #1, char 38: unknown option to `s' – Jaffer Wilson Nov 9 '16 at 5:06
  • What is your sed version? For me it's working fine, maybe try like this sed -ne 's/com\.\(.*\)\.www\(.*\)/www\.\1\.com/p' infile – αғsнιη Nov 9 '16 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.