I have a file of URLs, in the format as shown below:
com.blendtuts/S
°=
com.blengineering.www/:http
±=
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²=
com.blenheimsi
³=
com.blenkov.www/page/media/18/34/376
´=
com.blentwell.www/bookmarks.php/jackroldan/sp
¸=
com.blentwell.www/tags.php/I
The file size is in GigaBytes. Say around 250 GB of the file size.
I was trying to reverse the words in the file and extract only the domains from the text. I tried to make it using Ubuntu OS terminal commands. Let me tell you what I have tried:
First I removed the data after “/” using the following command:
~$ ex -sc '%s/\(\/\).*/\1/ | x' newfile.txt > ddm.txt
And the result as:
com.blendtuts/
°=
com.blengineering.www/
±=
com.blenheimgang.www/
²=
com.blenheimsi
³=
com.blenkov.www/
´=
com.blentwell.www/
¸=
com.blentwell.www/
Now I reversed the complete text in the file using the solution from : https://stackoverflow.com/questions/40467918/how-to-reverse-the-word-in-ubuntu
And got the following result:
/blendtuts.com
°= /www.blengineering.com
±= /www.blenheimgang.com
²= blenheimsi.com
³= /www.blenkov.com
µ= /www.blentwell.com
¶= /www.blentwell.com
•= /www.blentwell.com
/www.blentwell.com
But still the problem is not solved. I would like to how it is possible to extract URLs and put them into another file using Ubuntu. As you can see above the output what still I have is not the domain, it has a backslash with it.
If there is another solution to such a problem using any other operating system, do let me know. I prefer to go with Ubuntu.
I would like to extract domains out of the file and separate them to another file and that too in a proper format.
If I get the unique domain then it will be an excellent solution to my query. Otherwise, I am using command as:
$ sort filename.txt | uniq > save_to_file.txt
Please try not to give me a solution using awk command, as it does not work on my system.
Sample data:
com.blendschutzrollo.www/d_chefsessel6_Maxx_Chefsessel_mit_Kopfstutze_Chefdrehsessel___Munchen__374
¯=
com.blendtuts/S
°=
com.blengineering.www/:http
±=
com.blenheimgang.www/le-porsche-museum-en-details/porsche-museum-3
²=
com.blenheimsi
³=
com.blenkov.www/page/media/18/34/376
´=
com.blenoir.www/lat
µ=
com.blentwell.www/bookmarks.php/bashment%20jack/re
¶=
com.blentwell.www/bookmarks.php/djcable/rt
·=
com.blentwell.www/bookmarks.php/jackroldan/sp
¸=
com.blentwell.www/tags.php/I
¹=
com.blentwell.www/tags.php/eurot
º=
com.blentwell.www/tags.php/mitarbeiters
»=
com.blentwell.www/tags.php/verw
¼=
com.blenzblog/tag/olympic-w
½=
com.blepharoplastyusa.www/albany-n
¾=