-2

I have tried the script to store the domains with proper syntax. But I didn't got the proper output. Here is what I did and what I got and also this command got stuck:

while read -r url; do noprefix=${url#*//}; nosuffix=${noprefix%%/*}; echo "$nosuffix"; done < FinaleOutput.txt  > First.txt

msn.com.jp
msn.com.jp
msn.comjswartout
msn.comjubas
msn.com.junk
msn.comk
msn.com...kugoo
msn.comlamer6
msn.comlcrane
msn.comleonardpj49
msn.comlinkbobber
msn.comm
msn.comm.

How I can write a validating command which will give me the proper domain as per syntax.

Example of the output required:

google.com.tk
www.google.co.uk.se
google.co.au
www.google.co.uk.se
m.google.com
www.google.com

If the input is something like this:

google.com.tkmn/1/2/3/
www.google.co.uk.seas
google.co.au___travel
www.google.co.uk.se/1/2/
m.google.com--tour
http://www.google.com/au
  • 2
    What if your input is bad.domain.com___real.domain.com or something? Where are these things coming from? As I understand from your other questions, you are crawling the web for them so, presumably, have already accessed the domain. Why don't you keep the domains at that point instead of trying to parse this? – terdon Feb 1 '17 at 13:39
  • 1
    Pick random regex from stackoverflow.com/q/201323/2072269, use only the part for validating domains. – muru Feb 1 '17 at 13:40
  • @muru I am not checking emails. I am checking domains. I have list of domains. – Jaffer Wilson Feb 1 '17 at 14:22
  • I didn't say you were. I did say: use only the part for validating domains – muru Feb 1 '17 at 14:24
2

The following script operates on the basis of removing trailing characters and verifying resulting string against nslookup queries of your DNS server:

#!/bin/bash

find_correct_domain(){
    local string="$1"
    while ! nslookup "$string" &> /dev/null
    do
        string="${string%?}"
    done
    echo "$string"
}

main(){

    local new_line=""
    while IFS= read -r line
    do
        case "$line" in
            http*) new_line=${new_line##*/} ;;
            *) new_line="$line";;
        esac
        find_correct_domain "$new_line"
    done < "$1"
}
main "$@"

Using the input file provided by OP, resulting output is as follows:

$ ./format_domains.sh input.txt               
google.com.
www.google.co.uk.se
google.co.
www.google.co.uk.se
m.google.com
m.google.com

Notice the trailing dot at the end of google.com. and google.co. DNS queries (which is what nslookup performs ) all terminate in a dot unless you do so yourself, hence google.com. is a valid DNS query (as per RFC 1034). Refer to this article for more info: http://www.dns-sd.org/trailingdotsindomainnames.html

If the trailing dot is not desirable, you can add simple case statement into the find_correct_domain function to trim that out, like so:

find_correct_domain(){
    local string="$1"
    while ! nslookup "$string" &> /dev/null
    do
        string="${string%?}"
    done
    case "$string" in
        *.) string="${string%?}";;
    esac
    echo "$string"
}

Of course, keep in mind that this script isn't optimized for performance: it is done in bash and performs nslookup on all items so that's probably O(n2) type of run time, which means that if you have a big list, it's probably will run slow, but for relatively small lists it's fine.

  • Thank you for your answer. Let me try it. Also can I skip that part of nslookup to get only strings as output? – Jaffer Wilson Feb 2 '17 at 6:06
  • 1
    getent hosts might be better than nslookup, depending on the use case. getent hosts resolves hostnames the way other applications in your system, does, using /etc/hosts, DNS servers and any other database configured in nsswitch.conf. – muru Feb 2 '17 at 6:08
  • I was looking for the syntax checking modification in my command or any other way to check valid syntax of domain and that too fastest way. I think this method will take a lot of time. I have a text file containing many URLs and I cannot wait longer to get output. I need to put it on another process for HTTP status code identification. Will nslookup consume my bandwith. I want to save it too. – Jaffer Wilson Feb 2 '17 at 6:16
  • Yeah, like I said in last paragraph, this isn't the speediest method. Potentially, regex expressions could be used to trim down the domains, but it would probably be a bit complex to cover many cases. So, it's either speed or complexity that you have to choose. – Sergiy Kolodyazhnyy Feb 2 '17 at 6:23
  • @Serg It your script going to consume a lot of bandwidth for a larger list. I am ready to wait but need perfoect output. But do not like to consume a lot of bandwidth, as there are other people on work too. – Jaffer Wilson Feb 2 '17 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.