3

I would like to know whether is it possible to retrieve the technologies used to build a website just by having its URL using Ubuntu.
Say for example: if I have the URL:
https://www.wikipedia.org/

I want to know the technologies used to build this website.
The output would be :

PHP, HHVM, Vanish, AddThis and many others.

Is there any way to get this done faster?
Please, also remember that I have a file of list of website and I want to extract the web technologies of those websites and place them in a file just after the URL (Line by Line). Kindly, let me know if this is possible using Ubuntu command or any software on Ubuntu.

  • Please if anyone has any solution for my query. do let me know. – Jaffer Wilson Nov 11 '16 at 4:38
  • 3
    No it's not possible with a single command at least. You can use various web services like builtswith instead – Anwar Nov 11 '16 at 4:45
  • @Anwar If it would have possible with a single command. I would have attempted that already. It is not possible, hence I am here to find a solution. If you have any please let me know. will be delighted to hear from you soon. – Jaffer Wilson Nov 11 '16 at 6:55
  • 1
    I personally think this is out of the scope of Ask Ubuntu. It's more related with programming because I believe you need to setup a complex environment to do so. What you use as OS is irrelevant in this case afaik. – Anwar Nov 11 '16 at 9:36
1

You may look into Kali or Parrot distributions for information collecting tools.

  • nikto is one of them that I'd tried before and gives partial info. It is available is Ubuntu repository too.

    ~$ whatis nikto
    nikto (1)            - Scan web server for known vulnerabilities
    
    ~$ sudo apt-get install nikto
    ~$ sudo nikto -update
    ~$ nikto -Tuning b -h www.wikipedia.org
    - Nikto v2.1.5
    ---------------------------------------------------------------------------
    + Target IP:          91.198.174.192
    + Target Hostname:    www.wikipedia.org
    + Target Port:        80
    + Start Time:         2016-11-14 09:22:30 (GMT1)
    ---------------------------------------------------------------------------
    + Server: Varnish
    + IP address found in the 'x-client-ip' header. The IP is "105.107.105.185".
    + The anti-clickjacking X-Frame-Options header is not present.
    + Uncommon header 'x-client-ip' found, with contents: 105.107.105.185
    + Uncommon header 'x-cache' found, with contents: cp3041 int
    + Uncommon header 'x-varnish' found, with contents: 827655138
    + Uncommon header 'x-cache-status' found, with contents: int
    + Root page / redirects to: https://www.wikipedia.org/
    + No CGI Directories found (use '-C all' to force check all possible dirs)
    + Server banner has changed from 'Varnish' to 'mw1187.eqiad.wmnet' which may suggest a WAF, load balancer or proxy is in place
    + Cookie GeoIP created without the httponly flag
    + Retrieved via header: 1.1 varnish-v4, 1.1 varnish-v4, 1.1 varnish-v4
    + Retrieved x-powered-by header: HHVM/3.3.0-static
    + Server leaks inodes via ETags, header found with file /, fields: 0xW/3b2 0x5369720eefb07 
    + Uncommon header 'x-analytics' found, with contents: nocookies=1
    + Uncommon header 'backend-timing' found, with contents: D=236 t=1478774110870502
    + 269 items checked: 0 error(s) and 12 item(s) reported on remote host
    + End Time:           2016-11-14 09:23:21 (GMT1) (51 seconds)
    ---------------------------------------------------------------------------
    + 1 host(s) tested
    
  • whatweb is another tool. It has non fixed bug (invalid multibyte escape error) in Ubuntu.

    1. Open file of encoding auto-detection library for editing

      sudo nano /usr/lib/ruby/vendor_ruby/rchardet/universaldetector.rb
      
    2. Add # encoding: US-ASCII


    Even with this workaround above, the output is not clean as in Kali.

    ~$ whatis whatweb
    whatweb (1)          - Web scanner to identify what websites are running.
    
    ~$ whatweb www.wikipedia.org
    /usr/share/whatweb/lib/tld.rb:85: warning: key "2nd_level_registration" is duplicated and overwritten on line 85
    /usr/share/whatweb/lib/tld.rb:93: warning: key "2nd_level_registration" is duplicated and overwritten on line 93
    /usr/share/whatweb/lib/tld.rb:95: warning: key "2nd_level_registration" is duplicated and overwritten on line 95
    /usr/share/whatweb/plugins/wordpress.rb:436: warning: key "2.7-beta1" is duplicated and overwritten on line 453
    /usr/share/whatweb/lib/extend-http.rb:102:in `connect': Object#timeout is deprecated, use Timeout.timeout instead.
    http://www.wikipedia.org [301] Cookies[WMF-Last-Access], Country[NETHERLANDS][NL], HTTPServer[Varnish], HttpOnly[WMF-Last-Access], IP[91.198.174.192], RedirectLocation[https://www.wikipedia.org/], UncommonHeaders[x-varnish,x-cache-status,x-client-ip], Varnish
    /usr/share/whatweb/lib/extend-http.rb:102:in `connect': Object#timeout is deprecated, use Timeout.timeout instead.
    /usr/share/whatweb/lib/extend-http.rb:140:in `connect': Object#timeout is deprecated, use Timeout.timeout instead.
    https://www.wikipedia.org/ [200] Cookies[GeoIP,WMF-Last-Access], Country[NETHERLANDS][NL], Email[Wikipedia-logo-v2@1.5x.png,Wikipedia-logo-v2@2x.png,Wikipedia_wordmark@1.5x.png,Wikipedia_wordmark@2x.png,sprite-bookshelf_icons@1.5x.png,sprite-bookshelf_icons@2x.png,sprite-project-logos@1.5x.png,sprite-project-logos@2x.png], HTML5, HTTPServer[mw1253.eqiad.wmnet], HttpOnly[WMF-Last-Access], IP[91.198.174.192], probably MediaWiki, Script, Title[Wikipedia], UncommonHeaders[backend-timing,x-varnish,x-cache-status,strict-transport-security,x-analytics,x-client-ip], Varnish, Via-Proxy[1.1 varnish-v4, 1.1 varnish-v4, 1.1 varnish-v4], X-Powered-By[HHVM/3.3.0-static]
    

    Output from Kali:

    ~# whatweb https://www.wikipedia.org
    https://www.wikipedia.org [200 OK] Cookies[GeoIP,WMF-Last-Access], Country[NETHERLANDS][NL], Email[Wikipedia-logo-v2@1.5x.png,Wikipedia-logo-v2@2x.png,Wikipedia_wordmark@1.5x.png,Wikipedia_wordmark@2x.png,sprite-bookshelf_icons@1.5x.png,sprite-bookshelf_icons@2x.png,sprite-project-logos@1.5x.png,sprite-project-logos@2x.png], HTML5, HTTPServer[mw1253.eqiad.wmnet], HttpOnly[WMF-Last-Access], IP[91.198.174.192], probably MediaWiki, Script, Strict-Transport-Security[max-age=31536000; includeSubDomains; preload], Title[Wikipedia], UncommonHeaders[backend-timing,x-varnish,x-cache-status,x-analytics,x-client-ip], Varnish, Via-Proxy[1.1 varnish-v4, 1.1 varnish-v4, 1.1 varnish-v4], X-Powered-By[HHVM/3.3.0-static]
    
| improve this answer | |
  • Helpful .. but still not what I am looking for. Thank you for your answer. – Jaffer Wilson Nov 14 '16 at 11:12
  • @JafferWilson, :/ Almost same info from whatweb www.wikipedia.org if you may want to try it. – user.dz Nov 14 '16 at 14:04
  • Please add this option to your answer also. I would like to know what you answer you have got. As I cannot see the output on my screen . May be some problem from my end. – Jaffer Wilson Nov 15 '16 at 4:27
  • @JafferWilson, I forget to mention that version of Ubuntu have some unfixed bug. Probably same case for you, I'm using Xubuntu 16.04 (last update). I have update the answer and included the output from my xubuntu and a kali virtualbox – user.dz Nov 15 '16 at 8:25
5
+25

I'd suggest that this is not really possible just with Ubuntu.

You can pull in data with a terminal browser like Lynx or a command like curl, but parsing it will be extremely time consuming.

This is the sort of thing you would need an extensive script to accomplish. This would be a project of its own I think.

I've done some research, and similar questions have been asked before, but they are very few and far between.

You could use an online tool like https://builtwith.com to look up the details.

For example, it shows that techcrunch.com

  • Uses NGINX as a webserver
  • Uses SSL certs from Godaddy; Wordpress
  • Shows an SSL Redirect
  • Wordpress DNS
  • Uses Postmark; Sailthru; Google Apps for Business and has SPF setup
  • Is hosted by Wordpress.com
  • Uses Wordpress VIP as it's CMS
  • Uses a PHP interpreter
  • Has a ton of analytics; advertisers and tracking
  • Uses a lot of JS Libraries (all listed on the site) including things like jQuery and Backbone.js
  • Uses AOL On; Tube Mogul and TidalTV for media

There is a lot of information there, you'll have to read through it yourself, but it's a great tool.

Hope that helps!

Good luck :)

This may also be a duplicate question of https://stackoverflow.com/questions/396739/how-do-you-determine-what-technology-a-website-is-built-on

There are other online tools listed in the above question as well.

| improve this answer | |
  • Thank you for your answer. but this is not what exactly I am expecting in terms of bounty answer. please can you add more references to your answer so I will be satisfied. – Jaffer Wilson Nov 11 '16 at 6:53
  • It's ok. I'm not really interested in the bounty. Just trying to be helpful. There are honestly a ton of referenced sites and tools in the duplicate question I listed. Well worth looking into :) – ThatGuy Nov 11 '16 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.