5

Is there a way in Ubuntu to find all the directories in a website?

I have a website, and I want to check the internal links (directories) of that website.

Something like this:

Enter image description here

...

The problem with this website is when I enter something like ubuntu.com/cloud, it doesn't show the subdirectories.

8
  • @pa4080 Because the two questions are not identical, the answers to the duplicate link do not really answer this question.
    – karel
    Apr 5, 2018 at 11:34
  • @karel, I've retracted my close vote.
    – pa4080
    Apr 5, 2018 at 12:11
  • Thank you very much. Now all I have to worry about is leech close voters and eventually I will delete my two comments when they become obsolete.
    – karel
    Apr 5, 2018 at 12:27
  • 2
  • 2
    @pa4080 Cast his close vote, Karel dissented, pa4080 retracted, I came back in tonight to retract my own only to discover DavidFoerster has cast the very same close vote. Are we keeping open or closing. I'm confused now... FTR I do my own Stack Exchange web-scraping using bash: askubuntu.com/questions/900319/… As I only scrape my own AU answers and compare them to local disk answer scraping hits are less than 100. Apr 6, 2018 at 1:06

2 Answers 2

9

Open the terminal and type:

sudo apt install lynx  
lynx -dump -listonly -nonumbers "https://www.ubuntu.com/" | uniq -u 

This command improves upon the previous command by redirecting the output to a text file named links.txt.

lynx -dump "https://www.ubuntu.com/" | awk '/http/{print $2}' | uniq -u > links.txt
3
  • I trust this won't get you banned like the comment by pa4080 below the question when referencing the link: askubuntu.com/questions/991447/… Apr 5, 2018 at 11:03
  • 1
    I'm not spidering a multiple page website, only returning the links on one single webpage. Assuming the extremely improbable case of a poorly designed single webpage that contains 100,000 links (how would it be possible to load such a page to begin with) I suppose that lynx would try to return all the links until it finished executing the command or until the terminal froze.
    – karel
    Apr 5, 2018 at 13:04
  • 1
    Actually I've run very complicated backup scripts from the terminal involving hundreds of gigabytes of files transfered, and nothing froze until the backup command completed successfully. So I think that lynx would perform successfully even in this extreme case.
    – karel
    Apr 5, 2018 at 13:09
5

See this answer from superuser.com:

wget --spider -r --no-parent http://some.served.dir.ca/
ls -l some.served.dir.ca

There are free websites which will do this for you and convert the output to xml format though. I suggest you look into one of those as well to see which method is more suitable for your needs.

Edit OP has included a new screenprint

1
  • 1
    You can add also depth of the recursion - for example to the second level: wget --spider -r -l2.
    – pa4080
    Apr 5, 2018 at 11:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.