Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

How can I, as an Ubuntu beginner, get a daily snapshot of a webpage? The page requires a log in and then displays a bunch of numbers that I'm interested in.

Can I write a script to take care of this?

(I am on a waiting list for college dorm housing and want to keep a close eye on the progression. There's a long list of daily updated info that I would like to store somewhere I can easily flip through it.)

Edit: so I used Rinzwinds script and did some research on my own, here's what I have so far:

    #!/bin/bash
   filename="kkik"
   url="https://www.kollegierneskontor.dk/default.aspx?func=kkikportal.housingrequests&lang=GB"
   date=` date +%d%m%Y`
   output="$filename-$date.html"
   wget --save-cookies cookies.txt --post-data 'Page_ctl05_Main_ctl04_form_loginUserName=*****&Page$ctl05$Main$ctl04$form$loginPassword=******'
   wget  --load-cookies cookies.txt  $url -O $output

it successfully downloads the 'you don't have permission' page (if I take out the cookies and post stuff it gives me a 'you need to log in' page, which is different). Am I missing anything obvious? It doesn't seem like an overly complex website.

I'm sure if I Get the script to run, I'll figure out how to make a CRON job out of it.

share|improve this question
    
When you say "snapshot" do you mean like a screenshot or just a time-localised copy of the data? –  Oli Jun 21 '11 at 13:14
    
either would be fine, but time-localised copy is more than enough. –  joon Jun 22 '11 at 19:16
1  
it depends on what you need to do to get to the website (and my danish is very rusty ;) so it will be hard for me to explain) –  Rinzwind Jun 22 '11 at 19:37
    
In a normal browser, when I click 'Ansogning', I need to login (Username and password). I looked into the source and it's a normal 'input type="text"' thing. I took the names/id's off of them and put them in the --post-data section... Right now it's giving me session time out (a third different error page) after adding --keep-session-cookies. –  joon Jun 22 '11 at 19:52
add comment

3 Answers

1. Downloading a snapshot of a website

You can download the page with some simple commands. Does it require loging in each time? That might require some extra coding but this is a general script that downloads a website (and appends a date to the download)

   #!/bin/bash
   filename="askubuntu"
   url="www.askubuntu.com"
   date=` date +%d%m%Y`
   output="$filename-$date.html"
   wget $url -O $output

Result:

im1

askubuntu-210611.html can be viewed with a browser or a text viewer like gedit. This can even be altered to add in some sed/grep commands to filter everything that is html out of the result and just save the data you require. Makes it easy to add the data to libreoffice calc for instance.

If you add it to a cron script you can have it do this every x minutes/hours.

2. Making several screenshots of a website

Shutter Install shutter can make images from command line with

   shutter --window=.*firefox.*

This will make a screenshot of an instance named firefox and save it with a new name in the folder you save your shutter screenshots. Add this to a script and have it execute once every time you want and you can have screenshots of your website. As long as you refresh the page with an add-on like refresh every you are set to go.

Shutter also is capable of making an image of a part of the screen with

    shutter -s=100,100,300,300 -e

or capture a screenshot of a website (probably not useable for you if you need to login every time):

    shutter --web=[URL]
share|improve this answer
1  
I would use #1 of Rinzwind's and set it up in cron. –  wojox Jun 21 '11 at 14:53
    
Bugreport and fix on shutter --web not working: bugs.launchpad.net/ubuntu/+source/shutter/+bug/793125 –  Rinzwind Jun 22 '11 at 19:42
add comment

Not sure if it is a solution for you because it does not address all your requirements. However with the ScrapBook Firefox extension you can create snapshots of websites easily. Each time you grab a snapshot it creates an entry in a list sorted by date and you can easily go back a look at any version.

Because you need to login to the site, ScrapBook extension is great because you can login first and then take the snapshot as a logged user.

The only thing that it will not do is automated snapshots because you will need to navigate to the site and save the snapshot manually.

Hope this helps.

share|improve this answer
    
the automation is kind of the point. I can copypaster/save webpage/printscreen manually, but since I want it daily and have very little discipline, I'm bound to forget ;). I'll check it out though! –  joon Jun 22 '11 at 18:51
add comment

@joon I guess you need xdotool.

Xdotool lets you simulate keyboard input and mouse activity, move and resize windows, and much more. Take a check. With this you can do your autentication in the classic "gui" style, then your code can take the screenshot and whatever... I hope this helps you on your way.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.