Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I want to write a python program to download the contents of a web page, and then download the contents of the web pages that the first page links to.

For example, this is main web page http://www.adobe.com/support/security/, and the pages I want to download: http://www.adobe.com/support/security/bulletins/apsb13-23.html and http://www.adobe.com/support/security/bulletins/apsb13-22.html

There is a certain condition I want to meet: it should download only web pages under bulletins not under advisories(http://www.adobe.com/support/security/advisories/apsa13-02.html)

 #!/usr/bin/env python
 import urllib
 import re
 import sys
 page = urllib.urlopen("http://www.adobe.com/support/security/")
 page = page.read()
 fileHandle = open('content', 'w')
 links = re.findall(r"<a.*?\s*href=\"(.*?)\".*?>(.*?)</a>", page)
 for link in links:
 sys.stdout = fileHandle
 print ('%s' % (link[0]))
 sys.stdout = sys.__stdout__
 fileHandle.close() 
 os.system("grep -i '\/support\/security\/bulletins\/' content >> content1") 

I've already extracted the link of bulletins into a content1, but don't know how to download the content of those web pages, by providing content1 as input.

The content1 file is as shown below:- /support/security/bulletins/apsb13-23.html /support/security/bulletins/apsb13-23.html /support/security/bulletins/apsb13-22.html /support/security/bulletins/apsb13-22.html /support/security/bulletins/apsb13-21.html /support/security/bulletins/apsb13-21.html /support/security/bulletins/apsb13-22.html /support/security/bulletins/apsb13-22.html /support/security/bulletins/apsb13-15.html /support/security/bulletins/apsb13-15.html /support/security/bulletins/apsb13-07.html

share|improve this question

2 Answers 2

up vote 2 down vote accepted

If I understood your question, the following script should be what you want:

#!/usr/bin/env python

import urllib
import re
import sys
import os
page = urllib.urlopen("http://www.adobe.com/support/security/")
page = page.read()
fileHandle = open('content', 'w')
links = re.findall(r"<a.*?\s*href=\"(.*?)\".*?>(.*?)</a>", page)
for link in links:
    sys.stdout = fileHandle
    print ('%s' % (link[0]))
sys.stdout = sys.__stdout__
fileHandle.close() 
os.system("grep -i '\/support\/security\/bulletins\/' content 2>/dev/null | head -n 3 | uniq | sed -e 's/^/http:\/\/www.adobe.com/g' > content1")
os.system("wget -i content1")
share|improve this answer
    
Thank you, it worked,can you modify the code so that i can store the content of each html file in another file(STORE ALL PAGES CONTENT IN ONE FILE) –  Kummi_10 Sep 26 '13 at 8:10
    
@Kummi_10 Can you clarify? First you say "each html file in another file", and then you say "ALL PAGES CONTENT IN ONE FILE". This is contradictory. –  Radu Rădeanu Sep 26 '13 at 8:20
    
@ Radu Rădeanu I want to retrieve the content of those pages into one file(but your code downloading all those pages to pwd) –  Kummi_10 Sep 26 '13 at 8:27
    
@Kummi_10 You mean one directory? Then use wget -P directory -i content1 . See man wget for more info. –  Radu Rădeanu Sep 26 '13 at 8:47
    
Thank you very much,yesterday i posted this on stackoverflow but nobody undestood my qsn.... But you solved my problem,once again THANK YOU VERY MUCH –  Kummi_10 Sep 26 '13 at 10:24

Probably this question is for stackoverflow!

But anyways you can look in HTTrack for this it does similar kind of operation and moreover its opensource

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.