Take the 2-minute tour ×
Ask Ubuntu is a question and answer site for Ubuntu users and developers. It's 100% free, no registration required.

I'm looking for a web spider for Ubuntu like this one Webripper - Calluna Software. You can download a whole site like you can with

wget -r -m example.com

but the feature I'm looking for is you can input a search term like "Linux" and it searches the Web and downloads them. Are there any programs on Ubuntu like this?

share|improve this question

4 Answers 4

up vote 4 down vote accepted

You could use Google Alerts to create a sort of search page delivered to a feed and then use an RSS reader or Thunderbird to read them.

I use Thunderbird for RSS. I don't know if there are any RSS readers who could export the feed to simple html.

share|improve this answer

Give httrack (CLI) or webhttrack (web interface) a shot, it's in the universe repo. I'm not sure about the search-term-feature you describe, but it does offer a bunch of easily configurable options.

http://packages.ubuntu.com/de/oneiric/webhttrack

HTTrack Website Copier - Free Software Offline Browser (GNU GPL)

share|improve this answer

You can give http ripper a try.

Here are some features posted on the website:

Free Software (GPL 3)
Generic (works with almost every website)
Runs on GNU/Linux and Windows
Nearly undetectable / blockable by servers
Built with python and pygtk

http://29a.ch/httpripper/screenshots/2.png Screen-shot

Watch a tutorial, from the developer of httpripper himself:

Download link:

It worked for me under Ubuntu 11.10 x64

share|improve this answer

There are adequate modules at Perl's CPAN. You just need a little bit of perl scripting.

In particular, take a look at the WWW:Mechanize module at the WWW:Mechanize module.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.