0

I want to find the number of times a webpage used a specific word or phrase at various times over the past several years; and I would like to do this using archive.org's results page as the source of the page over time.

I'm fairly certain there is a tool of some kind that can do a word search of a webpage, and I know that archive.org has all of the pages readily available...

but that's as far as I get. I am fairly adept at some quick-and-dirty bash scripts and a little python as well, but I've got no experience with directly accessing the web. The Internet. Whatever. (See the level of my ignorance?!)

I guess what I want to know is: What would be a good CLI tool for word-counting any URL that is passed to it, and how would I go about writing a script that could return the URL of ~750 sequentially-dated pages? Or - is that even possible?

I'm certain there's somebody that could kludge this together in a matter of seconds rather than spoon-feed me a three-paragraph "how to" manual so I can do this one time, but I'd still at least like to see how it would be done.

archive.org has multiple saves per day, and not always the same number. So I'd like to know if there is a way to interrogate the 'results' page, follow each url and check the creation date, and save every url that returns a new date into some text file. This would allow my to build a list of sequentially-dated webpages for step 2...

If such a tool exists that can look up a url, do a grep (or similar) for some text b, and return the number of b that it finds, run that and pipe the output to a textfile. I use a few cli tools that can do that kind of gymnastics on a local drive, but I was hoping there would be a program that could just do:

for xurl in somefile.txt; do wordcount -a=$xurl -b="searchstring" >> temp.txt; done

  • You can use grep. grep -c will provide the count of matching lines, and not print normal output. An other alternative may be -o and wc – vidarlo Oct 20 '17 at 16:01
  • OK, thanks for that... I've not tried grep on a webpage, but it looks to be pretty straightforward. – rpdayton Oct 20 '17 at 16:16
  • So that just leaves... how can I interrogate archive.org for a list of all archived incarnations of... for example, wikipedia's "Texas" article? Tunneling through the redirects to get to the actual pages on archive.org is way above my skill level, or at least beyond my current knowledge. – rpdayton Oct 20 '17 at 16:24
1

you can curl the url and pipe it to grep like this

curl --silent https://www.google.cl | grep -c -i google

Also note the -i, that's a case-insensitive search.

To do it in batch from a file (one-url-per-line)

wordToSearch='google'
#
while read url; do
    curl --silent "${url}" | grep -c -i "${wordToSearch}" >> temp.txt
done < fileWithUrl.txt
1

I can’t see what the input to your program or the “results” page you mention would look like, but here’s how I would try to solve this:

curl allows you to download the source code of a web page. So curl archive.org/whatever gets you the results page in plain text.

You can pipe this into grep and search for HTML’s anchor tags (<a href="location.of/the-linked-page">a description of the link</a>). Here’s a grep invocation which may serve as a starting point: | grep -o '<a href.*</a> (-o prints only the matched parts of the string. You might need to replace the space between a and href with the space class or experiment with non-greedy forms of *, depending on your input.) You will also retrieve all links on that page, so another (or multiple) round(s) of grepping are necessary to remove the navigation links, ads, etc.

Afterwards, using sed, you remove the “clutter” like <a>, href, etc. to retrieve only the pure link. You can then feed the output into a for loop which again uses curl to retieve these links’ contents. On that output, finally, you can perform your word analysis using grep -c, as vidarlo mentions.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.