I want to find the number of times a webpage used a specific word or phrase at various times over the past several years; and I would like to do this using archive.org's results page as the source of the page over time.
I'm fairly certain there is a tool of some kind that can do a word search of a webpage, and I know that archive.org has all of the pages readily available...
but that's as far as I get. I am fairly adept at some quick-and-dirty bash scripts and a little python as well, but I've got no experience with directly accessing the web. The Internet. Whatever. (See the level of my ignorance?!)
I guess what I want to know is: What would be a good CLI tool for word-counting any URL that is passed to it, and how would I go about writing a script that could return the URL of ~750 sequentially-dated pages? Or - is that even possible?
I'm certain there's somebody that could kludge this together in a matter of seconds rather than spoon-feed me a three-paragraph "how to" manual so I can do this one time, but I'd still at least like to see how it would be done.
archive.org has multiple saves per day, and not always the same number. So I'd like to know if there is a way to interrogate the 'results' page, follow each url and check the creation date, and save every url that returns a new date into some text file. This would allow my to build a list of sequentially-dated webpages for step 2...
If such a tool exists that can look up a url, do a grep (or similar) for some text b, and return the number of b that it finds, run that and pipe the output to a textfile. I use a few cli tools that can do that kind of gymnastics on a local drive, but I was hoping there would be a program that could just do:
for xurl in somefile.txt; do wordcount -a=$xurl -b="searchstring" >> temp.txt; done