7

I need a bash command to delete the entire file if the file itself begins with <html>.

I'm not sure the best way to go about this...

Context: I download a series of files via curl requests. Most time the downloads and processing work fine. But other times the download request results in a 404 for whatever reason. When I get those, the contents of the downloaded file begins with a html tag. When the rest of my processing hits this file, it hangs. So I want to run a command prior to my other processing to cat each of the files and delete the file if it has this html tag.

|improve this question
  • You want to search for files beginning with <html> or delete a file that has <html> in its name? – Seth Oct 28 '14 at 4:17
  • Yes. I added an additional comment – BeMy Friend Oct 28 '14 at 4:20
  • @Seth I don't want to edit a title when a mod has done so already, but the title seems wrong to me. Files are deleted the same way regardless what it starts with. The question is how to identify files matching a certain condition, not how to delete them. – user50849 Oct 28 '14 at 9:15
  • 1
    Do note that many HTML files have their start tag preceded by a <!doctype something>. – Kroltan Oct 28 '14 at 10:58
  • Is a python script acceptable? It is much easier to write and read than bash magic. – Siyuan Ren Oct 28 '14 at 15:23
20

To address the question that prompted you to ask this one, rather than the one you actually asked:

curl can tell you the status code in addition to downloading the file. You do not need to check the file's contents for that. An example of how to check the status is

status=$(curl -w '%{http_code}' "${url}" -o "${file}")
test "${status}" -eq 200 || rm -- "${file}"

The various options you can use with -w are documented in the manual, and depending on your needs, you may want to extend this to output more information and parse it, and/or change the check of the status code to allow more than merely 200.

|improve this answer
  • 1
    It is important to quote the file, in case it contains spaces or special characters. Hence, instead of $file, use either "$file" or "${file}". For safety, you should also quote the URL, as either "$url" or "${url}". – Paddy Landau Nov 4 '14 at 9:51
  • @PaddyLandau That was because I had originally hard-coded the URL and file name, and changed it to variables just before posting my answer. Thanks, edited. Also, for safety, rm should have -- before the file name. Some folks really do use files named -i. – hvd Nov 4 '14 at 9:54
12

You could use this find command to delete all files only containing only the <html> pattern in the first line:

find . -type f -exec sh -c 'sed q "$0" | grep -qP "^<html>$" && rm "$0"' {} \;
|improve this answer
  • What's with the \;? – Shahbaz Oct 28 '14 at 12:45
  • 1
    @Shahbaz \; is required to identify the end of the -exec action – Sylvain Pineau Oct 28 '14 at 12:56
  • 4
    This won't work (and, at least in theory, is dangerous) if any of the filenames contains double quotes. Using -exec sh -c 'head -1 "$0" ...' {} \; is more reliable. – Dennis Oct 28 '14 at 13:58
  • @Dennis thanks a lot for the suggestion, I fixed the command. – Sylvain Pineau Oct 28 '14 at 14:09
  • 1
    I generally prefer find solutions, but in this case it seems a lot more confusing and convoluted than a simple for loop.. – Seth Oct 28 '14 at 15:14
8

I just tested this, it works.

Run shopt first because we don't want to parse ls: 

shopt -s nullglob

then use a simple bash for loop to find files that begin with <html> and remove them:

for i in *; do if [ "$(head -n 1 "$i")" == '<html>' ]; then rm "$i"; fi; done

It would be safer to use: 

for i in *; do if [ "$(head -n 1 "$i")" == '<html>' ]; then rm -i "$i"; fi; done

to have rm ask before removing any files, just in case.

Note that shopt isn't strictly needed but it prevents certain issues from occurring if the directory is empty or there happens to be a file with an asterisk in its name.

|improve this answer
  • Perfect, thank you. I think I should have been able to come up with that on my own. Probably time for a break. – BeMy Friend Oct 28 '14 at 4:22
1

Not every automating task should be done with shell. Here is a Python script instead

#!/usr/bin/env python
import os

def is_html_file(file_name):
    # Actually, try/except is better
    # But not very readable for someone not familiar with python
    if not os.path.isfile(file_name):
        return False
    with open(file_name, 'rb') as f:
        # A lot of HTML file starts with doctype
        # It is better to check that too
        return f.read(6) == '<html>'

def main():
    # Use os.walk if recursion is needed
    for fn in os.listdir('.'):
        if is_html_file(fn):
            print 'Removing', fn, '...'
            os.remove(fn)

main()

Maybe it is more verbose than the equivalent bash commands, but it is

  1. More readable
  2. More extensible
  3. Never ever going to be screwed up by file names with spaces and shell metacharacters, however careless you are.
|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.