11

I'm trying to gzip all files on ubuntu that have the file extension .css, .html or .js. in a top directory and all subdirectories. I want to keep the original files and overwrite the .gz file, if already existing.

So when I have n files, I want to keep these n files and create additional n archive files. Not just one.

My try was to run a script that looks like this:

gzip -rkf *.css
gzip -rkf *.html
... one line for each file extension

First: I need to have one line in that script for each file extension I want to gzip. That's ok, but I hope to find a better way

Second and more important: It does not work. Although -r should do the job, the subdirectories are unchanged. The gzip file is only created in the top directory.

What am I missing here?

Btw: The following is a bug in the verbose output, right? When using -k and -v option

-k, --keep        keep (don't delete) input files
-v, --verbose     verbose mode

The verbose output says it replaces the file, although "replace" means that the original file does not exist after the replace. Anyway, THis is only the output thing.

$ ls
  index.html      subdir1  testfile      testfile.css.gz
  javaclass.java  subdir2  testfile.css
$ gzip -fkv *.css
  testfile.css:   6.6% -- replaced with testfile.css.gz
$ ls
  index.html      subdir1  testfile      testfile.css.gz
  javaclass.java  subdir2  testfile.css
  • 1
    -r works as designed. From man gzip: Travel the directory structure recursively. If any of the file names specified on the command line are directories, gzip will descend into the directory and compress all the files it finds there (or decompress them in the case of gunzip). (emphasis mine) – Dennis Jul 11 '14 at 20:34
  • Ok. So -r would enter a directory with the name XYZ.css. Then recursion is not designed as I expected. – Sadik Jul 12 '14 at 8:59
7

you can do that with a for loop to find every file then compress it:

for i in `find | grep -E "\.css$|\.html$"`; do gzip "$i" ; done
  • Thank you! Though the -r option does not work, -k and -f are working, so I can use them like this: for i in find | grep -E "\.css$|\.html$"; do gzip -vkf "$i" ; done` – Sadik Jul 11 '14 at 12:19
  • @Sadik: Be careful! This approach won't work if any of the files' names contains a space. – Dennis Jul 11 '14 at 20:31
  • Could you explain why not? – Sadik Jul 12 '14 at 9:13
  • 1
    @Sadik: `...` provides a string, not a list. for uses the internal field separator ($IFS) to decide where that string should be split. By default, it splits at linefeeds, tabs and spaces, so if you have a file called new style.css, the commands gzip new and gzip style.css will be executed. – Dennis Jul 12 '14 at 13:01
  • 1
    @Sadik, Dennis is right, as quick workaround you can run export IFS=$'\n' just before the for loop. – mndo Aug 8 '14 at 14:52
13

I would use

find /path/to/dir \( -name '*.css' -o -name '*.html' \) -exec gzip --verbose --keep {} \;

Change name to iname if you want to match the extensions case-insensitively (i.e. include .CSS and/or .HTML extensions). You can omit the /path/to/dir if you want to start the recursive search from the current directory.

  • 2
    For those who may be wondering about the --keep switch, yes, it causes the original files to be retained. Omit it if you want to them to be deleted once gzipped. – Ben Johnson Feb 7 '17 at 23:00
4

To get the list of files:

find -type f | grep -P '\.js|\.html|\.css'

And to gzip all those files:

find -type f | grep -P '\.js|\.html|\.css' | tar cvzf archive.gz -T -
  • Wouldn't this tar the list of files as output by find, rather than the files themselves? – Jos Jul 11 '14 at 12:15
  • I edited my question to make clear that I want to have an archive file for each css,html or js file. – Sadik Jul 11 '14 at 12:16
  • 2
    @Jos no with the -T option tar processes the input as filenames. – chaos Jul 11 '14 at 12:25
  • @chaos Ah, thank you. I learned something today. – Jos Jul 11 '14 at 12:31
2

I used steeldriver's answer, but I like to complete it with the --best and --force options.

cd into any folder and type this code. All your matching files will be gzipped.

find . \( -name '*.css' -o -name '*.js' \) -exec gzip --verbose --keep --best --force {} \;
  • Use --best for best compression ratio.
  • Use --force for overwriting without asking if there is already a gzipped file.
1

You can use globstar.

With the globstar shell option enabled, all you need is gzip -vk **/*.{css,html}.

The Bash shell has a globstar option that lets you write recursive globs with **. shopt -s globstar enables it. But you might not want to do that for other commands you run later, so you can run it and your gzip command in a subshell instead.

This command gzips all .css and .html files in the current directory any of its subdirectories, any of their subdirectories, etc., keeping the original files (-k) and telling you what it's doing (-v):

(shopt -s globstar; gzip -vk **/*.{css,html})

If you want to match filenames case-insensitively so those extensions with some or all letters capitalized are included, then you can also enable the nocaseglob shell option:

(shopt -s globstar nocaseglob; gzip -vk **/*.{css,html})

; separates the two commands, and the outer ( ) cause them to be run in a subshell. Setting a shell option in a subshell does not cause it to be set in the calling shell. If you do want to enable globstar then you can run shopt -s globstar; then you can just run the command:

gzip -vk **/*.{css,html}

You can disable globstar with shopt -u globstar. You can check if it's currently enabled with shopt globstar.

How It Works

The key to how this gzip command works is that the shell performs expansions on it to produce a list of each file in the directory hierarchy with a matching name, then passes each of these filenames as arguments to gzip.

  • Brace expansion turns **/*.{css,html} into **/*.css **/*.html.
  • Then globbing expands those two patterns into the names of files accessible under the current directory (**, due to globstar) whose filenames consist of anything (*) followed by the specified suffix (.css or .html in this case).

This does not match files whose names start with . or those that reside in directories named this way. You probably don't have any such HTML and CSS files and, if you do, you probably don't want to include them. But if you do want to include them, then you can match them explicitly depending on your needs. For example, changing **/*.{css,html} to **/{,.}*.{css,html} includes files that start with . while still not searching in folders that do.

If you want both files whose names start with . and files in directories whose names start with . to be included, there's a cleaner and simpler way: enable the dotglob shell option.

(shopt -s globstar dotglob; gzip -vk **/*.{css,html})

Or if you want case-insensitive matching and matching of filenames that start with .:

(shopt -s globstar nocaseglob dotglob; gzip -vk **/*.{css,html})

It's possible, though very rare, for ** to expand to something too long.

If you have a huge number of files named this way, then this may fail with an error message explaining that the shell cannot build the command line because it would be too long. (Even with thousands of files, this usually is not a problem.)

gzip won't be called at all, so you won't get a half-done job.

If this error happens, or if you're worried about it, you can use find with -exec, either as steeldriver describes (with {} \;) or as I describe below (with {} +).

You can use find with the -exec action and + for efficiency.

The gzip command supports being given names of multiple files to be compressed. But this find command, although it works well and won't be slow unless you have many files, runs the gzip command once for each file:

find . \( -name \*.css -o -name \*.html \) -exec gzip -vk {} \;

This works, and you can definitely use it. (. searches from the current directory. Besides that, it's really a slightly different way of writing the command in steeldriver's very good answer; you can use whichever style you prefer.)

You can also make find pass multiple filenames to gzip and run it only as many times as necessary--which is nearly always just once. To do that, use + instead of \;. The + argument should come just after {}. find replaces + with additional filenames, if any.

find . \( -name \*.css -o -name \*.html \) -exec gzip -vk {} +

It's fine to use + even if there are only a few matching files, and when there are many of them, it can be noticeably faster than having a separate gzip invocation for each file.

As steeldriver mentions, you can use -iname instead of -name to match files whose name end like .css or .html but with different capitalization. This corresponds to enabling nocaseglob in the globstar-based method described above.

Finally, you probably don't have any matching files or directories that start with .. But if you do, find automatically includes them. If you want to exclude them (as happens with the globstar-based method detailed above when dotglob is off), you can:

find . -not -path '*/.*' \( -name \*.css -o -name \*.html \) -exec gzip -vk {} +

The globstar-based way described above is simpler to write, especially if you're excluding directories and files that begin with ., since that's the default.

What not to do...

Filenames can contain any character except the path separator / and the null character. Many techniques that break on weird filenames exist, and they are usually more complicated than techniques that always just work. So I suggest avoiding them even when you know (or think you know) they're okay in your specific situation. And of course you must not use them if you might have filenames with characters that may be treated specially, including spaces.

It is possible to safely pipe the output of find to another command that processes it if you use -print0 or a similar action to cause it to place a null character between paths instead of a newline, and not otherwise. Filenames can contain newlines (though I discourage you from deliberately naming files with them). A find command with the -print action--including find commands with no explicit action, since then -print is the default--does not produce output that can safely be piped or otherwise provided to another command that performs an action on the files.

The output find produces with the -print0 action may safely be piped to xargs -0 (the -0 flag tells xargs to expect null-separated input).

0

To zip all files in a folder/subfolder recursively:

gzip -r `find . -type f -name "*.html"` 

To unzip:

gunzip -r `find . -type f -name "*.gz"` 
  • This command substitution based method will frequently break, and quite badly. The problem is that filenames containing spaces or other whitespace will be split and treated as multiple filenames. (These commands are written using ` ` syntax, but the problem fully applies when using the $( ) syntax as well.) – Eliah Kagan Aug 14 '17 at 15:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.