1

I have scraped lots of images that were either jpg, jpeg, or png. However, some of my images do not show in thumbnail view because they are damaged. I have already increased the limit to 100MB for thumbnail view in Linux so now only the ones that are corrupted do not show as thumbnails.

How can I detect these images either using a Python code or bash script?

For example when I click on one of the PNGs, it looks like the following:

<?xml version="1.0" encoding="utf-8"?>
<!-- Generator: Adobe Illustrator 16.0.4, SVG Export Plug-In . SVG Version: 6.00 Build 0)  -->
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
     width="3024px" height="3024px" viewBox="0 0 3024 3024" enable-background="new 0 0 3024 3024" xml:space="preserve">
<path d="M1515.778,205.669C981.54,205.669,476.37,135.396,0-0.008c0,1068.396,0,2756.137,0,3024.016
    c476.37-135.406,981.54-205.678,1515.778-205.678c528.163,0,1031.723,68.74,1508.222,205.678c0-1028.865,0-1919.085,0-3024.016
    C2547.501,136.957,2043.941,205.669,1515.778,205.669z M2353.338,463.804l-18.163,28.818
    c-17.094,27.133-32.388,50.271-53.773,82.474c-28.56,42.729-81.775,127.019-145.562,235.543
    c-17.617,29.904-39.284,66.361-62.221,104.957c-43.037,72.421-91.819,154.506-129.665,219.889
    c-15.9,27.728-32.095,55.903-48.402,84.298c-42.299,73.63-86.041,149.764-127.873,223.314
    c-43.399,76.25-85.883,150.915-128.288,225.84v75.033c0,104.111,2.157,217.242,6.057,318.561
    c1.859,46.055,3.741,128.156,5.733,215.074c2.375,103.506,4.829,210.529,7.438,264.551l0.779,16.256l0.094,1.971l-17.549-5.006
    c-6.877-1.961-13.856-3.748-20.918-5.383c-21.498-4.521-44.457-8.213-66.956-10.348c-13.755-1.125-27.735-1.691-41.9-1.691
    c-0.057,0-0.113,0-0.17,0c-0.056,0-0.104,0-0.165,0c-14.169,0-28.146,0.566-41.904,1.691c-22.488,2.135-45.451,5.826-66.952,10.348
    c-7.059,1.635-14.037,3.422-20.919,5.383l-17.549,5.006l0.098-1.971l0.779-16.256c2.608-54.021,5.063-161.045,7.435-264.551
    c1.995-86.918,3.877-169.02,5.729-215.074c3.907-101.318,6.061-214.449,6.061-318.561v-75.033
    c-42.405-74.926-84.89-149.59-128.292-225.84c-41.829-73.551-85.57-149.685-127.87-223.314
    c-16.311-28.395-32.497-56.57-48.405-84.298c-37.84-65.383-86.621-147.467-129.663-219.889
    c-22.94-38.596-44.604-75.053-62.22-104.957c-63.787-108.524-117.003-192.814-145.562-235.543
    c-21.385-32.204-36.68-55.341-53.777-82.474l-18.159-28.818l-0.117-0.188l32.957,9.44c42.194,12.089,85.16,17.965,131.348,17.965
    c46.015,0,90.265-5.927,131.51-17.611l9.983-2.829l5.037,9.076c81.625,147.26,300.785,507.46,431.723,722.674
    c45.156,74.209,80.898,132.967,98.715,162.765c0.061-0.102,0.121-0.207,0.181-0.309c0.061,0.102,0.128,0.208,0.188,0.309
    c17.812-29.798,53.562-88.556,98.712-162.765c130.941-215.214,350.1-575.415,431.726-722.674l5.029-9.076l9.99,2.829
    c41.246,11.685,85.491,17.611,131.51,17.611c46.188,0,89.15-5.876,131.345-17.965l32.96-9.44L2353.338,463.804z"/>
</svg>

Some of the images though don't even open and are like this:

could not load image

text file icon

and when I open jpg images that don't have a thumbnail preview they are opened as text files with many weird characters. My ultimate goal is to automatically remove these damaged files rather than removing them manually by looking at their thumbnails as I have 10,000 images.

Also, for example I cannot see this image when I click on it but I get the following result:

$ identify 590.jpeg
590.jpeg JPEG 450x338 450x338+0+0 8-bit DirectClass 47.8KB 0.000u 0:00.000

and

>>> from PIL import Image
>>> im = Image.open("590.jpeg")
>>> im.verify()
>>> 

Update: Image could not open broken png files but cannot detect broken jpg/jpeg files:

>>> im = Image.open("722.png")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/scratch/sjn/anaconda/lib/python3.6/site-packages/PIL/Image.py", line 2590, in open
    % (filename if filename else fp))
OSError: cannot identify image file '722.png'

722.png is

[jalal@goku media30_GV_Aug2018]$ identify 722.png
722.png SVG 3024x3024 3024x3024+0+0 16-bit DirectClass 2.61KB 0.000u 0:00.009
  • 1
    Those files aren't broken/corrupt/damaged. They're just SVG's with the wrong extension. Change the extension to .svg and then you should be able to open them. – Joseph Sible-Reinstate Monica Sep 28 '18 at 1:45
  • some are broken some are SVG. How can automatically do this? as I said, I have 10,000 images so I cannot go through all images one by one – Mona Jalal Sep 28 '18 at 1:48
  • Can you add an example of one that's actually broken? – Joseph Sible-Reinstate Monica Sep 28 '18 at 1:49
  • The svg you posted is a stylized Y, white or or off-white on a black or dark background. It opens in gwenview, krita, google-chrome and in firefox. – DK Bose Sep 28 '18 at 2:57
1

Try ImageMagick's identify command. From the man page:

Identify describes the format and characteristics of one or more image files. It will also report if an image is incomplete or corrupt.

Example:

$ identify foo.png
identify: NotAPNGImageFile (foo.png).

$ echo $?
1

An alternative, is to use PIL (Python Imaging Library):

from PIL import Image

im = Image.open("foo.png")
im.verify()

From the documentation:

im.verify()

Attempts to determine if the file is broken, without actually decoding the image data. If this method finds any problems, it raises suitable exceptions. This method only works on a newly opened image; if the image has already been loaded, the result is undefined. Also, if you need to load the image after using this method, you must reopen the image file.

Source

  • so I am not sure what's wrong. I cannot see the image but I see it is jpeg $ identify 590.jpeg 590.jpeg JPEG 450x338 450x338+0+0 8-bit DirectClass 47.8KB 0.000u 0:00.000 – Mona Jalal Sep 28 '18 at 1:08
  • I also tried im.verify on 590.jpeg and I got no error. Do you think something with my Linux might be broken that when I click on the image it does not open? – Mona Jalal Sep 28 '18 at 1:11
  • It could be @MonaJalal, have you tried using different image viewers? Might as well do an fsck just to be sure. – sudo Sep 28 '18 at 2:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.