0

I have downloaded a 120 GB dataset in a tar.gz using Download Accelerator Plus, and used the following command to extract it:

tar -xvzf train_val2018.tar.gz

The dataset, when extracted, should contain a little more than 8,142 folders and exactly 461,939 image files.

When I open up the extracted folder, it does contain parts of the dataset. However, a huge portion of it is missing - it only contains 3,542 folders and 179,689 files.

Using the file explorer or the built-in file extractor GUI to extract it on Ubuntu only does worse. I've tried various applications on Windows as well, like WinZip, WinRAR, and 7Zip. None of these work, but the issue with them is that they had memory issues with such a large dataset.

Could the file be corrupted? The file has the same size as they've mentioned in the dataset website, and I've downloaded several larger files with DAP, and I've never encountered a corruption issue. For this reason, I'd like to know if there's some limitation with the built-in extractor or another issue.

The dataset I'm referring to is the iNaturalist 2018 Contest Dataset.

  • 2
    In my opinion, "Even if the size of downloaded file is same as the actual file, it is not guaranteed the file isn't corrupted" – Kulfy Jan 22 '19 at 9:45
2

Yes, off course it could be. And if you read they have the check step:

Running md5sum train_val2018.tar.gz should produce b1c6952ce38f31868cc50ea72d066cc3

if you do not want to compare manually the md5sum you can create a file md5sum-db (name can be whatever you like) containing the following:

b1c6952ce38f31868cc50ea72d066cc3 train_val2018.tar.gz

enter in the folder the compressed DB is and run:

md5sum -c md5sum-db

If the hash check match you will receive this output:

 train_val2018.tar.gz: OK

else

train_val2018.tar.gz: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

Another way to check the gz file is to use the test it:

gunzip -t file.tar.gz

NOTE: this method do not ensure the data contained in the archive integrity.

2

Could the file be corrupted?

Yes. That site also provides a tool for you to check yourself. From the link I see 4 checksums where the 1st one is for your download.

md5sum train_val2018.tar.gz should produce b1c6952ce38f31868cc50ea72d066cc3
md5sum train2018.json.tar.gz should produce bfa29d89d629cbf04d826a720c0a68b0
md5sum val2018.json.tar.gz should produce f2ed8bfe3e9901cdefceb4e53cd3775d
md5sum test2018.tar.gz should produce 4b71d44d73e27475eefea68886c7d1b1
md5sum test2018.json.tar.gz should produce fc717a7f53ac72ed8b250221a08a4502

If you get a different result your download is corrupted.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.