Check real size of USB thumb drive

Question

I recently read a lot about fake MicroSD card and USB thumb drives that claim to have a lot of space (even if you ask your computer), while physically offering way less. I recently bought a SanDisk USB drive (128 GB claimed) and want to test its size. It's not bought via ebay or something, but I really want to test the real size before using it productively.

I could just copy stuff on it, copy it back and look if the files are okay. I could also automate it with Hashes and stuff. But I hoped there is a more accurate solution. I read that for Windows, H2testw does the trick. Is there an easy way to test this on Ubuntu/Linux? A specialized, well working tool maybe?

Update: Just to be clear, the idea is to verify that the size the linux system gets told by the controller is correct (so no data will be lost). It's not like I want to see if I get 128 GB instead of 127.3 GB. I want to test if all data I write will be readable again. Unfortunately I can only find a few information about this on English tech sites. There are good German sources, though. I'm actually searching for an application like those, but for Ubuntu/Linux: https://www.raymond.cc/blog/test-and-detect-fake-or-counterfeit-usb-flash-drives-bought-from-ebay-with-h2testw/

Update2: I tried to get together some sources in English. I did not read all of them in detail, due to missing time.

• https://www.ebay.com/gds/All-About-Fake-Flash-Drives-2013-/10000000177553258/g.html
• https://en.wikipedia.org/wiki/USB_flash_drive#Counterfeit_products
• https://www.heise.de/newsticker/meldung/Verdaechtige-USB-Sticks-mit-2-Terabyte-bei-Amazon-Faelschungen-entlarven-Datenverluste-vermeiden-3915202.html
• http://www.pcgameshardware.de/USB-Stick-Hardware-255579/News/falsche-Speicherkapazitaet-bei-Amazon-1245682/

Update3: Explanations

Due to the strange critics below, some explanations.

What is the problem and why does dd alone not solve it?

This is a reaction to

"Clearly figure out what is the problem you're trying to solve and what is the definition of "fake drive"."

It seems that some people do not understand the problem. So I try to explain it as short as I can in details, though I think this is much to the extend of my question.

The capacity of usb devices your operating system or unix tools give you, can be wrong. This is fatal, since your OS regulates how much data you can send it to. Send more data than it can really hold, you'll get a data loss. This is a problem. So, why can this happen?

You do not need to know the USB-Protocol well to unterstand the problem. Serial Interfaces have the common property, that the client device (the usb drive) will need to tell its own capacity via this serial interface. This means that the client device needs it's own controller with some knowledge about the devices purpose and, in this case, it's capacity. It also decides what is done, when it receives the command to store something. If the controller is programmed that way, it can just ignore the command or overwrite something with the data.

What does this mean? Whatever your unix tools tell you about the capacity of the drive: It's what the tools asked the drive, nothing more. This is what h2testw was invented for: It tests the real size with a method explained later on, and compares it to what the drive says. If this is not the same, you may have a data loss, because all your common operations to store data, rely on the information of your operating system, which just asks the controller. Why just ask? Testing needs time and overwrites all data on the drive. So it's natural that an Operating System needs to rely on this information.

To check the real capacity like h2testw, you indeed can use dd to write data on the drive, read it again, and see if it's the same you wrote. Totally legit. The nature of hardware and the drive make it more complicated. Consider write-caches for example. You need to ensure that you do not read from the cache. This is just one example of why it is not as easy as it looks. Also think that just writing zeros means a low entropy of information, which can be reconstructed when reading. It's just not that easy in detail. You can still do it manually, of course.

But why, when you can automate things? Why to the work? f3 as proposed in my answer below, implements tons of thoughts of many contributors (consider that it kind of extended h2testw) and it also implements several methods with different trade-offs. The developer figured out the tricks of different fake drives (aka counterfeit drives) they had at hand. So, while I understand the theory and the problem (seemingly since the problems are well explained in german tech media, but not in english speaking media), I do not pretend to understand everything, which is why I mentioned it above. It's just the theory I understand, and I'm more of a software guy. But as a student of informatics I understand it well enough to see the problem.

"Try to understand basic Unix utilities"

Actually I answered this one already, but to make it clear: Unix tools just use the USB-Protocol (for USB-devices only, of course) to gather information. It does not make sense to do more than that.

Does it help to only buy from trustes suppliers?

tl;dr: It does not.

"When it comes to buying goods, just like it comes to any form of security, consider finding a trusted seller and buy drives only from them."

Security (and safety) is NOT about trust! It's about verification and validation! Sorry but this is so wrong in so many ways.

Assume you buy via a trusted seller. A few questions:

1. Did the supplier test the hardware to ensure there is no data loss? Does re recognize when he buys fake drives and sells them? Not necessarily.

2. Is it possible that he buys stuff he doesn't know is fake? Totally, look at the recent ryzen fakes: https://www.pcgamer.com/beware-of-fake-ryzen-processors-selling-on-amazon/ , https://www.heise.de/newsticker/meldung/Direkt-von-Amazon-Faelschungen-von-AMDs-Ryzen-Prozessoren-im-Umlauf-3772757.html

3. If I loose my presentation in the drive and screw up the presentation, will my trusted supplier go back in time and rescue me? It will probably replace the drive, since the last time-travelling DeLorean was destroyed in 1885.

Other stuff

"This question really seems to be more like "promo" for what OP likes, and seems that OP is much less interested in actually testing the drives."

This is ridiculous. I was searching specifically for a similar tool to h2testw that also runs on linux. And yes, that's what I'd "like", helpful answer, so sorry. I had no idea that the english speaking press is not that aware of such issues and was lucky to find something like that later on. This is not a promo, but actually it seems you could use one.

Answer 1

f3 - Fight Flash Fraud

There is only one alternative I found, but I think this is even a better one than the original h2testw tool for MS Windows. Fortunately, it is really easy to use, even from command line. There are GUIs available, though. There is also a lot of information about the implementation and the problem with fake drives on the tools website.

• Main page (source code):
• Documentation
• QT GUI

f3 offer two methods:

• f3probe method: Much faster
• h2testw method: Slower. Also test R/W performance. Probably more reliable.

The f3probe method (recomended)

f3probe is one way to test the drives, not as accurate but faster since it does not write on the whole drive. You can read more about it on the tools website. If you want to be 100% sure, better use the h2testw method. As the developer describes on the website:

f3probe is the fastest way to identify fake drives and their real sizes.

And:

Finally, thanks to f3probe being free software, and once f3probe is battle proven, f3probe could be embedded on smartphones, cameras, MP3 players, and other devices to stop once and for all the proliferation of fake flash.

There is also a usage example on the website:

Warning: This will destroy any previously stored data on your disk!

$ sudo f3probe --destructive --time-ops /dev/sdb
[sudo] password for michel: 
F3 probe 6.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing may **demolish data,** so it is more suitable for flash drives out of the box, without files being stored yet. The process normally takes from a few seconds to 15 minutes, but
         it can take longer. Please be patient. 

Bad news: The device `/dev/sdb' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=16477878 /dev/sdb

Device geometry:
             *Usable* size: 7.86 GB (16477879 blocks)
            Announced size: 15.33 GB (32155648 blocks)
                    Module: 16.00 GB (2^34 Bytes)
    Approximate cache size: 0.00 Byte (0 blocks), need-reset=yes
       Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 1'13"
 Operation: total time / count = avg time
      Read: 472.1ms / 4198 = 112us
     Write: 55.48s / 2158 = 25.7ms
     Reset: 17.88s / 14 = 1.27s

Note that it also returns a command that enables you to use the drive with it's real size, using f3fix.

The f3fix tool

f3fix creates a partition that fits the actual size of the fake drive. Use f3probe’s output to determine the parameters for i3fix

sudo f3fix --last-sec=16477878 /dev/sdb

The h2testw method / Testing performance with f3read/f3write

F3 is a collection of tools that deal with fake flash drives. Two of them together implement the h2testw-Method:

f3write [--start-at=NUM] [--end-at=NUM] <PATH>
f3read  [--start-at=NUM] [--end-at=NUM] <PATH>

f3write will ask for the devices claimed size and fill it with generated files with a size of 1gb each. f3read will read all those files and see of they are complete and not broken. As an example the commands I used to test my ~128gb thumb drive:

$ f3write /media/username/1EB8021AB801F0D7/
Free space: 117.94 GB
Creating file 1.h2w ... OK!                           
...
Creating file 118.h2w ... OK!                         
Free space: 0.00 Byte
Average writing speed: 11.67 MB/s

Now to test whether the files are correctly stored:

$ f3read /media/username/1EB8021AB801F0D7/
                  SECTORS      ok/corrupted/changed/overwritten
Validating file 1.h2w ... 2097152/        0/      0/      0
...
Validating file 118.h2w ... 1979488/        0/      0/      0

  Data OK: 117.94 GB (247346272 sectors)
Data LOST: 0.00 Byte (0 sectors)
           Corrupted: 0.00 Byte (0 sectors)
    Slightly changed: 0.00 Byte (0 sectors)
         Overwritten: 0.00 Byte (0 sectors)
Average reading speed: 32.38 MB/s

The test for a drive of this size took about three hours with this method and sometimes caused a heavy disk load on my computer, but it's said to me the most accurate.

Install in Ubuntu

On terminal:

sudo apt install f3

This will bring you: f3brew, f3fix, f3probe, f3read, f3write with their man pages.

This tools are part of the f3 package, which is at least available on Ubuntu 15.10. According to the website, there are some more tools that are available. To get them take a look at the website.
The package comes with short but useful manpages, though I think they miss some information from the website about the difference of f3read/write and f3probe for example, which is why this answer got a little longer.

Answer 2

I have written a simple tool for just that, it's called CapacityTester (screenshot) and it has a GUI as well as a CLI.

There's a precompiled binary for Debian 7 available for download, which is very likely to work out of the box on a modern Ubuntu system.

I've written it for my own personal use because I couldn't find a graphical tool for this purpose. You just need to mount your empty USB flash drive first, select it and start the test. It's a very dumb tool because all it does is fill the drive with files and then verify that the data on the drive is correct. It will abort the test on the first error (writing or reading/verifying). It will report the offset of the chunk that could not be written or verified successfully, but this is a logical offset so this information may be useless because it depends on the filesystem where the files are located on the drive. However, when the drive has been filled with data and everything could be read and verified, it should be safe to assume that the drive's reported capacity is correct. As a side note, the test files are automatically deleted (this may not work if the drive is bad).

Again, it's very simple as it only works with files on top of an existing filesystem. So there are some KB (+ 1M buffer) that cannot be tested. And it's very slow because it really fills the whole filesystem. F3 is certainly much more sophisticated and also faster, but it has no GUI. The only reason CapacityTester exists is because it has a GUI so that it can be used by users who are not familiar with the command line or who simply prefer a GUI.

Feedback is appreciated.

Answer 3

[Problem] You bought an SD card, but you don't know if it's fake or what the real capacity is.

[Solution] Here are the exact steps that worked for me on Ubuntu:

1. [Terminal] Install f3: sudo apt update && sudo apt install f3
2. [Disk Utility] Add a drive partition. Format the drive in slow mode as fat. See image.
3. [Disk Utility] Mount the drive: See image.
4. [Terminal] write to the disk: sudo f3write /media/toddmo/FakeOrNot
5. [Terminal] read from the disk: sudo f3read /media/toddmo/FakeOrNot
6. [Amazon/AliExpress][!important] Leave a review listing the write speed, read speed and true capacity.

(step 3)

Answer 4

Addressing OP's behavior and the "fake drive"

I'm editing the answer to properly address a few points, since OP has been very vehement (and in my opinion, opposing most comments and answers except their own, which I find suspicious). Particularly, there's a lot of claiming there exist "fake drive", but there's no clear definition as to what on the earth that actually means. OP stated:

I could just copy stuff on it, copy it back and look if the files are okay. I could also automate it with Hashes and stuff. But I hoped there is a more accurate solution.

OP themselves admitted that they "could just copy stuff", and verify data integrity, but were very much against all other comments and answer that propose anything else and OP only kept pushing F3 as the "real deal". The question itself at first started out about the size of the drive, but then OP for whatever reason mentioned hashes to "look if the files are ok", as if there are mysterious drives that claim one size and let you write that size, but then data is corrupt. Therefore, I find it highly suspicious and would consider OP promoting F3 as spam question and answer.

When a drive is actually fake drive

In the question, OP's apparent definition is

"..drives that claim to have a lot of space (often carried too far, like 128 GB), while physically offering only 0.5 to 4 GB."

In other words,according to OP, the controller claims X amount of data, but USB can only contain something like 80-90 % less of what is claimed.

The user sudodus proposed in the comments (emphasis added): "I have found that several USB pendrives are slightly smaller than the nominal size. I call them undersized. I think the fake drives are 'substantially undersized' (usually half of the nominal size or less)". This definition is great, however if we take that, fake drive is defined at 50%. A drive that claims 64 GB but can only hold 32 GB, technically looses half of its value to the owner and the owner only can put half of what they intended onto the drive.

I propose a simpler definition: counterfeit storage device is the one which claims to have Claimed Size but is below 15% tolerance (and tolerance is Claimed Size ± 15 %).

The ± 15 % is very reasonable. Consider also that users are typically confused between Unix,IEEE, and IEC organizations using binary prefix instead of power of 10 prefix for data storage size. The difference gets into 20% at yotta prefix level, however USB drives aren't there yet, so for maybe next 20 years 15 percent is reasonable. (See askubuntu question "Meaning of 'i' in 'MiB'" and Binary Prefix)

Testing the drive

Effectively, the user doesn't need any special tools, aside what already comes with Ubuntu and most POSIX-compliant Unix systems. Let's emphasize and rephrase the definition again:

If we can't write amount of data to drive and what we write is within 15 % tolerance, then the drive is OK

he simple way to do it is with dd, just overwrite the device with zeros ( and of course remember to save your files before you do that ).

sudo dd if=/dev/zero of=/dev/sdb1 iflag=nocache oflag=direct bs=1                        

Note the bs=1 for block size of 1 byte. The dd command usually gives a report as to how much is written.

$ dd if=/dev/zero of=/dev/null bs=1 count=1024
1024+0 records in
1024+0 records out
1024 bytes (1.0 kB, 1.0 KiB) copied, 0.00261981 s, 391 kB/s

We asked it to write 1024 bytes, it wrote 1024 bytes.

More precise list of steps adhering to the definition would be:

• Figure out how much data the drive claims (assuming that you suspect df to be "mistaken"). In this example, lets assume /dev/sdb1 is my device file for USB drive:

  $ df -P /dev/sdb1 | awk 'NR==2{print $2}'
  115247656
  

  Note that -P flag is for POSIX portability, which means block size of data will be 1024 bytes, and that means there's 115247656*1024 bytes on that drive.

• Figure out what is 15% tolerance below what drive claims (115247656), perhaps use utility that supports floating point calculation such as awk :

   $ awk 'BEGIN{printf "%d\n",115247656*(1-0.15)}'
   97960507
  

• Create random data on hard drive of the same size as the drive in previous step to use as benchmark: dd if=/dev/urandom of=./mytestfile.random bs=1024 count=97960507

• Now write data dd if=./mytestfile.random of=/dev/sda1. If the drive can hold this much, it's "real". You can also take md5sum or sha1sum of the ./mytestfile.random and compare with /dev/sda1 now. Even better improvement would be to write the mytestfile.random to mountpoint of the file, thus keeping the filesystem on the drive and unaltering partitioning of the drive, in other words

  dd if=./mytestfile.random of=/mountpoint/for/usb/drive/testfile.random
  

• For integrity then, you can just do any hashsum check, such as md5sum, sha1sum, sha256sum or others. For example

  md5sum ./mytestfile.random  /mountpoint/for/usb/drive/testfile.random
  

  Key point here is that if amount of written data is within tolerance and produces correct checksum before and after writing - the drive is probably OK.

All this can be put into a nice script for convenience, if one so desires.

Conclusion

This question really seems to be more like "promo" for what OP likes, and seems that OP is much less interested in actually testing the drives. Additionally, the issue itself is more human than "drive" issue. In the comments, OP themselves stated they don't really understand USB behavior, but are vehement to blame "the controller". I'll leave this question with 3 points:

• Clearly figure out what is the problem you're trying to solve and what is the definition of "fake drive".
• Try to understand basic Unix utilities
• When it comes to buying goods, just like it comes to any form of security, consider finding a trusted seller and buy drives only from them.