18

How can I do a disk low level formatting in Ubuntu on an external hard disk so that any data recovery tools like test-disk or photorec cannot retrieve any data once formatted to ext4 or any file-system? As I was playing around with Test disk, I found that it can recover almost all old files even after formatting it many times and I want to use my external hard disk as new because it is only one year old and in warranty but without any old data.

3
  • If you are worried about that, you might also worry about others recovering data in the case your hard drive fails... – PlasmaHH Jul 3 '15 at 11:24
  • 6
    Find a professional shredding service in the area. Shred the disk, incinerate the remains, buy a new disk. – Williham Totland Jul 3 '15 at 11:57
  • From an information theory standpoint, this is very hard. They will at least be able to tell with high probability that you erased all your files, no erasing that fact (easily.) – PyRulez Jul 4 '15 at 12:52
39

I use shred when selling old hard drives, use man shred for more information:

shred -vzn 0 /dev/xxx

You can specify n number of times to overwrite and z to overwrite it all with zeros at the end.

5
  • 12
    +1, but I wouldn't recommend -n 0. After all, if you want to make your data unrecoverable, you should really shred it, not just overwrite. – Andrea Corbellini Jul 3 '15 at 10:12
  • 1
    Andrea, do you have any literature that shows this is necessary? – drxzcl Jul 3 '15 at 14:53
  • 3
    @AndreaCorbellini Note that in 2006 NIST published a report (NIST Special Publication 800-88 Section 2.3 (p. 6)) saying "turns out one overwrite is enough!" wiki – Mike Ounsworth Jul 3 '15 at 22:00
  • my documentation does not indicate the a block device can be used for FILE. Mine is part of coreutils 8.22. – Otheus Nov 21 '16 at 13:34
  • I had input/output errors using shred on a dual boot disk that was about 50% NTFS and 50% ext4. This has me preferring to use dd. – H2ONaCl Mar 16 '18 at 23:56
23

Modern discs implement the ATA secure erase feature, which you can do with the hdparm command using the --security-erase option, after first setting a password on the disk. Note that there are caveats, including

  • possible firmware bugs
  • possible disk controller timeout for a possibly long-running operation
  • you shouldn't do it over USB

For some discs this will take hours, as each block is rewritten. For others it can take seconds as it just means changing a global encryption key held by the disc that transparently encrypts/decrypts all data going to/from the disk. This is true for hard discs and SSDs. It's the firmware than counts.

Another article also suggests that --security-erase has the advantage that it may also wipe out the hidden areas HPA host protected area and DCO device configuration overlay.

7
  • On hard disks ? – kos Jul 3 '15 at 9:15
  • 3
    Sure, hard discs too. The command was invented before SSD's. stackexchange – meuh Jul 3 '15 at 9:29
  • Interesting, I tought it would have work only on SSDs – kos Jul 3 '15 at 9:40
  • 1
    I read in this SE <link>security.stackexchange.com/a/5665 that it's not necessarily secure, but it must be better than nothing! – Arronical Jul 3 '15 at 10:11
  • 2
    You forgot possible NSA backdoors. – Lennart Rolland Jul 3 '15 at 17:48
14

If you want totally wipe the hdd the best is to use dd:

dd if=/dev/zero of=/dev/sdX bs=512 

Replace sdX with your drive letter

Another possible tools:

This command is a replacement for rm command. It works under Linux/BSD/UNIX-like OSes. It removes each specified file by overwriting, renaming, and truncating it before unlinking. This prevents other people from undelete or recovering any information about the file from the command line. Because it does lots of operation on file/directory for secure deletion, it also takes lot of time to remove it.

7
  • dd might leave residual magnetism (or charge in case of an SSD) on your drive. Use shred to overwrite several times with random data. – Philippe Jul 3 '15 at 11:30
  • 3
    @Philippe I wouldn't use shred on a SSD. – kos Jul 3 '15 at 11:39
  • 2
    You can also use cat with much simpler syntax: cat /dev/zero >/dev/sdX. ------ To get slightly better performance with dd use larger block size for example bs=64k. (cat uses this block size - 64 KiB by default.) ------ GNU dd has an useful feature: When you send it signal USR1 it shows statistics and continues in operation: pkill -USR1 -f '^dd if=/dev/zero of=/dev/sdX bs=64k' – pabouk Jul 3 '15 at 12:35
  • 3
    FYI, you could also use /dev/urandom instead of /dev/zero. This will give random data instead of binary zeros. IDK if this will make any difference at all, but you can do it. – TSJNachos117 Jul 4 '15 at 2:05
  • 2
    PS: you should probably repeat this command at least once or twice to make sure the data cannot be recovered. – TSJNachos117 Jul 4 '15 at 2:06
14

This doesn't really answer your question, as you want to use the HDD. However, it's important.


If this is really important data that should never be recovered, it's not safe to use the disk any more. Apple Macs offer a 35 overwrite feature - which they claim is what the government requires, but it's complicated:

Effective immediately, DSS will no longer approve overwriting procedures for the sanitization or downgrading (e.g. release to lower level classified information controls) of IS storage devices (e.g., hard drives) used for classified processing.

It is thought that some of the 3-letter agencies (FBI, CIA, MI6) require physical destruction of magnetic media (e.g., melting in a furnace).

You do have a few options - the one I've relied on in the past is hitting it with a hammer lots of times, then wiping the magnets over it.

However, I'm a student and nobody wants to hack into an empty bank account, so it didn't need to be as secure. Plus, I wasn't going to bin the Hard Drive - it's now a bird scarer for the allotment.

If you work under confidentially agreements (especially those involving the government) you need something a little more secure. I'd recommend a blowtorch:

Source, CNET

My pronouns are He / Him

7
  • 1
    I am sorry but I've not asked for the physical destruction or damaging media but to overwrite the data which remains even after formatting to any file-system – Mudit Kapil Jul 3 '15 at 16:45
  • 5
    @MKC I know, and the answer is physical destruction is the only 100% sure way of doing it. However, I'm not expecting you to accept this answer of course. – Tim Jul 3 '15 at 17:07
  • 1
    I am also a kind of a student with empty pockets – Mudit Kapil Jul 3 '15 at 17:10
  • 4
    I'm Mr. Back-up around here, and I agree with Tim: physical destruction is the only way. Completely melting it down, shredding or taking a 12-gauge shot-gun at it are all good. Anything else is just make-belief. – Fabby Jul 3 '15 at 18:49
  • I'm a bit confused about why you would mention the KGB of the Soviet Union which broke up in 1991 in place of it's successors. Thermite is more epic than a blowtorch. frank.geekheim.de/?p=2423 – LiveWireBT Jul 8 '15 at 8:28
8

The dd command by Maythux is a good one to go with, though I've read somewhere (sorry for no source!) that it's good to overwrite with zeros then do a second pass with random writes, then zero it off on a third pass.

To use dd to write random patterns:

dd if=/dev/urandom of=/dev/sdX bs=512

I have seen some examples of dd using smaller block size (down to bs=4) which I think will make the write slightly more random, but take longer.

Be super careful that you select the correct drive when using dd as it's very dangerous if you make a mistake!

5
  • I think you read about writing with 0s, 1s and then with random pattern, which is the DoD 5220.22-M 3-pass algorithm – kos Jul 3 '15 at 9:39
  • 1
    Thanks for the pointer @kos, I seem to keep seeing stuff that says a single wipe is fine these days, i.e. 'The Myth of 3 pass Overwrites', tough to know what the best advice is! – Arronical Jul 3 '15 at 10:02
  • 2
    @Arronical there's no evidence that zero-d data is recoverable by any tool or technique. It's never been accomplished publicly. – Daenyth Jul 3 '15 at 13:28
  • I know that OS X offers 35 write removal. – Tim Jul 3 '15 at 16:28
  • Block Size defaults to 512 and is much slower than saying bs=64k – SDsolar Dec 4 '17 at 19:39
8

NaA, but some additional informations, expecially about SSDs


There are lots of methods and lots of debates around the topic; basically this is about the type of the support (magnetical disk or flash disk) and about the sensitivity of your informations.

Data which are really sensitive might be considered undeniably safe only when the physical drive is destroyed, since, theoretically [but not too much] (placeholder for the link if I ever manage to find it again) laboratory methods capable of measuring differences in the magnetic / electronic fields of a support, joined with the knowledge of the algorithm used by firmware of the drive to handle the data might be able to extract even data that has been overwritten multiple times.

However, for the purpose of erasing a hard disk containing personal data, each of the methods above will work fine against a software recovery attempt; it might be noted, however, that methods using low-level utilities are not good to be used on SSDs, for multiple reasons.

The first one being that writing a flash-based memory (expecially multiple times) is not really healthy for the memory itself, altough the wear-leveling issue is highly overestimated (expecially on MLC SSDs), i.e.: you indeed don't want to dd a SSD drive every day, but once, twice or even thrice (or even more) in its whole life-span is absolutely not a big deal;

The second one being the fact that commercial SSDs usually come with additional replacement cells, which are initially not being used by the drive, and which are intended to act as a replacement for cells being damaged during the usage of the drive; this leading to, possibly, the SSD controller unmapping such damaged cells (possibly containing sensitive informations) at some point of the drive's life making them physically inaccessible from low-level tools.

So, if you're using a SSD, the best solution is probably sticking to the manufacturer's ATA SECURE_ERASE command implementation (as described in meuh's answer), hoping that this will be robust enough for the purpose.

2
  • 1
    So far, only you have covered the vital point "Do you need to be able to use the drive afterwards?" Destruction of a drive [industrial crusher & incinerator] is best. Failing that, disassemble and / or throw the disks through a drop-saw: try aiming for the centre spindle for max damage. If you can't manage that, I'd suggest ripping one side off - and using an old TV Degaussing Wand to randomise the data on the platters. – Alan Campbell Jul 3 '15 at 11:59
  • Or the magnets you get in a standard harddrive. – Tim Jul 3 '15 at 18:44
5

I want to elaborate on a detail in the question that no other answer has touched upon: low-level formatting. In the past, a distinction was made between two types of disk formatting:

  • Low-level formatting -- This refers to creating the data structures that define the sectors and tracks on a disk.
  • High-level formatting -- This refers to creating a filesystem on a disk.

Some decrepit old fools like me still twist our sentences into pretzel shapes to avoid using the word "formatting" alone because that's ambiguous. For years, in the Unix/Linux world, high-level formatting was always referred to as creating a filesystem. The word "formatting" has been creeping into various utilities in this context, though.

It's important to note that, to the best of my knowledge, no modern hard disk supports low-level formatting by users; that task is done at the factory and cannot be done again once the disk leaves the factory. Hard disks in the 1980s and maybe into the early 1990s did support low-level formatting by users, but those disks are all museum pieces by now. You can still low-level format a floppy disk, if your computer is old enough to actually have one. This was often necessary with floppies, since different computers (Macs vs. PCs, say) used different types of low-level formats, so if you got a Mac floppy and wanted to use it on a PC (or vice-versa), you'd have to low-level format it.

Also and FWIW, I looked into it a few years ago, and I could find no references to actual tools that could be used to recover data from a disk that had been wiped by using dd to zero out all its sectors. It's theoretically possible to do so by disassembling the disk and using special hardware to read it. If the disk holds national security secrets, going beyond dd might be advisable (and is required by many national security agencies); but I wouldn't worry about protecting your credit card numbers and Flickr password with anything beyond dd, at least on a hard disk. SSDs use fundamentally different technology and I've not looked into them in any depth.

0
4

If you want absolute peace of mind (and still want to use it afterwards) give it the Guttman method.

It is overkill. A standard three-pass overwrite - clearing with zeros, filling with ones, then overwriting with a random string of either - is industry recognised and still the defacto means to certifiably sanitize a hard drive. If your drive is without bad sectors and the eraser can access all addressable areas - your data is gone.

It might be worth noting that there are no trading names which claim to be able to recover previously overwritten data.

Infact, there are good reasons why a single pass might well be enough. There is really only one clinical circumstance where data could "theoretically" be recovered under laboratory conditions: by using 'some sort of' molecular magnetic microscopy on what would have ideally been a previously clean (and preferably intact) drive - measuring the minute differences in residual charge thought to represent previous logical values and then reconstructing bit by bit with the aid of probability analysis. It is vague and deemed to be so prohibitively expensive - especially for newer hard drives - that it is questionable whether some agencies could be capable of even justifying the attempt. Your data certainly isn't worth it!

Obsolete hard drives are thought prone - but beyond one pass the chances of recovery approach zero.

The belief that this can be reliably done by any agency is very likely a myth allowed to perpetuate to overstate a nation's perceived range of capabilities.

Conversely, the mere suggestion that such data recovery methods are no longer (or never were) possible on (post-modern) hard drives is quite possibly the rumour which enables agencies and nation-states to keep the capability unannounced.

So if you want to, give it Guttman's ridiculous 35-pass wipe. Even more, encrypt it first and throw away the key (aka crypto-erasure - a method sufficient on its own). If you want total peace of mind - shatter the platters. If you're sane - wipe it three times, or just once if you're going to be actively using it again pretty soon.

2
  • Good Answer; But not my requirement so +1 – Mudit Kapil Jul 4 '15 at 14:32
  • DoD standards for software-based secure erase is 7 passes of random numbers. – SDsolar Dec 4 '17 at 19:40
3

I think it is maybe not as secure as professional tools or dd or something like that, but if you just need to overwrite the whole file system's data part (! not disk), you could just format it normally and then create a huge file filling up the whole space with

yes "string pattern" > /mnt/device-to-kill/huge-file

This will create/replace the file /mnt/device-to-kill/huge-file, assuming that your external device is mounted as /mnt/device-to-kill.

The content will be string pattern overly and overly repeated. If you don't give a pattern as argument, default is y. The command will not stop writing data until either the disk is full or you abort it with Ctrl+C.

The advantage of this method is that even an unexperienced user can't destroy anything accidentally. If you specify a target file on your HDD, it will create/replace that one and fill up every free byte of disk space you have, which should at most produce a crash and minor loss of unsaved data. It does not touch existing files.

This is also a disadvantage though, as you have to format the file system before to delete all files normally, as the command only overwrites free space. Besides this, it can only overwrite the data part of your partition/device, not parts reserved for e.g. file tables (FAT, MFT, whatever depending on your file system).

Conclusion: This command is more for newbies that fear to accidentally destroy more than they want instead of experts that need 110% secure data destruction.

Not the answer you're looking for? Browse other questions tagged or ask your own question.