On a related subject, physical media, like a song album. I started by wondering if there were ever any solid state distribution options (One Company tried SD cards) and then started digging into the underlying storage tech to see if I could find a write once long term stable process.
First the elephant in the room. Why solid state? because the drives to read the media are often the weak link. When the drives are no longer being manufactured how hard is it to make one? reading solid state drives is a relatively low precision electrical process compared to the high precision mechanical process needed for most media.
First on the chopping block was bulk storage. It tends to be delicate and hard to read and short lifespans. But if I limited myself to small storage there are some interesting options. fusible proms were promising but top out at a few megabytes. Mask roms? does anyone offer a mask rom service anymore?
Put a mask rom into a sd card... sd cards are too physically small for a song album we want something bigger to put album art on. A thing the size of the original gameboy cartridge with a usb interface and a mask rom?
My conclusion, for that specific goal, indefinite future storage of a song album. Vinyl records. low tech enough that it is easy to make a player for them.
I like the fact he's not just verifying all of them each year. AFAICR, reading the flash causes the row to be rewritten with the values just read.
I remember years ago working on the Wii, and there was a restriction on how often you could read the flash to avoid premature wearing. Not sure if that was just the specific type of storage, as googling suggests that NAND is subject to this and NOR isn't. I think pretty much all USB drives now use NOR flash, so maybe this isn't actually an issue any more.
> reading the flash causes the row to be rewritten
This only happens very rarely, though more frequently as NAND flash goes QLC and beyond.
Besides, other experiments have shown that data remanence is way more of an issue with drives that are almost completely worn out (way beyond their specified TBW) and about to croak. Even then you only get rare bitrot that can be checked for and compensated quite cheaply in most cases.
If you take fresh media, write it just once or a few times at most, use substantial overprovisioning to keep the drive in its fast pseudo-SLC mode, and reread the media periodically, NAND can be a good enough storage system for most casual needs.
Probably depends on what “consumer-friendly” entails, how it’s stored, and the quantity of data.
If we’re talking the average tech-illiterate to literate-but-cost-and-space-constrained person, probably Blu-Ray. A burner+reader combo with a stack of dual-layer discs is probably cost-effective. High-capacity HDDs would probably be equally effective if you can guarantee that they’re stored away from accidents and mishandling, but if it requires a SATA-to-USB adapter with assembly then it might possibly be out of reach for some consumers, and any risk of damage from movement could rule it out entirely.
If we’re talking tech-savvy consumers who don’t have the IT budget of a corporation, maybe LTO-5 or LTO-6 tapes could work. Tapes themselves are very affordable and have a good shelf lifespan. Used libraries can be had for under $600. The primary issues would be finding one with an interface that works with your existing equipment and software to support tape read and write.
What's long-term? I have some dvd-rs that push 20-25 years and despite the plastic getting brittle they still work. I also have some ide drives that still work without problems after 40 years. I would rather aim for 20 years and upgrade the storage device if I still need to retain the data.
That's a thought I hadn't had. The plastic of the disk getting so brittle it shatters in the drive due to age. I wonder what's the embrittlement profile of polycarbonate stored in reasonable condition.
Slightly related: I have a tool that writes random (incompressible) data to a disk and lets you verify it back without storing a copy (by using a csprng seed), initially developed for benchmarking SSDs that used to cheat to get better performance numbers but that can also be used for this purpose or to overwrite (“shred”) a disk: https://github.com/mqudsi/hddrand
I haven't used badblocks https://en.wikipedia.org/wiki/Badblocks in about 10 years, but I was annoyed that this exact feature wasn't available for testing accidental swapping of block locations. badblocks only writes the same data to each block and thus they are all indistinguishable.
It depends on the type of flash being used and the controller managing it. That he did not even identify the chips should inform you of the extent that these results can be trusted.
All I can say for sure is that you should not trust any flash for long term storage, thumb drive or otherwise. In serious enough, high usage, high heat enviornments where everything working without problems or delay is part of what they are paying us to be responsible for, it is standard practice to clone fresh images to nvmes every time, with multiple spares that can be swapped out in minutes when they inevitably fail anyways.
It depends on how the flash modules are maintained and their quality, but yes having freshly written data will imply better data consistency on flash media.
Flash media relies on recharging, which may or many not happen often enough.
I think they are reading it correctly. Year 1, they touched one drive and left 9 untouched. Year 2, they read one additional drive and left 8 untouched. Etc.
That's good. I want to keep some institutional knowledge and photos in "cold storage" and cloud subscriptions with a credit card and password are completely inviable.
I'll probably get a spinner and a flash drive and hope one of them survives the years.
If privacy is your primary problem with cloud storage, I would suggest veracrypt containers. And if you aren't storing too much data, I would also suggest DVD/BluRay optical media with DVDisaster and PAR2 archives. I keep a DVD spindle in a safe deposit box that gets updated each year.
On a related subject, physical media, like a song album. I started by wondering if there were ever any solid state distribution options (One Company tried SD cards) and then started digging into the underlying storage tech to see if I could find a write once long term stable process.
First the elephant in the room. Why solid state? because the drives to read the media are often the weak link. When the drives are no longer being manufactured how hard is it to make one? reading solid state drives is a relatively low precision electrical process compared to the high precision mechanical process needed for most media.
First on the chopping block was bulk storage. It tends to be delicate and hard to read and short lifespans. But if I limited myself to small storage there are some interesting options. fusible proms were promising but top out at a few megabytes. Mask roms? does anyone offer a mask rom service anymore?
Put a mask rom into a sd card... sd cards are too physically small for a song album we want something bigger to put album art on. A thing the size of the original gameboy cartridge with a usb interface and a mask rom?
My conclusion, for that specific goal, indefinite future storage of a song album. Vinyl records. low tech enough that it is easy to make a player for them.
I like the fact he's not just verifying all of them each year. AFAICR, reading the flash causes the row to be rewritten with the values just read.
I remember years ago working on the Wii, and there was a restriction on how often you could read the flash to avoid premature wearing. Not sure if that was just the specific type of storage, as googling suggests that NAND is subject to this and NOR isn't. I think pretty much all USB drives now use NOR flash, so maybe this isn't actually an issue any more.
> reading the flash causes the row to be rewritten
This only happens very rarely, though more frequently as NAND flash goes QLC and beyond.
Besides, other experiments have shown that data remanence is way more of an issue with drives that are almost completely worn out (way beyond their specified TBW) and about to croak. Even then you only get rare bitrot that can be checked for and compensated quite cheaply in most cases.
If you take fresh media, write it just once or a few times at most, use substantial overprovisioning to keep the drive in its fast pseudo-SLC mode, and reread the media periodically, NAND can be a good enough storage system for most casual needs.
reading the flash causes the row to be rewritten with the values just read
DRAM works that way but flash doesn't. Read disturb is a different issue.
pretty much all USB drives now use NOR flash
Nope, NOR flash is much more expensive than NAND so NOR is only used for firmware and everything else is NAND.
What is the best consumer friendly long-term storage medium? Are we still better off with high capacity dvd/Blu ray discs?
Probably depends on what “consumer-friendly” entails, how it’s stored, and the quantity of data.
If we’re talking the average tech-illiterate to literate-but-cost-and-space-constrained person, probably Blu-Ray. A burner+reader combo with a stack of dual-layer discs is probably cost-effective. High-capacity HDDs would probably be equally effective if you can guarantee that they’re stored away from accidents and mishandling, but if it requires a SATA-to-USB adapter with assembly then it might possibly be out of reach for some consumers, and any risk of damage from movement could rule it out entirely.
If we’re talking tech-savvy consumers who don’t have the IT budget of a corporation, maybe LTO-5 or LTO-6 tapes could work. Tapes themselves are very affordable and have a good shelf lifespan. Used libraries can be had for under $600. The primary issues would be finding one with an interface that works with your existing equipment and software to support tape read and write.
I've been a big fan of M-Disc BD-R.
What's long-term? I have some dvd-rs that push 20-25 years and despite the plastic getting brittle they still work. I also have some ide drives that still work without problems after 40 years. I would rather aim for 20 years and upgrade the storage device if I still need to retain the data.
That's a thought I hadn't had. The plastic of the disk getting so brittle it shatters in the drive due to age. I wonder what's the embrittlement profile of polycarbonate stored in reasonable condition.
Slightly related: I have a tool that writes random (incompressible) data to a disk and lets you verify it back without storing a copy (by using a csprng seed), initially developed for benchmarking SSDs that used to cheat to get better performance numbers but that can also be used for this purpose or to overwrite (“shred”) a disk: https://github.com/mqudsi/hddrand
I haven't used badblocks https://en.wikipedia.org/wiki/Badblocks in about 10 years, but I was annoyed that this exact feature wasn't available for testing accidental swapping of block locations. badblocks only writes the same data to each block and thus they are all indistinguishable.
Rewriting the data each year hides the actual issue here. Have had plenty of "nice" flash drives rot to hell in 18+ months of dormancy
Does rewriting data help prevent bit rot? Does it mean powered drives can take advantage of it by periodically rewriting the same data over?
It depends on the type of flash being used and the controller managing it. That he did not even identify the chips should inform you of the extent that these results can be trusted.
All I can say for sure is that you should not trust any flash for long term storage, thumb drive or otherwise. In serious enough, high usage, high heat enviornments where everything working without problems or delay is part of what they are paying us to be responsible for, it is standard practice to clone fresh images to nvmes every time, with multiple spares that can be swapped out in minutes when they inevitably fail anyways.
It depends on how the flash modules are maintained and their quality, but yes having freshly written data will imply better data consistency on flash media.
Flash media relies on recharging, which may or many not happen often enough.
Did you miss that there are 10 different drives and so they have 10 different years of tests where they are testing a completely untouched drive?
I don't think you're reading the results properly.
I think they are reading it correctly. Year 1, they touched one drive and left 9 untouched. Year 2, they read one additional drive and left 8 untouched. Etc.
Yes, it's also confirmed on the OP's blog linked in the post.
Those drives aren't being read
What's the simplest way to rewrite the data without actually copying the data? Like in place rewrite - you write what you read.
I've seen "dd if=/dev/removable of=/dev/removable" suggested. I don't know if it actually works or if the OS optimizes it to a no-op.
Powered all the time on or powered off?
OP says powered off.
That's good. I want to keep some institutional knowledge and photos in "cold storage" and cloud subscriptions with a credit card and password are completely inviable.
I'll probably get a spinner and a flash drive and hope one of them survives the years.
If privacy is your primary problem with cloud storage, I would suggest veracrypt containers. And if you aren't storing too much data, I would also suggest DVD/BluRay optical media with DVDisaster and PAR2 archives. I keep a DVD spindle in a safe deposit box that gets updated each year.