Curcial and WD havea much higher rate on average across all their models.
The 800% is only because they had a single drive for a certain model, and it failed within 2 months. They have a lot of other Seagate models that are much older on average without any failures.
Interesting, from that data it seems SSD reliability so far isn’t too far off from HDDs (at least for Backblaze’s intensive cloud storage workload) despite having no moving parts…
Will be interesting to see how the reliability plays out as they build up their SSD inventory over the coming years
I agree. Consumer use cases of SSDs sees a tremendous benefit if only for accidental damage reasons, but for enterprise data center use I would not have expected the same overall rates of failure.
Pose this question to chat gpt 3.5 or 4. Ask it to assist in making a (python?) script to do this. Feed it errors, and you can get there pretty quickly and learn along the way.
Are you buying the hardware for this setup, or do you already have it laying around? If you don’t have the hardware yet I’d recommend not using external USB drives in any way possible, as speed and reliability will be hindered.
If you already have the hardware and want to use it I’m not super confident on recommending anything given my inexperience with this sort of setup, but I would probably try to use ZFS to minimize any potential read/write issues with dodgy USB connections. ZFS checksums files several times in transit, and will automatically repair and maintain them even if the drive gives you the wrong data. ZFS will probably be cranky when used with USB drives but it should still be possible. If you’re already planning on a RAID6 you could use a RAIDZ2 for a roughly equivalent ZFS option, or a double mirror layout for increased speed and IOPS. A RAIDZ2 is probably more resistant against disk failures since you can lose any 2 disks without pool failure, whereas with a double mirror the wrong 2 disks failing can cause a pool failure. The traditional gripe about RAIDZ’s longer rebuild times being vulnerable periods of failure are not relevant when your disks are only 2TB. Note you’ll likely want to limit ZFS’s ARC size if you’re pressed for memory on the Orange Pi, as it will try to use a lot of your memory to improve I/O efficiency by default. It should automatically release this memory if anything else needs it but it’s not always perfect.
Another option you may consider is SnapRAID+MergerFS, which can be built in a pseudo-RAID5 or RAID6 fashion with 1 or 2 parity drives, but parity calculation is not real time and you have to explicitly schedule parity syncs (aka if a data disk fails, anything changed before your last sync will be vulnerable). You can use any filesystems you want underneath this setup, so XFS/Ext4/BTRFS are all viable options. This sort of setup doesn’t have ZFS’s licensing baggage and might be easier to set up on an Orange Pi, depending on what distro you’re running. One small benefit of this setup is that you can pull the disks at any time and files will be intact (there is no striping). If a catastrophic pool failure happens, your remaining disks will still have readable data for the files that they are responsible for.
In terms of performance: ZFS double mirror > ZFS RAIDZ2 > SnapRAID+MergerFS (only runs at the speed of the disk that has the file).
In terms of stability: ZFS RAIDZ2 >= ZFS double mirror > SnapRAID+MergerFS (lacks obsessive checksumming and parity is not realtime).
If you want to be able to grow, check out mergerfs and snapraid. If you’re wanting to use a pi and USB drives it’s probably more what you’re wanting than zfs and raid arrays. It’s what i’m using and I’ve been really happy with it.
I’ve been using linux for a long time, and I have a background in this kind of stuff, but it’s not my career and I don’t keep as current as if it was, so i’m going to give my point of view on this.
A zfs array is probably the legit way to go. But there’s a huge caveat there. If you’re not working with this technology all the time, it’s really not more robust or reliable for you. If you have a failure in several years, you don’t want to rely on the fact that you set it up appropriately years ago, and you don’t want to have to relearn it all just to recover your data.
Mergerfs is basically just files on a bunch of disks. Each disk has the same directory structure and your files just exist in one of those directories on a single disk, and your mergerfs volume shows you all files on all disks in that directory. There are finer points of administration, but the bottom line is you don’t need to know a lot, or interact with mergerfs at all, to move all those files somewhere else. Just copy from each disk to a new drive and you have it all.
Snapraid is just a snapshot. You can use it to recover your data if a drive fails. The commands are pretty simple, and relearning that isn’t going to be too hard several years down the road.
The best way isn’t always the best if you know you’re not going to keep current with the technology.
XXVI Holdings similarly shows that it is owned by Alphabet Inc (www.wikidata.org/wiki/Q20800404), which is owned by a list of people and entities with various different voting rights.
There’s stuff you can tweak of course if it don’t quite work for you. Worked fine on me tests.
If you notice a page missing, you should be able to just scroll back to it and then download again to get everything. The first script just keeps collecting pages till you refresh the site. Which also means you should refresh once you are done downloading, as it eats CPU for breakfast.
Oh and NEVER RUN ANY JAVASCRIPT CODE SOMEONE ON THE INTERNET TELLS YOU TO RUN
Well, I may be technologically semi-literate and I may have felt a bit dizzy when I saw actual code in your comment, but I sure as hell will find a way to put it to use, no matter the cost.
Plan to do this with a lot of the entertainment videos I watch, considering how ban happy some websites have been with content creators, being able to still see their craft after it is gone is worthwhile.
They still happily exist on YouTube- for now. So no point in re-hosting, they’ll get squirreled away into the Giant Hard Drive of Doom.
If something happens to the actual archive project in the near future, I’ll likely section them up into 20gb pieces and post them out on a torrent someplace.
I’m a bit baffled that this hasn’t popped up yet: Sell them on eBay.
Mark them as broken goods/scrap and re-iterate that fact very clearly in the product description. Broken drives often sell for up to 1/3 of the value of a working one, no scamming needed.
I cannot tell you why that is, but my theory is that a lot of folk buy up broken drives in private sales in the hopes that the “broken”-diagnosis is just user error and that the drive is actually fine. Knowing my users that might actually be true in many cases.
Edit: I didn’t quite catch that you were not able to successfully overwrite your data. I guess that’s a point against selling it. Always encrypt your drives, that way you can always sell them when they break!
datahoarder
Hot
This magazine is from a federated server and may be incomplete. Browse more on the original instance.