Recovering RAID5
Saturday 6th June, 2009 20:40 Comments: 1
As usual when I mess around with my fileserver, I've somehow lost one of my arrays. This time I swear it wasn't my fault. I hooked up the existing array and went to import it, but 4 of the disks were already visible with Failed Redundancy and one disk needed to be added (I can't remember if I initialised it or put it online) and after that it was resynching and all was well. But I could also see some missing disks from the old array that I was no longer using. I deleted the volume using Disk Management and expected the missing volume to go away. Instead, the resynching RAID5 array was now gone, and the missing volume was still missing. F**k.
Well, I've been in this position before, so I decided to get my tools together. First, I booted up my main machine to get the registry key so I could register my copy of File Scavenger. Then I downloaded File Scavenger (3.2, but thankfully my 3.1 key works fine with 3.x versions by the look of things). I hooked up the other drives from the old array and created a nice spanned volume to copy all of the data onto (no redundancy, but in theory it's "redundant" on the missing RAID5 and I don't have any more 1TB hard disks!). Then all I needed was the order of the disks. Sadly, because I'd only just created the array and imported it into the new install of 2008, I hadn't actually made a note of this. So I decided to do some guesswork with File Scavenger. I don't recommend doing this, as it requires a "sixth sense" or "gut feeling" to guess the right order without trying every possible combination. I'm normally pretty good at it, but I made the fatal mistake of forgetting to change a default setting. I should have used 0 instead of 63. This really screwed up the results of the long scans, causing me to give up and rely on Raid Reconstructor. My experience with it in the past hasn't been great, but after reading some forum posts it seems it's worth adding a couple of 0s to the probe size.
As I was using software RAID in Windows, I knew to use a 64KB strip size and set the parity to dynamic disk, but I thought I'd let it try the other parity options too to make sure that the entropy was correct. After leaving it running for about an hour, it appeared to have come up with a significant result and a drive order (presumably so good, that it appeared to have skipped the rest of the combinations). This was the drive order I'd had the most success with when I was playing with File Scavenger, but had ruled out because of the default 63 value (Raid Reconstructor defaults to 0,128 when testing) causing a low number of results - but substantially more than other combinations I'd tried.
I tried a demo version of R-Studio, threw in all of the drive details and let it perform a quick scan. I stopped it and created a preview of a large PDF file (IIRC the demo is restricted to 64KB, but the preview is able to generate previews of larger files) and it looked okay (or as good as it could in their PDF previewer). I then used File Scavenger and this is when I finally remembered to set the value to 0 rather than 63 (you'd think after trying ~100 combinations, I would have remembered sooner, but it was very late last night and I was tired). I recovered the same PDF and opened it in Adobe Acrobat (on another machine) and it looked perfect (bookmarks, images, text... everything!). So now I'm trying to recover 2.68TB worth of data onto a 2.72TB spanned volume. It's 4% there so far, which gives me plenty of time to write down what I'd done this time. In preparation for when (not if) I have to do it next time.
In my experience, it's a lot easier to recover data from software RAID5 than hardware RAID5. I'm glad I made the switch, but it's still not a substitute for proper backups. The plan was to move the data onto the array and then copy the really important data back onto the individual disks to keep as backups (at least until I want to use the disks on other systems), but my RAID cockup occurred before I had a chance to do that and before I could make a note of the disk order.
Well, I've been in this position before, so I decided to get my tools together. First, I booted up my main machine to get the registry key so I could register my copy of File Scavenger. Then I downloaded File Scavenger (3.2, but thankfully my 3.1 key works fine with 3.x versions by the look of things). I hooked up the other drives from the old array and created a nice spanned volume to copy all of the data onto (no redundancy, but in theory it's "redundant" on the missing RAID5 and I don't have any more 1TB hard disks!). Then all I needed was the order of the disks. Sadly, because I'd only just created the array and imported it into the new install of 2008, I hadn't actually made a note of this. So I decided to do some guesswork with File Scavenger. I don't recommend doing this, as it requires a "sixth sense" or "gut feeling" to guess the right order without trying every possible combination. I'm normally pretty good at it, but I made the fatal mistake of forgetting to change a default setting. I should have used 0 instead of 63. This really screwed up the results of the long scans, causing me to give up and rely on Raid Reconstructor. My experience with it in the past hasn't been great, but after reading some forum posts it seems it's worth adding a couple of 0s to the probe size.
As I was using software RAID in Windows, I knew to use a 64KB strip size and set the parity to dynamic disk, but I thought I'd let it try the other parity options too to make sure that the entropy was correct. After leaving it running for about an hour, it appeared to have come up with a significant result and a drive order (presumably so good, that it appeared to have skipped the rest of the combinations). This was the drive order I'd had the most success with when I was playing with File Scavenger, but had ruled out because of the default 63 value (Raid Reconstructor defaults to 0,128 when testing) causing a low number of results - but substantially more than other combinations I'd tried.
I tried a demo version of R-Studio, threw in all of the drive details and let it perform a quick scan. I stopped it and created a preview of a large PDF file (IIRC the demo is restricted to 64KB, but the preview is able to generate previews of larger files) and it looked okay (or as good as it could in their PDF previewer). I then used File Scavenger and this is when I finally remembered to set the value to 0 rather than 63 (you'd think after trying ~100 combinations, I would have remembered sooner, but it was very late last night and I was tired). I recovered the same PDF and opened it in Adobe Acrobat (on another machine) and it looked perfect (bookmarks, images, text... everything!). So now I'm trying to recover 2.68TB worth of data onto a 2.72TB spanned volume. It's 4% there so far, which gives me plenty of time to write down what I'd done this time. In preparation for when (not if) I have to do it next time.
In my experience, it's a lot easier to recover data from software RAID5 than hardware RAID5. I'm glad I made the switch, but it's still not a substitute for proper backups. The plan was to move the data onto the array and then copy the really important data back onto the individual disks to keep as backups (at least until I want to use the disks on other systems), but my RAID cockup occurred before I had a chance to do that and before I could make a note of the disk order.
Robert - Sunday 7th June, 2009 11:40 15 hours later and I appear to have all of my data back, (almost) as if nothing had happened.