Another diskpart clean tale of woe

townsbg said:

CurlyBen said:

As for file systems, I understand that different developers have differing requirements and objectives, but given that OS X can read an NTFS volume without issue it really shouldn't be a big deal to implement basic write capability. There are drivers available to allow OSX to write to an NTFS partition but they seem to get mixed reviews, because they're not properly integrated into the system. I never tried reading HFS with a Windows machine, but it shouldn't be all that complicated. It's just quite frustrating as someone who uses both systems!

Quite a bold statement to make if you don't understand OS level programming. Just food for thought. I found this on Wikipedia. Apparently enabling write support on a mac causes it to crash so perhaps Apple tried and ran into problems they couldn't fix so they avoided them altogether by disabling it (not surprising considering Apple's history). I don't think that either company is in a hurry to make their systems inter-operable considering how much money that they are already making. If you think about it you can't (without an illegal hack) put mac on a pc and most that buy a mac aren't going to put windows on it so there isn't much of a demand for them to do so.

Mac OS X 10.3 and later include read-only support for NTFS-formatted partitions...Native NTFS write support has been discovered in Mac OS X 10.6 and later, but is not activated by default, although workarounds do exist to enable the functionality. However, user reports indicate the functionality is unstable and tends to cause kernel panics, probably the reason why write support has not been enabled or advertised.

https://en.wikipedia.org/wiki/NTFS#Interoperability

I'll admit my idea of hardware is more hammers and spanners (I'm a mechanical engineer by training... amongst other things) but it doesn't seem to be that big an assumption to say that if the OS understands the file system well enough to read it that it's not a big leap to be able to write entries for new files. There are two relatively small companies (Tuxera and Paragon) who have implemented full read-write capability on OS X, so it doesn't seem absurd to think that Apple could do it with a little effort. I understand it's not a priority for them, but it doesn't stop me wishing it was!

OK, so I've learned a little more about what the exFAT boot sector is saying (I created a 100mb exFAT partition on my new external hard drive to have a bit of a look at what's going on). It appears the first line stays the same for every exFAT partition, and the screenshots I posted yesterday are the only sectors where that string appears on the disk. Looking at the first few lines more closely, that sector isn't a working bootsector anyway as all the important attributes (hidden sectors, length, cluster size etc.) are all 0. So the bootsector is gone... but the FAT still exists to some extent, as DMDE was able to pull up the file structure when I was going through the recovery actions, although it didn't manage to recover everything successfully. I've attached a screenshot of the folder structure DMDE pulled out, and it's right up to date (the new folder structure directory was something I was working on the night before my mishap, ironically to make it easier for me to back up to a remote server via a VPN...)


So far my research on the exFAT bootsector makes me think it's the red 00s in lines 04 to 07 (see next image) that are the key to the bootsector, everything else is more or less generic to an exFAT partition. I'm trying to work out whether I can get enough information from DMDE to recreate how the bootsector should have looked (the final image is some data from DMDE, though I'd like to know where it's come from before using it. However that's from the volume information for the volume which shows the correct directory structure and I recovered significant amounts of data).




Anyway for the time being it's getting late, so I've rebooted my PC and I've set testdisk running another deep search (Intel/PC) with nothing else running, just in case it finds anything interesting.

The idea of implanting the exFat bootsector from another HDD has been on the back of my mind but I had reserved it as a final option if all else fails. That idea as conceived by me:

Buy another 1TB WD Passport. It should be the same model number, same capacity so that the total number of sectors is the same.

Format it as an exfat drive creating one single volume. Backup the bootsector at 2048 using bootice.

Restore that bootsector to sector 2048 on to sector 2048 of the drive where it is missing, again using bootice.

Will that work? I have no idea. ( I formatted my drive using Windows Disk Management. It put the bootsector at 2048. Will it be the same if I had formatted it with some other utility - say using a Mac?

You are doing some good research. You had examined the template for the NTFS bootsector. Now examine the exFat bootsector template in Bootice. Try to decipher what the fields you had marked in red denote. With more active brain cells than I have , you should be able to do a better job.

If TestDisk deep search also does not find a valid partition, try PhotoRec.

Keep Zero Assumption Recovery also in mind. The digital Image Recovery module in it can be used for free. It is also geared to image recovery specific to a list of cameras. Data Recovery Software, Solutions, Tutorials - ZAR Data Recovery

jumanji said:

The idea of implanting the exFat bootsector from another HDD has been on the back of my mind but I had reserved it as a final option if all else fails. That idea as conceived by me:

Buy another 1TB WD Passport. It should be the same model number, same capacity so that the total number of sectors is the same.

Format it as an exfat drive creating one single volume. Backup the bootsector at 2048 using bootice.

Restore that bootsector to sector 2048 on to sector 2048 of the drive where it is missing, again using bootice.

Will that work? I have no idea. ( I formatted my drive using Windows Disk Management. It put the bootsector at 2048. Will it be the same if I had formatted it with some other utility - say using a Mac?

You are doing some good research. You had examined the template for the NTFS bootsector. Now examine the exFat bootsector template in Bootice. Try to decipher what the fields you had marked in red denote. With more active brain cells than I have , you should be able to do a better job.

If TestDisk deep search also does not find a valid partition, try PhotoRec.

Keep Zero Assumption Recovery also in mind. The digital Image Recovery module in it can be used for free. It is also geared to image recovery specific to a list of cameras. Data Recovery Software, Solutions, Tutorials - ZAR Data Recovery

Testdisk deep search still didn't find anything, so I've set it running using the 'none' partition table option. So far it's supposedly found a couple of HFS partitions (which I don't recall ever creating) but it will probably be running for another 24 hours or so yet. I had a quick go with Photorec but it was behaving a little strangely - at first attempt (with default file options) it was finding the CR2 image files but also a huge volume of irrelevant .apple and .plist files, so I cancelled it and just selected the file types I'm interested in. I tried a few different combinations but it was finding almost nothing, so I've abandoned that for the time being - I may try again when I have another drive large enough to extract all the files to.

I also had a look at Zero Assumption Recovery but it needs a partition to search, so I'll leave that for the time being.

I like the idea of getting another 1tb Passport drive to experiment with. I'll have a look for one.

I've got a reasonable idea so far of what the fields marked in red are. I've put my comments in italics after each one:

• Hidden sectors
  The number of sectors prior to the bootsector, i.e. for the first bootsector 2048 (sectors 0-2047)
  
• Total sectors
  For a disk with a single partition presumably total sectors - hidden sectors. I suspect nothing will go wrong if this value is too large
  
• FAT offset
  Not sure yet whether this is a fixed or changing number
  
• FAT size
  This could be difficult - my understanding is the FAT will increase in size as more files are added, so the FAT size of a new partition will be far smaller than that of a partition with hundreds of thousands of files
  
• First cluster offset
  Not sure on this yet
  
• Clusters count
  Not sure on this yet
  
• Root directory first cluster
  Not sure on this yet
  
• Volume serial number
  I think this is just a way of uniquely identifying the volume to the OS, so probably doesn't need to match the original
  
• File system revision
  Apparently the only current revision is 00 01
  
• Bytes per sector
  Will almost certainly be the default value. 512 according to DMDE
  
• Sectors per cluster
  As above. 256 according to DMDE
  
• Number of FATs
  I believe exFAT only uses a single FAT
  
• Media descriptor
  Describes the type of drive. Can almost certainly be left at 80 (the value used in my trial exFAT partition)

So that's where I've got to so far. My thoughts on where to go next:

• I can probably get hold of a couple more exFAT bootsectors from drives my brother has. I need to get a better understanding of which values change between partitions and which don't
• I'm going to get another 1tb Passport drive and format it as exFAT as suggested. I'll back up the bootsector then copy all the recovered files to it and look to see how the bootsector changes, in particular the FAT size. I'll also experiment with formatting from OS X and see whether it creates a different bootsector to Windows 10.
• I'll try and find out how DMDE has come up with some of it's information, as it's clearly found the FAT, despite not being able to correctly recover everything
• When I've got enough information together to have a go at creating a new bootsector I'll copy it to my existing drive, then run Testdisk. I assume that if I've got the new bootsector correct (or at least close) I'll be able to list the contents of the disk before anything other than sector 2048 has been modified. If it doesn't work simply zeroing sector 2048 should return the drive to its current state

Any suggestions or criticisms welcome. I'm probably not going to be able to look at this much for the next week or two as I've got my flight instructor exams coming up and I should be doing some more studying! Although whilst I'm really annoyed at myself for causing all this hassle, I am quite enjoying learning a bit more about how the system works and the challenge of recreating it. That will turn to pure annoyance when the progress stops!

townsbg said:

CurlyBen said:

...it doesn't seem absurd to think that Apple could do it with a little effort. I understand it's not a priority for them, but it doesn't stop me wishing it was!

I agree however I think that Apple is bad at fixing problems or coming up with other solutions. After installing Yosemite I started encountering graphics issues and I've since found other problems with the OS that they ignored. They seem to be fixed in the newest version (which I put on a VM) but they didn't even attempt to fix them in Yosemite so I've had to live with them. I get the feeling that their engineers' spend most of their time developing new versions rather than fixing current ones. Pertaining exfat write support as I've stated I doubt that they consider the demand to be high enough to warrant fixing it.

I'll happily admit my experience of using an Apple computer has been well below what I was expecting. There appears to be an issue in El Capitan where running a virtual machine from a USB drive results in exceptionally slow running. This is a bit of an issue for me as I need to run Windows for some CAD software and a few other programs, and I only have a 120gb SSD due to the hideous prices Apple charge for storage upgrades. Oh well! There's a lot I like, but I'm certainly not an Apple evangelist.

Thinking about how I can work out the unknowns as listed above, it seems the thing to do is work out what the FAT should look like and then search for that. Then it doesn't really matter whether bootsector is identical to the one that's been lost, provided the FAT offset etc. are correct relative to the new bootsector. I think the best way to do that is going to be to create a new exFAT partition, completely zero it, then add some files to it and identify the FAT and its components. That's going to require a little more time than I have right now, but I'll post back when I've done so in case anyone is interested/attempting to go through the same process!

PS I'm confident the FAT is still there and at least mostly intact due to DMDE finding the correct folder structure.

Thanks jumanji, that link is interesting. I've just acquired another drive (not a 1tb unfortunately, but it will help) and I'm just in the process of cleaning the whole disk to make sure there's nothing there to confuse things. At the moment I'm struggling to work out what the FAT offset value should be - on my first trial partition the bootsector is at 2048 and the FAT offset is also 2048, which suggests to me the FAT should start at 4096, but the drive is blank between 4000 and at least 8000. I'll check the old drive for backup bootsectors but I'm fairly sure they're gone, as I searched for the first line which should be common to all exFAT bootsectors and I only found it once and that was a blank bootsector anyway. The FAT seems to start somewhere around sector 60,000 (I haven't worked out exactly where it starts) so it's possible that partition was slightly offset from the start of the drive anyway (clearly it wasn't already complicated enough!)