Write-caching - or no?

[martinlest]

I am in the middle of backing up a large amount of data, about 5TBs of files all the same size (11MBs), from one USB3 external HDD to another.

I have noticed, frustratingly, that although after a reboot, the speed is reasonably OK, this usually doesn't last long and the transfer becomes slower, with frequent bouts of up to 15 seconds where no files seem to be copied. (I usually use TeraCopy, or a batch script for copying). This happens whatever I try (running the transfer in Safe Mode, for instance) and Windows 10 doesn't really seem much netter than Windows 7.

Killing time during copying (5TBs takes a couple of days usually), I was browsing the internet looking for articles about how I might improve this copy speed (nothing useful so far) and came a cross a couple of articles suggesting that setting drives to 'Better performance' - with Write-Caching enabled could actually reduce performance, and at best benchmarks reproduced there showed that any increase in read/write was marginal.

I have usually set a drive to Better performance as soon as I have bought it, enabling write-caching. (Why is it, in Device Manager/properties/Policies, some of my USB drives only have the 'top half' of the options, that is, 'Removal Policy', but the bottom half of the window is missing - the Write-Caching Policy part?).

I tried changing a few of my HDD drives back to 'Quick Removal'. (My internal SDD drives - I have no internal HDDs - I have left on Better performance). I have noticed no change yet in transfer speeds at all as a result.

I wondered what any 'gurus' here might think of all this.

Thanks,

Martin

 

[Paul Black]

Hi martinlest,

I am in the middle of backing up a large amount of data, about 5TBs of files all the same size (11MBs), from one USB3 external HDD to another.

I have noticed, frustratingly, that although after a reboot, the speed is reasonably OK, this usually doesn't last long and the transfer becomes slower, with frequent bouts of up to 15 seconds where no files seem to be copied. (I usually use TeraCopy, or a batch script for copying).

What is the batch script that you are using?
Is the batch script using XCOPY?

 

[martinlest]

xxcopy

 

[Megahertz07]

HDDs are slow. When it begins to copy it seems fast as they are using the disk cache.
I do a backup from one SATA 600 to another SATA 600 (internal disks) and the average speed is 60MB/sec.

 

[martinlest]

Yes, my internal hard drives are all SSDs and I can get unbelievably fast transfers. The external HDDs seem erratic - far too often they slow to a frustrating crawl, especially when using XXCopy. With 5TBs of data to transfer, it often seems like it'll take a lifetime (as the estimate drops from 9 hours to 1600 days or whatever!). Occasionally I can maintain 30MB/s for hours on end, but it seems to be trial and error.

Windows is the problem for the most part of course - I often end up plugging the HDDs into my Windows 10 laptop instead of my Windows 7 PC.. as much as I vastly prefer Win7 to Win10 for most purposes (inc. gaming), Windows 10 is in fact a bit less likely to slow to a crawl when copying than is Windows 7.

 

[martinlest]

.. but as to enabling the performance mode/write-cache or not (from a performance, not data safety point of view)?

 

[Megahertz07]

As I understand, the performance is almost the same.
With write-cache on, windows may think that the write operation has ended when in fact it still reading from the cache to the disk. So data can be corrupted if you remove the disk while it still writing
I would leave write-cache on, and will remove the USB cable only after using the safe remove tool.

 

[Alejandro85]

You're misunderstanding what write-cache actually does. It doesn't improves drive performance, but OS aparent performance. Real write speed is limited by the hardware, and no software can ever increase that beyond the physical limits of the drive.

What the write cache does is to let the OS "lie" about the status of the write, it reports a successful write as soon as the data is put on a temporary buffer, not necesarily in the drive itself. When enabled, programs will not wait until the data is physically saves in the drive, but will continue as soon as possible, which helps increase responsiveness of the OS and software. When disabled, programs will wait until data reaches the drive successfully, which may take a little longer. While this is safer, it's also more waiting, as the OS cannot chose a better moment to commit the changes to the drive.

Usually there is little reason to ever turn write cache off, and that's commonly related with unreliable devices. For normal operation just leave it enabled all the time. Drive speed itself won't be affected.

Drops in performance are usually due to the nature of spinner drives. Outer sectors have a higher speed than inner ones, which are mapped to the last logical blocks in the disk, which end up being slower to access. Other causes may include saturation of copy buffers or disk usage from other processes. Write-caching is often not a cause of problems here.

 

[martinlest]

You're misunderstanding what write-cache actually does.

Maybe, but not in the way you say. I know that it is an O/S 'tweak', not some magical upgrade to the drive's performance. You have highlighted the word 'apparent', and indeed, the advantages are often hardly apparent at all (if you ultimately measure how effective transfers are by the time taken to move or copy the same amount of similar data. ('Similar' meaning of course not comparing moving 10 x 100MB files with 10000 x 100kb files).

What I find difficult to be patient with is the huge speed reduction often after less than a minute. Via USB 3, HDD to HDD, I rarely get more than 60MB/s (10MB files). I can live with that, but I make a coffee, come back, and the speed is now perhaps just 2MB/s. It fluctuates wildly thereafter, but never gets back to the original high speeds. Happens with all my external drives (mix of Toshiba, Seagate & WD). I note though that you say write-caching is probably not the culprit there.

Worse is the tendency for the copying to come to a halt for a minute or more, especially (it seemed) if it is via a batch script, then copy a few files and stop again. As if Windows is 'thinking about it'.

I asked this question because (as I think I mentioned) I had come a cross a few sites where drives had been benchmarked with write-caching on and then off and the difference in read/write speeds was insignificant. I also had the impression (after days and days of testing, as I copied by 5TBs of data - now, mercifully, completed) that the copying was less likely to slow to a halt with write-caching disabled (but that may have been wrong).

One thing I did discover is this (copied from my other thread on copying speeds here on the forum):

One thing I have noticed over the past several months of trying to copy large amounts of data (mostly via USB, HDD to HDD): if I drag and drop one single folder, no matter how large it is, from the source to the destination, the copy speed almost never slows down. It can run for an hour or more at a reasonable speed. However if you drag and drop (or copy and paste via the keyboard) multiple folders (including folders containing subfolders), then the copy speed invariably drops to a crawl. Similarly with a batch script...

So in desperation to get stuff backed up this week I have spent hours dragging individual folders across from one screen to another - an annoying waste of time, but at least the copying goes ahead 'full speed'. (I have also, for the largest folders, containing hundreds of subfolders, written batch scripts which copy each folder separately - the script may have hundreds of lines in it. That also runs without slowing down, usually, but can take a long time to set up).

I can't see any logic to that, but after so much practice at this over the past few months (I have copied a total of around 15TBs now), I am fairly sure it's not my imagination or just chance.