Loading huge files - fast

[JerometheGiraff]

I'm in need of some suggestions, I sometimes have to load files that are any where between 3 - 25 gigs. An SSD that is 1 TB is too costly. My current 7200 RPM drive with 64 Meg Cache is still somewhat slow to load files of such size, I was looking into a 10K drive, not sure if it will speed things up tremendously. At this point, not sure if there is a solution, besides waiting for the file to load ?

 

[TwoCables]

Why would you need a 1 TB drive to move 3-25 GB of files?

 

[LMiller7]

Loading files from what source and by what means?
That may be the limiting factor and not you hard drive at all.

 

[TwoCables]

Like a USB source.

 

[JerometheGiraff]

No External Drives, all Internal HDD. I don't want to move files, I want to read and write files of such size, faster. Why did I mention 1TB drive, multiple 25 gig files by 30, you're almost at 1 TB.

 

[TwoCables]

No External Drives, all Internal HDD. I don't want to move files, I want to read and write files of such size, faster. Why did I mention 1TB drive, multiple 25 gig files by 30, you're almost at 1 TB.

Oh. You didn't mention 30 files at 25 GB a piece. You said that you want to load files that are anywhere between 3-25 GB and then you said a 1 TB SSD is too expensive, so you can see how I understood it.

Anyway, yeah I don't know if there's a solution to this other than spending a lot of money on a much faster drive.

 

[kbrady1979]

Find something else to do while transferring? A 10K RPM HDD won't do it much faster than today's 7200 RPM drives. Have you ever ran Crystal Disk Mark on your HDD's? If not, download it and run a test and post the results.

 

[Britton30]

Maybe a NAS enclosure, 4 500GB drives in a RAID setup so you have 1TB of space?
Something uncommon is needed to keep opening 25GB files.

 

[TVeblen]

What do you consider too slow?
I often transfer 3 to12GB files between 2 PCs connected via gigabit Ethernet, SATA 2 to SATA 2 and I achieve about 110-120MB per second.
From spinner to spinner on one PC, SATA 2 to SATA 3 I get 120 - 160MB per second.
SATA 3 to SATA 2 I get 100 - 120MB per second.
The rate of transfer changes. There are other factors at play.

 

[JerometheGiraff]

Able to load 25 gig files in up to 5-10 seconds. At a rate of 160 megs per second, is still slow.

I may have to live with it and find a way to down size the files, so I don't run into this, although even if I down size the file sizes are still 2-5 gigs max, no way around it.

 

[kbrady1979]

What do you mean by "able to load 25gb files in up to 5-10 seconds" exactly? Load them into what from where? 10K RPM drives MIGHT hit 200MB/s Read/Write on a good day. Not worth spending money on for such a little bump up in performance. You either need a very complex RAID array or SSD's to take the Read/Write speed of the mechanical HD out of the equation.

 

[bobafetthotmail]

if it's not a secret, can you tell what are you trying to do (i.e. what are those big files for, and why you need so fast transfer rates)? maybe there is a better way around the issue.

 

[TVeblen]

Able to load 25 gig files in up to 5-10 seconds. At a rate of 160 megs per second, is still slow.

No. It's just the limit of the technology.

I may have to live with it

Best answer

and find a way to down size the files, so I don't run into this, although even if I down size the file sizes are still 2-5 gigs max, no way around it.

Many video programs will compress video streams into .avi, .mp4, etc. to save disk space. However, I have found it takes more time to do the conversion that to wait for the file to transfer.

I would also suggest meditation!

 

[JerometheGiraff]

There really is no other solution, besides trying to find the fastest SSD on the market, which probably isn't fast enough, or accepting the wait times. I'll have to accept the wait times

 

[bobafetthotmail]

You checked out RAID? Add moar disks and run them as one to improve speed. Raid 0 is the fastest, but if a drive dies you lose everything, raid 10 and you have moar speed and enough fault tolerance to survive some losses.

If you have a crappy PC laying around (and an excessive amount of hard drives) you can add that at the software level with linux. Then you just need to find a cable with enough bandwidth to move data to/from it.

 

[Faladu]

OP, yeah you are in the technology limitation, 160 megs a second is SLOW?

What are you doing, that it's a problem? [someone needs perspective, here you try a 1200 baud modem!! :roflmao:]

 

[Britton30]

There really is no other solution, besides trying to find the fastest SSD on the market, which probably isn't fast enough, or accepting the wait times. I'll have to accept the wait times

Until 2237 when we'll have bio gel packs as storage medium. No waiting at all and they hold exabytes of data.

 

[bobafetthotmail]

Well, that's from Star Trek Voyager isn't it?
Not a show known for scientific accuracy.

Although yeah, bio-stuff has a ridiculously high data density (DNA mostly), the issue is that reading speed tends to suck a bit as it's a loooooooooooong tape.

 

[TVeblen]

Well, that's from Star Trek Voyager isn't it?
Not a show known for scientific accuracy.

The original Star Trek predicted the flip phone!

We are waiting for Quantum Computing, where (as the breathless protagonists tell us) the computer will have the answer before you ask it. (Verbally of course!)

 

[bobafetthotmail]

We have already quantum computers. They are one-trick ponies though. That is, they can do stuff normal computers cannot, but cannot do what a normal computer can. So they do just some equation solving for the moment.
Btw, Google bought one recently for its brand new Quantum Artificial Intelligence Lab. Likely to make smarter search algorithms, and not at all a world-ruling Skynet. Googlenet.

the computer will have the answer before you ask it. (Verbally of course!)

Any smartphone equipped with Siri or one of its android clones can do that. Wolfram Alpha is another similar project (answers questions written in plain english) but does not listen/speak (not that it's hard to add anyway)