This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: copying a million tiny files?


Gary R. Van Sickle wrote:
I have a similar situation to the OP (copying many thousands of small files over a fairly slow link), and actually timed using XCOPY vs. Cygwin methods (cp in my case). It didn't make a significant difference. Ultimately what I think you run into in these sorts of situations is that you bump up against the slowness of the link (or physical disk) because, POSIX emulation or not, all your caches do is thrash.
I think your situation is vastly different. Granted copying over a network connection will be soo much slower that the additional overhead of Cygwin emulation fades away. But the OP was talking about copying between to supposedly local disks. With the network link out of the way the overhead of Cygwin emulation could become significant.

As for your cp across a link... At one time I was using cp to copy our release images (about 9 files of about 50-150 meg mind you) across a WAN link (from Santa Clara, California -> Shanghai, China). The link was real slow and it was talking ~30-45 minutes per image IIRC!

Then I changed it to an ncftp and the time per image dropped down to ~5 minutes.

Moral of the story? SMB is perhaps the worse way to transfer data across network links.

YMMV.
--
Andrew DeFaria <http://defaria.com>
I used to have an open mind but my brains kept falling out.


-- Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple Problem reports: http://cygwin.com/problems.html Documentation: http://cygwin.com/docs.html FAQ: http://cygwin.com/faq/


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]