This is the mail archive of the
cygwin
mailing list for the Cygwin project.
Re: rsync performance degradation? Could be a windows issue?
- From: Greg Freemyer <greg dot freemyer at gmail dot com>
- To: cygwin at cygwin dot com
- Date: Mon, 11 Apr 2016 14:20:35 -0400
- Subject: Re: rsync performance degradation? Could be a windows issue?
- Authentication-results: sourceware.org; auth=none
- References: <CAGpXXZJV-nsB=xyO7=EBUb-JCk5KqMPS=bvg2aWgw_jm4OZFsQ at mail dot gmail dot com> <CAGpXXZLJ=O7y-6YEcODv4r32X96zGSU5PX9KwzwF3QedSV4iHA at mail dot gmail dot com> <570BE03D dot 7030709 at gmail dot com>
On Mon, Apr 11, 2016 at 1:34 PM, Marco Atzeri <marco.atzeri@gmail.com> wrote:
>
>
> On 11/04/2016 18:57, Greg Freemyer wrote:
>>
>> On Thu, Mar 31, 2016 at 4:48 PM, Greg Freemyer <greg.freemyer@gmail.com>
>> wrote:
>>>
>>> All,
>>>
>>> I'm not sure this is on-topic, but at least I'm in cygwin when I
>>> notice the below:
>>>
>>> I do a lot of large data transfers between USB drives. Often I get
>>> great speeds (70MB/sec or more).
>>>
>>> Sometimes it falls way off to closer to 20MB/sec with the same class
>>> of hardware.
>>>
>>> I experienced the "slow" transfer speed today, so I thought I'd ask if
>>> anyone knew a way to resolve it?
>>
>>
>> I'm trying rsync of a bunch of large files again today.
>>
>> Initially I saw the same slow copy speed (about 20 MB/sec). My
>> destination was connected to a USB 3.1 port, but it seemed to be the
>> bottleneck so I moved it to a USB 3 port (3.1 should be 2x the speed
>> of 3.0, so it should not have been the bottleneck).
>>
>> After getting my throughput up above 50 MB/sec I notice in the windows
>> "performance monitor" that my source disk is hitting 100% utilization,
>> then a few seconds later my destination disk is. And back and forth.
>> It seems I'm only reading or writing for a few seconds, then
>> alternating.
>>
>> I assume the issue is that too much data is being read / cached by
>> rsync prior to it being written out so I'm getting no advantage of
>> reading and writing in parallel.
>>
>> (I gather iostat isn't available for cygwin?)
>>
>> Is there some rsync (or cygwin) option / feature that would encourage
>> parallel reading/writing?
>>
>> fyi: I did some linux testing with "dd" over the weekend and I hit 140
>> MB/sec if I used a 100MB blocksize. That was to / from the raw disk
>> (/dev/sda => /dev/sdb).
>>
>> If I bumped my blocksize to 1GB for dd in linux, my throughput dropped
>> to 70 MB/sec just as I see right now with rsync in cygwin.
>>
>> Thanks
>> Greg
>>
>
>
> Is windows robocopy faster ?
As I test, I just copied 30 GB of 1.5 GB files via robocopy. None of
the files had been accessed since a reboot, so none should have been
in cache.
According to Resource Monitor, yes.
I'm getting about 105 MB/sec for read and 105 MB/sec for write vs 70
MB/sec for rsync.
Significantly, with robocopy Resource Monitor is showing a consistent
read/write speed. With rsync read and write fluctuate back and forth.
As I stated before, I believe I should be able to get 140 MB/sec, but
I wouldn't complain about 105 MB/sec. 70 MB/sec just seems to
highlight a flaw in how rsync manages the data flow since it is 50% of
theoretical max.
ie. this seems to be what rsync in cygwin is doing:
while (files) {
read 1.5 GB file to ram
write 1.5 GB file from ram
fsync() ensure 1.5 GB file is on disk
} endwhile
I haven't tested in linux. Maybe rsync just isn't as efficient as I expect?
There is also the problem that rsync often slows down to below 50
MB/sec I've tried it with no other activity on the PC. Today I was
only getting 20 MB/sec at first. (I moved the USB cable and it came
up to 70 MB/sec).
Thanks
Greg
--
Problem reports: http://cygwin.com/problems.html
FAQ: http://cygwin.com/faq/
Documentation: http://cygwin.com/docs.html
Unsubscribe info: http://cygwin.com/ml/#unsubscribe-simple