Re: rsync very slow, but not a network issue

[Alexis Gallagher <alexis at alexisgallagher dot com>]

Reini (and Steven),

Okay, I get. This benchmark you introduced below compares rsync on two 
variables: copy speed locally vs over the network, and copy speed for 
copies onto indentical target files vs different target files. And your 
benchmark showed that rsync was faster for copies onto pre-existing 
target files -- just as rsync is supposed to be.

I just ran the same benchmark on my side and I get exactly the opposite 
results. For what it's worth, here's what I got. I started with two 
different binary files (based on different mp3's) each of size 434006.

%[local copy, target file is different ]
%time rsync test1.mp3 test2.mp3

real 0m6.422s
user 0m0.430s
sys  0m1.431s

%[local copy, target file is the same ]
%time rsync test1.mp3 test1-same.mp3

real 0m6.523s
user 0m0.300s
sys  0m1.261s

%cp test2.bak test2.mp3
%[network copy, a pure copy with scp for benchmark]
%time scp test2.mp3 $TEST:

260.6KB/s (calculated average)
real 0m16.257s
user 0m0.510s
sys  0m1.091s

%[network copy, target file is different ]
%time rsync test1.mp3 $TEST:test2.mp3

real 0m13.797s
user 0m1.421s
sys  0m1.250s

% (network copy, files are 100% the same)
% (repeated twice to guarantee target file is there)
%time rsync test2.mp3 $TEST:test2.bak

real 0m28.329s
user 0m1.390s
sys  0m4.766s

%rsync --version
rsync version 2.6.2 protocol version 28
ssh $TEST rsync --version
rsync version 2.6.2 protocol version 28
$ ssh -V
OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
$ ssh $TEST ssh -V
OpenSSH_3.7.1p1, SSH protocols 1.5/2.0, Open SSL 0.9.7d 17 Mar 2004
(but ssh -v reports TEST's sshd server is responding with
OpenSSH_3.6.1p1+CAN-2003-0693, SSH protocols 1.5/2.0, Open SSL 0x0090702f )
$ uname -r
1.5.10(0.116/4/2)

As you can see, my network copy onto an identical waiting file is taking 
almost twice as long as a network copy onto a _different_ waiting file, 
and is also substantially slower than a pure copy. So I'm experiencing a 
net rsync slowdown rather than a speedup.

Is it the processor? Thorsten Kampe And Steven Hartland suggested the 
problem is in effect that my computer is too slow, and that I should 
turn off all checksumming. I seem to have more bandwidth to my TEST 
machine than you, so I suppose my processor might be the bottleneck for 
me but not for you, which would explain our different results. How fast 
was your machine?

Or Is it cygwin? I notice that the enormous slowdown for the network 
copy onto an identical target  is all appearing in the 'real' component 
of the time's timing report, and that there's a bit of extra time in 
'sys' but other than that there's nothing to account for it. I'm not 
sure how to interpret that. Would cygwin-specific delays appear in 
'sys'? In other words, does this mean the checksums are using up all the 
time, and confirm that my processors are too slow to benefit from rsync?

Best wishes,
Alexis


Reini Urban wrote:

> So check if the net overhead in the cygwin version is broken?
> Could be easily tested out with two local files:
>
> head -c 2100000 < /dev/random >test1.mp3
> head -c 2100000 < /dev/random >test2.mp3
> cp test1.mp3 test1-same.mp3
> cp test2.mp3 test2.bak
> time rsync test1.mp3 test2.mp3
>
>   real    0m0.518s
>   user    0m0.201s
>   sys     0m0.171s
>
> time rsync test1.mp3 test1-same.mp3
>
>   real    0m0.524s
>   user    0m0.154s
>   sys     0m0.263s
>
> RSYNC_RSH=ssh
> TEST=othermachine
> cp test2.bak test2.mp3
> time scp test2.mp3 $TEST:
>
>   63.2KB/s
> real    0m36.619s
> user    0m0.170s
> sys     0m0.233s
>
> # 100% different
> $ time rsync test1.mp3 $TEST:test2.mp3
> Server is very old version of rsync, upgrade recommended.
>
> real    0m37.162s
> user    0m0.388s
> sys     0m0.202s
>
> # 100% same
> $ time rsync test2.mp3 $TEST:test2.bak
> Server is very old version of rsync, upgrade recommended.
>
> real    0m7.298s
> user    0m1.201s
> sys     0m2.217s
>
> 6x faster than scp, with same data.
> 0.5s (1.4%) slower than scp, with complete random data.
>
> $ rsync --version
> rsync  version 2.6.2  protocol version 28
> $ ssh $TEST rsync --version
> rsync version 2.4.6  protocol version 24
> $ ssh -V
> OpenSSH_3.8.1p1, OpenSSL 0.9.7d 17 Mar 2004
> $ ssh $TEST ssh -V
> OpenSSH_3.7.1p2, SSH protocols 1.5/2.0, OpenSSL 0.9.6c 21 dec 2001
> $ uname -r
> 1.5.10(0.116/4/2)
> --
> Reini Urban
> http://xarch.tu-graz.ac.at/home/rurban/



I see the final result you


--
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple
Problem reports:       http://cygwin.com/problems.html
Documentation:         http://cygwin.com/docs.html
FAQ:                   http://cygwin.com/faq/