Re: Optimizing away "ReadFile" calls when Make calls stat()

["Larry Hall (RFK Partners, Inc)" <lhall at rfk dot com>]

At 01:36 PM 2/13/2001, Jonathan Kamens wrote:
>We use Cygwin to develop a large product (running a build and the test
>suites takes about two hours on a very fast machine); our builds are
>driven by GNU Make.  We compile and test the same product under Linux.
>We've found that builds under Cygwin run several times slower than
>builds under Linux, even on machines of comparable speed, RAM, etc.
>The slowness is seriously impacting the productivity of our developers
>who work on Windows, so we're searching for any way we can to speed up
>Cygwin builds.
>
>We've found that one of the biggest culprits in slowing down the
>Cygwin builds is Make.  The problem is that every time Make does
>stat() to find out the modification time on a dependency to determine
>whether or not its dependents need to be rebuilt, Cygwin calls
>ReadFile on the file twice -- once to determine whether it's a
>symbolic link, and a second time to determine whether it should appear
>to be executable according to stat().  We have thousands of
>dependencies in our Makefiles, and many of those dependencies
>frequently live on network drives, so these calls to ReadFile
>seriously slow things down.
>
>We don't use symbolic links anywhere in our source tree or build
>tree, and Make doesn't really care whether a file is executable when
>deciding whether it is newer than one of its dependents, so both of
>these calls to ReadFile are totally unnecessary to us.  As an
>experiment, I added code to the Cygwin DLL to allow these ReadFile
>calls to be temporarily disabled, and then I compiled a modified
>version of Make which disables the ReadFile calls before calling
>stat() and then turns them back on.
>
>To measure the effect of these changes, I ran "make all" in a build
>tree tha was already completely built, so that I would be timing only
>the work Make does to check dependencies, rather than timing actual
>build work.  With the unmodified Make, "make all" takes around six
>minutes; with the modified Make, it takes around three.  We consider
>this a significant improvement.  (However, note that on Linux, "make
>all" when nothing needs to be done takes only 17 seconds, so clearly
>there's still a lot of room for improvement under Cygwin.)
>
>I'm wondering if the maintainers of Cygwin would be willing to
>consider incorporating these changes, if I submit them, into the
>Cygwin DLL and the Cygwin version of Make.  I'm thinking that the DLL
>changes would actually need to be split into two flags -- one to say,
>"Don't call ReadFile to find out whether a file is executable, because
>I don't care about that," and the other to say, "Don't call ReadFile
>to find out if a file is a symbolic link, because I know I'm not using
>any symbolic links."  Then, GNU Make on Cygwin could always set the
>first flag, and it could set the second flag if the user specified
>"--nosymlinks" or something like that.
>
>I realize that this is a bit gross.  However, (a) surely it isn't much
>more gross than storing symbolic links inside files and reading files
>to determine whether they should look executable :-), and (b) it
>really does give a drastic performance improvement for the small price
>of not using symbolic links in your source or build tree.
>
>Please comment.



I know this would only address half the problem but I wonder if it would make
sense to cache the results of ReadFile() so that separate checks for symbolic
links and executables would result in only 1 ReadFile() call.  This seems
like a nice general optimization which wouldn't be so "gross"...



Larry Hall                              lhall@rfk.com
RFK Partners, Inc.                      http://www.rfk.com
118 Washington Street                   (508) 893-9779 - RFK Office
Holliston, MA 01746                     (508) 893-9889 - FAX



--
Want to unsubscribe from this list?
Check out: http://cygwin.com/ml/#unsubscribe-simple