Re: [RFC][PATCH] Rerun benchmarks only if *.out file does not exist.

[Siddhesh Poyarekar <siddhesh at gotplt dot org>]

On Monday 25 September 2017 10:47 PM, Steve Ellcey wrote:
>> Two questions: One, does this sound useful to other people?  And two, what
>> is wrong with my patch?  This patch mostly works but the bench-memcpy,
>> bench-memcpy-large, and bench-memcpy-random tests get rerun everytime even
>> if the .out files for those tests already exist.  I don't know why.  I am
>> guessing is thas something to do with their common name prefix and the string
>> manipulations done to test names in the Makefile but memset does not seem
>> to have that problem.
<snip>
> 
> Well, you can ignore the question about what is wrong with the patch, I
> was removing the memcpy*.out files and that was why they got rerun.  I
> didn't see the rm command in my script and got confused.
> 
> While testing my changes I noticed that 'make bench-clean' does not
> remove the *.out files.  It removes the executables and object files
> but not the *.out result files.  So with my change 'make bench; make
> bench-clean; make bench' resulted in the benchmarks getting rebuilt but
> not rerun.  I don't know if the decision to not remove *.out files was
> intentional or an oversight.

That is intentional, so is the decision to re-run benchmarks despite the
presence of the *.out files.  In fact there's a different kind of bug
there, which is that the current .out file should be backed up like the
main bench.out is to bench.out.old and I may have missed doing that for
string benchmarks.  That provides a convenient way to do before-after
comparison for patches.

If you want to run only specific benchmarks or benchsets, I suggest
making that a command line parameter, something like:

  make BENCHSETS=string-benchset

or

  make BENCH="sin cos asin acos"

Siddhesh