This is the mail archive of the ecos-patches@sources.redhat.com mailing list for the eCos project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Redboot jumb patch


>>>>> "Andrew" == Andrew Lunn <andrew.lunn@ascom.ch> writes:

    <snip>

    >> As for verification - this is a common problem. Since we no
    >> longer have access to the Red Hat test farm (which had pretty
    >> much one of every board type) to test against, we'll have to
    >> just make sure that any affected platforms can still build and
    >> hope that the community provides feedback when things break. Of
    >> course, if you have access to a particular platform, it should
    >> be checked as much as possible.

    Andrew> What involved in getting such a thing up and running. How
    Andrew> much of the software was public domain? Even if we cannot
    Andrew> get the whole system going, just being able to build the
    Andrew> images would be a big step forward. I guess have a
    Andrew> complete test farm in one place is not likely, but maybe
    Andrew> we could have a number of smaller test farms at
    Andrew> contributers premises who have spare boards laying around.

The eCos testfarm infrastructure was never released. I believe this is
probably a good thing on the whole, IMHO there were so many problems
with that code that it would be better and possibly even quicker to
start again, rather than try to build a new test farm with the old
code. This is one of the projects I hope to work on in the coming
weeks - although I am not going to promise any ETA.

The first step will be to implement "make check" functionality, so
that it is actually possible to run through the tests easily. Back in
the early days of eCos I wrote a Tcl script which did much of this,
and which in fact I still use for testing on the synthetic target.
This needs to be generalized, and I'll probably switch to using expect
rather than vanilla Tcl. Linux-only for now, I may or may not bother
getting it working and robust under Windows.

As Gary mentioned elsewhere, resetting boards is a particular problem.
There are various approaches. If the test worked ok then breaking out
of gdb stubs back to the RedBoot prompt is usually enough. Otherwise
you may need to "hit" a reset button, which for automated testing is
likely to involve relays. If there is no reset button or if the
hardware is in a seriously confused state, the test code would need to
resort to power cycling, either using a relay to control the DC input
or an X10 controller for mains. A problem with using relays is that
there is no real industry standard hardware. I have bought an
electronics kit that needs soldering together and which hopefully I
can drive from a Linux box, but chances are that other people will use
different hardware. X10's are better in this regard but we did have
reliability problems - they are not intended for 24*7 operation in a
test farm, being switched every few seconds.

Another issue that needs addressing is much better support for I/O
testing. Because the new code will be entirely script-based, it should
be much easier to allow for that and make the system extensible.


Given this "make check" functionality, it should be possible for
various people to set up cron jobs and do automated testing of the
current anoncvs trunk on whatever boards they have available. They
would have to look at the results themselves, and we could set up a
suitable mailing list for reporting regressions.

A next stage could be a centralized database to store the test
results, with a web interface for analysis of course. The outputs from
various cron jobs could then be emailed to the central location, to be
included in the database. There are lots of potential problems here,
not least security, and I have little experience with this sort of
thing.

More advanced testing would require some sort of scheduler per
miniature testfarm, deciding what tests should run next. Examples
include: a basic test run for a default configuration; or for some
other configuration; or a stress test; or an I/O test; ...

However, the immediate goal is  "make check" functionality.

Bart


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]