Re: [RFC][PATCH 22/22] Add Infinity notes implementing td_ta_map_lwp2thr

[Adhemerval Zanella <adhemerval dot zanella at linaro dot org>]

On 21-10-2015 17:09, Gary Benson wrote:
> Adhemerval Zanella wrote:
>> On 21-10-2015 12:42, Gary Benson wrote:
>>> Adhemerval Zanella wrote:
>>>> I like the idea of improving debugging capabilities with
>>>> something new than libpthread_db, however I do not see the best
>>>> approach of using the i8 files directly in GLIBC. If I
>>>> understood correctly from the link you posted with this
>>>> documentation, it is still a very WIP project with rough places
>>>> and the language/compiler is still being constantly updated.
>>>
>>> It is a work-in-progress right now.  It's being developed to allow
>>> GDB to debug multithreaded inferiors, so this patch series (or,
>>> rather, the complete version that will follow in the next few
>>> months) is the work that will turn Infinity from WIP to something
>>> complete.
>>>
>>> I mailed this series as an RFA to get some early feedback.  I'm
>>> anticipating that a prerequisite of a GLIBC commit will be a GDB that
>>> fully supports Infinity (and a prerequisite of a GDB commit will be a
>>> GLIBC with Infinity notes) and what I *didn't* want to do was appear
>>> out of the blue with a complete system I'd spent six months writing
>>> away by myself only for somebody to point out a killer flaw that meant
>>> the whole thing was junk.  Also because I recognise that input from
>>> the wider community will make the final result something way better
>>> than I could achieve on my own.
>>>
>>> So yes, it's very work-in-progress right now, but by the time it's
>>> ready to commit it should be pretty well baked and polished.
>>
>> I did say it was flawed or even half-baked, my concern is due it
>> uses a non well establish language it may add more maintaining
>> burden than use assembly drop releases. What happens if had or
>> decided to extend I8X to cover a new feature and add this dependency
>> on the GLIBC code?  Will you always provide backwards compatibility?
>> Will we need to tie GLIBC releases to some I8X release or will be
>> feature based?
> 
> I wasn't intending to ever break backwards compatibility.  That would
> mess things up for people (and I'd have to deal with it!)
> 
> If some new language feature was added then some configure check could
> be written, either by looking at "i8c --version" or by trying to
> compile some code that had the new feature and seeing if it failed.
> It would be much as we do for the other tools we use.
> 
>>>> Also I do not like to add another tool dependency for complete
>>>> GLIBC build and one in such state (GLIBC do use python for
>>>> benchmark but it is not really required for build afaik). So
>>>> which is the issue of shipping the ELF notes in assembly format
>>>> instead?
>>>
>>> The initial reason I opted for a compiled source approach is that
>>> the note format is somewhat complex.  The format is documented
>>> here:
>>>
>>>   https://sourceware.org/gdb/wiki/NoteFormat
>>>
>>> I want Infinity to be future-proof.  Now that libthread_db exists,
>>> GLIBC is essentially bound to support it *forever*.  I don't want to
>>> introduce another debugger interface that we'll have to support
>>> forever unless I can be fairly confident we won't have to replace that
>>> five or ten years down the line.  So there's a lot of requirements in
>>> the note format specification that (should!) mean that any new feature
>>> we require can be added without breaking existing note consumers.
>>>
>>> The second reason for using a compiler (or, more generally, tools) is
>>> testing.  We could ship the notes in assembly language but then how do
>>> you test them?  I don't like the idea of GLIBC shipping with untested
>>> code, especially since the result of a breakage might not be
>>> immediately obvious: if we ship a broken note in a GLIBC release it
>>> might well not be noticed until it makes its way into some distro and
>>> some user finds they can't debug their application because one of the
>>> thread-debugging notes is invalid.  That's why the required tools are
>>> a compiler *and* a unit tester.  I don't know if you saw, but the
>>> testcase for the note in this patch exercises every path through the
>>> compiled note.  It's not perfect, but it's as close as I can get.  So,
>>> each note I add to GLIBC will have an accompanying testcase so I can
>>> be as certain as I can be that if "make check" passed then the notes
>>> in the files will work.
>>>
>>> For me, the second reason (testing) is the more compelling of the two.
>>> And if you're going to require an external tool for testing, why not
>>> require an external compiler too?
>>
>> AFAIK the only non-trivial build dependency for testing is perl and 
>> I do not see a way forward to add another interpreter dependency.  It
>> will be even harder on cross-compiling environment to do native testing
>> with such dependency.
> 
> I8C and I8X run on the host, not the target.  You only need Python on
> the host.  It's probably already there :)
> 
> And on the host, if you have Python and pip installed then you can
> install I8C and I8X with a single shell command.  It takes about 20
> seconds on my <5Mbit connection and you don't need root access.

That is not the case where you are bootstrapping on a platform that does
not have python enabled.  

> 
>>> I appreciate that people don't want to add more build dependencies to
>>> GLIBC but I don't really have a way around that.  To mitigate the
>>> impact I've written the tools without using any external libraries,
>>> and I've written them to work on as wide a range of Python versions
>>> as is possible.  It's not possible to write code that works both
>>> for Python < 2.6 and for Python >= 3.0 simultaneously so 2.6 is the
>>> oldest you can use.
>>
>> So the compelling reason I see so far to do not make assembly releases
>> and to not add the python as build/tests requirement is the testcases.
>> How hard would be to add a non-python testcase on GLIBC to test such
>> features?
> 
> I8X has 1230 statements (lines of code that do something, as opposed
> to comments, blank lines and documentation).  I don't have figures for
> how Python compares to C in terms of line count but I figure you're
> looking at 2500-5000 lines of C and that's not including the testcases
> themselves or something to parse and execute them if you go that
> route.  Easily a month of work, and the result would likely be less
> flexible.

I was thinking more like something a objdump or similar to just dump
the notes in a meaningful way.

Anyway, I my only concern is setting python as a glibc dependency build
where IHMO does not really add anything.  I think one strategy that we
can use to decouple the need for a python binary in build is:

1. Provide assembly generated notes from a defined I8X compiler along 
   with I8X sources.
2. Add a makefile rule to autogenerate the assembly file if a I8C
   compiler is presented in the system.
3. For tests, only run is I8X is presented.


> 
> Cheers,
> Gary
>