This is the mail archive of the
libc-alpha@sourceware.org
mailing list for the glibc project.
Re: [RFC][PATCH 22/22] Add Infinity notes implementing td_ta_map_lwp2thr
- From: Adhemerval Zanella <adhemerval dot zanella at linaro dot org>
- To: Gary Benson <gbenson at redhat dot com>
- Cc: libc-alpha at sourceware dot org
- Date: Thu, 22 Oct 2015 09:22:14 -0200
- Subject: Re: [RFC][PATCH 22/22] Add Infinity notes implementing td_ta_map_lwp2thr
- Authentication-results: sourceware.org; auth=none
- References: <1445352975-17844-1-git-send-email-gbenson at redhat dot com> <1445352975-17844-23-git-send-email-gbenson at redhat dot com> <56277179 dot 5060502 at linaro dot org> <20151021144250 dot GA18821 at blade dot nx> <5627BEBA dot 5040100 at linaro dot org> <20151021190911 dot GB25328 at blade dot nx> <5627ED06 dot 5040505 at linaro dot org> <20151021212857 dot GA10785 at blade dot nx> <F688AA9A-97E8-4271-817A-C4F4E0201D78 at linaro dot org> <20151022085645 dot GA21239 at blade dot nx>
On 22-10-2015 06:56, Gary Benson wrote:
> Adhemerval Zanella wrote:
>>> Em 21 de out de 2015, às 19:28, Gary Benson <gbenson@redhat.com> escreveu:
>>> Adhemerval Zanella wrote:
>>>>> On 21-10-2015 17:09, Gary Benson wrote:
>>>>> Adhemerval Zanella wrote:
>>>>>> AFAIK the only non-trivial build dependency for testing
>>>>>> is perl and I do not see a way forward to add another
>>>>>> interpreter dependency. It will be even harder on
>>>>>> cross-compiling environment to do native testing with such
>>>>>> dependency.
>>>>>
>>>>> I8C and I8X run on the host, not the target. You only need
>>>>> Python on the host. It's probably already there :)
>>>>>
>>>>> And on the host, if you have Python and pip installed then you
>>>>> can install I8C and I8X with a single shell command. It takes
>>>>> about 20 seconds on my <5Mbit connection and you don't need
>>>>> root access.
>>>>
>>>> That is not the case where you are bootstrapping on a platform
>>>> that does not have python enabled.
>>>
>>> Don't people usually cross-build to bootstrap new platforms?
>>
>> So the testing itself does not require to run Python natively? If it
>> is the case where you can the i8x tests in the build machine I agree
>> that it is not an impending reason.
>
> Neither the compile step nor the test step require Python to run
> natively. I8C and I8X both run only on the build host.
>
>>>>>> So the compelling reason I see so far to do not make
>>>>>> assembly releases and to not add the python as build/tests
>>>>>> requirement is the testcases. How hard would be to add a
>>>>>> non-python testcase on GLIBC to test such features?
>>>>>
>>>>> I8X has 1230 statements (lines of code that do something, as
>>>>> opposed to comments, blank lines and documentation). I don't
>>>>> have figures for how Python compares to C in terms of line
>>>>> count but I figure you're looking at 2500-5000 lines of C and
>>>>> that's not including the testcases themselves or something to
>>>>> parse and execute them if you go that route. Easily a month
>>>>> of work, and the result would likely be less flexible.
>>>>
>>>> I was thinking more like something a objdump or similar to just
>>>> dump the notes in a meaningful way.
>>>
>>> With respect that's not really the same thing at all. The I8X
>>> testcase executes the functions multiple times with the correct
>>> endianness and wordsize for the target. The generated functions
>>> are different for each platform too--they have structure offsets
>>> and other constants embedded in them, for example, and the note
>>> in this patch has three different variants depending on how the
>>> target does THREAD_SELF. You can't even compare the operations
>>> because if a field in a structure has a zero offset on one
>>> platform and a nonzero offset on another then the platform with
>>> the nonzero offset will have extra adds that the compiler
>>> optimized away on the platform with the zero offset.
>>
>> Unless it requires runtime information where you can't obtain prior
>> hand due kernel/hardware interaction I do not see why this is not a
>> matter of parse the correct fields and information in the notes.
>
> Sure, but what you're talking about is essentially recreating the
> note compiler. I8C is currently 2699 Python statements plus a 917
> statement testsuite, so over 3500 Python statements to translate
> into a presumably less-expressive language.
>
>> However I see that the infrastructure is already created using i8x,
>> so it would be a duplicate effort.
>
> Yeah. And I8X is testing a different thing too. It's not checking
> the bytecode you got is the bytecode you expected, it's checking the
> result of executing the bytecode is the result you expected.
That's why I asked if you need to run natively, since the expected results
will only vary if you have runtime constraints. But I do see that changing
the offsets or internal structures will probably ending to adjust the
tests expectations also.
>
>>>> I think one strategy that we can use to decouple the need for a
>>>> python binary in build is:
>>>>
>>>> 1. Provide assembly generated notes from a defined I8X compiler
>>>> along with I8X sources.
>>>> 2. Add a makefile rule to autogenerate the assembly file if a I8C
>>>> compiler is presented in the system.
>>>> 3. For tests, only run is I8X is presented.
>>>
>>> That's not really viable for the reasons I detailed above. You
>>> would have to ship generated assembly for every platform, and even
>>> that is not enough. For example, on some platforms "sizeof
>>> (struct pthread)" is a constant that gets into notes. If someone
>>> adds a field to struct pthread and doesn't regenerate the notes
>>> then you're shipping GLIBC with broken notes. Are there CFLAGS
>>> the user could set that would change structure offsets? Setting
>>> these would mean broken notes if you didn't have I8C installed.
>>> There are constants in kernel headers that end up in notes. If
>>> one of these changes, you're shipping GLIBC with broken notes.
>>>
>>> There are so many silent failure modes here--and broken notes are
>>> far worse than no notes at all. The only robust way to do this is
>>> to build the notes with the constants and offsets that you are
>>> building the rest of GLIBC with, and that means saying if you want
>>> a GLIBC with notes then you need to install this package on your
>>> build system.
>>
>> This can be done with proper documentation and in the release cycle:
>> the release maintainer will recreate the assembly annotations for
>> the release architectures. My only concern is the kernel notes
>> dependency, however even those we need to take care since you also
>> do not know prior hand which version you will end using.
>
> That's fragile. It's also burdensome for the release maintainer.
> It also raises the barrier for people hacking on GLIBC, because not
> only do they need to know all the other things they need to know, they
> also now need to know about recreating notes. Say somebody is hacking
> on NPTL, they don't know about updating notes, so the first thing they
> see is the debugger won't attach. How long do they spend going round
> in circles until somebody says "oh, you need to run this command in
> this directory."?
>
> It also doesn't solve the problem that if people are working without
> I8C and I8X installed then they are creating GLIBCs containing
> unchecked notes. My position is users create GLIBCs containing
> fully-tested notes or they create GLIBCs creating no notes at all.
>
> In either setup there's a process to create notes. What you're
> saying is, "let's encode this process in documentation and let
> people figure it out." What I'm saying is, "let's encode this in
> code and let the build host figure it out."
Yes it is a tradeoff of adding external dependency and making GLIBC
build independently of them. Also the exported NPTL structures do
not change in a dynamic way (struct pthread, for instance, changes
it's size twice in seven years).
Also, currently if someone is hacking on GLIBC and he will also need
to adjust libpthread_db anyway. I know it is not the best approach
and I do want to see a better strategy to handle it, but adding an
external tools with python dependency IMHO is not the best one.
I would expect such tool to be included in binutils and have only
binutils own dependencies as requisites. I would also expect notes
definition language stability, meaning that a notes defined with a
older version will always be handled by new compiler version
(so building a older glibc notes version with a new notes compiler
won't break the build).
>
> Thanks,
> Gary
>