Re: [libc/string] State of PAGE_COPY_FWD / PAGE_COPY_THRESHOLD

[Andrew Pinski <pinskia at gmail dot com>]

On Thu, Nov 10, 2016 at 12:01 AM, Andrew Pinski <pinskia@gmail.com> wrote:
> On Wed, Nov 9, 2016 at 11:52 PM, Maxim Kuvyrkov
> <maxim.kuvyrkov@linaro.org> wrote:
>>> On Nov 10, 2016, at 11:48 AM, Andrew Pinski <pinskia@gmail.com> wrote:
>>>
>>> On Wed, Nov 9, 2016 at 11:39 PM, Maxim Kuvyrkov
>>> <maxim.kuvyrkov@linaro.org> wrote:
>>>>> On Nov 1, 2016, at 5:59 PM, Adhemerval Zanella <adhemerval.zanella@linaro.org> wrote:
>>>>>
>>>> ...
>>>>> $ cat sysdeps/x86/pagecopy.h
>>>>>
>>>>> #define PAGE_SIZE           4096
>>>>> #define PAGE_COPY_THRESHOLD PAGE_SIZE
>>>>>
>>>>> #define PAGE_COPY_FWD(dstp, srcp, nbytes_left, nbytes)  /* Implement it */
>>>>>
>>>>> It should work on any other architecture as well.  Now the question
>>>>> is whether this actually does make sense for Linux.  Hurd/mach provided
>>>>> a syscall (?) to actually copy the pages (vm_copy) which seems to apply
>>>>> some tricks to avoid full copy pages. By 'linux zero page sharing' are
>>>>> you referring to KSM (Kernel Samepage Merging)?
>>>>>
>>>>> If so, on a system without a provided kernel interface to work directed
>>>>> with underlying memory mapping (such as for mach), mem{cpy,set} will
>>>>> actually need to touch the pages and it will be up to kernel page fault
>>>>> mechanism to actually handle it (by identifying common pages and adjusting
>>>>> vma mapping accordingly). And AFAIK this are only enabled on KSM if you
>>>>> actually madavise the page explicit. So I am not grasping the need to
>>>>> actually implement page copying on Linux.
>>>>
>>>> Linux kernel has a reserved page filled with zeroes, so it there /were/ a syscall to tell kernel to map N consecutive pages starting at address PTR to that zero page, we could use that in GLIBC for really big memset(0).
>>>>
>>>> A quick investigation shows that there is no such syscall provided by the Linux kernel.  Doesn't mean we can't ask for / implement one.
>>>
>>> And then there would be a COW interrupt on the first write.  Not a
>>> good idea.  Since most likely you are writing zeros to a big page for
>>> security reasons before filling it again with other data.
>>
>> I'm looking at this as a possible performance optimization for a well-known benchmark.
>
> Please don't do it unless you benchmark real workloads.  Doing this
> for a benchmark is not a good.  Please use something like WRF, mysql,
> hadoop, spark or any other real workload that does lots of
> memset/memcpy.  Please don't do this just for a well-known broken
> benchmark.  Seriously this is just a broken benchmark anyways.
>
>>
>>>   That mean
>>> each page would need to be copied which is normally slower than
>>> zeroing in the first place.
>>
>> It may be like you say, or it may be a significant performance improvement.  I want to see numbers before deciding on how useful this may be.
>
> Copying is always slower than doing setting zero.  There are
> instructions on most major arch (including AARCH64) for zeroing a
> cache line.  Copying means loading one cache line to L1 and then doing
> stores.  Yes you can mark the cache line as not going to be used later
> but that still means going to the cache.

Also memcpy/memset is useful to optimize for each micro-arch instead
of doing this kind of optimization is better in general.  I have a
semi-optimized memcpy for ThunderX (T88, not T81 or T83; still have
not looked into an optimized version for T83/T81 yet) and have an idea
how to optimize it for another core but there is no way to handle this
in glibc because of kernel or glibc infrastructure for arm64.

Thanks,
Andrew

>
> Thanks,
> Andrew
>
>>
>>>
>>> COW is only useful when most of the pages will not be written to; it
>>> is not useful when doing memcpy or memset.  Mainly because you don't
>>> need to take the overhead of taking an interrupt twice (a system call
>>> is still an interrupt).
>>
>>
>> --
>> Maxim Kuvyrkov
>> www.linaro.org
>>