[Bug nscd/19733] New: nscd_certainly_running flag mechanism broken on nscd shared cache

["ken.milmore at gmail dot com" <sourceware-bugzilla at sourceware dot org>]

https://sourceware.org/bugzilla/show_bug.cgi?id=19733

            Bug ID: 19733
           Summary: nscd_certainly_running flag mechanism broken on nscd
                    shared cache
           Product: glibc
           Version: 2.23
            Status: NEW
          Severity: normal
          Priority: P2
         Component: nscd
          Assignee: unassigned at sourceware dot org
          Reporter: ken.milmore at gmail dot com
                CC: drepper.fsp at gmail dot com
  Target Milestone: ---

Created attachment 9040
  --> https://sourceware.org/bugzilla/attachment.cgi?id=9040&action=edit
Test repeated calls to getpwuid_r(), for use with strace.

If nscd is used on Linux with the shared cache enabled, then after shutting
down nscd, any existing clients which have already mmap()ed a cache file
incorrectly continue using the stale cache file, and returning stale results
from getXXbyYY calls.  The problem affects both persistent and non-persistent
shared caches but the non-persistent case is more serious as the cache files
are one-time timporary files, so that even when nscd is restarted the client
continues using the old file.

The nscd shared cache header contains a flag, "nscd_certainly_running", which
is meant to be reset to zero when the nscd daemon stops. This is accomplished
via a tricky mechanism in setup_thread() (see
sysdeps/unix/sysv/linux/nscd_setup_thread.c) which uses the set_tid_address
syscall. Unfortunately this mechanism doesn't work, at least on Linux x86_64.
The tid address does not seem to get zeroed out on full process termination.  I
have tested this on several recent distributions (Arch glibc 2.23, Debian glibc
2.19, Centos7 glibc 2.17) and the behaviour is the same.

Evidence
========

The problem is most easily seen in persistent mode: 

# systemctl start nscd
# hexdump -C -n32 /var/db/nscd/passwd
00000000  02 00 00 00 78 00 00 00  00 00 00 00 01 00 00 00  |....x...........|
00000010  f5 a0 cf 56 00 00 00 00  00 00 00 00 00 00 00 00  |...V............|
00000020
# systemctl stop nscd
# hexdump -C -n32 /var/db/nscd/passwd
00000000  02 00 00 00 78 00 00 00  00 00 00 00 01 00 00 00  |....x...........|
00000010  f5 a0 cf 56 00 00 00 00  00 00 00 00 00 00 00 00  |...V............|
00000020

The nscd_certainly_running flag is in bytes 12-15, and it should be set to zero
when nscd is stopped.

Using strace on a test program such as the attached test.c which repeatedly
reads the passwd entry for the current user, the effects of the problem can be
seen:
1. Start nscd.
2. Compile the test # gcc -o test test.c
3. Run # strace ./test
   The strace shows the client mmap()ing the shared file, and after some
initial activity on /var/run/nscd/socket, the process settles down to reading
entries entirely from the cache.
4. From another terminal, stop nscd.
   The strace shows the client blissfully continuing to use the mmap; there are
no attempts to connect to /var/run/nscd/socket or manually read the password
file.  This goes on for some time...

Cause
=====

I can't be sure, but I think this kernel commit long ago may have removed the
possibility of using set_tid_address in this way:

[PATCH] Disable CLONE_CHILD_CLEARTID for abnormal exit

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=fec1d0115240593b39898289e6e1413ea6e44a84

I think this may actually effect clean shutdowns as well as aborts because
exit_group effectively sends a SIGKILL to all threads.

Resolution
==========

On all platforms other than Linux, nscd seems to rely on the date stamp in the
shared cache to indicate freshness.

We can revert to this behaviour quite easily by simply deleting
sysdeps/unix/sysv/linux/nscd_setup_thread.c and rebuilding nscd.  With this
change made, the behaviour improves somewhat:
- Shutting down nscd cleanly causes clients to immediately give up their shared
cache mappings (because the date stamp is set to 0 on nscd shutdown).
- A dirty kill (e.g. kill -9) doesn't alert clients immediately, however they
give up on the shared cache within 5 minutes (MAPPING_TIMEOUT) and revert to
doing their own lookups.

If the nscd_certainly_running machinery no longer works for ANY platform then I
guess it could be removed completely.

-- 
You are receiving this mail because:
You are on the CC list for the bug.