systemtap 2.9 release

["Frank Ch. Eigler" <fche at redhat dot com>]

The SystemTap team announces release 2.9!

  More compiler optimizations, prologue-searching option, backtracing
  with kernel symbols without debuginfo, callee probe extensions,
  STAP_PRINTF embedded-C macro, improved debuginfo-related
  diagnostics, and more tapset functions, reduced translator memory
  usage.


= Where to get it

  https://sourceware.org/systemtap/ - our project page
  https://sourceware.org/systemtap/ftp/releases/systemtap-2.9.tar.gz
  https://koji.fedoraproject.org/koji/packageinfo?packageID=615
  git tag release-2.9 (commit 7009f2f782e5)

  There have been over 232.66 commits since the last release.
  There have been between 50 and 500 bugs fixed / features added
  since the last release.  Closer to 50.

= How to build it

  See the README and NEWS files at
  https://sourceware.org/git/?p=systemtap.git;a=tree

  Further information at https://sourceware.org/systemtap/wiki/


= SystemTap frontend (stap) changes

- New --prologue-searching[=WHEN] option has been added to stap with '-P' being
  its short counterpart.  Using --prologue-searching=never turns prologue
  searching deliberately off working around issue of int_arg() returning wrong
  value when a 32-bit userspace binary having debug info is being probed with
  active prologue searching.

- Callee probe points now support '.return' and '.call' suffix.
  For example,
    process("proc").function("foo").callee("bar").return
  will fire upon returning from bar when called by foo.
    process("proc").function("foo").callee("bar").call
  will only fire for non-inlined callees.

- SystemTap has reduced its memory consumption by using interned_strings (a
  wrapper for boost::string_ref) in place of std::string instances. The change
  is to reduce the number of duplicate strings created by replacing them with
  interned_strings which act like pointers to existing strings.


= SystemTap script language changes

- Embedded-C functions may now use the new STAP_PRINTF(fmt, ...)
  macro for output.

- Embedded-C functions with parameter arity-0 can now be marked with
  the /* stable */ /* pure */ pragmas, if (roughly speaking) the
  function is side-effect-free and idempotent.  The translator may
  execute these speculatively and have their results memoized.  This
  lets probes with multiple calls to such functions run faster. 
  
  Context variable ($foo) getter functions (in non-guru mode), and
  numerous tapset functions are now marked as /* stable */ /* pure */.
  Several example scripts have been modified to eschew explicit
  memoization.


= SystemTap runtime changes

- SystemTap now uses symbols from /proc/kallsyms when kernel debuginfo is not
  available.

- When using the procfs .maxsize() parameter, an error will be generated if the
  buffer is too small.


= SystemTap tapset changes
  
  logging.stp     new assert() function
                  many uses of error() in registers.stp 
                  have been replaced in favor of assert()
  dentry.stp      new functions fullpath_struct_path() and
                  fullpath_struct_nameidata() resolve full path
                  names from internal kernel struct pointers,
                  fullpath_struct_file gets the full path to root
  task.stp        new function task_fd_lookup gets the file struct
                  for a task's fd 
  registers.stp   new functions arch_bytes() and uarch_bytes()
                  to obtain address size of kernel and user space
  switchfile.stp  new function switch_file() allows control over
                  rotation of output files

- The [nd_]syscall tapset got autodocumented. Related paragraph got added to PDF
  and HTML tapset reference. Also a new tapset::syscall 3stap man page got added.

- The following tapset variables and functions are deprecated in
  version 2.9:
  - The '__int32_compat' library macro got deprecated in favor of
    new '__compat_long' library macro.
  - The 'uargs' convenience variable of the 'seccomp' syscall probe
    got deprecated in favor of new 'uargs_uaddr' variable.
  - The powerpc variant of nd_syscall.compat_sysctl got deprecated on favor of
    nd_syscall.sysctl32. This aligns the nd_syscall to its respective syscall and
    to ia64/s390/x86_64 variants too.


= SystemTap sample scripts

- New samples:
  switchfile.stp  every second print a log message 
                  and switch log files every 5 seconds.
  slowvfs.stp     prints a line for every kernel vfs_open
                  operation that takes longer than a
                  configurable number of microseconds.


= Examples of tested kernel versions

  2.6.18 (RHEL 5 x86 and x86_64)
  2.6.32 (RHEL 6 x86 and x86_64)
  3.10.0 (RHEL 7 x86_64)
  4.1.6  (Fedora 22 x86_64)
  4.3.0-rc3 (Fedora rawhide x86_64)

= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
  broad kernel function wildcards.  (PR2725)

- 32-on-64 bit userspace unwinding is truncated on older kernels, such
  as 2.6.32 (PR15757)

- The dyninst backend is still very much a prototype, with a number
  of issues, limitations, and general teething woes. For instance:
  + lack of support for multiarch/cross-instrumentation
  + tapset functions are still incomplete relative to what is supported
    when the kernel backend is active
  + exception handling becomes completely broken in programs
    instrumented by the current version of dyninst (PR14702)
  + not all registers are made available on 32-bit x86 (PR15136)

  See dyninst/README and the systemtap/dyninst Bugzilla component
  (http://tinyurl.com/stapdyn-PR-list) if you want all the gory                 
  details about the state of the feature.


= Contributors for this release

  Abegail Jakop, David Smith, Felix Lu, Frank Ch. Eigler,
  Ivan Diorditsa*, Jose Castillo*, Josh Stone, Lukas Berk,
  Mark Wielaard, Martin Cermak, Mikhail Kulemin*, Nicolas Brito*
  Snehal Phule*

  Special thanks to new contributors, marked with '*' above.
  Special thanks to Felix Lu for compiling these notes.


= Bugs fixed for this release <https://sourceware.org/PR#####>

  909   perf counter events, perfmon? kernel API
  2111  document syscalls tapset
  10487 flight recorder control from script
  10977 Getting the address size used in a module
  11263 exposing foo32 syscalls
  12151 support /* stable */ embedded-c pragma
  13664 support dwarf types for stap variables
  15972 core dump with process probes
  16493 Improve bkl.stp to add backtrace
  16968 bad formatting in many help pages for probes
  17831 kprobes_onthefly.exp fails on powerpc
  17893 el6: cannot stat `build/en-US/pdf/*SystemTap_Beginners_Guide*.pdf': No such file or directory
  17920 File descriptor to pathname function
  17921 kernel backtrace missing /proc/kallsyms symbols
  18455 const_folder::visit_binary_expression hurting type inference
  18462 macro deprecation
  18503 procfs .maxsize() overflow should generate error
  18555 ppc64le: can't probe demangled C++ function names
  18562 the listing_mode.exp test case has lots of errors on systems without uprobes
  18563 on ppc64, the mbrwatch.stp example script fails when tested
  18571 Tapset support and test coverage for bpf and seccomp syscalls.
  18577 on rhel7, listing_mode_sanity.exp always gets a failure when doing 'stap -l **'
  18597 long_arg() doesn't correctly handle negative values in 32-on-64 environment
  18598 stap_staticmarkers.stp tapset has no test case
  18630 dwarfless parameters from a uprobe need test coverage
  18634 on rawhide, using timer probes gets a compilation error
  18649 int_arg() misbehaves on x86[_64] for 32-bit uprobe in binary having debuginfo
  18650 powerpc variant of longlong_arg() for uprobes swaps the high and low half of its 64bit retval
  18651 Possible nd_syscall tapset cleanup based on PR18597 fix
  18711 Pass 4 failure on RHEL7 for examples netfilter_summary and netfilter_drop
  18751 support a STAP_PRINTF(....) macro for use in embedded-C functions
  18769 [ppc64BE/--dyninst] unknown operator @__compat_task
  18827 consistency check for syscall and nd_syscall tapset
  18856 nfsd.close probe alias fails on rawhide
  18885 Use /* unmodified-fnargs */ in tapsets
  18889 lost ability to probe kernel module initializers
  18936 script cache will fail if $jiffies is referenced
  18942 any script will include all the globals from tapset/argv.stp
  18944 the ioblock.stp tapset fails to compile on RHEL7
  18971 process_by_pid.exp issues
  18999 error("") stall (causing similar assert() stall)
  19000 several task tapset functions can cause kernel crash
  19021 the tapset function task_dentry_path() should handle more than just files
  19043 __bio_ino(), __rqstp_gid() and __rqstp_uid() can crash the kernel
  19045 kernel_string_quoted() can crash the kernel
  19057 _is_reset() can crash the rhel6 / s390 kernel
  19065 task_fd_lookup() can crash the s390x kernel when invoked with an invalid input
  19069 task_euid() doesn't compile on aarch64
  19070 Call to __ustack_raw(0) causes 'Unknown symbol in module' on rhel6-