SystemTap 2.8 release

[Abe Jakop <ajakop at redhat dot com>]

The SystemTap team announces release 2.8!

  Syscall coverage, improved netfilter probes, namespace-aware tapsets, JSON
  tapsets, new and improved man pages, additional golang support

= Where to get it

  https://sourceware.org/systemtap/ - our project page
  https://sourceware.org/systemtap/ftp/releases/systemtap-2.8.tar.gz
  https://koji.fedoraproject.org/koji/packageinfo?packageID=615
  git tag release-2.8 (commit 08dc718)

  There have been over 205 commits since the last release.
  There have been over 50 bugs fixed / features added since the last release.


= How to build it

  See the README and NEWS files at
  https://sourceware.org/git/?p=systemtap.git;a=tree

  Further information at https://sourceware.org/systemtap/wiki/


= SystemTap frontend (stap) changes

- Colorization of error string tokens is made more robust, especially
  in presence of $N/@N substitution.

- The function::*, probe::* and new macro::* man pages cross-references the
  enclosing tapset::* man page. For example:

  function::pn(3stap) mentions tapset::pn(3stap) in the SEE ALSO section

- New stapref(1) man page provides a reference for the scripting language. The
  stapref page contains an overview of the features available in the language,
  such as keywords, data types, operators and more.


= SystemTap script language changes

- The @task macro performs the very common @cast to a task_struct.

  The embedded-C bodies of task_current() and pid2task() are now wrapped
  by @task, which gives them a debuginfo type on the return value.  With
  autocast type propagation, this removes the need for any explicit @cast
  in many places.

  Other places which take untyped task pointers as parameters, for
  instance, now use @task as well to simplify their code.


= SystemTap runtime changes

- SystemTap has improved support for probing golang programs. Work has been
  done to be able to handle DWARF information, reporting file names, line
  numbers, and column numbers, and tolerance of odd characters in symbol names.


= SystemTap tapset changes

- New tapsets:
  json.stp            provides probes, functions, and macros to generate a
                      JSON metadata and data file. JSON metadata file is
                      located in proc/systemtap/MODULE/metadata.json. JSON data
                      file is located in /proc/systemtap/MODULE/data.json.
  json.stpm           ditto.

- Changed tapsets:
  context.stp           use new @task macro which gives a debuginfo type on
                        the return value
  context-envvar.stp    Ditto
  ip.stp                Add "kernel<foo.h>" for all @cast()'s
  netfilter.stp         Ditto. Exposed arp and bridge protocol info.
  nfs.stp               Fix vfs.do_sync_read and vfs.do_sync_write probe
                        aliases on rawhide kernels where the underlying probe
                        points no longer exist
  proc_mem.stp          Use @mm in _stp_get_mm_counter(). use new @task macro
                        which gives a debuginfo type on the return value
  arm/registers.stp     Allow argument 7 to be fetched in _stp_arg. handle
                        arguments passed on the arm stack
  scheduler.stp         Use new @task macro which gives a debuginfo type on the
                        return value
  socket.stp            Support  socket_write_iter() and socket_read_iter(),
                        the replacements for socket_aio_write() and
                        socket_aio_read
  speculative.stp       Optimize by replacing several iterations with array slices
  nd_syscalls.stp       Made vm86_warning probes optional. Add arm fix for
                        fadvise64_64. String representation of timex struct.
                        Add 'env_str' convenience variable for the list of
                        environment variables. For kernels < 3.7, move execve
                        tapset support to arch-specific tapset code.
  syscalls.stp          Ditto. Use __compat_[u]long() macros in
                        syscall.migrate_pages and __keyctl_argstr(). Fix s390x
                        compat task pointer retrieval.
  aux_syscalls.stp      Updated ptrace functions. Print the failing address
                        instead of "UNKNOWN". Rename  all uses of
                        syscall_get_nr() to _stp_syscall_get_nr(). Add SPLICE*
                        defs. Improve formatting and report the difference
                        between an invalid address and a NULL in
                        __get_[compat_]argv. String representation of timex
                        struct and arch_prctl option arg.
  nd_syscalls2.stp      Improve argstr handling, 32-bit support, ptrace probes.
                        Fix syscall.rt_sig{action,pending,procmask} for s390x.
                        Call _stp_sigaltstack_u() or _stp_compat_sigaltstack_u()
                        in syscall.sigalstack to decode the uss.
  syscalls2.stp         Ditto. Fix s390x compat task pointer retrieval. Fix
                        syscall.rt_sigsuspend compat probe. Minor improvement to
                        syscall.nfsservctl for RHEL6.
  syscalls.stpm         New __BIG_ENDIAN, @__quotactl_argstr(), and other macros.
                        Use __compat_[u]long() macros in syscall.migrate_pages
                        and __keyctl_argstr().
  i386/syscalls.stp     Decode struct user_desc. Fix iopl probe.
  i386/nd_syscalls.stp  Ditto.
  [nd_]syscalls.stp     Changed [compat_]execve[at] probes to use new
                        __count[_compat]_envp functions to display the number
                        of environment variables. Displaying the actual
                        environment variables can easily overflow MAXSTRINGLEN
                        and doesn't match what strace does.
  task.stp              Use new @task macro which gives a debuginfo type on the
                        return value.
  tty.stp               Replace  kernel_string calls with kernel_string_quoted
  task.stpm             Add @mm and @task macros.
  task_time.stp         Since  @defined(task->real_start_time->tv_sec) doesn't
                        work, use @defined(@task(0)->...). Use new @task macro
                        which gives a debuginfo type on the return value.
  utrace.stp            Rename syscall_get_nr() to _stp_syscall_get_nr().
  vfs.stp               Fix vfs.do_sync_read and vfs.do_sync_write probe
                        aliases on rawhide kernels where the underlying probe
                        points no longer exist
  target_set.stp        Handle [nd_]syscall.fork probe's split into
                        [nd_]syscall.{fork,vfork,clone} probes.

- Extensive work was done to provide coverage for syscalls in
  [nd_|aux_]syscalls[2].stp, along with  continuous improvement to the
  types of the parameters in many syscall probe aliases. Work was also
  done to add compat and 32-bit support for syscall.

- New namespace-aware tapset functions [task_]ns_*() in task.stp and
  context.stp use option --target-namespaces=PID to denote a target set of
  namespaces corresponding to the PID's namespaces. The namespace-aware
  tapsets will return values relative to the target namespaces if specified,
  or the stap process' namespaces.

- Many side-effect free inline embedded-C blocks have been tagged with
  /* pure */ so that they can be optimized away.

- The following tapset variables and functions are deprecated in
  version 2.8:
  - The 'hostname_uaddr' variable in the syscall.setdomainname and
    nd_syscall.setdomainname probe aliases have been deprecated in
    favor of the new 'domainname_uaddr' variable.
  - The 'fd' and 'fd_str' variables in the syscall.execveat and
    nd_syscall.execveat probe aliases have been deprecated in favor of
    the new 'dirfd' and 'dirfd_str' variables.


= SystemTap sample scripts

- New samples:
  netfilter_summary_json.stp  watches all IPv4 network traffic on the system.
                              The data is output in JSON format.
  net_xmit_json.stp           tracks time between packet queue and transmit.
                              The information is provided to userspace via
                              procfs in JSON format.
  proctop.stp                 top-like script that periodically prints out
                              process info.
  stp_dump.stp                prints out the packet contents. Each block
                              contains the STP protocol ID, version ID, flags,
                              root and bridge MAC addresses, and various times.
  whythefail.stp              prints a statement-execution trace for a given
                              function. works for failure analysis.


= Examples of tested kernel versions

  2.6.9  (RHEL 4 x86)
  2.6.18 (RHEL 5 x86 and x86_64)
  2.6.32 (RHEL 6 x86 and x86_64)
  3.10.0 (RHEL 7 x86_64)
  3.17.7 (Fedora 20 x86_64)
  3.19.3 (Fedora 21 x86_64)
  4.0.4  (Fedora 22 and 21 x86_64)


= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
  broad kernel function wildcards.  (PR2725)

- 32-on-64 bit userspace unwinding is truncated on older kernels, such
  as 2.6.32 (PR15757)

- The dyninst backend is still very much a prototype, with a number
  of issues, limitations, and general teething woes. For instance:
  + lack of support for multiarch/cross-instrumentation
  + tapset functions are still incomplete relative to what is supported
    when the kernel backend is active
  + exception handling becomes completely broken in programs
    instrumented by the current version of dyninst (PR14702)
  + not all registers are made available on 32-bit x86 (PR15136)

  See dyninst/README and the systemtap/dyninst Bugzilla component
  (http://tinyurl.com/stapdyn-PR-list) if you want all the gory                 
  details about the state of the feature.

- An upstream kernel commit #2062afb4f804a put "-fno-var-tracking-assignments"
  into KCFLAGS, reducing debuginfo quality which can cause debuginfo failures.
  A proposed workaround to this issue exists in:
  https://lkml.org/lkml/2014/11/21/505 . Fedora kernels are not affected by
  this issue.


= Contributors for this release

  Abegail Jakop, *Athira, David Smith, *Felix Lu, Frank Ch. Eigler,
  *Hemant Kumar, Jonathan Lebon, Josh Stone, Lukas Berk, Mark Wielaard,
  Martin Cermak, Masanari Iida, Nathan Scott, *Qiao Nuohan, William Cohen


  Special thanks to new contributors, marked with '*' above.
  Special thanks to Abegail and Felix for compiling these notes.


= Bugs fixed for this release <https://sourceware.org/PR#####>

  6762  Some syscalls functions just wrappers for other syscalls
  10488  sample script for unsampled ""true-top"" process monitor
  13481  ARM register.stp arg() functions don't handle arguments passed on the stack
  14164  netfilter.stp -- expose arp and bridge protocol info
  14325  for arm, the nd_syscall tapsets need improvement
  15987  generated function::* man pages should xref to enclosing tapset::*
  16663  Provide synthetic pt_regs context for backtracing
  16716  syscall wrappers causing syscall probes to get the wrong types
  16974  excessive context size after {stmt;}->stmt; optimization
  17638  Symbol resolution broken for PPC64 ABIv2
  17660  @perf wont work with uprobes
  17906  Wildcards in .nearest probes causing unprivileged_probes.exp to fail
  17957  Systemtap rejects binaries probuced by the standard Go compiler (gc)
  17958  Systemtap doesn't like variables with ~ in their name
  17959  Something causes Systemtap to generate C code with uint0_t types
  17986  unprivileged_[probes|myproc].exp regressed on el6
  18000  kernel tracepoints not found without using the cache
  18026  need timerfd_create, timerfd_gettime, timerfd_settime syscall support
  18115  probe-condition mistranslated for array expressions
  18120  no test coverage for adjtimex, arch_prctl, brk, capget, capset
  18121  fallocate syscall support needs to be added
  18122  [nd_]syscall.exec* probes need work
  18143  target_set tapset does not track threads created with clone()
  18151  chroot, getpid, getppid, gettid, iopl, lookup_dcookie, mincore need test coverage
  18154  let's be more forgiving on ppc64le
  18159  the [nd_]syscall.ptrace probes need improvement
  18162  aarch64 compile server issue
  18184  personality pivot_root quotactl need test coverage
  18213  on arm, the runtime doesn't return correct syscall numbers
  18262  The 'sync_file_range' and 'syncfs' syscalls need tapset support
  18263  In tty tapset, driver_name can be null, fails when probing tty.write or tty.read
  18264  the 'name_to_handle_at' and 'open_by_handle_at' syscalls need tapset support
  18284  some of the rt_* syscalls need improved/added tapset support
  18293  unshare, times, sysinfo, sysctl, syncfs, stime need test coverage
  18309  [nd_]syscall.{sigpending,sigsuspend,sigaltstack} probes need improvement/testing
  18310  remap_file_pages restart_syscall setsid uselib vhangup need test coverage
  18337  [nd_]syscall.{kexec_load,set_tid_address} probes need improvement and test coverage
  18343  Syscall clock_adjtime needs tapset handler and test coverage.
  18361  systemtap doesn't realize RHEL7 kernels require secure-boot signed modules
  18387  The setns syscall needs tapset handler and test coverage.
  18389  gcc5 code folding breaks tracepoint queries
  18395  Syscalls {get,set}_robust_list need tapset handler and test coverage
  18398  The {get,set}_thread_area syscalls need tapset support and test coverage
  18460  tracepoint_onthefly.exp kernel crash
  18461  code generated by tapset-netfilter.cxx for nf_hook_ops does not compile with linux-4.1.0-rc5 kernel
  18477  fix remaining kernel 3.10 syscall issues
  18492  Syscalls sched_getattr and sched_setattr need tapset handler and test coverage
  18501  The kcmp syscall needs tapset support and test coverage.
  18518  vfs.do_sync_read and vfs.do_sync_write probe aliases no longer work on rawhide
  18526  socket.aio_read and socket.aio_write probe aliases are broken on rawhide
  18538  on rawhide, the scsi.iodispatching probe alias can't be resolved