SystemTap 2.7 release

[Abe Jakop <ajakop at redhat dot com>]

The SystemTap team announces release 2.7, RedGrizzlyCoffee!

  Kernel tracepoint subsystem probes, optimized string passing, syscall tapset
  improvements, highlighted educational examples, addr->file:line mapping,
  SystemTap's 10th anniversary, and more!


= Where to get it

  https://sourceware.org/systemtap/ - our project page
  https://sourceware.org/systemtap/ftp/releases/systemtap-2.7.tar.gz
  https://koji.fedoraproject.org/koji/packageinfo?packageID=615
  git tag release-2.7 (commit bf16266782e1f25)

  There have been over 290 commits since the last release.
  There have been over 35 bugs fixed / features added since the last release.


= How to build it

  See the README and NEWS files at
  https://sourceware.org/git/?p=systemtap.git;a=tree

  Further information at https://sourceware.org/systemtap/wiki/


= SystemTap's 10th Anniversary

  February 12 was the 10th anniversary of SystemTap. We would to thank all of
  our contributors over the years, listed below.

  Aaron Tomlin, Abegail Jakop, Adrien Kunysz, Alexander Lochmann,
  Alexander Y. Fomichev, Alfred Landrum, Ananth N Mavinakayanahalli,
  Andre Detsch, Andreas MÃller, Andy Fitzsimon, Anil Keshavamurthy,
  Anithra Janakiraman, Anton Vorontsov, Atsushi Tsuji, Brendan Gregg,
  Breno Leitao, Brent Baude, Brian Chrisman, Bryn M. Reeves, Casey Dahlin,
  Charles Spirakis, Charley Wang, Chris Meek, Chung-Lin Tang, Dan Horak,
  Daniel Migault, Daniel Tschan, Dave Brolley, Dave Nomura, David J. Wilder,
  David Juran, David Smith, Dennis Gilmore, Diego Elio PettenÃ, Don Domingo,
  Elliott Baron, Erick Tryzelaar, Eugene Teo, Eugeniy Meshcheryakov,
  Fabio Olive Leite, Fahad Arshad, Felix Lin, Frank Ch. Eigler,
  Frederic Turgis, Fredrik Ãstman, Grant Edwards, Graydon Hoare, Han Pingtian,
  Hien Nguyen, Hilko Bengen, Honggyu Kim, Hushan Jia, James Bottomley,
  Jan Kratochvil, Jeff Moyer, Jim Keniston, Jiri Horky, Jiri Slaby,
  JoeLynn Keniston, Jonathan Lebon, Josh Stone, JÃhann B. GuÃmundsson,
  K.Prasad, Kai Meyer, Kapileshwar Singh (KP), Keiichi KII, Kent Sebastian,
  Kevin Stafford, Kim van der Riet, Kiran Prakesh, Lars R. Damerow,
  Li Guanglei, Lubomir Rintel, Lukas Berk, Lukas Czerner, LukÃÅ HejtmÃnek,
  Mahesh J Salgaonkar, Malte Nuhn, Maran Pakkirisamy, Marc Milgram,
  Mark McLoughlin, Mark Wielaard, Martin Cermak, Martin Hunt, Martin Lazar,
  Masami Hiramatsu, Masanari Iida, Masatake YAMATO, Maynard Johnson,
  Michael Meeks, Mike Mason, Miroslav Franc, Naresh Kamboju, Nathan Scott,
  Negreanu Marius Adrian, Nitin A Kamble, Nobuhiro Tachino,
  Oleksandr Chumachenko, Otavio Pontes, Paddie O'Brien, Peter Feiner,
  Peter KjellstrÃm, Peter Robinson, Petr Muller, Phil Muldoon, Prerna Saxena,
  PrzemysÅaw PaweÅczyk, Quentin Barnes, Rajan Arora, Rajasekhar Duddu,
  RaphaÃl Beamonte, Rayson Ho, Rich Megginson, Robb Romans, Robin Hack,
  Robin Lee, Roland Grunberg, Roland McGrath, RÃdiger Sonderfeld,
  Santosh Shukla, Serguei Makarov, Shaohua Li, Siddhesh Poyarekar,
  Srikar Dronamraju, Srinivasa DS, Stan Cox, Stefan Hajnoczi, Steve Dickson,
  Sunzen Wang, Tetsuo Handa, Thang Nguyen, Theodore Ts'o, Tim Moore,
  Timo Juhani Lindfors, Tom Callaway, Tom Tromey, Tom Zanussi, Tomoki Sekiyama,
  Tony Jones, Torsten Polle, Varun Chandramohan, Victor Kamensky,
  Vincent Bernat, Wade Farnsworth, Wenji Huang, William Cohen,
  Yaakov Selkowitz, Yang Wen, Yichun Zhang (agentzh), Yoshihide Sonoda,
  Zhaolei, Zifei Tong, ch huang

  As a thank you for your contribution:
  https://sourceware.org/ml/systemtap/2015-q1/msg00021.html


= SystemTap frontend (stap) changes

- SystemTap now allows .function probes to be specified by their full function
  name, file, and declaration line number. Use the .statement probe to probe a
  specific line number.

- Tracepoint probes can now also be specified by the target subsystem. For
  example, the following are all supported:

  probe kernel.trace("sched:sched_switch") --> probe sched_switch found in the
                                               sched subsystem
  probe kernel.trace("sched:*") --> probe all tracepoints in sched subsystem

  As a result, tapset functions such as pn() will now return a different string
  than before. To retain the previous behaviour, use '--compatible=2.6'.


= SystemTap runtime changes

- Passing strings to and from functions has become faster due to optimization
  (passing some strings by reference instead of by value/copy).  It may
  be disabled by using the unoptimize flag (-u).

  To make embedded-C functions eligible for the same optimization, use the pragma
  /* unmodified-fnargs */ to indicate that the function body will not modify
  the function arguments.  Remember to use MAXSTRINGLEN for string length,
  rather than sizeof(string_arg) (which might now be a pointer).


= SystemTap tapset changes

- Changed tapsets:

  aux_syscalls.stp        returns the pointer value instead of "UNKOWN" for an
                          invalid pointer. improved _msync_flag_str() and
                          _mlockall_flags_str(). arm64 support in __is_user_regs
                          collection of new _*str and other functions
  nd_syscalls.stp         fix compile problem. new probe nd_syscall.finit_module
                          correct nodebug mkdirat syscall argstr
  syscalls.stp            new probe syscall.finit_module. doesn't rejecting
                          RUSAGE_BOTH, lets the kernel do it instead.
  powerpc/syscalls.stp    new probe for "ppc_newuname" kernel function
  x86_64/syscalls.stp     sdded syscall.compat_fadvise64 probe.
  syscalls.stpm           added __futex_argstr, __compat_syscall_gate_negative macro
  dentry.stp              Adapt to changes in the dentry structure to handle
                          more kernels, especially 3.18
  nfs.stp                 new probe nfs.fop.read_iter for kernels >= 3.16
                          and __syscall_gate_compat_simple macros
  nfs_proc.stp            probe nfs.proc.rename_setup added to support
                          asynchronous NFS client rename operations.
  nfsd.stp                sewrite __svc_fh() in systemtap script language
  [u]context-symbols.stp  new functions [u]symfileline, [u]symline and [u]symfile

- For Linux >= 3.17, added support for getrandom, memfd_create, seccomp syscalls
  within [nd_]syscalls*.stp

- For Linux >= 3.19, added support for execveat syscalls within [nd_]syscalls.stp

- More work was done to improve the types of the parameters in many
  syscall probe aliases ([nd_|aux_]syscalls[2].stp). Work was also done to add
  compat and 32-bit support for syscall.

- The following functions are deprecated in release 2.7:
  - _adjtx_mode_str(), _statfs_f_type_str(), _waitid_opt_str(),
    _internal_wait_opt_str(), and _epoll_events_str().


= SystemTap sample scripts

- New collection:

  lwtools               lightweight tools. example tools for systems without
                        kernel debuginfo

- New samples:

  accept2close-nd.stp   traces socket duration from the accept() syscall to
                        close(), and provides details on the lifespan of these
                        passive connections, showing the distribution as a
                        histogram.
  biolatency-nd.stp     measures block I/O latency (storage I/O, ie, disk I/O),
                        and shows the distribution as a histogram
  bitesize-nd.stp       uses the kernel tracepoint block_rq_insert to read the
                        size of I/O. The output includes the name of the
                        process or thread that was on-CPU when the I/O request
                        was inserted on the issue queue.
  execsnoop-nd.stp      can identify if CPU is consumed by short-lived
                        processes, by tracing new process execution. will not
                        catch new processes that only fork(). shows every
                        exec(), including those if a process re-execs.
  fileline-profile.stp  prints a sorted list of the top twenty kernel
                        and/or user processes providing file:line information,
                        from the samples addresses gathered.
  fslatency-nd-stp      dynamically traces two common file system functions:
                        do_sync_read() and do_sync_write(), and reports a
                        histogram distribution of latency.
  fsslower-nd.stp       dynamically traces two common file system functions:
                        do_sync_read() and do_sync_write(), and shows details
                        of each call that is slower than a threshold.
  glibc-malloc.stp      reports on internal statistics of the glibc malloc
                        implementation, as used by a process restricted by -x/-c
  killsnoop-nd.stp      traces signals system-wide, including those sent by
                        the kill(1) command, and shows various details.
  opensnoop-nd.stp      traces the open() syscall system-wide, to show which
                        files are being opened, and by who.
  rwtime-nd.stp         traces read() and write() syscalls, producing a
                        histogram summary of their durations (aka latencies).
  spawn_seeker.stp      prints out the local time and sorted lists of which
                        processes and executables spawned tasks during the
                        previous minute.
  syscallbypid-nd.stp   traces syscalls system-wide, and produces a summary
                        report showing their counts by process ID, process
                        name, and syscall types.

- Changed samples:

  cycle_thief.stp       improve the output to include executable name
  hello-world.stp       use probe oneshot
  iostat-scsi.stp       probes the "sd_init_command" kernel function which
                        replaced "sd_prep_fn" in newer kernels.
  pfaults.stp           index call/return arrays by tid, not pid
  periodic.stp          print the symbol@file:line information instead of
                        symdata for kernel addresses

- Some SystemTap sample scripts are now identified with the "_best" keyword,
  because they are generally useful or educational.  They are now promoted
  within the generated index files.


= Examples of tested kernel versions

  2.6.18 (RHEL 5 x86 and x86_64)
  2.6.32 (RHEL 6 x86 and x86_64)
  3.10.0 (RHEL 7 x86_64)
  3.17.7 (Fedora 20 x86_64)
  3.18.3 (Fedora 21 x86_64)
  3.19 (upstream)
  3.20-rc (Rawhide)

= Known issues with this release

- Some kernel crashes continue to be reported when a script probes
  broad kernel function wildcards.  (PR2725)

- 32-on-64 bit userspace unwinding is truncated on older kernels, such
  as 2.6.32 (PR15757)

- The dyninst backend is still very much a prototype, with a number
  of issues, limitations, and general teething woes. For instance:
  + lack of support for multiarch/cross-instrumentation
  + tapset functions are still incomplete relative to what is supported
    when the kernel backend is active
  + exception handling becomes completely broken in programs
    instrumented by the current version of dyninst (PR14702)
  + not all registers are made available on 32-bit x86 (PR15136)

  See dyninst/README and the systemtap/dyninst Bugzilla component
  (http://tinyurl.com/stapdyn-PR-list) if you want all the gory
  details about the state of the feature.

- An upstream kernel commit #2062afb4f804a put "-fno-var-tracking-assignments"
  into KCFLAGS, reducing debuginfo quality which can cause debuginfo failures.
  A proposed workaround to this issue exists in:
  https://lkml.org/lkml/2014/11/21/505 . Fedora kernels are not affected by
  this issue.


= Contributors for this release

  Abegail Jakop, *Andy Fitzsimon, *Brendan Gregg, *Chung-Lin Tang, David Smith,
  Frank Ch. Eigler, *Hilko Bengen, *Hushan Jia, Jonathan Lebon, Josh Stone,
  Lukas Berk, Martin Cermak, Masanari Iida, *Oleksandr Chumachenko,
  *Santosh Shukla, *Siddhesh Poyarekar, Stan Cox, Stefan Hajnoczi,
  William Cohen

  Special thanks to new contributors, marked with '*' above.
  Special thanks to Abegail for compiling these notes.


= Bugs fixed for this release <https://sourceware.org/PR#####>

11528 Pass string values by reference where possible
11644 process(PID).function/statement probes
12276 lookup functions & uploaded-data for address->file:line mappings
13330 Support TRACE_SYSTEM for tracepoints
16645 Unhelpful feedback when stap fails
16920 Add aarch64 backtrace support
17049 dtrace does not support -fPIC option anymore
17126 tracepoints.exp testcase causing stalls/hang on ppc64
17140 systemtap.examples/profiling/functioncallcount.stp causing kernel panic on s390x
17190 'stap -l' output not useful as stap input
17270 uprobes_onthefly.exp causing hang on ppc64
17292 multi-line strings broken
17395 util.cxx:1120: bad switch statement ?
17413 missing timer_create syscall in 32-on-64 mode
17462 aarch64 deprecates a number of syscalls (__NR_epoll_wait not defined)
17622 On aarch64 listing_mode.exp fails to find any probe points for
      process.library(""liblisting_mode.so"").function(""libfoo"")
17688 probe nfs.fop.aio_read no longer valid
17690 probe nfs.proc3.rename no longer valid
17696 fails to find kernel tracepoints when kernel is built in a separate directory from source
17706 user string copy fault on s390 in 31-on-64 mode
17710 stap-report script contains bashisms
17714 poll.c syscall testcase
17737 ""menu"" nodes in online docs have text at the end
17738 bad formatting in online docs
17743 stap --help should print to stdout
17749 stap doesn't recognize ""++"" as a use
17807 The vfs tapset is broken on 3.18
17858 helloworld doesn't with --runtime=dyninst
17860 foreach loop fails -p4 if array slice is an expression
17864 alias_tapset.exp regressed
17883 git stap commit 81acc574 doesn't compile on rhel[56]
17884 Fresh upstream bits based on git commit 81acc574 do not compile on RHEL[56].
17888 task_dentry_path.exp got broken on el7 somewhere between b9ae0bc and b19a437
17889 the translator does not understand arm64 constants in sdt markers
17894 global_opt_unknown.exp failure
17928 syscall.exp tests fail on 32-bit arm because size_t sign extended to 64-bit value
17951 cast-user.exp regressed due to the syscall.bind tapset modification
17966 A $var guarded by @defined in return probes may still raise errors
17985 babad5b31b buildok.exp regressed on el6 compared to systemtap-2.5-5.el6
17962 dtrace.exp --no-parsing fallback test fails on rhel6