Suggestions for localizing problems with identifiers in SystemTap.

[William Cohen <wcohen at redhat dot com>]

Martin Hunt wrote:
> > FC5 i686 Linux 2.6.15-1.1948_FC5smp
> >
> > Stap translator summary of failed tests
> > buildok-seven.stp:
> > 	Unable to find identifier $type in function find_pid
> > 	Unable to find identifier $task in function detach_pid
>
>
> There don't appear to be significant changes in the sources to account
> for this. That makes it a likely gcc bug. 
>
> Can someone write up the proper procedure to use to report these? I
> think it should be on the web page.
>
> Martin

I have jotted down some notes on how to track down these types of 
problems with the identifiers. The notes are below. There are still lots 
of holes.

-Will



Suggestions for localizing problems with identifiers in SystemTap.


Possible causes for this problems symbols in SystemTap scripts:

1) source code doesn't have function, variable, or argument
2) GCC tool chain not generating the proper debug information
3) system tap translator is unable them in the debugging information

-----

Source code doesn't have function, variable, or argument

This can be determined by examining the source code.

1) Find out which kernel is being used and location of source
code. You will need to have the cscope rpm installed

	uname -r

	get the matching source SRPM.
	install srpm on  the machine with:

	rpm -Uvh kernel-<version>.src.rpm

	cd into SPEC directory
	rpmbuild -bp kernel-2.6.rpm

	cd into BUILD/kernel-<version>/linux-<version> directory
	make mrproper
	cp /boot/config-`uname -r` .config; make oldconfig; make cscope

2) Find where the problem function is located and examine code for
variable or arument name. Ccsope is extremely useful for navigating
locating the functions.

	Start up cscope and use the following cscope entries:
		"Find this global definition:" or "Find this egrep pattern:"
	Type in function name or pattern
	If function shows up, hit return to look at function by hitting enter.
		cscope will start an editor. Take a look for the problem symbol
		if the function is "static" or "static inline",
			the compiler could optimize these away if not called.
			cscope can also show where the function is used

-----

GCC toolchain not generating the proper debug information

After establishing that the the symbol exists in source code,
determine whether the compiler is generating the proper debug
information.  The program "readelf" in the binutils RPM can be used
the examine the debugging information in the kernel and modules.

readelf --debug-dump /usr/lib/debug/lib/modules/`uname -r`/vmlinux > 
/tmp/debuginfo

This will generate a very large file /tmp/debuginfo that can be
examined.  For example want to find out why $task is not being found
in function detach_pid for src/testsuite/buildok/seven.stp.

Looking for the section of the output in /tmp/debuginfo for the
detach_pid function we find the following section:


 <1><3554ae>: Abbrev Number: 83 (DW_TAG_subprogram)
     DW_AT_sibling     : <35550b>
     DW_AT_external    : 1
     DW_AT_name        : (indirect string, offset: 0x1c783): detach_pid
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 193
     DW_AT_prototyped  : 1
     DW_AT_low_pc      : 0xffffffff80143458
     DW_AT_high_pc     : 0xffffffff80143499
     DW_AT_frame_base  : 0x8ee43        (location list)
 <2><3554cf>: Abbrev Number: 76 (DW_TAG_formal_parameter)
     DW_AT_name        : (indirect string, offset: 0xba9d3): task
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 192
     DW_AT_type        : <35184d>
     DW_AT_location    : 0x8eea3        (location list)
<2><3554de>: Abbrev Number: 76 (DW_TAG_formal_parameter)
     DW_AT_name        : (indirect string, offset: 0x4fb9b): type
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 192
     DW_AT_type        : <35062a>
     DW_AT_location    : 0x8eec6        (location list)
 <2><3554ed>: Abbrev Number: 72 (DW_TAG_variable)
     DW_AT_name        : tmp
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 194
     DW_AT_type        : <34e505>
     DW_AT_location    : 0x8ef0f        (location list)
 <2><3554fc>: Abbrev Number: 72 (DW_TAG_variable)
     DW_AT_name        : nr
     DW_AT_decl_file   : 1
     DW_AT_decl_line   : 194
     DW_AT_type        : <34e505>
     DW_AT_location    : 0x8ef45        (location list)


It is clear the "task" identifier is in the debugging information and
that it is a argument being passed into the function.  The type of the
identifier can be determined by by searching for the "DW_AT_type", in
this case "><35184d>":


<1><35184d>: Abbrev Number: 7 (DW_TAG_pointer_type)
     DW_AT_byte_size   : 8
     DW_AT_type        : <350f16>

The chain can be followed searching for "><350f16>":

 <1><350f16>: Abbrev Number: 6 (DW_TAG_typedef)
     DW_AT_name        : (indirect string, offset: 0x782ad): task_t
     DW_AT_decl_file   : 9
     DW_AT_decl_line   : 184
     DW_AT_type        : <34e8a0>

Searching for "><34e8a0>" yield task "task_struct" followed by its fields:


 <1><34e8a0>: Abbrev Number: 15 (DW_TAG_structure_type)
     DW_AT_sibling     : <34f062>
     DW_AT_name        : (indirect string, offset: 0x15daa): task_struct
     DW_AT_byte_size   : 1888
     DW_AT_decl_file   : 10
     DW_AT_decl_line   : 13
 <2><34e8ad>: Abbrev Number: 16 (DW_TAG_member)
     DW_AT_name        : (indirect string, offset: 0xb09cc): state
     DW_AT_decl_file   : 9
     DW_AT_decl_line   : 701
     DW_AT_type        : <34f60a>
     DW_AT_data_member_location: 2 byte block: 23 0 
(DW_OP_plus_uconst: 0)
 <2><34e8bc>: Abbrev Number: 16 (DW_TAG_member)
     DW_AT_name        : (indirect string, offset: 0x8efe): thread_info
     DW_AT_decl_file   : 9
     DW_AT_decl_line   : 702
     DW_AT_type        : <35191d>
     DW_AT_data_member_location: 2 byte block: 23 8 
(DW_OP_plus_uconst: 8)


-----

SystemTap (stap) is not finding the identiers in the debugging information

After eliminating the source code and compiler, the SystemTap
translator is the likely suspect. The systemtap translator might be
able to find the identifier but is unable to process it. For example
src/testsuite/buildok/seve.stp instruments the detach_pid funtion in
the kernel and on the x86_64 produces the following error message:

semantic error: unresolved target-symbol expression: identifier '$task' 
at 
/home/wcohen/stap_testing_200602261550/src/testsuite/buildok/seven.stp:19:35


One can use one (or more) "-v" (verbose) options with the SystemTap
translator to get get a clearer picture of what the translator is
doing and getting some idea of why the symbol is not being resolved
correctly. The following command was used:

stap -p4 -vvvv ../src/testsuite/buildok/seven.stp

The previous command's output included:

pattern 'detach_pid' matches function 'detach_pid'
selected function detach_pid
finding prologue for 'detach_pid' entrypc=0xffffffff80143458 
highpc=0xffffffff80143499
finding location for local 'task' near address ffffffff8014345f, module 
bias 0
finding location for local 'type' near address ffffffff8014345f, module 
bias 0
finding location for local 'task' near address ffffffff8014345f, module 
bias 0
finding location for local 'type' near address ffffffff8014345f, module 
bias 0

Thus, the 'task' local is found. However, the SystemTap translator
appears to be unable to handle it.