GDB Tracing Commands

Tracing program execution: the record/playback model

Some program behaviors can be analyzed only by collecting data over long periods of time for later analysis. Other problems occur in sections of code that are too time-sensative to be debugged interactively. In these cases, tracing may be more appropriate than breakpoint debugging.

This section discusses the following topics.

Tracepoints and trace experiments

Like a breakpoint, a tracepoint makes your program stop whenever a certain point in the program is reached. However, while a breakpoint stops the program for a "long" time (while GDB prompts you and lets you type commands), a tracepoint stops the program for only a "short" time, after which the program gets to continue with minimal disruption of its behavior. During this "short" interval, the trace mechanism records the fact that it has been there (ie. that the tracepoint was executed), and may also perform certain actions that you've requested such as recording the values of selected variables and registers. Thus it's not well defined how long a "short" time is, but in any event it will be thousands of times shorter than the time required for a human to do the same tasks interactively.

As with breakpoints, GDB assigns a number to each tracepoint when you create it. Like breakpoint numbers, tracepoint numbers are successive integers starting from one. Many of the commands associated with tracepoints use the tracepoint number to identify which tracepoint to delete, disable, etc.

A trace experiment must involve each of the following steps: defining the tracepoints and actions, running the experiment, and reviewing the results. Many of the commands defined below (such as setting and deleting tracepoints) have no real effect until you actually begin running the trace experiment. Moreover, many of them also have no effect during an already running trace experiment. Once a trace experiment is running, changing the tracepoints or their attributes will have no effect unles you stop the experiment and start it again.

Setting tracepoints

Tracepoints are set with the trace command (abbreviated tr). The debugger convenience variable '$tpnum' records the tracepoint number of the most recently set tracepoint; see Convenience Variables for a discussion of what you can do with convenience variables.

The location of a tracepoint can be specified in exactly the same ways as the location of a breakpoint, here reproduced for convenience.

trace func_name

Set a tracepoint at entry to function func_name. When using source languages that permit overloading of symbols, such as C++, func_name may refer to more than one possible place to trace. See Tracepoint menus for a discussion of that situation.

trace +offset

trace -offset

Set a tracepoint some number of lines forward or back from the "current location". Note that the "current location" is affected not only by stack walking commands such as up and down, but also by trace browsing commands such as tfind (see Reviewing trace results).

trace linenum

Set a tracepoint at line linenum in the current source file. The current source file is the last file listed, or the last file in which execution stopped (eg. at a breakpoint), or the last file visited while browsing trace results (see Reviewing trace results).

trace filename:linenum

Set a tracepoint at line linenum in source file filename.

trace filename:func_name

Set a tracepoint at entry to function func_name found in file filename. The filename is superfluous except when several source files contain functions with the same name.

trace *address

Set a tracepoint at address address. You can use this to set tracepoints in parts of your program which do not have debugging information or source files.

Tracepoint menus

Some programming languages (notably C++) permit a single function name to be defined several times, for application in different contexts. This is called overloading. Tracepoints on overloaded functions behave exactly like breakpoints do (see "Breakpoint menus" in the GDB manual (available at http://sourceware.cygnus.com/gdb)).

Deleting tracepoints

Any tracepoint's definition can be permanently discarded using the delete tracepoint command (abbreviated del tr). Give the tracepoint number(s) of the tracepoint(s) that you want to delete. A deleted tracepoint no longer exists, in contrast with a disabled tracepoint (which can be re-enabled). Use the info tracepoints command to discover the numbers of your tracepoints.

NOTE: this command has no effect on a running trace experiment.

delete tracepoint [tpnums...]

Delete the tracepoint(s) of the numbers specified as arguments. If no argument is specified, delete all tracepoints (GDB asks confirmation, unles you have set confirm off. You can abbreviate this command as del tr.

Disabling tracepoints

Rather than deleting a tracepoint, you might prefer to disable it. This will make the tracepoint inoperative in the next trace experiment, but remembers the tracepoint and all it's attributes so that you can enable it again. You disable and enable tracepoints with the disable tracepoint and enable tracepoint commands. Use the info tracepoints command to find out the numbers of tracepoints and which ones are enabled.

NOTE: these commands have no effect on a running trace experiment.

disable tracepoint [tpnums...]

Disable the specified tracepoints -- or all tracepoints if none are listed. A disabled tracepoint will have no effect in the next trace experiment, but is not forgotten. All its attributes are remembered, and will still be in effect if it is enabled again later. You can abbrevkate this command as dis tr.

enable tracepoint [tpnums...]

Enable the specified tracepoints (or all tracepoints). They will become effective again the next time a trace experiment is started. You can abbreviate this command as en tr.

Tracepoint passcounts

Passcounts allow you to terminate a trace experiment automatically. If a tracepoint has a non-zero passcount of , then the trace experiment will automatically terminate (and no more trace data will be collected) when that tracepoint has been executed times.

Any number of tracepoints may have passcounts of any integer value. When the first tracepoint reaches its passcount, the trace experiment will terminate.

To clear a passcount, set it to zero.

NOTE: this command has no effect on a running trace experiment.

passcount <n> [tpnum]

Set the passcount of the given tracepoint to <n>. If tpnum is omitted, set the passcount of the most recently defined tracepoint. You can abbreviate this command as pas.

Tracepoint action lists

A tracepoint can do far more than simply record where your program has been. It can also save the values of selected program variables, machine registers, or memory locations for you to examine later at your leisure. All of these data go into a single buffer, but except for the size of that buffer there is virtually no limit to the data you can collect at tracepoints.

The actions at a tracepoint may also include single-stepping for some number of machine instructions, collecting data at each step. In this mode you can collect the changes in registers or variables over a range of instructions.

Here are the commands for defining the actions for a tracepoint.

actions [tpnum]

...action-list...

end

Specify a list of actions for tracepoint number tpnum. The actions themselves appear on the following lines. Type a line containing just end to terminate the actions. To remove all actions from a tracepoint, type actions [tpnum] and follow it immediately with end; that is, give no actions. With no tpnum argument, actions refers to the last tracepoint set.

collect [exprs]

Describes data to be collected at the tracepoint. Exprs may be any legal C expression, or several expressions separated by commas. Any global, static, or local variables may be collected. In addition, the following special arguments are supported.

while-stepping <n> 

	while-stepping <n>
	  collect [exprs]
	end

After collecting any data requested at the tracepoint itself, execute <n> machine instructions, collecting the following expressions after each instruction. Each step will generate a separate trace event in the trace buffer (see Reviewing trace results).

end

Terminates an action list, or a while-stepping collection list. If actions include stepping, two end commands are required.

NOTE: these commands have no effect on a running trace experiment.

Listing tracepoints

Use this command to list all existing tracepoints and their attributes.

info tracepoints [tpnum]

Print a list of all tracepoints that have been set and not deleted, together with the following information for each tracepoint.

Starting and stopping trace collection

None of the commands discussed so far will have any actual effect until you tell GDB to start a trace experiment. Only then will the tracepoints begin collecting data. Moreover, once you have started a trace experiment, none of the commands above have any effect on the running experiment. Changes to your tracepoints will only go into effect the next time you start a trace experiment. Therefore, if you delete, disable, enable, or define any new tracepoints, or change the actions or passcounts of existing tracepoints, you must stop any currently running trace collection experiment and start a new one before your changes will go into effect.

You start and stop trace collection by using the tstart and tstop commands.

tstart

Start trace data collection.

tstop

Stop trace data collection.

NOTE: trace data collection may also be stopped automatically if any tracepoint's passcount is reached. It may also stop automatically if the trace buffer becomes full, unles the trace buffer is set to operate in a continuous "circular buffer" mode.

Note also that starting trace collection has the side effect of discarding any old trace data in the buffer, so that the buffer starts out empty each time. You may stop a trace experiment and still look at your saved data, but as soon as you start a new trace experiment, the previously collected data will be irretrievably gone.

Reviewing trace results

While a trace experiment is running, every time a tracepoint is executed generates a trace event, and every trace event generates a data record called a trace frame in a buffer called the trace buffer. If a tracepoint's actions include single-stepping, each single-step also generates a trace event and a trace frame.

Each trace frame has a sequential identifying number starting from zero, plus the address (PC) of the event. In addition, the trace frame contains any values collected by the actions for the corresponding tracepoint.

The basic commands for selecting a trace frame and extracting data from it are tfind and tdump. In addition, several built-in GDB variables are provided for identifying the currently selected trace frame.

tfind [< event id > | start | none | -]

Select a trace frame (event) by id number. "tfind start" is a synonym for tfind 0 (where zero is the lowest numbered trace frame). If no argument is specified, selects the next frame. "tfind -" selects the previous frame, and "tfind none" selects no frame.

tfind tracepoint [< tracepoint id >]

Select the next trace frame (event) corresponding to the given tracepoint. If no argument is given, find the next trace frame corresponding to the same tracepoint as the current frame.

tfind pc [< address >]

Select the next trace frame corresponding to the given code address. If no argument is given, find the next trace frame corresponding to the same PC as the current frame.

tfind line [linenum]

tfind line [filename:linenum]

Select the next trace frame corresponding to the given source line. If no argument is given, then advance to the next trace frame from any source line other than the current one. This is useful when a trace frame is collected for every machine instruction. It has the effect of advancing past all the events for instructions in the current source line, thus getting to the next line traced.

tdump

Print all of the data collected at the current trace frame.

Convenience variables

  • $trace_frame
    • the ID of the currently selected frame (or -1 if no trace frame is selected).
  • $tracepoint
    • the ID of the tracepoint corresponding to $trace_frame
  • $trace_line
    • the source line number corresponding to $tracepoint
  • $trace_file
    • the source filename corresponding to $tracepoint
  • $trace_func
    • the name of the function containing $tracepoint

By using combinations of the above commands and convenience variables, together with GDB's built-in scripting language, there are many options for viewing the collected trace data. The most straightforward (but not necessarily the most useful) would be to simply print out everything that was collected. 

	(gdb) tfind start
	(gdb) while ($trace_frame != -1)
	> tdump
	> tfind
	> end
There are many ways to filter and format the data. As a simple example, you might want to extract only the data for certain tracepoints.

(gdb) tfind start (gdb) while ($trace_frame != -1) > if ($tracepoint == 2 || $tracepoint == 7) > tdump > end > tfind > end

Whenever a trace frame is selected, all of GDB's existing commands for examining machine state and data will get their values from the trace frame. Therefore you can use commands such as GDB's printf command to format the collected trace data any way you like (so long as you only attempt to print data that you've collected!). For example, you might want to produce a verbose listing of certain values from each of several tracepoints. 

	(gdb) tfind start
	(gdb) while ($trace_frame != -1)
	> printf "Tracepoint %d, ", $tracepoint
	> printf "line %d, ", $trace_line
	> if ($tracepoint == 1)
	  > printf "x == %d\n", x
	  > end
	> if ($tracepoint == 2)
	  > printf "gradient == %d, ", gradient
	  > printf "register D0 == 0x%x\n", $d0
	  > end
	> if ($tracepoint == 3)
	  > printf "z / q == %f\n", z / q
	  > end
	> tfind
	> end
However, many people find that the most useful way to review the collected data is interactively. Whenever you use the tfind command, GDB displays the source line corresponding to the selected trace frame, just as if you had stepped to that line or hit a breakpoint there. The next tfind command takes you to the next source line, very much like stepping. At each trace frame, you can use all of the customary GDB commands to examine the data collected, very much as you could if you were sitting at a breakpoint, except that you can only examine values that you collected earlier by listing them in the action lists of your tracepoints.

In fact, if you plan ahead and collect a chunk of memory from the machine stack, you can even use GDB's backtrace or up and down commands to find out what functions called the function that you're tracing. 

	(gdb) trace foo
	(gdb) actions
	> collect $fp, *(long **) $sp @ 64
	> end

Saving tracepoint definitions

Once you have defined a set of tracepoints and actions, you may want to save them to be used later in another debug session. The save-trace command saves all tracepoints, passcounts and actions as a GDB command file, which can be imported later by using the source command (See "Command files" in the GDB manual (available at http://sourceware.cygnus.com/gdb)).

save-tracepoints filename

Write all tracepoint definitions, passcounts, and actions out to the command file 'filename', suitable for reading with the source command.