• GDBTestcaseCookbook

This page collects notes on how to write testcases in the GDB testsuite, traps to avoid, etc.

When adding a feature or fixing a bug which is not covered by the testsuite, you should write a new testcase or extend an existing one.

Building the Example Program

The typical template to start with development of a new testcase can be copied from gdb/testsuite/gdb.base/template.exp / gdb/testsuite/gdb.base/template.c.

See gdb/testsuite/lib/gdb.exp for the list of routines available and their documentation.

Performing a Test

Testing a command in GDB should be done using either gdb_test, gdb_test_no_output, gdb_test_multiple or gdb_test_sequence. Please do not use gdb_expect as the latter does not automatically handle all known error situations (internal errors, timeouts, etc). Known failing new testcases must produce KFAIL (GDB problem) or XFAIL (environment problem).

When matching GDB output in a test, you have to wait for the prompt explicitly. If you ever leave expect in a state where two prompts should be arriving, it won't know which is which; if it sees them separately, you can get out of sync and all tests will fail with unknown output. This means your testcase will have a race condition and may pass or fail intermittently. gdb_test waits for the GDB prompt, gdb_test_multiple doesn't. When using the latter, you have to explicitly put the GDB prompt in your regular expressions.

Also, gdb_test puts an End-of-Line marker at the end of your regular expression.

    gdb_test "break $srcfile:factorial" \
        "Breakpoint.*at.* file .*$srcfile, line.*" \
        "breakpoint function in file"

    gdb_test_no_output "set \$foo=$bp_location11" \
        "set convenience variable \$foo to bp_location11"

    set test "print/x $var"
    gdb_test_multiple "$test" $test {
        -re "\\$\[0-9\]+ = [string_to_regexp $val].*\r\n$gdb_prompt $" {
            pass $test
        }
        -re "\\$\[0-9\]+ = $addr.*\r\n$gdb_prompt $" {
            fail "$test (prints just address)"
        }
        -re "\\$\[0-9\]+ = 0x\[a-f0-9\]+.*\r\n$gdb_prompt $" {
            fail "$test (prints unexpected address)"
        }

Below is an example where gdb_test_sequence is used. Note that, at the moment, the function treats the sequence as raw output, not as a list of lines, and thus does not implicitly add new-line markers in between each element.

    gdb_test_sequence "where" "where in corefile" {
        "\[\r\n\]+#0 .* terminal_func \\(\\) at "
        "\[\r\n\]+#1 .* array_func \\(\\) at "
        "\[\r\n\]+#2 .* factorial_func \\(value=1\\) at "
        "\[\r\n\]+#3 .* factorial_func \\(value=2\\) at "
        "\[\r\n\]+#4 .* main \\(.*\\) at "
    }

If you want to send a command to GDB without increasing the test count, use gdb_test or gdb_test_no_output with an empty message (i.e., pass an empty string as the third argument). Do not use send_gdb since it won't wait for the prompt and thus will make the testcase get out of sync with the GDB output.

Convenient variables defined for use in output matching

Dejagnu defines the following convenient variables (see <dejagnu-prefix>/share/dejagnu/runtest.exp):

• hex: Matches any hexadecimal number, including the 0x prefix.

• decimal: Matches a decimal number. It does not handle + or - signs, however, and thus cannot be used to match negative numbers.

The GDB testsuite framework also defines the following convenient variables:

• octal: Matches an octal number. The number does not need to start with the digit zero.

• gdb_prompt: Matches the GDB prompt, and the GDB prompt alone. Beware that id does not match the space that usually follows it.

• inferior_exited_re: Matches the notification printed by GDB when the inferior exited, either normally or not.

• fullname_syntax: Matches what GDB considers an absolute path. The regexp is not platform-specific, and matches all known forms of absolute paths, including the Windows one, even when testing a Unix debugger on a Unix platform.

• EXEEXT: The extension used for executables. It looks like this variable is set using the environment variable EXEEXT, or else to the empty string. (FIXME: Is this correct?)

Running the Example Program in GDB

Use either gdb_run_cmd, runto, or runto_main. Do not use gdb_test with either "run" or "start" as the command being sent. This would not work when debugging with gdbserver, for instance. The same applies to gdb_start_cmd.

Note that gdb_run_cmd only sends the command needed to start the execution. It should be followed by a test that verifies the output. For example:

gdb_run_cmd
gdb_test "" ".*Breakpoint.*1.*callee.*14.*" "run"

Similarly to when building the program fails, if you want the testcase to abort, please use return.

Inserting a Breakpoint

It is fine to use gdb_test to insert the breakpoint and match the output in the usual way. In fact, the vast majority of testcase do it this way. However, the gdb.exp library provides a set of routines that can be convenient when a breakpoint needs to be inserted:

• gdb_get_line_number: This routine allows us to avoid hard-coding the line number where to insert the breakpoint. This is useful when the line number changes because of some adjustment in the code, or even some comment update as we do every year to add the new year in the copyright notices.

• gdb_breakpoint: This routines simplifies the insertion of the breakpoint and the matching of the output.

gdb_breakpoint ${srcfile}:[gdb_get_line_number "watchpoint-here"]

You should specify the source file, unless you are specifically testing the "break LINE" command. For example, if GDB finds debug information for _start, the default source file may not be what you expect!

Continuing the Execution until Reaching a Breakpoint

It is fine to use gdb_test to continue the execution and match the output in the usual way. However, there is a function provided by gdb.exp which might be useful in simplifying the writing of the test: gdb_continue_to_breakpoint.

gdb_continue_to_breakpoint "Place to set the watchpoint" ".*pattern that should be seen.*"

Implementation note: We should rewrite this routine using gdb_test_multiple.

Restarting GDB

We sometimes need to restart GDB in a testcase. Rather than using the "gdb_exit; gdb_start; gdb_reinitialize_dir; gdb_load" sequence, you can use the clean_restart function.

clean_restart $executable

Writing regexps

Writing regular expressions in Tcl can be cumbersome and/or confusing. Here are a few random notes to help.

• In double-quote "" quoted strings [ is special to Tcl, it is used to invoke an expression.

If we want to write, say, a regular expression that matches either "a" or "b" then write it as "\[ab\]". This does not apply to strings quoted with {} braces where [ has no special meaning to the Tcl parser. So one could write the same regular expression as {[ab]}. If you happen to include the \, as in {\[ab\]} that's still ok as the test harness will change that to {[ab]} before it is used as a regular expression.

• Matching period

One should be careful when trying to match period. When expect/TCL parses its input, it will process escape sequences that it sees. So "\n" will become the byte "0xa" for example. When TCL sees an escape sequence that it doesn't recognise then it just removes the backslash, so "\." becomes a period: ".". Within a TCL regular expression, as with many other regular expression engines, a "." matches any single character, which includes a literal period. Within the GDB testsuite, there has, historically, been some confusion with the use of "\.". This is often used when trying to match a literal period, which, obviously, it will match. But it also matches other things. And if what we wanted was to match anything, we'd be better off writing just "." on its own. Instead, what people should have written is "\\.", where the "\\" will result in a single backslash once the escape is resolved, thus TCL's regexp engine will see "\.".

'\n' -> Expect expands this to byte 0xa ??
'\.' -> Expect is going to expand this to: '.'
'\\' -> Expect is going to expand this to '\'
Conclusion:
'\\.' -> Regexp engine sees: '\.' 

Writing a test that generates a very long output

Expect uses a fixed-size buffer when collecting and waiting for the matching output to arrive. As a result, tests that produce very long output might exceed that size, and trigger an UNSUPPORTED result.

The best way to address this issue is by consuming the output in chunks. For an example of how to do this, see this commit (commit diff).

Writing a test with crafted DWARF debug information

The GDB testsuite has a DWARF assembler (see testsuite/lib/dwarf.exp). It takes a textual description of the DWARF as input (which is just TCL code, really) and generates an assembly file. This assembly file contains the assembler directives required to generate the DWARF sections with the requested debug information.

Here's a fairly minimal example, which you can use as a template to get started. It uses the testsuite/gdb.dwarf2/main.c file as its main source file. The main.c file is compiled without debug info, because we want to use the debug information we generate using the DWARF assembler. In this example, we pretend that the main function has a local variable whose value is a constant 3.

load_lib dwarf.exp

standard_testfile main.c -dw.S
set dwarf_file [standard_output_file $srcfile2]

Dwarf::assemble $dwarf_file {
    # Sets main_start, main_len and main_end.
    get_func_info main

    cu {} {
        DW_TAG_compile_unit {} {
            declare_labels int_label

            int_label: DW_TAG_base_type {
                { DW_AT_byte_size 4 DW_FORM_udata }
                { DW_AT_encoding @DW_ATE_signed }
                { DW_AT_name int }
            }

            DW_TAG_subprogram {
                { DW_AT_name main }
                { DW_AT_low_pc $main_start DW_FORM_addr }
                { DW_AT_high_pc $main_end DW_FORM_addr }
                { DW_AT_type :$int_label }
            } {
                DW_TAG_variable {
                    { DW_AT_name foo }
                    { DW_AT_type :$int_label }
                    { DW_AT_location {
                        DW_OP_constu 3
                        DW_OP_stack_value
                    } SPECIAL_expr }
                }
            }
        }
    }
}

if { [prepare_for_testing "failed to prepare" $testfile \
        [list $srcfile $dwarf_file] {nodebug}] } {
    return
}

if { ![runto_main] } {
    return
}

gdb_test "p foo" " = 3"

It can be executed with:

$ make check TESTS="gdb.dwarf2/example.exp"
...
# of expected passes            1
...
$ ./gdb -nx -q --data-directory=data-directory testsuite/outputs/gdb.dwarf2/example/example -batch -ex "start" -ex "print foo"
...
$1 = 3

XFAIL vs KFAIL

XFAIL is for issues outside of GDB's control. For example, known bugs of GCC or Linux.

KFAIL is for GDB bugs. These should be filed in GDB's bugzilla.

A useful mnemonic is:

XFAIL - e(X)ternal issue
KFAIL - (K)nown gdb issue

Dos and Don'ts

Make sure test messages are unique

We sometimes need to test the same operation multiple times in a testcase, so the test results would be the same and duplicated, which should be avoided. After running your new or modified testcase, please check that test messages are unique

$ make check RUNTESTFLAGS="gdb.base/mytest.exp"
$ cat testsuite/gdb.sum | grep "PASS" | sort | uniq -c | sort -n

If you see something like this below in the output, it means messages are not unique. The first column indicates how many times a single line appears, so you want to only see 1 there.

2 PASS: gdb.trace/trace-break.exp: 5 trace ftrace ftrace@0: ftrace after_set_point
2 PASS: gdb.trace/trace-break.exp: 5 trace ftrace trace@1: trace set_point

The with_test_prefix function is one way to address this -- it allows easily prefixing a set of test messages with a unique pass identifier. See also the proc_with_prefix and foreach_with_prefix convenience wrappers around with_test_prefix.

Make sure test names are environment-independent

The names of test cases must be environment-independent. That is, a given test should have the same name, regardless of factors such as the host, the target, or the directory in which the test was run. This rule makes it simpler to compare different test runs; and in particular makes it easy to run a baseline test in one directory and a patched test somewhere else.

A typical error here is to use a directory in a test name accidentally, by not specifying a test name explicitly:

gdb_test_no_output "set debug-file-directory $somewhere"

This may yield output like:

PASS: gdb.base/whatever.exp: set debug-file-directory /tmp/gdb/build/testsuite/

In this case the fix is to specify a test name:

gdb_test_no_output "set debug-file-directory $somewhere" "set debug-file-directory"

Follow the test name convention

Test names should start with a lower case and don't need to end with a period (they are not sentences). For example, prefer this

gdb_test "print 1" " = 1" "print something"

over

gdb_test "print 1" " = 1" "Print something."

Make sure test executables are unique

Each test compilation should produce a separate executable.

It is good practice for each test to have its own source code, but if for some reason a new test reuses the sources of another existing test, the new test shall compile to its own executable. E.g., if newtest.exp wants to reuse oldtest.c, then the new test should make sure to not overwrite the old tests' executable. The simplest way to do this is to use standard_testfile, which sets various globals based on the current .exp file's name.

Similarly, if a single .exp test file builds more than one executable, perhaps to exercise several variants of the same test, then each of the compilations should produce a separate executable. This makes it easier to reproduce test problems by hand. Each such executable, or shared library, should include the .exp file's base name; if the file uses standard_testfile then this is in the global testfile.

"untested" calls

In untested calls, please spell out the reason the test ends up untested, instead of just writing the test name, as with the latter we just end up with the test name duplicated in the gdb.sum output. For example:

untested mytest.exp

results in the not very helpful:

UNTESTED: gdb.base/mytest.exp: mytest.exp

A better untested call would look like:

untested "could not compile test program"

which results in the helpful:

UNTESTED: gdb.base/mytest.exp: could not compile test program

There are many bad examples in the testsuite, but we don't want to add more.

Do not refer directly to "$objdir"

Do not refer directly to $objdir. Instead, use the standard_output_file proc to compute a file name from a base name.

Do not write files into "."

Do not write files into . This is not safe when tests are run in parallel. Again, use standard_output_file to control where output is directed.

Delete all DW_AT_sibling

Attribute DW_AT_sibling is optional intended only to accelerate consumer (GDB) DWARF reader. Unless DW_AT_sibling-specific bug is under test it may only break a testcase by forgetting to update its value during various hand modifications of DWARF under test around.

Avoid using ".*" in some regular expressions

While it may be tempting to write ".*" when you are not interested in the output of certain command, it is a good practice to check for the expected output every time you can do so. This way, you will avoid incorrect PASSes in your testcase results.

Write anchored regular expressions

Try to write regular expressions that are anchored, i.e., that start and/or finish at a determined point. For example, if you are expecting some value to be printed, don't do:

gdb_test "print foo" ".*10.*"

Instead, do:

gdb_test "print foo" " = 10"

Don't write tests that run forever

It's important to not leave tests running after the testsuite completes. Sometimes it can't be helped, the harness doesn't track test processes, and if, for example, gdb crashes during the test then the test program is left running. Even if the program is in an infinite loop of "sleep(1);" leaving detritus behind is annoying.

There are two common ways to achieve this.

1. Put limits on loops.

Instead of:

 while (1)
   do_something ();

Write:

  for (i = 0; i < 1000000; ++i)
    do_something ();

Choose the loop limit appropriately.

2. Use alarm.

Only use alarm if the testcase is already posix-dependent. Ideally we want the test to be usable in as many environments as possible.

A typical value for the alarm is 60 seconds. Note that if the user is debugging the testcase then 60 seconds is too little. OTOH one doesn't want to make it too large. 600 (10 minutes) is probably large enough.

Unset/clear global array variables before using them

If you need to use global array variables in your program, you must make sure that you correctly unset/clear the variable before using it. For example:

set global_array_var ""
# Now you can use the variable
# ...

This is needed because the tests run in the same namespace, which can cause name clashing when the same name is used for global arrays.

It is also a good practice to choose unique names for those variables, though this is not really necessary if you correctly unset/clear them.

Do not use "tail parentheses" on test messages

When you write a test, do not put text between parentheses preceded by a space at the end of the text message. For example, this is incorrect:

gdb_test "some_test" "expected output" "I am a test message (123)"

The example above should be rewritten in order to remove the tail parentheses or to remove the space before parentheses.

This is because our our regression analysis tools make the assumption that the text inserted between parentheses is extra information rather than part of the test name. For instance, if the test labeled "whatever test" fails due to a time-out, the test message will be reported with a " (timeout)" suffix added at its end. Therefore:

-PASS: gdb.base/foo.exp: whatever test
+FAIL: gdb.base/foo.exp: whatever test (timeout)

is considered a regression of the same test. See https://www.sourceware.org/ml/gdb-patches/2016-03/msg00099.html for discussion on this topic.

To fix the example above, you can for example use a comma instead of the parentheses:

gdb_test "some_test" "expected output" "I am a test message, 123"

Note however that it is OK to put text between parentheses at the end of the text message, as long it is not preceded by a space. This is convenient when testing commands involving expressions that end with a function call. For example, this is perfectly fine:

gdb_test "p foo(1)" "= void"

and produces:

PASS: gdb.base/foo.exp: p foo(1)