GDB BuildBot
Contents
This wiki page describes how the GDB BuildBot works. Read it if you are interested in adding a new buildslave to the system, if you want to know who is responsible for each existing buildslave, or if you are just curious about how the BuildBot is configured.
1. DISCLAIMER
![/!\](/moin_static1910/modern/img/alert.png "/!\"){kind=small}
PLEASE READ CAREFULLY
It has become clear that it is important to define the roles of each individual/community when it comes to our BuildBot. Please read the excerpt below, taken from Joel Brobecker's e-mail message at https://sourceware.org/ml/gdb-patches/2017-12/msg00417.html:
The GDB project being a volunteer-based community-driven project, a port, and the associated resources, can only survive if there is a group of people with time and motivation to match both the port and the available resources. When resources are put in place for those platforms, and those resources fail to be useful, the people to turn to for answers are those who pushed for them to be put in place. Asking the community in general to be responsible does not seem like an attainable goal, nor is it attractive to me. That is not to say that the community is entirely care free either. I think we see nearly every day people cooperating on subjects that do not directly matter to them, and as far as I am concerned going well beyond the call of duty. But there has to be a reasonable limit to what we ask of people to help for other platforms. Contributing hardware is not a fire-and-forget thing. For the issues with the buildbots, there has to be someone who has a stake in those platforms, and willing to drive the effort in getting those back in shape. These are the people we should turn to, not the person who provides the service for those who want to. And if there are none and the resource becomes a nuisance rather than a help, then the logical answer is to remove it. If anything, what this discussion leads me to, is that the lessons learned here is that we should be clearer about an identified group who would be responsible for what before a new buildbot is being put in place, and also about what level of "service" can be expected from the people maintaining the infrastructure, vs what concerns the platforms the buildbots build and test.
In other words:
Please DO NOT assume that contributing a machine to the GDB BuildBot project means that the GDB community will automatically be responsible for maintaining that specific port in good shape.
If you care about XYZ and contribute a XYZ machine/system to the GDB BuildBot, please keep in mind that the GDB community and the GDB BuildBot maintainer reserve the right to disable the e-mail notifications generated by the XYZ builders if they are not reliable.
Do not expect the GDB BuildBot maintainer to keep track of all the failures happening when building GDB on your system.
Do not expect the GDB community or the GDB BuildBot maintainer to fix the problems happening when building GDB on your system. We will help you investigate the issue whenever we can, though.
2. TL;DR
The web interface for the GDB BuildBot is http://gdb-buildbot.osci.io/.
The URL for connecting your buildslave to the GDB BuildBot is gdb-buildbot.osci.io:16123. You must contact us first in order to obtain permission to do that (and a password); send a message to <gdb@sourceware.org> and put <sergiodj@redhat.com> in the Cc.
The interface for viewing the results of the builds is http://gdb-buildbot.osci.io/results .
Regression reports are sent to the gdb-testers mailing list.
3. New terms
In this page, we will use a few specific terms to describe things, so it is good to understand what they mean.
BuildBot: This is the name of the project we use to test GDB (see more info here).
buildmaster: BuildBot has a master node, which is responsible for deciding which tasks each slave will perform, collecting the results of each task, and providing the interface for viewing the results.
buildslave': This is the slave node of BuildBot. Slaves are "dumb", in the sense that they only execute things when master tells them to, and they usually do not have any knowledge of the entire task being performed. A typical configuration for BuildBot is to have one buildmaster, and one or more buildslaves connected to it. This is what GDB does.
build step: A step is what the name says: a step in the build process. For example, checking out the source code of the project that will be tested is a step.
build factory: A factory is like a recipe of how to perform a build. It can contain instructions for how to prepare the build directory, how to checkout the sources, how to compile them, how to test them, and how to compare the test results. A build factory is composed by one or more build steps.
builder: A builder is an instance of a factory. It is the build process itself. For example, one possible builder for GDB is Fedora-x86_64-native-gdbserver-m64, which is responsible for building and testing a 64-bit GDB on a x86_64 machine, and testing the native-gdbserver there.
4. How to view previous build logs
All the logs for all builds in the last 6 months are stored at http://gdb-buildbot.osci.io/results/. As you can see when you enter the page, the logs are organized by builder, then by the first two letters of the commit hash, and then by the commit hash itself. So, for example, if you wanted to see the build logs generated by the Fedora-x864_64-m64 builder for a fictitious commit 0a1b2c3d4e, you would have to go to http://gdb-buildbot.osci.io/results/Fedora-x864_64-m64/0a/0a1b2c3d4e.
The logs are compressed using xz in order to save space, so you have to uncompress before viewing them. A direct way to look at the gdb.log of the commit mentioned above, for example, is:
wget http://gdb-buildbot.osci.io/results/Fedora-x864_64-m64/0a/0a1b2c3d4e/gdb.log.xz -O - | xzcat | pager
We also store logs for try builds (see below for more information on try builds). To access them, go to the try directory inside the builder directory. For example, http://gdb-buildbot.osci.io/results/Fedora-x864_64-m64/try/0a/0a1b2c3d4e.
5. How to add your buildslave
If you are interested in adding a new buildslave (and possibly a new builder) to the GDB BuildBot, thank you! There are some steps you need to perform in order to make sure everything is OK.
5.1. Buildslave specs
Your machine needs to:
Be able to properly compile and test GDB. Make sure all the necessary dependencies are installed in your system.
Be able to run Python, and to run the buildbot-worker package. The BuildBot master is running a recent version of the software (at the time of this writing, 2.3.1), which means that you can safely install a recent version of the buildbot-worker package. Keep in mind that your buildbot-worker package needs to be the Python 3 version of the package. For that reason, a buildbot-slave package whose version is 0.8.x will not work.
Have a decent amount of disk space to perform the git clone, the build and the test.
It is also recommended that your machine is constantly updated (e.g., on Fedora you might consider running a cronjob to dnf update every day), because GDB's test results are impacted by updates on GCC, glibc, binutils, etc.
5.2. Buildslave owner responsibilities
You will be the responsible for the buildslave, which means that we may contact you if there is anything you should do (i.e., update the buildbot-worker package, restart your buildslave, etc.). Be sure to fill the contact fields on your buildslave configuration file (see below).
5.3. Contact us first
Please, contact us in order to discuss further details, and to let us know about you. The best way to contact the GDB community is to write to <gdb@sourceware.org>. Please, make sure you also include Sergio Durigan Junior in the Cc (e-mail: <sergiodj@redhat.com>), because he is the current maintainer of the GDB BuildBot.
5.4. Adding your buildslave to the configuration
You will need to patch the following files and add your buildslave configuration there:
- config.json
- master.cfg
On config.json, you will have to add your buildslave information on the slaves section. You should provide:
The name of your buildslave. Please pick a descriptive name. Do not use whitespaces nor special characters.
The arch, i.e., the architecture of your buildslave. Possible values are x86_64, ppc64, s390x, etc.
The jobs property, which says how many parallel jobs can be triggered by make when compiling GDB. This does not affect make check, because it is always done in non-parallel.
This is an example of how a new entry should look like:
"slaves" : [ { "name" : "fedora-x86-64-1", "arch" : "x86_64", "jobs" : "4" },
...
{ "name" : "debian-x86-64-1", "arch" : "x86_64", "jobs" : "2" }
],Then, you will have to edit the builders section. In this section we describe what kind of build we are interested in, and what are the buildslaves that will perform it. If you are adding a buildslave with a new architecture, then you will most probably want to create a new builder for it. However, you can also be interested in adding more buildslaves to an existing builder. In both cases, you will have to edit the following properties:
The name of the builder. Please pick a descriptive name; for example, if the builder will build and test native-gdbserver 32-bit on Fedora, with a 64-bit GDB, your builder should be called Fedora-x86_64-native-gdbserver-m32 (actually, this is the name of an existing builder). Do not use whitespaces or special characters.
The type of the builder. This is one of the most important fields. It will be used as the prefix for the RunTestGDB* set of classes, in order to determine how to build and test GDB. Using the same example as above, the type would be NativeGDBServer_c64t32. The c64t32 part means that GDB will be compiled as a 64-bit binary, but the tests will be run in 32-bit mode (i.e., RUNTESTFLAGS='--target_board=native-gdbserver/-m32').
The builddir, which is the name of the directory that will be created on the buildmaster to store information about this build. Please use the name of the builder as the builddir, but with lowercase chars.
The slavenames property, which is a list of buildslaves that can perform this build. It is extremely important to include only buildslaves capable of building for the specified architecture. This is what a new addittion should look like:
"builders" : [ { "name" : "Fedora-x86_64-m64", "type" : "Plain_c64t64",
"builddir" : "fedora-x86-64",
"slavenames" : [ "fedora-x86-64-1" ] },
...
{ "name" : "Debian-x86_64-m64", "type" : "Plain_c64t64",
"builddir" : "debian-x86-64",
"slavenames" : [ "debian-x86-64-1" ] },At last, if you created a new builder, you will also want to include its name in the builderNames property of the schedulers section.
After you have done this, please send a patch to the <gdb-patches@sourceware.org> mailing list. You will be provided a password used by the buildslave to connect to the buildmaster.
5.5. Buildslave configuration
The first thing you will need to do is to install the buildbot-slave, version 0.8.14, package on your system. To do that, run (as root):
pip install buildbot-worker
You may need to install the Python 3 development files in order to build buildbot-worker.
If this doesn't work, you will have to find the specific version on BuildBot's website. If you are running Windows, take a look at BuildBot's instructions.
5.5.1. Fedora-specific instructions
You will need to install the dependencies to compile and test GDB. For Fedora systems:
(set -ex;dnf install fedpkg rpm-build;fedpkg clone -a gdb;cd gdb;fedpkg sources;rpmbuild --define "_specdir $PWD" --define "_sourcedir $PWD" --define "_srcrpmdir $PWD" -bs --with testsuite *.spec;dnf builddep gdb-*.src.rpm)
Please make sure to re-run this command whenever you upgrade your Fedora! New dependencies can be added to GDB, and it is important to keep up-to-date with them.
Do not forget to disable the abrt-ccpp service on Fedora. It will make ABRT be invoked every time a coredump is generate, which consumes time and is totally unnecessary. To disable it, do (as root):
$ systemctl stop abrt-ccpp.service $ systemctl disable abrt-ccpp.service
If you're running a version of Fedora that has abrtd, you can also disable it:
$ systemctl stop abrtd.service $ systemctl disable abrtd.service
Also make sure that core dumps can be generated in the current directory, so that core dump tests are not skipped. E.g., on Fedora:
$ echo "kernel.core_pattern=core" > /etc/sysctl.d/50-coredump.conf $ sysctl --system
If you're having permission problems or any other strange problem, try disabling SELinux:
$ setenforce 0
If it works, you will have to permanently disable it by editing the file /etc/selinux/config and setting SELINUX=disabled.
5.5.2. Debian/Ubuntu-specific instructions
For Debian-based distributions, you can do:
$ apt-get build-dep gdb $ apt-get install texinfo flex bison $ apt-get install emacs
Please make sure to re-run this command whenever you upgrade your Debian! New dependencies can be added to GDB, and it is important to keep up-to-date with them.
5.5.3. General instructions
Because of the make TAGS step, you will also have to install emacs.
If your machine will not test 32-bit packages, you do not need to install the 32-bit (i.e., .i686) versions of the packages above.
It has been reported that the tests won't run properly if the SHELL variable is not set to a valid shell (e.g. /sbin/nologin instead of /bin/bash). Make sure that the user running the buildslave has a valid shell or override the SHELL environment variable somehow.
It is also recommended that you create a separate user to run the buildslave. This user should not need/have root nor SSH access; it only needs to be able to build and test GDB.
For example, let's assume this new user is called buildbot. Log into its account in the system, and create a new buildslave. To do that, you are going to need three things:
- The name of the buildslave (should be the same name you put in the JSON configuration file)
The address of the buildmaster, which is gdb-buildbot.osci.io:16123
- The password that was provided to you
Then, you can issue the command:
$ buildbot-worker create-worker DIR URL:PORT NAME PASSWORD
Where DIR is the directory that will be used by the buildslave to store local files (it can be the same as NAME), URL:PORT is the address of the buildmaster, and NAME is the name of the buildslave. The PASSWORD needs to be obtained from the BuildBot admin.
5.5.4. Adding information about yourself
You must add your contact information on your buildslave. To do that, edit the file named DIR/info/admin (assuming that DIR is the name of your buildslave, as per the example above) and add your name and e-mail address there:
$ cat DIR/info/admin My Name <my@email.com>
5.5.5. Adding information about the machine
You should also edit the file DIR/info/host and include information about the machine that is running the buildslave. This is very useful when a failure happens and someone is trying to reproduce it, because knowing the versions of the Linux kernel, g++ and other tools can help narrow down the problem. In order to do that, it is recommended that you put at least the following information in your DIR/info/host file:
<Distro name and release> <Linux kernel information (from uname)> <Version of g++> <Version of binutils>
A good idea is to create a script that updates this information every time the system reboots, because that is usually a sign that some packages have been upgraded. In other words, it's your responsibility to keep this information up-to-date on your buildslave :-).
5.5.6. Starting the buildslave
After that, you can start the buildslave:
$ buildbot-worker start NAME
Make sure it is started OK. It is also recommended that you create some special cronjob to restart this on every reboot. An example would be:
$ crontab -l # restart buildslaves on every reboot @reboot buildbot-worker restart NAME
6. Try Jobs
The Try scheduler is now configured on our BuildBot. What this means is that a developer will be able to upload a patch to our BuildBot, and this patch will then be tested against a set of builders we have there.
6.1. Concepts
As you know, we have several "slaves" running on different locations. The master doesn't do any builds itself, but rather controls all the details about the builds that need to be performed. The slaves do not take any decision by themselves; they just do what the master tells them to do.
On top of the slaves, there is an abstraction called "builder". The builder is basically a recipe of how to do a build. This includes the compilation flags used, environment variables, and testsuite options. A slave can have multiple builders; a builder can also be assigned to multiple slaves (for example, when we have more than one slave with the same hardware configuration, as is the case of the Fedora x86_64 slaves).
Then, we have the schedulers. They are responsible for monitoring events and triggering builds. Our main scheduler monitors our repository, and triggers a full build on all builders whenever a new commit has been pushed. We also have "cron job" schedulers, used to trigger racy testcase detection builds every week at a certain time. And now, we have the "try job" scheduler, responsible for triggering a build on a set of builders when a developer has submitted a patch for testing.
6.2. Build priority
Before the try scheduler, builds were basically treated in a "first come, first served" basis. No special priority was assigned to a build.
Now, I decided to try a new algorithm. Considering that a developer usually want to test patches when they are almost ready for submission, it would be really annoying if she had to wait in line until her build is processed; it is not rare to see some builders having 10 or more builds in the queue. So, my decision was to "move" the try builds to the top of the queue. This will obviously delay the testing of the official repository, but I am hoping that the impact will not be very big. If it is, then this decision will have to be revisited.
6.3. Security
There are a few considerations to be made about the security of a try job. The obvious one is that a developer may upload a malicious patch to be "tested", and there is no easy way for BuildBot to detect this. Since the BuildBot master does not build anything, it would be the slaves who would suffer from this kind of attack.
But there is another thing to consider. In order to be able to upload a patch to be tested, the developer must have some kind of access to the BuildBot master. I am the maintainer of the service (and of the machine where the service is running), and obviously I am not happy about this. I will write more on the authentication mechanism below.
So, after giving a long thought about this problem, I decided that I will only provide accounts to people who already have commit access to the GDB repository.
If you want access, you need to contact me and provide a proof that you have commit access (a commit made by you should be enough).
6.4. Authentication
The BuildBot try command can use SSH for authentication, so that is what I have chosen. Basically, you will need to send me your public SSH key; no passwords are needed/accepted. You will be using a common user to log in, and will not receive a PTY (i.e., will not have shell access). If you ever used gitolite, then you know the drill. If not, it's pretty simple.
I will provide a configuration file below.
6.5. Installing buildbot
In order to use the new feature, you will have to install the buildbot package. You don't need the "slave" package! On Debian, an apt-get install buildbot should do the trick. On Fedora, dnf install buildbot. However, Fedora enables the BuildBot systemd service after the installation; I strongly recommend you disable that, unless you intend to run a BuildBot master on your machine.
It is also important to mention that the buildbot binary needs to be a recent one, Python 3 enabled. Version 0.8.x won't work.
6.6. The "buildbot try" command
After installing the package, the only command you need to know is "buildbot try". It will be responsible for generating a patch out of the git tree you're in (assuming you don't provide a patch explicitly), connecting to the server and submitting the patch to be tested.
I strongly recommend that you read the command's documentation at:
http://docs.buildbot.net/current/manual/cmdline.html#try
However, the usage is pretty basic. There are two modes: you can either be in a local branch and ask "buildbot try" to automatically figure out the patch to be tested, or you can provide the patch file explicitly.
In the first mode, "buildbot try" will generate a diff of your local branch against your "master" branch, and will instruct the BuildBot master to test the patch *using the master's branch HEAD revision*. This means that if you want to generate a patch to be tested against the latest commit of the official repository, you have to do a "git pull" on your master branch first (and rebase your local branch, of course).
The second mode accepts a patch file via the "--diff" parameter. Originally, "buildbot try" will not specify a revision against which the patch needs to be tested; however, I have modified that behaviour on the server side, and the provided patch will always be tested against the latest commit of the official repository.
6.7. Configuration file for "buildbot try"
Here is the configuration file that you need to use for the "buildbot try" command to work. This file needs to be named $HOME/.buildbot/options.
$> cat $HOME/.buildbot/options
# The method of connection to the BuildBot master.
try_connect = 'ssh'
# The host of the BuildBot master.
try_host = 'gdb-buildbot.osci.io'
# Our username. We will always use this username!
try_username = 'gdb-buildbot'
# The jobdir on BuildBot master (i.e., the directory where "buildbot
# try" will put the request files.
try_jobdir = 'gdb/try_ssh_jobdir'
# The VCS we are using for the project.
try_vc = 'git'
# The branch against which we will try our patches. Currently, we
# only support 'master'.
try_branch = 'master'
# Who am I?
# NOTE: YOU NEED TO USE THE FORMAT "NAME <EMAIL>" IN THIS FIELD!
try_who = 'John Smith <john@example.com>'
# The builders I want to test my patch in.
# The list of available builder can be found in:
# <http://sourceware.org/gdb/wiki/BuildBot#Available_Builders>
try_builders = [ "Fedora-x86_64-m64",
"Fedora-x86_64-m32",
"Fedora-x86_64-native-gdbserver-m64",
"Fedora-x86_64-native-gdbserver-m32",
"Fedora-x86_64-native-extended-gdbserver-m64",
"Fedora-x86_64-native-extended-gdbserver-m32",
"Fedora-x86_64-cc-with-index",
"Fedora-i686",
"Fedora-x86_64-w64-mingw32",
"Fedora-s390x-m64",
"Debian-s390x-m64",
"Debian-s390x-native-gdbserver-m64",
"Debian-s390x-native-extended-gdbserver-m64",
"AIX-POWER8-plain",
"RHEL-s390x-m64",
"NetBSD-x86_64-m64",
"Solaris11-amd64-m64",
"Solaris11-sparcv9-m64",
"Ubuntu-Aarch64-m64",
"Ubuntu-Aarch64-native-gdbserver-m64",
"Ubuntu-Aarch64-native-extended-gdbserver-m64",
"Ubuntu-ARMhf-m32",
"Ubuntu-ARMhf-native-gdbserver-m32",
"Ubuntu-ARMhf-native-extended-gdbserver-m32" ]
$>It is good to mention that most of these options can be overriden by command line options, so you can (for example) choose to test your patch against only a subset of the available try_builders.
6.8. Notifications
The "buildbot try" command offers a way to wait for the builds to complete on the command line, but this doesn't work when you're using SSH. However, you will receive e-mails from our BuildBot notifying you about the results of all of your try builds. The e-mails will be very similar to those sent to the gdb-testers mailing list.
6.9. Build specifics
As mentioned above, the try builds will almost always be performed against the latest commit on the official repository. They will also run using the very same flags that are being used for each builder (unfortunately, there is no way to change these flags on try builds). And the results will be compared against the last results that our BuildBot has on file.
For example, if our BuildBot has just tested the commit 0abcd on a certain builder, and then you request it to test your patch on the same builder, BuildBot will run the full build of your patch, generate the testsuite logs, and compare them against the testsuite logs generated by testing 0abcd. The known XFAIL files for that specific builder will also be taken into account, i.e., they will be discarded when comparing the testsuite logs.
Last, but not least, your try build will generate its own testsuite logs, which will be stored inside the builder's directory, at:
http://gdb-buildbot.osci.io/results/<<BUILDER>>/try/<<FIRST TWO LETTERS OF COMMIT>>/<<COMMIT>>/<<TRY COUNT>>/
The TRY COUNT can be obtained from the build status page. Just search for try_count there.
6.10. Available Builders
The list of available builders for a try build can be found at:
https://git.sergiodj.net/gdb-buildbot.git/tree/lib/config.json)-- Search for the entry named GDB_Try_Jobdir, and look at the builderNames list. That is the most updated list of try builders.