MIPS and -msym32

[Crestez Dan Leonard <cdleonard at gmail dot com>]

Hello,

I've been playing around with systemtap on mips.

I used the elfutils mips patch from debian:
	http://sources.debian.net/src/elfutils/0.159-4/debian/patches/mips_backend.diff

For systemtap I ported some old patches from cisco on top of release 2.5. Here is the link to the patches on top of 1.6:
	https://sourceware.org/git/gitweb.cgi?p=systemtap.git;a=log;h=refs/heads/systemtap-1.6-cisco-patches

One of the issues I encountered is that by default 64-bit mips kernels are compiled with -msym32. This causes gcc to emit dwarf information with a pointer size of 4, even though the file is 64bit. In theory this can be disabled by passing KBUILD_SYM32=no to the kernel make commandline. This is a bad idea. It would disable some optimizations which have been enabled on mips for many years. I actually tried to do it locally but it broke module loading and I was unable to boot.

The generated dwarf files confuse systemtap is multiple ways. 

When fetching some parameters systemtap relies on address_size from dwarf_diecu to determine max_fetch_size. This caused an error like "semantic error: single register too big for fetch/store ???: identifier '$data' at ..." for an unsigned long variable.

Here is a hack I used to get around this:
--- a/libdw/dwarf_diecu.c
+++ b/libdw/dwarf_diecu.c
@@ -47,7 +47,22 @@ dwarf_diecu (die, result, address_sizep, offset_sizep)
   *result = CUDIE (die->cu);
 
   if (address_sizep != NULL)
+  {
     *address_sizep = die->cu->address_size;
+    /* Hack: */
+    if (1)
+    {
+      struct Elf *elf = die->cu->dbg->elf;
+      GElf_Ehdr ehdr_mem;
+      GElf_Ehdr* ehdr = gelf_getehdr (elf, &ehdr_mem);
+      if (ehdr &&
+              ehdr->e_machine == EM_MIPS &&
+              ehdr->e_ident[EI_CLASS] == ELFCLASS64)
+      {
+        *address_sizep = 8;
+      }
+    }
+  }
   if (offset_sizep != NULL)
     *offset_sizep = die->cu->offset_size;

This is obviously evil. What is worse is that -msym32 seems to break systemtap address mapping. Earlier I had to do the following hack inside systemtap.

diff --git a/tapsets.cxx b/tapsets.cxx
index 4e42403..94ee230 100644
--- a/tapsets.cxx
+++ b/tapsets.cxx
@@ -1265,6 +1265,22 @@ dwarf_query::add_probe_point(const string& dw_funcname,
       else
         {
           assert (has_kernel || has_module);
+          if (1)
+            {
+              Dwarf_Addr bias;
+              Elf* elf = (dwarf_getelf (dwfl_module_getdwarf (dw.module, &bias))
+                 ?: dwfl_module_getelf (dw.module, &bias));
+              GElf_Ehdr ehdr_mem;
+              GElf_Ehdr* em = gelf_getehdr (elf, &ehdr_mem);
+              int elf_machine = em->e_machine;
+              if (elf_machine == EM_MIPS)
+                {
+                  if (sess.verbose > 1)
+                    clog << "Hack reloc=" << hex << reloc_addr
+                      << " from addr=" << hex << addr;
+                  reloc_addr = addr - 0xc0000400;
+                }
+            }
           results.push_back (new dwarf_derived_probe(funcname, filename, line,
                                                      module, reloc_section, addr, reloc_addr,
                                                      *this, scope_die));

At the time I did the hack I was unaware of -msym32, I just wanted something to work. If I compile with KBUILD_SYM32=no it seems that the above hack might no longer be necessary (the addresses "look" right). I can't actually test the generated probes because my system won't boot with KBUILD_SYM32=no.

Apparently the gcc folks decided that this -msym32 behavior was too confusing and changed it to generate dwarf with a pointer size of 8:
https://gcc.gnu.org/ml/gcc/2009-01/msg00611.html

In the future this might not matter, but my cross-compiler is based on gcc-4.3.3. I'm posting this here because maybe somebody has a better idea about how to deal with this strange flavor of dwarf.

If this could be detected and handled inside systemtap fully then maybe you could apply the patches. Elfutils seems mostly blameless to me, it's just reading the information that GCC wrote.

Regards,
Leonard