Scripts for slab allocator

[William Cohen <wcohen at redhat dot com>]

Hi,

I have been looking at instrumenting the slab allocator in the kernel. I have 
written a couple of scripts to collect data to give people a better idea what is 
going on with the slab allocator. I am looking for feedback on the scripts. 
Currently they don't work with the 2.6.24 kernels (or things using the slub 
allocator). I know that the tokenize() function isn't need for newer versions of 
stap.

The slabuse2.stp script takes a string listing the slab object that it should 
monitor. The second argument is a threshold when to collect data. When the 
number allocations of an object is greater than the number of frees for that 
object stack backtraces are recorded. When the script is stopped with a 
control-c it prints out the top 20 slab allocations and top 20 stack back 
traces. For example:

 stap  slabuse2.stp -g "ext3_inode_cache dentry_cache" 100

The cache_grow.stp records backtraces for for the various times that slabs need 
to be grown in size.

I would like to extend these to work with newer kernels and provide better 
rendering of information. Comments?


-Will
# slabuse2.stp
# This script captures the stack back trace for a each slab object
# once a threshold active objects is reached for the object.
# The following would track allocations for ext3_inode_cache and
# dentry_cache objects whenever the script observes slab object 
# allocations exceeding slab object frees by  100
#
# stap  -g slabuse2.stp "ext3_inode_cache dentry_cache" 100
#
# slabuse2.stp is for use on older versions of systemtap, 0.5.12
# which do not have the tokenize() functions

/* the collect_data variable is set when problem condition occurs */
global name_of_interest
global alloc_active

global stats, stacks

probe kernel.function("kmem_cache_alloc") {
  name = kernel_string($cachep->name)
  count_of_interest = name_of_interest[name]
  if (count_of_interest > 0) {
    alloc_active[name]++
    if (alloc_active[name] >= name_of_interest[name]) {
      exec = execname()
      stats[exec, name] <<< 1
      stacks[exec, name, backtrace()] <<< 1
    }
  }
}

probe kernel.function("kmem_cache_free") {
  name = kernel_string($cachep->name)
  if (alloc_active[name]) alloc_active[name]--
}

probe begin {
  /* tokenize argument for list of allocations to watch */
  /* argument is of the form:  "ext3_inode_cache dentry_cache" 100 */
  name = tokenize(@1, " ")
  count = $2
  while (name != "" ) {
    name_of_interest[name] = count
    name = tokenize("", " ")
  }
}

probe end {
  printf("Number of slab allocations by process and slab name\n")
  foreach ([exec, name] in stats- limit 20) {
    printf("%s, %s:\t%d\n", exec, name, @count(stats[exec, name]))
  }

  printf("\nBacktrace of processes when allocating\n")
  foreach ([exec, cache, bt] in stacks- limit 20) {
    printf("Exec: %s Slab name: %s  Count: %d\n",
       exec, cache, @count(stacks[exec, cache, bt]))
         print_stack(bt)
         printf("\n-------------------------------------------------------\n\n")
  }
}

/*
 * tokenize - Given a string and a token delimiter,
 *            return the next token in the string
 * input  String to tokenize. If NULL, returns the next token in the
 *        string passed in the previous call to tokenize().
 * delim  Token delimiter. Note this is a string, but only the first
 *        character is used as the delimiter.
 */
function tokenize:string(input:string, delim:string)
%{ /* pure */
	static char str[MAXSTRINGLEN];
	static char *str_start;
	char *token = NULL;

	if (THIS->input[0]) {
		strncpy(str, THIS->input, MAXSTRINGLEN);
		str_start = &str[0];
	}
	token = strsep(&str_start, THIS->delim);
	if (token)
		strncpy (THIS->__retvalue, token, MAXSTRINGLEN);
%}

# cache_grow.stp
# This script captures tallies stack back traces each time the slab cache
# size is increased. The program is run with the following command:
#
# stap  cache_grow.stp
#
# When the script completes it prints out top twenty executable that
# caused slabs to grow. This is followed by stack backtraces showing path
# in kernel to the cache_grow function.

global stats, stacks

probe kernel.function("cache_grow") {
  name = kernel_string($cachep->name)
  exec = execname()
  stats[exec, name] <<< 1
  stacks[exec, name, backtrace()] <<< 1
}


probe begin {
  printf("Type ctrl-c to stop script, and print out data (top 20 lists)\n")
}

probe end {
  printf("Number of cache_grow called by process and slab name\n")
  foreach ([exec, name] in stats- limit 20) {
    printf("%s, %s:\t%d\n", exec, name, @count(stats[exec, name]))
  }

  printf("\nBacktrace of processes when cache_grow called\n")
  foreach ([exec, cache, bt] in stacks- limit 20) {
    printf("Exec: %s Slab name: %s  Count: %d\n",
       exec, cache, @count(stacks[exec, cache, bt]))
    print_stack(bt)
    printf("\n-------------------------------------------------------\n\n")
  }
}