[SCRIPT] NUMA page fault accounting.

["Jose R. Santos" <jrs at us dot ibm dot com>]

Hi folks,

My teams has had the need to do analysis of page faults that happen on a 
per NUMA node basis and I've come up with this simple SystemTap script 
that does this for me.  I only have access to a PPC64 NUMA box but I 
took care to use generic arch independed NUMA code and I think it should 
work for any kernel that has NUMA enable.  Here a short output  of 
running the stream benchmark with a 4.5GB array on a 8GB system.

Starting pagefault counters
Printing counters:
           Execname         PID        Read Faults     Write Faults
====================    ========        ===========     ============
          stream       22786                 33           668186    
Node[0]=521176  Node[1]=147043  
               stpd       22803                  1                3    
Node[0]=3       Node[1]=1       

and the script:

#! stap -g

global execnames, page_faults, node_faults

function addr_to_node:long(addr:long)
%{
       int nid;
       int pfn = __pa(THIS->addr) >> PAGE_SHIFT;
       for_each_online_node(nid)
               if ( node_start_pfn(nid) <= pfn &&
                       pfn < (node_start_pfn(nid) +
                       NODE_DATA(nid)->node_spanned_pages) )
               {
                       THIS->__retvalue = nid;                
                       break;
               }

%}

probe kernel.function("__handle_mm_fault") {
       execnames[pid()] = execname()
       page_faults [pid(), $write_access ? 1 : 0] ++
       node_faults [pid(), addr_to_node($address)] ++

}

function print_pf () {
       print ("            Execname\t     PID\tRead Faults\tWrite 
Faults\n")
       print 
("====================\t========\t===========\t============\n")
       foreach (pid in execnames) {
               printf ("%20s\t%8d\t%11d\t%12d\t", execnames[pid], pid,
                       page_faults[pid,0], page_faults[pid,1])

               foreach ([pid2,node+] in node_faults) {
                       if (pid2 == pid)
                               printf ("Node[%d]=%d\t", node,
                                       node_faults[pid2, node])
               }
               print ("\n")

       }
}

probe begin {
 print ("Starting pagefault counters \n")
}

probe end {
 print ("Printing counters: \n")
 print_pf ()
 print ("Done\n")
}


Comments welcome - Enjoy

-JRS