This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: [pcp] suitability of PCP for event tracing
On 13/09/2010 11:29 PM, Max Matveev wrote:
On Mon, 13 Sep 2010 02:43:03 +1000, Ken McDonell wrote:
kenj> Some of the suggestions to date include ...
kenj> + data filtering predicates pushed from a client to pmcd and then on to
kenj> a pmda to enable or restrict the types of events or conditions on event
kenj> parameters that would be evaluated before asynchronously sending
kenj> matching events to the client
How would that work if multiple clients request mutually exclusive
predicates?
I wonder about this also. But my initial guess would be that predicates
from different clients could be combined with a boolean OR to register
the union of the events of interest, and then applying individual
predicates to the stream to produce the events for each client. If the
underlying event mechanism cannot support this, e.g. some process based
cpu event registers where only one process at a time can be traced, then
"first in best dressed" is probably the only protocol that will work.
kenj> + additional per-client state data being held in pmcd to allow rate
kenj> aggregation (and similar temporal averaging) to be done at pmcd, rather
kenj> than the client [note I have a long-standing objection to this approach
kenj> based on the original design criteria that pmcd needs to be mean and
kenj> lean to reduce impact, and data reduction and analysis should be pushed
kenj> out to the clients where the computation can be done without impacting
kenj> the system being monitored ... but maybe it is time to revist this, as
kenj> the current environments where PCP is being used may differ from those
kenj> we were concerned with in 1994]
Recent (2010) experience on 3rd rate platform suggests that this is
still an issue - doing too much calculation in pmda is adding to the
time it takes to fetch the data, unless pmcd can magically hide delays
induced by calculations it has to make or calculations made by pmda
ALL clients suffer.
Yep that is always a risk ... and as a general rule I'd like to see
collection being restricted at the pmcd site (e.g. don't probe for stats
that no one is asking for) and aggregated processing like rate
calculations moved out to the clients.
kenj> Depending on the set of goals we agree on, there may even be a place to
kenj> consider maintaining the poll-based mechanism, but the export data is a
kenj> variable length buffer of all event records (each aggregated and
kenj> self-identifying as below) seen since the last poll-based sample.
How will this work with multiple clients? Will the clients get a "snap
time" to indicate when pmda updated its buffer or will pmda need to
remember the state of each client (which would mean dragging client
information into pmda, maintaining that information and somehow
retiring per-client state without affecting the clients).
I think it would need per-client state in the pmda (which assumes a
protocol change between pmcd and the pmdas) ... the implementation might
involve a dynamic ring buffer, sequence numbers and last fetched
sequence number per client.
The advantage is we maintain polled and synchronous protocols between
the clients and pmcd ... it is exactly these sort of pros and cons that
I'd like to see us discussing.