Subject: Re: Coverage semantics
From: Warmke, Doug (doug_warmke@mentorg.com)
Date: Wed Oct 23 2002 - 16:57:56 PDT
Michael, Joao,
Small comments embedded in the good discussion below...
Michael Rohleder wrote:
> Joao,
>
> thanks for this input. This is good stuff. I will add some comments to your proposals to show why I think this should be defined in a better way.
>
> Regards,
> Michael
>
> Joao.Geada@synopsys.com wrote:
>
>
>>To address Michael's and Bassam's points on the specific details of coverage
>>semantics, I describe below specific proposed meanings to the limit & coverage
>>numbers for line, toggle and assertion coverages.
DOUG: Joao, what are your views on the following additional types of coverage?
- expression (what parts of an expression actively caused it to be each possible state)
- path (difficult, shows percentage of all possible paths through code that were executed)
- branch or decision (measures if a control construct's control expression had all
- event or trigger (in sensitivity lists, which items caused the process to trigger)
If you agree these kinds of coverage have value and would be implemented
in various tools out there, would you mind adding them to your definitions?
BTW, thanks much for taking initiative in this area. It's very helpful.
>>
>>Line coverage:
>> limit: total number of statements in the appropriate portion of the design.
>> Statements in automatic tasks & dynamic processes are counted only once,
>> regardless of how many concurrent invocations may exist at any one time.
>> coverage: how many of the statements have been executed at least once
>> Note that the granularity of this number is by block (set of statements that
>> will execute in sequence with no change in control and without suspending for an event)
>
>
> I take from your description that you talk about basic blocks (sorry, this is a term used by software, especially compiler guys for what you talk about in the granularity). So basically you will have 'basic block coverage' meaning such a block receives a
> coverage of 1 when the set of statements contained in the basic block is executed at least once, else 0; is this correct?
>
> In general I think the name and description is very misleading. From your description I see no relationship at all to lines (so why 'line coverage') and even a reference to the 'number of statements' is a misnomer; you are always looking at the set of
> statements of one basic block.
DOUG: I agree with Michael in general.
A better term here might be "statement coverage".
It does reduce to your definition of "basic block", but users
will tend to think more about statements rather than blocks.
A reporting tool could indeed collect basic block numbers,
but then apply them to each statement in the block for
the user's benefit.
"line coverage" is definitely not too useful. Consider these lines:
if (a == b) z = 1;
x = 1; y = 1;
>
>
>>Toggle coverage:
>> limit: total number of bits in register & wire signals in the appropriate part of the design
>> (excluding arrays, but including vectors)
>> coverage: how many of the bits have completely toggled (had both 0->1 transitions *and* 1->0 transitions)
>> Granularity is 1 (1 additional bit completely toggled)
>
>
> So this is a net toggle coverage. As already said, I can envision some more coverage types w.r.t., but fine this is the current state. I don't like two things here: a) Your definition just looks at 0->1 transitions, what about other states to 1 transitions
> (e.g. z->1), so what happens in case of a 0->z->1 transition AND b) I would like to know even the lower granularity of <any>->1 as well as <anz>->0.
>
>
>>Assertion coverage:
>> limit: total number of assertions in the appropriate part of the design
>> coverage: how many of those assertions have had at least once success
>> Granularity is 1 (1 additional assertion had a first success)
>
>
> What's this. Is this your condition coverage or is this a new one?
DOUG: Seems to me this is really about SV assertions.
Condition coverage as I know it (i.e. did a given "if" control expression
simulate both true and false) isn't mentioned so far in the discussion.
>
>
>>In addition, Synopsys would be willing to add to our donation the FSM description
>>pragmas and FSM description config file, which would enable any user, regardless of
>>tool vendor to access coverage for those FSMs through the coverage API.
>
>
> Sounds good.
DOUG: Yes - thank you very much.
This will turn out to be a big service to users.
In addition, we should extend this kind of infrastructure
to the other coverage types where applicable.
>
> IN GENERAL:
>
> Besides the above concerns; would it be a major different to just return _how_often_ a statement has been covered? So you would have instead of just having a 0/1 case a 0/n case where n shows how often this basic block/toggle/... has been identified. I
> doubt that this would be a big runtime difference, but it could be even more useful information ...
>
DOUG: This seems valuable, especially in terms of statement coverage
and toggle coverage.
Thank you,
Doug Warmke
This archive was generated by hypermail 2b28 : Wed Oct 23 2002 - 16:59:32 PDT