One thought: I would ask the application community for evaluation
metrics. What is it they are trying to achieve? If they can't tell
you, then any comparision is specious.
Second thought: in my work on the on-line maintenance agent
(D'Ambrosio and Burgess, UAI-96), we worked hard to find a domain
value based metric for the quality of inference, rather than the more
usual "error bound" kind of metric.
Even if the application community is looking for information rather
than decisions, I would look for some kind of cost or loss function.
Failing all that, I currently find ROC curves (receiver operating
characteristic, originated in the communications theory community, I
believe) very valuable utility-free characterizers of diagnostic or
interpretation systems. They are decision-oriented, but value-model
independent.
cheers - Bruce