Critically Appraisal of Guidelines

Practice guidelines have a mixed history in medicine. One the one hand, many physicians look to them for rapidly digested information and recommendations about important and common problems in medicine. On the other hand, physicians can too readily adopt them wholesale, deride them as "cookbook medicine," or feel unnecessarily constrained by their recommendations.

In the 1980's in his book Clinical Decision Making, David Eddy defined a useful framework for guidelines that reflects our level of certainty in the recommendation:

• Guideline - A recommendation that generally is useful to follow, but are flexible and may be altered due to specific patient or local circumstance. These are made from lesser quality, but still important evidence and the benefit to the patient may be small to moderate.

• Standard - A recommendations that are inflexible and should always be followed unless there is a specific contraindication. These are made from the best evidence and have a significant benefit to the patient.

• Option - A list of possible therapies or choices that really does not point to a single recommendation. This is where the evidence for the recommendations are weakest.

While these definitions are not at all standard in our medical information system, they represent a more sensible way to think about the variety of guidelines we have available.

Note that in the first two definitions above, we have weighed the "strength of evidence" and the "benefit" to the patient. This is an increasingly popular methodology used to create recommendations.

The strength of evidence is related to how certain we are that the intervention will help. And our assessment of the strength of evidence, in turn, is based on the quality of the evidence. Large, well-done randomized controlled trials (or systematic reviews of well-done RCTs) are the best source.

The benefit of the intervention to the patients is measured by "effect size", absolute risk difference, number needed to treat, etc. These statistics tell us whether the magnitude of benefit is significant or if the balance between the benefits and harms (side effects, etc.) of the intervention is too close to call.

For a good discussion about guideline concepts, see THIS PAGE at the US Preventive Services Task Force site.

See this list of guidelines to be aware of if you're in primary care.

JAMA Users’ Guides for Assessing a Clinical Guideline Recommendation

1. Is the recommendation clear and actionable?

   1. Are the patients, intervention, alternatives, and recommended action clear?

   2. Is the strength of the recommendation clear?

2. Was the evidence summarized with rigorous systematic review methods?

3. Did the guideline panel consider all outcomes important to patients?

4. Did the panel make appropriate judgments in the interpretation of the evidence and the decision of the final recommendation?

   1. Did the panel appropriately consider the magnitude of effect and the relative importance of the outcomes?

   2. Did the panel consider all relevant factors for formulating recommendations?

   3. Is the strength of the recommendation appropriate?

   4. Did the panel avoid having conflicts of interest influence their judgments?

5. Does the recommendation apply to a specific patient?

   1. Are there any important differences between the recommendation question and the clinical question of the patient?

   2. Do any of the contextual factors that have an important bearing in the recommendation differ in the patient’s setting?

AGREE II is a validated set of critical appraisal criteria that helps us grade the quality of clinical practice guidelines.

The list of AGREE II questions is below - but for much more detail and help, go to the AGREE site and read their material. These questions can help us ask the right questions about the guidelines we find.

DOMAIN 1. SCOPE AND PURPOSE

• The overall objective(s) of the guideline is (are) specifically described.

• The health question(s) covered by the guideline is (are) specifically described.

• The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described.

DOMAIN 2. STAKEHOLDER INVOLVEMENT

• The guideline development group includes individuals from all relevant professional groups.

• The views and preferences of the target population (patients, public, etc.) have been sought.

• The target users of the guideline are clearly defined.

DOMAIN 3. RIGOR OF DEVELOPMENT

• Systematic methods were used to search for evidence.

• The criteria for selecting the evidence are clearly described.

• The strengths and limitations of the body of evidence are clearly described.

• The methods for formulating the recommendations are clearly described.

• The health benefits, side effects, and risks have been considered in formulating the recommendations.

• There is an explicit link between the recommendations and the supporting evidence.

• The guideline has been externally reviewed by experts prior to its publication.

• A procedure for updating the guideline is provided.

DOMAIN 4. CLARITY OF PRESENTATION

• The recommendations are specific and unambiguous.

• The different options for management of the condition or health issue are clearly presented.

• Key recommendations are easily identifiable.

DOMAIN 5. APPLICABILITY

• The guideline describes facilitators and barriers to its application.

• The guideline provides advice and/or tools on how the recommendations can be put into practice.

• The potential resource implications of applying the recommendations have been considered.

• The guideline presents monitoring and/or auditing criteria

DOMAIN 6. EDITORIAL INDEPENDENCE

• The views of the funding body have not influenced the content of the guideline.

• Competing interests of guideline development group members have been recorded and addressed.