Critical Appraisal of Systematic Reviews - Validity

The validity of a systematic review has to do with how well the REVIEW was done first, not how well the individual trials were done. While the validity of the included studies is important, the first step is to appraise the REVIEW.

• Was the SEARCH for evidence comprehensive?

  • Did it include at least 2 different databases? Was there a search for unpublished literature? Is is likely that relevant studies were missed?

  • Were there explicit INCLUSION and EXCLUSION criteria stated and followed? Was there a table of excluded studies so that you can tell which studies didn't make the cut?

  • Were the included studies appraised for their validity (here's where the individual trials are graded)? Did the authors analyze the results with and without the low-quality studies (did they perform a sensitivity analysis)?

  • Was there an assessment for heterogeneity?

    • heterogeneity refers to differences between the studies that make it unwise to combine their results - these differences can be from patient population, methods, etc.

    • there are statistical measures of heterogeneity (p-values, I-squared statistics), etc. The most important issue, however, is that the authors didn't just look for it, but EXPLAINED the heterogeneity if they found it.

  • Did the authors assess for publication bias? Was a funnel plot provided?

    • • if studies don't show a difference between treatment and control, then they frequently don't get published, so they cannot be found in a regular search. A funnel plot can help show this graphically, but the authors should discuss whether publication bias was present or not.