Home

ISLAMIC MEDICAL EDUCATION RESOURCES 04

0308-META ANALYSIS

Lecture at the FIFTH ADVANCED ASIAN COURSE IN TROPICAL EPIDEMIOLOGY INSTITUTE FOR MEDICAL RESEARCH KUALA LUMPUR 18-29 AUGUST 2003 By Professor Dr Omar Hasan Kasule, Sr. MB ChB (MUK), MPH, DrPH (Harvard) eputy Dean for Research, Faculty of Medicine, UIA PO Box 141 Kuantan Pahang MALAYSIA Tel 609 513 2797 Fax 609 513 3615 E-M omarkasule@yahoo.com

Learning Objectives

Definition of Meta analysis

Uses of meta analysis

Methods of meta analysis

Limitations of meta analysis

 

Key words and key terms

Review article

Summary quantitative statistic

Over conclusion

 

AGENDA

A. Definition And Historical Background

B. Advantages Of Meta Analysis

C. Steps In Meta Analysis

D. Difficulties Of Meta Analysis

E. Reading Results Of Meta Analysis

 

PRE-TEST

1. The following statements are true about meta-analysis

Meta analysis combines data from more than one study to produce a quantitative summary statistic

Meta analysis picks up missed statistical significance because of a larger effective sample size

Meta-analysis enables study of variations of effect measures across different population groups

The odds ratio or the regression coefficient are the usual summary measures

The summary effect measure is a weighted summary of the effect measures of individual studies

DEFINITION

Combining data from more than one study

Producing a quantitative summary statistic.

 

HISTORICAL BACKGROUND

Review articles

Written by discipline leaders

Subjective

Summarization of conclusions without looking at original data

 

Meta analysis

First developed for randomized clinical trials

Use in observational studies is more problematic

Difficulty of knowing and adjusting for confounders.

 

ADVANTAGES OF META ANALYSIS

Computation of an effect estimate for a larger number of study subjects

Picking up statistical significance that would be missed by small individual studies

Study of variation across several population subgroups

Transparent reviews based on quantitative assessments.

 

STEPS IN META ANALYSIS

Identification of the variables of interest

Outcome

Exposure

Confounder

Intermediate

Modifying

 

Identification of relevant articles from data bases such as medline

Use of variables and key words

Inclusion/exclusion criteria

Determination of effect measures and population to be covered

Data abstraction on a standard form (raw data is preferred if available)

Display of effect measures (with 95% CI) for each study

Computation of the summary effect measure, OR or b

Each individual study is treated as a stratum

Tests of homogeneity

Compute by weighted logistic regression or MH weighted average

Weight of each measure the inverse of its precision ie 1/(se)2

Adjustment of the summary statistic for bias

Confounding bias

Selection bias

Misclassification bias

Sensitivity analysis to test the robustness of the combined effect measure.

 

 DIFFICULTIES OF META ANALYSIS

Differences in study designs, study analysis, and data quality

Over conclusion

Artifactual conclusion

Conclusion not supported by aggregated data

Bias

Publication bias

Positive results favored

Studies in academic or government institutions favored

Studies by pharmaceutical firms suspected

Selection bias

Inadequate search for reports

Multiple publications of  the same study

Bias (conscious or unconscious) in study selection

Use of wrong methods

Conclusion based on the proportion of positive studies

Scatter plot of test statistics to show a general trend or correlation

Using SD as a standardized measure of the deviation from the center

Quality scoring of the various studies.

READING RESULTS OF META ANALYSIS

Methods clearly stated?

Search for articles comprehensive enough?

Criteria for selecting articles for review stated?

Criteria objective?

Criteria adhered to?

Possibility of bias in the selection of articles?

Methodologic quality of each article assessed?

Differences between studies explained?

Combination of results from the primary studies appropriate?

Conclusions of the reviewer supported by data?

 

SYNOPSIS

Meta analysis refers to methods used to combine data from more than one study to produce a quantitative summary statistic. Meta analysis enables computation of an effect estimate for a larger number of study subjects thus enabling picking up statistical significance that would be missed if analysis was based on small individual studies. Meta analysis also enables study of variation across several population subgroups since it involves several individual studies carried out in various countries and populations. Criteria must be set for what articles to include or exclude. Information is abstracted from the articles on a standardized data abstract form with standard outcome, exposure, confounder, or effect modifying variables. The first step is to display the effect measures with their 95% confidence limits to get a general idea of their distribution before proceeding to compute summary measures. The summary effect measure, OR or b, is computed from the effect measures of individual studies using weighted logistic regression or computing a MH weighted average in which the weight of each measure is the inverse of its precision ie 1/(se)2. In both the logistic or MH procedures, each study is treated as a stratum. The combined effect measure is then statistically adjusted for confounding, selection, and misclassification biases. Tests of homogeneity can be carried out before computing the summary effect measure. Sensitivity analysis is undertaken to test the robustness of the combined effect measure.

 

A. DEFINITION and HISTORICAL BACKGROUND

DEFINITION

Meta analysis refers to methods used to combine data from more than one study to produce a quantitative summary statistic.

HISTORICAL BACKGROUND

Review articles written by discipline leaders were the most popular method of combining findings from various studies. They however had two serious draw-backs. They were subjective and therefore prone to error and bias. The reviewer was free to make decisions on what data and conclusions to emphasize. Secondly they dealt with summarization of conclusions without looking at the data on which the conclusions were based. Meta analyis was first developed for randomized clinical trials. Its use in observational studies is more problematic mainly because of the difficulty of knowing and adjusting for confounders.

 

B. ADVANTAGES OF META ANALYSIS

Meta-analysis has become popular with the proliferation of epidemiological studies on particular subjects. Writers of review articles and practising epidemiologists would like to have some form of consensus or summary of the findings of various studies. Meta analysis enables computation of an effect estimate for a larger number of study subjects thus enabling picking up statistical significance that would be missed if analysis was based on small individual studies. Many clinical trials especially with invasive intervention can not recruit enough patients in one center to reach statistical significance. Meta analysis also enables study of variation across several population subgroups since it involves several individual studies carried out in various countries and populations. Meta analysis makes the process of reviewing several studies with  view to reaching a general conclusion very transparent because it is based on quantitative assessments.

 

C. STEPS IN META ANALYSIS

The first step is to identify the variables of interest such as outcome, exposure, confounder, intermediate, and effect modifying variables. The variables are used together with other relevant key words to identify relevant articles from the data bases of such as medline or from other unpublished sources. Criteria must be set for what articles to include or exclude. These include definition of what effect measures are used and the populations covered. It is best if there is as much homogeneity in the population as is practical. Data collection is carried out by abstracting information from the articles on a standardized data abstract form with standard outcome, exposure, confounder, or effect modifying variables. The ideal is to abstract raw dta and reanalyze it in a standard way to enable combination of several studies to get one summary effect measure. In practice the raw data is not available and only effect measures are available. The first step is to display the effect measures with their 95% confidence limits to get a general idea of their distribution before proceeding to compute summary measures. The summary effect measure, OR or b, is computed from the effect measures of individual studies using weighted logistic regression or computing a MH weighted average in which the weight of each measure is the inverse of its precision ie 1/(se)2. In both the logistic or MH procedures, each study is treated as a stratum. The combined effect measure is then statistically adjusted for confounding, selection, and misclassification biases. Tests of homogeneity can be carried out before computing the summary effect measure. Sensitivity analysis is undertaken to test the robustness of the combined effect measure.

 

D. DIFFICULTIES OF META ANALYSIS

Meta analysis is methodologically complex because of different study designs, study analysis, and even different data quality. The major problems are: overconclusion and bias (publication bias, selection bias), and use of wrong methods. Overconclousion arises when a conclusion is artifactual and is not supported by the aggregated data.The results of metaanalysis based on published sources may not reflect the true situation because of existence of publication bias. Positive findings are more likely to be published than negative ones. Studies carried out in academic or government institutions are thought to be more credible and are therefore more likely to be published whereas studies by pharmaceutical firms have a lower publication rates. Inadequate search for reports may lead to bias just as multiple publications of  the same study data leads to bias. Bias, conscious or unconscious, may occur in the selection of studies for analysis. In some cases the methods used for metaanalysis are wrong or inappropriate such as reaching a conclusion based on the proportion of positive studies, a scatterplot of test statistics to show a general trend or correlation, use of the standard deviation as a standardized measure of the deviation of the effect measures from the center, or quality scoring of the various studies.

 

E. READING RESULTS OF META ANALYSIS

The following questions should be considered when reading review or meta-analysis articles: are the methods clearly stated (b) was the search for articles comprehensive enough?,  were the criteria for selecting articles for review stated, were the criteria objective and were they adhered to?, was there a possibility of bias in the selection of those articles?,  was the methodologic quality of each article assessed?, were differences between studies explained or were they just glossed over?,  was the combination of results from the primary studies appropriate?, and were the conclusions of the reviewer supported by data?

Professor Omar Hasan Kasule August 2003