From 6dfe7b73e1e2345e80a572f9f46f1759743d00a8 Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Tue, 2 Apr 2024 16:11:41 +0100 Subject: [PATCH 1/9] add design notation docs page --- docs/source/design_notation.md | 90 ++++++++++++++++++++++++++++++++++ docs/source/index.rst | 5 +- 2 files changed, 93 insertions(+), 2 deletions(-) create mode 100644 docs/source/design_notation.md diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md new file mode 100644 index 00000000..7d64402a --- /dev/null +++ b/docs/source/design_notation.md @@ -0,0 +1,90 @@ +# Experimental design notation + +This page provides a concise summary of the tabular notation used by Shadish Cook & Campbell (2002) and Reichardt (2009). This notation provides a compact description of various experimental designs. While it is possible to describe randomised designs using this notation, we focus purely on quasi-experimental designs here, with non-random allocation (abbreviated as `NR`). Observations are denoted by $O$. Time proceeds from left to right, so observations made through time are labelled as $O_1$, $O_2$, etc. The treatment is denoted by `X`. Rows represent different groups of units. Remember, a unit is a person, place, or thing that is the subject of the study. + +## Pretest-posttest designs + +One of the simplest designs is the pretest-posttest design. Here we have one row, denoting a single group of units. There is an `X` which means all are treated. The pretest is denoted by $O_1$ and the posttest by $O_2$. See p99 of Reichardt (2019). + +| | | | +|----|---|----| +$O_1$ | X | $O_2$ | + +Informally, if we think about drawing conclusions about the causal impact of the treatment based on the change from $O_1$ to $O_2$, we might say that the treatment caused the change. However, this is a tenuous conclusion because we have no way of knowing what would have happened in the absence of the treatment. + +A variation of this design which may (slightly) improve this situation from the perspective of making causal claims, would be to take multiple pretest measures. This is shown below, see p107 of Reichardt (2019). + +| | | | | +|----|--|---|----| +$O_1$ | $O_2$ | X | $O_3$ | + +This would allow us to estimate how the group was changing over time before the treatment was introduced. This could be used to make a stronger causal claim about the impact of the treatment. + +## Nonequivalent group designs + +In randomized experiments, with large enough groups, the randomization process should ensure that the treatment and control groups are equivalent. However, in quasi-experimental designs, with non-random (`NR`) allocation, we could expect there to be differences between the treatment and control groups. This poses some challenges in making strong causal claims about the impact of the treatment. + +For example, in the simplest nonequivalent group design, we have two groups, one treated and one not treated, and just one posttest. See p114 of Reichardt (2019). + +| | | | +|-----|---|----| +| NR: | X | $O_1$ | +| NR: | | $O_1$ | + +The above design would be considered weak - the lack of a pre-test measure makes it hard to know whether differences between the groups at $O_1$ are due to the treatment or to pre-existing differences between the groups. + +This limitation can be addressed by adding a pretest measure. See p115 of Reichardt (2019). + +| | | | | +|-----|----|---|----| +| NR: | $O_1$ | X | $O_2$ | +| NR: | $O_1$ | | $O_2$ | + +Non-equivalent group designs like this, with a pretest and a posttest measure could be analysed in a number of ways: +1. **ANCOVA:** Here, the group would be a categorical predictor, the pretest measure would be a covariate, and the posttest measure would be the outcome. +2. **Difference-in-differences:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. + +A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures. See p154 of Reichardt (2019). + +| | | | | | +|-----|----|---|-|----| +| NR: | $O_1$ | $O_2$ | X | $O_3$ | +| NR: | $O_1$ | $O_2$ | | $O_3$ | + +Again, this design could be analysed using the difference-in-differences approach. + +## Interrupted time series designs + +While there is no control group, the interrupted time series design is a powerful quasi-experimental design that can be used to estimate the causal impact of a treatment. The design involves multiple pretest and posttest measures. The treatment is introduced at a specific point in time, denoted by `X`. The design can be used to estimate the causal impact of the treatment by comparing the trajectory of the outcome variable before and after the treatment. See p203 of Reichardt (2019). + +| | | | | | | | | | +|-----|----|---|----|---|----|----|----|----| +| $O_1$ | $O_2$ | $O_3$ | $O_4$ | X | $O_5$ | $O_6$ | $O_7$ | $O_8$ | + +## Comparative interrupted time series designs + +The comparative interrupted time series design incorporates aspects of **interrupted time series** (with only a treatment group), and **nonequivalent group designs** (with a treatment and control group). This design can be used to estimate the causal impact of a treatment by comparing the trajectory of the outcome variable before and after the treatment in the treatment group, and comparing this to the trajectory of the outcome variable in the control group. See p226 of Reichardt (2019). + +| | | | | | | | | | | +|-----|----|---|----|---|----|----|----|----|-| +| NR: | $O_1$ | $O_2$ | $O_3$ | $O_4$ | X | $O_5$ | $O_6$ | $O_7$ | $O_8$ | +| NR: | $O_1$ | $O_2$ | $O_3$ | $O_4$ | | $O_5$ | $O_6$ | $O_7$ | $O_8$ | + + +Because this design is very similar to the nonequivalent group design, simply with multiple pre and ppost test measures, it is well-suited to analysis under the difference-in-differences approach. + +However, if we have many untreated units and one treated unit, then this design could be analysed with the synthetic control approach. + +## Regression discontinuity designs + +The design notation for regression discontinuity designs are different from the others and take a bit of getting used to. We have two groups, but allocation to the groups are determined by a units' relation to a cutoff point `C` along a running variable. Also, $O_1$ now represents the value of the running variable, and $O_2$ represents the outcome variable. See p169 of Reichardt (2019). This will make more sense if you consider the design notation alongside one of the example notebooks. + +| | | | | +|-----|----|---|----| +| C: | $O_1$ | X | $O_2$ | +| C: | $O_1$ | | $O_2$ | + +From an analysis perspective, regression discontinuity designs are very similar to interrupted time series designs. The key difference is that treatment is determined by a cutoff point along a running variable, rather than by time. + +## Summary +This page has offered a brief overview of the tabular notation used to describe quasi-experimental designs. The notation is a useful tool for summarizing the design of a study, and can be used to help identify the strengths and limitations of a study design. But readers are strongly encouraged to consult the original sources when assessing the relative strengths and limitations of making causal claims under different quasi-experimental designs. diff --git a/docs/source/index.rst b/docs/source/index.rst index b69b3f4a..4e3cc9c8 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -138,9 +138,10 @@ Documentation outline ===================== .. toctree:: - :titlesonly: + :caption: Knowledge Base - glossary + design_notation.md + glossary.rst .. toctree:: :caption: Examples From 132b423463d02c47b5d5d0bec2f12cb6776a4f3c Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Tue, 2 Apr 2024 16:23:12 +0100 Subject: [PATCH 2/9] update title of page --- docs/source/design_notation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 7d64402a..073810a1 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -1,4 +1,4 @@ -# Experimental design notation +# Quasi-experimental design notation This page provides a concise summary of the tabular notation used by Shadish Cook & Campbell (2002) and Reichardt (2009). This notation provides a compact description of various experimental designs. While it is possible to describe randomised designs using this notation, we focus purely on quasi-experimental designs here, with non-random allocation (abbreviated as `NR`). Observations are denoted by $O$. Time proceeds from left to right, so observations made through time are labelled as $O_1$, $O_2$, etc. The treatment is denoted by `X`. Rows represent different groups of units. Remember, a unit is a person, place, or thing that is the subject of the study. From dd16414fd84610b7e7c284f87340eb480c63929d Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Wed, 3 Apr 2024 09:38:28 +0100 Subject: [PATCH 3/9] fix typo --- docs/source/design_notation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 073810a1..a986bf47 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -71,7 +71,7 @@ The comparative interrupted time series design incorporates aspects of **interru | NR: | $O_1$ | $O_2$ | $O_3$ | $O_4$ | | $O_5$ | $O_6$ | $O_7$ | $O_8$ | -Because this design is very similar to the nonequivalent group design, simply with multiple pre and ppost test measures, it is well-suited to analysis under the difference-in-differences approach. +Because this design is very similar to the nonequivalent group design, simply with multiple pre and post test measures, it is well-suited to analysis under the difference-in-differences approach. However, if we have many untreated units and one treated unit, then this design could be analysed with the synthetic control approach. From c341a62595279f53c527f81654fbc118216bb6b3 Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Wed, 3 Apr 2024 10:51:44 +0100 Subject: [PATCH 4/9] add references and glossary terms --- docs/source/design_notation.md | 31 ++++++++++++++++++------------- docs/source/glossary.rst | 6 ++++-- docs/source/references.bib | 8 ++++++++ 3 files changed, 30 insertions(+), 15 deletions(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index a986bf47..4e0a9967 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -1,18 +1,18 @@ # Quasi-experimental design notation -This page provides a concise summary of the tabular notation used by Shadish Cook & Campbell (2002) and Reichardt (2009). This notation provides a compact description of various experimental designs. While it is possible to describe randomised designs using this notation, we focus purely on quasi-experimental designs here, with non-random allocation (abbreviated as `NR`). Observations are denoted by $O$. Time proceeds from left to right, so observations made through time are labelled as $O_1$, $O_2$, etc. The treatment is denoted by `X`. Rows represent different groups of units. Remember, a unit is a person, place, or thing that is the subject of the study. +This page provides a concise summary of the tabular notation used by {cite:t}`shadish_cook_cambell_2002` and {cite:t}`reichardt2019quasi`. This notation provides a compact description of various experimental designs. While it is possible to describe randomised designs using this notation, we focus purely on {term}`quasi-experimental` designs here, with non-random allocation (abbreviated as `NR`). Observations are denoted by $O$. Time proceeds from left to right, so observations made through time are labelled as $O_1$, $O_2$, etc. The treatment is denoted by `X`. Rows represent different groups of units. Remember, a unit is a person, place, or thing that is the subject of the study. ## Pretest-posttest designs -One of the simplest designs is the pretest-posttest design. Here we have one row, denoting a single group of units. There is an `X` which means all are treated. The pretest is denoted by $O_1$ and the posttest by $O_2$. See p99 of Reichardt (2019). +One of the simplest designs is the pretest-posttest design. Here we have one row, denoting a single group of units. There is an `X` which means all are treated. The pretest is denoted by $O_1$ and the posttest by $O_2$. See p99 of {cite:t}`reichardt2019quasi`. | | | | |----|---|----| $O_1$ | X | $O_2$ | -Informally, if we think about drawing conclusions about the causal impact of the treatment based on the change from $O_1$ to $O_2$, we might say that the treatment caused the change. However, this is a tenuous conclusion because we have no way of knowing what would have happened in the absence of the treatment. +Informally, if we think about drawing conclusions about the {term}`causal impact` of the treatment based on the change from $O_1$ to $O_2$, we might say that the treatment caused the change. However, this is a tenuous conclusion because we have no way of knowing what would have happened in the absence of the treatment. -A variation of this design which may (slightly) improve this situation from the perspective of making causal claims, would be to take multiple pretest measures. This is shown below, see p107 of Reichardt (2019). +A variation of this design which may (slightly) improve this situation from the perspective of making causal claims, would be to take multiple pretest measures. This is shown below, see p107 of {cite:t}`reichardt2019quasi`. | | | | | |----|--|---|----| @@ -24,7 +24,7 @@ This would allow us to estimate how the group was changing over time before the In randomized experiments, with large enough groups, the randomization process should ensure that the treatment and control groups are equivalent. However, in quasi-experimental designs, with non-random (`NR`) allocation, we could expect there to be differences between the treatment and control groups. This poses some challenges in making strong causal claims about the impact of the treatment. -For example, in the simplest nonequivalent group design, we have two groups, one treated and one not treated, and just one posttest. See p114 of Reichardt (2019). +For example, in the simplest {term}`nonequivalent group design`, we have two groups, one treated and one not treated, and just one posttest. See p114 of {cite:t}`reichardt2019quasi`. | | | | |-----|---|----| @@ -33,7 +33,7 @@ For example, in the simplest nonequivalent group design, we have two groups, one The above design would be considered weak - the lack of a pre-test measure makes it hard to know whether differences between the groups at $O_1$ are due to the treatment or to pre-existing differences between the groups. -This limitation can be addressed by adding a pretest measure. See p115 of Reichardt (2019). +This limitation can be addressed by adding a pretest measure. See p115 of {cite:t}`reichardt2019quasi`. | | | | | |-----|----|---|----| @@ -41,10 +41,10 @@ This limitation can be addressed by adding a pretest measure. See p115 of Reicha | NR: | $O_1$ | | $O_2$ | Non-equivalent group designs like this, with a pretest and a posttest measure could be analysed in a number of ways: -1. **ANCOVA:** Here, the group would be a categorical predictor, the pretest measure would be a covariate, and the posttest measure would be the outcome. -2. **Difference-in-differences:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. +1. **{term}`ANCOVA`:** Here, the group would be a categorical predictor, the pretest measure would be a covariate, and the posttest measure would be the outcome. +2. **{term}`Difference in differences`:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. -A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures. See p154 of Reichardt (2019). +A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures. See p154 of {cite:t}`reichardt2019quasi`. | | | | | | |-----|----|---|-|----| @@ -55,7 +55,7 @@ Again, this design could be analysed using the difference-in-differences approac ## Interrupted time series designs -While there is no control group, the interrupted time series design is a powerful quasi-experimental design that can be used to estimate the causal impact of a treatment. The design involves multiple pretest and posttest measures. The treatment is introduced at a specific point in time, denoted by `X`. The design can be used to estimate the causal impact of the treatment by comparing the trajectory of the outcome variable before and after the treatment. See p203 of Reichardt (2019). +While there is no control group, the {term}`interrupted time series design` is a powerful quasi-experimental design that can be used to estimate the causal impact of a treatment. The design involves multiple pretest and posttest measures. The treatment is introduced at a specific point in time, denoted by `X`. The design can be used to estimate the causal impact of the treatment by comparing the trajectory of the outcome variable before and after the treatment. See p203 of {cite:t}`reichardt2019quasi`. | | | | | | | | | | |-----|----|---|----|---|----|----|----|----| @@ -63,7 +63,7 @@ While there is no control group, the interrupted time series design is a powerfu ## Comparative interrupted time series designs -The comparative interrupted time series design incorporates aspects of **interrupted time series** (with only a treatment group), and **nonequivalent group designs** (with a treatment and control group). This design can be used to estimate the causal impact of a treatment by comparing the trajectory of the outcome variable before and after the treatment in the treatment group, and comparing this to the trajectory of the outcome variable in the control group. See p226 of Reichardt (2019). +The {term}`comparative interrupted time-series` design incorporates aspects of **interrupted time series** (with only a treatment group), and **nonequivalent group designs** (with a treatment and control group). This design can be used to estimate the causal impact of a treatment by comparing the trajectory of the outcome variable before and after the treatment in the treatment group, and comparing this to the trajectory of the outcome variable in the control group. See p226 of {cite:t}`reichardt2019quasi`. | | | | | | | | | | | |-----|----|---|----|---|----|----|----|----|-| @@ -73,11 +73,11 @@ The comparative interrupted time series design incorporates aspects of **interru Because this design is very similar to the nonequivalent group design, simply with multiple pre and post test measures, it is well-suited to analysis under the difference-in-differences approach. -However, if we have many untreated units and one treated unit, then this design could be analysed with the synthetic control approach. +However, if we have many untreated units and one treated unit, then this design could be analysed with the {term}`synthetic control` approach. ## Regression discontinuity designs -The design notation for regression discontinuity designs are different from the others and take a bit of getting used to. We have two groups, but allocation to the groups are determined by a units' relation to a cutoff point `C` along a running variable. Also, $O_1$ now represents the value of the running variable, and $O_2$ represents the outcome variable. See p169 of Reichardt (2019). This will make more sense if you consider the design notation alongside one of the example notebooks. +The design notation for {term}`regression discontinuity designs` are different from the others and take a bit of getting used to. We have two groups, but allocation to the groups are determined by a units' relation to a cutoff point `C` along a running variable. Also, $O_1$ now represents the value of the running variable, and $O_2$ represents the outcome variable. See p169 of {cite:t}`reichardt2019quasi`. This will make more sense if you consider the design notation alongside one of the example notebooks. | | | | | |-----|----|---|----| @@ -88,3 +88,8 @@ From an analysis perspective, regression discontinuity designs are very similar ## Summary This page has offered a brief overview of the tabular notation used to describe quasi-experimental designs. The notation is a useful tool for summarizing the design of a study, and can be used to help identify the strengths and limitations of a study design. But readers are strongly encouraged to consult the original sources when assessing the relative strengths and limitations of making causal claims under different quasi-experimental designs. + +## References +:::{bibliography} +:filter: docname in docnames +::: diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index 2251d2ce..2769a2e8 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -18,6 +18,9 @@ Glossary Change score analysis A statistical procedure where the outcome variable is the difference between the posttest and protest scores. + Causal impact + An umbrella term for the estimated effect of a treatment on an outcome. + Comparative interrupted time-series CITS An interrupted time series design with added comparison time series observations. @@ -36,7 +39,6 @@ Glossary ITS A quasi-experimental design to estimate a treatment effect where a series of observations are collected before and after a treatment. No control group is present. - Instrumental Variable regression IV A quasi-experimental design to estimate a treatment effect where the is a risk of confounding between the treatment and the outcome due to endogeniety. @@ -67,6 +69,7 @@ Glossary An emprical comparison used to estimate the effects of treatments where units are assigned to treatment conditions randomly. Regression discontinuity design + RDD A quasi–experimental comparison to estimate a treatment effect where units are assigned to treatment conditions based on a cut-off score on a quantitative assignment variable (aka running variable). Regression kink design @@ -88,7 +91,6 @@ Glossary Wilkinson notation A notation for describing statistical models :footcite:p:`wilkinson1973symbolic`. - Two Stage Least Squares 2SLS An estimation technique for estimating the parameters of an IV regression. It takes its name from the fact that it uses two OLS regressions - a first and second stage. diff --git a/docs/source/references.bib b/docs/source/references.bib index 9f30ae53..08034e5a 100644 --- a/docs/source/references.bib +++ b/docs/source/references.bib @@ -68,3 +68,11 @@ @article{acemoglu2001colonial pages={1369--1401}, year={2001} } + +@book{shadish_cook_cambell_2002, + title={Experimental and quasi-experimental designs for generalized causal inference}, + author={Cook, Thomas D and Campbell, Donald Thomas and Shadish, William}, + volume={1195}, + year={2002}, + publisher={Houghton Mifflin Boston, MA} +} From 79cf08ddf87b03564e4e736e811585c951e5bef4 Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Wed, 3 Apr 2024 10:57:34 +0100 Subject: [PATCH 5/9] expand on "equivalent" --- docs/source/design_notation.md | 6 ++++-- docs/source/glossary.rst | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 4e0a9967..6522de9e 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -22,9 +22,11 @@ This would allow us to estimate how the group was changing over time before the ## Nonequivalent group designs -In randomized experiments, with large enough groups, the randomization process should ensure that the treatment and control groups are equivalent. However, in quasi-experimental designs, with non-random (`NR`) allocation, we could expect there to be differences between the treatment and control groups. This poses some challenges in making strong causal claims about the impact of the treatment. +In randomized experiments, with large enough groups, the randomization process should ensure that the treatment and control groups are approximately equivalent in terms of their attributes. This is positive for causal inference as we can be more sure that differences between control and test groups are due to treatment exposure, not because of differences in attributes of the groups. -For example, in the simplest {term}`nonequivalent group design`, we have two groups, one treated and one not treated, and just one posttest. See p114 of {cite:t}`reichardt2019quasi`. +However, in quasi-experimental designs, with non-random (`NR`) allocation, we could expect there to be differences between the treatment and control groups' attributes. This poses some challenges in making strong causal claims about the impact of the treatment - we can't be sure that differences between the groups at the posttest are due to the treatment, or due to pre-existing differences between the groups. + +In the simplest {term}`nonequivalent group design`, we have two groups, one treated and one not treated, and just one posttest. See p114 of {cite:t}`reichardt2019quasi`. | | | | |-----|---|----| diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index 2769a2e8..d3d5ab0f 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -48,7 +48,7 @@ Glossary Non-equivalent group designs NEGD - A quasi-experimental design where units are assigned to conditions non-randomly, and not according to a running variable (see Regression discontinuity design). + A quasi-experimental design where units are assigned to conditions non-randomly, and not according to a running variable (see Regression discontinuity design). This can be problematic when assigning causal influence of the treatment - differences in outcomes between groups could be due to the treatment or due to differences in the group attributes themselves. One-group posttest-only design A design where a single group is exposed to a treatment and assessed on an outcome measure. There is no pretest measure or comparison group. From f7c292c16d4bd2bd02462d67406db6a55d87d673 Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Wed, 3 Apr 2024 11:05:12 +0100 Subject: [PATCH 6/9] add content on parallel trends assumption --- docs/source/design_notation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 6522de9e..2cf94be9 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -44,9 +44,9 @@ This limitation can be addressed by adding a pretest measure. See p115 of {cite: Non-equivalent group designs like this, with a pretest and a posttest measure could be analysed in a number of ways: 1. **{term}`ANCOVA`:** Here, the group would be a categorical predictor, the pretest measure would be a covariate, and the posttest measure would be the outcome. -2. **{term}`Difference in differences`:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. +2. **{term}`Difference in differences`:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. Note that this approach has a strong assumption of [parallel trends](https://en.wikipedia.org/wiki/Difference_in_differences#Assumptions) - that the treatment and control groups would have changed in the same way in the absence of the treatment. -A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures. See p154 of {cite:t}`reichardt2019quasi`. +A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures and can help in assessing if the parallel trends assumption is reasonable. See p154 of {cite:t}`reichardt2019quasi`. | | | | | | |-----|----|---|-|----| From 2b3fb84c5c85e89fcada769e3be2b4de7cd35b56 Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Wed, 3 Apr 2024 12:18:20 +0100 Subject: [PATCH 7/9] edit ANCOVA description --- docs/source/design_notation.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 2cf94be9..52f070a1 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -43,7 +43,7 @@ This limitation can be addressed by adding a pretest measure. See p115 of {cite: | NR: | $O_1$ | | $O_2$ | Non-equivalent group designs like this, with a pretest and a posttest measure could be analysed in a number of ways: -1. **{term}`ANCOVA`:** Here, the group would be a categorical predictor, the pretest measure would be a covariate, and the posttest measure would be the outcome. +1. **{term}`ANCOVA`:** Here, the group would be a categorical predictor (e.g. treated/untreated), the pretest measure would be a covariate (though there could be more than one), and the posttest measure would be the outcome. 2. **{term}`Difference in differences`:** We can apply linear modeling approaches such as `y ~ group + time + group:time` to estimate the treatment effect. Here, `y` is the outcome measure, `group` is a binary variable indicating treatment or control group, and `time` is a binary variable indicating pretest or posttest. Note that this approach has a strong assumption of [parallel trends](https://en.wikipedia.org/wiki/Difference_in_differences#Assumptions) - that the treatment and control groups would have changed in the same way in the absence of the treatment. A limitation of the nonequivalent group designs with single pre and posttest measures is that we don't know how the groups were changing over time before the treatment was introduced. This can be addressed by adding multiple pretest measures and can help in assessing if the parallel trends assumption is reasonable. See p154 of {cite:t}`reichardt2019quasi`. From 092654e6f7b29f0a4aa5cb615417e88ac734112b Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Thu, 4 Apr 2024 09:25:01 +0100 Subject: [PATCH 8/9] add glossary reference + fix typo --- docs/source/design_notation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index 52f070a1..bf952ed5 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -10,7 +10,7 @@ One of the simplest designs is the pretest-posttest design. Here we have one row |----|---|----| $O_1$ | X | $O_2$ | -Informally, if we think about drawing conclusions about the {term}`causal impact` of the treatment based on the change from $O_1$ to $O_2$, we might say that the treatment caused the change. However, this is a tenuous conclusion because we have no way of knowing what would have happened in the absence of the treatment. +Informally, if we think about drawing conclusions about the {term}`causal impact` of the treatment based on the change from $O_1$ to $O_2$, we might say that the treatment caused the change. However, this is a tenuous conclusion because we have no way of knowing what would have happened in the ({term}`counterfactual`) absence of the treatment. A variation of this design which may (slightly) improve this situation from the perspective of making causal claims, would be to take multiple pretest measures. This is shown below, see p107 of {cite:t}`reichardt2019quasi`. @@ -73,7 +73,7 @@ The {term}`comparative interrupted time-series` design incorporates aspect | NR: | $O_1$ | $O_2$ | $O_3$ | $O_4$ | | $O_5$ | $O_6$ | $O_7$ | $O_8$ | -Because this design is very similar to the nonequivalent group design, simply with multiple pre and post test measures, it is well-suited to analysis under the difference-in-differences approach. +Because this design is very similar to the nonequivalent group design, simply with multiple pre and posttest measures, it is well-suited to analysis under the difference-in-differences approach. However, if we have many untreated units and one treated unit, then this design could be analysed with the {term}`synthetic control` approach. From fe097e305049c2fca82a0537405c0e2944e3cf7d Mon Sep 17 00:00:00 2001 From: "Benjamin T. Vincent" Date: Fri, 5 Apr 2024 10:26:37 +0100 Subject: [PATCH 9/9] minor tweaks + additional glossary term --- docs/source/design_notation.md | 6 ++++-- docs/source/glossary.rst | 3 +++ 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/source/design_notation.md b/docs/source/design_notation.md index bf952ed5..1fdacf63 100644 --- a/docs/source/design_notation.md +++ b/docs/source/design_notation.md @@ -18,7 +18,7 @@ A variation of this design which may (slightly) improve this situation from the |----|--|---|----| $O_1$ | $O_2$ | X | $O_3$ | -This would allow us to estimate how the group was changing over time before the treatment was introduced. This could be used to make a stronger causal claim about the impact of the treatment. +This would allow us to estimate how the group was changing over time before the treatment was introduced. This could be used to make a stronger causal claim about the impact of the treatment. We could use {term}`interrupted time series` analysis to help here. ## Nonequivalent group designs @@ -63,6 +63,8 @@ While there is no control group, the {term}`interrupted time series design` is a |-----|----|---|----|---|----|----|----|----| | $O_1$ | $O_2$ | $O_3$ | $O_4$ | X | $O_5$ | $O_6$ | $O_7$ | $O_8$ | +You can see that this is an example of a pretest-posttest design with multiple pre and posttest measures. + ## Comparative interrupted time series designs The {term}`comparative interrupted time-series` design incorporates aspects of **interrupted time series** (with only a treatment group), and **nonequivalent group designs** (with a treatment and control group). This design can be used to estimate the causal impact of a treatment by comparing the trajectory of the outcome variable before and after the treatment in the treatment group, and comparing this to the trajectory of the outcome variable in the control group. See p226 of {cite:t}`reichardt2019quasi`. @@ -79,7 +81,7 @@ However, if we have many untreated units and one treated unit, then this design ## Regression discontinuity designs -The design notation for {term}`regression discontinuity designs` are different from the others and take a bit of getting used to. We have two groups, but allocation to the groups are determined by a units' relation to a cutoff point `C` along a running variable. Also, $O_1$ now represents the value of the running variable, and $O_2$ represents the outcome variable. See p169 of {cite:t}`reichardt2019quasi`. This will make more sense if you consider the design notation alongside one of the example notebooks. +The design notation for {term}`regression discontinuity designs` are different from the others and take a bit of getting used to. We have two groups, but allocation to the groups are determined by a units' relation to a cutoff point `C` along a {term}`running variable`. Also, $O_1$ now represents the value of the running variable, and $O_2$ represents the outcome variable. See p169 of {cite:t}`reichardt2019quasi`. This will make more sense if you consider the design notation alongside one of the example notebooks. | | | | | |-----|----|---|----| diff --git a/docs/source/glossary.rst b/docs/source/glossary.rst index d3d5ab0f..c6aedbb2 100644 --- a/docs/source/glossary.rst +++ b/docs/source/glossary.rst @@ -75,6 +75,9 @@ Glossary Regression kink design A quasi-experimental research design that estimates treatment effects by analyzing the impact of a treatment or intervention precisely at a defined threshold or "kink" point in a quantitative assignment variable (running variable). Unlike traditional regression discontinuity designs, regression kink design looks for a change in the slope of an outcome variable at the kink, instead of a discontinuity. This is useful when the assignment variable is not discrete, jumping from 0 to 1 at a threshold. Instead, regression kink designs are appropriate when there is a change in the first derivative of the assignment function at the kink point. + Running variable + In regression discontinuity designs, the running variable is the variable that determines the assignment of units to treatment or control conditions. This is typically a continuous variable. Examples could include a test score, age, income, or spatial location. But the running variable would not be time, which is the case in interrupted time series designs. + Sharp regression discontinuity design A Regression discontinuity design where allocation to treatment or control is determined by a sharp threshold / step function.