Stata version bug

[johnklopfer]

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = "http://cran.us.r-project.org") : unable to install packages

[jonathandroth]

Hi, thank you for bringing this to our attention!

Can you try running

install.packages("devtools", repos = "http://cran.us.r-project.org")

in the R terminal and let me know if it works? I am trying to figure out whether this is an issue on the R end or Stata end.

On Thu, Aug 5, 2021 at 8:50 AM johnklopfer @.***> wrote:

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = " http://cran.us.r-project.org") : unable to install packages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[johnklopfer]

For sure - thank you for writing software! That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there. John

On Aug 6, 2021, at 3:42 AM, Jonathan Roth @.***> wrote:

Hi, thank you for bringing this to our attention!

Can you try running

install.packages("devtools", repos = "http://cran.us.r-project.org")

in the R terminal and let me know if it works? I am trying to figure out whether this is an issue on the R end or Stata end.

On Thu, Aug 5, 2021 at 8:50 AM johnklopfer @.***> wrote:

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = " http://cran.us.r-project.org") : unable to install packages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-893734099, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[johnklopfer]

I tinkered around a little more today and can add -

1. The problem was with RCall to command line R. After I installed RStudio and all of the dependencies, the Stata version works without any error messages.

2. staggered in RStudio is throwing the warning message:

Warning message: In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", : The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.

after I used command staggered(df = short,

•       i = "FIPS",
  

•       t = "SURVYEAR",
  

•       g = "YSFR1",
  

•       y = "PMTE11", 
  

•       estimand = "simple”) 
  

on the attached file:

3. staggered in Stata produces the same result, but does not throw the warning message.

4. staggered_cs in Stata (estimand simple) throws the same warning message and produces

       'estimate'         'se'  'se_neyman'
   

'simple' 178.26605 343.76784 409.91009

while csdid in Stata (estimand simple) produces

         |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]

-------------+---------------------------------------------------------------- ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372

5. If I drop the cohorts named in the warning message, csdid in Stata (estimand simple) produces

         |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
   

-------------+---------------------------------------------------------------- ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084

I think the major difference here is the dropped cohorts, though I know there are other differences between the two programs. Is there a reason to drop cohorts with one cross-sectional unit, when 2x2s are going to be aggregated later? It would be great to be able to use the full dataset without disaggregating from the FIPSxYEAR level I’m using here!

Hope some of this is useful :)

John

On Aug 6, 2021, at 8:39 AM, John Klopfer @.***> wrote:

For sure - thank you for writing software! That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there. John

On Aug 6, 2021, at 3:42 AM, Jonathan Roth @.*** @.***>> wrote:

Hi, thank you for bringing this to our attention!

Can you try running

install.packages("devtools", repos = "http://cran.us.r-project.org http://cran.us.r-project.org/")

in the R terminal and let me know if it works? I am trying to figure out whether this is an issue on the R end or Stata end.

On Thu, Aug 5, 2021 at 8:50 AM johnklopfer @.***> wrote:

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = " http://cran.us.r-project.org http://cran.us.r-project.org/") : unable to install packages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/jonathandroth/staggered/issues/10 https://github.com/jonathandroth/staggered/issues/10>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-893734099, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[pedrohcgs]

Hi John, Would you be able to share the data (or subset) so we can dig into this more closely?

You can send to my email, if you want to.

Thanks

On Fri, Aug 6, 2021 at 04:14 johnklopfer @.***> wrote:

I tinkered around a little more today and can add -

1. The problem was with RCall to command line R. After I installed RStudio and all of the dependencies, the Stata version works without any error messages.

2. staggered in RStudio is throwing the warning message:

Warning message: In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", : The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.

after I used command staggered(df = short,

• i = "FIPS",
• t = "SURVYEAR",
• g = "YSFR1",
• y = "PMTE11",
• estimand = "simple”)

on the attached file:

3. staggered in Stata produces the same result, but does not throw the warning message.

4. staggered_cs in Stata (estimand simple) throws the same warning message and produces

'estimate' 'se' 'se_neyman' 'simple' 178.26605 343.76784 409.91009

while csdid in Stata (estimand simple) produces

| Coef. Std. Err. z P>|z| [95% Conf. Interval]

-------------+---------------------------------------------------------------- ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372

5. If I drop the cohorts named in the warning message, csdid in Stata (estimand simple) produces

| Coef. Std. Err. z P>|z| [95% Conf. Interval]

-------------+---------------------------------------------------------------- ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084

I think the major difference here is the dropped cohorts, though I know there are other differences between the two programs. Is there a reason to drop cohorts with one cross-sectional unit, when 2x2s are going to be aggregated later? It would be great to be able to use the full dataset without disaggregating from the FIPSxYEAR level I’m using here!

Hope some of this is useful :)

John

On Aug 6, 2021, at 8:39 AM, John Klopfer @.***> wrote:

For sure - thank you for writing software! That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there. John

On Aug 6, 2021, at 3:42 AM, Jonathan Roth @.*** @.***>> wrote:

Hi, thank you for bringing this to our attention!

Can you try running

install.packages("devtools", repos = "http://cran.us.r-project.org < http://cran.us.r-project.org/>")

in the R terminal and let me know if it works? I am trying to figure out whether this is an issue on the R end or Stata end.

On Thu, Aug 5, 2021 at 8:50 AM johnklopfer @.***> wrote:

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = " http://cran.us.r-project.org http://cran.us.r-project.org/") : unable to install packages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/jonathandroth/staggered/issues/10 < https://github.com/jonathandroth/staggered/issues/10>>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A < https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/jonathandroth/staggered/issues/10#issuecomment-893734099>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

—

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-894189252, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABE734YTWH2R2A25LHEPUM3T3O7Z7ANCNFSM5BTZNY3A .

--

Pedro H. C. Sant'Anna Department of Economics Vanderbilt University 615-875-8448 (phone) @.*** https://pedrohcgs.github.io

[johnklopfer]

Hmm I guess Github drops attachments - here it is again!

Thank you both,

John

On Aug 6, 2021, at 10:34 PM, pedrohcgs @.***> wrote:

Hi John, Would you be able to share the data (or subset) so we can dig into this more closely?

You can send to my email, if you want to.

Thanks

On Fri, Aug 6, 2021 at 04:14 johnklopfer @.***> wrote:

I tinkered around a little more today and can add -

1. The problem was with RCall to command line R. After I installed RStudio and all of the dependencies, the Stata version works without any error messages.

2. staggered in RStudio is throwing the warning message:

Warning message: In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", : The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.

after I used command staggered(df = short,

• i = "FIPS",
• t = "SURVYEAR",
• g = "YSFR1",
• y = "PMTE11",
• estimand = "simple”)

on the attached file:

3. staggered in Stata produces the same result, but does not throw the warning message.

4. staggered_cs in Stata (estimand simple) throws the same warning message and produces

'estimate' 'se' 'se_neyman' 'simple' 178.26605 343.76784 409.91009

while csdid in Stata (estimand simple) produces

| Coef. Std. Err. z P>|z| [95% Conf. Interval]

-------------+---------------------------------------------------------------- ATT | 423.6928 384.5374 1.10 0.271 -329.9867 1177.372

5. If I drop the cohorts named in the warning message, csdid in Stata (estimand simple) produces

| Coef. Std. Err. z P>|z| [95% Conf. Interval]

-------------+---------------------------------------------------------------- ATT | 161.1675 386.1504 0.42 0.676 -595.6734 918.0084

I think the major difference here is the dropped cohorts, though I know there are other differences between the two programs. Is there a reason to drop cohorts with one cross-sectional unit, when 2x2s are going to be aggregated later? It would be great to be able to use the full dataset without disaggregating from the FIPSxYEAR level I’m using here!

Hope some of this is useful :)

John

On Aug 6, 2021, at 8:39 AM, John Klopfer @.***> wrote:

For sure - thank you for writing software! That command line works, and the bug in Stata (same error message, still trying to install devtools and staggered) is still there. John

On Aug 6, 2021, at 3:42 AM, Jonathan Roth @.*** @.***>> wrote:

Hi, thank you for bringing this to our attention!

Can you try running

install.packages("devtools", repos = "http://cran.us.r-project.org < http://cran.us.r-project.org/>")

in the R terminal and let me know if it works? I am trying to figure out whether this is an issue on the R end or Stata end.

On Thu, Aug 5, 2021 at 8:50 AM johnklopfer @.***> wrote:

Hi, I installed your code exactly as described on this page. When I enter the following command line in Stata, I get an error, but the code runs fine in R, and the package the error message refers to is installed. In general, the Stata wrapper seems to have install errors for package dependencies in R.

I'll just run the code in R for now, but thought you might want to know about this!

. staggered, y("complaints") g("first_trained") t("period") i("uid") estimand("simple") Installing devtools and staggered packages in R Error in install.packages("devtools", repos = " http://cran.us.r-project.org http://cran.us.r-project.org/") : unable to install packages

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/jonathandroth/staggered/issues/10 < https://github.com/jonathandroth/staggered/issues/10>>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A < https://github.com/notifications/unsubscribe-auth/AE6EXFAAPZRM4MI4KSYQGE3T3KCLBANCNFSM5BTZNY3A

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub < https://github.com/jonathandroth/staggered/issues/10#issuecomment-893734099>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AQMZRGPYJCMGX6BG7C4ZYM3T3LSSDANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

—

You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-894189252, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABE734YTWH2R2A25LHEPUM3T3O7Z7ANCNFSM5BTZNY3A .

--

Pedro H. C. Sant'Anna Department of Economics Vanderbilt University 615-875-8448 (phone) @.*** https://pedrohcgs.github.io

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-894303875, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMZRGIQD7IX45K2B2Y2Z2LT3PXFVANCNFSM5BTZNY3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[pedrohcgs]

@johnklopfer can you send it to my email? [email protected] Github is blocking it.

[pedrohcgs]

@johnklopfer got it!!

[pedrohcgs]

Some thoughts on some errors:

1. It seems that FIPS is not a cross-sectional identifier, as we observe several observations of the same FIPS for the same time period. Should set i = leaid , as leaid seems to be the correct cross-sectional identifier here. I did this in R and had no problems.

2. I just double checked in R that did and staggered_cs agreed on point estimates. They do. In did, you need to make sure that you set control_group = "notyettreated" .

3. I am not getting any error with staggered or staggered_cs in R. Would you mind updating/reinstall the R package and see if you still get the error?

4. I am also getting the error message on Stata : "vcv_neyman not found'. Need to dig more about why this is happening.

[johnklopfer]

Thanks Pedro!

Quick checks and comments:

0. I think I confused things by sending two datasets: leaid is nested in FIPS, and ’shortlea.csv’ is a disaggregated version of ’short.csv’.

1a. Like you, I had no trouble running the leaid-level regression in R (or in Stata) for the dataset ’shortlea.csv’ (observations at the leaid x year level) that I shared with you.

1b. The trouble is with running the FIPS-level regression in R (or in Stata) for the dataset ’short.csv’ (observations at the FIPS x year level). It’s not clear why R is throwing the warning message:

Warning message: In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", : The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.

This creates a dilemma - can’t get the right coefficient estimate in the FIPS-level dataset because half the treatment cohorts are dropped, and can’t get the right standard error estimate in the leaid-level dataset because treatment is assigned (and errors should be clustered) at the FIPS level (which can only be accomplished with this package by working at or aggregating up to the FIPS level; per p. 21, Remark 5).

2. Similarly, like you I find that (in Stata) 'csdid, notyet' and ’staggered_cs’ produce identical results with ’shortlea.csv’. The trouble is these same two (nominally equivalent) commands produce different results on ’short.csv’, because of the treatment cohorts csdid does not drop and staggered_cs does drop (see 1b above for the list):

csdid, notyet: coefficient: 450.6242 standard error: 376.2183 staggered_cs: coefficient: 178.26605 standard error: 343.76784 csdid, notyet; dropping same treatment cohorts as staggered_cs: coefficient: 178.2661 standard error: 385.7158

3. I reinstalled ‘staggered' in R. I still get the “We drop these cohorts." error with the file ’short.csv’. Because there are multiple leaid per FIPS in each year, I still do not get the “We drop these cohorts.” error with ’shortlea.csv’.

4. Sure, please let me know what you digging turns up!

John

On Aug 17, 2021, at 5:39 AM, pedrohcgs @.***> wrote:

Some thoughts on some errors:

It seems that FIPS is not a cross-sectional identifier, as we observe several observations of the same FIPS for the same time period. Should set i = leaid , as leaid seems to be the correct cross-sectional identifier here. I did this in R and had no problems.

I just double checked in R that did and staggered_cs agreed on point estimates. They do. In did, you need to make sure that you set control_group = "notyettreated" .

I am not getting any error with staggered or staggered_cs in R. Would you mind updating/reinstall the R package and see if you still get the error?

I am also getting the error message on Stata : "vcv_neyman not found'. Need to dig more about why this is happening.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-899835713, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMZRGJEL7CITVLJK5P2S5LT5GAQTANCNFSM5BTZNY3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[pedrohcgs]

Can you send me the short.csv file?

Thanks!

On Mon, Aug 16, 2021 at 21:37 johnklopfer @.***> wrote:

Thanks Pedro!

Quick checks and comments:

0. I think I confused things by sending two datasets: leaid is nested in FIPS, and ’shortlea.csv’ is a disaggregated version of ’short.csv’.

1a. Like you, I had no trouble running the leaid-level regression in R (or in Stata) for the dataset ’shortlea.csv’ (observations at the leaid x year level) that I shared with you.

1b. The trouble is with running the FIPS-level regression in R (or in Stata) for the dataset ’short.csv’ (observations at the FIPS x year level). It’s not clear why R is throwing the warning message:

Warning message: In staggered(df = short, i = "FIPS", t = "SURVYEAR", g = "YSFR1", : The treatment cohorts g = 1990, 1992, 2000, 2001, 2003, 2004, 2006, 2007, 2008, 2010, 2011 have a single cross-sectional unit only. We drop these cohorts.

This creates a dilemma - can’t get the right coefficient estimate in the FIPS-level dataset because half the treatment cohorts are dropped, and can’t get the right standard error estimate in the leaid-level dataset because treatment is assigned (and errors should be clustered) at the FIPS level (which can only be accomplished with this package by working at or aggregating up to the FIPS level; per p. 21, Remark 5).

2. Similarly, like you I find that (in Stata) 'csdid, notyet' and ’staggered_cs’ produce identical results with ’shortlea.csv’. The trouble is these same two (nominally equivalent) commands produce different results on ’short.csv’, because of the treatment cohorts csdid does not drop and staggered_cs does drop (see 1b above for the list):

csdid, notyet: coefficient: 450.6242 standard error: 376.2183 staggered_cs: coefficient: 178.26605 standard error: 343.76784 csdid, notyet; dropping same treatment cohorts as staggered_cs: coefficient: 178.2661 standard error: 385.7158

3. I reinstalled ‘staggered' in R. I still get the “We drop these cohorts." error with the file ’short.csv’. Because there are multiple leaid per FIPS in each year, I still do not get the “We drop these cohorts.” error with ’shortlea.csv’.

4. Sure, please let me know what you digging turns up!

John

On Aug 17, 2021, at 5:39 AM, pedrohcgs @.***> wrote:

Some thoughts on some errors:

It seems that FIPS is not a cross-sectional identifier, as we observe several observations of the same FIPS for the same time period. Should set i = leaid , as leaid seems to be the correct cross-sectional identifier here. I did this in R and had no problems.

I just double checked in R that did and staggered_cs agreed on point estimates. They do. In did, you need to make sure that you set control_group = "notyettreated" .

I am not getting any error with staggered or staggered_cs in R. Would you mind updating/reinstall the R package and see if you still get the error?

I am also getting the error message on Stata : "vcv_neyman not found'. Need to dig more about why this is happening.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub < https://github.com/jonathandroth/staggered/issues/10#issuecomment-899835713>, or unsubscribe < https://github.com/notifications/unsubscribe-auth/AQMZRGJEL7CITVLJK5P2S5LT5GAQTANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android < https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-899987303, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABE73474G4XWTZULVNN5QL3T5HRRNANCNFSM5BTZNY3A .

--

Pedro H. C. Sant'Anna Department of Economics Vanderbilt University 615-875-8448 (phone) @.*** https://pedrohcgs.github.io

[pedrohcgs]

On the Stata bug: We are currently going deep into this because this is a weird bug. Depending on the event time structure, it currently may or may not work. Which doesn't happen in R!

Now, with a single treated cluster: The theory of our paper and also of Callaway and Sant'Anna (and I'd say all papers doing staggered DiD) do not cover that case. Issue is that we can't identify sigma with a single observation; not much we can do on this...

I will keep you posted on the Stata bug. Keeping this open for that.

Thanks

[johnklopfer]

Hi Pedro, not sure I understand on single treated clusters: your Stata implementation of Callaway and Sant’Anna (csdid, agg(group)) plays just fine with singleton treated clusters - drops none, gives standard errors for each! I plan to check a few of the others, but also had no issues with Borusyak, Jaravel, and Spiess (did_imputation) or with Gardner (did2s).

Help me understand the problem? The one place that looks like trouble is the degrees of freedom correction for \hat{S_g} in section 2.5.1. of your paper, which is very different from that for S_g in section 2.4. Is that correct (shouldn’t Equation 5 be estimable for a single treated cluster, for example)? Is there no alternative using a different plug-in estimator, a different approximation to the optimum, etc.?

On Aug 24, 2021, at 1:30 AM, pedrohcgs @.***> wrote:

On the Stata bug: We are currently going deep into this because this is a weird bug. Depending on the event time structure, it currently may or may not work. Which doesn't happen in R!

Now, with a single treated cluster: The theory of our paper and also of Callaway and Sant'Anna (and I'd say all papers doing staggered DiD) do not cover that case. Issue is that we can't identify sigma with a single observation; not much we can do on this...

I will keep you posted on the Stata bug. Keeping this open for that.

Thanks

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-903971118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQMZRGPK7HJOOKLZ2LPLELLT6KAVFANCNFSM5BTZNY3A. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email.

[pedrohcgs]

The theory behind CS relies on the number of units in each group growing. I believe this is the same for BJS and Gardner. The fact that csdid, did_imputation and did2s produce results do not imply that their results are necessarily reliable for this particular case with one unit per group.

As an author of CS, I can talk more about that one. For instance, with a single treated unit in a group, CS essentially only accounts for the uncertainty coming from the construction of the comparison group. That is, it treat Y(g)|G=g as fixed (zero variance).

With staggered, this is trickier bc we are adopting a design-based perspective, and we cant estimate the variance of Y(g) if group G=g has a single observation. Treating the variance as zero is not really a "solution"....

Does this help?

Thanks

[jonathandroth]

Just to follow up on what Pedro said, the challenge is if N_g = 1 (or you only have 1 cluster for g), there is no way to estimate S_g = Var[ Y(g) ]. The sample variance will be 0 regardless of what the true variance is. Pedro's csdid package essentially assumes that S_g =0 if N_g = 1. It is not accounting for the sampling variation from that cohort. I do not think this is very desirable, because it makes it seem as if the estimates are more precise than they are likely to be, because it is not accounting for the uncertainty for cohort g. I am not sure what the other packages are doing, but I suspect that they are doing something similar. Just because a package outputs a standard error does not mean that that standard error is reliable. So I think it is better to throw a warning that we can't estimate S_g rather than to assume it is zero and give a potentially unreliable standard error.

If you were willing to impose that S_g = S_{g'} for some g' with more observations, then you could still form an estimate of the covariance matrix while included cohort g. If there is demand for others for this feature, we will consider implementing it. Likewise, randomization inference will be valid under the sharp null for any test statistic (including the "studentized" one that imposes S_g =0, but there are no guarantees for validity for the weak null in the case where N_g = 1).

On Fri, Sep 10, 2021 at 3:10 PM pedrohcgs @.***> wrote:

The theory behind CS relies on the number of units in each group growing. I believe this is the same for BJS and Gardner. The fact that csdid, did_imputation and did2s produce results do not imply that their results are necessarily reliable for this particular case with one unit per group.

As an author of CS, I can talk more about that one. For instance, with a single treated unit in a group, CS essentially only accounts for the uncertainty coming from the construction of the comparison group. That is, it treat Y(g)|G=g as fixed (zero variance).

With staggered, this is trickier bc we are adopting a design-based perspective, and we cant estimate the variance of Y(g) if group G=g has a single observation. Treating the variance as zero is not really a "solution"....

Does this help?

Thanks

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jonathandroth/staggered/issues/10#issuecomment-917145723, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE6EXFA4OS5DTTSAEYXJPVDUBJJ2NANCNFSM5BTZNY3A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.