Privacy-preserving impact evaluation using Difference-in-Differences

Difference-in-Differences (DID) is a widely used tool for causal impact evaluation but is constrained by data privacy regulations when applied to sensitive personal information, such as individual-level performance records or healthcare data, that must not be shared with data analysts. Obtaining consent can reduce sample sizes or exclude treated/untreated groups, diminishing statistical power or making estimation impossible. Federated Learning, which shares aggregated statistics to ensure privacy, can address these concerns, but advanced federated DID software packages remain scarce. We derived and developed a federated version of the Callaway and SantAnna DID, implemented within the DataSHIELD platform. Our package adheres to DataSHIELDs security measures and adds extra protections, enhancing data privacy and confidentiality. It reproduces point estimates, asymptotic standard errors, and bootstrapped standard errors equivalent to the non-federated implementation. We demonstrate this functionality on simulated data and real-world data from a malaria intervention in Mozambique. By leveraging federated estimates, we increase effective sample sizes leading to reduced estimation uncertainty, and enable estimation when single data owners cannot share the data but only have access to the treated or untreated group.