Privacy Side Channels in Machine Learning Systems

Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace and Florian Tramèr

USENIX Security Symposium 2024


Most current approaches for protecting privacy in machine learning (ML) assume that models exist in a vacuum, when in reality, ML models are part of larger systems that include components for training data filtering, output monitoring, and more. In this work, we introduce privacy side channels: attacks that exploit these system-level components to extract private information at far higher rates than is otherwise possible for standalone models. We propose four categories of side channels that span the entire ML lifecycle (training data filtering, input preprocessing, output post-processing, and query filtering) and allow for either enhanced membership inference attacks or even novel threats such as extracting users’ test queries. For example, we show that deduplicating training data before applying differentially-private training creates a side-channel that completely invalidates any provable privacy guarantees. Moreover, we show that systems which block language models from regenerating training data can be exploited to allow exact reconstruction of private keys contained in the training set – even if the model did not memorize these keys. Taken together, our results demonstrate the need for a holistic, end-to-end privacy analysis of machine learning.

  author   =   {Debenedetti, Edoardo and Severi, Giorgio and Carlini, Nicholas and Choquette-Choo, Christopher A and Jagielski, Matthew and Nasr, Milad and Wallace, Eric and Tram{\`e}r, Florian},
  title   =   {Privacy Side Channels in Machine Learning Systems},
  booktitle   =   {USENIX Security Symposium},
  year   =   {2024},
  url   =   {}