Florian Tramèr | Publications

	Sort by: Date \| Venue
Manuscripts	Black-box Optimization of LLM Outputs by Asking for Directions [arXiv] Jie Zhang, Meng Ding, Yang Liu, Jue Hong and Florian Tramèr The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections [arXiv] Milad Nasr, Nicholas Carlini, Chawin Sitawarin, Sander V. Schulhoff, Jamie Hayes, Michael Ilie, Juliette Pluto, Shuang Song, Harsh Chaudhari, Ilia Shumailov, Abhradeep Thakurta, Kai Yuanqing Xiao and others Pitfalls in Evaluating Language Model Forecasters [arXiv] Daniel Paleka, Shashwat Goel, Jonas Geiping and Florian Tramèr LLMs unlock new paths to monetizing exploits [arXiv] Nicholas Carlini, Milad Nasr, Edoardo Debenedetti, Barry Wang, Christopher A Choquette-Choo, Daphne Ippolito, Florian Tramèr and Matthew Jagielski Gradient-based Jailbreak Images for Multimodal Fusion Models [arXiv] Javier Rando, Hannah Korevaar, Erik Brinkman, Ivan Evtimov and Florian Tramèr
2026	Defeating Prompt Injections by Design [arXiv] Edoardo Debenedetti, Ilia Shumailov, Tianqi Fan, Jamie Hayes, Nicholas Carlini, Daniel Fabian, Christoph Kern, Chongyang Shi, Andreas Terzis and Florian Tramèr IEEE SaTML 2026
2025	RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics [arXiv] Jie Zhang, Cezara Petrui, Kristina Nikolić and Florian Tramèr NeurIPS 2025 Exploring and mitigating adversarial manipulation of voting-based leaderboards [arXiv] Yangsibo Huang, Milad Nasr, Anastasios Angelopoulos, Nicholas Carlini, Wei-Lin Chiang, Christopher A. Choquette-Choo, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Ken Ziyu Liu, Ion Stoica, Florian Tramèr and others ICML 2025 (Oral Presentation) The Jailbreak Tax: How Useful are Your Jailbreak Outputs? [arXiv] Kristina Nikolić, Luze Sun, Jie Zhang and Florian Tramèr ICML 2025 (Spotlight) AutoAdvExBench: Benchmarking autonomous exploitation of adversarial example defenses [arXiv] Nicholas Carlini, Javier Rando, Edoardo Debenedetti, Milad Nasr and Florian Tramèr ICML 2025 (Oral Presentation) Design Patterns for Securing LLM Agents against Prompt Injections [arXiv] Luca Beurer-Kellner, Beat Buesser Ana-Maria Creţu, Edoardo Debenedetti, Daniel Dobos, Daniel Fabian, Marc Fischer, David Froelicher, Kathrin Grosse, Daniel Naeff, Ezinwanne Ozoani, Andrew Paverd, Florian Tramèr and others SoK: Watermarking for AI-Generated Content [arXiv] Xuandong Zhao, Sam Gunn, Miranda Christ, Jaiden Fairoze, Andres Fabrega, Nicholas Carlini, Sanjam Garg, Sanghyun Hong, Milad Nasr, Florian Tramèr and others IEEE S&P 2025 Membership Inference Attacks on Sequence Models [arXiv] Lorenzo Rossi, Michael Aerni, Jie Zhang and Florian Tramèr DLS Workshop 2025 (Best Paper Award) Adversarial ML Problems Are Getting Harder to Solve and to Evaluate [arXiv] Javier Rando, Jie Zhang, Nicholas Carlini and Florian Tramèr DLS Workshop 2025 Blind Baselines Beat Membership Inference Attacks for Foundation Models [arXiv] Debeshee Das, Jie Zhang and Florian Tramèr DLS Workshop 2025 Consistency Checks for Language Model Forecasters [arXiv] Daniel Paleka, Abhimanyu Pallavi Sudhir, Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan Wang and Florian Tramèr ICLR 2025 (Oral Presentation) Measuring Non-Adversarial Reproduction of Training Data in Large Language Models [arXiv] Michael Aerni, Javier Rando, Edoardo Debenedetti, Nicholas Carlini, Daphne Ippolito and Florian Tramèr ICLR 2025 Persistent Pre-Training Poisoning of LLMs [arXiv] Yiming Zhang, Javier Rando, Ivan Evtimov, Jianfeng Chi, Eric Michael Smith, Nicholas Carlini, Florian Tramèr and Daphne Ippolito ICLR 2025 Adversarial Search Engine Optimization for Large Language Models [arXiv] Fredrik Nestaas, Edoardo Debenedetti and Florian Tramèr ICLR 2025 Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI [arXiv] Robert Hönig, Javier Rando, Nicholas Carlini and Florian Tramèr ICLR 2025 (Spotlight Presentation) Scalable Extraction of Training Data from (Production) Language Models [arXiv] Milad Nasr, Javier Rando, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Florian Tramèr and Katherine Lee ICLR 2025 Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data [arXiv] Jie Zhang, Debeshee Das, Gautam Kamath and Florian Tramèr IEEE SaTML 2025 An adversarial perspective on machine unlearning for AI safety [arXiv] Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr and Javier Rando TMLR 2025
2024	Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition [arXiv] Edoardo Debenedetti, Javier Rando, Daniel Paleka, Silaghi Fineas Florin, Dragos Albastroiu, Niv Cohen, Yuval Lemberg, Reshmi Ghosh, Rui Wen, Ahmed Salem, Giovanni Cherubin, Santiago Zanella-Beguelin and others NeurIPS 2024 (Spotlight Presentation)* AgentDojo: A Dynamic Environment to Evaluate Attacks and Defenses for LLM Agents [arXiv] Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer and Florian Tramèr NeurIPS 2024 (Winner of CAIS SafeBench competition) JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models [arXiv] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramèr and others NeurIPS 2024 Query-Based Adversarial Prompt Generation [arXiv] Jonathan Hayase, Ema Borevkovic, Nicholas Carlini, Florian Tramèr and Milad Nasr NeurIPS 2024 Evaluations of Machine Learning Privacy Defenses are Misleading [arXiv] Michael Aerni, Jie Zhang and Florian Tramèr CCS 2024 Privacy Side Channels in Machine Learning Systems [arXiv] Edoardo Debenedetti, Giorgio Severi, Nicholas Carlini, Christopher A Choquette-Choo, Matthew Jagielski, Milad Nasr, Eric Wallace and Florian Tramèr USENIX Security 2024 Extracting Training Data From Document-Based VQA Models [arXiv] Francesco Pinto, Nathalie Rauschmayr, Florian Tramèr, Philip Torr and Federico Tombari ICML 2024 Privacy Backdoors: Stealing Data with Corrupted Pretrained Models [arXiv] Shanglun Feng and Florian Tramèr ICML 2024 Stealing Part of a Production Language Model [arXiv] Nicholas Carlini, Daniel Paleka, Krishnamurthy Dj Dvijotham, Thomas Steinke, Jonathan Hayase, A Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Eric Wallace, David Rolnick and others ICML 2024 (Best Paper Award) Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining [arXiv] Florian Tramèr, Gautam Kamath and Nicholas Carlini (βα-order) ICML 2024 (Best Paper Award) Universal Jailbreak Backdoors from Poisoned Human Feedback [arXiv] Javier Rando and Florian Tramèr ICLR 2024 Poisoning Web-Scale Training Datasets is Practical [arXiv] Nicholas Carlini, Matthew Jagielski, Christopher A. Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas and Florian Tramèr IEEE S&P 2024 Evading Black-box Classifiers Without Breaking Eggs [arXiv] Edoardo Debenedetti, Nicholas Carlini and Florian Tramèr IEEE SaTML 2024 (Best paper award runner-up) Evaluating Superhuman Models with Consistency Checks [arXiv] Lukas Fluri, Daniel Paleka and Florian Tramèr IEEE SaTML 2024
2023	Students Parrot Their Teachers: Membership Inference on Model Distillation [arXiv] Matthew Jagielski, Milad Nasr, Christopher Choquette-Choo, Katherine Lee, Nicholas Carlini and Florian Tramèr NeurIPS 2023 (Oral Presentation) Are aligned neural networks adversarially aligned? [arXiv] Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo, Matthew Jagielski, Irena Gao, Anas Awadalla, Pang Wei Koh, Daphne Ippolito, Katherine Lee, Florian Tramèr and Ludwig Schmidt NeurIPS 2023 Counterfactual Memorization in Neural Language Models [arXiv] Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tramèr and Nicholas Carlini NeurIPS 2023 (Spotlight Presentation) Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy [arXiv] Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo and Nicholas Carlini INLG 2023 (Best Paper Finalist) Tight Auditing of Differentially Private Machine Learning [arXiv] Milad Nasr, Jamie Hayes, Thomas Steinke, Borja Balle, Florian Tramèr, Matthew Jagielski, Nicholas Carlini and Andreas Terzis USENIX Security 2023 (Distinguished paper award) Extracting Training Data from Diffusion Models [arXiv] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito and Eric Wallace USENIX Security 2023 Preprocessors Matter! Realistic Decision-Based Attacks on Machine Learning Systems [arXiv] Chawin Sitawarin, Florian Tramèr and Nicholas Carlini ICML 2023 Backdoor Attacks for In-Context Learning with Language Models [OpenReview] Nikhil Kandpal, Matthew Jagielski, Florian Tramèr and Nicholas Carlini ICML AdvML Workshop 2023 Measuring Forgetting of Memorized Training Examples [arXiv] Matthew Jagielski, Om Thakkar, Florian Tramèr, Daphne Ippolito, Katherine Lee, Nicholas Carlini, Eric Wallace, Shuang Song, Abhradeep Thakurta, Nicolas Papernot and Chiyuan Zhang ICLR 2023 (Certified!!) Adversarial Robustness for Free! [arXiv] Nicholas Carlini, Florian Tramèr*, Rice Leslie, Mingjie Sun, Krishnamurthy Dvijotham and J Zico Kolter ICLR 2023* Quantifying Memorization Across Neural Language Models [arXiv] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr and Chiyuan Zhang (αβ-order) ICLR 2023 (Spotlight Presentation) SNAP: Efficient Extraction of Private Properties with Poisoning [arXiv] Harsh Chaudhari, John Abascal, Alina Oprea, Matthew Jagielski, Florian Tramèr and Jonathan Ullman IEEE S&P 2023 Randomness in ML Defenses Helps Persistent Attackers and Hinders Evaluators [arXiv] Keane Lucas, Matthew Jagielski, Florian Tramèr, Lujo Bauer and Nicholas Carlini
2022	Red-Teaming the Stable Diffusion Safety Filter [arXiv] Javier Rando, Daniel Paleka, David Lindner, Lennart Heim and Florian Tramèr NeurIPS ML Safety Workshop 2022 (Best Paper Award) Increasing Confidence in Adversarial Robustness Evaluations [arXiv] Roland S. Zimmermann, Wieland Brendel, Florian Tramèr and Nicholas Carlini NeurIPS 2022 The Privacy Onion Effect: Memorization is Relative [arXiv] Nicholas Carlini, Matthew Jagielski, Nicolas Papernot, Andreas Terzis, Florian Tramèr and Chiyuan Zhang (αβ-order) NeurIPS 2022 Truth Serum: Poisoning Machine Learning Models to Reveal Their Secrets [arXiv] Florian Tramèr, Reza Shokri, Ayrton San Joaquin, Hoang Le, Matthew Jagielski, Sanghyun Hong and Nicholas Carlini (βα-order) CCS 2022 Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them [arXiv] Florian Tramèr ICML 2022 (Long Presentation) What Does it Mean for a Language Model to Preserve Privacy? [arXiv] Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri and Florian Tramèr (αβ-order) FAccT 2022 Large Language Models Can Be Strong Differentially Private Learners [arXiv] Xuechen Li, Florian Tramèr, Percy Liang and Tatsunori Hashimoto ICLR 2022 (Oral Presentation) Data Poisoning Won’t Save You From Facial Recognition [arXiv] Evani Radiya-Dixit, Sanghyun Hong, Nicholas Carlini and Florian Tramèr ICLR 2022 Membership Inference Attacks From First Principles [arXiv] Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis and Florian Tramèr (αβ-order) IEEE S&P 2022 Debugging Differential Privacy: A Case Study for Privacy Auditing [arXiv] Florian Tramèr, Andreas Terzis, Thomas Steinke, Shuang Song, Matthew Jagielski and Nicholas Carlini (βα-order)
2021	Antipodes of Label Differential Privacy: PATE and ALIBI [arXiv] Mani Malek, Ilya Mironov, Karthik Prasad, Igor Shilov and Florian Tramèr (αβ-order) NeurIPS 2021 Measuring and Enhancing the Security of Machine Learning [PDF] Florian Tramèr PhD Thesis 2021 Extracting Training Data from Large Language Models [arXiv] Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea and Colin Raffel USENIX Security 2021 (Caspar Bowden Award for Outstanding Research in Privacy Enhancing Technologies runner-up) NeuraCrypt is not private [arXiv] Nicholas Carlini, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody and Florian Tramèr (αβ-order) PPML Workshop 2021 On the Opportunities and Risks of Foundation Models [arXiv] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch and others Label-Only Membership Inference Attacks [arXiv] Christopher A. Choquette Choo, Florian Tramèr, Nicholas Carlini and Nicolas Papernot ICML 2021 Differentially Private Learning Needs Better Features (or Much More Data) [arXiv] Florian Tramèr and Dan Boneh ICLR 2021 (Spotlight Presentation) Is Private Learning Possible with Instance Encoding? [arXiv] Nicholas Carlini, Samuel Deng, Sanjam Garg, Somesh Jha, Saeed Mahloujifar, Mohammad Mahmoody, Shuang Song, Abhradeep Thakurta and Florian Tramèr (αβ-order) IEEE S&P 2021 SquirRL: Automating Attack Discovery on Blockchain Incentive Mechanisms with Deep Reinforcement Learning [arXiv] Charlie Hou, Mingxun Zhou, Yan Ji, Phil Daian, Florian Tramèr, Giulia Fanti and Ari Juels NDSS 2021 Advances and Open Problems in Federated Learning [arXiv] Peter Kairouz, H. Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, Rafael G. L. D’Oliveira, Hubert Eichner and others Foundations and Trends in ML 2021
2020	On Adaptive Attacks to Adversarial Example Defenses [arXiv] Florian Tramèr, Nicholas Carlini, Wieland Brendel* and Aleksander Madry NeurIPS 2020 Remote Side-Channel Attacks on Anonymous Transactions [eprint] Florian Tramèr, Dan Boneh and Kenneth G. Paterson USENIX Security 2020 Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbations [arXiv] Florian Tramèr, Jens Behrmann, Nicholas Carlini, Nicolas Papernot and Jörn-Henrik Jacobsen ICML 2020 SentiNet: Detecting Localized Universal Attacks Against Deep Learning Systems [arXiv] Edward Chou, Florian Tramèr and Giancarlo Pellegrino DLS Workshop 2020
2019	Adversarial Training and Robustness for Multiple Perturbations [arXiv] Florian Tramèr and Dan Boneh NeurIPS 2019 (Spotlight Presentation) AdVersarial: Perceptual Ad Blocking meets Adversarial Machine Learning [arXiv] Florian Tramèr, Pascal Dupré, Gili Rusak, Giancarlo Pellegrino and Dan Boneh CCS 2019 The Hydra Framework for Principled, Automated Bug Bounties [PDF] Lorenz Breidenbach, Phil Daian, Florian Tramèr* and Ari Juels IEEE Security & Privacy 2019 Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware [arXiv] Florian Tramèr and Dan Boneh ICLR 2019 (Oral Presentation) Exploiting Excessive Invariance caused by Norm-Bounded Adversarial Robustness [arXiv] Jörn-Henrik Jacobsen, Jens Behrmann, Nicholas Carlini, Florian Tramèr and Nicolas Papernot ICLR SafeML Workshop 2019
2018	Enter the Hydra: Towards Principled Bug Bounties and Exploit-Resistant Smart Contracts [eprint] Lorenz Breidenbach, Phil Daian, Florian Tramèr* and Ari Juels USENIX Security 2018 (Invited to appear in IEEE Security and Privacy Magazine) Physical Adversarial Examples for Object Detectors [arXiv] Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Florian Tramèr*, Atul Prakash, Tadayoshi Kohno and Dawn Song WOOT Workshop 2018* Ensemble Adversarial Training: Attacks and Defenses [arXiv] Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh and Patrick McDaniel ICLR 2018
2017	PrivateRide: A Privacy-Preserving and Secure Ride-Hailing Service [PDF] Anh Pham, Italo Dacosta, Bastien Jacot-Guillarmod, Kévin Huguenin, Taha Hajar, Florian Tramèr and Jean-Pierre Hubaux PETS 2017 Formal Abstractions for Attested Execution Secure Processors [eprint] Rafael Pass, Elaine Shi and Florian Tramèr (αβ-order) EUROCRYPT 2017 Sealed-Glass Proofs: Using Transparent Enclaves to Prove and Sell Knowledge [eprint] Florian Tramèr, Fan Zhang, Huang Lin, Jean-Pierre Hubaux, Ari Juels and Elaine Shi EuroS&P 2017 FairTest: Discovering Unwarranted Associations in Data-Driven Applications [arXiv] Florian Tramèr, Vaggelis Atlidakis, Roxana Geambasu, Daniel Hsu, Jean-Pierre Hubaux, Mathias Humbert, Ari Juels and Huang Lin EuroS&P 2017 The Space of Transferable Adversarial Examples [arXiv] Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh and Patrick McDaniel Addressing Beacon Re-Identification Attacks: Quantification and Mitigation of Privacy Risks [PDF] Jean Louis Raisaro, Florian Tramèr*, Zhanglong Ji, Diyue Bu, Yongan Zhao, Knox Carey and others JAMIA 2017*
2016	Stealing Machine Learning Models via Prediction APIs [arXiv] Florian Tramèr, Fan Zhang, Ari Juels, Michael Reiter and Thomas Ristenpart USENIX Security 2016 On Solving LPN using BKW and Variants [eprint] Sonia Bogos, Florian Tramèr and Serge Vaudenay (αβ-order) Cryptography and Communications 2016
2015	Differential Privacy with Bounded Priors: Reconciling Utility and Privacy in Genome-Wide Association Studies [PDF] Florian Tramèr, Zhicong Huang, Jean-Pierre Hubaux and Erman Ayday CCS 2015 Algorithmic Fairness Revisited [PDF] Florian Tramèr Master Thesis 2015 Better Algorithms for LWE and LWR [eprint] Alexandre Duc, Florian Tramèr and Serge Vaudenay (αβ-order) EUROCRYPT 2015