ShadowGuard: Cryptographic Shadow Stack Protection with XOR Obfuscation and HMAC Integrity

Ilahi, Sirine, Omotosho, Adebayo ORCID: https://orcid.org/0000-0002-1642-7610 and Hammer, Christian (2025) ShadowGuard: Cryptographic Shadow Stack Protection with XOR Obfuscation and HMAC Integrity. In: The 36th IEEE International Symposium on Software Reliability Engineering, 21st to 24th October 2025, São Paulo, Brazil. (Unpublished)

Text (Peer-reviewed version) 15360 Ilahi, S et al . (2025) ShadowGuard - Cryptographic Shadow Stack Protection with XOR Obfuscation and HMAC Integrity.pdf - Accepted Version Restricted to Repository staff only until 1 July 2026. Available under License Creative Commons Attribution 4.0. Download (638kB)

Abstract

Return-Oriented Programming (ROP) attacks, a persistent security threat for over a decade, pose significant risks to computing devices by exploiting vulnerabilities to hijack control flow and execute arbitrary code. While memory isolation and shadow stacks raise the bar, advanced memory disclosure attacks can still bypass these defenses. As a more resilient software-based defense against such advanced threats, we introduce ShadowGuard, a novel approach that leverages Low-Level Virtual Machine (LLVM) passes, programmatic transformations during compilation, to enhance return address protection and prevent ROP attacks on (embedded) systems that to not feature hardware support for control flow protection. ShadowGuard employs dual XOR-based obfuscation and a HMAC-SHA256 keyed hash algorithm to mask return addresses and ensure their integrity. This combination allows for the detection of tampering attempts. Additionally, a separate, secure shadow stack stores obfuscated addresses and their authentication hashed keys, preventing unauthorized access or modification by attackers. Through comprehensive evaluation using real-life applications and the Coreutils-8.32 benchmark, we demonstrate that our approach effectively detects and mitigates ROP attacks while maintaining practicality. The runtime overhead is approximately 31%, and the binary size increase 2%, on average. This solution offers a scalable and robust defense mechanism for securing return addresses in modern real-world applications.

Item Type:	Conference or Workshop Item (Paper)
Uncontrolled Keywords:	Return oriented programming; Shadow stack; LLVM; Obfuscation, Security
Subjects:	Q Science > QA Mathematics Q Science > QA Mathematics > QA75 Electronic computers. Computer science Q Science > QA Mathematics > QA76 Computer software
Divisions:	Schools and Research Institutes > School of Business, Computing and Social Sciences
Depositing User:	Bayo Omotosho
Date Deposited:	22 Oct 2025 13:15
Last Modified:	22 Oct 2025 13:30
URI:	https://eprints.glos.ac.uk/id/eprint/15360

University Staff: Request a correction | Repository Editors: Update this record