2025-11-21

Linux Security Mechanisms: GCC Compiler Protections

Linux provides various security mechanisms. ASLR is handled directly by the kernel and configured through system files. NX, Canary, PIE, and RELRO are enabled or disabled at compile time via compiler flags. When unspecified, default settings apply.

📝 This article was created with AI assistance and reviewed by a human.

CANARY

When Canary is enabled, the function prologue inserts a canary value onto the stack. Before the function returns, this canary is verified—if it was modified, a stack overflow was detected and the program halts.

Example:

#include <stdio.h>

int main() {
    char buf[10];
    scanf("%s", buf);
    return 0;
}

With Canary enabled:

$ gcc -fstack-protector canary.c -o f.out
$ python -c 'print("A"*20)' | ./f.out
*** stack smashing detected ***: <unknown> terminated
Aborted (core dumped)

With Canary disabled:

$ gcc -fno-stack-protector canary.c -o fno.out
$ python -c 'print("A"*20)' | ./fno.out
Segmentation fault (core dumped)

Disassembly comparison:

With Canary:

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000000000001145 <+0>:     push   %rbp
   0x0000000000001146 <+1>:     mov    %rsp,%rbp
   0x0000000000001149 <+4>:     sub    $0x20,%rsp
   0x000000000000114d <+8>:     mov    %fs:0x28,%rax            ;
   0x0000000000001156 <+17>:    mov    %rax,-0x8(%rbp)          ;
   0x000000000000115a <+21>:    xor    %eax,%eax                ;
   ...
   0x000000000000117d <+56>:    xor    %fs:0x28,%rdx
   0x0000000000001186 <+65>:    je     0x118d <main+72>
   0x0000000000001188 <+67>:    callq  0x1030 <__stack_chk_fail@plt>
   0x000000000000118d <+72>:    leaveq
   0x000000000000118e <+73>:    retq
End of assembler dump.

Without Canary:

(gdb) disassemble main
Dump of assembler code for function main:
   0x0000000000001135 <+0>:     push   %rbp
   0x0000000000001136 <+1>:     mov    %rsp,%rbp
   0x0000000000001139 <+4>:     sub    $0x10,%rsp
   0x000000000000113d <+8>:     lea    -0xa(%rbp),%rax
   ...
   0x000000000000115b <+38>:    retq
End of assembler dump.

FORTIFY

The -D_FORTIFY_SOURCE option, used with optimization -O, detects buffer overflows at compile time and runtime.

Example:

#include<string.h>
void main() {
    char str[3];
    strcpy(str, "abcde");
}

Without FORTIFY:

$ gcc -O2 fortify.c
$ checksec --file=a.out
RELRO           STACK CANARY      NX            PIE             FORTIFY
Partial RELRO   No canary found   NX enabled    PIE enabled     No

With FORTIFY:

$ gcc -O2 -D_FORTIFY_SOURCE=2 fortify.c
In function ‘strcpy’,
    inlined from ‘main’ at fortify.c:6:5:
warning: writing 6 bytes into a region of size 3 overflows
$ checksec --file=a.out
Partial RELRO   No canary found   NX enabled    PIE enabled     Yes

NX (No-eXecute)

NX marks data memory pages as non-executable. If an overflow leads to shellcode execution, the CPU raises an exception. The .text segment is marked executable, while .data, .bss, stack, and heap are not.

This prevents classic GOT overwrite shellcode attacks, but does not stop ret2libc-style code reuse attacks.

PIE (Position Independent Executable)

PIE works with ASLR to randomize executable load addresses. PIE is compile-time randomization (the compiler), while ASLR is load-time randomization (the OS). When PIE is enabled, the binary is compiled as a shared object file; when disabled, it’s an executable.

With ASLR off and PIE on, the entry point remains fixed. With both enabled, addresses randomize on each run.

ASLR (Address Space Layout Randomization)

• Disabled: # echo 0 > /proc/sys/kernel/randomize_va_space
• Partial (mmap, stack, vdso randomized): # echo 1 > /proc/sys/kernel/randomize_va_space
• Full (+ heap randomized): # echo 2 > /proc/sys/kernel/randomize_va_space

RELRO (ReLocation Read-Only)

RELRO makes symbol relocation tables read-only or resolves all dynamic symbols at startup to reduce GOT attack surface.

• Partial RELRO: some segments (including .dynamic) become read-only after initialization.
• Full RELRO: lazy binding is disabled, all imports resolved at startup, .got.plt is read-only.

Compiler Flags Summary

Disable all:

gcc hello.c -o hello -fno-stack-protector -z execstack -no-pie -z norelro

Enable all:

gcc hello.c -o hello -fstack-protector-all -z noexecstack -pie -z now

FORTIFY:

• -D_FORTIFY_SOURCE=1: compile-time checks only
• -D_FORTIFY_SOURCE=2: compile-time and runtime checks

Detection Tools

checksec:

$ checksec --file=/bin/ls
RELRO STACK CANARY NX PIE FORTIFY
Partial RELRO Canary found NX enabled No PIE Yes

peda checksec (gdb plugin):

gdb-peda$ checksec
CANARY : ENABLED
FORTIFY : ENABLED
NX : ENABLED
PIE : DISABLED
RELRO : Partial

ASLR and PIE Interaction

ASLR randomizes memory layout at load time. Attackers often bypass it via information leaks—since relative offsets within a module are fixed, leaking any code or data pointer allows calculating all addresses in that module.

To analyze a PIE binary, disable ASLR to neutralize both PIE and ASLR at once.