01
x86-64 entry
User-space: the SYSCALL instruction
When a user-space program calls a libc wrapper like read(), the C library places the syscall number in RAX and the arguments in RDI, RSI, RDX, R10, R8, R9 — then executes the SYSCALL instruction. SYSCALL is a fast system-call instruction introduced in AMD64. It does not push a return address onto a stack or switch stacks itself. Instead it atomically: saves RIP into RCX, saves RFLAGS into R11, loads the new CS/SS selectors from IA32_STAR, and jumps to the address stored in MSR_LSTAR.
// glibc sysdeps/unix/sysv/linux/x86_64/syscall.S
mov %rdi, %rax // syscall number (e.g. 0 = read)
mov %rsi, %rdi // arg1
mov %rdx, %rsi // arg2
mov %rcx, %rdx // arg3
mov %r8, %r10 // arg4 (note: rcx clobbered by SYSCALL)
mov %r9, %r8 // arg5
syscall // → MSR_LSTAR
NOTERCX is clobbered by SYSCALL (it holds the saved RIP), so arg4 must be passed in R10 instead of RCX — unlike the normal System V AMD64 ABI.
02
x86-64 entry
entry_SYSCALL_64: the kernel entry point
MSR_LSTAR points to entry_SYSCALL_64 in arch/x86/entry/entry_64.S. This is pure assembly — no C yet. The first thing it does is SWAPGS, which exchanges the user GS base with the kernel GS base stored in MSR_KERNEL_GS_BASE. This gives access to the per-CPU area, from which the kernel stack pointer is loaded. The user RSP is saved into the per-CPU scratch space, and the kernel RSP is installed.
// arch/x86/entry/entry_64.S (Linux 7.2, simplified)
SYM_CODE_START(entry_SYSCALL_64)
swapgs // swap user/kernel GS base
movq %rsp, PER_CPU_VAR(cpu_tss_rw + TSS_sp2) // save user RSP
movq PER_CPU_VAR(cpu_current_top_of_stack), %rsp // load kernel stack
pushq $__USER_DS // SS
pushq PER_CPU_VAR(cpu_tss_rw + TSS_sp2) // user RSP
pushq %r11 // RFLAGS
pushq $__USER_CS // CS
pushq %rcx // RIP (saved by SYSCALL)
PUSH_AND_CLEAR_REGS // save all GPRs → struct pt_regs
...
NOTEPUSH_AND_CLEAR_REGS is a macro that pushes all general-purpose registers and clears them to prevent speculative-execution leaks (Spectre mitigations).
03
x86-64 entry
Entering C: do_syscall_64()
After the register save, entry_SYSCALL_64 calls do_syscall_64() — the first C function in the path. It receives a pointer to the pt_regs struct on the kernel stack and the syscall number (from RAX). Before dispatching, it runs syscall_enter_from_user_mode() which handles ptrace stops, audit, and seccomp. Only if all hooks pass does it proceed to the dispatch table.
// arch/x86/entry/common.c (Linux 7.2)
__visible noinstr void do_syscall_64(struct pt_regs *regs, int nr)
{
add_random_kstack_offset();
nr = syscall_enter_from_user_mode(regs, nr); // ptrace, audit, seccomp
instrumentation_begin();
if (!do_syscall_x64(regs, nr) && !do_syscall_x32(regs, nr) && nr != -1) {
/* unknown syscall */
regs->ax = __x64_sys_ni_syscall(regs);
}
instrumentation_end();
syscall_exit_to_user_mode(regs); // audit, ptrace, signals
}
NOTEadd_random_kstack_offset() adds a random offset to the kernel stack pointer on each syscall entry — a stack-layout randomisation mitigation against stack-based info leaks.
04
security hooks
seccomp: BPF filter interception
syscall_enter_from_user_mode() calls __secure_computing() if the thread has a seccomp filter. The BPF program receives a seccomp_data struct containing the syscall number, architecture, instruction pointer, and the first six arguments. It returns one of: SECCOMP_RET_ALLOW (continue), SECCOMP_RET_ERRNO (return -errno), SECCOMP_RET_KILL_THREAD, SECCOMP_RET_KILL_PROCESS, SECCOMP_RET_TRAP (send SIGSYS), or SECCOMP_RET_USER_NOTIF (notify a supervisor process via a file descriptor).
// kernel/seccomp.c (Linux 7.2, simplified)
static int __seccomp_filter(int this_syscall, const struct seccomp_data *sd,
const bool recheck_after_trace)
{
u32 filter_ret, action;
...
filter_ret = seccomp_run_filters(sd, &match); // run BPF program
action = filter_ret & SECCOMP_RET_ACTION_FULL;
switch (action) {
case SECCOMP_RET_ALLOW:
return 0; // proceed to dispatch
case SECCOMP_RET_ERRNO:
syscall_set_return_value(current, task_pt_regs(current),
-filter_ret & SECCOMP_RET_DATA, 0);
return -1; // skip dispatch, go to exit path
case SECCOMP_RET_KILL_THREAD:
do_exit(SIGSYS);
...
}
}
NOTESECCOMP_RET_USER_NOTIF (added in Linux 5.0) allows a supervisor process to intercept and respond to syscalls — used by container runtimes like gVisor and systemd.
05
dispatch
sys_call_table[]: dispatch to the handler
do_syscall_x64() indexes the sys_call_table array with the syscall number. Each entry is a function pointer of type sys_call_ptr_t. The table is generated at build time from syscall definition files. For syscall 0 (read), the entry is __x64_sys_read, which is a thin wrapper generated by the SYSCALL_DEFINE macro that extracts arguments from pt_regs and calls the generic ksys_read().
// arch/x86/entry/common.c
static __always_inline bool do_syscall_x64(struct pt_regs *regs, int nr)
{
unsigned int unr = nr;
if (likely(unr < NR_syscalls)) {
unr = array_index_nospec(unr, NR_syscalls); // Spectre v1 mitigation
regs->ax = sys_call_table[unr](regs); // call the handler
return true;
}
return false;
}
// The table itself (arch/x86/entry/syscall_64.c):
asmlinkage const sys_call_ptr_t sys_call_table[] = {
[0] = __x64_sys_read,
[1] = __x64_sys_write,
[2] = __x64_sys_open,
// ... 400+ entries
};
NOTEarray_index_nospec() is a Spectre v1 mitigation — it masks the index to prevent speculative out-of-bounds reads through the table.
06
handler
The syscall handler: ksys_read()
The SYSCALL_DEFINE macro generates a wrapper that unpacks arguments from pt_regs and calls the real implementation. For read(), that is ksys_read(), which looks up the file descriptor in the current process's file descriptor table, checks permissions, and delegates to the VFS layer via vfs_read(). The return value (bytes read, or a negative errno) is placed in regs->ax.
// fs/read_write.c (Linux 7.2)
SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
{
return ksys_read(fd, buf, count);
}
ssize_t ksys_read(unsigned int fd, char __user *buf, size_t count)
{
struct fd f = fdget_pos(fd); // look up fd → struct file
ssize_t ret = -EBADF;
if (f.file) {
loff_t pos, *ppos = file_ppos(f.file);
if (ppos) {
pos = *ppos;
ppos = &pos;
}
ret = vfs_read(f.file, buf, count, ppos); // → file_operations.read
if (ret >= 0 && ppos)
f.file->f_pos = pos;
fdput_pos(f);
}
return ret;
}
NOTEfdget_pos() is a lightweight file-descriptor lookup that avoids taking the full file-table lock for the common single-threaded case.
07
x86-64 exit
SYSRET: returning to user-space
After the handler returns, syscall_exit_to_user_mode() runs the exit-side hooks: audit, ptrace, signal delivery, and TIF_NOTIFY_RESUME work. Then entry_SYSCALL_64 pops pt_regs, executes SWAPGS to restore the user GS base, and executes SYSRET. SYSRET loads RIP from RCX (the saved user RIP), restores RFLAGS from R11, switches back to user CS/SS, and resumes user-space execution. The syscall return value is in RAX.
// arch/x86/entry/entry_64.S (exit path, simplified)
POP_REGS pop_rdi=0 // restore GPRs from pt_regs
movq %rsp, %rdi
movq PER_CPU_VAR(cpu_tss_rw + TSS_sp0), %rsp
UNWIND_HINT_EMPTY
pushq RSP-8(%rdi) // user RSP
pushq (%rdi) // user RDI
swapgs // restore user GS base
popq %rdi
popq %rsp
sysretq // RIP←RCX, RFLAGS←R11, CPL←3
NOTESYSRET has a known erratum on some AMD CPUs when RCX is not canonical — the kernel checks for this and falls back to IRET for non-canonical return addresses.