x86: keeping the CPUs' TLBs up to date when a mapping changes. More...

#include <core/system.h>
#include <core/arch.h>
#include <cpu/cpu.h>
#include <mm/mm.h>
#include <mm/pmap.h>
#include <mm/vm.h>
#include <platform/misc.h>
#include <sys/cpu.h>
#include <sys/lock.h>

Data Structures
struct	[struct].request

Macros
#define	FLUSH_PAGES_MAX 32
#define	ICR_ASSERT 0x4000

Functions
static void	flush (vaddr_t sva, vaddr_t eva)
	Flushes [sva, eva) from the running CPU's TLB.
static void	request_do (struct pmap *pmap, vaddr_t sva, vaddr_t eva, int leave)
	Does what a request asks of the running CPU: flush the range, if the CPU may have it, or switch away from page tables that are going away.
void	x86_tlb_ipi (void)
	Does the pending request if it is for the running CPU.
void	arch_lock_spin (void)
	Runs in the kernel lock's wait loop, on each turn.
void	arch_lock_taken (void)
	Runs once this CPU has taken the kernel lock.
void	tlb_shootdown (struct pmap *pmap, vaddr_t sva, vaddr_t eva, int leave)
void	pmap_loaded (struct pmap from, struct pmap to)

Variables
static uint64_t	kernel_gen
static struct	request

Detailed Description

x86: keeping the CPUs' TLBs up to date when a mapping changes.

A CPU caches the mappings it uses in its TLB. When the page tables change (a page is unmapped, made read-only, or replaced), the CPU that changed them flushes its own TLB, asks the other CPUs that may have the old mapping to flush theirs with an IPI, and waits until they have. The page can't be reused while another CPU may still write to it.

For a process's page tables, the CPUs that may have the old mapping are those the tables are loaded on (pmap->cpus, which pmap_switch() keeps up to date). Only the CPU with the kernel lock changes page tables, so there is one request at a time. The other CPUs do it without the lock (someone else holds it): from the IPI's handler, or, if they are waiting for the lock with interrupts off, from that wait (arch_lock_spin()).

The kernel's mappings, shared by all page tables, are in every CPU's TLB, but the other CPUs aren't asked to flush them. Outside the lock, a CPU only touches memory that was mapped before it last took the lock, and that stays mapped while it uses it (its idle stack, its thread's kernel stack, its struct cpu, the lock, the page tables it has loaded). Instead, a CPU flushes its TLB when it takes the lock, if the kernel's mappings changed since it last did (kernel_gen).

Reloading CR3 flushes every mapping but the global ones. The only global pages are x86_64's direct map, which never changes.

Definition in file tlb.c.