From d230991493b521eeff39f32434fddcbcdb109eb0 Mon Sep 17 00:00:00 2001 From: Johannes Weiner Date: Mon, 29 Jun 2026 12:33:37 -0400 Subject: mm: mempolicy: fix automatic numa balancing for shmem Neha reports that mapped shmem aren't considered for NUMA balancing, noting convergence problems and bandwidth bottlenecking for cachelib based workloads on tiered memory systems. Looking at the code and going through the git history, this doesn't actually seem intentional: Commit fc3147245d19 ("mm: numa: Limit NUMA scanning to migrate-on-fault VMAs") added a vma_policy_mof() gate to task_numa_work() so VMAs whose policy lacks MPOL_F_MOF are skipped from NUMA balancing scans. The motivation was a real usecase: Oracle was pinning shared segments with mbind(MPOL_BIND) so trapping faults was both expensive and pointless. The handling of NULL from vm_ops->get_policy, however, treated "user explicitly opted out" the same as "user never specified anything." For VMAs whose shared policy is absent - the common case for shmem - the scan was disabled too. This issue is old. It probably hurts less in conventional NUMA. But it's very noticeable on tiered systems, where entire tmpfs workingsets can get stuck on lower-bandwidth memory. Fix this by having vma_policy_mof() use __get_vma_policy() directly, and thereby handle the fallback to task policy (-> preferred_node_policy() has MPOL_F_MOF per default). Every other consumer of vm_ops->get_policy already handles it this way, the scan-eligibility check was the outlier. This preserves Mel's intended fix: don't scan stuff the user explicitly pinned. But allow default policy vmas to participate in balancing. Link: https://lore.kernel.org/20260629163337.1264881-1-hannes@cmpxchg.org Fixes: fc3147245d19 ("mm: numa: Limit NUMA scanning to migrate-on-fault VMAs") Signed-off-by: Johannes Weiner Reported-by: Neha Gholkar Tested-by: Neha Gholkar Reviewed-by: Gregory Price Acked-by: David Hildenbrand (Arm) Acked-by: Balbir Singh Cc: Alistair Popple Cc: Byungchul Park Cc: "Huang, Ying" Cc: Joshua Hahn Cc: Matthew Brost Cc: Rakie Kim Cc: Zi Yan Cc: Signed-off-by: Andrew Morton --- mm/mempolicy.c | 21 ++++++--------------- 1 file changed, 6 insertions(+), 15 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 501e0b80d7da..5720f7f54d94 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2060,24 +2060,15 @@ struct mempolicy *get_vma_policy(struct vm_area_struct *vma, bool vma_policy_mof(struct vm_area_struct *vma) { struct mempolicy *pol; + pgoff_t ilx; + bool mof; - if (vma->vm_ops && vma->vm_ops->get_policy) { - bool ret = false; - pgoff_t ilx; /* ignored here */ - - pol = vma->vm_ops->get_policy(vma, vma->vm_start, &ilx); - if (pol && (pol->flags & MPOL_F_MOF)) - ret = true; - mpol_cond_put(pol); - - return ret; - } - - pol = vma->vm_policy; + pol = __get_vma_policy(vma, vma->vm_start, &ilx); if (!pol) pol = get_task_policy(current); - - return pol->flags & MPOL_F_MOF; + mof = pol->flags & MPOL_F_MOF; + mpol_cond_put(pol); + return mof; } bool apply_policy_zone(struct mempolicy *policy, enum zone_type zone) -- cgit v1.2.3 From 644ad84870ab503f5619d21cf3d37efb134e82de Mon Sep 17 00:00:00 2001 From: Hajime Tazaki Date: Thu, 2 Jul 2026 10:25:46 +0900 Subject: mm: nommu: point to the write iterator upon split_vma When a user invokes munmap(2) on a partial region allocated by mmap(), the kernel may split the original region if necessary and shrink it to the correct size. At the beginning of vmi_shrink_vma(), the unused part is cleared; however, an assertion is triggered if the shrink occurs after split_vma(). This commit fixes the issue by correctly configuring the pointer to the iterator at the end of split_vma(). This bug was detected using the Linux Test Project (LTP) test linked below, running on a nommu UML (User-Mode Linux) environment (via an out-of-tree extension to UML). Here is a minimal reproducible chunk of code for this issue: void *addr; size_t pagesize = getpagesize(); addr = mmap(NULL, pagesize * 4, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); munmap(addr + pagesize * 1, pagesize); This is the console output with CONFIG_DEBUG_MAPLE_TREE=y. nommu: WARN at __mas_set_range:791 (1) MAS: tree=0000000091c23b08 enode=0000000065057663 (ma_active) Store Type: node_store [9/9] index=70af8000 last=ffffffffffffffff min=0 max=ffffffffffffffff sheaf=0000000000000000, request 0 depth=0, flags=0 maple_tree(0000000091c23b08) flags 307, height 1 root 0000000083394c06 0-ffffffffffffffff: node 0000000010c90bd6 depth 0 type 1 parent 0000000050e1ddf8 contents: 0000000000000000 707A 7FFF 00000000eb0ac2b5 707AFFFF 0000000000000000 7093FFFF 0000000045ead616 7095FFFF 0000000000000000 7096CFFF 000 00000681c7151 7096FFFF 0000000000000000 70AF3FFF 000000006c78b9e9 70AF4FFF 000000001914ab0b 70AF7FFF 00000000000 00000 FFFFFFFFFFFFFFFF 0000000000000000 0 0000000000000000 0 0000000000000000 0 0000000000000000 0 0000000000000 000 0 00000000bca8be4f 0-707a7fff: 0000000000000000 707a8000-707affff: 00000000eb0ac2b5 707b0000-7093ffff: 0000000000000000 70940000-7095ffff: 0000000045ead616 70960000-7096cfff: 0000000000000000 7096d000-7096ffff: 00000000681c7151 70970000-70af3fff: 0000000000000000 70af4000-70af4fff: 000000006c78b9e9 70af5000-70af7fff: 000000001914ab0b 70af8000-ffffffffffffffff: 0000000000000000 nommu: Pass: 796 Run:797 Link: https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/syscalls/mseal/mseal02.c Link: https://lore.kernel.org/20260702012546.665383-1-thehajime@gmail.com Signed-off-by: Hajime Tazaki Cc: Jann Horn Cc: Liam R. Howlett Cc: Lorenzo Stoakes Cc: Pedro Falcato Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/nommu.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/nommu.c b/mm/nommu.c index 277f663e1c5b..498e01ee40b0 100644 --- a/mm/nommu.c +++ b/mm/nommu.c @@ -1393,6 +1393,10 @@ static int split_vma(struct vma_iterator *vmi, struct vm_area_struct *vma, setup_vma_to_mm(vma, mm); setup_vma_to_mm(new, mm); vma_iter_store_new(vmi, new); + + /* vmi should point lower address */ + if (new_below) + vma_next(vmi); mm->map_count++; return 0; -- cgit v1.2.3 From 9b5e4809806cb300cc163b26fa70dfd36e3577b3 Mon Sep 17 00:00:00 2001 From: Ethan Nelson-Moore Date: Wed, 10 Jun 2026 15:09:04 -0700 Subject: maple_tree: remove undocumented CONFIG_MAPLE_RCU_DISABLED macro consults the macro CONFIG_MAPLE_RCU_DISABLED to determine whether to disable the mt_in_rcu() function (by making it always return false). This macro is not reachable via Kconfig, despite its name, and is not documented anywhere. Remove it to avoid polluting the CONFIG_* namespace. Discovered while searching for CONFIG_* symbols referenced in code but not defined in any Kconfig file. Link: https://lore.kernel.org/20260610220905.99860-1-enelsonmoore@gmail.com Signed-off-by: Ethan Nelson-Moore Acked-by: SeongJae Park Reviewed-by: Liam Howlett Reviewed-by: Alice Ryhl Cc: Andrew Ballance Signed-off-by: Andrew Morton --- include/linux/maple_tree.h | 4 ---- 1 file changed, 4 deletions(-) diff --git a/include/linux/maple_tree.h b/include/linux/maple_tree.h index 4a5631906aff..1b3014377105 100644 --- a/include/linux/maple_tree.h +++ b/include/linux/maple_tree.h @@ -11,7 +11,6 @@ #include #include #include -/* #define CONFIG_MAPLE_RCU_DISABLED */ /* * Allocated nodes are mutable until they have been inserted into the tree, @@ -864,9 +863,6 @@ static inline void mt_init(struct maple_tree *mt) static inline bool mt_in_rcu(struct maple_tree *mt) { -#ifdef CONFIG_MAPLE_RCU_DISABLED - return false; -#endif return mt->ma_flags & MT_FLAGS_USE_RCU; } -- cgit v1.2.3 From 1eba458a54d073aa8bd62627ad0e9264ef0f51e3 Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Tue, 7 Jul 2026 17:03:31 +0800 Subject: mm/kconfig: drop redundant memory hotplug dependencies MHP_MEMMAP_ON_MEMORY is defined inside the MEMORY_HOTPLUG block, and MEMORY_HOTPLUG already depends on SPARSEMEM_VMEMMAP. Keep the explicit MEMORY_HOTPLUG dependency for local readability, but drop the redundant SPARSEMEM_VMEMMAP dependency. ZONE_DEVICE depends on MEMORY_HOTREMOVE, which depends on MEMORY_HOTPLUG. MEMORY_HOTPLUG in turn depends on SPARSEMEM_VMEMMAP. Drop the direct MEMORY_HOTPLUG and SPARSEMEM_VMEMMAP dependencies from ZONE_DEVICE. This does not change the set of valid configurations. Link: https://lore.kernel.org/20260707090331.52971-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng Acked-by: David Hildenbrand (Arm) Acked-by: Muchun Song Reviewed-by: Lorenzo Stoakes Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/Kconfig | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index 3185500ce7b7..d28dde592de4 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -588,7 +588,7 @@ config MEMORY_HOTREMOVE config MHP_MEMMAP_ON_MEMORY def_bool y - depends on MEMORY_HOTPLUG && SPARSEMEM_VMEMMAP + depends on MEMORY_HOTPLUG depends on ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE endif # MEMORY_HOTPLUG @@ -1227,9 +1227,7 @@ config ZONE_DMA32 config ZONE_DEVICE bool "Device memory (pmem, HMM, etc...) hotplug support" - depends on MEMORY_HOTPLUG depends on MEMORY_HOTREMOVE - depends on SPARSEMEM_VMEMMAP select XARRAY_MULTI help -- cgit v1.2.3 From 092836fedd82cdafc6e2085c4b7a7878bfa3ca1d Mon Sep 17 00:00:00 2001 From: "David Hildenbrand (Arm)" Date: Thu, 9 Jul 2026 10:13:34 +0530 Subject: mm: standardize printing for pgtable entries Bad page map reporting currently stores page table entry values in an unsigned long long and prints them with fixed 64-bit-oriented format strings. This is inconsistent across call sites and does not work well for architectures where page table entry values are not naturally represented as 64-bit values, such as 32-bit or 128-bit entries. Introduce a common helper to convert raw page table entry values into a fixed-width hexadecimal string based on the actual entry size. Use it for bad page map reporting and for dumping the page table walk in __print_bad_page_map_pgtable(). Pass page table entry values to the reporting path as raw bytes together with their size, instead of forcing them through an unsigned long long. It keeps the printed output consistent and avoids truncation or misleading formatting for non-64-bit page table entries. Link: https://lore.kernel.org/20260709044334.1741263-1-anshuman.khandual@arm.com Signed-off-by: David Hildenbrand (Arm) Co-developed-by: Anshuman Khandual Signed-off-by: Anshuman Khandual Cc: Andriy Shevchenko Cc: David Hildenbrand Cc: Hugh Dickins Cc: Matthew Wilcox (Oracle) Cc: Ryan Roberts Signed-off-by: Andrew Morton --- mm/memory.c | 102 ++++++++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 79 insertions(+), 23 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index be743a9c6606..d2f14ba2261c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -519,9 +519,52 @@ static bool is_bad_page_map_ratelimited(void) return false; } +static void ptval_bytes_to_hex_str(char *buf, size_t buf_size, const void *entry, size_t entry_size) +{ + if (WARN_ON_ONCE(buf_size < entry_size * 2 + 1)) { + snprintf(buf, buf_size, "overflow"); + return; + } + + switch (entry_size) { + case sizeof(u32): + snprintf(buf, buf_size, "%08x", *(const u32 *)entry); + break; + case sizeof(u64): + snprintf(buf, buf_size, "%016llx", *(const u64 *)entry); + break; +#if defined(__SIZEOF_INT128__) + case sizeof(u128): + snprintf(buf, buf_size, "%016llx%016llx", + (unsigned long long)(*(const u128 *)entry >> 64), + (unsigned long long)*(const u128 *)entry); + break; +#endif + default: + snprintf(buf, buf_size, "unsupported"); + break; + } +} + +#define ptval_to_str(buf, val) \ + do { \ + auto __val = (val); \ + \ + ptval_bytes_to_hex_str((buf), sizeof(buf), &__val, sizeof(__val)); \ + } while (0) + +#if defined(__SIZEOF_INT128__) +#define PTVAL_STR_MAX (32 + 1) /* Max 128-bit value in hex + NUL */ +#else +#define PTVAL_STR_MAX (16 + 1) /* Max 64-bit value in hex + NUL */ +#endif + static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long addr) { - unsigned long long pgdv, p4dv, pudv, pmdv; + char pgd_str[PTVAL_STR_MAX]; + char p4d_str[PTVAL_STR_MAX]; + char pud_str[PTVAL_STR_MAX]; + char pmd_str[PTVAL_STR_MAX]; p4d_t p4d, *p4dp; pud_t pud, *pudp; pmd_t pmd, *pmdp; @@ -532,34 +575,34 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * see locking requirements for print_bad_page_map(). */ pgdp = pgd_offset(mm, addr); - pgdv = pgd_val(*pgdp); + ptval_to_str(pgd_str, pgd_val(*pgdp)); if (!pgd_present(*pgdp) || pgd_leaf(*pgdp)) { - pr_alert("pgd:%08llx\n", pgdv); + pr_alert("pgd:%s\n", pgd_str); return; } p4dp = p4d_offset(pgdp, addr); p4d = p4dp_get(p4dp); - p4dv = p4d_val(p4d); + ptval_to_str(p4d_str, p4d_val(p4d)); if (!p4d_present(p4d) || p4d_leaf(p4d)) { - pr_alert("pgd:%08llx p4d:%08llx\n", pgdv, p4dv); + pr_alert("pgd:%s p4d:%s\n", pgd_str, p4d_str); return; } pudp = pud_offset(p4dp, addr); pud = pudp_get(pudp); - pudv = pud_val(pud); + ptval_to_str(pud_str, pud_val(pud)); if (!pud_present(pud) || pud_leaf(pud)) { - pr_alert("pgd:%08llx p4d:%08llx pud:%08llx\n", pgdv, p4dv, pudv); + pr_alert("pgd:%s p4d:%s pud:%s\n", pgd_str, p4d_str, pud_str); return; } pmdp = pmd_offset(pudp, addr); pmd = pmdp_get(pmdp); - pmdv = pmd_val(pmd); + ptval_to_str(pmd_str, pmd_val(pmd)); /* * Dumping the PTE would be nice, but it's tricky with CONFIG_HIGHPTE, @@ -567,8 +610,7 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * doing another map would be bad. print_bad_page_map() should * already take care of printing the PTE. */ - pr_alert("pgd:%08llx p4d:%08llx pud:%08llx pmd:%08llx\n", pgdv, - p4dv, pudv, pmdv); + pr_alert("pgd:%s p4d:%s pud:%s pmd:%s\n", pgd_str, p4d_str, pud_str, pmd_str); } /* @@ -584,10 +626,11 @@ static void __print_bad_page_map_pgtable(struct mm_struct *mm, unsigned long add * page table lock. */ static void print_bad_page_map(struct vm_area_struct *vma, - unsigned long addr, unsigned long long entry, struct page *page, - enum pgtable_level level) + unsigned long addr, const void *entry, size_t entry_size, + struct page *page, enum pgtable_level level) { struct address_space *mapping; + char entry_str[PTVAL_STR_MAX]; pgoff_t index; if (is_bad_page_map_ratelimited()) @@ -596,8 +639,9 @@ static void print_bad_page_map(struct vm_area_struct *vma, mapping = vma->vm_file ? vma->vm_file->f_mapping : NULL; index = linear_page_index(vma, addr); - pr_alert("BUG: Bad page map in process %s %s:%08llx", current->comm, - pgtable_level_to_str(level), entry); + ptval_bytes_to_hex_str(entry_str, sizeof(entry_str), entry, entry_size); + pr_alert("BUG: Bad page map in process %s %s:%s", current->comm, + pgtable_level_to_str(level), entry_str); __print_bad_page_map_pgtable(vma->vm_mm, addr); if (page) dump_page(page, "bad page map"); @@ -627,8 +671,13 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) } } -#define print_bad_pte(vma, addr, pte, page) \ - print_bad_page_map(vma, addr, pte_val(pte), page, PGTABLE_LEVEL_PTE) +static void print_bad_pte(struct vm_area_struct *vma, unsigned long addr, + pte_t pte, struct page *page) +{ + auto entry = pte_val(pte); + + print_bad_page_map(vma, addr, &entry, sizeof(entry), page, PGTABLE_LEVEL_PTE); +} /** * __vm_normal_page() - Get the "struct page" associated with a page table entry. @@ -636,8 +685,9 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) * @addr: The address where the page table entry is mapped. * @pfn: The PFN stored in the page table entry. * @special: Whether the page table entry is marked "special". - * @level: The page table level for error reporting purposes only. * @entry: The page table entry value for error reporting purposes only. + * @entry_size: The size of @entry. + * @level: The page table level for error reporting purposes only. * * "Special" mappings do not wish to be associated with a "struct page" (either * it doesn't exist, or it exists but they don't want to touch it). In this @@ -697,7 +747,7 @@ static inline bool pgtable_level_has_pxx_special(enum pgtable_level level) */ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, unsigned long addr, unsigned long pfn, bool special, - unsigned long long entry, enum pgtable_level level) + const void *entry, size_t entry_size, enum pgtable_level level) { if (pgtable_level_has_pxx_special(level)) { if (unlikely(special)) { @@ -710,7 +760,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, if (is_zero_pfn(pfn) || is_huge_zero_pfn(pfn)) return NULL; - print_bad_page_map(vma, addr, entry, NULL, level); + print_bad_page_map(vma, addr, entry, entry_size, NULL, level); return NULL; } /* @@ -741,7 +791,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, if (unlikely(pfn > highest_memmap_pfn)) { /* Corrupted page table entry. */ - print_bad_page_map(vma, addr, entry, NULL, level); + print_bad_page_map(vma, addr, entry, entry_size, NULL, level); return NULL; } /* @@ -767,8 +817,10 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, struct page *vm_normal_page(struct vm_area_struct *vma, unsigned long addr, pte_t pte) { + auto entry = pte_val(pte); + return __vm_normal_page(vma, addr, pte_pfn(pte), pte_special(pte), - pte_val(pte), PGTABLE_LEVEL_PTE); + &entry, sizeof(entry), PGTABLE_LEVEL_PTE); } /** @@ -809,8 +861,10 @@ struct folio *vm_normal_folio(struct vm_area_struct *vma, unsigned long addr, struct page *vm_normal_page_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t pmd) { + auto entry = pmd_val(pmd); + return __vm_normal_page(vma, addr, pmd_pfn(pmd), pmd_special(pmd), - pmd_val(pmd), PGTABLE_LEVEL_PMD); + &entry, sizeof(entry), PGTABLE_LEVEL_PMD); } /** @@ -850,8 +904,10 @@ struct folio *vm_normal_folio_pmd(struct vm_area_struct *vma, struct page *vm_normal_page_pud(struct vm_area_struct *vma, unsigned long addr, pud_t pud) { + auto entry = pud_val(pud); + return __vm_normal_page(vma, addr, pud_pfn(pud), pud_special(pud), - pud_val(pud), PGTABLE_LEVEL_PUD); + &entry, sizeof(entry), PGTABLE_LEVEL_PUD); } #endif -- cgit v1.2.3 From 0b4268ac77fa6e2c05fc8b90441b0832b211385d Mon Sep 17 00:00:00 2001 From: Kaitao Cheng Date: Sun, 12 Jul 2026 17:33:26 +0800 Subject: mm/kconfig: drop redundant dependency wrappers Some mm Kconfig entries repeat dependencies that are already expressed by their surrounding blocks or menus. The zsmalloc allocator options menu already depends on ZSMALLOC, so the outer if ZSMALLOC block does not add any extra constraint. MEMORY_HOTREMOVE and MHP_MEMMAP_ON_MEMORY are both inside the if MEMORY_HOTPLUG block, so their local depends on MEMORY_HOTPLUG entries are redundant. PTE_MARKER_UFFD_WP is the only entry under if USERFAULTFD. Move the USERFAULTFD dependency into the symbol itself and combine it with the architecture support dependency. This keeps the same visibility and defaults while avoiding duplicate dependency expressions. Link: https://lore.kernel.org/20260712093326.8313-1-kaitao.cheng@linux.dev Signed-off-by: Kaitao Cheng Suggested-by: Julian Braha Reviewed-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Liam R. Howlett Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Suren Baghdasaryan Cc: Vlastimil Babka Signed-off-by: Andrew Morton --- mm/Kconfig | 10 +--------- 1 file changed, 1 insertion(+), 9 deletions(-) diff --git a/mm/Kconfig b/mm/Kconfig index d28dde592de4..060190e12bce 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -125,8 +125,6 @@ config ZSWAP_COMPRESSOR_DEFAULT config ZSMALLOC tristate -if ZSMALLOC - menu "Zsmalloc allocator options" depends on ZSMALLOC @@ -161,8 +159,6 @@ config ZSMALLOC_CHAIN_SIZE endmenu -endif - menu "Slab allocator options" config SLUB @@ -583,12 +579,10 @@ endchoice config MEMORY_HOTREMOVE bool "Allow for memory hot remove" - depends on MEMORY_HOTPLUG select MIGRATION config MHP_MEMMAP_ON_MEMORY def_bool y - depends on MEMORY_HOTPLUG depends on ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE endif # MEMORY_HOTPLUG @@ -1393,17 +1387,15 @@ menuconfig USERFAULTFD Enable the userfaultfd() system call that allows to intercept and handle page faults in userland. -if USERFAULTFD config PTE_MARKER_UFFD_WP bool "Userfaultfd write protection support for shmem/hugetlbfs" default y - depends on HAVE_ARCH_USERFAULTFD_WP + depends on USERFAULTFD && HAVE_ARCH_USERFAULTFD_WP help Allows to create marker PTEs for userfaultfd write protection purposes. It is required to enable userfaultfd write protection on file-backed memory types like shmem and hugetlbfs. -endif # USERFAULTFD # multi-gen LRU { config LRU_GEN -- cgit v1.2.3 From 0bd14001eb264247d565a5a44a71675df273640d Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:18 +0100 Subject: mm/vma: introduce VMA anon page offset field and add helpers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patch series "mm/rmap: index MAP_PRIVATE file-backed folios by anonymous pgoff", v5. In memory management we've managed to manufacture a great deal of confusion around the concept of anonymous memory. We have: 1. 'Pure anon' memory - anonymous VMAs whose folios are anonymous and swap-backed (thus for reclaim purposes, treated as anonymous). These are simple enough. 2. shmem - file-backed VMAs, file-backed folios (from rmap perspective) so present in the page cache and mapped by an address_space object, but whose folios are also swap-backed (thus treated as anonymous for reclaim purposes). 3. MAP_PRIVATE-mapped /dev/zero - a strange beast whose VMAs have vma->vm_file set, but which clears vma->vm_ops to satisfy vma_is_anonymous(), resulting in VMAs that were mmap()'d referencing a file, but are in every other sense anonymous, including the folios. 4. Other MAP_PRIVATE-file backed mappings - These possess file-backed VMAs and have file-backed folios until CoW'd, at which point those CoW'd folios are anonymous. This series fixes issue 3. In order for us to traverse VMAs using the reverse mapping, we require two fields - folio->mapping and folio->index. The first tells the rmap code where to look for VMAs, and the second tells it at which offset the folio starts within the referenced object. For anonymous folios, folio->mapping points at an anon_vma object. For file-backed folios, it points at an address_space. And: * For file-backed folios folio->index is simply the page offset of the start of the folio within the file. * For anonymous folios belonging to pure anon mappings, folio->index is equal to the anonymous page offset of the folio. * For anonymous folios belonging to file-backed mappings (i.e. CoW'd folios of a MAP_PRIVATE file-backed mapping), folio->index is equal to the file page offset. This series establishes a new anonymous page offset property of VMAs to allow us to map anonymous folios at their anonymous page offset, consistent with pure anon. The purpose of doing so is to lay the foundations for the scalable CoW work. This is necessary because scalable CoW looks in the maple tree for the VMA located at folio->index << PAGE_SHIFT, before falling back to looking up tracked remaps if necessary. The MAP_PRIVATE file-backed case means that folio indices will very often conflict with one another and this remap tracking becomes substantially more contended, and of course the fast path can never be used. This also makes it possible, in future, to unshare anonymously mapped folios with deep fork hierarchies on remap, eliminating the need for remap tracking in the vast majority of cases. Similar to page offset of pure anonymous VMAs, we update the anonymous page offset of unfaulted file-backed VMAs on remap, but do not once CoW'd (i.e. vma->anon_vma is non-NULL). Overall, there is little impact on mergeability, which remains exactly the same for pure anonymous and shared file-backed mappings, with the only impact being on MAP_PRIVATE-mapped file-backed mappings, which must now match on anonymous page offset as well as file page offset to be merged. To fail to merge like this would require CoW'ing the mapping, then finding another VMA with identical file and compatible page offset to remap next to. This is therefore very much an edge case that should have very little impact (and which scalable CoW may very well address in any case). This patch (of 16): Establish fields in vm_area_struct to store the anonymous page offset of VMAs. Initially, the anonymous page offset of a VMA is vma->vm_start >> PAGE_SHIFT. When a VMA is remapped to new_address its anonymous page offset is either updated to new_address >> PAGE_SHIFT if unfaulted or, if faulted, remains equal to the anonymous page offset it had when first faulted. Currently, anonymous folios belonging to CoW'd MAP_PRIVATE-mapped file-backed VMAs are tracked by their file offsets. By adding anonymous offset as a property of VMAs, we can now track them by their anonymous page offset instead. By tracking this, we provide the means by which to eliminate this inconsistency, and more importantly lay the foundations for future work for the scalable CoW anonymous rmap rework. This patch simply adds the fields and some simple helpers. Subsequent patches will update mm code to make use of these fields correctly. The fields chosen are packed in the VMA such that, for 64-bit kernel builds, no additional space is taken up. The first field is present on cacheline 0 containing key VMA fields, and the second on cacheline 3, which contains file-backed reverse mapping fields. Given the relative time spent accessing reverse mapping fields as well as updating them, there shouldn't be any performance impact here from false sharing. Update the VMA userland tests to account for this change. No callsites are updated yet, so no functional change intended. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-0-c21581c0c3c8@kernel.org Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-1-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Reviewed-by: Gregory Price (Meta) Reviewed-by: Xu Xin Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 59 +++++++++++++++++++++++++++++++++++++++++ include/linux/mm_types.h | 12 +++++++++ mm/vma.h | 14 ++++++++++ mm/vma_init.c | 1 + tools/testing/vma/include/dup.h | 26 ++++++++++++++++++ 5 files changed, 112 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 87feaa5a2b78..df78847f5f07 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -4393,6 +4393,65 @@ static inline pgoff_t vma_last_pgoff(const struct vm_area_struct *vma) return vma_end_pgoff(vma) - 1; } +/** + * vma_start_anon_pgoff() - Get the anonymous page offset of the start of @vma + * @vma: The VMA whose anonymous page offset is required. + * + * If unfaulted, then this is vma->vm_start >> PAGE_SHIFT, if faulted then the + * anonymous page offset at the time of first fault. + * + * If the VMA is anonymous, this returns the same value as vma_start_pgoff(). + * + * This value is used for tracking MAP_PRIVATE file-backed mappings by their + * anonymous page offset. + * + * Returns: The anonymous page offset of the start of @vma. + */ +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +/** + * vma_end_anon_pgoff() - Get the anonymous page offset of the exclusive end of + * @vma. + * @vma: The VMA whose end anonymous page offset is required. + * + * This returns the anonymous exclusive end page offset of @vma, which is useful + * for expressing page offset ranges. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The exclusive end anonymous page offset of @vma. + */ +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +/** + * vma_last_anon_pgoff() - Get the anonymous page offset of the last page in + * @vma. + * @vma: The VMA whose last anonymous page offset is required. + * + * See the description of vma_start_anon_pgoff() for a description of VMA + * anonymous page offsets. + * + * Returns: The last anonymous page offset of @vma. + */ +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline unsigned long vma_desc_size(const struct vm_area_desc *desc) { return desc->end - desc->start; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 939b5ea8c9e0..ebf0d912be7d 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -967,6 +967,11 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + /* + * Low 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma * list, after a COW of one of the file pages. A MAP_SHARED vma @@ -1041,6 +1046,13 @@ struct vm_area_struct { #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map vmlock_dep_map; #endif +#endif +#ifdef CONFIG_64BIT + /* + * High 32-bits of anonymous page offset. + * See vma_start_anon_pgoff() comment for details. + */ + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, diff --git a/mm/vma.h b/mm/vma.h index 0bc7d521e976..54ed7c744e3b 100644 --- a/mm/vma.h +++ b/mm/vma.h @@ -283,6 +283,20 @@ static inline void vma_set_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) vma->vm_pgoff = pgoff; } +static inline void __vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ +#ifdef CONFIG_64BIT + vma->__vm_anon_pgoff_hi = pgoff >> 32; +#endif + vma->__vm_anon_pgoff_lo = pgoff & GENMASK(31, 0); +} + +static inline void vma_set_anon_pgoff(struct vm_area_struct *vma, pgoff_t pgoff) +{ + vma_assert_can_modify(vma); + __vma_set_anon_pgoff(vma, pgoff); +} + static inline void vma_add_pgoff(struct vm_area_struct *vma, pgoff_t delta) { vma_assert_can_modify(vma); diff --git a/mm/vma_init.c b/mm/vma_init.c index 715feee283f0..baa7e82f47e3 100644 --- a/mm/vma_init.c +++ b/mm/vma_init.c @@ -51,6 +51,7 @@ static void vm_area_init_from(const struct vm_area_struct *src, dest->vm_end = src->vm_end; dest->anon_vma = src->anon_vma; dest->vm_pgoff = vma_start_pgoff(src); + __vma_set_anon_pgoff(dest, vma_start_anon_pgoff(src)); dest->vm_file = src->vm_file; dest->vm_private_data = src->vm_private_data; vm_flags_init(dest, src->vm_flags); diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index cdeb53bbdd1b..17f94e5de569 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -577,6 +577,7 @@ struct vm_area_struct { */ unsigned int vm_lock_seq; #endif + unsigned int __vm_anon_pgoff_lo; /* * A file's MAP_PRIVATE vma can be in both i_mmap tree and anon_vma @@ -612,6 +613,9 @@ struct vm_area_struct { #ifdef CONFIG_PER_VMA_LOCK /* Unstable RCU readers are allowed to read this. */ refcount_t vm_refcnt; +#endif +#ifdef CONFIG_64BIT + unsigned int __vm_anon_pgoff_hi; #endif /* * For areas with an address space and backing store, @@ -1320,6 +1324,28 @@ static inline pgoff_t vma_end_pgoff(const struct vm_area_struct *vma) return vma_start_pgoff(vma) + vma_pages(vma); } +static inline pgoff_t vma_start_anon_pgoff(const struct vm_area_struct *vma) +{ + pgoff_t pgoff = 0; + +#ifdef CONFIG_64BIT + pgoff += vma->__vm_anon_pgoff_hi; + pgoff <<= 32; +#endif + pgoff += vma->__vm_anon_pgoff_lo; + return pgoff; +} + +static inline pgoff_t vma_end_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_start_anon_pgoff(vma) + vma_pages(vma); +} + +static inline pgoff_t vma_last_anon_pgoff(const struct vm_area_struct *vma) +{ + return vma_end_anon_pgoff(vma) - 1; +} + static inline int vfs_mmap_prepare(struct file *file, struct vm_area_desc *desc) { return file->f_op->mmap_prepare(desc); -- cgit v1.2.3 From 51943a18ad4bd6ff8baea2da7b8cce2f86f1a959 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:19 +0100 Subject: mm: provide vma_[flags_]is_cow_mapping() and remove is_cow_mapping() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit All remaining callers of is_cow_mapping() are invoking it in the form of is_cow_mapping(vma->vm_flags) or an indirected version of this. Therefore, provide a helper - vma_is_cow_mapping() to directly test the VMA. Additionally provide a new helper vma_flags_is_cow_mapping() which performs the check using the new vma_flags_t type, and share this logic between vma_is_cow_mapping() and vma_desc_is_cow_mapping(). With these changes, no callers of is_cow_mapping() remain, so remove it. Also update the userland VMA tests to reflect the change. No functional change intended. [akpm@linux-foundation.org: fix kerneldoc comment typo, per Lorenzo] Link: https://lore.kernel.org/aob1goSSPH6sTN9y@gremlin Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-2-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Gregory Price (Meta) Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- arch/s390/mm/gmap_helpers.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c | 4 +- drivers/gpu/drm/drm_gem_shmem_helper.c | 2 +- drivers/gpu/drm/panthor/panthor_gem.c | 2 +- drivers/gpu/drm/ttm/ttm_bo_vm.c | 2 +- drivers/gpu/drm/xe/xe_device.c | 2 +- fs/proc/task_mmu.c | 2 +- include/linux/mm.h | 71 ++++++++++++++++++++++++++++++--- kernel/events/uprobes.c | 2 +- mm/gup.c | 2 +- mm/huge_memory.c | 8 ++-- mm/hugetlb.c | 2 +- mm/internal.h | 2 +- mm/memory.c | 25 ++++++------ mm/mempolicy.c | 2 +- tools/testing/vma/include/dup.h | 11 +++++ 16 files changed, 105 insertions(+), 36 deletions(-) diff --git a/arch/s390/mm/gmap_helpers.c b/arch/s390/mm/gmap_helpers.c index 4bf7c9012feb..cd5fded159c0 100644 --- a/arch/s390/mm/gmap_helpers.c +++ b/arch/s390/mm/gmap_helpers.c @@ -200,7 +200,7 @@ static int find_zeropage_pte_entry(pte_t *pte, unsigned long addr, * currently only works in COW mappings, which is also where * mm_forbids_zeropage() is checked. */ - if (!is_cow_mapping(walk->vma->vm_flags)) + if (!vma_is_cow_mapping(walk->vma)) return -EFAULT; *found_addr = addr; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c index 6a0699746fbc..0c7309080a7a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c @@ -377,9 +377,9 @@ static int amdgpu_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_str /* Workaround for Thunk bug creating PROT_NONE,MAP_PRIVATE mappings * for debugger access to invisible VRAM. Should have used MAP_SHARED * instead. Clearing VM_MAYWRITE prevents the mapping from ever - * becoming writable and makes is_cow_mapping(vm_flags) false. + * becoming writable and makes vma_is_cow_mapping(vma) false. */ - if (is_cow_mapping(vma->vm_flags) && + if (vma_is_cow_mapping(vma) && !(vma->vm_flags & VM_ACCESS_FLAGS)) vm_flags_clear(vma, VM_MAYWRITE); diff --git a/drivers/gpu/drm/drm_gem_shmem_helper.c b/drivers/gpu/drm/drm_gem_shmem_helper.c index 06d019d51d3e..177d0e0b9334 100644 --- a/drivers/gpu/drm/drm_gem_shmem_helper.c +++ b/drivers/gpu/drm/drm_gem_shmem_helper.c @@ -753,7 +753,7 @@ int drm_gem_shmem_mmap(struct drm_gem_shmem_object *shmem, struct vm_area_struct return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; dma_resv_lock(shmem->base.resv, NULL); diff --git a/drivers/gpu/drm/panthor/panthor_gem.c b/drivers/gpu/drm/panthor/panthor_gem.c index 770556353968..d2eec46f7abe 100644 --- a/drivers/gpu/drm/panthor/panthor_gem.c +++ b/drivers/gpu/drm/panthor/panthor_gem.c @@ -761,7 +761,7 @@ static int panthor_gem_mmap(struct drm_gem_object *obj, struct vm_area_struct *v return ret; } - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (!refcount_inc_not_zero(&bo->cmap.mmap_count)) { diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 88babf435ac2..872bf444b1f0 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -489,7 +489,7 @@ static const struct vm_operations_struct ttm_bo_vm_ops = { int ttm_bo_mmap_obj(struct vm_area_struct *vma, struct ttm_buffer_object *bo) { /* Enforce no COW since would have really strange behavior with it. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; drm_gem_object_get(&bo->base); diff --git a/drivers/gpu/drm/xe/xe_device.c b/drivers/gpu/drm/xe/xe_device.c index 9d119c95a569..de5fdf49d729 100644 --- a/drivers/gpu/drm/xe/xe_device.c +++ b/drivers/gpu/drm/xe/xe_device.c @@ -330,7 +330,7 @@ static int xe_pci_barrier_mmap(struct file *filp, if (vma->vm_end - vma->vm_start > SZ_4K) return -EINVAL; - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return -EINVAL; if (vma->vm_flags & (VM_READ | VM_EXEC)) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 817e3e0f9194..5c54aebe2118 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1693,7 +1693,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, if (!pte_write(pte)) return false; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return false; if (likely(!mm_flags_test(MMF_HAS_PINNED, vma->vm_mm))) return false; diff --git a/include/linux/mm.h b/include/linux/mm.h index df78847f5f07..20361b4344ea 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2271,17 +2271,76 @@ void unpin_user_pages(struct page **pages, unsigned long npages); void unpin_user_folio(struct folio *folio, unsigned long npages); void unpin_folios(struct folio **folios, unsigned long nfolios); -static inline bool is_cow_mapping(vm_flags_t flags) +/** + * vma_flags_is_cow_mapping() - Do these VMA flags imply a CoW mapping? + * @flags: The VMA flags to check. + * + * Mappings which could be CoW'd (subject to Copy-On-Write faults) are + * described as CoW mappings. + * + * All mappings backed by anonymous folios (all anonymous mappings and most + * MAP_PRIVATE-file backed ranges) are CoW mappings. + * + * All other mappings (including all MAP_SHARED mappings) are non-CoW. + * + * The criteria are !VMA_SHARED_BIT, VMA_MAYWRITE_BIT. + * + * VMA_MAYWRITE_BIT is checked instead of VMA_WRITE_BIT to account for both + * future mprotect() calls which can render a read-only mapping writable, and + * GUP with FOLL_FORCE (e.g. ptrace) which can CoW a read-only mapping. + * + * - No anonymous mapping can ever clear VMA_MAYWRITE_BIT. + * + * - Writes to anonymous mappings do not immediately result in CoW faults but + * may do so after the process is forked or if a read is followed by a + * write. + * + * - Writes to MAP_PRIVATE file-backed mappings result in CoW faults and may + * do so again after fork. + * + * - MAP_SHARED mappings of a file opened read-only are transformed into + * VMA_MAYSHARE_BIT, !VMA_SHARED_BIT, !VMA_MAYWRITE_BIT mappings, so remain + * non-CoW. + * + * - Drivers may clear VMA_MAYWRITE_BIT but do so at mmap() time and cannot + * mark themselves anonymous. Having cleared this flag it is not valid for + * them to leave the VMA_WRITE_BIT flag set. + * + * As a consequence, the anonymous reverse mapping only tracks CoW mappings. + * + * Returns: true if the flags indicate a CoW mapping, otherwise false. + */ +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +/** + * vma_is_cow_mapping() - Is this VMA a CoW mapping? + * @vma: The VMA to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA is a CoW mapping, otherwise false. + */ +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) { - return (flags & (VM_SHARED | VM_MAYWRITE)) == VM_MAYWRITE; + return vma_flags_is_cow_mapping(&vma->flags); } +/** + * vma_desc_is_cow_mapping() - Is this VMA descriptor a CoW mapping? + * @desc: The VMA descriptor to check. + * + * See vma_flags_is_cow_mapping() for details. + * + * Returns: true if the VMA descriptor describes a CoW mapping, otherwise + * false. + */ static inline bool vma_desc_is_cow_mapping(struct vm_area_desc *desc) { - const vma_flags_t *flags = &desc->vma_flags; - - return vma_flags_test(flags, VMA_MAYWRITE_BIT) && - !vma_flags_test(flags, VMA_SHARED_BIT); + return vma_flags_is_cow_mapping(&desc->vma_flags); } #ifndef CONFIG_MMU diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index ae2f3b9f8d50..eb0d11092fb3 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -513,7 +513,7 @@ int uprobe_write(struct arch_uprobe *auprobe, struct vm_area_struct *vma, uprobe = container_of(auprobe, struct uprobe, arch); - if (WARN_ON_ONCE(!is_cow_mapping(vma->vm_flags))) + if (WARN_ON_ONCE(!vma_is_cow_mapping(vma))) return -EINVAL; /* diff --git a/mm/gup.c b/mm/gup.c index 99902c15703b..8ea3de60e82d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1236,7 +1236,7 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags) * Anon pages in shared mappings are surprising: now * just reject it. */ - if (!is_cow_mapping(vm_flags)) + if (!vma_is_cow_mapping(vma)) return -EFAULT; } } else if (!(vm_flags & VM_READ)) { diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 644d6905b49c..ff13b57d9d56 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1681,7 +1681,7 @@ vm_fault_t vmf_insert_pfn_pmd(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1789,7 +1789,7 @@ vm_fault_t vmf_insert_pfn_pud(struct vm_fault *vmf, unsigned long pfn, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); pfnmap_setup_cachemode_pfn(pfn, &pgprot); @@ -1931,7 +1931,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, * applied special bit, or we made the PRIVATE mapping be * able to wrongly write to the backend MMIO. */ - VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + VM_WARN_ON_ONCE(vma_is_cow_mapping(src_vma) && pmd_write(pmd)); goto set_pmd; } @@ -2052,7 +2052,7 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + if (vma_is_cow_mapping(vma) && pud_write(pud)) { pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_wrprotect(pud); } diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 79e5c3b3e850..49bf325325c0 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4898,7 +4898,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, pte_t *src_pte, *dst_pte, entry; struct folio *pte_folio; unsigned long addr; - bool cow = is_cow_mapping(src_vma->vm_flags); + bool cow = vma_is_cow_mapping(src_vma); struct hstate *h = hstate_vma(src_vma); unsigned long sz = huge_page_size(h); unsigned long npages = pages_per_huge_page(h); diff --git a/mm/internal.h b/mm/internal.h index 68db5abd0a4c..a75e7641ef49 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1353,7 +1353,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, * ... because we only care about writable private ("COW") * mappings where we have to break COW early. */ - return is_cow_mapping(vma->vm_flags); + return vma_is_cow_mapping(vma); } /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ diff --git a/mm/memory.c b/mm/memory.c index d2f14ba2261c..396d7b9059e6 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -780,7 +780,7 @@ static inline struct page *__vm_normal_page(struct vm_area_struct *vma, /* Only CoW'ed anon folios are "normal". */ if (pfn == index) return NULL; - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) return NULL; } } @@ -1002,7 +1002,6 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, pte_t *dst_pte, pte_t *src_pte, struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long addr, int *rss) { - vm_flags_t vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); softleaf_t entry = softleaf_from_pte(orig_pte); pte_t pte = orig_pte; @@ -1026,7 +1025,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, rss[mm_counter(folio)]++; if (!softleaf_is_migration_read(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { /* * COW mappings require pages in both parent and child * to be set to read. A previously exclusive entry is @@ -1067,7 +1066,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * save and restore device driver state). */ if (softleaf_is_device_private_write(entry) && - is_cow_mapping(vm_flags)) { + vma_is_cow_mapping(dst_vma)) { entry = make_readable_device_private_entry( swp_offset(entry)); pte = swp_entry_to_pte(entry); @@ -1082,7 +1081,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * exclusive entries currently only support private writable * (ie. COW) mappings. */ - VM_BUG_ON(!is_cow_mapping(src_vma->vm_flags)); + VM_BUG_ON(!vma_is_cow_mapping(src_vma)); if (try_restore_exclusive_pte(src_vma, addr, src_pte, orig_pte)) return -EBUSY; return -ENOENT; @@ -1181,7 +1180,7 @@ static __always_inline void __copy_present_ptes(struct vm_area_struct *dst_vma, } /* If it's a COW mapping, write protect it both processes. */ - if (is_cow_mapping(src_vma->vm_flags) && writable) { + if (vma_is_cow_mapping(src_vma) && writable) { wrprotect_ptes(src_mm, addr, src_pte, nr); pte = pte_wrprotect(pte); } @@ -1602,9 +1601,9 @@ copy_page_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma) * We need to invalidate the secondary MMU mappings only when * there could be a permission downgrade on the ptes of the * parent mm. And a permission downgrade will only happen if - * is_cow_mapping() returns true. + * vma_is_cow_mapping() returns true. */ - is_cow = is_cow_mapping(src_vma->vm_flags); + is_cow = vma_is_cow_mapping(src_vma); if (is_cow) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_PAGE, @@ -2437,7 +2436,7 @@ static bool vm_mixed_zeropage_allowed(struct vm_area_struct *vma) if (mm_forbids_zeropage(vma->vm_mm)) return false; /* zeropages in COW mappings are common and unproblematic. */ - if (is_cow_mapping(vma->vm_flags)) + if (vma_is_cow_mapping(vma)) return true; /* Mappings that do not allow for writable PTEs are unproblematic. */ if (!(vma->vm_flags & (VM_WRITE | VM_MAYWRITE))) @@ -2888,7 +2887,7 @@ vm_fault_t vmf_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr, BUG_ON(!(vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP))); BUG_ON((vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) == (VM_PFNMAP|VM_MIXEDMAP)); - BUG_ON((vma->vm_flags & VM_PFNMAP) && is_cow_mapping(vma->vm_flags)); + BUG_ON((vma->vm_flags & VM_PFNMAP) && vma_is_cow_mapping(vma)); BUG_ON((vma->vm_flags & VM_MIXEDMAP) && pfn_valid(pfn)); if (addr < vma->vm_start || addr >= vma->vm_end) @@ -3300,7 +3299,7 @@ static int remap_pfn_range_prepare_vma(struct vm_area_struct *vma, unsigned long size) { const unsigned long end = addr + PAGE_ALIGN(size); - const bool is_cow = is_cow_mapping(vma->vm_flags); + const bool is_cow = vma_is_cow_mapping(vma); int err; err = get_remap_pgoff(is_cow, addr, end, vma->vm_start, vma->vm_end, @@ -6800,7 +6799,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, * FAULT_FLAG_UNSHARE only applies to COW mappings. Let's * just treat it like an ordinary read-fault otherwise. */ - if (!is_cow_mapping(vma->vm_flags)) + if (!vma_is_cow_mapping(vma)) *flags &= ~FAULT_FLAG_UNSHARE; } else if (*flags & FAULT_FLAG_WRITE) { /* Write faults on read-only mappings are impossible ... */ @@ -6808,7 +6807,7 @@ static vm_fault_t sanitize_fault_flags(struct vm_area_struct *vma, return VM_FAULT_SIGSEGV; /* ... and FOLL_FORCE only applies to COW mappings. */ if (WARN_ON_ONCE(!(vma->vm_flags & VM_WRITE) && - !is_cow_mapping(vma->vm_flags))) + !vma_is_cow_mapping(vma))) return VM_FAULT_SIGSEGV; } #ifdef CONFIG_PER_VMA_LOCK diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 5720f7f54d94..3498a5651d50 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -844,7 +844,7 @@ bool folio_can_map_prot_numa(struct folio *folio, struct vm_area_struct *vma, return false; /* Also skip shared copy-on-write folios */ - if (is_cow_mapping(vma->vm_flags) && folio_maybe_mapped_shared(folio)) + if (vma_is_cow_mapping(vma) && folio_maybe_mapped_shared(folio)) return false; /* Folios are pinned and can't be migrated */ diff --git a/tools/testing/vma/include/dup.h b/tools/testing/vma/include/dup.h index 17f94e5de569..af2fd3f607b5 100644 --- a/tools/testing/vma/include/dup.h +++ b/tools/testing/vma/include/dup.h @@ -1162,6 +1162,17 @@ static inline bool vma_is_shared_maywrite(struct vm_area_struct *vma) return is_shared_maywrite(&vma->flags); } +static inline bool vma_flags_is_cow_mapping(const vma_flags_t *flags) +{ + return vma_flags_test(flags, VMA_MAYWRITE_BIT) && + !vma_flags_test(flags, VMA_SHARED_BIT); +} + +static inline bool vma_is_cow_mapping(const struct vm_area_struct *vma) +{ + return vma_flags_is_cow_mapping(&vma->flags); +} + static inline struct vm_area_struct *vma_next(struct vma_iterator *vmi) { /* -- cgit v1.2.3 From 7e6543d1f939eaaca008c13395e52cbd07605cb0 Mon Sep 17 00:00:00 2001 From: "Lorenzo Stoakes (ARM)" Date: Thu, 13 Aug 2026 18:32:20 +0100 Subject: mm: introduce linear_anon_page_index() MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This function provides the anonymous equivalent of linear_page_index(), instead offsetting based on the anonymous page offset of the VMA. It is valid only for anonymous or MAP_PRIVATE file-backed mappings, in other words CoW mappings. For pure anon VMAs, this will be equal to linear_page_index(). Assert that both of these invariants are true in linear_anon_page_index() and implement the algorithm in __linear_anon_page_index(). Note that MAP_PRIVATE-/dev/zero mappings will satisfy vma_is_anonymous() but not fulfill this invariant, so when asserting this we check vma->vm_file to account for this. We do not update callsites yet, so no functional change intended. Also const-ify vma_is_anonymous() to make it compatible with the const-ified linear_anon_page_index(). While we're here, update linear_page_index() to be more succinct. VMA userland tests are also updated accordingly. Link: https://lore.kernel.org/20260813-b4-scalable-cow-virt-pgoff-v5-3-c21581c0c3c8@kernel.org Signed-off-by: Lorenzo Stoakes (ARM) Reviewed-by: Gregory Price (Meta) Acked-by: David Hildenbrand (Arm) Cc: Adrian Hunter Cc: Alexander Deucher Cc: Alexander Gordeev Cc: Alexander Shishkin Cc: Alistair Popple Cc: Arnaldo Carvalho de Melo Cc: Arnd Bergmann Cc: Baolin Wang Cc: Baoquan He Cc: Barry Song Cc: Boris Brezillon Cc: Byungchul Park Cc: Chengming Zhou Cc: Chris Li Cc: Christan König Cc: Christian Borntraeger Cc: Claudio Imbrenda Cc: Dave Airlie Cc: Dev Jain Cc: Gerald Schaefer Cc: Greg Kroah-Hartman Cc: Harry Yoo Cc: Heiko Carstens Cc: Huang Ray Cc: "Huang, Ying" Cc: Ian Rogers Cc: Ingo Molnar Cc: James Clark Cc: Jan Kara Cc: Jann Horn Cc: Janosch Frank Cc: Jason Gunthorpe Cc: Jiri Olsa Cc: John Hubbard Cc: Joshua Hahn Cc: Kairui Song Cc: Kees Cook Cc: Kemeng Shi Cc: Lance Yang Cc: Liam R. Howlett Cc: Liviu Dudau Cc: Maarten Lankhorst Cc: Marc Rutland Cc: "Masami Hiramatsu (Google)" Cc: Matthew Auld Cc: Matthew Brost Cc: Matthew Wilcox (Oracle) Cc: Maxime Ripard Cc: Miaohe Lin Cc: Michal Hocko Cc: Mike Rapoport Cc: Muchun Song Cc: Namhyung kim Cc: Naoya Horiguchi Cc: Nhat Pham Cc: Nico Pache Cc: Oleg Nesterov Cc: Oscar Salvador Cc: Pedro Falcato Cc: Peter Xu Cc: Peter Zijlstra Cc: Rakie Kim Cc: Rik van Riel Cc: Rodrigo Vivi Cc: Ryan Roberts Cc: Steven Price Cc: Suren Baghdasaryan Cc: Sven Schnelle Cc: Thomas Hellström Cc: Thomas Zimemrmann Cc: Vasily Gorbik Cc: Vlastimil Babka Cc: xu xin Cc: Zi Yan Signed-off-by: Andrew Morton --- include/linux/mm.h | 2 +- include/linux/pagemap.h | 40 +++++++++++++++++++++++++++++++++++++--- tools/testing/vma/include/dup.h | 25 ++++++++++++++++++++++++- 3 files changed, 62 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 20361b4344ea..dd09c438fa23 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1556,7 +1556,7 @@ static inline void vma_desc_set_anonymous(struct vm_area_desc *desc) desc->vm_ops = NULL; } -static inline bool vma_is_anonymous(struct vm_area_struct *vma) +static inline bool vma_is_anonymous(const struct vm_area_struct *vma) { return !vma->vm_ops; } diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index c6fc783aaee5..0adfa6605653 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1094,10 +1094,44 @@ static inline pgoff_t linear_page_delta(const struct vm_area_struct *vma, static inline pgoff_t linear_page_index(const struct vm_area_struct *vma, const unsigned long address) { - pgoff_t pgoff; + return linear_page_delta(vma, address) + vma_start_pgoff(vma); +} + +static inline pgoff_t __linear_anon_page_index(const struct vm_area_str