forked from Qortal/Brooklyn
You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
187 lines
8.2 KiB
187 lines
8.2 KiB
.. _transhuge: |
|
|
|
============================ |
|
Transparent Hugepage Support |
|
============================ |
|
|
|
This document describes design principles for Transparent Hugepage (THP) |
|
support and its interaction with other parts of the memory management |
|
system. |
|
|
|
Design principles |
|
================= |
|
|
|
- "graceful fallback": mm components which don't have transparent hugepage |
|
knowledge fall back to breaking huge pmd mapping into table of ptes and, |
|
if necessary, split a transparent hugepage. Therefore these components |
|
can continue working on the regular pages or regular pte mappings. |
|
|
|
- if a hugepage allocation fails because of memory fragmentation, |
|
regular pages should be gracefully allocated instead and mixed in |
|
the same vma without any failure or significant delay and without |
|
userland noticing |
|
|
|
- if some task quits and more hugepages become available (either |
|
immediately in the buddy or through the VM), guest physical memory |
|
backed by regular pages should be relocated on hugepages |
|
automatically (with khugepaged) |
|
|
|
- it doesn't require memory reservation and in turn it uses hugepages |
|
whenever possible (the only possible reservation here is kernelcore= |
|
to avoid unmovable pages to fragment all the memory but such a tweak |
|
is not specific to transparent hugepage support and it's a generic |
|
feature that applies to all dynamic high order allocations in the |
|
kernel) |
|
|
|
get_user_pages and follow_page |
|
============================== |
|
|
|
get_user_pages and follow_page if run on a hugepage, will return the |
|
head or tail pages as usual (exactly as they would do on |
|
hugetlbfs). Most GUP users will only care about the actual physical |
|
address of the page and its temporary pinning to release after the I/O |
|
is complete, so they won't ever notice the fact the page is huge. But |
|
if any driver is going to mangle over the page structure of the tail |
|
page (like for checking page->mapping or other bits that are relevant |
|
for the head page and not the tail page), it should be updated to jump |
|
to check head page instead. Taking a reference on any head/tail page would |
|
prevent the page from being split by anyone. |
|
|
|
.. note:: |
|
these aren't new constraints to the GUP API, and they match the |
|
same constraints that apply to hugetlbfs too, so any driver capable |
|
of handling GUP on hugetlbfs will also work fine on transparent |
|
hugepage backed mappings. |
|
|
|
Graceful fallback |
|
================= |
|
|
|
Code walking pagetables but unaware about huge pmds can simply call |
|
split_huge_pmd(vma, pmd, addr) where the pmd is the one returned by |
|
pmd_offset. It's trivial to make the code transparent hugepage aware |
|
by just grepping for "pmd_offset" and adding split_huge_pmd where |
|
missing after pmd_offset returns the pmd. Thanks to the graceful |
|
fallback design, with a one liner change, you can avoid to write |
|
hundreds if not thousands of lines of complex code to make your code |
|
hugepage aware. |
|
|
|
If you're not walking pagetables but you run into a physical hugepage |
|
that you can't handle natively in your code, you can split it by |
|
calling split_huge_page(page). This is what the Linux VM does before |
|
it tries to swapout the hugepage for example. split_huge_page() can fail |
|
if the page is pinned and you must handle this correctly. |
|
|
|
Example to make mremap.c transparent hugepage aware with a one liner |
|
change:: |
|
|
|
diff --git a/mm/mremap.c b/mm/mremap.c |
|
--- a/mm/mremap.c |
|
+++ b/mm/mremap.c |
|
@@ -41,6 +41,7 @@ static pmd_t *get_old_pmd(struct mm_stru |
|
return NULL; |
|
|
|
pmd = pmd_offset(pud, addr); |
|
+ split_huge_pmd(vma, pmd, addr); |
|
if (pmd_none_or_clear_bad(pmd)) |
|
return NULL; |
|
|
|
Locking in hugepage aware code |
|
============================== |
|
|
|
We want as much code as possible hugepage aware, as calling |
|
split_huge_page() or split_huge_pmd() has a cost. |
|
|
|
To make pagetable walks huge pmd aware, all you need to do is to call |
|
pmd_trans_huge() on the pmd returned by pmd_offset. You must hold the |
|
mmap_lock in read (or write) mode to be sure a huge pmd cannot be |
|
created from under you by khugepaged (khugepaged collapse_huge_page |
|
takes the mmap_lock in write mode in addition to the anon_vma lock). If |
|
pmd_trans_huge returns false, you just fallback in the old code |
|
paths. If instead pmd_trans_huge returns true, you have to take the |
|
page table lock (pmd_lock()) and re-run pmd_trans_huge. Taking the |
|
page table lock will prevent the huge pmd being converted into a |
|
regular pmd from under you (split_huge_pmd can run in parallel to the |
|
pagetable walk). If the second pmd_trans_huge returns false, you |
|
should just drop the page table lock and fallback to the old code as |
|
before. Otherwise, you can proceed to process the huge pmd and the |
|
hugepage natively. Once finished, you can drop the page table lock. |
|
|
|
Refcounts and transparent huge pages |
|
==================================== |
|
|
|
Refcounting on THP is mostly consistent with refcounting on other compound |
|
pages: |
|
|
|
- get_page()/put_page() and GUP operate on head page's ->_refcount. |
|
|
|
- ->_refcount in tail pages is always zero: get_page_unless_zero() never |
|
succeeds on tail pages. |
|
|
|
- map/unmap of the pages with PTE entry increment/decrement ->_mapcount |
|
on relevant sub-page of the compound page. |
|
|
|
- map/unmap of the whole compound page is accounted for in compound_mapcount |
|
(stored in first tail page). For file huge pages, we also increment |
|
->_mapcount of all sub-pages in order to have race-free detection of |
|
last unmap of subpages. |
|
|
|
PageDoubleMap() indicates that the page is *possibly* mapped with PTEs. |
|
|
|
For anonymous pages, PageDoubleMap() also indicates ->_mapcount in all |
|
subpages is offset up by one. This additional reference is required to |
|
get race-free detection of unmap of subpages when we have them mapped with |
|
both PMDs and PTEs. |
|
|
|
This optimization is required to lower the overhead of per-subpage mapcount |
|
tracking. The alternative is to alter ->_mapcount in all subpages on each |
|
map/unmap of the whole compound page. |
|
|
|
For anonymous pages, we set PG_double_map when a PMD of the page is split |
|
for the first time, but still have a PMD mapping. The additional references |
|
go away with the last compound_mapcount. |
|
|
|
File pages get PG_double_map set on the first map of the page with PTE and |
|
goes away when the page gets evicted from the page cache. |
|
|
|
split_huge_page internally has to distribute the refcounts in the head |
|
page to the tail pages before clearing all PG_head/tail bits from the page |
|
structures. It can be done easily for refcounts taken by page table |
|
entries, but we don't have enough information on how to distribute any |
|
additional pins (i.e. from get_user_pages). split_huge_page() fails any |
|
requests to split pinned huge pages: it expects page count to be equal to |
|
the sum of mapcount of all sub-pages plus one (split_huge_page caller must |
|
have a reference to the head page). |
|
|
|
split_huge_page uses migration entries to stabilize page->_refcount and |
|
page->_mapcount of anonymous pages. File pages just get unmapped. |
|
|
|
We are safe against physical memory scanners too: the only legitimate way |
|
a scanner can get a reference to a page is get_page_unless_zero(). |
|
|
|
All tail pages have zero ->_refcount until atomic_add(). This prevents the |
|
scanner from getting a reference to the tail page up to that point. After the |
|
atomic_add() we don't care about the ->_refcount value. We already know how |
|
many references should be uncharged from the head page. |
|
|
|
For head page get_page_unless_zero() will succeed and we don't mind. It's |
|
clear where references should go after split: it will stay on the head page. |
|
|
|
Note that split_huge_pmd() doesn't have any limitations on refcounting: |
|
pmd can be split at any point and never fails. |
|
|
|
Partial unmap and deferred_split_huge_page() |
|
============================================ |
|
|
|
Unmapping part of THP (with munmap() or other way) is not going to free |
|
memory immediately. Instead, we detect that a subpage of THP is not in use |
|
in page_remove_rmap() and queue the THP for splitting if memory pressure |
|
comes. Splitting will free up unused subpages. |
|
|
|
Splitting the page right away is not an option due to locking context in |
|
the place where we can detect partial unmap. It also might be |
|
counterproductive since in many cases partial unmap happens during exit(2) if |
|
a THP crosses a VMA boundary. |
|
|
|
The function deferred_split_huge_page() is used to queue a page for splitting. |
|
The splitting itself will happen when we get memory pressure via shrinker |
|
interface.
|
|
|