| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
Pull KVM updates from Paolo Bonzini:
"ARM:
- pKVM state isolation: years after pKVM started being enabled
upstream, we finally get the basics in the form of state isolation.
The vcpu state is now off-limit for the host for protected VMs
- pKVM also grows a hypervisor-private memory allocator, allowing it
to deal with sub-page allocation, much like everywhere else
- pKVM also lands a trace_printk() equivalent, which hopefully will
make debugging a bit easier
- First shot at annotating portions of the KVM/arm64 code to be
extracted and built on s390. The real fun is yet to come
- Allow pre-faulting of stage-2 canonical mappings from userspace,
joining x86 and s390 in implementing this feature
- Add a reverse mapping infrastructure for shadow stage-2,
significantly improving performance when MMU notifiers kick in, and
avoiding full unmap of all shadow tables for this VM
- Next step of GICv5 enablement, adding support for the IRS block,
and providing both SPI and LPI support. This is the first point
where KVM/arm64 can support an SMP GICv5 guest
- As a preliminary to eventually enabling CCA, add a bunch of
infrastructure to KVM/arm64, splitting some of the basic operations
into individual backends (normal VM, normal pKVM VM, protected pKVM
VM). This allows CCA to later slot in without polluting the rest of
KVM with its firmware-driven nonsense
- Make ID registers immutable from the point where a vcpu is able to
run, even if it fails to run. This prevents KVM from sampling
untrusted values
- Restore some type checking to the EL2 object. KVM/arm64 originally
had some strong type checking between the kernel and the EL2
object, but lost it along the way when the private hypercall
interface was (somewhat pointlessly) converted to SMCCC. Type
checking is back, syscall style, and with added sparse goodies
- Expand the FFA support to FFA_NOTIFICATION_*, and add some
sanitisation of the protocol that pKVM relays to the "trusted"
runtime
- Make GICv3 RD assignment more robust to userspace nonsense
- Convert the remaining MMU fault handler backends to
kvm_s2_fault_vma_info and co., fixing a number of bugs in the
process
- Fix the SError/SEA selftests to be both more robust and less
braindead
- Fix a smattering of documentation bugs, some of them as a result of
our AI friends spitting out credible assertions based on
non-existing symbols
- The usual collection of low impact bug fixes (SMCCC, timers, PMU)
Loongarch:
- Add separate vmid support
- Simplify MSI interrupt injection
- Add crash hypercall support
- Allow to set pv_feature until vCPU run
- Enable HW page table walk automatically in selftests
s390:
- rework page allocation for secure guest variable storage
- Selftests and documentation improvements
- Replace get_zeroed_page with kzalloc
- Fix corner cases of floating interrupt injection
- Initial arm64 on s390 support
x86:
- Clean up the handling of vcpu->arch.xen.long_mode to be consistent
and correctly handle 32-bit/64-bit alignment, and various other bug
fixes have accumulated over the months since v1
- Grab KVM's SRCU "lock" in kvm_zap_gfn_range() to fix a recurring
bug where callers fail to do so
- Fix a variety of issues in the pre-fault code that are mostly
benign in the current code base, but will show up in big ways
if/when the pre-faulting logic is used to implementing
prefetch-on-fault for the TDP MMU
- For PREEMPT_RT=y kernels, drop the rmaps' individual bit-spinlock
and just walk the rmaps' with mmu_lock taken. On the write side
(which always uses mmu_lock), the bit-spinlock was assuming that
preemption was disabled, but that is *not* true when rwlocks become
sleepable
- Prepare a memory_fault exit (paired with -EFAULT) when KVM hits an
error PFN, e.g. so that userspace can resolve transient issues
instead of killing the guest. Convert a variety of WARN()+EFAULT
paths into KVM_BUG_ON()+EIO so that KVM can hard commit to filling
memory_fault on -EFAULTs that occur in the page fault path
- Fix a NX hugepage un-accounting bug where KVM would always uncharge
the shadow MMU, even when handling TDP MMU pages
- Mask off the address space ID when checking for illegal usage of
kvm_set_internal_memslot() to prevent false negatives on the sanity
check
- Explicitly zero reserved flags when delivering a SynIC message to
avoid leaking a byte of stack information to the guest
- Rework TDX's handling of the canonical MMU root level to not rely
on userspace to fill CPUID correctly, as doing so lets userspace
trivially trigger a KVM_BUG_ON()
- Reject KVM_SET_NESTED_STATE if userspace "restores" L1 with
EFER.LMA=1 && EFER.LME=0, which confuses KVM into trying to
configure an 64-bit MMU without PAE
- Ignore EFER.LMA (clear it from KVM's cached save state) if
EFER.LME=0 when preparing L2 state, to fix another way to confuse
KVM's MMU (and because real hardware also ignores EFER.LMA)
- Improve KVM's sanity checks for a misconfigured MMU to (hopefully)
detect all of the ways KVM can botch EFER.LMA vs. CR4.PAE
- Remove the "full" out-param from kvm_x86_ops.flush_tlb_gva() as the
param is only used to avoid what effectively amounts to a single
memory write
- Fix a mostly-benign bug where KVM can write through a guest's
read-only mapping due to not checking the error code when
expediting emulated MMIO page faults
- Honor the guest's view of the "EFER.LMSLE must be zero" defeature
so that userspace can provide migration compatibility between Rome
and Milan+
- Pend a KVM_REQ_EVENT request when an exception is morphed into a
nested VM-Exit, otherwise KVM can fail to actually deliver the
nested VM-Exit, e.g. when injecting a page fault
- Add support for PML on SVM
- Add support for virtualizing Bus Lock Detect on SVM, and fix a
number of DR6 bugs and warts along the way
- Fix a variety of warts in KVM's handling of guest TSC, kvmclock,
and timekeeping in general, as a (big) first step towards
addressing several longstanding problems with the kvmclock
implementation. Most notably, fix a precision issue where
get_kvmclock_ns() computed the clock from the *host* TSC without
applying guest TSC scaling, causing systemic drift from the values
the guest computes from its own TSC
- Rip out KVM's shadow timekeeping structures, now that ktime
snapshots are powerful enough to replace it
- Misc fixes and cleanups
x86 (AMD):
- Fix a benign-in-practice bug where KVM unintentionally sets a
non-zero error code when delivering exceptions to SEV-ES+ guests
- Force a new ASID in both VMCBs on pCPU migration to fix a bug where
KVM can run a VMCB with a stale ASID due to the tracking being
per-CPU, not per-VMCB
- Return INVALID_INPUT to the guest instead of exiting to userspace
with -EIO if KVM can't access "guest request" buffer, i.e. try to
keep the guest alive if the guest or userspace goofs
x86 (Intel):
- Enable Notify VM-Exits (now called Instruction Timeout in the SDM)
and Bus Lock VM-Exits for TDX guests when they're enabled by
userspace (KVM forgot to wire up the TDX Module knobs)
- Move all common (between VMX and TDX) APIs/helpers to
common.h/main.c, and change all namespace prefixes from "vmx" to
"vt"
- Fix a thankfully benign bug (due to other areas of KVM being
hardened) where userspace can bypass INIT_MEM_REGION's address wrap
check by providing a nr_pages that will overflow a u64 when
converted to bytes, and fix an also- benign off-by-one bug in the
check (inclusive vs. exclusive)
- Set the vmcs.GUEST_ACTIVITY_STATE back to ACTIVE when userspace
sets KVM's mp_state to anything other than HALTED, e.g. so that a
vCPU doesn't get stuck in HLT if userspace emulates RESET
guest_memfd:
- Take the invalidate lock when unbinding a dying file to fix a race
with kvm_gmem_invalidate_end() that results in KVM leaving the MMU
invalidation count elevated. Not taking the lock in the first place
was pure misguided paranoid
- When binding a memslot, purge the bindings if storing the new
memslot fails, as Xarray's "store range" API can subtly fail
partway through
- Establish memslot<=>guest_memfd bindings *after* the memslot is
ready, as the memslot becomes reachable the instant the binding is
installed
- Don't grab a page refcount when reloading an SNP VMSA and stop
returning a "struct page" from guest_memfd's PFN lookup altogether,
to ensure that KVM doesn't hold any unnecessary refcounts (the
lifetime of the page is ensured by MMU notifiers and similar
hooks). Preventing unnecessary refcounts greatly simplifies
supporting in-place SHARED=>PRIVATE conversions
- Add support for in-place PRIVATE<=>SHARED conversion, i.e. allow a
single guest_memfd instance to be used for both PRIVATE and SHARED
variations of CoCo guests' memory (though not at the same time).
This eliminates the need to double allocate guest memory for TDX
and SNP, and is basically a hard requirement for pKVM, which relies
on being able to preserve memory contents on conversions
- Hack-a-fix a deadlock bug in ->error_remove_folio() by returning
-EBUSY if the filemap invalidate lock is contended. Punt on a
proper fix for the moment, as a proper fix likely involves
reworking truncate_error_folio()
- Add a pile of selftests for in-place conversions
- Misc guest_memfd cleanups
Generic:
- Move some configuration items to preprocessor symbols, to help with
arm-on-s390's need to compile the same code twice with different
values
- Allow KVM implementations to choose alternative device names, also
for s390's support of ARM virtualmachines
- Switch the KVM-VFIO interface and external consumers over to
standard file-based reference counting, eliminating all external
KVM symbol exports
Selftests:
- Rework KVM selftest's handling of memory regions all so that PPC
can force its page table allocations to be naturally aligned
- PPC support
- Fix emulation of VMPTRST and VMPTRLD when eVMCS is in-use to match
real Hyper-V's behavior
- Saturate L2's TSC frequency if stacking L1 and L2 scaling
multipliers exceeds what hardware supports, as attempting to run L2
with an illegal multiplier will either cause VM-Entry to fail (VMX)
or trigger a #GP when writing MSR_AMD64_TSC_RATIO (SVM). The #GP is
thankfully benign in the grand scheme as even "unsafe" WRMSRs have
exception fixup
- Rework KVM selftest's handling of memory regions so that PPC can
force its page table allocations to be naturally aligned
- Clean up selftest's VMX instruction helpers, and drop an absurd
number of completely useless return values, e.g. functions that
unconditionally return %true and then have every caller assert
success
- Add selftests support for Zhaoxin CPUs
- Overhaul the TSC scaling test to add a regression test for the
saturation behavior, and to increase its coverage in general
- Fix selftests usage various NUMA syscalls to account for an
undocumented, but known since 2007, off-by-one kernel bug. Add
helpers to try and avoid re-introducing the same flaws in the
future
- Automatically run xAPIC IPI migration test (between NUMA nodes)
when possible, as the entire point of the test is verify KVM's
handling of page migration of the APIC backing page
- Misc cleanups"
* tag 'for-linus' of git://git.kernel.org/pub/scm/virt/kvm/kvm: (506 commits)
KVM: always make in-place conversion a compile-time constant when possible
KVM: x86: leave the legacy private attribute in for TDX and SEV
KVM: split CONFIG_KVM_PRIVATE_MEMATTR_VM out of KVM_VM_MEMORY_ATTRIBUTES
selftests: kvm: remove useless comma operator
arm64: Use proper include variant
MAINTAINERS: Add entry for ARM on s390 KVM
KVM: s390: arm64: Integrate arm on s390
KVM: s390: arm64: Implement basic page fault handler
KVM: s390: arm64: Implement vCPU IOCTLs
KVM: s390: arm64: Implement vm/vcpu create destroy
KVM: s390: arm64: Implement required functions
KVM: s390: Add basic arm64 kvm module
s390/hwcaps: Report SAE support as hwcap
KVM: s390: arm64: Introduce host definitions
s390: Introduce Start Arm Execution instruction
KVM: s390: Use arm64 code
s390/tools: Use arm64 headers
KVM: s390: Use dedicated function for migration mode
s390/Kconfig: remove PCI dependency from HAS_IOMEM
KVM: s390: Kick PV cpus at the right time for service irqs
...
|
|
Remove the unnecessary static call indirection and make
gmem_in_place_conversion a constant if CONFIG_KVM_PRIVATE_MEMATTR_VM
is false.
Require that CONFIG_KVM_PRIVATE_MEMATTR_VM be enabled together with
CONFIG_KVM_GUEST_MEMFD and with a definition of kvm_arch_has_private_mem.
This way, the three possible cases can be expressed without nested #ifs.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
With Linux 7.4 the "private" memory attribute was moved from the VM
to the individual guest_memfd file descriptor. The old semantics are
available for backwards compatibility via CONFIG_KVM_VM_MEMORY_ATTRIBUTES.
However, this ties support for VM memory attributes entirely to
a backwards compatibility symbol, which is a problem when adding
VM memory attributes other than PRIVATE. Thus, make
CONFIG_KVM_VM_MEMORY_ATTRIBUTES the equivalent of the old
CONFIG_KVM_GENERIC_MEMORY_ATTRIBUTES, and use a more descriptive name
for the private memory attribute. Do this now, instead of waiting
for more attributes to come, before distros start adjusting
their build configuration.
Reviewed-by: Xiaoyao Li <xiaoyao.li@intel.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs idmap constification from Christian Brauner:
"This makes struct mnt_idmap const throughout the tree.
A mount's idmapping is immutable. The only thing modified after
alloc_mnt_idmap() is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only reads from
it. That is the same model struct cred uses and the idmapping is
sensitive enough to deserve it.
The conversion goes bottom up through the vfs{g,u}id and fs{g,u}id
helpers, the privilege checks, acl, xattr and quota and then every
inode operation that takes an idmap, so ->permission(), ->getattr(),
->setattr(), ->create(), ->mkdir(), ->mknod(), ->symlink(),
->rename(), ->tmpfile(), ->get_acl(), ->set_acl() and ->fileattr_set()
with their implementations in every filesystem, plus the LSM hooks.
struct vfsmount, struct mount_kattr and struct kstatmount store a
const pointer and nop_mnt_idmap and invalid_mnt_idmap live in rodata,
so anything that tries to modify them faults instead of silently
corrupting the initial idmapping"
* tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (27 commits)
fs: make nop_mnt_idmap and invalid_mnt_idmap const
fs: port mnt_idmap() and file_mnt_idmap() to const mnt_idmap
fs: port vfs_*() helpers to const mnt_idmap
fs: port ->setattr() to pass const mnt_idmap
fs: port ->getattr() to pass const mnt_idmap
fs: port ->create() to pass const mnt_idmap
fs: port ->symlink() to pass const mnt_idmap
fs: port ->mkdir() to pass const mnt_idmap
fs: port ->rename() to pass const mnt_idmap
fs: port ->mknod() to pass const mnt_idmap
fs: port ->tmpfile() to pass const mnt_idmap
fs: port ->get_acl() to pass const mnt_idmap
fs: port ->set_acl() to pass const mnt_idmap
fs: port ->fileattr_set() to pass const mnt_idmap
fs: port xattr to const mnt_idmap
fs: port ->permission() to pass const mnt_idmap
fs: port acl to const mnt_idmap
fs: port inode_init_owner() to const mnt_idmap
fs: port inode_owner_or_capable() to const mnt_idmap
fs: port privilege checking helpers to const mnt_idmap
...
|
|
KVM x86 misc changes for 7.4
- Mask off the address space ID when checking for illegal usage of
kvm_set_internal_memslot() to prevent false negatives on the sanity check.
- Explicitly zero reserved flags when delivering a SynIC message to avoid
leaking a byte of stack information to the guest.
- Rework TDX's handling of the canonical MMU root level to not rely on
userspace to fill CPUID correctly, as doing so lets userspace trivially
trigger a KVM_BUG_ON().
- Reject KVM_SET_NESTED_STATE if userspace "restores" L1 with EFER.LMA=1 &&
EFER.LME=0, which confuses KVM into trying to configure an 64-bit MMU
without PAE.
- Ignore EFER.LMA (clear it from KVM's cached save state) if EFER.LME=0 when
preparing L2 state, to fix another way to confuse KVM's MMU (and because
real hardware also ignores EFER.LMA).
- Improve KVM's sanity checks for a misconfigured MMU to (hopefully) detect
all of the ways KVM can botch EFER.LMA vs. CR4.PAE.
- Remove the "full" out-param from kvm_x86_ops.flush_tlb_gva() as the param
is only used to avoid what effectively amounts to a single memory write.
- Fix a mostly-benign bug where KVM can write through a guest's read-only
mapping due to not checking the error code when expediting emulated MMIO
page faults.
- Honor the guest's view of the "EFER.LMSLE must be zero" defeature so that
userspace can provide migration compatibility between Rome and Milan+.
- Pend a KVM_REQ_EVENT request when an exception is morphed into a nested
VM-Exit, otherwise KVM can fail to actually deliver the nested VM-Exit,
e.g. when injecting a page fault.
- Add support for PML on SVM.
- Add support for virtualizing Bus Lock Detect on SVM, and fix a number of
DR6 bugs and warts along the way.
- Misc fixes and cleanups.
|
|
KVM guest_memfd and CoCo changes for 7.4
- Take the invalidate lock when unbinding a dying file to fix a race with
kvm_gmem_invalidate_end() that results in KVM leaving the MMU invalidation
count elevated. Not taking the lock in the first place was pure misguided
paranoid.
- When binding a memslot, purge the bindings if storing the new memslot fails,
as Xarray's "store range" API can subtly fail partway through.
- Establish memslot<=>guest_memfd bindings *after* the memslot is ready, as
the memslot becomes reachable the instant the binding is installed.
- Don't grab a page refcount when reloading an SNP VMSA and stop returning a
"struct page" from guest_memfd's PFN lookup altogether, to ensure that KVM
doesn't hold any unnecessary refcounts (the lifetime of the page is ensured
by MMU notifiers and similar hooks). Preventing unnecessary refcounts
greatly simplifies supporting in-place SHARED=>PRIVATE conversions.
- Add support for in-place PRIVATE<=>SHARED conversion, i.e. allow a single
guest_memfd instance to be used for both PRIVATE and SHARED variations of
CoCo guests' memory (though not at the same time). This eliminates the need
to double allocate guest memory for TDX and SNP, and is basically a hard
requirement for pKVM, which relies on being able to preserve memory contents
on conversions.
- Hack-a-fix a deadlock bug in ->error_remove_folio() by returning -EBUSY if
the filemap invalidate lock is contended. Punt on a proper fix for the
moment, as a proper fix likely involves reworking truncate_error_folio().
- Add a pile of selftests for in-place conversions.
- Misc guest_memfd cleanups
Conflicts in arch/arm64/kvm/mmu.c: convert to use s2vi.
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson into HEAD
LoongArch KVM changes for v7.4
1. Add separate vmid support.
2. Simplify MSI interrupt injection.
3. Add crash hypercall support.
4. Allow to set pv_feature until vCPU run.
5. Enable HW page table walk automatically in selftests.
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/kvmarm/kvmarm into HEAD
KVM/arm64 updates for 7.4
* New features
- pKVM state isolation: years after pKVM started being enabled
upstream, we finally get the basics in the form of state
isolation. The vcpu state is now off-limit for the host for
protected VMs.
- pKVM also grows a hypervisor-private memory allocator, allowing it
to deal with sub-page allocation, much like everywhere else.
- pKVM also lands a trace_printk() equivalent, which hopefully will
make debugging a bit easier.
- First shot at annotating portions of the KVM/arm64 code to be
extracted and built on s390. The real fun is yet to come.
- Allow pre-faulting of stage-2 canonical mappings from userspace,
joining x86 and s390 in implementing this feature.
- Add a reverse mapping infrastructure for shadow stage-2,
significantly improving performance when MMU notifiers kick in,
and avoiding full unmap of all shadow tables for this VM.
- Next step of GICv5 enablement, adding support for the IRS block,
and providing both SPI and LPI support. This is the first point
where KVM/arm64 can support an SMP GICv5 guest.
* Improvements
- As a preliminary to eventually enabling CCA, add a bunch of
infrastructure to KVM/arm64, splitting some of the basic
operations into individual backends (normal VM, normal pKVM VM,
protected pKVM VM). This allows CCA to later slot in without
polluting the rest of KVM with its firmware-driven nonsense.
- Make ID registers immutable from the point where a vcpu is able
to run, even if it fails to run. This prevents KVM from sampling
untrusted values.
- Restore some type checking to the EL2 object. KVM/arm64 originally
had some strong type checking between the kernel and the EL2
object, but lost it along the way when the private hypercall
interface was (somewhat pointlessly) converted to SMCCC. Type
checking is back, syscall style, and with added sparse goodies.
- Expand the FFA support to FFA_NOTIFICATION_*, and add some
sanitisation of the protocol that pKVM relays to the "trusted"
runtime.
* Fixes
- Make GICv3 RD assignment more robust to userspace nonsense
- Convert the remaining MMU fault handler backends to
kvm_s2_fault_vma_info and co, fixing a number of bugs in the
process.
- Fix the SError/SEA selftests to be both more robust and less
braindead.
- Fix a smattering of documentation bugs, some of them as a result
of our AI friends spitting out credible assertions based on
non-existing symbols.
- The usual collection of low impact bug fixes (SMCCC, timers, PMU).
|
|
Defining KVM_MMIO is not flexible enough for multi-KVM systems where one
KVM implements MMIO but others do not. Therefore, remove KVM_MMIO from the
config space and use the macro __KVM_NO_MMIO instead. Define __KVM_NO_MMIO
for KVM/s390 and for HV-only 64-bit Book3s (aka pSeries/sPAPR) configurations.
In particular, for PowerPC configurations:
* KVM_BOOK3S_32 got KVM_MMIO via KVM_BOOK3S_32_HANDLER and KVM_BOOK3S_PR_POSSIBLE
* KVM_BOOK3S_64 got KVM_MMIO via KVM_BOOK3S_PR_POSSIBLE if !KVM_BOOK3S_HV_POSSIBLE
* KVM_BOOK3S_64 + KVM_BOOK3S_64_PR got KVM_MMIO via KVM_BOOK3S_PR_POSSIBLE
* likewise for KVM_BOOK3S_64 + KVM_BOOK3S_64_PR + KVM_BOOK3S_64_HV
* KVM_E500V2 and KVM_E500MC directly selected KVM_MMIO
but KVM_BOOK3S_64 + KVM_BOOK3S_64_HV didn't and therefore needs __KVM_NO_MMIO.
coalesced_mmio.c is now always compiled and linked. However, for
architectures/configurations defining __KVM_NO_MMIO this effectively
collapses to an empty object with no mmio code.
No functional changes.
Suggested-by: Marc Zyngier <maz@kernel.org>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Link: https://patch.msgid.link/20260930173441.66762-4-seiden@linux.ibm.com
[Simplify powerpc condition, add underscores. - Paolo]
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
With s390 hosting multiple independent KVM implementations within the
same kernel build (native s390 KVM and arm64 on s390), global Kconfig
selections force both implementations to inherit options that only apply
to one of them.
Replace global Kconfig options which only apply to one KVM
implementation with architecture-specific opt-in defines in the
respective <asm/kvm_host.h>. Only remove Kconfig symbols that would be
different on both s390 KVM implementations
- __KVM_HAVE_INVALID_WAKEUPS (formerly HAVE_KVM_INVALID_WAKEUPS)
- __KVM_HAVE_MSI (formerly HAVE_KVM_MSI)
- __KVM_HAVE_NO_POLL (formerly HAVE_KVM_NO_POLL)
- __KVM_HAVE_READONLY_MEM (formerly HAVE_KVM_READONLY_MEM)
- __KVM_HAVE_VCPU_RUN_PID_CHANGE (formerly HAVE_KVM_VCPU_RUN_PID_CHANGE)
This allows each architecture and guest implementation to configure its
supported capabilities independently in its kvm_host header and eliminates
adding unnecessary fallback stubs across different KVM targets.
Acked-by: Hendrik Brueckner <brueckner@linux.ibm.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Link: https://patch.msgid.link/20261013121343.1997112-1-seiden@linux.ibm.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Allow KVM implementations to choose alternative device names. This is
especially useful for architectures providing multiple KVM
implementations simultaneously. Architectures providing multiple KVM
implementations must compile the KVM common code once per implementation
and mangle symbols.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Link: https://patch.msgid.link/20260930173441.66762-2-seiden@linux.ibm.com
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
* kvm-arm64/pre-faulting:
: \
: Enable stage-2 pre-faulting in the canonical IPA space, implementing the
: KVM_PRE_FAULT_MEMORY API. Patches courtesy of Lorenzo Stoakes, and based
: on an initial work by Jack Thomson.
: /
KVM: selftests: Add nested pre-fault test for arm64
KVM: selftests: Add option for different backing in pre-fault tests
KVM: selftests: Enable pre_fault_memory_test for arm64
Documentation: KVM: document arm64 KVM_PRE_FAULT_MEMORY
KVM: arm64: Implement KVM_PRE_FAULT_MEMORY
KVM: arm64: Pass walk flags to kvm_pgtable_get_leaf()
KVM: arm64: Propagate EHWPOISON in kvm_s2_fault_pin_pfn()
KVM: arm64: Size the stage-2 memcache from the fault MMU
KVM: arm64: Propagate and use kvm_s2_fault_result on S2 fault
KVM: arm64: Propagate and use mmu in s2fd when handling guest aborts
KVM: arm64: Propagate and use esr in s2fd when handling guest aborts
KVM: arm64: Use ESR helpers in guest abort handling
arm64: Add ESR fault helpers
KVM: Allow architectures to disallow pre-fault
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
git://git.kernel.org/pub/scm/linux/kernel/git/kvms390/linux into HEAD
KVM/vfio: Use file-based reference counting for KVM
This series switches the KVM-VFIO interface and external consumers over to
standard file-based reference counting, eliminating all external KVM symbol
exports.
Currently, VFIO integrates with KVM by looking up kvm_get_kvm_safe() and
kvm_put_kvm() dynamically using symbol_get(), manually tracking module
reference counts and storing a put_kvm function pointer in struct
vfio_device.
In the ARM64-on-s390 architecture, a second concurrent KVM module
(kvm-arm64) is introduced alongside native KVM to host hardware-accelerated
ARM64 guests. Having exported global symbols (like kvm_get_kvm/kvm_put_kvm)
creates symbol conflicts and prevents clean coexistence of two KVM modules.
Additionally, the file based counting simplifies the code and reuses
the existing fs refcounting.
Instead of passing raw KVM pointers and managing module symbols manually, the
interface now passes the underlying VM file descriptor throughout VFIO and
associated architecture subsystems. To safely extract the KVM instance from a
file, an architecture-namespaced helper mechanism is introduced that verifies
the file belongs to the expected KVM implementation before accessing its
internal state. A back-pointer from the KVM instance to its associated file is
maintained across its lifecycle so subsystems can safely acquire file
references on demand. Finally, with VFIO, architecture page tracking, and
device hooks converted to use file references, the remaining KVM reference-
counting exports are restricted strictly to internal KVM modules.
|
|
Neither the VCPU_RUN in kvm_main.c nor the VCPU RUN in kvm_host.h names
an ioctl that exists. The one that enters the guest is KVM_RUN, so refer
to it by that name.
Signed-off-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260915084441.3846947-2-fuad.tabba@linux.dev
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Remove the now unused file_is_kvm function.
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Switch kvm_get_kvm, kvm_get_kvm_safe, and kvm_put_kvm from
EXPORT_SYMBOL_GPL to EXPORT_SYMBOL_FOR_KVM_INTERNAL. These symbols are
now only used within KVM itself. No external module needs them now that
VFIO uses file-based reference counting. Remove the corresponding
exemptions from the x86 KVM Makefile exports check.
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Replace manual module reference counting with file-based reference
counting for KVM integration. Previously, VFIO used symbol_get() to
obtain function pointers for kvm_get_kvm_safe() and kvm_put_kvm(),
then manually tracked module references through these symbols. This
approach required storing the put_kvm function pointer in each device
and carefully managing symbol references.
Pass struct file pointers instead of struct kvm pointers throughout the
VFIO-KVM interface, leveraging the kernel's existing file reference
counting via get_file()/get_file_active() and fput(). Convert the x86
page-track API and update s390 vfio to use file_to_kvm_<arch>(). This
simplifies the code.
Suggested-by: Jason Gunthorpe <jgg@nvidia.com>
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Acked-by: Alex Williamson <alex@shazbot.org>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
[borntraeger@linux.ibm.com remove _safe from vfio_device_get_kvm_safe]
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Add kvm->file as a no-reference back-pointer to the VM file in struct
kvm. The pointer is set at VM creation time and cleared under
WRITE_ONCE() in kvm_vm_release() before the file is freed.
Callers storing the file must take their own reference.
Co-developed-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
Add a macro mechanism that generates an opt-in and arch-namespaced
file_to_kvm_<arch>() to convert a file reference to a kvm object if the
file handle represents a kvm handle.
Activate for x86 and for s390 (native KVM only).
Suggested-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Steffen Eiden <seiden@linux.ibm.com>
Reviewed-by: Claudio Imbrenda <imbrenda@linux.ibm.com>
Acked-by: Janosch Frank <frankja@linux.ibm.com>
Acked-by: Sean Christopherson <seanjc@google.com>
Signed-off-by: Christian Borntraeger <borntraeger@linux.ibm.com>
|
|
memory_failure() invokes ->error_remove_folio() while holding the
poisoned folio lock. KVM's guest_memfd callback then takes
mapping->invalidate_lock for read before invalidating guest mappings.
This can deadlock with paths that hold mapping->invalidate_lock for write
and then try to lock the same folio, e.g. truncation from MADV_REMOVE or
FALLOC_FL_PUNCH_HOLE. Because rwsems are fair, the problem is not limited
to guest_memfd punch-hole: a pending writer can also block future readers,
so ->error_remove_folio() must not block on the invalidate lock while
holding the poisoned folio lock.
Use filemap_invalidate_trylock_shared() in kvm_gmem_error_folio(). If the
invalidate lock is contended, fail the recovery instead of potentially
deadlocking the memory-failure path.
Fixes: a7800aa80ea4 ("KVM: Add KVM_CREATE_GUEST_MEMFD ioctl() for guest-specific backing memory")
Signed-off-by: Hao Zhang <zhanghao1@kylinos.cn>
Link: https://lore.kernel.org/all/tencent_7DA064CAC9B463EB9B651801011E2CE4230A@qq.com/
Link: https://patch.msgid.link/ambEdSPjerZIVN0b@192.168.1.215
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Now that KVM holds kvm->lock for the entirety of vCPU creation, from when
created_vcpus is incremented until the new vCPU is fully onlined, WARN if
the impossible happens and KVM somehow sees a discrepancy between the
number of vCPUs "created" and "onlined".
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-8-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Now that KVM uses kvm_get_vcpu_by_id() to check for an existing vCPU ID
before doing any meaningful work, which was made possible by holding
kvm->lock for the entirety of vCPU creation, revert the now-redundant
"early" vCPU ID tracking. The claims about the impact of kvm->vcpu_ids on
the memory footprint were a wee bit wrong: the worst case scenario isn't
256 bytes per VM, it's 256 "unsigned longs" per VM, i.e. 2048 bytes per VM.
Increasing the size of "struct kvm" by 2048 nearly doubled the total size
on many architectures, and tripped x86's KVM_SANITY_CHECK_VM_STRUCT_SIZE,
which was added to detect this *exact* scenario, where a single change
significantly increased the size of "struct kvm". I.e. attempting to build
KVM with CONFIG_DEBUG_KERNEL=n fails on x86 (the build failures got missed
because all build bots apparently test only CONFIG_DEBUG_KERNEL=y kernels,
and maintainers' test flows were similarly lacking).
This reverts commit 97d65b544f48b2ee49f6aea32145e3e7969955dc.
Fixes: 97d65b544f48 ("KVM: Check for duplicate vcpu_id as early as possible")
Reported-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Closes: https://lore.kernel.org/all/56a4bc35ee605588b7cc36c8e45c12b5f3b506cb.camel@guillain.net
Reported-by: Paweł S <spawel523@gmail.com>
Closes: https://lore.kernel.org/all/CABD%3DWFOS4j4hDv%2BpW-eEM9HAM2q2GY_iYdAG%2BqvYcUEinUrcQQ@mail.gmail.com
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-7-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Now that kvm->lock is held for the entirety of vCPU creation, check for a
conflicting vCPU ID at the begnning of vCPU creation, before the arch
precreate() hook is invoked. This will allow reverting commit 97d65b544f48
("KVM: Check for duplicate vcpu_id as early as possible"). For now, keep
the redundant vcpu_ids tracking as a sanity check.
No functional change intended (absent KVM bugs, checking vcpu_ids and
walking kvm_get_vcpu_by_id() should yield the same result).
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-6-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
When creating a vCPU, don't drop kvm->lock to when doing the bulk of actual
vCPU creation, as allowing multiple vCPUs to be created in parallel adds
significant complexity in KVM (as evidenced by the many related bugs), and
all known VMMs fully serialize vCPU creation. Remove all manually locking
of kvm->lock from kvm_arch_vcpu_{post,}create() for obvious reasons.
For many years, "everyone" has assumed that dropping kvm->lock was done for
performance reasons optimization, e.g. to allow userspace to create all
vCPUs concurrently for latency purposes. But as above, no known VMM does
that. Looking at the history of this code, before commit 11ec28047118
("KVM: Convert vm lock to a mutex"), kvm->lock was a spinlock. I.e. KVM
*had* to drop kvm->lock when doing the bulk of vCPU creation, otherwise KVM
couldn't do normal memory allocations. When kvm->lock got turned into a
mutex for unrelated reasons, no one took advantage updated of the change to
simplify vCPU creation. And 19 years later, everyone just assumed that KVM
continued to deal with the complexity for performance reasons.
Furthermore, naively parallelizing vCPU creation in userspace is likely a
net negative due to the overheads of task creation. Unless a VMM carefully
avoids the extra overhead related to parallelization, e.g. spawns each
vCPU's thread before creating the vCPU, creating vCPUs concurrently is a
net *negative* up until about ~64 vCPUs, after which the times are a wash.
The absolute speed of light _is_ faster if KVM doesn't hold kvm-lock, but
at vCPU counts of ~16 or less, it's probably in the noise when considering
total VM creation time, as the added latency is less than 1ms up until 16
or so vCPUs.
On top of all that, KVM has had a *lot* of fatal bugs (most often found by
syzkaller) related to vCPUs being created while trying to do per-VM
operations (basically, see every flow that locks all vCPUs). I.e. the
parallel vCPU creation "support" is actively harmful as the only "use case"
is for misbehaving userspace to exploit KVM bugs.
Serializing vCPU creation will allow reverting commit 97d65b544f48 ("KVM:
Check for duplicate vcpu_id as early as possible"), which had "minor" math
error: the worst case scenario isn't "256 bytes per VM", it's "256 unsigned
longs per VM", i.e. 2048 bytes per VM, which doubles the size of each VM
and pushes several architectures into order-1 allocations.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-5-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
Reject locking of all vCPUs if vCPU creation is in-progress, i.e. if the
number of "created" vCPUs doesn't match the number of "onlined" vCPUs.
It's simply not possible to guarantee that KVM has truly locked all vCPUs
if one or more vCPUs are actively being created. Holding kvm->lock does
prevent in-flight vCPUs from being fully onlined, but it's infeasible for
common KVM to know whether or not that provides sufficient protection.
In practice, this is likely a minor bug fix for the ARM and RISC-V usage of
kvm_trylock_all_vcpus(), and a glorified nop for everything else. E.g.
ARM's kvm_timer_vcpu_init() can race kvm_vm_ioctl_set_counter_offset() with
respect to observing KVM_ARCH_FLAG_VM_COUNTER_OFFSET.
Opportunistically drop x86's existing manual checks on vCPU creation being
in-progress as all of x86's checks immediately precede or follow locking of
all vCPUs. Leave arm64 and RISC-V alone for the moment, as their checks
aren't as obviously redundant/equivalent.
Signed-off-by: Sean Christopherson <seanjc@google.com>
Tested-by: Jean-Christophe Guillain <jean-christophe@guillain.net>
Tested-by: Naveen N Rao (AMD) <naveen@kernel.org>
Message-ID: <20260921174445.911676-2-seanjc@google.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
|
|
All existing architectures which implement KVM pre-fault (x86, s390) have
mechanisms for disallowing pre-faulting.
Currently these are open coded as part of kvm_arch_vcpu_pre_fault_memory().
Formalise this by moving them into a new kvm_arch_pre_fault_allowed() hook,
which every architecture selecting CONFIG_KVM_GENERIC_PRE_FAULT_MEMORY must
implement, returning an error code if the operation is disallowed or 0
otherwise.
The hook is called early in the generic code, allowing architectures to
disallow the operation prior to vCPU load.
This is important, as kvm_vcpu_pre_fault_memory() is the only place where
generic code can call vcpu_load() on a vCPU that has not yet been
initialised.
This lays the foundation for a future change which implements pre-faulting
for arm64 which needs to disallow the mechanism for uninitialised vCPUs.
Suggested-by: Oliver Upton <oupton@kernel.org>
Signed-off-by: Lorenzo Stoakes (ARM) <ljs@kernel.org>
Acked-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Fuad Tabba <fuad.tabba@linux.dev>
Tested-by: Fuad Tabba <fuad.tabba@linux.dev>
Link: https://patch.msgid.link/20260923-kvm-arm-prefault-v4-1-d4b0b4dfa8c3@kernel.org
Signed-off-by: Marc Zyngier <maz@kernel.org>
|
|
Convert to const struct mnt_idmap.
A mount's idmapping is immutable. The only thing that is allowed to be
modified afterwards is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only ever reads
from the idmapping. This is the same model that struct cred uses and the
idmapping is also rather sensitive.
So make the idmap argument const wherever we can. The conversion is done
from the bottom up so callers can continue to pass a non-const pointer
to a const parameter until the conversion is finished.
No functional changes.
Link: https://patch.msgid.link/20260901-work-idmap-const-v1-24-54ccd48e100b@kernel.org
Acked-by: Paul Moore <paul@paul-moore.com>
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewed-by: Seth Forshee <sforshee@kernel.org>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
|
|
Skip the xarray reservation loop when clearing all memory attributes, as
storing NULL only erases the entry and never needs to allocate, so no
reservation (and no cleanup of a failed one) is required in that case.
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: split to separate patch]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
Explicitly instantiate the memory attributes xarray with XA_FLAGS_ACCOUNT
to ensure that all allocations are accounted to the memcg. Frustratingly,
memory allocations done in the "fastpath" do not honor the passed in gfp,
even for an explicit xa_reserve(). Only the rare, slow path __xas_nomem()
honors the original gfp. E.g.
xa_reserve(..., GFP_KERNEL_ACCOUNT)
|
-> ...
|
-> __xa_cmpxchg_raw()
|
-> xas_store() <== does not take @gfp
|
-> xas_create()
|
-> xas_alloc()
The bug was confirmed by observing that a process in a cgroup limited to
256 MiB grew radix_tree_node slab by ~512 MiB while its memory.current
stayed near 0.
Fixes: 5a475554db1e ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Assisted-by: Claude-Code:claude-opus-5
Signed-off-by: David Ballesteros <davimaba.v@proton.me>
Link: https://patch.msgid.link/20260915175335.138547-4-davimaba.v@proton.me
[sean: rewrite changelog, tag for stable]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
|
kvm_vm_set_mem_attributes() reserves an xarray entry for every gfn in
the range before storing the new attributes, so that the store loop
can't fail partway through. If one of the reservations fails, e.g. with
-ENOMEM, the entries that were already reserved are left in the array.
That is harmless as far as xa_reserve() is concerned, as the reserved
entries read back as NULL via xa_load(), but it confuses the "does this
range have no attributes at all" check:
if (!attrs)
return !xas_find(&xas, end - 1);
A reserved entry is XA_ZERO_ENTRY, not NULL, and xas_find() returns it
as present. So a leftover reservation makes KVM report that a fully
shared range has attributes even though kvm_get_memory_attributes()
returns none for every gfn in the range. On x86, the next time
mixed-attribute tracking is recomputed for the range (memslot creation,
or a later attribute change that straddles the 2MiB page),
hugepage_has_attrs() treats a fully shared 2MiB range as mixed and
refuses to map it with a hugepage, until userspace happens to set
attributes on the range again.
Drop the shortcut and handle the !attrs case in the per-index loop,
using xas_next_entry() to find the next non-NULL entry. xas_next_entry()
is essentially an optimized xas_find(), so the effective change is that
the !attrs lookup now goes through xas_retry() like the attrs != 0 case,
i.e. reserved entries are skipped and retry entries restart the walk.
Don't check the index when no entry is found, as the xarray leaves the
xas index in a bogus state in that case; no entry simply means the rest
of the range has no attributes.
KVM never stores a non-NULL entry with a value of zero (clearing stores
NULL), but such an entry would be returned by xas_next_entry() and trip
the index check, so WARN if one is ever seen.
Fixes: 5a475554db1e ("KVM: Introduce per-page memory attributes")
Cc: stable@vger.kernel.org
Suggested-by: Sean Christopherson <seanjc@google.com>
Cc: David Ballesteros <davimaba.v@proton.me>
Signed-off-by: Zeng Chi <zengchi@kylinos.cn>
Link: https://patch.msgid.link/20260921102442.1232375-1-zeng_chi911@163.com
[sean: expand comment to elaborate on xarray APIs, split optimization out]
Signed-off-by: Sean Christopherson <seanjc@google.com>
|
