Merge tag 'rust-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux

Pull Rust updates from Miguel Ojeda:
 "Toolchain and infrastructure:

   - Extract the 'pin-init' API from the 'kernel' crate and make it into
     a standalone crate.

     In order to do this, the contents are rearranged so that they can
     easily be kept in sync with the version maintained out-of-tree that
     other projects have started to use too (or plan to, like QEMU).

     This will reduce the maintenance burden for Benno, who will now
     have his own sub-tree, and will simplify future expected changes
     like the move to use 'syn' to simplify the implementation.

   - Add '#[test]'-like support based on KUnit.

     We already had doctests support based on KUnit, which takes the
     examples in our Rust documentation and runs them under KUnit.

     Now, we are adding the beginning of the support for "normal" tests,
     similar to those the '#[test]' tests in userspace Rust. For
     instance:

         #[kunit_tests(my_suite)]
         mod tests {
             #[test]
             fn my_test() {
                 assert_eq!(1 + 1, 2);
             }
         }

     Unlike with doctests, the 'assert*!'s do not map to the KUnit
     assertion APIs yet.

   - Check Rust signatures at compile time for functions called from C
     by name.

     In particular, introduce a new '#[export]' macro that can be placed
     in the Rust function definition. It will ensure that the function
     declaration on the C side matches the signature on the Rust
     function:

         #[export]
         pub unsafe extern "C" fn my_function(a: u8, b: i32) -> usize {
             // ...
         }

     The macro essentially forces the compiler to compare the types of
     the actual Rust function and the 'bindgen'-processed C signature.

     These cases are rare so far. In the future, we may consider
     introducing another tool, 'cbindgen', to generate C headers
     automatically. Even then, having these functions explicitly marked
     may be a good idea anyway.

   - Enable the 'raw_ref_op' Rust feature: it is already stable, and
     allows us to use the new '&raw' syntax, avoiding a couple macros.
     After everyone has migrated, we will disallow the macros.

   - Pass the correct target to 'bindgen' on Usermode Linux.

   - Fix 'rusttest' build in macOS.

  'kernel' crate:

   - New 'hrtimer' module: add support for setting up intrusive timers
     without allocating when starting the timer. Add support for
     'Pin<Box<_>>', 'Arc<_>', 'Pin<&_>' and 'Pin<&mut _>' as pointer
     types for use with timer callbacks. Add support for setting clock
     source and timer mode.

   - New 'dma' module: add a simple DMA coherent allocator abstraction
     and a test sample driver.

   - 'list' module: make the linked list 'Cursor' point between
     elements, rather than at an element, which is more convenient to us
     and allows for cursors to empty lists; and document it with
     examples of how to perform common operations with the provided
     methods.

   - 'str' module: implement a few traits for 'BStr' as well as the
     'strip_prefix()' method.

   - 'sync' module: add 'Arc::as_ptr'.

   - 'alloc' module: add 'Box::into_pin'.

   - 'error' module: extend the 'Result' documentation, including a few
     examples on different ways of handling errors, a warning about
     using methods that may panic, and links to external documentation.

  'macros' crate:

   - 'module' macro: add the 'authors' key to support multiple authors.
     The original key will be kept until everyone has migrated.

  Documentation:

   - Add error handling sections.

  MAINTAINERS:

   - Add Danilo Krummrich as reviewer of the Rust "subsystem".

   - Add 'RUST [PIN-INIT]' entry with Benno Lossin as maintainer. It has
     its own sub-tree.

   - Add sub-tree for 'RUST [ALLOC]'.

   - Add 'DMA MAPPING HELPERS DEVICE DRIVER API [RUST]' entry with
     Abdiel Janulgue as primary maintainer. It will go through the
     sub-tree of the 'RUST [ALLOC]' entry.

   - Add 'HIGH-RESOLUTION TIMERS [RUST]' entry with Andreas Hindborg as
     maintainer. It has its own sub-tree.

  And a few other cleanups and improvements"

* tag 'rust-6.15' of git://git.kernel.org/pub/scm/linux/kernel/git/ojeda/linux: (71 commits)
  rust: dma: add `Send` implementation for `CoherentAllocation`
  rust: macros: fix `make rusttest` build on macOS
  rust: block: refactor to use `&raw mut`
  rust: enable `raw_ref_op` feature
  rust: uaccess: name the correct function
  rust: rbtree: fix comments referring to Box instead of KBox
  rust: hrtimer: add maintainer entry
  rust: hrtimer: add clocksource selection through `ClockId`
  rust: hrtimer: add `HrTimerMode`
  rust: hrtimer: implement `HrTimerPointer` for `Pin<Box<T>>`
  rust: alloc: add `Box::into_pin`
  rust: hrtimer: implement `UnsafeHrTimerPointer` for `Pin<&mut T>`
  rust: hrtimer: implement `UnsafeHrTimerPointer` for `Pin<&T>`
  rust: hrtimer: add `hrtimer::ScopedHrTimerPointer`
  rust: hrtimer: add `UnsafeHrTimerPointer`
  rust: hrtimer: allow timer restart from timer handler
  rust: str: implement `strip_prefix` for `BStr`
  rust: str: implement `AsRef<BStr>` for `[u8]` and `BStr`
  rust: str: implement `Index` for `BStr`
  rust: str: implement `PartialEq` for `BStr`
  ...

1 files changed, 148 insertions, 1299 deletions

diff --git a/rust/kernel/init.rs b/rust/kernel/init.rs
index e25d047f3c82..8d228c237954 100644
--- a/rust/kernel/init.rs
+++ b/rust/kernel/init.rs
@@ -1,131 +1,77 @@
-// SPDX-License-Identifier: Apache-2.0 OR MIT
+// SPDX-License-Identifier: GPL-2.0
 
-//! API to safely and fallibly initialize pinned `struct`s using in-place constructors.
-//!
-//! It also allows in-place initialization of big `struct`s that would otherwise produce a stack
-//! overflow.
+//! Extensions to the [`pin-init`] crate.
 //!
 //! Most `struct`s from the [`sync`] module need to be pinned, because they contain self-referential
 //! `struct`s from C. [Pinning][pinning] is Rust's way of ensuring data does not move.
 //!
-//! # Overview
+//! The [`pin-init`] crate is the way such structs are initialized on the Rust side. Please refer
+//! to its documentation to better understand how to use it. Additionally, there are many examples
+//! throughout the kernel, such as the types from the [`sync`] module. And the ones presented
+//! below.
+//!
+//! [`sync`]: crate::sync
+//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
+//! [`pin-init`]: https://rust.docs.kernel.org/pin_init/
 //!
-//! To initialize a `struct` with an in-place constructor you will need two things:
-//! - an in-place constructor,
-//! - a memory location that can hold your `struct` (this can be the [stack], an [`Arc<T>`],
-//!   [`UniqueArc<T>`], [`KBox<T>`] or any other smart pointer that implements [`InPlaceInit`]).
+//! # [`Opaque<T>`]
 //!
-//! To get an in-place constructor there are generally three options:
-//! - directly creating an in-place constructor using the [`pin_init!`] macro,
-//! - a custom function/macro returning an in-place constructor provided by someone else,
-//! - using the unsafe function [`pin_init_from_closure()`] to manually create an initializer.
+//! For the special case where initializing a field is a single FFI-function call that cannot fail,
+//! there exist the helper function [`Opaque::ffi_init`]. This function initialize a single
+//! [`Opaque<T>`] field by just delegating to the supplied closure. You can use these in
+//! combination with [`pin_init!`].
 //!
-//! Aside from pinned initialization, this API also supports in-place construction without pinning,
-//! the macros/types/functions are generally named like the pinned variants without the `pin`
-//! prefix.
+//! [`Opaque<T>`]: crate::types::Opaque
+//! [`Opaque::ffi_init`]: crate::types::Opaque::ffi_init
+//! [`pin_init!`]: pin_init::pin_init
 //!
 //! # Examples
 //!
-//! ## Using the [`pin_init!`] macro
+//! ## General Examples
 //!
-//! If you want to use [`PinInit`], then you will have to annotate your `struct` with
-//! `#[`[`pin_data`]`]`. It is a macro that uses `#[pin]` as a marker for
-//! [structurally pinned fields]. After doing this, you can then create an in-place constructor via
-//! [`pin_init!`]. The syntax is almost the same as normal `struct` initializers. The difference is
-//! that you need to write `<-` instead of `:` for fields that you want to initialize in-place.
+//! ```rust,ignore
+//! # #![allow(clippy::disallowed_names)]
+//! use kernel::types::Opaque;
+//! use pin_init::pin_init_from_closure;
 //!
-//! ```rust
-//! # #![expect(clippy::disallowed_names)]
-//! use kernel::sync::{new_mutex, Mutex};
-//! # use core::pin::Pin;
-//! #[pin_data]
-//! struct Foo {
-//!     #[pin]
-//!     a: Mutex<usize>,
-//!     b: u32,
+//! // assume we have some `raw_foo` type in C:
+//! #[repr(C)]
+//! struct RawFoo([u8; 16]);
+//! extern {
+//!     fn init_foo(_: *mut RawFoo);
 //! }
 //!
-//! let foo = pin_init!(Foo {
-//!     a <- new_mutex!(42, "Foo::a"),
-//!     b: 24,
-//! });
-//! ```
-//!
-//! `foo` now is of the type [`impl PinInit<Foo>`]. We can now use any smart pointer that we like
-//! (or just the stack) to actually initialize a `Foo`:
-//!
-//! ```rust
-//! # #![expect(clippy::disallowed_names)]
-//! # use kernel::sync::{new_mutex, Mutex};
-//! # use core::pin::Pin;
-//! # #[pin_data]
-//! # struct Foo {
-//! #     #[pin]
-//! #     a: Mutex<usize>,
-//! #     b: u32,
-//! # }
-//! # let foo = pin_init!(Foo {
-//! #     a <- new_mutex!(42, "Foo::a"),
-//! #     b: 24,
-//! # });
-//! let foo: Result<Pin<KBox<Foo>>> = KBox::pin_init(foo, GFP_KERNEL);
-//! ```
-//!
-//! For more information see the [`pin_init!`] macro.
-//!
-//! ## Using a custom function/macro that returns an initializer
-//!
-//! Many types from the kernel supply a function/macro that returns an initializer, because the
-//! above method only works for types where you can access the fields.
-//!
-//! ```rust
-//! # use kernel::sync::{new_mutex, Arc, Mutex};
-//! let mtx: Result<Arc<Mutex<usize>>> =
-//!     Arc::pin_init(new_mutex!(42, "example::mtx"), GFP_KERNEL);
-//! ```
-//!
-//! To declare an init macro/function you just return an [`impl PinInit<T, E>`]:
-//!
-//! ```rust
-//! # use kernel::{sync::Mutex, new_mutex, init::PinInit, try_pin_init};
 //! #[pin_data]
-//! struct DriverData {
+//! struct Foo {
 //!     #[pin]
-//!     status: Mutex<i32>,
-//!     buffer: KBox<[u8; 1_000_000]>,
+//!     raw: Opaque<RawFoo>,
 //! }
 //!
-//! impl DriverData {
-//!     fn new() -> impl PinInit<Self, Error> {
-//!         try_pin_init!(Self {
-//!             status <- new_mutex!(0, "DriverData::status"),
-//!             buffer: KBox::init(kernel::init::zeroed(), GFP_KERNEL)?,
-//!         })
+//! impl Foo {
+//!     fn setup(self: Pin<&mut Self>) {
+//!         pr_info!("Setting up foo\n");
 //!     }
 //! }
-//! ```
 //!
-//! ## Manual creation of an initializer
-//!
-//! Often when working with primitives the previous approaches are not sufficient. That is where
-//! [`pin_init_from_closure()`] comes in. This `unsafe` function allows you to create a
-//! [`impl PinInit<T, E>`] directly from a closure. Of course you have to ensure that the closure
-//! actually does the initialization in the correct way. Here are the things to look out for
-//! (we are calling the parameter to the closure `slot`):
-//! - when the closure returns `Ok(())`, then it has completed the initialization successfully, so
-//!   `slot` now contains a valid bit pattern for the type `T`,
-//! - when the closure returns `Err(e)`, then the caller may deallocate the memory at `slot`, so
-//!   you need to take care to clean up anything if your initialization fails mid-way,
-//! - you may assume that `slot` will stay pinned even after the closure returns until `drop` of
-//!   `slot` gets called.
+//! let foo = pin_init!(Foo {
+//!     raw <- unsafe {
+//!         Opaque::ffi_init(|s| {
+//!             // note that this cannot fail.
+//!             init_foo(s);
+//!         })
+//!     },
+//! }).pin_chain(|foo| {
+//!     foo.setup();
+//!     Ok(())
+//! });
+//! ```
 //!
-//! ```rust
-//! # #![expect(unreachable_pub, clippy::disallowed_names)]
-//! use kernel::{init, types::Opaque};
+//! ```rust,ignore
+//! # #![allow(unreachable_pub, clippy::disallowed_names)]
+//! use kernel::{prelude::*, types::Opaque};
 //! use core::{ptr::addr_of_mut, marker::PhantomPinned, pin::Pin};
 //! # mod bindings {
-//! #     #![expect(non_camel_case_types)]
-//! #     #![expect(clippy::missing_safety_doc)]
+//! #     #![allow(non_camel_case_types)]
 //! #     pub struct foo;
 //! #     pub unsafe fn init_foo(_ptr: *mut foo) {}
 //! #     pub unsafe fn destroy_foo(_ptr: *mut foo) {}
@@ -133,7 +79,7 @@
 //! # }
 //! # // `Error::from_errno` is `pub(crate)` in the `kernel` crate, thus provide a workaround.
 //! # trait FromErrno {
-//! #     fn from_errno(errno: kernel::ffi::c_int) -> Error {
+//! #     fn from_errno(errno: core::ffi::c_int) -> Error {
 //! #         // Dummy error that can be constructed outside the `kernel` crate.
 //! #         Error::from(core::fmt::Error)
 //! #     }
@@ -157,7 +103,7 @@
 //!         //   enabled `foo`,
 //!         // - when it returns `Err(e)`, then it has cleaned up before
 //!         unsafe {
-//!             init::pin_init_from_closure(move |slot: *mut Self| {
+//!             pin_init::pin_init_from_closure(move |slot: *mut Self| {
 //!                 // `slot` contains uninit memory, avoid creating a reference.
 //!                 let foo = addr_of_mut!((*slot).foo);
 //!
@@ -187,401 +133,114 @@
 //!     }
 //! }
 //! ```
-//!
-//! For the special case where initializing a field is a single FFI-function call that cannot fail,
-//! there exist the helper function [`Opaque::ffi_init`]. This function initialize a single
-//! [`Opaque`] field by just delegating to the supplied closure. You can use these in combination
-//! with [`pin_init!`].
-//!
-//! For more information on how to use [`pin_init_from_closure()`], take a look at the uses inside
-//! the `kernel` crate. The [`sync`] module is a good starting point.
-//!
-//! [`sync`]: kernel::sync
-//! [pinning]: https://doc.rust-lang.org/std/pin/index.html
-//! [structurally pinned fields]:
-//!     https://doc.rust-lang.org/std/pin/index.html#pinning-is-structural-for-field
-//! [stack]: crate::stack_pin_init
-//! [`Arc<T>`]: crate::sync::Arc
-//! [`impl PinInit<Foo>`]: PinInit
-//! [`impl PinInit<T, E>`]: PinInit
-//! [`impl Init<T, E>`]: Init
-//! [`Opaque`]: kernel::types::Opaque
-//! [`Opaque::ffi_init`]: kernel::types::Opaque::ffi_init
-//! [`pin_data`]: ::macros::pin_data
-//! [`pin_init!`]: crate::pin_init!
 
 use crate::{
-    alloc::{AllocError, Flags, KBox},
+    alloc::{AllocError, Flags},
     error::{self, Error},
-    sync::Arc,
-    sync::UniqueArc,
-    types::{Opaque, ScopeGuard},
-};
-use core::{
-    cell::UnsafeCell,
-    convert::Infallible,
-    marker::PhantomData,
-    mem::MaybeUninit,
-    num::*,
-    pin::Pin,
-    ptr::{self, NonNull},
 };
+use pin_init::{init_from_closure, pin_init_from_closure, Init, PinInit};
 
-#[doc(hidden)]
-pub mod __internal;
-#[doc(hidden)]
-pub mod macros;
+/// Smart pointer that can initialize memory in-place.
+pub trait InPlaceInit<T>: Sized {
+    /// Pinned version of `Self`.
+    ///
+    /// If a type already implicitly pins its pointee, `Pin<Self>` is unnecessary. In this case use
+    /// `Self`, otherwise just use `Pin<Self>`.
+    type PinnedSelf;
 
-/// Initialize and pin a type directly on the stack.
-///
-/// # Examples
-///
-/// ```rust
-/// # #![expect(clippy::disallowed_names)]
-/// # use kernel::{init, macros::pin_data, pin_init, stack_pin_init, init::*, sync::Mutex, new_mutex};
-/// # use core::pin::Pin;
-/// #[pin_data]
-/// struct Foo {
-///     #[pin]
-///     a: Mutex<usize>,
-///     b: Bar,
-/// }
-///
-/// #[pin_data]
-/// struct Bar {
-///     x: u32,
-/// }
-///
-/// stack_pin_init!(let foo = pin_init!(Foo {
-///     a <- new_mutex!(42),
-///     b: Bar {
-///         x: 64,
-///     },
-/// }));
-/// let foo: Pin<&mut Foo> = foo;
-/// pr_info!("a: {}\n", &*foo.a.lock());
-/// ```
-///
-/// # Syntax
-///
-/// A normal `let` binding with optional type annotation. The expression is expected to implement
-/// [`PinInit`]/[`Init`] with the error type [`Infallible`]. If you want to use a different error
-/// type, then use [`stack_try_pin_init!`].
-///
-/// [`stack_try_pin_init!`]: crate::stack_try_pin_init!
-#[macro_export]
-macro_rules! stack_pin_init {
-    (let $var:ident $(: $t:ty)? = $val:expr) => {
-        let val = $val;
-        let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
-        let mut $var = match $crate::init::__internal::StackInit::init($var, val) {
-            Ok(res) => res,
-            Err(x) => {
-                let x: ::core::convert::Infallible = x;
-                match x {}
-            }
+    /// Use the given pin-initializer to pin-initialize a `T` inside of a new smart pointer of this
+    /// type.
+    ///
+    /// If `T: !Unpin` it will not be able to move afterwards.
+    fn try_pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> Result<Self::PinnedSelf, E>
+    where
+        E: From<AllocError>;
+
+    /// Use the given pin-initializer to pin-initialize a `T` inside of a new smart pointer of this
+    /// type.
+    ///
+    /// If `T: !Unpin` it will not be able to move afterwards.
+    fn pin_init<E>(init: impl PinInit<T, E>, flags: Flags) -> error::Result<Self::PinnedSelf>
+    where
+        Error: From<E>,
+    {
+        // SAFETY: We delegate to `init` and only change the error type.
+        let init = unsafe {
+            pin_init_from_closure(|slot| init.__pinned_init(slot).map_err(|e| Error::from(e)))
         };
-    };
-}
+        Self::try_pin_init(init, flags)
+    }
 
-/// Initialize and pin a type directly on the stack.
-///
-/// # Examples
-///
-/// ```rust,ignore
-/// # #![expect(clippy::disallowed_names)]
-/// # use kernel::{
-/// #     init,
-/// #     pin_init,
-/// #     stack_try_pin_init,
-/// #     init::*,
-/// #     sync::Mutex,
-/// #     new_mutex,
-/// #     alloc::AllocError,
-/// # };
-/// # use macros::pin_data;
-/// # use core::pin::Pin;
-/// #[pin_data]
-/// struct Foo {
-///     #[pin]
-///     a: Mutex<usize>,
-///     b: KBox<Bar>,
-/// }
-///
-/// struct Bar {
-///     x: u32,
-/// }
-///
-/// stack_try_pin_init!(let foo: Result<Pin<&mut Foo>, AllocError> = pin_init!(Foo {
-///     a <- new_mutex!(42),
-///     b: KBox::new(Bar {
-///         x: 64,
-///     }, GFP_KERNEL)?,
-/// }));
-/// let foo = foo.unwrap();
-/// pr_info!("a: {}\n", &*foo.a.lock());
-/// ```
-///
-/// ```rust,ignore
-/// # #![expect(clippy::disallowed_names)]
-/// # use kernel::{
-/// #     init,
-/// #     pin_init,
-/// #     stack_try_pin_init,
-/// #     init::*,
-/// #     sync::Mutex,
-/// #     new_mutex,
-/// #     alloc::AllocError,
-/// # };
-/// # use macros::pin_data;
-/// # use core::pin::Pin;
-/// #[pin_data]
-/// struct Foo {
-///     #[pin]
-///     a: Mutex<usize>,
-///     b: KBox<Bar>,
-/// }
-///
-/// struct Bar {
-///     x: u32,
-/// }
-///
-/// stack_try_pin_init!(let foo: Pin<&mut Foo> =? pin_init!(Foo {
-///     a <- new_mutex!(42),
-///     b: KBox::new(Bar {
-///         x: 64,
-///     }, GFP_KERNEL)?,
-/// }));
-/// pr_info!("a: {}\n", &*foo.a.lock());
-/// # Ok::<_, AllocError>(())
-/// ```
-///
-/// # Syntax
-///
-/// A normal `let` binding with optional type annotation. The expression is expected to implement
-/// [`PinInit`]/[`Init`]. This macro assigns a result to the given variable, adding a `?` after the
-/// `=` will propagate this error.
-#[macro_export]
-macro_rules! stack_try_pin_init {
-    (let $var:ident $(: $t:ty)? = $val:expr) => {
-        let val = $val;
-        let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
-        let mut $var = $crate::init::__internal::StackInit::init($var, val);
-    };
-    (let $var:ident $(: $t:ty)? =? $val:expr) => {
-        let val = $val;
-        let mut $var = ::core::pin::pin!($crate::init::__internal::StackInit$(::<$t>)?::uninit());
-        let mut $var = $crate::init::__internal::StackInit::init($var, val)?;
-    };
+    /// Use the given initializer to in-place initialize a `T`.
+    fn try_init<E>(init: impl Init<T, E>, flags: Flags) -> Result<Self, E>
+    where
+        E: From<AllocError>;
+
+    /// Use the given initializer to in-place initialize a `T`.
+    fn init<E>(init: impl Init<T, E>, flags: Flags) -> error::Result<Self>
+    where
+        Error: From<E>,
+    {
+        // SAFETY: We delegate to `init` and only change the error type.
+        let init = unsafe {
+            init_from_closure(|slot| init.__pinned_init(slot).map_err(|e| Error::from(e)))
+        };
+        Self::try_init(init, flags)
+    }
 }
 
-/// Construct an in-place, pinned initializer for `struct`s.
-///
-/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
-/// [`try_pin_init!`].
-///
-/// The syntax is almost identical to that of a normal `struct` initializer:
-///
-/// ```rust
-/// # use kernel::{init, pin_init, macros::pin_data, init::*};
-/// # use core::pin::Pin;
-/// #[pin_data]
-/// struct Foo {
-///     a: usize,
-///     b: Bar,
-/// }
-///
-/// #[pin_data]
-/// struct Bar {
-///     x: u32,
-/// }
-///
-/// # fn demo() -> impl PinInit<Foo> {
-/// let a = 42;
-///
-/// let initializer = pin_init!(Foo {
-///     a,
-///     b: Bar {
-///         x: 64,
-///     },
-/// });
-/// # initializer }
-/// # KBox::pin_init(demo(), GFP_KERNEL).unwrap();
-/// ```
-///
-/// Arbitrary Rust expressions can be used to set the value of a variable.
-///
-/// The fields are initialized in the order that they appear in the initializer. So it is possible
-/// to read already initialized fields using raw pointers.
-///
-/// IMPORTANT: You are not allowed to create references to fields of the struct inside of the
-/// initializer.
-///
-/// # Init-functions
-///
-/// When working with this API it is often desired to let others construct your types without
-/// giving access to all fields. This is where you would normally write a plain function `new`
-/// that would return a new instance of your type. With this API that is also possible.
-/// However, there are a few extra things to keep in mind.
-///
-/// To create an initializer function, simply declare it like this:
-///
-/// ```rust
-/// # use kernel::{init, pin_init, init::*};
-/// # use core::pin::Pin;
-/// # #[pin_data]
-/// # struct Foo {
-/// #     a: usize,
-/// #     b: Bar,
-/// # }
-/// # #[pin_data]
-/// # struct Bar {
-/// #     x: u32,
-/// # }
-/// impl Foo {
-///     fn new() -> impl PinInit<Self> {
-///         pin_init!(Self {
-///             a: 42,
-///             b: Bar {
-///                 x: 64,
-///             },
-///         })
-///     }
-/// }
-/// ```
+/// Construct an in-place fallible initializer for `struct`s.
 ///
-/// Users of `Foo` can now create it like this:
+/// This macro defaults the error to [`Error`]. If you need [`Infallible`], then use
+/// [`init!`].
 ///
-/// ```rust
-/// # #![expect(clippy::disallowed_names)]
-/// # use kernel::{init, pin_init, macros::pin_data, init::*};
-/// # use core::pin::Pin;
-/// # #[pin_data]
-/// # struct Foo {
-/// #     a: usize,
-/// #     b: Bar,
-/// # }
-/// # #[pin_data]
-/// # struct Bar {
-/// #     x: u32,
-/// # }
-/// # impl Foo {
-/// #     fn new() -> impl PinInit<Self> {
-/// #         pin_init!(Self {
-/// #             a: 42,
-/// #             b: Bar {
-/// #                 x: 64,
-/// #             },
-/// #         })
-/// #     }
-/// # }
-/// let foo = KBox::pin_init(Foo::new(), GFP_KERNEL);
-/// ```
+/// The syntax is identical to [`try_pin_init!`]. If you want to specify a custom error,
+/// append `? $type` after the `struct` initializer.
+/// The safety caveats from [`try_pin_init!`] also apply:
+/// - `unsafe` code must guarantee either full initialization or return an error and allow
+///   deallocation of the memory.
+/// - the fields are initialized in the order given in the initializer.
+/// - no references to fields are allowed to be created inside of the initializer.
 ///
-/// They can also easily embed it into their own `struct`s:
+/// # Examples
 ///
 /// ```rust
-/// # use kernel::{init, pin_init, macros::pin_data, init::*};
-/// # use core::pin::Pin;
-/// # #[pin_data]
-/// # struct Foo {
-/// #     a: usize,
-/// #     b: Bar,
-/// # }
-/// # #[pin_data]
-/// # struct Bar {
-/// #     x: u32,
-/// # }
-/// # impl Foo {
-/// #     fn new() -> impl PinInit<Self> {
-/// #         pin_init!(Self {
-/// #             a: 42,
-/// #             b: Bar {
-/// #                 x: 64,
-/// #             },
-/// #         })
-/// #     }
-/// # }
-/// #[pin_data]
-/// struct FooContainer {
-///     #[pin]
-///     foo1: Foo,
-///     #[pin]
-///     foo2: Foo,
-///     other: u32,
+/// use kernel::error::Error;
+/// use pin_init::zeroed;
+/// struct BigBuf {
+///     big: KBox<[u8; 1024 * 1024 * 1024]>,
+///     small: [u8; 1024 * 1024],
 /// }
 ///
-/// impl FooContainer {
-///     fn new(other: u32) -> impl PinInit<Self> {
-///         pin_init!(Self {
-///             foo1 <- Foo::new(),
-///             foo2 <- Foo::new(),
-///             other,
-///         })
+/// impl BigBuf {
+///     fn new() -> impl Init<Self, Error> {
+///         try_init!(Self {
+///             big: KBox::init(zeroed(), GFP_KERNEL)?,
+///             small: [0; 1024 * 1024],
+///         }? Error)
 ///     }
 /// }
 /// ```
 ///
-/// Here we see that when using `pin_init!` with `PinInit`, one needs to write `<-` instead of `:`.
-/// This signifies that the given field is initialized in-place. As with `struct` initializers, just
-/// writing the field (in this case `other`) without `:` or `<-` means `other: other,`.
-///
-/// # Syntax
-///
-/// As already mentioned in the examples above, inside of `pin_init!` a `struct` initializer with
-/// the following modifications is expected:
-/// - Fields that you want to initialize in-place have to use `<-` instead of `:`.
-/// - In front of the initializer you can write `&this in` to have access to a [`NonNull<Self>`]
-///   pointer named `this` inside of the initializer.
-/// - Using struct update syntax one can place `..Zeroable::zeroed()` at the very end of the
-///   struct, this initializes every field with 0 and then runs all initializers specified in the
-///   body. This can only be done if [`Zeroable`] is implemented for the struct.
-///
-/// For instance:
-///
-/// ```rust
-/// # use kernel::{macros::{Zeroable, pin_data}, pin_init};
-/// # use core::{ptr::addr_of_mut, marker::PhantomPinned};
-/// #[pin_data]
-/// #[derive(Zeroable)]
-/// struct Buf {
-///     // `ptr` points into `buf`.
-///     ptr: *mut u8,
-///     buf: [u8; 64],
-///     #[pin]
-///     pin: PhantomPinned,
-/// }
-/// pin_init!(&this in Buf {
-///     buf: [0; 64],
-///     // SAFETY: TODO.
-///     ptr: unsafe { addr_of_mut!((*this.as_ptr()).buf).cast() },
-///     pin: PhantomPinned,
-/// });
-/// pin_init!(Buf {
-///     buf: [1; 64],
-///     ..Zeroable::zeroed()
-/// });
-/// ```
-///
-/// [`try_pin_init!`]: kernel::try_pin_init
-/// [`NonNull<Self>`]: core::ptr::NonNull
-// For a detailed example of how this macro works, see the module documentation of the hidden
-// module `__internal` inside of `init/__internal.rs`.
+/// [`Infallible`]: core::convert::Infallible
+/// [`init!`]: pin_init::init
+/// [`try_pin_init!`]: crate::try_pin_init!
+/// [`Error`]: crate::error::Error
 #[macro_export]
-macro_rules! pin_init {
+macro_rules! try_init {
     ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
         $($fields:tt)*
     }) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)?),
-            @fields($($fields)*),
-            @error(::core::convert::Infallible),
-            @data(PinData, use_data),
-            @has_data(HasPinData, __pin_data),
-            @construct_closure(pin_init_from_closure),
-            @munch_fields($($fields)*),
-        )
+        ::pin_init::try_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
+            $($fields)*
+        }? $crate::error::Error)
+    };
+    ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
+        $($fields:tt)*
+    }? $err:ty) => {
+        ::pin_init::try_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
+            $($fields)*
+        }? $err)
     };
 }
 
@@ -603,7 +262,9 @@ macro_rules! pin_init {
 /// # Examples
 ///
 /// ```rust
-/// use kernel::{init::{self, PinInit}, error::Error};
+/// # #![feature(new_uninit)]
+/// use kernel::error::Error;
+/// use pin_init::zeroed;
 /// #[pin_data]
 /// struct BigBuf {
 ///     big: KBox<[u8; 1024 * 1024 * 1024]>,
@@ -614,843 +275,31 @@ macro_rules! pin_init {
 /// impl BigBuf {
 ///     fn new() -> impl PinInit<Self, Error> {
 ///         try_pin_init!(Self {
-///             big: KBox::init(init::zeroed(), GFP_KERNEL)?,
+///             big: KBox::init(zeroed(), GFP_KERNEL)?,
 ///             small: [0; 1024 * 1024],
 ///             ptr: core::ptr::null_mut(),
 ///         }? Error)
 ///     }
 /// }
 /// ```
-// For a detailed example of how this macro works, see the module documentation of the hidden
-// module `__internal` inside of `init/__internal.rs`.
-#[macro_export]
-macro_rules! try_pin_init {
-    ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
-        $($fields:tt)*
-    }) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)? ),
-            @fields($($fields)*),
-            @error($crate::error::Error),
-            @data(PinData, use_data),
-            @has_data(HasPinData, __pin_data),
-            @construct_closure(pin_init_from_closure),
-            @munch_fields($($fields)*),
-        )
-    };
-    ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
-        $($fields:tt)*
-    }? $err:ty) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)? ),
-            @fields($($fields)*),
-            @error($err),
-            @data(PinData, use_data),
-            @has_data(HasPinData, __pin_data),
-            @construct_closure(pin_init_from_closure),
-            @munch_fields($($fields)*),
-        )
-    };
-}
-
-/// Construct an in-place initializer for `struct`s.
-///
-/// This macro defaults the error to [`Infallible`]. If you need [`Error`], then use
-/// [`try_init!`].
-///
-/// The syntax is identical to [`pin_init!`] and its safety caveats also apply:
-/// - `unsafe` code must guarantee either full initialization or return an error and allow
-///   deallocation of the memory.
-/// - the fields are initialized in the order given in the initializer.
-/// - no references to fields are allowed to be created inside of the initializer.
 ///
-/// This initializer is for initializing data in-place that might later be moved. If you want to
-/// pin-initialize, use [`pin_init!`].
-///
-/// [`try_init!`]: crate::try_init!
-// For a detailed example of how this macro works, see the module documentation of the hidden
-// module `__internal` inside of `init/__internal.rs`.
+/// [`Infallible`]: core::convert::Infallible
+/// [`pin_init!`]: pin_init::pin_init
+/// [`Error`]: crate::error::Error
 #[macro_export]
-macro_rules! init {
-    ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
-        $($fields:tt)*
-    }) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)?),
-            @fields($($fields)*),
-            @error(::core::convert::Infallible),
-            @data(InitData, /*no use_data*/),
-            @has_data(HasInitData, __init_data),
-            @construct_closure(init_from_closure),
-            @munch_fields($($fields)*),
-        )
-    }
-}
-
-/// Construct an in-place fallible initializer for `struct`s.
-///
-/// This macro defaults the error to [`Error`]. If you need [`Infallible`], then use
-/// [`init!`].
-///
-/// The syntax is identical to [`try_pin_init!`]. If you want to specify a custom error,
-/// append `? $type` after the `struct` initializer.
-/// The safety caveats from [`try_pin_init!`] also apply:
-/// - `unsafe` code must guarantee either full initialization or return an error and allow
-///   deallocation of the memory.
-/// - the fields are initialized in the order given in the initializer.
-/// - no references to fields are allowed to be created inside of the initializer.
-///
-/// # Examples
-///
-/// ```rust
-/// use kernel::{alloc::KBox, init::{PinInit, zeroed}, error::Error};
-/// struct BigBuf {
-///     big: KBox<[u8; 1024 * 1024 * 1024]>,
-///     small: [u8; 1024 * 1024],
-/// }
-///
-/// impl BigBuf {
-///     fn new() -> impl Init<Self, Error> {
-///         try_init!(Self {
-///             big: KBox::init(zeroed(), GFP_KERNEL)?,
-///             small: [0; 1024 * 1024],
-///         }? Error)
-///     }
-/// }
-/// ```
-// For a detailed example of how this macro works, see the module documentation of the hidden
-// module `__internal` inside of `init/__internal.rs`.
-#[macro_export]
-macro_rules! try_init {
+macro_rules! try_pin_init {
     ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
         $($fields:tt)*
     }) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)?),
-            @fields($($fields)*),
-            @error($crate::error::Error),
-            @data(InitData, /*no use_data*/),
-            @has_data(HasInitData, __init_data),
-            @construct_closure(init_from_closure),
-            @munch_fields($($fields)*),
-        )
+        ::pin_init::try_pin_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
+            $($fields)*
+        }? $crate::error::Error)
     };
     ($(&$this:ident in)? $t:ident $(::<$($generics:ty),* $(,)?>)? {
         $($fields:tt)*
     }? $err:ty) => {
-        $crate::__init_internal!(
-            @this($($this)?),
-            @typ($t $(::<$($generics),*>)?),
-            @fields($($fields)*),
-            @error($err),
-            @data(InitData, /*no use_data*/),
-            @has_data(HasInitData, __init_data),
-            @construct_closure(init_from_closure),
-            @munch_fields($($fields)*),
-        )
+        ::pin_init::try_pin_init!($(&$this in)? $t $(::<$($generics),* $(,)?>)? {
+            $($fields)*
+        }? $err)
     };
 }
-
-/// Asserts that a field on a struct using `#[pin_data]` is marked with `#[pin]` ie. that it is
-/// structurally pinned.
-///
-/// # Example
-///
-/// This will succeed:
-/// ```
-/// use kernel::assert_pinned;
-/// #[pin_data]
-/// struct MyStruct {
-///     #[pin]
-///     some_field: u64,
-/// }
-///
-/// assert_pinned!(MyStruct, some_field, u64);
-/// ```
-///
-/// This will fail:
-// TODO: replace with `compile_fail` when supported.
-/// ```ignore
-/// use kernel::assert_pinned;
-/// #[pin_data]
-/// struct MyStruct {
-///     some_field: u64,
-/// }
-///
-/// assert_pinned!(MyStruct, some_field, u64);
-/// ```
-///
-/// Some uses of the macro may trigger the `can't use generic parameters from outer item` error. To
-/// work around this, you may pass the `inline` parameter to the macro. The `inline` parameter can
-/// only be used when the macro is invoked from a function body.
-/// ```
-/// use kernel::assert_pinned;
-/// #[pin_data]
-/// struct Foo<T> {
-///     #[pin]
-///     elem: T,
-/// }
-///
-/// impl<T> Foo<T> {
-///     fn project(self: Pin<&mut Self>) -> Pin<&mut T> {
-///         assert_pinned!(Foo<T>, elem, T, inline);
-///
-///         // SAFETY: The field is structurally pinned.
-///         unsafe { self.map_unchecked_mut(|me| &mut me.elem) }
-///     }
-/// }
-/// ```
-#[macro_export]
-macro_rules! assert_pinned {
-    ($ty:t