Repository navigation
userspace: doc: document the overall approach how SOF utilizes Zephyr user-space #11247
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
kv2019i
wants to merge
3
commits into
thesofproject:main
Choose a base branch
from
kv2019i:202609-userll-doc-update
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+371
−0
Open
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| # Architecture Support (`src/arch`) | ||
|
|
||
| > **Runs in:** Kernel-only — CPU-architecture primitives (Xtensa) execute in supervisor context | ||
| > and are never linked into a user thread. See | ||
| > [User/kernel split](../include/sof/userspace/README.md). | ||
| This directory contains architecture-specific code. It sits below the RTOS and audio | ||
| application and has no user-space entry points. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,18 @@ | ||
| # Audio Components (`src/audio`) | ||
|
|
||
| > **Runs in:** Application — under userspace configs (e.g. PTL with `CONFIG_SOF_USERSPACE_LL`), | ||
| > the pipeline, buffers, copiers and processing components in this tree run in MMU-isolated | ||
| > **user** threads. They reach kernel resources only through the SOF syscalls. See | ||
| > [User/kernel split](../include/sof/userspace/README.md). | ||
|
|
||
| This directory holds the SOF audio application: the pipeline graph and scheduling glue, host/DAI | ||
| copiers, and the ~36 processing components/modules (volume, EQ, SRC, mixers, codecs, etc.). This | ||
| is the code the user/kernel split is designed to protect the rest of the firmware from. | ||
|
|
||
| Notable subdirectories carry their own README with a matching "Runs in:" banner, e.g. | ||
| [`module_adapter/`](module_adapter/README.md), which is the boundary container that launches | ||
| modules in user context. | ||
|
|
||
| When adding a component here, keep it inside the application tier: allocate through the module | ||
| allocation syscalls (`mod_*`), mark any globals shared with the kernel using the partition | ||
| markers, and never call kernel-only subsystems (`drivers/`, `platform/`, `idc/`) directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| # Device Drivers (`src/drivers`) | ||
|
|
||
| > **Runs in:** Kernel-only — hardware/HAL access runs in supervisor context and is never | ||
| > executed from a user thread. User-space audio code reaches driver functionality only through | ||
| > SOF syscalls (e.g. the `sof_dma_*` and `dai_get_device` calls). See | ||
| > [User/kernel split](../include/sof/userspace/README.md). | ||
|
|
||
| This directory contains low-level device drivers and hardware abstraction. Because it touches | ||
| MMIO, interrupts and other privileged resources, nothing here is mapped into a user memory | ||
| domain. If user-space code needs a driver operation, expose it through a validated syscall in | ||
| the boundary tier rather than linking this code into a user thread. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| # Inter-Domain / Inter-Core Communication (`src/idc`) | ||
|
|
||
| > **Runs in:** Kernel-only — cross-core messaging (`idc.c`, `zephyr_idc.c`) runs in supervisor | ||
| > context. User-space audio code triggers cross-core work indirectly through higher-level | ||
| > subsystems, not by calling IDC directly. See | ||
| > [User/kernel split](../include/sof/userspace/README.md). | ||
|
|
||
| This directory implements the mechanism cores use to signal and hand work to each other. It is | ||
| part of the OS-integration layer, not the audio application. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,236 @@ | ||
| # SOF User/Kernel Split — Architecture and Conventions | ||
|
|
||
| This is the canonical reference for how Sound Open Firmware (SOF) is divided between | ||
| **kernel (supervisor) context** and **user (unprivileged) context** on top of Zephyr | ||
| userspace. It defines the conventions that make the split evident in the source layout and | ||
| easy to maintain. | ||
|
|
||
| > **Status:** convention proposal. The execution model and the syscall/marker/Kconfig facts | ||
| > below reflect the code as it is today. The *directory taxonomy* and the per-directory | ||
| > "Runs in:" banners are the proposed mechanism for making the split visible; they are being | ||
| > rolled out incrementally. | ||
|
|
||
| ## Why this exists | ||
|
|
||
| SOF supports several privilege configurations. On PTL (`intel_adsp_ace30_ptl`) the board | ||
| config enables `CONFIG_USERSPACE` and `CONFIG_SOF_USERSPACE_PROXY`, running individual audio | ||
| modules in isolated user threads via the proxy container. For Linux builds | ||
| (`os_linux_overlay.conf`), `app/CMakeLists.txt` automatically includes | ||
| `app/overlays/ptl/ll_userspace_overlay.conf`, which additionally enables | ||
| `CONFIG_SOF_USERSPACE_LL` so the entire LL pipeline and the module-facing IPC run in user | ||
| threads. This is the standard PTL Linux build configuration. The kernel keeps the OS/HAL, | ||
| boot, drivers, and the syscall validators in both cases. | ||
|
|
||
| The privilege boundary is real, but nothing in the source *layout* signals which side a given | ||
| file belongs to. This document makes that explicit. | ||
|
|
||
| ## Execution model | ||
|
|
||
| ```mermaid | ||
| graph TD | ||
| subgraph Kernel [Kernel / supervisor context - ring 0] | ||
| Boot[Boot & init] | ||
| Drv[Drivers / HAL] | ||
| Dispatch[IPC dispatch] | ||
| LLk[LL scheduler core + domain setup] | ||
| Vrfy["Syscall validators (z_vrfy_*)"] | ||
| end | ||
|
|
||
| subgraph User [User context - ring 3, MMU-isolated] | ||
| LLu[LL pipeline thread] | ||
| DP[DP module threads] | ||
| Mods[Audio modules / copiers] | ||
| IPCu[Module-facing IPC handler] | ||
| end | ||
|
|
||
| Dispatch -->|forwards cmd| IPCu | ||
| LLk -->|starts| LLu | ||
| LLu -->|"syscalls"| Vrfy | ||
| DP -->|"syscalls"| Vrfy | ||
| IPCu -->|"syscalls"| Vrfy | ||
| Vrfy -->|"z_impl_* (validated)"| Drv | ||
| ``` | ||
|
|
||
| - **Kernel context** owns hardware, boot, interrupt handling, memory-domain construction, and | ||
| the *validating* half of every syscall (`z_vrfy_*`). | ||
| - **User context** runs the audio application: LL pipeline processing, DP modules, copiers, | ||
| and the module-facing IPC path. | ||
| - The two contexts cross the boundary in two ways: | ||
| 1. **Syscalls** — user code calls a kernel service (see | ||
| [The boundary is the syscall set](#the-boundary-is-the-syscall-set)). | ||
| 2. **Kernel-to-user shared-memory handoff** — the kernel writes IPC message words to | ||
| shared `ipc_user` state, signals the user thread via a granted `k_event`, and waits on | ||
| a semaphore for the result (`ipc_user_forward_cmd()` in `ipc/ipc-common.c`). The user | ||
| thread wakes, processes the message, and posts the semaphore back. | ||
|
|
||
| ## Directory taxonomy — where code runs | ||
|
|
||
| Every `src/` subdirectory falls into one of four tiers. This is the primary mechanism for | ||
| making the split evident: the tier tells you where a directory's code runs before you read a | ||
| line of it. Top-level directories carry a **"Runs in:"** banner in their `README.md` to | ||
| declare their tier. The banner rollout is in progress; directories in the table that still | ||
| lack a banner or a README are noted. The `src/include/` tree is not classified here because | ||
| its subdirectories span all tiers. | ||
|
|
||
| | Tier | Meaning | Directories | | ||
| |------|---------|-------------| | ||
| | **Kernel-only** | OS integration, HAL, boot, inter-core. Never executes in a user thread. Must not be reached directly from user code — only via a syscall. | `arch/`, `drivers/`, `idc/`, `init/`, `platform/`, `probe/` *, `logging/` *, `trace/` * | | ||
| | **Boundary / bridge** | Owns the syscalls and the thread / memory-domain setup. Split internally: some translation units run kernel-side, some user-side (e.g. `handler-kernel.c` vs `handler-user.c`, `zephyr_ll.c` vs `zephyr_ll_user.c`). | `schedule/`, `ipc/`, `library_manager/` *, `debug/debug_stream/` *, and (in the Zephyr integration tree) `zephyr/syscall/` | | ||
| | **Application** | Audio pipeline / component logic that runs in user threads under userspace configs. | `audio/` | | ||
| | **Shared library** | Context-agnostic code linked into whichever context calls it. Must stay "user-safe": no privileged operations (no direct MMIO, no spinlocks held across syscalls, no raw kernel object access) so it is safe when pulled into a user build. | `math/`, `module/` | | ||
| | **Mixed** | A directory whose files span more than one tier; see its own README for per-file classification. | `lib/` | | ||
|
|
||
| Directories marked * do not yet have a "Runs in:" README banner. | ||
|
|
||
| Classification is grep-verifiable — see [Verifying the taxonomy](#verifying-the-taxonomy). | ||
|
|
||
| > **`zephyr/syscall/` is the model to grow toward.** The Zephyr integration tree already | ||
| > collects the validating half of several syscalls in one place — `zephyr/syscall/alloc.c`, | ||
| > `cpu.c`, `vregion.c`, `dai.c`, `sof_dma.c` each hold the `z_vrfy_*` wrappers for their | ||
| > subsystem, physically separated from the `z_impl_*` logic (in `zephyr/lib/*.c` and | ||
| > `src/lib/*.c`). This is exactly the boundary-tier separation this document promotes; the | ||
| > migration opportunity is to move the *still-scattered* validators (IPC, module management, | ||
| > library manager, scheduling, debug) toward the same convention. | ||
|
|
||
| > Borderline cases (e.g. parts of `lib/` that host syscall *implementations* such as `dma.c` | ||
| > and `dai.c`) behave as boundary code even though the directory is mostly shared-library. When | ||
| > a single directory genuinely mixes tiers, its README states the exception explicitly. | ||
|
|
||
| ## The boundary is the syscall set | ||
|
|
||
| Zephyr crosses the user/kernel boundary with system calls declared `__syscall` in a header and | ||
| implemented as two functions: | ||
|
|
||
| - `z_vrfy_<name>()` — **kernel-side validator**. Runs in supervisor mode, validates every | ||
| argument coming from user space (`K_SYSCALL_MEMORY_READ/WRITE`, `K_SYSCALL_OBJ`, | ||
| `k_usermode_from/to_copy`), then calls the implementation. This is the trust boundary. | ||
| - `z_impl_<name>()` — the actual logic. Callable directly from kernel code (fast path) or from | ||
| the validator when the call originates in user space. | ||
|
|
||
| **These pairs are the entire user/kernel API surface.** The table below summarises every SOF | ||
| syscall by subsystem: | ||
|
|
||
| | Subsystem | Syscalls | Declared in | Validator (`z_vrfy_`) / logic (`z_impl_`) | | ||
| |-----------|----------|-------------|-------------------------------------------| | ||
| | IPC | `ipc_msg_send`, `ipc_msg_list_remove`, `ipc_msg_reply`, `ipc_compound_pre_start`, `ipc_compound_post_start`, `ipc_wait_for_compound_msg`, `send_resource_notif` | `sof/ipc/msg.h`, `sof/ipc/ipc_reply.h`, `ipc4/handler.h`, `ipc4/notification.h` | scattered: `ipc/ipc-common.c`, `ipc/ipc3/helper.c`, `ipc/ipc4/handler-kernel.c`, `ipc/ipc4/notification.c` | | ||
| | Heap alloc | `sof_heap_alloc`, `sof_heap_free` | `rtos/alloc.h` | `zephyr/syscall/alloc.c` / `zephyr/lib/alloc.c` | | ||
| | Virtual regions | `vregion_create_map`, `vregion_set_interim`, `vregion_get`, `vregion_put`, `vregion_alloc_align`, `vregion_alloc_coherent_align`, `vregion_free` | `sof/lib/vregion.h` | `zephyr/syscall/vregion.c` / `zephyr/lib/vregion.c` | | ||
| | DMA | `sof_dma_get`, `sof_dma_put`, `sof_dma_get_attribute`, `sof_dma_request_channel`, `sof_dma_release_channel`, `sof_dma_config`, `sof_dma_start`, `sof_dma_stop`, `sof_dma_get_status`, `sof_dma_reload`, `sof_dma_suspend`, `sof_dma_resume` | `sof/lib/sof_dma.h` | `zephyr/syscall/sof_dma.c` / `lib/dma.c` (get/put); remaining 10 are `static inline z_impl_*` in `sof/lib/sof_dma.h` | | ||
| | DAI | `dai_get_device` | `sof/lib/dai-zephyr.h` | `zephyr/syscall/dai.c` / `lib/dai.c` | | ||
| | CPU | `cpu_get_id` | `sof/lib/cpu.h` | `zephyr/syscall/cpu.c` / `zephyr/lib/cpu.c` | | ||
| | Module management | `mod_balloc_align`, `mod_alloc_ext`, `mod_free`, `mod_free_all`, `mod_data_blob_handler_new`, `mod_fast_get` | `sof/audio/module_adapter/module/generic.h` | scattered: `audio/module_adapter/module/generic.c` | | ||
| | Library manager | `lib_manager_free_module` | `sof/lib_manager.h` | scattered: `library_manager/lib_manager.c` | | ||
| | Scheduling | `zephyr_ll_task_sem_alloc`, `zephyr_ll_task_sem_free`, `scheduler_dp_internal_free`, `scheduler_dp_ll_tick` | `sof/schedule/ll_schedule_domain.h`, `sof/schedule/dp_schedule.h` | scattered: `schedule/zephyr_ll.c`, `schedule/zephyr_dp_schedule*.c` | | ||
| | Debug | `debug_stream_slot_send_record` | `user/debug_stream_slot.h` | scattered: `debug/debug_stream/debug_stream_slot.c` | | ||
|
|
||
| The `zephyr/syscall/*.c` files hold the validators in one place per subsystem; the rows marked | ||
| **scattered** keep their `z_vrfy_` next to the kernel logic and are the migration targets. | ||
|
|
||
| Most syscall declarations are gated by `#if defined(__ZEPHYR__) && defined(CONFIG_SOF_FULL_ZEPHYR_APPLICATION)`; outside that guard (non-Zephyr host-test and ztest builds) the same name is a plain macro or inline calling `z_impl_*` directly. Non-userspace Zephyr builds (no `CONFIG_USERSPACE`) still use the `__syscall` declaration path, which Zephyr compiles to a direct `z_impl_*` call without the privilege-elevation trap. | ||
|
|
||
| ## Memory-partition markers | ||
|
|
||
| User threads can only touch memory that is mapped into their memory domain. SOF exposes shared | ||
| globals to user threads through Zephyr application memory partitions. The markers live in | ||
| [`rtos/userspace_helper.h`](../../../../zephyr/include/rtos/userspace_helper.h): | ||
|
|
||
| | Marker | Partition | Availability | Use for | | ||
| |--------|-----------|--------------|---------| | ||
| | `APP_TASK_DATA` / `APP_TASK_BSS` | `common_partition` | `CONFIG_USERSPACE` | Data shared with the userspace proxy worker thread and similar kernel-managed user tasks; not mapped into DP module domains in `SOF_USERSPACE_APPLICATION` mode | | ||
| | `APP_SYSUSER_DATA` / `APP_SYSUSER_BSS` | `sysuser_partition` | `CONFIG_SOF_USERSPACE_LL` | Data shared with the LL user thread and the LL userspace IPC thread | | ||
| | `K_APP_DMEM(part)` / `K_APP_BMEM(part)` | kernel-defined partition | `CONFIG_USERSPACE` | Per-subsystem partitions defined and mapped by kernel code (e.g. `ipc_context_part` in `ipc-common.c`); not intended for use by user components directly | | ||
|
|
||
| When userspace is disabled, all four SOF markers expand to nothing, so annotated globals are | ||
| ordinary variables in non-userspace builds. | ||
|
|
||
| **Rule:** *any global that a kernel-managed user thread reads or writes must carry a partition | ||
| marker at its declaration site.* Conversely, marked data is **user-writable**, so kernel code | ||
| must never trust it — re-validate on every use. DP modules running under | ||
| `SOF_USERSPACE_APPLICATION` access data through their per-module memory domain set up by the | ||
| proxy or DP thread infrastructure, not through `APP_TASK_*` / `APP_SYSUSER_*` markers. See the warning in | ||
| [`schedule/zephyr_ll_user.c`](../../../schedule/zephyr_ll_user.c) around `zephyr_ll_heap`, and | ||
| `zephyr_ll_user_heap_verify()` which exists precisely because the user-visible heap pointer | ||
| cannot be trusted. | ||
|
|
||
| ## Kconfig hierarchy | ||
|
|
||
| The userspace behaviour is assembled from a small option tree (source of truth: | ||
| [`zephyr/Kconfig`](../../../../zephyr/Kconfig)). | ||
|
|
||
| ```mermaid | ||
| graph TD | ||
| US["USERSPACE (Zephyr MMU)"] | ||
| US --> ALLOC[SOF_USERSPACE_INTERFACE_ALLOC] | ||
| US --> DMA[SOF_USERSPACE_INTERFACE_DMA] | ||
| US --> VREG["SOF_USERSPACE_INTERFACE_VREGION (needs SOF_VREGIONS)"] | ||
| US --> LL[SOF_USERSPACE_LL] | ||
| LL -.selects.-> ALLOC | ||
| LL -.selects.-> DMA | ||
| LL -.selects if SOF_VREGIONS.-> VREG | ||
| US --> SHEAP[SOF_USERSPACE_USE_SHARED_HEAP] | ||
| US --> DHEAP[SOF_USERSPACE_USE_DRIVER_HEAP] | ||
| PROXY[SOF_USERSPACE_PROXY] | ||
| PROXY -.selects.-> SHEAP | ||
| PROXY -.selects.-> DHEAP | ||
| PROXY --> MODIPC[SOF_USERSPACE_MOD_IPC_BY_DP_THREAD] | ||
| APP["SOF_USERSPACE_APPLICATION (auto = USERSPACE && !PROXY, needs SOF_VREGIONS)"] | ||
| ``` | ||
|
|
||
| Key options: | ||
|
|
||
| - **`SOF_USERSPACE_LL`** — run LL pipelines in user threads (`depends on USERSPACE`; selects | ||
| the ALLOC/DMA/VREGION interfaces). This is what the PTL overlay turns on. | ||
| - **`SOF_USERSPACE_PROXY`** — run modules inside a proxy container that forwards kernel-side | ||
| calls to user-privileged module code (selects the driver + shared heaps). | ||
| - **`SOF_USERSPACE_APPLICATION`** — not user-settable; a shortcut for | ||
| `USERSPACE && !SOF_USERSPACE_PROXY` (`depends on SOF_VREGIONS`), used to switch the DP | ||
| scheduler between the `_application` and `_thread` implementations in | ||
| [`schedule/CMakeLists.txt`](../../../schedule/CMakeLists.txt). | ||
| - **`SOF_USERSPACE`** — a separate WIP flag for userspace *modules*; do not confuse with | ||
| `USERSPACE` (the Zephyr option) or `SOF_USERSPACE_LL`. | ||
|
|
||
| ## Rules for placing new code | ||
|
|
||
| 1. **Pick the tier first.** OS/HAL/boot → a kernel-only directory. Audio application logic → | ||
| `audio/`. Reusable pure code → a shared-library directory (keep it user-safe). | ||
| 2. **Cross the boundary only via a syscall.** Add a `__syscall` declaration, put the | ||
| `z_vrfy_` validator in the boundary tier — preferably in `zephyr/syscall/<subsystem>.c` | ||
| alongside the existing ones — and validate every argument. | ||
| 3. **Mark shared globals.** Any global a kernel-managed user thread (LL user thread, userspace | ||
| IPC thread, proxy worker) reads/writes gets `APP_TASK_*`, `APP_SYSUSER_*`, or a `K_APP_*` | ||
| partition marker — and the kernel re-validates it. DP modules in application mode use their | ||
| per-module domain instead; see the DP scheduler and userspace proxy for how that is wired. | ||
| 4. **Grant kernel objects explicitly.** User threads need `k_object_access_grant()` / | ||
| `k_thread_access_grant()` for each kernel object they use. | ||
| 5. **Split by suffix inside boundary directories.** Where a subsystem has both sides, keep the | ||
| existing `-kernel` / `-user` file split so the two contexts stay physically separate. | ||
|
|
||
| ## Verifying the taxonomy | ||
|
|
||
| The tier of any directory can be confirmed from its source signals: | ||
|
|
||
| ```sh | ||
| # Boundary code: declares or implements syscalls | ||
| grep -rl 'z_vrfy_\|__syscall' src/<dir> | ||
|
|
||
| # User-visible data: partition markers | ||
| grep -rl 'APP_SYSUSER\|APP_TASK\|K_APP_BMEM\|K_APP_DMEM\|K_APPMEM_PARTITION' src/<dir> | ||
|
|
||
| # User threads / object grants / domains | ||
| grep -rl 'K_USER\|_access_grant\|mem_domain' src/<dir> | ||
| ``` | ||
|
kv2019i marked this conversation as resolved.
|
||
|
|
||
| A kernel-only directory returns nothing for all three. A boundary directory hits the first (and | ||
| usually the third). The application/shared tiers hit markers but should not host `z_vrfy_` | ||
| validators. | ||
|
|
||
| ## Related files | ||
|
|
||
| - [`zephyr/include/rtos/userspace_helper.h`](../../../../zephyr/include/rtos/userspace_helper.h) | ||
| — partition markers and user-space helper API. | ||
| - [`schedule/README.md`](../../../schedule/README.md), | ||
| [`ipc/README.md`](../../../ipc/README.md), | ||
| [`audio/module_adapter/README.md`](../../../audio/module_adapter/README.md) — boundary | ||
| subsystems. | ||
| - [`AGENTS.md`](../../../../AGENTS.md) — contributor and AI-agent workflow rules that | ||
| reference this document. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.