Compare commits

...
Author SHA1 Message Date
nmzik b7888046ba Fix macOS guest address-space reservation 2026-07-31 02:48:09 +02:00
nmzik 4532883b83 add unknwon flag 2026-07-31 02:33:00 +02:00
nmzik e87ae56544 Rework guest memory tracking 2026-07-31 02:14:17 +02:00
ecb48f90bb Emulate SHA-NI and fix SSE4a EXTRQ/INSERTQ register form (#126)
* Emulate SHA-NI on illegal instruction faults

* Fix SSE4a EXTRQ/INSERTQ register form

* Fix SHA-NI memory operand emulation

* Revert "Fix SSE4a EXTRQ/INSERTQ register form"

This reverts commit ea2b54a4d0.

---------

Co-authored-by: neobugs1 <neobugs1@users.noreply.github.com>
Co-authored-by: nmzik <Nmzik@mail.ru>
2026-07-30 16:21:36 +02:00
nmzik 77aa28b27c update README 2026-07-30 05:16:37 +02:00
nmzikandGitHub d04938c88c Embedded fetch shader: Fix overlapping buffer loads (#133)
Fix overlapping buffer loads. Fixes many games
2026-07-30 05:08:38 +02:00
nmzikandGitHub 85622befb8 Fix fabricated HTTP2 success (#129)
@StefanosCosta Thanks!
2026-07-30 00:48:08 +02:00
nmzik 3965d41d36 texture_cache: fix exact-match reuse across different tile modes 2026-07-30 00:10:40 +02:00
nmzik c508c4a9c0 shader_recompiler: allow GDS append/consume offsets 2026-07-30 00:10:40 +02:00
ClaxtenandGitHub e91dd39cb0 Drop redundant PROT_NONE tracking in reserve paths for Linux (#122)
src: platform: Linux: Drop redundant PROT_NONE tracking in reserve paths

* Some UE4 games, such as The Pathless, reserve a 512 GiB virtual address range during libc startup.
  Tracking every 4 KiB page causes a long delay and is unnecessary since the range is already PROT_NONE,
  and untracked pages are treated as NoAccess.

Signed-off-by: Claxten <claxten10@gmail.com>
2026-07-30 00:00:21 +02:00
nmzik cc76827e63 Fix vertex buffer ranges crossing memory mappings 2026-07-29 21:05:24 +02:00
nmzik 832bc84100 fix(shader): stabilize scalar provenance phis in cyclic CFGs 2026-07-29 21:05:24 +02:00
nmzik 65a0f0baa7 NpManager ABI 2026-07-29 21:05:24 +02:00
nmzik b9ae2537ef renderer: broaden compatibility 2026-07-29 18:47:26 +02:00
nmzik 0b9edaa721 graphics: broaden storage image atomic compatibility 2026-07-29 18:47:24 +02:00
nmzik 8a244677d7 fix(renderer): resolve delayed GPU page faults through buffer and texture caches 2026-07-29 18:47:19 +02:00
nmzikandGitHub f6e01e5403 Optimize bulk memory invalidation (#124)
Build and Release KytyPS5 / Build KytyPS5 (Windows) (push) Canceled after 0s
Build and Release KytyPS5 / Build KytyPS5 (macOS) (push) Canceled after 0s
Build and Release KytyPS5 / Build KytyPS5 (Linux) (push) Canceled after 0s
Build and Release KytyPS5 / Release KytyPS5 (push) Canceled after 0s
* Per page -> per range search (optimization)
2026-07-29 05:09:36 +02:00
nmzikandGitHub 861729fc6c Optimize texture cache tracking (#123)
Optimize texture cache page tracking
2026-07-29 03:37:09 +02:00
nmzik 687ce025c6 KernelOpen: minor fix 2026-07-29 00:31:09 +02:00
nmzik a21d1aaa47 fix 2026-07-29 00:31:09 +02:00
nmzik ec11f31aa6 fix graphical bug (PPSA17221) 2026-07-29 00:31:08 +02:00
nmzik aeceaff028 new ABIs + one stub 2026-07-29 00:31:08 +02:00
nmzik e76f2d1af8 new ABIs 2026-07-29 00:31:08 +02:00
nmzik 64845e2294 new ABI, broaden support 2026-07-29 00:31:08 +02:00
nmzik 4b9030bd3d ABI fixes: libNet, libSaveData (rewrite the legacy Kyty implementation) 2026-07-29 00:30:24 +02:00
nmzik a6b61d7aa0 refactor(graphics): split renderer and recompiler into focused modules 2026-07-29 00:30:24 +02:00
Stefanos CostaandGitHub 2b9cba457e Linux: port the emulator to a working state on Linux (#117)
Linux: port the emulator to a working state
2026-07-29 00:25:58 +02:00
nmzikandGitHub 3dc793d859 Fix badges (#120)
Build and Release KytyPS5 / Build KytyPS5 (Windows) (push) Waiting to run
Build and Release KytyPS5 / Build KytyPS5 (macOS) (push) Waiting to run
Build and Release KytyPS5 / Release KytyPS5 (push) Blocked by required conditions
Build KytyPS5 (Linux) / build (push) Canceled after 0s
docs: fix build badges
2026-07-28 19:40:14 +02:00
nmzikandGitHub 01d78b71fe CI: release Windows and macOS archives (#119)
* ci: release Windows and macOS archives

* docs: note experimental macOS support
2026-07-28 19:33:21 +02:00
4d79f19089 macOS (Apple Silicon) support: native build running under Rosetta 2 with MoltenVK (#102)
* macos: POSIX platform layer (host fault handler, virtual memory)

- hostException: Mach/POSIX signal-based host fault handler mirroring the
  Windows vectored handler (SIGSEGV/SIGBUS/SIGILL), behind
  '#elif defined(__APPLE__)'. Windows and Linux branches are untouched.
- sysLinuxVirtual: mach_vm_region-based is_mapped (no /proc/self/maps on
  macOS), PTHREAD_MUTEX_NORMAL, and a MAP_FIXED carve-in-place allocator that
  never leaves an unmapped hole for dyld/Rosetta/Metal to claim. All
  __APPLE__-guarded; the original Linux allocator path is preserved verbatim.
- sysLinuxDbg/sysLinuxFileIO: libgen.h for basename(), and a POSIX
  opendir/readdir implementation of SysFileGetDents (previously a stub).

* macos: guest kernel backing, threads, and POSIX libs

- memory/memoryAddressSpace: anonymous shm_open backing (macOS has no
  memfd_create) and re-reserve-on-unmap so a guest MAP_FIXED remap never
  destroys a host mapping (__APPLE__-guarded). Drops host <sys/mman.h> MAP_*
  macros so the guest's constexpr MAP_* constants compile (no-op on Windows).
- pthread: undef the host PTHREAD_STACK_MIN macro; on non-Windows, pin the
  stack-switch asm's guest rsp/rbp to callee-saved r14/r15 (the template
  clobbers r12/r13); __APPLE__ no-op for the absent pthread_condattr_setclock.
  The Windows stack-switch asm is byte-identical.
- network: define SOCKET/INVALID_SOCKET for the non-Windows paths (were
  undefined at 20+ use sites), and rename the non-Windows kernel_clock_*
  helpers to the CamelCase names the callers already use. Both repair the
  shared non-Windows build. Integer reinterpret_cast -> static_cast.

* loader: pin stack-switch asm operands to callee-saved registers

RunEntry switches to the guest stack with an asm template that clobbers
r12/r13 before consuming its inputs. With plain "r" constraints the compiler
may place func/guest_rsp/guest_rbp into r12/r13, so 'callq *func' jumps
through the saved host rsp -- a latent miscompile on every platform that any
unrelated codegen change can trigger. Pin the inputs to rbx/r14/r15, which the
template never touches and the SysV callee preserves.

General correctness fix (not macOS-specific); the same fix is applied to
pthread RunOnGuestStack.

* macos: Vulkan/MoltenVK graphics enablement

- pageManager: GPU write-tracking via Mach VM queries + mprotect (mprotect
  cannot report the previous protection, so the expected-old check is dropped),
  thread id via pthread_mach_thread_np, 4 KB page-size check -- __APPLE__-guarded.
- shaders/renderDraw/vulkanWindow: MoltenVK lacks VK_EXT_color_write_enable,
  VK_EXT_depth_clip_enable and depthBounds; fall back to static color-write
  masks and default depth clipping, request VK_KHR_portability_subset, and
  reuse the graphics queue for present when only one queue is exposed. Non-Apple
  pipeline/extension setup is unchanged.
- window: optional borderless window (KYTY_BORDERLESS) to sidestep a Rosetta
  NSException in macOS window chrome.

* graphics: macOS thread identity for region tracking

Region-ownership locks resolve the current thread via GetCurrentThreadId()
on Windows and EXIT elsewhere. Use the Mach thread port on macOS (nonzero
per-thread id; 0 stays the no-owner sentinel).

* macos: marshal AppKit window operations to the main thread

The present worker thread shows the window, updates its icon and title, and
recreates a lost Vulkan surface. AppKit traps with 'Must only be used from the
main thread' when these run off the main thread. Route them through a small
task queue drained by the SDL main loop (an SDL_USEREVENT wakes the loop when
it is blocked in SDL_WaitEvent). Title updates are fire-and-forget; showing
the window and surface recreation wait for completion.

Windows and Linux call the SDL functions directly, as before.

* build: ignore the _Build output directory

_Build is the conventional out-of-tree build location (only _Build/vscode-clang
was ignored); a stray git add could sweep build artifacts into a commit.

* macos: isolate stack-switch workaround

---------

Co-authored-by: nmzik <Nmzik@mail.ru>
2026-07-28 19:03:15 +02:00
8fe4765f9e ci: add macOS build workflow (#114)
Build KytyPS5 (Linux) / build (push) Waiting to run
Build KytyPS5 / build (push) Waiting to run
Build KytyPS5 / release (push) Blocked by required conditions
Build KytyPS5 (macOS) / build (push) Has been cancelled
* ci: add macOS build workflow

* macos: add POSIX compatibility guards

* ci: select Xcode 26 for macOS

* macos: add kernel POSIX compatibility

* launcher: support macOS process startup

* ci: fix macOS architecture verification

* ci: cache glslang on Windows

---------

Co-authored-by: Abdullah K. <akjee204@gmail.com>
2026-07-27 22:01:50 +02:00
nmzik c71bb9fc9e build: make clang-tidy checks opt-in 2026-07-27 16:33:22 +02:00
nmzikandGitHub f60f80b631 launcher: add local game patches (#111)
launcher: add game patches
2026-07-27 16:19:31 +02:00
Abdullah K.andnmzik 31ea0081a9 perf: skip LOGF formatting when logging is silent
Build KytyPS5 (Linux) / build (push) Waiting to run
Build KytyPS5 / build (push) Waiting to run
Build KytyPS5 / release (push) Blocked by required conditions
LOGF/LOGF_COLOR always ran fmt::sprintf even when the output was discarded; at
emulator log volume the formatting alone costs frames. Add Log::IsSilent() and
short-circuit before formatting. Behavior-preserving on all platforms (it only
skips work whose result is thrown away); before init it reports non-silent so
early logs still print.

(cherry picked from commit 1749e0fc68)
2026-07-27 14:21:03 +02:00
Abdullah K.andnmzik 7921269878 build: add macOS (Apple Silicon) build support
macOS compiles the existing POSIX/Linux code paths (KYTY_PLATFORM_LINUX)
targeting x86_64 under Rosetta 2. Adds an APPLE branch to the build:
selects Apple's ld64 linker, unsets the Ninja response-file forcing (Apple
ar has no @file support), derives the target arch from CMAKE_OSX_ARCHITECTURES
for cpuinfo/ffmpeg, and re-signs the binary post-build with JIT entitlements
(required to execute written trampolines and Rosetta-translated guest code).

All changes are guarded by if(APPLE); Windows and Linux configuration is
unchanged.

(cherry picked from commit 13be13b757)
2026-07-27 14:16:51 +02:00
nmzik e42f6c3892 ci: add Linux build workflow 2026-07-27 14:13:34 +02:00
nmzik 7351eae117 graphics: add 1D image descriptor support 2026-07-27 14:10:51 +02:00
nmzik 125287dc30 graphics: broaden depth compatibility 2026-07-27 10:29:37 +02:00
nmzik 142b3c20fa graphics: unify buffer bindings and depth target handling 2026-07-27 10:12:41 +02:00
nmzik f43567e12c shader: validate MSAA image descriptors during materialization 2026-07-27 10:12:41 +02:00
nmzik c86d4b1f0d Fix linux compilation 2026-07-27 10:12:41 +02:00
nmzik 31fb3ec5af graphics: broaden storage image and view format compatibility 2026-07-27 10:12:41 +02:00
nmzik c3b2ae9733 graphics: support more depth-tiled storage image formats 2026-07-27 10:12:41 +02:00
nmzik 93774ee37c fix teardown 2026-07-27 02:27:37 +02:00
nmzik a20cf5d298 refactor renderer 2026-07-27 02:27:29 +02:00
nmzik a720b28b7f graphics: cleanup 2026-07-23 23:21:26 +02:00
nmzik 7638c25d76 renderer: replace generation retries with staged resource binding 2026-07-23 22:32:30 +02:00
nmzik 85876388cf gpuTiler: fix performance (dispatch a 2D 8×8 workgroup) 2026-07-22 23:36:00 +02:00
nmzik f60007e923 refactor(gpu): use a single ordered PM4 scheduler 2026-07-22 23:15:03 +02:00
nmzik dbcb444c05 unify graphics and compute on one Vulkan queue 2026-07-22 17:44:38 +02:00
255 changed files with 32190 additions and 26326 deletions
+308 -16
View File
@@ -1,4 +1,4 @@
name: Build KytyPS5
name: Build and Release KytyPS5
on:
workflow_dispatch:
@@ -7,7 +7,8 @@ on:
pull_request:
jobs:
build:
windows:
name: Build KytyPS5 (Windows)
runs-on: windows-2022
steps:
@@ -22,11 +23,31 @@ jobs:
- name: Setup Ninja
uses: seanmiddleditch/gha-setup-ninja@v5
- name: Install glslang
- name: Locate vcpkg
id: vcpkg
shell: pwsh
run: |
$vcpkgRoot = Split-Path (Get-Command vcpkg).Source
vcpkg install glslang[tools,opt]:x64-windows
$portHash = (Get-FileHash "$vcpkgRoot\ports\glslang\vcpkg.json" -Algorithm SHA256).Hash
"root=$vcpkgRoot" |
Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
"port_hash=$($portHash.ToLowerInvariant())" |
Out-File -FilePath $env:GITHUB_OUTPUT -Encoding utf8 -Append
- name: Cache glslang
id: glslang_cache
uses: actions/cache@v4
with:
path: ${{ steps.vcpkg.outputs.root }}\installed\x64-windows
key: glslang-${{ runner.os }}-${{ runner.arch }}-${{ steps.vcpkg.outputs.port_hash }}-tools-opt
- name: Install glslang
shell: pwsh
run: |
$vcpkgRoot = "${{ steps.vcpkg.outputs.root }}"
if ("${{ steps.glslang_cache.outputs.cache-hit }}" -ne "true") {
vcpkg install glslang[tools,opt]:x64-windows
}
"$vcpkgRoot\installed\x64-windows\tools\glslang" |
Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
@@ -62,52 +83,323 @@ jobs:
- name: Build
shell: cmd
run: |
cmake --build _Build/windows --target launcher --parallel
cmake --build _Build/windows --target launcher virtual_memory_allocation_tests --parallel
- name: Test
shell: cmd
run: |
ctest --test-dir _Build/windows --output-on-failure -R "^virtual_memory_allocation$"
- name: Install
shell: cmd
run: |
cmake --install _Build/windows --prefix _Build/windows/install
- name: Upload Artifacts
- name: Upload Windows artifacts
uses: actions/upload-artifact@v4
with:
name: KytyPS5
name: KytyPS5-Windows-x64
path: _Build/windows/install/**
if-no-files-found: error
macos:
name: Build KytyPS5 (macOS)
runs-on: macos-15
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
submodules: recursive
- name: Select Xcode 26
shell: bash
run: |
xcode_path="$(ls -d /Applications/Xcode_26*.app | tail -n 1)"
test -n "$xcode_path"
sudo xcode-select --switch "$xcode_path/Contents/Developer"
xcodebuild -version
- name: Install build dependencies
shell: bash
run: brew install glslang ninja
- name: Install Qt 6.10.3
uses: jurplel/install-qt-action@v4
with:
version: "6.10.3"
host: mac
target: desktop
arch: clang_64
cache: true
- name: Verify toolchain
shell: bash
run: |
git --version
cmake --version
ninja --version
clang++ --version
glslangValidator --version
echo "Host architecture: $(uname -m)"
echo "QT_ROOT_DIR=$QT_ROOT_DIR"
- name: Configure
shell: bash
run: |
cmake -S src -B _Build/macos \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_OSX_ARCHITECTURES=x86_64 \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_PREFIX_PATH="$QT_ROOT_DIR"
- name: Build
shell: bash
run: |
cmake --build _Build/macos \
--target launcher virtual_memory_allocation_tests \
--parallel
- name: Test
shell: bash
run: |
ctest --test-dir _Build/macos --output-on-failure \
-R '^virtual_memory_allocation$'
- name: Install
shell: bash
run: |
cmake --install _Build/macos --prefix _Build/macos/install
- name: Bundle MoltenVK
shell: bash
env:
MOLTENVK_VERSION: v1.4.2
MOLTENVK_SHA256: f95765a6229cb7b915990a2890ce12ebe36a730b021545d3d52ae69ce4c4024e
run: |
archive="$RUNNER_TEMP/MoltenVK-macos.tar"
package="$RUNNER_TEMP/MoltenVK"
curl --fail --location --retry 3 \
--output "$archive" \
"https://github.com/KhronosGroup/MoltenVK/releases/download/$MOLTENVK_VERSION/MoltenVK-macos.tar"
echo "$MOLTENVK_SHA256 $archive" | shasum -a 256 --check
tar -xf "$archive" -C "$RUNNER_TEMP"
install -m 755 \
"$package/MoltenVK/dynamic/dylib/macOS/libMoltenVK.dylib" \
_Build/macos/install/libMoltenVK.dylib
install -m 644 "$package/LICENSE" _Build/macos/install/LICENSE.MoltenVK
codesign --force --sign - --timestamp=none _Build/macos/install/libMoltenVK.dylib
- name: Verify artifacts
shell: bash
run: |
file _Build/macos/install/launcher
file _Build/macos/install/kyty_emulator
file _Build/macos/install/libMoltenVK.dylib
lipo _Build/macos/install/launcher -verify_arch x86_64
lipo _Build/macos/install/kyty_emulator -verify_arch x86_64
lipo _Build/macos/install/libMoltenVK.dylib -verify_arch x86_64
codesign --verify --strict _Build/macos/install/kyty_emulator
codesign --verify --strict _Build/macos/install/libMoltenVK.dylib
- name: Upload macOS artifacts
uses: actions/upload-artifact@v4
with:
name: KytyPS5-macOS-x86_64
path: _Build/macos/install/**
if-no-files-found: error
linux:
name: Build KytyPS5 (Linux)
runs-on: ubuntu-24.04
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
submodules: recursive
- name: Install build dependencies
shell: bash
run: |
sudo apt-get update
sudo apt-get install --no-install-recommends --yes \
clang \
glslang-tools \
libasound2-dev \
libdbus-1-dev \
libgl1-mesa-dev \
libpulse-dev \
libudev-dev \
libwayland-dev \
libx11-dev \
libxcursor-dev \
libxext-dev \
libxfixes-dev \
libxi-dev \
libxkbcommon-dev \
libxrandr-dev \
libxss-dev \
lld \
ninja-build \
wayland-protocols
- name: Install Qt 6.10.3
uses: jurplel/install-qt-action@v4
with:
version: "6.10.3"
host: linux
target: desktop
arch: linux_gcc_64
cache: true
- name: Verify toolchain
shell: bash
run: |
git --version
cmake --version
ninja --version
clang++ --version
ld.lld --version
glslangValidator --version
echo "Qt6_DIR=$Qt6_DIR"
- name: Configure
shell: bash
run: |
mkdir -p _Build
cmake -S src -B _Build/linux \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_PREFIX_PATH="$Qt6_DIR" 2>&1 | tee _Build/configure.log
exit "${PIPESTATUS[0]}"
- name: Verify SDL2 backends
shell: bash
run: |
status=0
for feature in SDL_ALSA SDL_PULSEAUDIO SDL_WAYLAND SDL_X11 SDL_LIBUDEV SDL_DBUS; do
if grep -qE "^-- ${feature} +\\(Wanted: ON\\): ON" _Build/configure.log; then
echo "ok ${feature}"
else
echo "FAIL ${feature} is not enabled"
status=1
fi
done
exit "$status"
- name: Build
shell: bash
run: |
cmake --build _Build/linux \
--target launcher page_manager_tests memory_tracker_tests \
virtual_memory_allocation_tests \
--parallel
- name: Test
shell: bash
run: |
ctest --test-dir _Build/linux --output-on-failure \
-R '^(page_manager|memory_tracker|virtual_memory_allocation)$'
- name: Install
shell: bash
run: |
cmake --install _Build/linux --prefix _Build/linux/install
- name: Verify artifacts
shell: bash
run: |
file _Build/linux/install/launcher
file _Build/linux/install/kyty_emulator
file _Build/linux/install/kyty_emulator | grep -q "ELF 64-bit LSB .*x86-64"
ldd _Build/linux/install/kyty_emulator > /dev/null
readelf -d _Build/linux/install/launcher | grep -q 'RPATH.*\$ORIGIN/lib'
while IFS= read -r -d '' binary; do
while read -r dependency; do
test -e "_Build/linux/install/lib/$dependency"
done < <(
readelf -d "$binary" |
sed -n 's/.*Shared library: \[\(libQt6[^]]*\|libicu[^]]*\)\].*/\1/p'
)
done < <(
find _Build/linux/install/launcher _Build/linux/install/plugins \
-type f \( -name launcher -o -name '*.so' \) -print0
)
_Build/linux/install/kyty_emulator --help > /dev/null
- name: Upload Linux artifacts
uses: actions/upload-artifact@v4
with:
name: KytyPS5-Linux-x86_64
path: _Build/linux/install/**
if-no-files-found: error
release:
if: github.event_name == 'push'
needs: build
name: Release KytyPS5
if: github.event_name == 'push' && github.repository == 'KytyPS5/KytyPS5'
needs: [windows, macos, linux]
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Download build
- name: Download builds
uses: actions/download-artifact@v4
with:
name: KytyPS5
path: KytyPS5
path: artifacts
- name: Set release name
shell: bash
run: |
echo "RELEASE_NAME=KytyPS5-$(date -u +'%Y-%m-%d')-${GITHUB_SHA::7}" >> "$GITHUB_ENV"
- name: Package build
- name: Package builds
shell: bash
run: zip -r "$RELEASE_NAME.zip" KytyPS5
run: |
windows_dir="artifacts/KytyPS5-Windows-x64"
macos_dir="artifacts/KytyPS5-macOS-x86_64"
linux_dir="artifacts/KytyPS5-Linux-x86_64"
test -d "$windows_dir"
test -d "$macos_dir"
test -d "$linux_dir"
chmod a+x \
"$macos_dir/launcher" \
"$macos_dir/kyty_emulator" \
"$macos_dir/libMoltenVK.dylib" \
"$linux_dir/launcher" \
"$linux_dir/kyty_emulator"
(
cd "$windows_dir"
zip -r "$GITHUB_WORKSPACE/$RELEASE_NAME-Windows-x64.zip" .
)
(
cd "$macos_dir"
zip -r "$GITHUB_WORKSPACE/$RELEASE_NAME-macOS-x86_64.zip" .
)
tar -C "$linux_dir" -czf \
"$GITHUB_WORKSPACE/$RELEASE_NAME-Linux-x86_64.tar.gz" .
- name: Create release
shell: bash
env:
GH_TOKEN: ${{ github.token }}
run: |
assets=(
"$RELEASE_NAME-Windows-x64.zip"
"$RELEASE_NAME-macOS-x86_64.zip"
"$RELEASE_NAME-Linux-x86_64.tar.gz"
)
if gh release view "$RELEASE_NAME" --repo "$GITHUB_REPOSITORY" > /dev/null 2>&1; then
gh release upload "$RELEASE_NAME" "$RELEASE_NAME.zip" --clobber --repo "$GITHUB_REPOSITORY"
gh release upload "$RELEASE_NAME" "${assets[@]}" \
--clobber \
--repo "$GITHUB_REPOSITORY"
else
gh release create "$RELEASE_NAME" "$RELEASE_NAME.zip" \
gh release create "$RELEASE_NAME" "${assets[@]}" \
--generate-notes \
--repo "$GITHUB_REPOSITORY" \
--target "$GITHUB_SHA" \
+2 -1
View File
@@ -2,4 +2,5 @@
.vs/
.idea/
build/
_Build/vscode-clang/
_Build/vscode-clang/
_Build/
+11 -1
View File
@@ -44,7 +44,17 @@ set(SPDLOG_NO_EXCEPTIONS ON CACHE BOOL "" FORCE)
add_subdirectory(spdlog EXCLUDE_FROM_ALL)
if (NOT TARGET FFmpeg::ffmpeg)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64|ARM64)$")
# On macOS the target arch is driven by CMAKE_OSX_ARCHITECTURES, not the host
# CMAKE_SYSTEM_PROCESSOR (which reports arm64 even for an x86_64/Rosetta build).
if(APPLE AND CMAKE_OSX_ARCHITECTURES)
if(CMAKE_OSX_ARCHITECTURES MATCHES "^(x86_64)$")
set(ARCHITECTURE x86_64)
elseif(CMAKE_OSX_ARCHITECTURES MATCHES "^(arm64)$")
set(ARCHITECTURE arm64)
else()
set(ARCHITECTURE x86_64 arm64) # universal
endif()
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(aarch64|arm64|ARM64)$")
set(ARCHITECTURE arm64)
else()
set(ARCHITECTURE x86_64)
+12 -3
View File
@@ -5,7 +5,14 @@ set(cpuinfo_src
deps/clog/src/clog.c
)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(i[3-6]86|AMD64|x86(_64)?)$")
# On Apple the target arch is driven by CMAKE_OSX_ARCHITECTURES, not the host
# CMAKE_SYSTEM_PROCESSOR (which is arm64 even for an x86_64/Rosetta build).
set(CPUINFO_TARGET_PROC "${CMAKE_SYSTEM_PROCESSOR}")
if(APPLE AND CMAKE_OSX_ARCHITECTURES)
list(GET CMAKE_OSX_ARCHITECTURES 0 CPUINFO_TARGET_PROC)
endif()
if(CPUINFO_TARGET_PROC MATCHES "^(i[3-6]86|AMD64|x86(_64)?)$")
list(APPEND cpuinfo_src
src/x86/init.c
src/x86/info.c
@@ -19,11 +26,13 @@ if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(i[3-6]86|AMD64|x86(_64)?)$")
src/x86/cache/deterministic.c)
if(LINUX OR ANDROID)
list(APPEND cpuinfo_src src/x86/linux/init.c src/x86/linux/cpuinfo.c)
elseif(APPLE)
list(APPEND cpuinfo_src src/x86/mach/init.c src/mach/topology.c)
else()
list(APPEND cpuinfo_src src/x86/windows/init.c)
endif()
elseif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(armv[5-8].*|aarch64|arm64)$")
elseif(CPUINFO_TARGET_PROC MATCHES "^(armv[5-8].*|aarch64|arm64)$")
list(APPEND cpuinfo_src src/arm/uarch.c src/arm/cache.c)
if(LINUX OR ANDROID)
+78 -13
View File
@@ -1,13 +1,16 @@
# KytyPS5
[![Platform](https://img.shields.io/badge/platform-Windows%20x64-0078D4.svg)](#system-requirements)
[![Build KytyPS5 (Windows)](https://img.shields.io/github/actions/workflow/status/KytyPS5/KytyPS5/build.yml?branch=main&event=push&label=Build%20KytyPS5%20%28Windows%29)](https://github.com/KytyPS5/KytyPS5/actions/workflows/build.yml)
[![Build KytyPS5 (Linux)](https://img.shields.io/github/actions/workflow/status/KytyPS5/KytyPS5/build.yml?branch=main&event=push&label=Build%20KytyPS5%20%28Linux%29)](https://github.com/KytyPS5/KytyPS5/actions/workflows/build.yml)
[![Build KytyPS5 (macOS)](https://img.shields.io/github/actions/workflow/status/KytyPS5/KytyPS5/build.yml?branch=main&event=push&label=Build%20KytyPS5%20%28macOS%29)](https://github.com/KytyPS5/KytyPS5/actions/workflows/build.yml)
[![Platform](https://img.shields.io/badge/platform-Windows%20x64%20%7C%20Linux%20x64%20%7C%20macOS%20x86__64-0078D4.svg)](#system-requirements)
[![Status](https://img.shields.io/badge/status-early%20development-orange.svg)](#current-status)
[![License](https://img.shields.io/badge/license-GPL--2.0-blue.svg)](LICENSE)
KytyPS5 is a free and open-source PlayStation 5 emulator written in C++ for Windows. It is based on
a heavily modified version of [Kyty](https://github.com/InoriRus/Kyty). The project is in an early
stage of development, so compatibility is limited and behavior may change significantly between
builds.
KytyPS5 is a free and open-source PlayStation 5 emulator written in C++ for Windows and Linux,
with experimental macOS support. It is based on a heavily modified version of
[Kyty](https://github.com/InoriRus/Kyty). The project is in an early stage of development, so
compatibility is limited and behavior may change significantly between builds.
> [!IMPORTANT]
> KytyPS5 is not affiliated with Sony Interactive Entertainment or PlayStation. The project does
@@ -21,7 +24,11 @@ KytyPS5 can boot 2D games and a selection of 3D games, including titles built wi
Development is focused on compatibility and boot reliability.
Linux support is planned, but Windows is the only supported platform at this time.
Windows is the primary platform and receives the most testing. Linux builds and runs; see
[Building on Linux](#building-on-linux).
macOS support is experimental. Compatibility with the same games on Windows and macOS has not yet
been tested.
## Bugs and Issues
@@ -43,7 +50,7 @@ graphical glitches, low compatibility, and poor performance.
</tr>
<tr>
<td align="center">
<strong>Minecraft Legends</strong><br>
<strong>Neptunia ReVerse</strong><br>
<img src="docs/screenshots/ps5-04.png" width="300" alt="Minecraft Legends running in KytyPS5">
</td>
<td align="center">
@@ -51,15 +58,28 @@ graphical glitches, low compatibility, and poor performance.
<img src="docs/screenshots/ps5-05.png" width="300" alt="SILENT HILL: The Short Message running in KytyPS5">
</td>
</tr>
<tr>
<td align="center">
<strong>Hellboy</strong><br>
<img src="docs/screenshots/ps5-02.png" width="300" alt="Disgaea 6 running in KytyPS5">
</td>
<td align="center">
<strong>Paleo Pines</strong><br>
<img src="docs/screenshots/ps5-06.png" width="300" alt="Dreaming Sarah running in KytyPS5">
</td>
</tr>
</table>
<p align="center"><em>And many more...</em></p>
## Contributing
Testing games and submitting detailed bug reports are useful ways to contribute. Search existing
issues first, then use the **Game Emulation Bug Report** template and attach the complete log file.
Code contributions should be focused, build successfully on Windows, and include relevant tests
where practical. Because KytyPS5 is still evolving quickly, consider opening an issue before
Code contributions should be focused, build successfully on the platforms they touch, and include
relevant tests where practical. Windows is the primary target, so a change that alters shared code
should not regress it; changes confined to a platform's own code paths only need to build there. Because KytyPS5 is still evolving quickly, consider opening an issue before
starting a large change.
### Formatting
@@ -94,11 +114,11 @@ the Vulkan/SPIR-V validation rules.
### System requirements
- Windows 10 version 1803
- Windows 10 version 1803, or a current Linux distribution
- A 64-bit x86 processor
- A Vulkan 1.3-capable GPU with current drivers
### Build requirements
### Build requirements (Windows)
- Git
- CMake 3.12 or newer
@@ -132,11 +152,48 @@ cmake --install _Build/windows --prefix _Build/windows/install
The finished application and its runtime dependencies will be placed in
`_Build/windows/install`.
### Building on Linux
Install the toolchain and the libraries the bundled SDL2 needs. Without the audio, Wayland and
udev development packages SDL2 quietly configures itself without those backends, and the resulting
build has no working sound and no gamepad hotplug:
```bash
sudo apt-get install --no-install-recommends \
clang lld ninja-build cmake git glslang-tools \
libgl1-mesa-dev libx11-dev libxcursor-dev libxext-dev libxfixes-dev \
libxi-dev libxrandr-dev libxss-dev libxkbcommon-dev \
libasound2-dev libpulse-dev libudev-dev libdbus-1-dev libwayland-dev wayland-protocols
```
Qt 6 (Concurrent, Network, Widgets) is also required — either the distribution packages
(`qt6-base-dev`) or an official Qt installation.
```bash
git submodule update --init --recursive
cmake -S src -B _Build/linux -G Ninja -DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ \
-DCMAKE_PREFIX_PATH="$Qt6_DIR"
cmake --build _Build/linux --target launcher --parallel
cmake --install _Build/linux --prefix _Build/linux/install
```
The install step copies the Qt libraries and plugins next to the binaries, so
`_Build/linux/install` runs without a matching system Qt.
As on Windows, the MSVC compiler is not used; Clang is required. `cl.exe` is rejected at configure
time.
Note that the CMake source root is `src`, not the repository root.
### Visual Studio Code
A ready-made Visual Studio Code setup is included in [`.vscode`](.vscode). It configures CMake
Tools to build the project with Ninja and `clang-cl` and provides launch profiles for both
`launcher.exe` and `kyty_emulator.exe`.
`launcher.exe` and `kyty_emulator.exe`. It is Windows-only: VS Code settings cannot select a
compiler per platform, so on Linux configure from the command line as shown above.
Before using it:
@@ -158,6 +215,10 @@ To use the graphical launcher:
.\_Build\windows\install\launcher.exe
```
```bash
./_Build/linux/install/launcher
```
On first launch, add one or more game folders in the global settings. The launcher searches those
folders recursively for game directories containing `eboot.bin`. Select a detected game and run it
from the game list.
@@ -168,7 +229,11 @@ The emulator can also be started directly with a legally obtained game directory
.\_Build\windows\install\kyty_emulator.exe --game "D:\Games\ExampleGame"
```
Run `kyty_emulator.exe --help` to see the available graphics, logging, validation, profiling, and
```bash
./_Build/linux/install/kyty_emulator --game "/games/ExampleGame"
```
Run `kyty_emulator --help` to see the available graphics, logging, validation, profiling, and
debugging options.
### AI Use
Binary file not shown.

After

Width:  |  Height:  |  Size: 1.6 MiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 2.7 MiB

After

Width:  |  Height:  |  Size: 2.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.4 MiB

+191 -25
View File
@@ -10,15 +10,19 @@ if(CMAKE_SYSTEM_NAME MATCHES ".*Linux")
set(LINUX TRUE)
endif()
if (NOT (WIN32 OR LINUX))
message(FATAL_ERROR "only Windows and Linux builds are supported")
if (NOT (WIN32 OR LINUX OR APPLE))
message(FATAL_ERROR "only Windows, Linux, and macOS builds are supported")
endif()
set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED ON)
set(CMAKE_CXX_EXTENSIONS OFF)
set(CMAKE_CXX_SCAN_FOR_MODULES OFF)
include(utils.cmake)
include(CTest)
option(KYTY_ENABLE_CLANG_TIDY "Run clang-tidy checks during builds" OFF)
set(KYTY_THIRD_PARTY_DIR "${CMAKE_CURRENT_SOURCE_DIR}/../3rdparty")
@@ -43,8 +47,9 @@ else()
set(KYTY_BUILD KYTY_BUILD_RELEASE)
endif()
if(LINUX)
set(KYTY_PLATFORM KYTY_PLATFORM_LINUX)
if(LINUX OR APPLE)
# macOS rides the POSIX/Linux code paths until it gets a dedicated platform
set(KYTY_PLATFORM KYTY_PLATFORM_LINUX)
else()
set(KYTY_PLATFORM KYTY_PLATFORM_WINDOWS)
endif()
@@ -61,6 +66,8 @@ if(CMAKE_CXX_COMPILER_ID MATCHES "^(Apple)?Clang$")
if(WIN32 AND KYTY_CXX_COMPILER_NAME STREQUAL "clang-cl")
set(KYTY_CLANG_CL TRUE)
set(KYTY_LINKER LLD_LINK)
elseif(APPLE)
set(KYTY_LINKER LD64) # Apple's default linker; lld flags don't apply
else()
set(KYTY_LINKER LLD)
endif()
@@ -113,6 +120,7 @@ add_custom_target( KytyGitVersion
-D INPUT_FILE=${CMAKE_CURRENT_SOURCE_DIR}/kytyGitVersion.h.in
-D OUTPUT_FILE=${CMAKE_CURRENT_BINARY_DIR}/kytyGitVersion.h
-D GIT_EXECUTABLE=${GIT_EXECUTABLE}
-D GIT_WORKING_DIRECTORY=${CMAKE_CURRENT_SOURCE_DIR}
-P ${CMAKE_CURRENT_SOURCE_DIR}/generate_version.cmake
COMMENT "Generate kytyGitVersion.h"
)
@@ -123,7 +131,7 @@ if (CMAKE_CXX_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0.0)
common
#launcher
)
list(APPEND KYTY_CLANG_TYDY
list(APPEND KYTY_CLANG_TIDY
kyty_emulator
#common
launcher
@@ -150,16 +158,24 @@ file(GLOB kyty_emulator_src CONFIGURE_DEPENDS
graphics/host_gpu/*.h
graphics/host_gpu/renderer/*.cpp
graphics/host_gpu/renderer/*.h
graphics/host_gpu/renderer/cache/*.cpp
graphics/host_gpu/renderer/cache/*.h
graphics/host_gpu/renderer/image/*.cpp
graphics/host_gpu/renderer/image/*.h
graphics/host_gpu/renderer/pipeline/*.cpp
graphics/host_gpu/renderer/pipeline/*.h
graphics/shader/*.cpp
graphics/shader/*.h
graphics/shader/recompiler/*.cpp
graphics/shader/recompiler/*.h
graphics/shader/recompiler/shaderIR/*.cpp
graphics/shader/recompiler/shaderIR/*.h
graphics/shader/recompiler/spirvEmitter/*.cpp
graphics/shader/recompiler/spirvEmitter/*.h
graphics/host_gpu/objects/*.cpp
graphics/host_gpu/objects/*.h
graphics/shader/recompiler/cfg/*.cpp
graphics/shader/recompiler/cfg/*.h
graphics/shader/recompiler/decompiler/*.cpp
graphics/shader/recompiler/decompiler/*.h
graphics/shader/recompiler/emitter/*.cpp
graphics/shader/recompiler/emitter/*.h
graphics/shader/recompiler/ir/*.cpp
graphics/shader/recompiler/ir/*.h
graphics/presentation/*.cpp
graphics/presentation/*.h
graphics/presentation/window/*.cpp
@@ -183,6 +199,9 @@ set(gpu_tiler_shader_names
prt_3d
render_target
depth
promote_d16
demote_d16
swap_bgra16
)
file(GLOB gpu_tiler_shader_includes CONFIGURE_DEPENDS "${gpu_tiler_shader_dir}/gpu_tiler_*.inc")
foreach(shader_name IN LISTS gpu_tiler_shader_names)
@@ -204,6 +223,30 @@ foreach(shader_name IN LISTS gpu_tiler_shader_names)
endforeach()
list(APPEND kyty_emulator_src ${gpu_tiler_shader_headers})
set(gpu_blit_generated_dir "${PROJECT_BINARY_DIR}/gpu_blit_shaders")
set(gpu_blit_shader_sources
"${gpu_tiler_shader_dir}/gpu_blit_fs_triangle.vert"
"${gpu_tiler_shader_dir}/gpu_blit_color_to_ms_depth.frag"
)
foreach(shader_source IN LISTS gpu_blit_shader_sources)
get_filename_component(shader_name "${shader_source}" NAME_WE)
set(shader_spv "${gpu_blit_generated_dir}/${shader_name}.spv")
set(shader_header "${gpu_blit_generated_dir}/${shader_name}_spv.h")
string(TOUPPER "${shader_name}_SPV" shader_symbol)
add_custom_command(
OUTPUT "${shader_header}"
COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_blit_generated_dir}"
COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.0 -Os
"-I${gpu_tiler_shader_dir}" -o "${shader_spv}" "${shader_source}"
COMMAND ${CMAKE_COMMAND} -DINPUT=${shader_spv} -DOUTPUT=${shader_header}
-DSYMBOL=${shader_symbol} -P "${CMAKE_CURRENT_SOURCE_DIR}/embed_spirv.cmake"
DEPENDS "${shader_source}"
VERBATIM
)
list(APPEND gpu_blit_shader_headers "${shader_header}")
endforeach()
list(APPEND kyty_emulator_src ${gpu_blit_shader_headers})
list(APPEND kyty_emulator_src
emulator.h
emulator.cpp
@@ -216,6 +259,26 @@ endif()
set(kyty_emulator_link_libraries common Vulkan::Headers spirv-tools-opt spirv-tools SDL2-static xxhash FFmpeg::ffmpeg fmt::fmt nlohmann_json::nlohmann_json LibAtrac9)
# Linux system libraries required by the static FFmpeg archive.
if(LINUX)
find_package(Threads REQUIRED)
list(APPEND kyty_emulator_link_libraries m ${CMAKE_DL_LIBS} Threads::Threads)
# Optional FFmpeg dependencies.
find_package(ZLIB)
if(ZLIB_FOUND)
list(APPEND kyty_emulator_link_libraries ZLIB::ZLIB)
endif()
find_library(KYTY_BZ2_LIBRARY bz2)
if(KYTY_BZ2_LIBRARY)
list(APPEND kyty_emulator_link_libraries ${KYTY_BZ2_LIBRARY})
endif()
find_library(KYTY_LZMA_LIBRARY lzma)
if(KYTY_LZMA_LIBRARY)
list(APPEND kyty_emulator_link_libraries ${KYTY_LZMA_LIBRARY})
endif()
endif()
set(inc_headers
${CMAKE_CURRENT_SOURCE_DIR}
${KYTY_THIRD_PARTY_DIR}/SDL2/include
@@ -251,13 +314,23 @@ function(add_kyty_full_emulator_test target source)
endif()
endfunction()
function(configure_macos_guest_address_space target)
if(APPLE AND (CMAKE_OSX_ARCHITECTURES STREQUAL "x86_64" OR
(NOT CMAKE_OSX_ARCHITECTURES AND CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64)$")))
target_sources(${target} PRIVATE kernel/macosGuestAddressSpace.cpp)
target_compile_definitions(${target} PRIVATE KYTY_LINKED_GUEST_ADDRESS_SPACE=1)
target_link_options(${target} PRIVATE
-Wl,-ld_classic,-no_pie,-no_fixup_chains,-no_huge,-pagezero_size,0x40000,-segaddr,SYSTEM_MANAGED,0x40000,-segaddr,SYSTEM_RESERVED,0x7ffffc000,-segaddr,USER_AREA,0x7000000000,-image_base,0x700000000000)
endif()
endfunction()
add_kyty_full_emulator_test(shader_cfg_tests ../tests/shaderCfgTests.cpp)
add_executable(scalar_provenance_tests EXCLUDE_FROM_ALL
../tests/ScalarProvenanceTests.cpp
graphics/host_gpu/hostMemory.cpp
graphics/shader/recompiler/ScalarProvenance.cpp
graphics/shader/recompiler/SrtWalker.cpp
graphics/shader/recompiler/ir/ScalarProvenance.cpp
graphics/shader/recompiler/ir/SrtWalker.cpp
)
target_link_libraries(scalar_provenance_tests fmt::fmt)
target_include_directories(scalar_provenance_tests PRIVATE ${inc_headers})
@@ -275,7 +348,6 @@ add_executable(memory_tracker_tests EXCLUDE_FROM_ALL
)
target_link_libraries(memory_tracker_tests fmt::fmt common)
target_include_directories(memory_tracker_tests PRIVATE ${inc_headers})
target_compile_definitions(memory_tracker_tests PRIVATE KYTY_MEMORY_TRACKER_TESTS=1)
add_executable(shader_vertex_metadata_tests EXCLUDE_FROM_ALL
../tests/ShaderVertexMetadataTests.cpp
@@ -289,9 +361,9 @@ add_executable(shader_stage_runtime_tests EXCLUDE_FROM_ALL
graphics/guest_gpu/gpu_format.cpp
graphics/host_gpu/hostMemory.cpp
graphics/shader/shaderStageRuntime.cpp
graphics/shader/recompiler/ResourceMaterialization.cpp
graphics/shader/recompiler/ScalarProvenance.cpp
graphics/shader/recompiler/SrtWalker.cpp
graphics/shader/recompiler/ir/ResourceMaterialization.cpp
graphics/shader/recompiler/ir/ScalarProvenance.cpp
graphics/shader/recompiler/ir/SrtWalker.cpp
)
target_link_libraries(shader_stage_runtime_tests fmt::fmt)
target_include_directories(shader_stage_runtime_tests PRIVATE ${inc_headers})
@@ -300,37 +372,117 @@ add_executable(resource_tracking_tests EXCLUDE_FROM_ALL
../tests/ResourceTrackingTests.cpp
graphics/guest_gpu/gpu_format.cpp
graphics/host_gpu/hostMemory.cpp
graphics/shader/recompiler/ScalarProvenance.cpp
graphics/shader/recompiler/SrtWalker.cpp
graphics/shader/recompiler/SrtPatcher.cpp
graphics/shader/recompiler/ResourceTracking.cpp
graphics/shader/recompiler/ResourceMaterialization.cpp
graphics/shader/recompiler/ShaderInfoCollection.cpp
graphics/shader/recompiler/BindingLayout.cpp
graphics/shader/recompiler/ir/ScalarProvenance.cpp
graphics/shader/recompiler/ir/SrtWalker.cpp
graphics/shader/recompiler/ir/SrtPatcher.cpp
graphics/shader/recompiler/ir/ResourceTracking.cpp
graphics/shader/recompiler/ir/ResourceMaterialization.cpp
graphics/shader/recompiler/ir/ShaderInfoCollection.cpp
graphics/shader/recompiler/ir/BindingLayout.cpp
)
target_link_libraries(resource_tracking_tests fmt::fmt)
target_include_directories(resource_tracking_tests PRIVATE ${inc_headers})
add_executable(resource_mutex_tests EXCLUDE_FROM_ALL
../tests/ResourceMutexTests.cpp
graphics/host_gpu/renderer/resourceMutex.cpp
graphics/host_gpu/renderer/cache/resourceMutex.cpp
)
target_link_libraries(resource_mutex_tests common)
target_include_directories(resource_mutex_tests PRIVATE ${inc_headers})
add_executable(event_queue_lifetime_tests EXCLUDE_FROM_ALL
../tests/EventQueueLifetimeTests.cpp
kernel/eventQueue.cpp
loader/timer.cpp
)
target_link_libraries(event_queue_lifetime_tests common fmt::fmt)
target_include_directories(event_queue_lifetime_tests PRIVATE ${inc_headers})
add_executable(image_page_table_tests EXCLUDE_FROM_ALL
../tests/ImagePageTableTests.cpp
)
target_link_libraries(image_page_table_tests common fmt::fmt)
target_include_directories(image_page_table_tests PRIVATE ${inc_headers})
add_kyty_full_emulator_test(shader_recompiler_compute_tests ../tests/ShaderRecompilerComputeTests.cpp)
set(gpu_test_generated_dir "${PROJECT_BINARY_DIR}/gpu_test_shaders")
set(gpu_test_ms_depth_source
"${gpu_tiler_shader_dir}/gpu_test_ms_depth.comp")
set(gpu_test_ms_depth_spv
"${gpu_test_generated_dir}/gpu_test_ms_depth.spv")
set(gpu_test_ms_depth_header
"${gpu_test_generated_dir}/gpu_test_ms_depth_spv.h")
add_custom_command(
OUTPUT "${gpu_test_ms_depth_header}"
COMMAND ${CMAKE_COMMAND} -E make_directory "${gpu_test_generated_dir}"
COMMAND "${KYTY_GLSLANG_VALIDATOR}" -V --target-env vulkan1.0 -Os
-o "${gpu_test_ms_depth_spv}" "${gpu_test_ms_depth_source}"
COMMAND ${CMAKE_COMMAND} -DINPUT=${gpu_test_ms_depth_spv}
-DOUTPUT=${gpu_test_ms_depth_header} -DSYMBOL=GPU_TEST_MS_DEPTH_SPV
-P "${CMAKE_CURRENT_SOURCE_DIR}/embed_spirv.cmake"
DEPENDS "${gpu_test_ms_depth_source}"
VERBATIM
)
target_sources(shader_recompiler_compute_tests PRIVATE
"${gpu_test_ms_depth_header}")
add_kyty_full_emulator_test(virtual_memory_allocation_tests ../tests/VirtualMemoryAllocationTests.cpp)
target_compile_definitions(virtual_memory_allocation_tests PRIVATE
KYTY_VIRTUAL_MEMORY_ALLOCATION_TESTS=1)
configure_macos_guest_address_space(virtual_memory_allocation_tests)
# These tests use exceptions.
if(NOT KYTY_CLANG_CL)
foreach(kyty_exception_test scalar_provenance_tests resource_tracking_tests
virtual_memory_allocation_tests)
target_compile_options(${kyty_exception_test} PRIVATE -fexceptions)
endforeach()
endif()
if(BUILD_TESTING)
add_test(NAME image_page_table COMMAND $<TARGET_FILE:image_page_table_tests>)
add_test(NAME memory_tracker COMMAND $<TARGET_FILE:memory_tracker_tests>)
add_test(NAME page_manager COMMAND $<TARGET_FILE:page_manager_tests>)
add_test(NAME resource_mutex COMMAND $<TARGET_FILE:resource_mutex_tests>)
add_test(NAME event_queue_lifetime COMMAND $<TARGET_FILE:event_queue_lifetime_tests>)
add_test(NAME shader_recompiler_compute COMMAND $<TARGET_FILE:shader_recompiler_compute_tests>)
add_test(NAME virtual_memory_allocation
COMMAND $<TARGET_FILE:virtual_memory_allocation_tests>)
add_test(NAME command_scheduler_timeline
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --scheduler-only)
add_test(NAME stream_buffer_ring
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --stream-buffer-only)
add_test(NAME gpu_command_lane
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --gpu-command-lane-only)
add_test(NAME gpu_tiler
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --gpu-tiler-only)
add_test(NAME texture_cache_layered_image
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --layered-image-only)
add_test(NAME texture_cache_image_views
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --image-view-cache-only)
add_test(NAME texture_cache_storage_sampled
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --storage-sampled-only)
add_test(NAME texture_cache_depth_readback
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --depth-readback-only)
add_test(NAME buffer_cache_dirty_gc
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --buffer-cache-gc-only)
if(WIN32)
# These tests still depend on the Windows multisample-depth path.
add_test(NAME texture_cache_image_overlap
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --image-overlap-only)
add_test(NAME texture_cache_htile_clear
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --htile-clear-only)
add_test(NAME buffer_cache_ranges
COMMAND $<TARGET_FILE:shader_recompiler_compute_tests> --buffer-cache-range-only)
endif()
endif()
add_executable(kyty_emulator main.cpp ${kyty_emulator_src})
configure_macos_guest_address_space(kyty_emulator)
target_link_libraries(kyty_emulator ${kyty_emulator_link_libraries})
if (WIN32)
@@ -345,6 +497,8 @@ endif()
if (CLANG AND NOT KYTY_CLANG_CL)
target_link_libraries(kyty_emulator pthread)
endif()
# dlopen/dlsym/dladdr for RenderDoc.
target_link_libraries(kyty_emulator ${CMAKE_DL_LIBS})
target_include_directories(kyty_emulator PRIVATE ${inc_headers})
clang_tidy_check(kyty_emulator "" "${check_headers}" "${inc_headers}")
@@ -365,6 +519,18 @@ endif()
add_dependencies(kyty_emulator KytyGitVersion)
if(APPLE)
# The emulator writes x86-64 trampolines/PLT stubs into memory and executes them, and
# runs guest code under Rosetta. That requires the JIT / unsigned-executable-memory
# entitlements, so re-sign the binary after every link (an unsigned/relinked binary
# reverts to the hardened defaults and aborts when it first executes written code).
add_custom_command(TARGET kyty_emulator POST_BUILD
COMMAND codesign -s - --force --options runtime
--entitlements "${CMAKE_CURRENT_SOURCE_DIR}/macos_jit.entitlements"
$<TARGET_FILE:kyty_emulator>
COMMENT "Codesign kyty_emulator with JIT entitlements (macOS)")
endif()
install(TARGETS kyty_emulator DESTINATION .)
if(KYTY_BUILD_LAUNCHER)
add_subdirectory(launcher)
+1
View File
@@ -6,6 +6,7 @@
#include <algorithm>
#include <cstddef> // IWYU pragma: export
#include <cstring>
#include <utility>
#include <vector>
+1
View File
@@ -5,6 +5,7 @@
#include "common/stringUtils.h"
#include <array>
#include <cstring>
namespace Common {
+4
View File
@@ -93,4 +93,8 @@ bool NggRectlistDrawEnabled() {
return g_config->ngg_rectlist_draw_enabled;
}
bool ReadbackLinearImagesEnabled() {
return g_config->readback_linear_images;
}
} // namespace Config
+2
View File
@@ -36,6 +36,7 @@ struct ConfigOptions {
bool spirv_debug_printf_enabled = false;
bool renderdoc_enabled = false;
bool ngg_rectlist_draw_enabled = true;
bool readback_linear_images = false;
};
void Load(const ConfigOptions& cfg);
@@ -64,6 +65,7 @@ bool SpirvDebugPrintfEnabled();
bool RenderDocEnabled();
bool NggRectlistDrawEnabled();
bool ReadbackLinearImagesEnabled();
} // namespace Config
+217 -9
View File
@@ -6,6 +6,14 @@
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
#include <windows.h> // IWYU pragma: keep
#elif defined(__APPLE__)
#include <csignal>
#include <sys/ucontext.h>
#else
#include <csignal>
#include <initializer_list>
#include <ucontext.h> // IWYU pragma: keep
#include <unistd.h>
#endif
// IWYU pragma: no_include <errhandlingapi.h>
@@ -16,7 +24,7 @@
namespace Common::HostException {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
#if !defined(__APPLE__)
static std::atomic<Handler> g_handler {nullptr};
static std::atomic_uint32_t g_install_state {0};
@@ -30,7 +38,9 @@ static_assert(decltype(g_install_state)::is_always_lock_free);
std::fputs(reason != nullptr ? reason : "unspecified", stderr);
std::fputc('\n', stderr);
std::fflush(stderr);
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
TerminateProcess(GetCurrentProcess(), static_cast<UINT>(EXCEPTION_NONCONTINUABLE_EXCEPTION));
#endif
std::_Exit(321);
}
@@ -48,6 +58,21 @@ public:
KYTY_CLASS_NO_COPY(FilterScope);
};
static Handler LoadInstalledHandler() noexcept {
if (g_install_state.load(std::memory_order_acquire) == 0) {
FailFast("host exception handler is not installed");
}
const auto handler = g_handler.load(std::memory_order_acquire);
if (handler == nullptr) {
FailFast("host exception callback is null");
}
return handler;
}
#endif
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
static LONG WINAPI ExceptionFilter(PEXCEPTION_POINTERS exception) {
FilterScope filter_scope;
@@ -106,22 +131,176 @@ static LONG WINAPI ExceptionFilter(PEXCEPTION_POINTERS exception) {
info.r14 = exception->ContextRecord->R14;
info.r15 = exception->ContextRecord->R15;
if (g_install_state.load(std::memory_order_acquire) == 0) {
FailFast("host exception handler is not installed");
const auto handler = LoadInstalledHandler();
return handler(info) ? EXCEPTION_CONTINUE_EXECUTION : EXCEPTION_CONTINUE_SEARCH;
}
#elif defined(__APPLE__)
static std::atomic<Handler> g_handler {nullptr};
static std::atomic_uint32_t g_install_state {0};
static thread_local bool g_in_exception_filter = false;
static_assert(decltype(g_handler)::is_always_lock_free);
static_assert(decltype(g_install_state)::is_always_lock_free);
[[noreturn]] static void FailFast(const char* reason) noexcept {
std::fputs("HostException fail-fast: ", stderr);
std::fputs(reason != nullptr ? reason : "unspecified", stderr);
std::fputc('\n', stderr);
std::fflush(stderr);
std::_Exit(321);
}
// Translate the x86-64 page-fault error code (mcontext __es.__err) into an access type.
// bit 1 (0x2) = write, bit 4 (0x10) = instruction fetch, otherwise a read.
static AccessViolationType DecodeAccess(uint64_t err) {
if ((err & 0x10u) != 0) {
return AccessViolationType::Execute;
}
if ((err & 0x2u) != 0) {
return AccessViolationType::Write;
}
return AccessViolationType::Read;
}
// POSIX signal handler that mirrors the Windows vectored handler: build an ExceptionInfo
// from the mcontext and dispatch. A resolved fault (handler returns true) simply returns,
// re-executing the faulting instruction against the now-fixed protection. An unresolved
// fault restores the default disposition so the retry terminates the process.
static void SignalHandler(int sig, siginfo_t* si, void* uctx) {
if (g_in_exception_filter) {
FailFast("nested exception while resolving a host fault");
}
g_in_exception_filter = true;
auto* uc = static_cast<ucontext_t*>(uctx);
const auto* mc = uc->uc_mcontext;
const auto& ss = mc->__ss;
ExceptionInfo info {};
info.exception_address = ss.__rip;
info.native_code = static_cast<uint32_t>(si->si_code);
info.native_context = uctx;
if (sig == SIGILL) {
info.type = ExceptionType::IllegalInstruction;
} else {
info.type = ExceptionType::AccessViolation;
info.access_violation_type = DecodeAccess(mc->__es.__err);
info.access_violation_vaddr = reinterpret_cast<uint64_t>(si->si_addr);
}
info.rax = ss.__rax;
info.rbx = ss.__rbx;
info.rcx = ss.__rcx;
info.rdx = ss.__rdx;
info.rsi = ss.__rsi;
info.rdi = ss.__rdi;
info.rbp = ss.__rbp;
info.rsp = ss.__rsp;
info.r8 = ss.__r8;
info.r9 = ss.__r9;
info.r10 = ss.__r10;
info.r11 = ss.__r11;
info.r12 = ss.__r12;
info.r13 = ss.__r13;
info.r14 = ss.__r14;
info.r15 = ss.__r15;
const auto handler = g_handler.load(std::memory_order_acquire);
if (handler == nullptr) {
FailFast("host exception callback is null");
}
return handler(info) ? EXCEPTION_CONTINUE_EXECUTION : EXCEPTION_CONTINUE_SEARCH;
const bool resolved = handler(info);
g_in_exception_filter = false;
if (resolved) {
return; // retry the faulting instruction against the fixed mapping
}
// Unresolved: restore the default action so the re-executed instruction terminates.
struct sigaction dfl {};
dfl.sa_handler = SIG_DFL;
sigemptyset(&dfl.sa_mask);
sigaction(sig, &dfl, nullptr);
}
#else
// x86-64 page-fault error bits.
constexpr uint64_t PAGE_FAULT_ERROR_WRITE = 0x02;
constexpr uint64_t PAGE_FAULT_ERROR_INSTRUCTION = 0x10;
// Let the kernel handle an unresolved fault on retry.
static void ChainToDefault(int signal_number) noexcept {
struct sigaction restore {};
restore.sa_handler = SIG_DFL;
sigemptyset(&restore.sa_mask);
restore.sa_flags = 0;
::sigaction(signal_number, &restore, nullptr);
}
static void SignalHandler(int signal_number, siginfo_t* signal_info, void* native_context) {
FilterScope filter_scope;
auto* context = static_cast<ucontext_t*>(native_context);
auto* gregs = context->uc_mcontext.gregs;
ExceptionInfo info {};
info.exception_address = static_cast<uint64_t>(gregs[REG_RIP]);
info.native_code = static_cast<uint32_t>(signal_number);
info.native_context = context;
if (signal_number == SIGSEGV || signal_number == SIGBUS) {
info.type = ExceptionType::AccessViolation;
const auto error_code = static_cast<uint64_t>(gregs[REG_ERR]);
if ((error_code & PAGE_FAULT_ERROR_INSTRUCTION) != 0) {
info.access_violation_type = AccessViolationType::Execute;
} else if ((error_code & PAGE_FAULT_ERROR_WRITE) != 0) {
info.access_violation_type = AccessViolationType::Write;
} else {
info.access_violation_type = AccessViolationType::Read;
}
info.access_violation_vaddr = reinterpret_cast<uint64_t>(signal_info->si_addr);
} else if (signal_number == SIGILL) {
info.type = ExceptionType::IllegalInstruction;
} else {
ChainToDefault(signal_number);
return;
}
info.rax = static_cast<uint64_t>(gregs[REG_RAX]);
info.rbx = static_cast<uint64_t>(gregs[REG_RBX]);
info.rcx = static_cast<uint64_t>(gregs[REG_RCX]);
info.rdx = static_cast<uint64_t>(gregs[REG_RDX]);
info.rsi = static_cast<uint64_t>(gregs[REG_RSI]);
info.rdi = static_cast<uint64_t>(gregs[REG_RDI]);
info.rbp = static_cast<uint64_t>(gregs[REG_RBP]);
info.rsp = static_cast<uint64_t>(gregs[REG_RSP]);
info.r8 = static_cast<uint64_t>(gregs[REG_R8]);
info.r9 = static_cast<uint64_t>(gregs[REG_R9]);
info.r10 = static_cast<uint64_t>(gregs[REG_R10]);
info.r11 = static_cast<uint64_t>(gregs[REG_R11]);
info.r12 = static_cast<uint64_t>(gregs[REG_R12]);
info.r13 = static_cast<uint64_t>(gregs[REG_R13]);
info.r14 = static_cast<uint64_t>(gregs[REG_R14]);
info.r15 = static_cast<uint64_t>(gregs[REG_R15]);
const auto handler = LoadInstalledHandler();
if (handler(info)) {
return;
}
ChainToDefault(signal_number);
}
#endif
bool InstallHandler(Handler handler) {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
if (handler == nullptr) {
return false;
}
@@ -133,19 +312,48 @@ bool InstallHandler(Handler handler) {
g_handler.store(handler, std::memory_order_release);
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
if (AddVectoredExceptionHandler(1, ExceptionFilter) == nullptr) {
g_handler.store(nullptr, std::memory_order_release);
g_install_state.store(0, std::memory_order_release);
printf("AddVectoredExceptionHandler() failed\n");
return false;
}
#elif defined(__APPLE__)
struct sigaction sa {};
sa.sa_sigaction = SignalHandler;
sa.sa_flags = SA_SIGINFO;
sigemptyset(&sa.sa_mask);
// macOS raises SIGBUS for protection faults on some paths and SIGSEGV on others;
// SIGILL covers instructions the host cannot execute (routed to the x64 emulator).
bool ok = sigaction(SIGSEGV, &sa, nullptr) == 0 && sigaction(SIGBUS, &sa, nullptr) == 0 &&
sigaction(SIGILL, &sa, nullptr) == 0;
if (!ok) {
g_handler.store(nullptr, std::memory_order_release);
g_install_state.store(0, std::memory_order_release);
printf("sigaction() failed to install the host fault handler\n");
return false;
}
#else
struct sigaction action {};
action.sa_sigaction = SignalHandler;
sigemptyset(&action.sa_mask);
// Fault resolution needs the normal thread stack.
action.sa_flags = SA_SIGINFO | SA_RESTART;
for (const int signal_number: {SIGSEGV, SIGBUS, SIGILL}) {
if (::sigaction(signal_number, &action, nullptr) != 0) {
g_handler.store(nullptr, std::memory_order_release);
g_install_state.store(0, std::memory_order_release);
printf("sigaction(%d) failed\n", signal_number);
return false;
}
}
#endif
g_install_state.store(2, std::memory_order_release);
return true;
#else
(void)handler;
return false;
#endif
}
} // namespace Common::HostException
+5
View File
@@ -178,6 +178,11 @@ Direction GetDirection() {
return g_direction;
}
bool IsSilent() {
// Before init LOGF must keep writing to stdout, so report non-silent.
return g_initialized && g_direction == Direction::Silent;
}
void Write(std::string_view text) {
WriteImpl(text);
}
+13 -2
View File
@@ -15,6 +15,7 @@ KYTY_SUBSYSTEM_DEFINE(Log);
enum class Direction { Silent, Console, File };
Direction GetDirection();
bool IsSilent();
void Write(std::string_view text);
void Write(fmt::text_style style, std::string_view text);
void WriteFatal(std::string_view text);
@@ -41,8 +42,18 @@ inline constexpr auto BrightWhite = fmt::fg(fmt::terminal_color::bright_white)
} // namespace Log
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define LOGF(...) ::Log::Write(::fmt::sprintf(__VA_ARGS__))
#define LOGF(...) \
do { \
if (!::Log::IsSilent()) { \
::Log::Write(::fmt::sprintf(__VA_ARGS__)); \
} \
} while (false)
// NOLINTNEXTLINE(cppcoreguidelines-macro-usage)
#define LOGF_COLOR(style, ...) ::Log::Write((style), ::fmt::sprintf(__VA_ARGS__))
#define LOGF_COLOR(style, ...) \
do { \
if (!::Log::IsSilent()) { \
::Log::Write((style), ::fmt::sprintf(__VA_ARGS__)); \
} \
} while (false)
#endif /* KYTY_COMMON_LOGGING_LOG_H_ */
+119
View File
@@ -0,0 +1,119 @@
#ifndef EMULATOR_SRC_COMMON_LRUCACHE_H_
#define EMULATOR_SRC_COMMON_LRUCACHE_H_
#include <cstddef>
#include <deque>
#include <type_traits>
#include <utility>
namespace Common {
template <typename Object, typename Tick>
class LeastRecentlyUsedCache {
struct Item {
Object object {};
Tick tick {};
Item* next = nullptr;
Item* prev = nullptr;
};
public:
[[nodiscard]] size_t Insert(Object object, Tick tick) {
const auto id = Build();
auto& item = m_items[id];
item.object = std::move(object);
item.tick = tick;
Attach(item);
return id;
}
void Touch(size_t id, Tick tick) {
auto& item = m_items[id];
if (item.tick >= tick) {
return;
}
item.tick = tick;
if (&item != m_last) {
Detach(item);
Attach(item);
}
}
void Free(size_t id) {
auto& item = m_items[id];
Detach(item);
item.next = nullptr;
item.prev = nullptr;
m_free.push_back(id);
}
template <typename Function>
void ForEachItemBelow(Tick tick, Function&& function) {
constexpr bool ReturnsBool =
std::is_same_v<std::invoke_result_t<Function, Object>, bool>;
for (auto* item = m_first; item != nullptr;) {
if (item->tick > tick) {
return;
}
auto* next = item->next;
if constexpr (ReturnsBool) {
if (function(item->object)) {
return;
}
} else {
function(item->object);
}
item = next;
}
}
private:
[[nodiscard]] size_t Build() {
if (m_free.empty()) {
const auto id = m_items.size();
m_items.emplace_back();
return id;
}
const auto id = m_free.front();
m_free.pop_front();
return id;
}
void Attach(Item& item) {
if (m_first == nullptr) {
m_first = &item;
}
if (m_last == nullptr) {
m_last = &item;
return;
}
item.prev = m_last;
m_last->next = &item;
item.next = nullptr;
m_last = &item;
}
void Detach(Item& item) {
if (item.prev != nullptr) {
item.prev->next = item.next;
}
if (item.next != nullptr) {
item.next->prev = item.prev;
}
if (m_first == &item) {
m_first = item.next;
}
if (m_last == &item) {
m_last = item.prev;
}
}
std::deque<Item> m_items;
std::deque<size_t> m_free;
Item* m_first = nullptr;
Item* m_last = nullptr;
};
} // namespace Common
#endif // EMULATOR_SRC_COMMON_LRUCACHE_H_
+4
View File
@@ -23,6 +23,10 @@ struct sys_dbg_stack_info_t {
size_t commited_size;
size_t total_size;
size_t code_size;
// Full stack reservation reported by pthread.
uintptr_t reserved_addr;
size_t reserved_size;
#endif
};
+79 -4
View File
@@ -8,12 +8,57 @@
#include <cstdlib>
#include <cstring>
#include <execinfo.h>
#include <pthread.h>
#include <sys/param.h>
#include <sys/types.h>
#include <unistd.h>
#if defined(__APPLE__)
#include <libgen.h> // POSIX basename() lives here on macOS, not in <cstring>
#endif
void SysStackWalk(void** /*stack*/, int* depth) {
*depth = 0;
// Avoid unwinding a guest-owned stack.
static bool OnOwnStack() {
const char* probe = reinterpret_cast<const char*>(&probe);
pthread_attr_t attr {};
#if defined(__APPLE__)
const auto* top = static_cast<const char*>(pthread_get_stackaddr_np(pthread_self()));
const auto size = pthread_get_stacksize_np(pthread_self());
(void)attr;
return top != nullptr && size != 0 && probe < top && probe >= top - size;
#else
if (pthread_getattr_np(pthread_self(), &attr) != 0) {
return false;
}
void* base = nullptr;
size_t size = 0;
const bool ok =
pthread_attr_getstack(&attr, &base, &size) == 0 && base != nullptr && size != 0;
pthread_attr_destroy(&attr);
if (!ok) {
return false;
}
const auto* low = static_cast<const char*>(base);
return probe >= low && probe < low + size;
#endif
}
void SysStackWalk(void** stack, int* depth) {
if (stack == nullptr || depth == nullptr || *depth <= 0) {
if (depth != nullptr) {
*depth = 0;
}
return;
}
if (!OnOwnStack()) {
*depth = 0;
return;
}
const int n = ::backtrace(stack, *depth);
*depth = (n < 0 ? 0 : n);
}
void SysStackUsagePrint(sys_dbg_stack_info_t& stack) {
@@ -30,6 +75,33 @@ void SysStackUsage(sys_dbg_stack_info_t& s) {
[[maybe_unused]] int result = 0;
memset(&s, 0, sizeof(sys_dbg_stack_info_t));
// Record the reservation before the Linux /proc walk.
{
pthread_attr_t self_attr {};
#if defined(__APPLE__)
void* stack_top = pthread_get_stackaddr_np(pthread_self());
const size_t stack_size = pthread_get_stacksize_np(pthread_self());
if (stack_top != nullptr && stack_size != 0) {
s.reserved_addr = reinterpret_cast<uintptr_t>(stack_top) - stack_size;
s.reserved_size = stack_size;
}
(void)self_attr;
#else
if (pthread_getattr_np(pthread_self(), &self_attr) == 0) {
void* stack_base = nullptr;
size_t stack_size = 0;
if (pthread_attr_getstack(&self_attr, &stack_base, &stack_size) == 0 &&
stack_base != nullptr && stack_size != 0) {
s.reserved_addr = reinterpret_cast<uintptr_t>(stack_base);
s.reserved_size = stack_size;
}
pthread_attr_destroy(&self_attr);
}
#endif
}
char str[1024];
char str2[1024];
result = sprintf(str, "/proc/%d/exe", static_cast<int>(pid));
@@ -43,8 +115,6 @@ void SysStackUsage(sys_dbg_stack_info_t& s) {
result = sprintf(str, "/proc/%d/maps", static_cast<int>(pid));
memset(&s, 0, sizeof(sys_dbg_stack_info_t));
FILE* f = fopen(str, "r");
if (f == nullptr) {
@@ -118,6 +188,11 @@ void SysStackUsage(sys_dbg_stack_info_t& s) {
}
result = fclose(f);
if (s.reserved_addr == 0) {
s.reserved_addr = s.addr;
s.reserved_size = s.total_size;
}
}
#endif
+193 -26
View File
@@ -11,7 +11,11 @@
#include <cerrno>
#include <cstdlib>
#include <dirent.h>
#include <fcntl.h>
#include <filesystem>
#include <sys/stat.h>
#include <system_error>
#include <unistd.h>
#include <utime.h>
@@ -39,10 +43,43 @@ struct sys_file_t {
};
};
// Darwin uses BSD timestamp member names.
#if defined(__APPLE__)
#define KYTY_STAT_ATIME_NS(st) ((st).st_atimespec.tv_nsec)
#define KYTY_STAT_MTIME_NS(st) ((st).st_mtimespec.tv_nsec)
#else
#define KYTY_STAT_ATIME_NS(st) ((st).st_atim.tv_nsec)
#define KYTY_STAT_MTIME_NS(st) ((st).st_mtim.tv_nsec)
#endif
static std::filesystem::path get_internal_name(const std::filesystem::path& name) {
return name.is_absolute() ? name : (std::filesystem::path(".") / name);
}
// Pass access-pattern hints to the host.
static void apply_cache_hint(FILE* f, sys_file_cache_type_t cache_type) {
if (f == nullptr) {
return;
}
#if !defined(__APPLE__)
int advice = POSIX_FADV_NORMAL;
switch (cache_type) {
case SYS_FILE_CACHE_RANDOM_ACCESS: advice = POSIX_FADV_RANDOM; break;
case SYS_FILE_CACHE_SEQUENTIAL_SCAN: advice = POSIX_FADV_SEQUENTIAL; break;
case SYS_FILE_CACHE_AUTO:
default: return;
}
::posix_fadvise(fileno(f), 0, 0, advice);
#else
if (cache_type == SYS_FILE_CACHE_SEQUENTIAL_SCAN) {
::fcntl(fileno(f), F_RDAHEAD, 1);
} else if (cache_type == SYS_FILE_CACHE_RANDOM_ACCESS) {
::fcntl(fileno(f), F_RDAHEAD, 0);
}
#endif
}
void SysFileRead(void* data, uint32_t size, sys_file_t& f, uint32_t* bytes_read) {
if (f.type == SYS_FILE_FILE) {
size_t w = fread(data, 1, size, f.f);
@@ -136,7 +173,7 @@ sys_file_t* SysFileCreate(const std::filesystem::path& file_name) {
}
sys_file_t* SysFileOpenR(const std::filesystem::path& file_name,
sys_file_cache_type_t /*cache_type*/) {
sys_file_cache_type_t cache_type) {
auto* ret = new sys_file_t;
ret->type = SYS_FILE_FILE;
@@ -150,6 +187,8 @@ sys_file_t* SysFileOpenR(const std::filesystem::path& file_name,
ret->type = SYS_FILE_ERROR;
}
apply_cache_hint(f, cache_type);
ret->f = f;
return ret;
@@ -180,7 +219,7 @@ sys_file_t* SysFileCreate() {
}
sys_file_t* SysFileOpenW(const std::filesystem::path& file_name,
sys_file_cache_type_t /*cache_type*/) {
sys_file_cache_type_t cache_type) {
auto* ret = new sys_file_t;
auto real_name = get_internal_name(file_name);
@@ -194,13 +233,15 @@ sys_file_t* SysFileOpenW(const std::filesystem::path& file_name,
ret->type = SYS_FILE_FILE;
}
apply_cache_hint(f, cache_type);
ret->f = f;
return ret;
}
sys_file_t* SysFileOpenRw(const std::filesystem::path& file_name,
sys_file_cache_type_t /*cache_type*/) {
sys_file_cache_type_t cache_type) {
auto* ret = new sys_file_t;
auto real_name = get_internal_name(file_name);
@@ -214,6 +255,8 @@ sys_file_t* SysFileOpenRw(const std::filesystem::path& file_name,
ret->type = SYS_FILE_FILE;
}
apply_cache_hint(f, cache_type);
ret->f = f;
return ret;
@@ -239,11 +282,17 @@ uint64_t SysFileSize(sys_file_t& f) {
[[maybe_unused]] int result = 0;
if (f.type == SYS_FILE_FILE) {
uint32_t pos = ftell(f.f);
result = fseek(f.f, 0, SEEK_END);
uint32_t size = ftell(f.f);
result = fseek(f.f, pos, SEEK_SET);
return size;
// Preserve sizes above 4 GiB.
const off_t pos = ftello(f.f);
if (pos < 0) {
return 0;
}
if (fseeko(f.f, 0, SEEK_END) != 0) {
return 0;
}
const off_t size = ftello(f.f);
result = fseeko(f.f, pos, SEEK_SET);
return (size < 0 ? 0 : static_cast<uint64_t>(size));
}
if (f.type == SYS_FILE_MEMORY_STAT || f.type == SYS_FILE_MEMORY_DYN) {
@@ -260,8 +309,18 @@ uint64_t SysFileSize(const std::filesystem::path& file_name) {
return size;
}
bool SysFileTruncate(sys_file_t& /*f*/, uint64_t /*size*/) {
return false;
bool SysFileTruncate(sys_file_t& f, uint64_t size) {
bool ok = false;
if (f.type == SYS_FILE_FILE) {
// Flush before resizing and restore the caller's position.
const auto position = ftell(f.f);
fflush(f.f);
ok = (ftruncate(fileno(f.f), static_cast<off_t>(size)) == 0);
if (position >= 0) {
fseek(f.f, position, SEEK_SET);
}
}
return ok;
}
bool SysFileUnlink(sys_file_t& /*f*/, const std::filesystem::path& name) {
@@ -382,6 +441,7 @@ SysFileTimeStruct SysFileGetLastAccessTimeUtc(const std::filesystem::path& name)
} else {
r.is_invalid = false;
r.time = s.st_atime;
r.nanos = KYTY_STAT_ATIME_NS(s);
}
return r;
@@ -400,6 +460,7 @@ SysFileTimeStruct SysFileGetLastWriteTimeUtc(const std::filesystem::path& name)
} else {
r.is_invalid = false;
r.time = s.st_mtime;
r.nanos = KYTY_STAT_MTIME_NS(s);
}
return r;
@@ -419,13 +480,36 @@ void SysFileGetLastAccessAndWriteTimeUtc(const std::filesystem::path& name, SysF
a.is_invalid = false;
w.is_invalid = false;
a.time = s.st_atime;
a.nanos = KYTY_STAT_ATIME_NS(s);
w.time = s.st_mtime;
w.nanos = KYTY_STAT_MTIME_NS(s);
}
}
void SysFileGetLastAccessAndWriteTimeUtc(sys_file_t& /*f*/, SysFileTimeStruct& /*a*/,
SysFileTimeStruct& /*w*/) {
EXIT("not implemented\n");
void SysFileGetLastAccessAndWriteTimeUtc(sys_file_t& f, SysFileTimeStruct& a,
SysFileTimeStruct& w) {
if (f.type == SYS_FILE_FILE) {
struct stat s {};
const bool ok = (0 == fstat(fileno(f.f), &s));
a.is_invalid = w.is_invalid = !ok;
if (ok) {
a.time = s.st_atime;
a.nanos = KYTY_STAT_ATIME_NS(s);
w.time = s.st_mtime;
w.nanos = KYTY_STAT_MTIME_NS(s);
}
} else if (f.type == SYS_FILE_MEMORY_STAT || f.type == SYS_FILE_MEMORY_DYN) {
// Memory-backed files use the current time.
SysTimeStruct t {};
SysGetSystemTimeUtc(t);
SysSystemToFileTimeUtc(t, a);
SysSystemToFileTimeUtc(t, w);
} else {
a.is_invalid = w.is_invalid = true;
}
}
bool SysFileSetLastAccessTimeUtc(const std::filesystem::path& name, SysFileTimeStruct& access) {
@@ -531,27 +615,110 @@ bool SysFileSetLastAccessAndWriteTimeUtc(const std::filesystem::path& name,
// }
}
void SysFileFindFiles(const std::filesystem::path& /*path*/,
std::vector<sys_file_find_t>& /*out*/) {
EXIT("not implemented\n");
// Recursively collect regular files.
void SysFileFindFiles(const std::filesystem::path& path, std::vector<sys_file_find_t>& out) {
auto real_path = get_internal_name(path);
DIR* dir = opendir(real_path.string().c_str());
if (dir == nullptr) {
return;
}
for (const dirent* entry = readdir(dir); entry != nullptr; entry = readdir(dir)) {
const std::string file_name(entry->d_name);
if (file_name == "." || file_name == "..") {
continue;
}
auto child = real_path / file_name;
struct stat s {};
// lstat, so a symlink is never followed into a cycle during the recursive walk.
if (0 != lstat(child.string().c_str(), &s)) {
continue;
}
if (S_ISDIR(s.st_mode)) {
SysFileFindFiles(child, out);
} else if (S_ISREG(s.st_mode)) {
sys_file_find_t r {};
r.path_with_name = child;
r.size = static_cast<uint64_t>(s.st_size);
r.last_access_time.is_invalid = false;
r.last_access_time.time = s.st_atime;
r.last_access_time.nanos = KYTY_STAT_ATIME_NS(s);
r.last_write_time.is_invalid = false;
r.last_write_time.time = s.st_mtime;
r.last_write_time.nanos = KYTY_STAT_MTIME_NS(s);
out.push_back(r);
}
}
closedir(dir);
}
void SysFileGetDents(const std::filesystem::path& /*path*/, std::vector<sys_dir_entry_t>& /*out*/) {
EXIT("not implemented\n");
// Keep "." and ".." to match FindFirstFileW.
void SysFileGetDents(const std::filesystem::path& path, std::vector<sys_dir_entry_t>& out) {
auto real_path = get_internal_name(path);
DIR* dir = opendir(real_path.string().c_str());
if (dir == nullptr) {
return;
}
for (const dirent* entry = readdir(dir); entry != nullptr; entry = readdir(dir)) {
sys_dir_entry_t r {};
r.name = entry->d_name;
if (entry->d_type == DT_UNKNOWN) {
// Some filesystems do not populate d_type.
struct stat s {};
r.is_file = 0 == lstat((real_path / r.name).string().c_str(), &s) && S_ISREG(s.st_mode);
} else {
r.is_file = entry->d_type != DT_DIR;
}
out.push_back(r);
}
closedir(dir);
}
bool SysFileCopyFile(const std::filesystem::path& /*src*/, const std::filesystem::path& /*dst*/) {
EXIT("not implemented\n");
return false;
bool SysFileCopyFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
std::error_code error;
return std::filesystem::copy_file(get_internal_name(src), get_internal_name(dst),
std::filesystem::copy_options::overwrite_existing, error) &&
!error;
}
bool SysFileMoveFile(const std::filesystem::path& /*src*/, const std::filesystem::path& /*dst*/) {
EXIT("not implemented\n");
return false;
bool SysFileMoveFile(const std::filesystem::path& src, const std::filesystem::path& dst) {
auto real_src = get_internal_name(src);
auto real_dst = get_internal_name(dst);
// Match MoveFileW: fail when the destination exists.
std::error_code error;
if (std::filesystem::exists(real_dst, error)) {
return false;
}
return 0 == rename(real_src.string().c_str(), real_dst.string().c_str());
}
void SysFileRemoveReadonly(const std::filesystem::path& /*name*/) {
EXIT("not implemented\n");
void SysFileRemoveReadonly(const std::filesystem::path& name) {
auto real_name = get_internal_name(name);
auto real_name_str = real_name.string();
struct stat s {};
if (0 != stat(real_name_str.c_str(), &s)) {
return;
}
chmod(real_name_str.c_str(), s.st_mode | S_IWUSR);
}
#endif
+209 -33
View File
@@ -8,9 +8,16 @@
#include "common/platform/sysVirtual.h"
#include "common/virtualMemory.h"
#include <atomic>
#include <map>
#include <pthread.h>
#include <sys/mman.h>
#include <unistd.h>
#if defined(__APPLE__)
#include <mach/mach.h>
#include <mach/mach_vm.h>
#endif
// IWYU pragma: no_include <asm/mman-common.h>
// IWYU pragma: no_include <asm/mman.h>
@@ -31,8 +38,8 @@ void SysVirtualInit() {
pthread_mutexattr_t attr {};
pthread_mutexattr_init(&attr);
#if KYTY_PLATFORM == KYTY_PLATFORM_LINUX
pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_FAST_NP);
#if KYTY_PLATFORM == KYTY_PLATFORM_LINUX && !defined(__APPLE__)
pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_FAST_NP); // glibc-only fast mutex
#else
pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_NORMAL);
#endif
@@ -76,6 +83,70 @@ static VirtualMemory::Mode get_protection_flag(int mode) {
}
}
// Keep automatic mappings inside the guest and GPU-addressable low window.
#ifdef KYTY_FIXED_NOREPLACE
static constexpr uintptr_t LOW_ARENA_LIMIT = 0x000000FC00000000ULL; // libc mspace window ceiling
static constexpr uintptr_t LOW_ARENA_FLOOR = 0x000000A000000000ULL; // 640 GiB
static constexpr uintptr_t LOW_ARENA_GRAIN = 0x0000000000010000ULL; // 64 KiB
static_assert(LOW_ARENA_LIMIT <= 0x0000010000000000ULL,
"arena must stay inside the GPU page tracker's 1<<40 window");
static_assert(LOW_ARENA_FLOOR < LOW_ARENA_LIMIT, "arena floor must sit below its ceiling");
static std::atomic<uintptr_t> g_low_arena_next {LOW_ARENA_LIMIT};
#endif
// Caller holds g_virtual_mutex.
static void record_alloc(uintptr_t addr, size_t size) {
auto next = g_allocs->upper_bound(addr);
if (next != g_allocs->begin()) {
auto it = std::prev(next);
const auto alloc_addr = it->first;
const auto alloc_end = alloc_addr + it->second;
if (alloc_addr <= addr && addr + size <= alloc_end) {
g_allocs->erase(it);
if (alloc_addr < addr) {
(*g_allocs)[alloc_addr] = addr - alloc_addr;
}
if (addr + size < alloc_end) {
(*g_allocs)[addr + size] = alloc_end - (addr + size);
}
}
}
(*g_allocs)[addr] = size;
}
#ifdef KYTY_FIXED_NOREPLACE
static uintptr_t align_up_to(uintptr_t addr, uint64_t alignment) {
return (addr + alignment - 1) & ~(alignment - 1);
}
#endif
// Freed arena addresses are not reused while GPU caches remain keyed by address.
static void* map_anonymous(uintptr_t addr, size_t size, int protect, int flags) {
if (addr != 0) {
return mmap(reinterpret_cast<void*>(addr), size, protect, flags, -1, 0); // NOLINT
}
#ifdef KYTY_FIXED_NOREPLACE
const auto step = align_up_to(size, LOW_ARENA_GRAIN);
for (int attempt = 0; attempt < 256; attempt++) {
const auto top = g_low_arena_next.fetch_sub(step, std::memory_order_relaxed);
if (top < step || top - step < LOW_ARENA_FLOOR) {
break;
}
const auto hint = (top - step) & ~(LOW_ARENA_GRAIN - 1);
void* ptr = mmap(reinterpret_cast<void*>(hint), size, protect,
flags | MAP_FIXED_NOREPLACE, -1, 0); // NOLINT
if (ptr != MAP_FAILED) {
return ptr;
}
}
#endif
return mmap(nullptr, size, protect, flags, -1, 0); // NOLINT
}
uint64_t SysVirtualAlloc(uint64_t address, uint64_t size, VirtualMemory::Mode mode) {
EXIT_IF(g_allocs == nullptr);
@@ -83,14 +154,13 @@ uint64_t SysVirtualAlloc(uint64_t address, uint64_t size, VirtualMemory::Mode mo
int protect = get_protection_flag(mode);
void* ptr =
mmap(reinterpret_cast<void*>(addr), size, protect, MAP_PRIVATE | MAP_ANON, -1, 0); // NOLINT
void* ptr = map_anonymous(addr, size, protect, MAP_PRIVATE | MAP_ANON);
auto ret_addr = reinterpret_cast<uintptr_t>(ptr);
if (ptr != MAP_FAILED) {
pthread_mutex_lock(&g_virtual_mutex);
(*g_allocs)[ret_addr] = size;
record_alloc(ret_addr, size);
uintptr_t page_start = ret_addr >> 12u;
uintptr_t page_end = (ret_addr + size - 1) >> 12u;
for (uintptr_t page = page_start; page <= page_end; page++) {
@@ -117,18 +187,43 @@ uint64_t SysVirtualAllocAligned(uint64_t address, uint64_t size, VirtualMemory::
auto addr = static_cast<uintptr_t>(address);
int protect = get_protection_flag(mode);
void* ptr =
mmap(reinterpret_cast<void*>(addr), size, protect, MAP_PRIVATE | MAP_ANON, -1, 0); // NOLINT
void* ptr = map_anonymous(addr, size, protect, MAP_PRIVATE | MAP_ANON);
auto ret_addr = reinterpret_cast<uintptr_t>(ptr);
if (ptr != MAP_FAILED && ((ret_addr & (alignment - 1)) != 0)) {
munmap(ptr, size);
ptr = mmap(reinterpret_cast<void*>(addr), size + alignment, protect,
MAP_PRIVATE | MAP_ANON | MAP_NORESERVE, -1, 0); // NOLINT
ptr = map_anonymous(addr, size + alignment, protect,
MAP_PRIVATE | MAP_ANON | MAP_NORESERVE);
ret_addr = reinterpret_cast<uintptr_t>(ptr);
if (ptr != MAP_FAILED) {
#if defined(__APPLE__)
// Carve the aligned subrange out of the live mapping with MAP_FIXED (in-place
// replacement) and trim the slack; never munmap the whole range first, or a
// concurrent host mapping (dyld, Rosetta, Metal) could claim the hole and be
// destroyed by the MAP_FIXED. Other platforms keep the original path below.
auto aligned_addr = align_up(ret_addr, alignment);
// NOLINTNEXTLINE
void* fixed = mmap(reinterpret_cast<void*>(aligned_addr), size, protect,
MAP_FIXED | MAP_PRIVATE | MAP_ANON, -1, 0);
if (fixed == MAP_FAILED) {
munmap(ptr, size + alignment);
ret_addr = 0;
ptr = MAP_FAILED;
} else {
if (aligned_addr > ret_addr) {
munmap(reinterpret_cast<void*>(ret_addr), aligned_addr - ret_addr);
}
const uintptr_t tail_start = aligned_addr + size;
const uintptr_t resv_end = ret_addr + size + alignment;
if (resv_end > tail_start) {
munmap(reinterpret_cast<void*>(tail_start), resv_end - tail_start);
}
ptr = fixed;
ret_addr = aligned_addr;
}
#else
munmap(ptr, size + alignment);
auto aligned_addr = align_up(ret_addr, alignment);
#ifdef KYTY_FIXED_NOREPLACE
@@ -146,6 +241,7 @@ uint64_t SysVirtualAllocAligned(uint64_t address, uint64_t size, VirtualMemory::
ret_addr = 0;
ptr = MAP_FAILED;
}
#endif
}
}
@@ -154,7 +250,7 @@ uint64_t SysVirtualAllocAligned(uint64_t address, uint64_t size, VirtualMemory::
}
pthread_mutex_lock(&g_virtual_mutex);
(*g_allocs)[ret_addr] = size;
record_alloc(ret_addr, size);
uintptr_t page_start = ret_addr >> 12u;
uintptr_t page_end = (ret_addr + size - 1) >> 12u;
for (uintptr_t page = page_start; page <= page_end; page++) {
@@ -165,6 +261,27 @@ uint64_t SysVirtualAllocAligned(uint64_t address, uint64_t size, VirtualMemory::
return ret_addr;
}
#if defined(__APPLE__)
// macOS has no /proc/self/maps; query the Mach VM map directly. mach_vm_region returns
// the first mapped region at or above `region_addr`; if it begins before the end of the
// requested range, the range overlaps an existing mapping.
static bool is_mapped(void* ptr, size_t length) {
auto query_addr = reinterpret_cast<mach_vm_address_t>(ptr);
mach_vm_address_t region_addr = query_addr;
mach_vm_size_t region_size = 0;
vm_region_basic_info_data_64_t info {};
mach_msg_type_number_t count = VM_REGION_BASIC_INFO_COUNT_64;
mach_port_t object_name = MACH_PORT_NULL;
kern_return_t kr =
mach_vm_region(mach_task_self(), &region_addr, &region_size, VM_REGION_BASIC_INFO_64,
reinterpret_cast<vm_region_info_t>(&info), &count, &object_name);
if (kr != KERN_SUCCESS) {
return false; // no region at or above the address → unmapped
}
return region_addr < (query_addr + length);
}
#else
static bool is_mapped(void* ptr, size_t length) {
FILE* file = fopen("/proc/self/maps", "r");
char line[1024];
@@ -189,6 +306,7 @@ static bool is_mapped(void* ptr, size_t length) {
fclose(file);
return ret;
}
#endif
bool SysVirtualAllocFixed(uint64_t address, uint64_t size, VirtualMemory::Mode mode) {
EXIT_IF(g_allocs == nullptr);
@@ -218,7 +336,7 @@ bool SysVirtualAllocFixed(uint64_t address, uint64_t size, VirtualMemory::Mode m
if (ptr != MAP_FAILED) {
pthread_mutex_lock(&g_virtual_mutex);
(*g_allocs)[ret_addr] = size;
record_alloc(ret_addr, size);
uintptr_t page_start = ret_addr >> 12u;
uintptr_t page_end = (ret_addr + size - 1) >> 12u;
for (uintptr_t page = page_start; page <= page_end; page++) {
@@ -249,18 +367,44 @@ uint64_t SysVirtualReserveAligned(uint64_t address, uint64_t size, uint64_t alig
auto addr = static_cast<uintptr_t>(address);
void* ptr = mmap(reinterpret_cast<void*>(addr), size, PROT_NONE,
MAP_PRIVATE | MAP_ANON | MAP_NORESERVE, -1, 0); // NOLINT
void* ptr = map_anonymous(addr, size, PROT_NONE, MAP_PRIVATE | MAP_ANON | MAP_NORESERVE);
auto ret_addr = reinterpret_cast<uintptr_t>(ptr);
if (ptr != MAP_FAILED && ((ret_addr & (alignment - 1)) != 0)) {
munmap(ptr, size);
ptr = mmap(reinterpret_cast<void*>(addr), size + alignment, PROT_NONE,
MAP_PRIVATE | MAP_ANON | MAP_NORESERVE, -1, 0); // NOLINT
ptr = map_anonymous(addr, size + alignment, PROT_NONE,
MAP_PRIVATE | MAP_ANON | MAP_NORESERVE);
ret_addr = reinterpret_cast<uintptr_t>(ptr);
if (ptr != MAP_FAILED) {
#if defined(__APPLE__)
// Carve the aligned subrange out of the live reservation with MAP_FIXED (an
// in-place replacement), then trim the slack. The range must never be
// returned to the OS in between: another thread (dyld, Rosetta, Metal,
// malloc) could claim the hole, and the subsequent MAP_FIXED would silently
// destroy its mapping. Other platforms keep the original path below.
auto aligned_addr = align_up(ret_addr, alignment);
// NOLINTNEXTLINE
void* fixed = mmap(reinterpret_cast<void*>(aligned_addr), size, PROT_NONE,
MAP_FIXED | MAP_PRIVATE | MAP_ANON | MAP_NORESERVE, -1, 0);
if (fixed == MAP_FAILED) {
munmap(ptr, size + alignment);
ret_addr = 0;
ptr = MAP_FAILED;
} else {
if (aligned_addr > ret_addr) {
munmap(reinterpret_cast<void*>(ret_addr), aligned_addr - ret_addr);
}
const uintptr_t tail_start = aligned_addr + size;
const uintptr_t resv_end = ret_addr + size + alignment;
if (resv_end > tail_start) {
munmap(reinterpret_cast<void*>(tail_start), resv_end - tail_start);
}
ptr = fixed;
ret_addr = aligned_addr;
}
#else
munmap(ptr, size + alignment);
auto aligned_addr = align_up(ret_addr, alignment);
#ifdef KYTY_FIXED_NOREPLACE
@@ -278,6 +422,7 @@ uint64_t SysVirtualReserveAligned(uint64_t address, uint64_t size, uint64_t alig
ret_addr = 0;
ptr = MAP_FAILED;
}
#endif
}
}
@@ -286,12 +431,7 @@ uint64_t SysVirtualReserveAligned(uint64_t address, uint64_t size, uint64_t alig
}
pthread_mutex_lock(&g_virtual_mutex);
(*g_allocs)[ret_addr] = size;
uintptr_t page_start = ret_addr >> 12u;
uintptr_t page_end = (ret_addr + size - 1) >> 12u;
for (uintptr_t page = page_start; page <= page_end; page++) {
(*g_protects)[page] = PROT_NONE;
}
record_alloc(ret_addr, size);
pthread_mutex_unlock(&g_virtual_mutex);
return ret_addr;
@@ -324,12 +464,7 @@ bool SysVirtualReserveFixed(uint64_t address, uint64_t size) {
if (ptr != MAP_FAILED) {
pthread_mutex_lock(&g_virtual_mutex);
(*g_allocs)[ret_addr] = size;
uintptr_t page_start = ret_addr >> 12u;
uintptr_t page_end = (ret_addr + size - 1) >> 12u;
for (uintptr_t page = page_start; page <= page_end; page++) {
(*g_protects)[page] = PROT_NONE;
}
record_alloc(ret_addr, size);
pthread_mutex_unlock(&g_virtual_mutex);
return true;
@@ -339,7 +474,29 @@ bool SysVirtualReserveFixed(uint64_t address, uint64_t size) {
}
bool SysVirtualDecommit(uint64_t address, uint64_t size) {
return SysVirtualProtect(address, size, VirtualMemory::Mode::NoAccess);
// Drop physical pages while preserving the reservation.
if (!SysVirtualProtect(address, size, VirtualMemory::Mode::NoAccess)) {
return false;
}
if (size != 0) {
#if defined(__APPLE__)
constexpr int RECLAIM_ADVICE = MADV_FREE;
#else
constexpr int RECLAIM_ADVICE = MADV_DONTNEED;
#endif
const auto page_size = static_cast<uintptr_t>(sysconf(_SC_PAGESIZE));
if (page_size != 0) {
// Do not discard pages outside the requested range.
const auto begin = (static_cast<uintptr_t>(address) + page_size - 1) & ~(page_size - 1);
const auto end = (static_cast<uintptr_t>(address) + size) & ~(page_size - 1);
if (end > begin) {
::madvise(reinterpret_cast<void*>(begin), end - begin, RECLAIM_ADVICE);
}
}
}
return true;
}
bool SysVirtualFree(uint64_t address) {
@@ -391,15 +548,34 @@ bool SysVirtualFreeRange(uint64_t address, uint64_t size) {
pthread_mutex_unlock(&g_virtual_mutex);
return false;
}
auto it = std::prev(next);
const auto alloc_addr = it->first;
const auto alloc_end = alloc_addr + it->second;
if (addr < alloc_addr || end > alloc_end || munmap(reinterpret_cast<void*>(addr), size) != 0) {
// A reservation may have been split into several adjacent records.
auto first = std::prev(next);
const auto alloc_addr = first->first;
if (addr < alloc_addr || alloc_addr + first->second <= addr) {
pthread_mutex_unlock(&g_virtual_mutex);
return false;
}
g_allocs->erase(it);
auto last = first;
uintptr_t cursor = alloc_addr + first->second;
while (cursor < end) {
auto following = std::next(last);
if (following == g_allocs->end() || following->first != cursor) {
pthread_mutex_unlock(&g_virtual_mutex);
return false;
}
last = following;
cursor = following->first + following->second;
}
const auto alloc_end = cursor;
if (munmap(reinterpret_cast<void*>(addr), size) != 0) {
pthread_mutex_unlock(&g_virtual_mutex);
return false;
}
g_allocs->erase(first, std::next(last));
if (alloc_addr < addr) {
(*g_allocs)[alloc_addr] = addr - alloc_addr;
}
+11 -7
View File
@@ -27,7 +27,9 @@ struct SysFileTimeStruct {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
FILETIME time;
#elif KYTY_PLATFORM == KYTY_PLATFORM_LINUX
// Nanoseconds preserve sub-second file timestamps.
time_t time;
long nanos;
#endif
bool is_invalid;
};
@@ -139,7 +141,7 @@ inline void SysFileToSystemTimeUtc(const SysFileTimeStruct& f, SysTimeStruct& t)
t.Hour = i.tm_hour;
t.Minute = i.tm_min;
t.Second = (i.tm_sec == 60 ? 59 : i.tm_sec);
t.Milliseconds = 0;
t.Milliseconds = static_cast<uint16_t>((f.nanos / 1000000) % 1000);
}
inline void SysTimeTToSystem(time_t t, SysTimeStruct& s) {
@@ -168,10 +170,11 @@ inline void SysSystemToFileTimeUtc(const SysTimeStruct& f, SysFileTimeStruct& t)
// Retrieves the current local date and time.
inline void SysGetSystemTime(SysTimeStruct& t) {
time_t st {};
// Preserve millisecond precision.
timespec now {};
struct tm i {};
if (time(&st) == static_cast<time_t>(-1) || localtime_r(&st, &i) == nullptr) {
if (clock_gettime(CLOCK_REALTIME, &now) != 0 || localtime_r(&now.tv_sec, &i) == nullptr) {
t.is_invalid = true;
return;
}
@@ -183,15 +186,16 @@ inline void SysGetSystemTime(SysTimeStruct& t) {
t.Hour = i.tm_hour;
t.Minute = i.tm_min;
t.Second = (i.tm_sec == 60 ? 59 : i.tm_sec);
t.Milliseconds = 0;
t.Milliseconds = static_cast<uint16_t>((now.tv_nsec / 1000000) % 1000);
}
// Retrieves the current system date and time in Coordinated Universal Time (UTC).
inline void SysGetSystemTimeUtc(SysTimeStruct& t) {
time_t st {};
// Preserve millisecond precision.
timespec now {};
struct tm i {};
if (time(&st) == static_cast<time_t>(-1) || gmtime_r(&st, &i) == nullptr) {
if (clock_gettime(CLOCK_REALTIME, &now) != 0 || gmtime_r(&now.tv_sec, &i) == nullptr) {
t.is_invalid = true;
return;
}
@@ -203,7 +207,7 @@ inline void SysGetSystemTimeUtc(SysTimeStruct& t) {
t.Hour = i.tm_hour;
t.Minute = i.tm_min;
t.Second = (i.tm_sec == 60 ? 59 : i.tm_sec);
t.Milliseconds = 0;
t.Milliseconds = static_cast<uint16_t>((now.tv_nsec / 1000000) % 1000);
}
inline void SysQueryPerformanceFrequency(uint64_t* freq) {
+47
View File
@@ -7,6 +7,7 @@
#include <atomic>
#include <chrono> // IWYU pragma: keep
#include <condition_variable> // IWYU pragma: keep
#include <cerrno>
#include <mutex>
#include <vector>
@@ -14,6 +15,12 @@
#define KYTY_WIN_CS
#endif
// macOS has no clock_nanosleep.
#if KYTY_PLATFORM != KYTY_PLATFORM_WINDOWS && !defined(__APPLE__)
#define KYTY_POSIX_HIGH_RES_SLEEP
#include <ctime>
#endif
#include <sstream>
#include <string>
#include <thread>
@@ -121,6 +128,42 @@ static SleepConditionVariableCS_func_t ResolveSleepConditionVariableCS() {
#endif
#ifdef KYTY_POSIX_HIGH_RES_SLEEP
// Spin for very short waits; use an absolute deadline for longer waits.
static void SleepHighResolutionNanos(uint64_t nanos) {
if (nanos == 0) {
return;
}
constexpr uint64_t NANOS_PER_SEC = 1000000000;
constexpr uint64_t SPIN_LIMIT_NS = 50000; // below this a context switch dominates
timespec deadline {};
if (clock_gettime(CLOCK_MONOTONIC, &deadline) != 0) {
std::this_thread::sleep_for(std::chrono::nanoseconds(nanos));
return;
}
auto target_nsec = static_cast<uint64_t>(deadline.tv_nsec) + nanos;
deadline.tv_sec += static_cast<time_t>(target_nsec / NANOS_PER_SEC);
deadline.tv_nsec = static_cast<long>(target_nsec % NANOS_PER_SEC);
if (nanos <= SPIN_LIMIT_NS) {
timespec now {};
do {
if (clock_gettime(CLOCK_MONOTONIC, &now) != 0) {
return;
}
} while (now.tv_sec < deadline.tv_sec ||
(now.tv_sec == deadline.tv_sec && now.tv_nsec < deadline.tv_nsec));
return;
}
while (clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &deadline, nullptr) == EINTR) {
}
}
#endif
namespace Common {
using thread_id_t = std::thread::id;
@@ -249,6 +292,8 @@ void Thread::Sleep(uint32_t millis) {
void Thread::SleepMicro(uint32_t micros) {
#ifdef KYTY_WIN_CS
SleepHighResolution100ns(static_cast<uint64_t>(micros) * 10);
#elif defined(KYTY_POSIX_HIGH_RES_SLEEP)
SleepHighResolutionNanos(static_cast<uint64_t>(micros) * 1000);
#else
std::this_thread::sleep_for(std::chrono::microseconds(micros));
#endif
@@ -257,6 +302,8 @@ void Thread::SleepMicro(uint32_t micros) {
void Thread::SleepNano(uint64_t nanos) {
#ifdef KYTY_WIN_CS
SleepHighResolution100ns((nanos + 99) / 100);
#elif defined(KYTY_POSIX_HIGH_RES_SLEEP)
SleepHighResolutionNanos(nanos);
#else
std::this_thread::sleep_for(std::chrono::nanoseconds(nanos));
#endif
+56
View File
@@ -0,0 +1,56 @@
#ifndef KYTY_COMMON_UNIQUEFUNCTION_H_
#define KYTY_COMMON_UNIQUEFUNCTION_H_
#include <memory>
#include <type_traits>
#include <utility>
namespace Common {
template <typename Result, typename... Args>
class UniqueFunction {
class CallableBase {
public:
virtual ~CallableBase() = default;
virtual Result Invoke(Args&&... args) = 0;
};
template <typename Function>
class Callable final: public CallableBase {
public:
explicit Callable(Function function): m_function(std::move(function)) {}
Result Invoke(Args&&... args) override {
return m_function(std::forward<Args>(args)...);
}
private:
Function m_function;
};
public:
UniqueFunction() = default;
template <typename Function>
UniqueFunction(Function&& function)
: m_callable(std::make_unique<Callable<std::decay_t<Function>>>(
std::forward<Function>(function))) {}
UniqueFunction(UniqueFunction&&) noexcept = default;
UniqueFunction& operator=(UniqueFunction&&) noexcept = default;
UniqueFunction(const UniqueFunction&) = delete;
UniqueFunction& operator=(const UniqueFunction&) = delete;
Result operator()(Args... args) const {
return m_callable->Invoke(std::forward<Args>(args)...);
}
explicit operator bool() const noexcept { return m_callable != nullptr; }
private:
std::unique_ptr<CallableBase> m_callable;
};
} // namespace Common
#endif // KYTY_COMMON_UNIQUEFUNCTION_H_
-19
View File
@@ -58,25 +58,6 @@ bool FlushInstructionCache(uint64_t address, uint64_t size) {
return SysVirtualFlushInstructionCache(address, size);
}
bool PatchReplace(uint64_t vaddr, uint64_t value) {
Mode old_mode {};
Protect(vaddr, 8, Mode::ReadWrite, &old_mode);
auto* ptr = reinterpret_cast<uint64_t*>(vaddr);
bool ret = (*ptr != value);
*ptr = value;
Protect(vaddr, 8, old_mode);
if (IsExecute(old_mode)) {
FlushInstructionCache(vaddr, 8);
}
return ret;
}
} // namespace VirtualMemory
} // namespace Common
-1
View File
@@ -37,7 +37,6 @@ bool Free(uint64_t address);
bool FreeRange(uint64_t address, uint64_t size);
bool Protect(uint64_t address, uint64_t size, Mode mode, Mode* old_mode = nullptr);
bool FlushInstructionCache(uint64_t address, uint64_t size);
bool PatchReplace(uint64_t vaddr, uint64_t value);
} // namespace VirtualMemory
+24 -32
View File
@@ -105,7 +105,7 @@ static void ClearDebugTextureFolder() {
}
}
static void Init(const Config::ConfigOptions& cfg) {
static void Init(const Config::ConfigOptions& cfg, const std::filesystem::path& param_json) {
EXIT_IF(!Common::Thread::IsMainThread());
auto* slist = Common::SubsystemsList::Instance();
@@ -127,12 +127,21 @@ static void Init(const Config::ConfigOptions& cfg) {
slist->InitAll(true);
Config::Load(cfg);
slist->Add(log, {core, config});
slist->InitAll(true);
if (Common::File::IsFileExisting(param_json)) {
Loader::SystemContentLoadParamSfo(param_json);
if (const auto flexible_memory_size = Loader::SystemContentGetFlexibleMemorySize();
flexible_memory_size != 0) {
Libs::LibKernel::Memory::SetFlexibleMemorySize(flexible_memory_size);
}
}
slist->Add(audio, {core, log, pthread, memory});
slist->Add(controller, {core, log, config});
slist->Add(file_system, {core, log, pthread});
slist->Add(graphics, {core, log, pthread, memory, config, profiler, controller});
slist->Add(log, {core, config});
slist->Add(memory, {core, log});
slist->Add(network, {core, log, pthread});
slist->Add(profiler, {core, config});
@@ -159,25 +168,16 @@ static void LoadElf(const std::filesystem::path& elf, bool dbg_print_reloc = fal
}
}
static void Execute() {
int thread_model = 1;
if (thread_model == 0) {
Common::Thread t([](void* /*unused*/) { Libs::Graphics::WindowRun(); }, nullptr);
t.Detach();
auto* rt = Common::Singleton<Loader::RuntimeLinker>::Instance();
rt->Execute();
} else {
Common::Thread t(
[](void* /*unused*/) {
auto* rt = Common::Singleton<Loader::RuntimeLinker>::Instance();
rt->Execute();
},
nullptr);
t.Detach();
Libs::Graphics::WindowRun();
t.Join();
}
static void Execute(const std::filesystem::path& game_patch) {
auto patch_path = game_patch;
Common::Thread guest_thread(
[](void* param) {
auto* rt = Common::Singleton<Loader::RuntimeLinker>::Instance();
rt->Execute(*static_cast<const std::filesystem::path*>(param));
},
&patch_path);
Libs::Graphics::WindowRun();
std::quick_exit(0);
}
void Run(const RunOptions& options) {
@@ -189,7 +189,8 @@ void Run(const RunOptions& options) {
EXIT("ELF is required\n");
}
Init(options.config);
const auto param_json = options.app0_dir / "sce_sys" / "param.json";
Init(options.config, param_json);
ClearDebugTextureFolder();
@@ -201,15 +202,6 @@ void Run(const RunOptions& options) {
Libs::LibKernel::FileSystem::Mount(options.app0_dir, "/app0");
Libs::LibKernel::FileSystem::Mount(options.app0_dir, "/hostapp");
auto param_json = options.app0_dir / "sce_sys" / "param.json";
if (Common::File::IsFileExisting(param_json)) {
Loader::SystemContentLoadParamSfo(param_json);
if (auto flexible_memory_size = Loader::SystemContentGetFlexibleMemorySize();
flexible_memory_size != 0) {
Libs::LibKernel::Memory::SetFlexibleMemorySize(flexible_memory_size);
}
}
MountSandboxDirs();
auto* rt = Common::Singleton<Loader::RuntimeLinker>::Instance();
@@ -217,7 +209,7 @@ void Run(const RunOptions& options) {
LoadElf(options.elf);
Execute();
Execute(options.game_patch);
}
} // namespace Emulator
+1
View File
@@ -12,6 +12,7 @@ struct RunOptions {
Config::ConfigOptions config;
std::filesystem::path app0_dir;
std::filesystem::path elf;
std::filesystem::path game_patch;
};
void Run(const RunOptions& options);
+13 -6
View File
@@ -1,8 +1,15 @@
#message("${GIT_EXECUTABLE}")
set(KYTY_GIT_VERSION "unknown")
if(GIT_EXECUTABLE)
execute_process(COMMAND ${GIT_EXECUTABLE} describe --tags --always OUTPUT_VARIABLE KYTY_GIT_VERSION)
else()
set(KYTY_GIT_VERSION "unknown")
execute_process(
COMMAND "${GIT_EXECUTABLE}" describe --tags --always
WORKING_DIRECTORY "${GIT_WORKING_DIRECTORY}"
OUTPUT_VARIABLE KYTY_GIT_VERSION
OUTPUT_STRIP_TRAILING_WHITESPACE
RESULT_VARIABLE GIT_RESULT
ERROR_QUIET
)
if(NOT GIT_RESULT EQUAL 0)
set(KYTY_GIT_VERSION "unknown")
endif()
endif()
string(STRIP ${KYTY_GIT_VERSION} KYTY_GIT_VERSION)
configure_file(${INPUT_FILE} ${OUTPUT_FILE})
configure_file("${INPUT_FILE}" "${OUTPUT_FILE}")
-96
View File
@@ -1,96 +0,0 @@
#ifndef EMULATOR_INCLUDE_EMULATOR_GRAPHICS_ASYNCJOB_H_
#define EMULATOR_INCLUDE_EMULATOR_GRAPHICS_ASYNCJOB_H_
#include "common/common.h"
#include "common/profiler.h"
#include "common/threads.h"
#include <functional>
#include <string>
#include <utility>
namespace Libs::Graphics {
class AsyncJob final {
public:
using Task = std::function<void()>;
explicit AsyncJob(std::string thread_name = {})
: m_thread_name(std::move(thread_name)), m_worker_thread(WorkerEntry, this) {}
~AsyncJob() {
{
Common::LockGuard lock(m_mutex);
m_stop_requested = true;
m_task_available_condition.Signal();
}
m_worker_thread.Join();
}
KYTY_CLASS_NO_COPY(AsyncJob);
void Execute(Task task) {
Common::LockGuard lock(m_mutex);
while (m_is_busy) {
m_idle_condition.Wait(&m_mutex);
}
m_task = std::move(task);
m_is_busy = true;
m_task_available_condition.Signal();
}
void Wait() {
Common::LockGuard lock(m_mutex);
while (m_is_busy) {
m_idle_condition.Wait(&m_mutex);
}
}
private:
static void WorkerEntry(void* data) { static_cast<AsyncJob*>(data)->WorkerLoop(); }
void WorkerLoop() {
if (!m_thread_name.empty()) {
KYTY_PROFILER_THREAD(m_thread_name.c_str());
}
for (;;) {
Task task;
{
Common::LockGuard lock(m_mutex);
while (!m_is_busy && !m_stop_requested) {
m_task_available_condition.Wait(&m_mutex);
}
if (!m_is_busy) {
return;
}
task = std::move(m_task);
}
task();
{
Common::LockGuard lock(m_mutex);
m_is_busy = false;
m_idle_condition.SignalAll();
}
}
}
std::string m_thread_name;
Common::Mutex m_mutex;
Common::CondVar m_task_available_condition;
Common::CondVar m_idle_condition;
Task m_task;
bool m_is_busy = false;
bool m_stop_requested = false;
Common::Thread m_worker_thread;
};
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_ASYNCJOB_H_ */
@@ -2,111 +2,36 @@
#define GRAPHICS_GUEST_GPU_COMMAND_PROCESSOR_COMMAND_PROCESSOR_H
#include "common/assert.h"
#include "common/threads.h"
#include "graphics/guest_gpu/hardwareContext.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include <array>
#include <cstdint>
#include <vector>
namespace Libs::Graphics {
inline constexpr uint32_t AcquireGcrGl2Writeback = 1u << 15u;
bool TestWaitRegMemValue(uint64_t value, uint64_t ref, uint64_t mask, uint32_t func);
class CommandScheduler {
enum class Pm4ProcessResult { Complete, Blocked };
class Pm4Execution {
public:
static constexpr int BuffersNum = 8;
CommandScheduler(HW::Context& registers, HW::UserConfig& user_config, HW::Shader& shaders)
: m_registers(registers), m_user_config(user_config), m_shaders(shaders) {}
void SetQueue(int queue) { m_queue = queue; }
int Queue() const { return m_queue; }
bool Active() const { return m_current >= 0 && m_current < BuffersNum; }
void CheckActive() const { EXIT_IF(!Active()); }
RenderCommandBuffer& Current() const {
CheckActive();
EXIT_IF(m_buffers[m_current] == nullptr);
return *m_buffers[m_current];
}
void Init() {
if (m_current >= 0) {
return;
}
for (auto& buf: m_buffers) {
EXIT_IF(buf != nullptr);
buf = new RenderCommandBuffer(m_queue, m_registers, m_user_config, m_shaders);
}
m_current = 0;
Current().Begin();
}
void Flush() {
SubmitCurrent();
BeginNext();
}
CommandBuffer& FlushAndGetSubmitted() {
auto& submitted = SubmitCurrent();
BeginNext();
return submitted;
}
void CopyBuffers(std::array<CommandBuffer*, BuffersNum>& out) const {
for (int i = 0; i < BuffersNum; i++) {
auto* buf = m_buffers[i];
EXIT_IF(buf == nullptr);
out[i] = buf;
}
}
void WaitAll() {
for (auto* buf: m_buffers) {
EXIT_IF(buf == nullptr);
buf->WaitForFenceAndReset();
}
}
void SubmitForReadback() {
if (!Active()) {
return;
}
SubmitCurrent();
}
void ResumeAfterReadback() {
if (!Active()) {
return;
}
Current().WaitForFenceAndReset();
Current().Begin();
}
[[nodiscard]] bool MadeProgress() const noexcept { return m_made_progress; }
private:
CommandBuffer& SubmitCurrent() {
auto& submitted = Current();
submitted.End();
submitted.Execute();
return submitted;
}
friend class CommandProcessor;
void BeginNext() {
m_current = (m_current + 1) % BuffersNum;
Current().WaitForFenceAndReset();
Current().Begin();
}
struct BufferCursor {
uint32_t* next_packet = nullptr;
uint32_t remaining_dw = 0;
uint32_t total_dw = 0;
uint32_t deferred_advance_dw = 0;
};
RenderCommandBuffer* m_buffers[BuffersNum] = {};
int m_current = -1;
int m_queue = -1;
HW::Context& m_registers;
HW::UserConfig& m_user_config;
HW::Shader& m_shaders;
std::vector<BufferCursor> m_buffer_stack;
bool m_suspended = false;
bool m_made_progress = false;
};
class CommandProcessor {
@@ -118,8 +43,8 @@ public:
int64_t flip_arg = 0;
};
CommandProcessor(): m_scheduler(m_ctx, m_ucfg, m_sh_ctx) {}
~CommandProcessor() { KYTY_NOT_IMPLEMENTED; }
explicit CommandProcessor(RenderContext& renderer): m_renderer(renderer) {}
~CommandProcessor() = default;
KYTY_CLASS_NO_COPY(CommandProcessor);
@@ -130,26 +55,18 @@ public:
void BufferFlushAndWait();
void BufferWait();
void BeginReadbackTransaction() {
m_mutex.Lock();
if (m_readback_active) {
EXIT("nested command-processor readback transaction\n");
}
m_readback_active = true;
m_readback_finished = false;
m_readback_active = true;
}
void FinishReadbackTransaction();
void EndReadbackTransaction() {
if (!m_readback_active) {
EXIT("command-processor readback transaction is not active\n");
}
m_readback_active = false;
m_readback_finished = false;
m_mutex.Unlock();
m_readback_active = false;
}
void RunLock() { m_run_mutex.Lock(); }
void RunUnlock() { m_run_mutex.Unlock(); }
HW::Context& GetCtx() { return m_ctx; }
HW::UserConfig& GetUcfg() { return m_ucfg; }
HW::Shader& GetShCtx() { return m_sh_ctx; }
@@ -184,12 +101,10 @@ public:
void Flip(void* dst_gpu_addr, uint32_t value);
void FlipWithInterrupt(uint32_t eop_event_type, uint32_t cache_action, void* dst_gpu_addr,
uint32_t value);
void PrepareCpuFlip();
void PrepareCpuFlip(uint64_t request_id);
void SynchronizeGpu();
void MemoryBarrier();
void EmitGlobalBarrier();
void TriggerEopEventAtEndOfPipe(uint32_t interrupt_context_id);
void RenderTextureBarrier(uint64_t vaddr, uint64_t size);
void DepthStencilBarrier(uint64_t vaddr, uint64_t size);
void DispatchDirect(uint32_t thread_group_x, uint32_t thread_group_y, uint32_t thread_group_z,
uint32_t mode);
void DispatchIndirect(uint32_t data_offset, uint32_t mode);
@@ -198,16 +113,13 @@ public:
void SetUserDataMarker(HW::UserSgprType type) { m_user_data_marker = type; }
[[nodiscard]] HW::UserSgprType GetUserDataMarker() const { return m_user_data_marker; }
void SetEmbeddedDataMarker(const uint32_t* buffer, uint32_t num_dw, uint32_t align) {}
void PushMarker(const char* str) {}
void PopMarker() {}
void PrefetchL2(void* addr, uint32_t size) {}
void ResetDeCe();
void SetCeComplete(bool complete) { m_ce_complete = complete; }
void WaitCe();
void WaitDeDiff(uint32_t diff);
void IncremenetDe();
void IncremenetCe();
void IncrementDe();
void IncrementCe();
void WriteConstRam(uint32_t offset, const uint32_t* src, uint32_t dw_num);
void DumpConstRam(uint32_t* dst, uint32_t offset, uint32_t dw_num);
@@ -224,35 +136,29 @@ public:
const volatile void* address, uint32_t count_in_dwords);
[[nodiscard]] bool ShouldSkipPredicatedPackets() const { return m_predicate_skip; }
void Run(uint32_t* data, uint32_t num_dw);
Pm4ProcessResult Process(Pm4Execution& execution, uint32_t* buffer, uint32_t size_dw);
void ProcessIndirectBuffer(uint32_t* buffer, uint32_t size_dw);
void SetQueue(int queue);
[[nodiscard]] int GetQueue() const { return m_scheduler.Queue(); }
[[nodiscard]] const FlipInfo& GetFlip() const { return m_flip; }
void SetFlip(const FlipInfo& flip) { m_flip = flip; }
void SetFlip(const FlipInfo& flip) { m_flip = flip; }
[[nodiscard]] uint64_t GetSubmitId() const { return m_submit_id; }
void SetSubmitId(uint64_t submit_id) { m_submit_id = submit_id; }
private:
struct Counter {
Common::Mutex mutex;
Common::CondVar cond_var;
uint32_t value = 0;
};
template <typename T>
void WriteAtEndOfPipe(uint32_t cache_policy, uint32_t event_write_dest, uint32_t eop_event_type,
uint32_t cache_action, uint32_t event_index, uint32_t event_write_source,
void* dst_gpu_addr, T value, uint32_t interrupt_selector,
uint32_t interrupt_context_id);
void FinishCommandProcessors();
void ProcessPm4(Pm4Execution& execution, size_t stop_depth);
void SuspendPm4();
RenderCommandBuffer& CurrentBuffer() { return m_scheduler.Current(); }
void CheckBuffer() const { m_scheduler.CheckActive(); }
GpuResourceManager& GetGpuResources() const { return GetRenderContext().GetGpuResources(); }
CommandScheduler& GetScheduler() const { return m_renderer.GetCommandScheduler(); }
RenderCommandBuffer& CurrentBuffer() { return GetScheduler().Current(); }
void CheckBuffer() const { GetScheduler().CheckActive(); }
GpuResourceManager& GetGpuResources() const { return m_renderer.GetGpuResources(); }
RenderContext& m_renderer;
HW::Context m_ctx;
HW::UserConfig m_ucfg;
HW::Shader m_sh_ctx;
@@ -264,16 +170,10 @@ private:
uint64_t m_dispatch_indirect_args_base_addr = 0;
uint32_t m_num_instances = 1;
inline static Common::Mutex m_mutex;
inline static std::array<CommandProcessor*, GraphicContext::QUEUES_NUM> m_processors {};
inline static bool m_readback_active = false;
inline static bool m_readback_finished = false;
Common::Mutex m_run_mutex;
CommandScheduler m_scheduler;
Counter m_de_counter;
Counter m_ce_counter;
uint32_t m_de_count = 0;
uint32_t m_ce_count = 0;
bool m_ce_complete = false;
bool m_readback_active = false;
uint32_t m_const_ram[0x3000] = {0};
@@ -6,7 +6,6 @@
#include "graphics/guest_gpu/command_processor/pm4Dispatch.h"
#include "graphics/guest_gpu/graphicsRun.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/objects/label.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/presentation/videoOut.h"
@@ -66,10 +65,23 @@ constexpr uint32_t GcrKnownMask = GcrGl2MetadataInvalidate | GcrGl0V
GcrGl2Writeback | GcrOrder012 | GcrOrder210;
constexpr uint32_t RegisterSelectorMask = 0x70000000u;
uint32_t NormalizeRegisterOffset(uint32_t raw_offset) {
return (raw_offset & ~RegisterSelectorMask);
constexpr uint32_t NormalizeRegisterOffset(uint32_t raw_offset) {
return raw_offset & ~RegisterSelectorMask;
}
// Indirect Cx descriptors retain their selector. Selector 1 offsets 0..31 address the
// SPI_PS_INPUT_CNTL register bank; ordinary context-register offsets remain unchanged.
constexpr uint32_t DecodeIndirectCxRegisterOffset(uint32_t raw_offset) {
const auto offset = NormalizeRegisterOffset(raw_offset);
return (raw_offset & RegisterSelectorMask) == Pm4::CX_PS_SHADER_USAGE_BASE && offset < 32u
? Pm4::SPI_PS_INPUT_CNTL_0 + offset
: offset;
}
static_assert(DecodeIndirectCxRegisterOffset(Pm4::CX_PS_SHADER_USAGE_BASE + 2u) ==
Pm4::SPI_PS_INPUT_CNTL_0 + 2u);
static_assert(DecodeIndirectCxRegisterOffset(Pm4::DB_Z_INFO) == Pm4::DB_Z_INFO);
bool ReleaseMemGcrNeedsBarrier(uint32_t eop_event_type, uint32_t gcr_cntl) {
return eop_event_type != 0x28u ||
(gcr_cntl & (GcrGl2MetadataInvalidate | GcrGl0VectorInvalidate | GcrGl1Invalidate |
@@ -598,30 +610,7 @@ KYTY_HW_CTX_PARSER(HwCtxSetDepthRenderTarget) {
uint32_t count = 1;
if (cmd_id == 0xC0016900) {
HW::DepthZInfo r;
// r.expclear_enabled = (buffer[0] & 0x08000000u) != 0;
// r.format = (buffer[0] >> Pm4::DB_Z_INFO_FORMAT_SHIFT) &
// Pm4::DB_Z_INFO_FORMAT_MASK; r.num_samples = (buffer[0] >>
// Pm4::DB_Z_INFO_NUM_SAMPLES_SHIFT) & Pm4::DB_Z_INFO_NUM_SAMPLES_MASK; r.tile_mode_index =
//(buffer[0] >> Pm4::DB_Z_INFO_TILE_MODE_INDEX_SHIFT) & Pm4::DB_Z_INFO_TILE_MODE_INDEX_MASK;
// r.tile_surface_enable = ((buffer[0] >> Pm4::DB_Z_INFO_TILE_SURFACE_ENABLE_SHIFT) &
// Pm4::DB_Z_INFO_TILE_SURFACE_ENABLE_MASK) !=0
// r.zrange_precision = (buffer[0] >> Pm4::DB_Z_INFO_ZRANGE_PRECISION_SHIFT) &
// Pm4::DB_Z_INFO_ZRANGE_PRECISION_MASK;
r.format = KYTY_PM4_GET(buffer[0], DB_Z_INFO, FORMAT);
r.num_samples = KYTY_PM4_GET(buffer[0], DB_Z_INFO, NUM_SAMPLES);
r.embedded_sample_locations = KYTY_PM4_GET(buffer[0], DB_Z_INFO, ITERATE_FLUSH) != 0;
r.partially_resident = KYTY_PM4_GET(buffer[0], DB_Z_INFO, PARTIALLY_RESIDENT) != 0;
r.num_mip_levels = KYTY_PM4_GET(buffer[0], DB_Z_INFO, MAXMIP);
r.tile_mode_index = KYTY_PM4_GET(buffer[0], DB_Z_INFO, TILE_MODE_INDEX);
r.plane_compression = KYTY_PM4_GET(buffer[0], DB_Z_INFO, DECOMPRESS_ON_N_ZPLANES);
r.expclear_enabled = KYTY_PM4_GET(buffer[0], DB_Z_INFO, ALLOW_EXPCLEAR) != 0;
r.tile_surface_enable = KYTY_PM4_GET(buffer[0], DB_Z_INFO, TILE_SURFACE_ENABLE) != 0;
r.zrange_precision = KYTY_PM4_GET(buffer[0], DB_Z_INFO, ZRANGE_PRECISION);
cp.GetCtx().SetDepthZInfo(r);
cp.GetCtx().SetDepthZInfo(HW::DepthZInfo::Decode(buffer[0]));
} else if (cmd_id == 0xC0086900) {
if (dw >= 22 && buffer[8] == 0xC0016900 && buffer[9] == Pm4::DB_DEPTH_INFO &&
buffer[11] == 0xC0016900 && buffer[12] == Pm4::DB_DEPTH_VIEW &&
@@ -632,51 +621,8 @@ KYTY_HW_CTX_PARSER(HwCtxSetDepthRenderTarget) {
HW::DepthRenderTarget z;
// z.z_info.expclear_enabled = (buffer[0] & 0x08000000u) != 0;
// z.z_info.format = (buffer[0] >> Pm4::DB_Z_INFO_FORMAT_SHIFT) &
// Pm4::DB_Z_INFO_FORMAT_MASK; z.z_info.num_samples = (buffer[0] >>
// Pm4::DB_Z_INFO_NUM_SAMPLES_SHIFT) & Pm4::DB_Z_INFO_NUM_SAMPLES_MASK;
// z.z_info.tile_mode_index = (buffer[0] >>
// Pm4::DB_Z_INFO_TILE_MODE_INDEX_SHIFT) &
// Pm4::DB_Z_INFO_TILE_MODE_INDEX_MASK; z.z_info.tile_surface_enable =
// KYTY_PM4_GET(buffer[0], DB_Z_INFO, TILE_SURFACE_ENABLE) != 0;
// z.z_info.zrange_precision = (buffer[0] >> Pm4::DB_Z_INFO_ZRANGE_PRECISION_SHIFT) &
// Pm4::DB_Z_INFO_ZRANGE_PRECISION_MASK;
z.z_info.format = KYTY_PM4_GET(buffer[0], DB_Z_INFO, FORMAT);
z.z_info.num_samples = KYTY_PM4_GET(buffer[0], DB_Z_INFO, NUM_SAMPLES);
z.z_info.embedded_sample_locations =
KYTY_PM4_GET(buffer[0], DB_Z_INFO, ITERATE_FLUSH) != 0;
z.z_info.partially_resident =
KYTY_PM4_GET(buffer[0], DB_Z_INFO, PARTIALLY_RESIDENT) != 0;
z.z_info.num_mip_levels = KYTY_PM4_GET(buffer[0], DB_Z_INFO, MAXMIP);
z.z_info.tile_mode_index = KYTY_PM4_GET(buffer[0], DB_Z_INFO, TILE_MODE_INDEX);
z.z_info.plane_compression =
KYTY_PM4_GET(buffer[0], DB_Z_INFO, DECOMPRESS_ON_N_ZPLANES);
z.z_info.expclear_enabled = KYTY_PM4_GET(buffer[0], DB_Z_INFO, ALLOW_EXPCLEAR) != 0;
z.z_info.tile_surface_enable =
KYTY_PM4_GET(buffer[0], DB_Z_INFO, TILE_SURFACE_ENABLE) != 0;
z.z_info.zrange_precision = KYTY_PM4_GET(buffer[0], DB_Z_INFO, ZRANGE_PRECISION);
// z.stencil_info.expclear_enabled = (buffer[1] & 0x08000000u) != 0;
// z.stencil_info.tile_split = (buffer[1] >> 13u) & 0x7u;
// z.stencil_info.format = KYTY_PM4_GET(buffer[1],
// DB_STENCIL_INFO, FORMAT); z.stencil_info.tile_mode_index =
// KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, TILE_MODE_INDEX);
// z.stencil_info.tile_stencil_disable = KYTY_PM4_GET(buffer[1],
// DB_STENCIL_INFO, TILE_STENCIL_DISABLE);
z.stencil_info.format = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, FORMAT);
z.stencil_info.texture_compatible_stencil =
KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, ITERATE_FLUSH) != 0;
z.stencil_info.partially_resident =
KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, PARTIALLY_RESIDENT) != 0;
z.stencil_info.tile_split = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, RESERVED_FIELD_1);
z.stencil_info.tile_mode_index =
KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, TILE_MODE_INDEX);
z.stencil_info.expclear_enabled =
KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, ALLOW_EXPCLEAR) != 0;
z.stencil_info.tile_stencil_disable =
KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, TILE_STENCIL_DISABLE) != 0;
z.z_info = HW::DepthZInfo::Decode(buffer[0]);
z.stencil_info = HW::DepthStencilInfo::Decode(buffer[1]);
z.z_read_base_addr = static_cast<uint64_t>(buffer[2]) << 8u;
z.stencil_read_base_addr = static_cast<uint64_t>(buffer[3]) << 8u;
@@ -1253,26 +1199,7 @@ KYTY_HW_CTX_PARSER(HwCtxSetStencilInfo) {
EXIT_NOT_IMPLEMENTED(cmd_id != 0xC0016900);
EXIT_NOT_IMPLEMENTED(cmd_offset != Pm4::DB_STENCIL_INFO);
HW::DepthStencilInfo r;
// r.expclear_enabled = (buffer[0] & 0x08000000u) != 0;
// r.tile_split = (buffer[0] >> 13u) & 0x7u;
// r.format = (buffer[0] >> Pm4::DB_STENCIL_INFO_FORMAT_SHIFT) &
// Pm4::DB_STENCIL_INFO_FORMAT_MASK; r.tile_mode_index = (buffer[0] >>
// Pm4::DB_STENCIL_INFO_TILE_MODE_INDEX_SHIFT) & Pm4::DB_STENCIL_INFO_TILE_MODE_INDEX_MASK;
// r.tile_stencil_disable =
// ((buffer[0] >> Pm4::DB_STENCIL_INFO_TILE_STENCIL_DISABLE_SHIFT) &
// Pm4::DB_STENCIL_INFO_TILE_STENCIL_DISABLE_MASK) != 0;
r.format = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, FORMAT);
r.texture_compatible_stencil = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, ITERATE_FLUSH) != 0;
r.partially_resident = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, PARTIALLY_RESIDENT) != 0;
r.tile_split = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, RESERVED_FIELD_1);
r.tile_mode_index = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, TILE_MODE_INDEX);
r.expclear_enabled = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, ALLOW_EXPCLEAR) != 0;
r.tile_stencil_disable = KYTY_PM4_GET(buffer[1], DB_STENCIL_INFO, TILE_STENCIL_DISABLE) != 0;
cp.GetCtx().SetDepthStencilInfo(r);
cp.GetCtx().SetDepthStencilInfo(HW::DepthStencilInfo::Decode(buffer[0]));
return 1;
}
@@ -1764,231 +1691,11 @@ static bool HwUcTrySetFakeRegisterRange(uint32_t cmd_offset, const uint32_t* buf
return true;
}
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
KYTY_CP_OP_PARSER(CpOpAcquireMem) {
KYTY_PROFILER_FUNCTION();
EXIT_NOT_IMPLEMENTED(cmd_id != 0xC0055800 && cmd_id != 0xc0061050);
bool custom = (cmd_id == 0xc0061050);
uint32_t engine = buffer[0] >> 31u;
uint32_t stall_mode = (custom ? 1u : engine);
uint32_t cache_action = buffer[0] & 0x7fffffffu;
uint64_t size_lo = buffer[1];
uint32_t size_hi = buffer[2];
uint64_t base_lo = buffer[3];
uint32_t base_hi = buffer[4];
uint32_t poll = buffer[5];
[[maybe_unused]] uint32_t gcr_cntl = (custom ? buffer[6] : 0);
uint32_t target_mask = cache_action & 0x00007FC0u;
uint32_t extended_action = cache_action & 0x2E000000u;
uint32_t action =
((cache_action & 0x00C00000u) >> 0x12u) | ((cache_action & 0x00058000u) >> 0xfu);
if (custom && engine > 1) {
LOGF("\t warning: custom acquire_mem unsupported engine: %" PRIu32 "\n", engine);
}
// EXIT_NOT_IMPLEMENTED(stall_mode != 1);
EXIT_NOT_IMPLEMENTED(size_hi != 0);
EXIT_NOT_IMPLEMENTED(base_hi != 0);
if (poll != 10) {
LOGF("\t warning: acquire_mem unexpected poll interval: %" PRIu32 "\n", poll);
}
switch (cache_action) {
case 0x00000000: {
if (custom && gcr_cntl != 0) {
LOGF("\t custom acquire_mem GCR-only barrier, gcr_cntl = 0x%08" PRIx32
", base = 0x%016" PRIx64 ", size = 0x%016" PRIx64 "\n",
gcr_cntl, base_lo << 8u, size_lo << 8u);
cp.MemoryBarrier();
if ((gcr_cntl & AcquireGcrGl2Writeback) != 0) {
cp.SynchronizeGpu();
}
}
} break;
case 0x00000040:
case 0x00003fc0:
case 0x00004000:
case 0x00007fc0: {
// target_mask set, no CB/DB action bits. Treat as an ordering barrier.
EXIT_IF(target_mask != cache_action);
EXIT_IF(extended_action != 0x00000000);
EXIT_IF(action != 0x00);
LOGF("\t temporary: acquire_mem target-mask-only barrier, target_mask = 0x%08" PRIx32
", gcr_cntl = 0x%08" PRIx32 ", base = 0x%016" PRIx64 ", size = 0x%016" PRIx64 "\n",
target_mask, gcr_cntl, base_lo << 8u, size_lo << 8u);
cp.MemoryBarrier();
} break;
case 0x02000000: {
// target_mask: 0x00000000 (none)
// extended_action: 0x02000000 (FlushAndInvalidateCbCache)
// action: 0x00 (none)
EXIT_IF(target_mask != 0x00000000);
EXIT_IF(extended_action != 0x02000000);
EXIT_IF(action != 0x00);
LOGF("\t temporary: acquire_mem CB-cache-only barrier, gcr_cntl = 0x%08" PRIx32
", base = 0x%016" PRIx64 ", size = 0x%016" PRIx64 "\n",
gcr_cntl, base_lo << 8u, size_lo << 8u);
cp.MemoryBarrier();
} break;
case 0x04000000: {
// target_mask: 0x00000000 (none)
// extended_action: 0x04000000 (FlushAndInvalidateDbCache)
// action: 0x00 (none)
EXIT_IF(target_mask != 0x00000000);
EXIT_IF(extended_action != 0x04000000);
EXIT_IF(action != 0x00);
LOGF("\t temporary: acquire_mem DB-cache-only barrier, gcr_cntl = 0x%08" PRIx32
", base = 0x%016" PRIx64 ", size = 0x%016" PRIx64 "\n",
gcr_cntl, base_lo << 8u, size_lo << 8u);
cp.MemoryBarrier();
} break;
case 0x04004000:
case 0x04007fc0: {
// target_mask: 0x00004000 (Depth Target), 0x00007fc0 (all rt and depth)
// extended_action: 0x04000000 (FlushAndInvalidateDbCache)
// action: 0x00 (none)
EXIT_IF(target_mask != 0x00004000 && target_mask != 0x00007FC0);
EXIT_IF(extended_action != 0x04000000);
EXIT_IF(action != 0x00);
LOGF("\t temporary: acquire_mem DB target barrier, target_mask = 0x%08" PRIx32
", gcr_cntl = 0x%08" PRIx32 ", base = 0x%016" PRIx64 ", size = 0x%016" PRIx64 "\n",
target_mask, gcr_cntl, base_lo << 8u, size_lo << 8u);
if (size_lo != 0) {
cp.DepthStencilBarrier(base_lo << 8u, size_lo << 8u);
} else {
cp.MemoryBarrier();
}
} break;
case 0x02c40040:
case 0x02c43fc0:
case 0x02c47fc0: {
// target_mask: 0x00000040 (rt0), 0x00003fc0 (all rt), 0x00007fc0 (all rt and depth)
// extended_action: 0x02000000 (FlushAndInvalidateCbCache)
// action: 0x38 (WriteBackAndInvalidateL1andL2)
EXIT_IF(target_mask != 0x00000040 && target_mask != 0x00003FC0 &&
target_mask != 0x00007FC0);
EXIT_IF(extended_action != 0x02000000);
EXIT_IF(action != 0x38);
EXIT_NOT_IMPLEMENTED(size_lo == 0);
EXIT_NOT_IMPLEMENTED(base_lo == 0);
cp.RenderTextureBarrier(base_lo << 8u, size_lo << 8u);
cp.SynchronizeGpu();
} break;
case 0x02003fc0:
case 0x02007fc0: {
// target_mask: 0x00003FC0 (all rt), 0x00007fc0 (all rt and depth)
// extended_action: 0x02000000 (FlushAndInvalidateCbCache)
// action: 0x00 (none)
EXIT_IF(target_mask != 0x00003FC0 && target_mask != 0x00007fc0);
EXIT_IF(extended_action != 0x02000000);
EXIT_IF(action != 0x00);
if (size_lo == 0) {
if (base_lo != 0) {
LOGF("\t warning: acquire_mem CB-cache barrier with non-zero base");
}
cp.MemoryBarrier();
} else {
EXIT_NOT_IMPLEMENTED(base_lo == 0);
cp.RenderTextureBarrier(base_lo << 8u, size_lo << 8u);
}
} break;
case 0x00C40000: {
// target_mask: 0x00000000 (none)
// extended_action: 0x00000000 (none)
// action: 0x38 (WriteBackAndInvalidateL1andL2)
EXIT_IF(target_mask != 0x00000000);
EXIT_IF(extended_action != 0x00000000);
EXIT_IF(action != 0x38);
EXIT_NOT_IMPLEMENTED(size_lo != 1);
EXIT_NOT_IMPLEMENTED(base_lo != 0);
cp.MemoryBarrier();
cp.SynchronizeGpu();
} break;
case 0x00400000: {
// target_mask: 0x00000000 (none)
// extended_action: 0x00000000 (none)
// action: 0x10 (InvalidateL1)
EXIT_IF(target_mask != 0x00000000);
EXIT_IF(extended_action != 0x00000000);
EXIT_IF(action != 0x10);
EXIT_NOT_IMPLEMENTED(size_lo != 1);
EXIT_NOT_IMPLEMENTED(base_lo != 0);
cp.MemoryBarrier();
} break;
case 0x04c44000: {
// target_mask: 0x00004000 (Depth Target)
// extended_action: 0x04000000 (FlushAndInvalidateDbCache)
// action: 0x38 (WriteBackAndInvalidateL1andL2)
EXIT_IF(target_mask != 0x00004000);
EXIT_IF(extended_action != 0x04000000);
EXIT_IF(action != 0x38);
cp.DepthStencilBarrier(base_lo << 8u, size_lo << 8u);
cp.SynchronizeGpu();
} break;
case 0x06000040:
case 0x06000080:
case 0x06003fc0:
case 0x06007fc0: {
// target_mask: 0x00000040 (rt0), 0x00000080 (rt1), 0x00003fc0 (all rt), 0x00007fc0
// (all rt and depth) extended_action: 0x06000000 (Flush Cb & Db) action: 0x00
// (none)
if (gcr_cntl != 0 && gcr_cntl != 0x280 && gcr_cntl != 0x300) {
LOGF("\t temporary: acquire_mem CB+DB barrier with unhandled GCR control "
"0x%08" PRIx32 "\n",
gcr_cntl);
}
EXIT_IF(target_mask != 0x00000040 && target_mask != 0x00000080 &&
target_mask != 0x00003fc0 && target_mask != 0x00007fc0);
EXIT_IF(extended_action != 0x06000000);
EXIT_IF(action != 0x00);
if (size_lo != 0) {
if ((target_mask & 0x00003fc0) != 0) {
cp.RenderTextureBarrier(base_lo << 8u, size_lo << 8u);
}
if ((target_mask & 0x00004000) != 0) {
cp.DepthStencilBarrier(base_lo << 8u, size_lo << 8u);
}
} else {
cp.MemoryBarrier();
}
} break;
default:
EXIT("unknown barrier: 0x%08" PRIx32 ", 0x%08" PRIx32 ", 0x%08" PRIx32 ", 0x%08" PRIx32
"\n",
cache_action, target_mask, extended_action, action);
}
if (stall_mode == 0) {
cp.BufferFlush();
cp.BufferWait();
}
return (custom ? 7 : 6);
return (cmd_id == 0xc0061050 ? 7 : 6);
}
KYTY_CP_OP_PARSER(CpOpDispatchDirect) {
@@ -2166,10 +1873,10 @@ KYTY_CP_OP_PARSER(CpOpBranch) {
reinterpret_cast<uint64_t>(else_buffer), else_num_dw);
if (take_then) {
cp.Run(then_buffer, then_num_dw);
cp.ProcessIndirectBuffer(then_buffer, then_num_dw);
} else if (mode == 2 && else_num_dw != 0) {
EXIT_NOT_IMPLEMENTED(else_buffer == nullptr);
cp.Run(else_buffer, else_num_dw);
cp.ProcessIndirectBuffer(else_buffer, else_num_dw);
}
return payload_dw;
@@ -2250,9 +1957,6 @@ KYTY_CP_OP_PARSER(CpOpDmaData) {
const uint64_t dst = buffer[3] | (static_cast<uint64_t>(buffer[4]) << 32u);
if (control == 0x60000000 && dst == 0x0003022c && (control2 >> 21u) == 0x141u) {
auto* addr = reinterpret_cast<void*>(src);
cp.PrefetchL2(addr, control2 & 0x1fffffu);
return 6;
}
@@ -2496,7 +2200,7 @@ KYTY_CP_OP_PARSER(CpOpIncrementCeCounter) {
EXIT_NOT_IMPLEMENTED(cmd_id != 0xC0008400);
EXIT_NOT_IMPLEMENTED(buffer[0] != 1);
cp.IncremenetCe();
cp.IncrementCe();
return 1;
}
@@ -2507,7 +2211,7 @@ KYTY_CP_OP_PARSER(CpOpIncrementDeCounter) {
EXIT_NOT_IMPLEMENTED(cmd_id != 0xC0008500);
EXIT_NOT_IMPLEMENTED(buffer[0] != 0);
cp.IncremenetDe();
cp.IncrementDe();
return 1;
}
@@ -2607,7 +2311,7 @@ KYTY_CP_OP_PARSER(CpOpIndirectBuffer) {
GraphicsDbgDumpDcb("ci", indirect_num_dw, indirect_buffer);
cp.Run(indirect_buffer, indirect_num_dw);
cp.ProcessIndirectBuffer(indirect_buffer, indirect_num_dw);
return 3;
}
@@ -2630,14 +2334,18 @@ KYTY_CP_OP_PARSER(CpOpIndirectCxRegs) {
EXIT("indirect CX registers have null address, num_regs = %" PRIu32 "\n", indirect_num_dw);
}
for (uint32_t i = 0; i < indirect_num_dw; i++, indirect_buffer += 2) {
auto cmd_offset = indirect_buffer[0];
auto value = indirect_buffer[1];
// Keep the encoded offset for packet control values, and use the decoded offset only
// for register dispatch.
auto raw_cmd_offset = indirect_buffer[0];
auto cmd_offset = DecodeIndirectCxRegisterOffset(raw_cmd_offset);
auto value = indirect_buffer[1];
if (HwCtxTrySetFakeRegister(cmd_offset, value)) {
continue;
}
if (cmd_offset == 0xffffffffu) {
// The sentinel is an encoded descriptor value and must be checked before normalization.
if (raw_cmd_offset == 0xffffffffu) {
static bool logged = false;
if (!logged) {
LOGF("\t temporary: skipping indirect CX sentinel pair offset = 0xffffffff, value "
@@ -2800,11 +2508,10 @@ KYTY_CP_OP_PARSER(CpOpMarker) {
// EXIT_NOT_IMPLEMENTED(cmd_id != 0xC0001000);
uint32_t id = buffer[0] & 0xfffu;
uint32_t align = (buffer[0] >> 12u) & 0xfu;
uint32_t len_dw = ((cmd_id >> 16u) & 0x3fffu);
switch (id) {
case 0x0: cp.SetEmbeddedDataMarker(buffer + 1, len_dw, align); break;
case 0x0: break;
case 0x4: cp.SetUserDataMarker(HW::UserSgprType::Vsharp); break;
case 0xd: cp.SetUserDataMarker(HW::UserSgprType::Region); break;
case 0x777: {
@@ -2876,8 +2583,6 @@ KYTY_CP_OP_PARSER(CpOpPopMarker) {
LOGF("Pop marker\n");
}
cp.PopMarker();
return dw_num + 1;
}
@@ -2893,8 +2598,6 @@ KYTY_CP_OP_PARSER(CpOpPushMarker) {
LOGF("Push marker: %s\n", str);
}
cp.PushMarker(str);
return dw_num + 1;
}
@@ -2939,7 +2642,7 @@ KYTY_CP_OP_PARSER(CpOpReleaseMem) {
if (data_sel == 0 || interrupt_selector == 4) {
if (eop_event_type != 0x28 || gcr_cntl != 0) {
cp.MemoryBarrier();
cp.EmitGlobalBarrier();
}
if (gl2_writeback) {
@@ -2953,7 +2656,7 @@ KYTY_CP_OP_PARSER(CpOpReleaseMem) {
if (release_dst == ReleaseMemDstMemory && dst_gpu_addr == nullptr) {
if (eop_event_type != 0x28 || gcr_cntl != 0) {
cp.MemoryBarrier();
cp.EmitGlobalBarrier();
}
if (gl2_writeback) {
@@ -2966,7 +2669,7 @@ KYTY_CP_OP_PARSER(CpOpReleaseMem) {
}
if (ReleaseMemGcrNeedsBarrier(eop_event_type, gcr_cntl)) {
cp.MemoryBarrier();
cp.EmitGlobalBarrier();
}
auto cache_action = ReleaseMemCacheActionFromGcr(gcr_cntl);
@@ -3986,18 +3689,7 @@ void GraphicsInitJmpTablesCxIndirect() {
};
g_hw_ctx_indirect_func[Pm4::DB_Z_INFO] = [](KYTY_HW_CTX_INDIRECT_ARGS) {
HW::DepthZInfo r;
r.format = KYTY_PM4_GET(value, DB_Z_INFO, FORMAT);
r.num_samples = KYTY_PM4_GET(value, DB_Z_INFO, NUM_SAMPLES);
r.embedded_sample_locations = KYTY_PM4_GET(value, DB_Z_INFO, ITERATE_FLUSH) != 0;
r.partially_resident = KYTY_PM4_GET(value, DB_Z_INFO, PARTIALLY_RESIDENT) != 0;
r.num_mip_levels = KYTY_PM4_GET(value, DB_Z_INFO, MAXMIP);
r.tile_mode_index = KYTY_PM4_GET(value, DB_Z_INFO, TILE_MODE_INDEX);
r.plane_compression = KYTY_PM4_GET(value, DB_Z_INFO, DECOMPRESS_ON_N_ZPLANES);
r.expclear_enabled = KYTY_PM4_GET(value, DB_Z_INFO, ALLOW_EXPCLEAR) != 0;
r.tile_surface_enable = KYTY_PM4_GET(value, DB_Z_INFO, TILE_SURFACE_ENABLE) != 0;
r.zrange_precision = KYTY_PM4_GET(value, DB_Z_INFO, ZRANGE_PRECISION);
cp.GetCtx().SetDepthZInfo(r);
cp.GetCtx().SetDepthZInfo(HW::DepthZInfo::Decode(value));
};
g_hw_ctx_indirect_func[Pm4::DB_DEPTH_INFO] = [](KYTY_HW_CTX_INDIRECT_ARGS) {
@@ -4025,15 +3717,7 @@ void GraphicsInitJmpTablesCxIndirect() {
};
g_hw_ctx_indirect_func[Pm4::DB_STENCIL_INFO] = [](KYTY_HW_CTX_INDIRECT_ARGS) {
HW::DepthStencilInfo r;
r.format = KYTY_PM4_GET(value, DB_STENCIL_INFO, FORMAT);
r.texture_compatible_stencil = KYTY_PM4_GET(value, DB_STENCIL_INFO, ITERATE_FLUSH) != 0;
r.partially_resident = KYTY_PM4_GET(value, DB_STENCIL_INFO, PARTIALLY_RESIDENT) != 0;
r.tile_split = KYTY_PM4_GET(value, DB_STENCIL_INFO, RESERVED_FIELD_1);
r.tile_mode_index = KYTY_PM4_GET(value, DB_STENCIL_INFO, TILE_MODE_INDEX);
r.expclear_enabled = KYTY_PM4_GET(value, DB_STENCIL_INFO, ALLOW_EXPCLEAR) != 0;
r.tile_stencil_disable = KYTY_PM4_GET(value, DB_STENCIL_INFO, TILE_STENCIL_DISABLE) != 0;
cp.GetCtx().SetDepthStencilInfo(r);
cp.GetCtx().SetDepthStencilInfo(HW::DepthStencilInfo::Decode(value));
};
g_hw_ctx_indirect_func[Pm4::DB_Z_READ_BASE] = [](KYTY_HW_CTX_INDIRECT_ARGS) {
+18
View File
@@ -142,6 +142,24 @@ enum class StencilFormat : uint32_t {
k8UInt = 1,
};
enum class TextureCompatiblePlaneCompression : uint32_t {
kDisable = 0x00000000,
kEnable = 0x02900800,
kBitMask = 0x02900800,
};
enum class TextureCompatibleStencil : uint32_t {
kDisable = 0x00000000,
kEnable = 0x00100800,
kBitMask = 0x00100800,
};
enum class ZCompareBase : uint32_t {
kZMin = 0x00000000,
kZMax = 0x80000000,
kBitMask = 0x80000000,
};
enum class TileMode : uint32_t {
kLinear = 0x00,
kStandard256B = 0x01,
File diff suppressed because it is too large Load Diff
+46 -20
View File
@@ -3,33 +3,59 @@
#include "common/abi.h"
#include "common/common.h"
#include "common/uniqueFunction.h"
#include <memory>
namespace Libs::Graphics {
class CommandProcessor;
class GpuState;
class RenderContext;
class GraphicsRunSubmissionLock final {
class Gpu final {
public:
GraphicsRunSubmissionLock();
~GraphicsRunSubmissionLock();
KYTY_CLASS_NO_COPY(GraphicsRunSubmissionLock);
explicit Gpu(RenderContext& renderer);
~Gpu();
KYTY_CLASS_NO_COPY(Gpu);
void Shutdown();
[[nodiscard]] bool IsStopping();
void SendCommand(Common::UniqueFunction<void>&& command);
void SendCommandSync(Common::UniqueFunction<void>&& command);
void SendCommandSyncWithProcessor(Common::UniqueFunction<void, CommandProcessor&>&& command);
void Submit(uint32_t* draw_commands, uint32_t draw_size_dw, uint32_t* constant_commands,
uint32_t constant_size_dw, bool trigger_agc_interrupt_on_done = false);
void SubmitCompute(uint32_t queue, uint32_t* commands, uint32_t size_dw,
bool trigger_agc_interrupt_on_done = false);
void SubmitFlipPreparation(uint64_t request_id);
void Done();
[[nodiscard]] int GetFrameNum() const;
[[nodiscard]] static bool IsCommandProcessorThread() noexcept;
[[nodiscard]] static CommandProcessor* CurrentCommandProcessor() noexcept;
[[nodiscard]] static bool SubmissionLockHeld() noexcept;
[[nodiscard]] static bool MutexHeld() noexcept;
class SubmissionLock final {
public:
explicit SubmissionLock(Gpu& gpu);
~SubmissionLock();
KYTY_CLASS_NO_COPY(SubmissionLock);
private:
Gpu& m_gpu;
};
private:
friend class SubmissionLock;
void PauseSubmissions();
void ResumeSubmissions();
std::unique_ptr<GpuState> m_state;
};
void GraphicsRunInit();
void GraphicsRunSubmit(uint32_t* cmd_draw_buffer, uint32_t num_draw_dw, uint32_t* cmd_const_buffer,
uint32_t num_const_dw, bool trigger_agc_interrupt_on_done = false);
void GraphicsRunSubmitCompute(uint32_t queue, uint32_t* cmd_buffer, uint32_t num_dw,
bool trigger_agc_interrupt_on_done = false);
void GraphicsRunSubmitFlipPreparation();
void GraphicsRunWait();
void GraphicsRunDone();
int GraphicsRunGetFrameNum();
[[nodiscard]] bool GraphicsRunIsCommandProcessorThread() noexcept;
[[nodiscard]] CommandProcessor* GraphicsRunCurrentCommandProcessor() noexcept;
void GraphicsRunFinishCommandProcessors();
[[nodiscard]] bool GraphicsRunSubmissionLockHeld() noexcept;
[[nodiscard]] bool GraphicsRunGpuLockHeld() noexcept;
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_GRAPHICSRUN_H_ */
+58 -17
View File
@@ -3,6 +3,7 @@
#include "common/abi.h"
#include "common/common.h"
#include "graphics/guest_gpu/gpu_defs.h"
namespace Libs::Graphics::HW {
@@ -131,26 +132,66 @@ struct RenderTarget {
};
struct DepthZInfo {
uint32_t format = 0;
uint32_t tile_mode_index = 0;
uint32_t num_samples = 0;
uint32_t zrange_precision = 0;
bool tile_surface_enable = false;
bool expclear_enabled = false;
bool embedded_sample_locations = false;
bool partially_resident = false;
uint8_t num_mip_levels = 0;
uint8_t plane_compression = 0;
uint32_t format = 0;
uint32_t num_samples = 0;
Prospero::TextureCompatiblePlaneCompression texture_compatibility =
Prospero::TextureCompatiblePlaneCompression::kDisable;
Prospero::ZCompareBase z_compare_base = Prospero::ZCompareBase::kZMin;
bool htile_acceleration = false;
bool expclear_enabled = false;
bool partially_resident = false;
uint8_t max_mip_level = 0;
[[nodiscard]] static DepthZInfo Decode(uint32_t value) {
DepthZInfo info;
info.format = value & 0x3u;
info.num_samples = (value >> 2u) & 0x3u;
info.texture_compatibility = static_cast<Prospero::TextureCompatiblePlaneCompression>(
value & Prospero::GpuEnumValue(Prospero::TextureCompatiblePlaneCompression::kBitMask));
info.partially_resident = (value & 0x00001000u) != 0;
info.max_mip_level = static_cast<uint8_t>((value >> 16u) & 0x0fu);
info.expclear_enabled = (value & 0x08000000u) != 0;
info.htile_acceleration = (value & 0x20000000u) != 0;
info.z_compare_base = static_cast<Prospero::ZCompareBase>(
value & Prospero::GpuEnumValue(Prospero::ZCompareBase::kBitMask));
return info;
}
[[nodiscard]] bool HasValidTextureCompatibility() const {
switch (texture_compatibility) {
case Prospero::TextureCompatiblePlaneCompression::kDisable:
case Prospero::TextureCompatiblePlaneCompression::kEnable: return true;
default: return false;
}
}
};
struct DepthStencilInfo {
uint32_t format = 0;
uint32_t tile_mode_index = 0;
uint32_t tile_split = 0;
bool expclear_enabled = false;
bool tile_stencil_disable = false;
bool texture_compatible_stencil = false;
bool partially_resident = false;
uint32_t format = 0;
Prospero::TextureCompatibleStencil texture_compatibility =
Prospero::TextureCompatibleStencil::kDisable;
bool expclear_enabled = false;
bool htile_stencil_disabled = false;
bool partially_resident = false;
[[nodiscard]] static DepthStencilInfo Decode(uint32_t value) {
DepthStencilInfo info;
info.format = value & 0x1u;
info.texture_compatibility = static_cast<Prospero::TextureCompatibleStencil>(
value & Prospero::GpuEnumValue(Prospero::TextureCompatibleStencil::kBitMask));
info.partially_resident = (value & 0x00001000u) != 0;
info.expclear_enabled = (value & 0x08000000u) != 0;
info.htile_stencil_disabled = (value & 0x20000000u) != 0;
return info;
}
[[nodiscard]] bool HasValidTextureCompatibility() const {
switch (texture_compatibility) {
case Prospero::TextureCompatibleStencil::kDisable:
case Prospero::TextureCompatibleStencil::kEnable: return true;
default: return false;
}
}
};
struct DepthRenderTargetDepthInfo {
+3
View File
@@ -385,6 +385,9 @@ constexpr uint32_t SPI_SHADER_POS_FORMAT = 0x1C3;
constexpr uint32_t SPI_SHADER_Z_FORMAT = 0x1C4;
constexpr uint32_t SPI_SHADER_COL_FORMAT = 0x1C5;
// Indirect Cx descriptor selector for the 32-entry PS input-control register bank.
constexpr uint32_t CX_PS_SHADER_USAGE_BASE = 0x10000000u;
constexpr uint32_t CB_BLEND0_CONTROL = 0x1E0;
constexpr uint32_t CB_BLEND0_CONTROL_COLOR_SRCBLEND_SHIFT = 0;
constexpr uint32_t CB_BLEND0_CONTROL_COLOR_SRCBLEND_MASK = 0x1F;
+58 -36
View File
@@ -406,8 +406,8 @@ struct TextureBlockLayout {
static bool GetTextureBlockLayout(uint32_t format, uint32_t tile, TextureBlockLayout& out) {
uint32_t width_log2 = 0, height_log2 = 0;
if (tile == 1 && Gen5Standard256BLayout(format, &out.bytes, &out.texel_width,
&out.texel_height, &width_log2, &height_log2)) {
if (tile == 1 && Gen5Standard256BLayout(format, &out.bytes, &out.texel_width, &out.texel_height,
&width_log2, &height_log2)) {
out.block_size = 256;
} else if (tile == 5 && Gen5Standard4KBLayout(format, &out.bytes, &out.texel_width,
&out.texel_height, &width_log2, &height_log2)) {
@@ -524,7 +524,7 @@ static void SetMacroMipLayout(const TextureBlockLayout& block, uint32_t tile, ui
}
bool TileGetTextureVolumeLayout(uint32_t format, uint32_t width, uint32_t height, uint32_t depth,
uint32_t levels, uint32_t tile, TileVolumeLayout& out) {
uint32_t levels, uint32_t tile, TileVolumeLayout& out) {
if (width == 0 || height == 0 || depth == 0 || levels == 0 || levels > 16) {
return false;
}
@@ -543,21 +543,26 @@ bool TileGetTextureVolumeLayout(uint32_t format, uint32_t width, uint32_t height
TileBlockLayout block {};
if (!TileGetBlockLayout(family, element.bytes, block)) return false;
out = {};
out.family = family;
out.bytes_per_element = element.bytes;
out.texel_width = element.texel_width;
out.texel_height = element.texel_height;
out.block_depth = block.block_depth;
out.first_tail_level = levels;
out = {};
out.family = family;
out.bytes_per_element = element.bytes;
out.texel_width = element.texel_width;
out.texel_height = element.texel_height;
out.block_depth = block.block_depth;
out.first_tail_level = levels;
const uint32_t width0 = (width + element.texel_width - 1u) / element.texel_width;
const uint32_t height0 = (height + element.texel_height - 1u) / element.texel_height;
const bool thick4 = family == TileBlockFamily::Standard4KB3D;
const bool thick64 =
family == TileBlockFamily::Standard64KB3D || family == TileBlockFamily::Prt64KB3D;
const uint32_t max_tail = thick4 ? 5u : (thick64 ? 10u : 12u);
uint32_t tail_width = thick4 ? block.block_width : block.block_width >> 1u;
uint32_t tail_height = thick4 ? block.block_height >> 1u : block.block_height;
uint32_t max_tail = 12u;
if (thick4) {
max_tail = 5u;
} else if (thick64) {
max_tail = 10u;
}
uint32_t tail_width = thick4 ? block.block_width : block.block_width >> 1u;
uint32_t tail_height = thick4 ? block.block_height >> 1u : block.block_height;
if (family == TileBlockFamily::Depth64KB && element.bytes < 4) {
tail_width = 64;
tail_height = 128;
@@ -574,18 +579,23 @@ bool TileGetTextureVolumeLayout(uint32_t format, uint32_t width, uint32_t height
}
out.level_widths[level] = AlignUp(mip_width, block.block_width);
out.level_heights[level] = AlignUp(mip_height, block.block_height);
out.level_sizes[level] = static_cast<uint64_t>(block.block_depth) *
out.level_widths[level] * out.level_heights[level] *
element.bytes;
out.level_sizes[level] = static_cast<uint64_t>(block.block_depth) *
out.level_widths[level] * out.level_heights[level] * element.bytes;
out.block_slice_size += out.level_sizes[level];
}
const auto bytes_log2 = std::countr_zero(element.bytes);
const auto bytes_log2 = std::countr_zero(element.bytes);
const Gen5MipTailLocation* tail_locations = nullptr;
if (thick4) {
tail_locations = GEN5_MIP_TAIL_LOCATIONS_THICK_4KB[bytes_log2];
} else if (thick64) {
tail_locations = GEN5_MIP_TAIL_LOCATIONS_THICK_64KB[bytes_log2];
} else {
tail_locations = GEN5_MIP_TAIL_LOCATIONS_THIN_64KB[bytes_log2];
}
for (uint32_t level = out.first_tail_level; level < levels; ++level) {
const auto index = level - out.first_tail_level;
const auto tail = thick4 ? GEN5_MIP_TAIL_LOCATIONS_THICK_4KB[bytes_log2][index]
: (thick64 ? GEN5_MIP_TAIL_LOCATIONS_THICK_64KB[bytes_log2][index]
: GEN5_MIP_TAIL_LOCATIONS_THIN_64KB[bytes_log2][index]);
const auto index = level - out.first_tail_level;
const auto tail = tail_locations[index];
out.level_sizes[level] = block.block_size;
out.level_widths[level] = block.block_width;
out.level_heights[level] = block.block_height;
@@ -1047,7 +1057,7 @@ static constexpr uint32_t Depth64KB64XOffsetBytes(uint32_t x);
static constexpr uint32_t Depth64KB64YOffsetBytes(uint32_t y);
bool TileGetBlockLayout(TileBlockFamily family, uint32_t bytes_per_element,
TileBlockLayout& layout) {
TileBlockLayout& layout) {
if (!std::has_single_bit(bytes_per_element) || bytes_per_element > 16) {
return false;
}
@@ -1067,13 +1077,25 @@ bool TileGetBlockLayout(TileBlockFamily family, uint32_t bytes_per_element,
};
switch (family) {
case TileBlockFamily::Standard256B:
result.block_size = 256;
result.block_width = bytes_per_element <= 2 ? 16 : (bytes_per_element <= 8 ? 8 : 4);
result.block_size = 256;
if (bytes_per_element <= 2) {
result.block_width = 16;
} else if (bytes_per_element <= 8) {
result.block_width = 8;
} else {
result.block_width = 4;
}
result.block_height = result.block_size / (result.block_width * bytes_per_element);
break;
case TileBlockFamily::Standard4KB:
result.block_size = 4096;
result.block_width = bytes_per_element <= 2 ? 64 : (bytes_per_element <= 8 ? 32 : 16);
result.block_size = 4096;
if (bytes_per_element <= 2) {
result.block_width = 64;
} else if (bytes_per_element <= 8) {
result.block_width = 32;
} else {
result.block_width = 16;
}
result.block_height = result.block_size / (result.block_width * bytes_per_element);
break;
case TileBlockFamily::Standard4KB3D: {
@@ -1126,7 +1148,7 @@ bool TileGetBlockLayout(TileBlockFamily family, uint32_t bytes_per_element,
}
bool TileGetBlockOffset(const TileBlockLayout& layout, uint32_t x, uint32_t y, uint32_t z,
uint32_t& byte_offset) {
uint32_t& byte_offset) {
TileBlockLayout expected {};
if (!TileGetBlockLayout(layout.family, layout.bytes_per_element, expected) ||
layout.block_size != expected.block_size || layout.block_width != expected.block_width ||
@@ -1226,12 +1248,12 @@ bool TileGetBlockOffset(const TileBlockLayout& layout, uint32_t x, uint32_t y, u
}
bool TileGetBlockXor(const TileBlockLayout& layout, uint32_t block_x, uint32_t block_y,
uint32_t& byte_offset) {
uint32_t& byte_offset) {
return TileGetBlockXor(layout, block_x, block_y, 0, byte_offset);
}
bool TileGetBlockXor(const TileBlockLayout& layout, uint32_t block_x, uint32_t block_y,
uint32_t block_z, uint32_t& byte_offset) {
uint32_t block_z, uint32_t& byte_offset) {
TileBlockLayout expected {};
if (!TileGetBlockLayout(layout.family, layout.bytes_per_element, expected) ||
layout.block_size != expected.block_size || layout.block_width != expected.block_width ||
@@ -1377,8 +1399,8 @@ bool TileGetHtileSize(uint32_t width, uint32_t height, TileSizeAlign& htile_size
}
bool TileGetDepthSize(uint32_t width, uint32_t height, uint32_t pitch, uint32_t z_format,
uint32_t stencil_format, bool htile, TileSizeAlign& stencil_size,
TileSizeAlign& htile_size, TileSizeAlign& depth_size,
uint32_t stencil_format, bool htile, TileSizeAlign& stencil_size,
TileSizeAlign& htile_size, TileSizeAlign& depth_size,
uint32_t num_fragments_log2) {
EXIT_IF(pitch != 0);
// Prospero derives uncompressed depth/stencil as independent 64 KiB block surfaces.
@@ -1410,8 +1432,8 @@ bool TileGetDepthSize(uint32_t width, uint32_t height, uint32_t pitch, uint32_t
if (depth_bytes_total <= UINT32_MAX && stencil_bytes_total <= UINT32_MAX && htile_valid) {
depth_size = {static_cast<uint32_t>(depth_bytes_total), 65536};
stencil_size = stencil_format == 1
? TileSizeAlign {static_cast<uint32_t>(stencil_bytes_total), 65536}
: TileSizeAlign {};
? TileSizeAlign {static_cast<uint32_t>(stencil_bytes_total), 65536}
: TileSizeAlign {};
htile_size = calculated_htile;
return true;
}
@@ -1441,7 +1463,7 @@ uint32_t TileGetDepthPitch(uint32_t width, uint32_t bytes_per_element,
}
bool TileGetRenderTargetSize(uint32_t width, uint32_t height, uint32_t pitch,
uint32_t bytes_per_element, TileSizeAlign& total_size,
uint32_t bytes_per_element, TileSizeAlign& total_size,
uint32_t num_fragments_log2) {
total_size = {};
uint32_t block_width = 0;
@@ -1466,7 +1488,7 @@ bool TileGetRenderTargetSize(uint32_t width, uint32_t height, uint32_t pitch,
bool TileGetRenderTargetMipLayout(uint32_t width, uint32_t height, uint32_t pitch,
uint32_t bytes_per_element, uint32_t levels,
TileSizeAlign& total_size, TileSizeOffset* level_sizes,
TileSizeAlign& total_size, TileSizeOffset* level_sizes,
TilePaddedSize* padded_size) {
total_size = {};
if (width == 0 || height == 0 || levels == 0 || levels > 16 ||
@@ -1578,7 +1600,7 @@ void TileGetTextureSize(uint32_t format, uint32_t width, uint32_t height, uint32
void TileGetTextureTotalSize(uint32_t format, uint32_t width, uint32_t height, uint32_t depth,
uint32_t pitch, uint32_t levels, uint32_t tile, bool volume_texture,
TileSizeAlign& total_size) {
TileSizeAlign& total_size) {
EXIT_NOT_IMPLEMENTED(depth == 0);
if (volume_texture) {
TileVolumeLayout volume {};
-551
View File
@@ -1,551 +0,0 @@
#include "graphics/host_gpu/gpuTiler.h"
#include "common/assert.h"
#include "common/threads.h"
#include "gpu_tiler_shaders/gpu_tiler_depth_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_prt_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_prt_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_render_target_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard256_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard4_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard4_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard64_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard64_spv.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/vma.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <array>
#include <bit>
#include <cstring>
#include <memory>
#include <vector>
namespace Libs::Graphics {
namespace {
constexpr uint32_t GROUP_SIZE = 64;
constexpr uint32_t FAMILY_COUNT = static_cast<uint32_t>(TileBlockFamily::Count);
constexpr uint32_t BYTES_PER_ELEMENT_COUNT = 5;
constexpr uint32_t DIRECTION_COUNT = 2;
constexpr uint32_t PIPELINE_COUNT = FAMILY_COUNT * BYTES_PER_ELEMENT_COUNT * DIRECTION_COUNT;
static_assert(FAMILY_COUNT == 9);
struct Push {
uint32_t src_base;
uint32_t dst_base;
uint32_t width;
uint32_t height;
uint32_t depth;
uint32_t surface_z;
uint32_t pitch_bytes;
uint32_t slice_bytes;
uint32_t blocks_per_row;
uint32_t blocks_per_slice;
uint32_t tail_x;
uint32_t tail_y;
uint32_t tail;
uint32_t first;
uint32_t count;
};
static_assert(sizeof(Push) == 60);
struct Shader {
const uint32_t* code;
size_t words;
};
constexpr std::array<Shader, FAMILY_COUNT> SHADERS {{
{GPU_TILER_STANDARD256_SPV, std::size(GPU_TILER_STANDARD256_SPV)},
{GPU_TILER_STANDARD4_SPV, std::size(GPU_TILER_STANDARD4_SPV)},
{GPU_TILER_STANDARD4_3D_SPV, std::size(GPU_TILER_STANDARD4_3D_SPV)},
{GPU_TILER_STANDARD64_SPV, std::size(GPU_TILER_STANDARD64_SPV)},
{GPU_TILER_STANDARD64_3D_SPV, std::size(GPU_TILER_STANDARD64_3D_SPV)},
{GPU_TILER_PRT_SPV, std::size(GPU_TILER_PRT_SPV)},
{GPU_TILER_PRT_3D_SPV, std::size(GPU_TILER_PRT_3D_SPV)},
{GPU_TILER_RENDER_TARGET_SPV, std::size(GPU_TILER_RENDER_TARGET_SPV)},
{GPU_TILER_DEPTH_SPV, std::size(GPU_TILER_DEPTH_SPV)},
}};
struct Dispatch {
Push push {};
uint32_t pipeline_slot = 0;
uint32_t elements = 0;
};
struct Resources {
vk::DescriptorSetLayout descriptor_layout = nullptr;
vk::PipelineLayout pipeline_layout = nullptr;
vk::DescriptorPool descriptor_pool = nullptr;
vk::DescriptorSet descriptor_set = nullptr;
std::array<vk::Pipeline, PIPELINE_COUNT> pipelines {};
VulkanBuffer staging;
VulkanBuffer linear;
void* mapped = nullptr;
};
bool CheckedAdd(uint64_t a, uint64_t b, uint64_t& result) {
return b <= UINT64_MAX - a && (result = a + b, true);
}
bool CheckedMultiply(uint64_t a, uint64_t b, uint64_t& result) {
return (a == 0 || b <= UINT64_MAX / a) && (result = a * b, true);
}
bool CheckedAddProduct(uint64_t& value, uint64_t count, uint64_t stride) {
uint64_t bytes = 0;
return CheckedMultiply(count, stride, bytes) && CheckedAdd(value, bytes, value);
}
bool IsRangeValid(uint64_t offset, uint64_t size, uint64_t capacity) {
return size != 0 && offset <= capacity && size <= capacity - offset;
}
uint64_t AlignToDword(uint64_t value) {
return (value + 3u) & ~uint64_t {3};
}
uint32_t GetPipelineSlot(bool to_tiled, TileBlockFamily family, uint32_t bytes_per_element) {
const uint32_t direction_index = to_tiled ? 1u : 0u;
const uint32_t family_index = static_cast<uint32_t>(family);
const uint32_t element_size_index = std::countr_zero(bytes_per_element);
return (direction_index * FAMILY_COUNT + family_index) * BYTES_PER_ELEMENT_COUNT +
element_size_index;
}
void Barrier(vk::CommandBuffer command, vk::Buffer buffer, vk::AccessFlags src_access,
vk::AccessFlags dst_access, vk::PipelineStageFlags src_stage,
vk::PipelineStageFlags dst_stage) {
vk::BufferMemoryBarrier barrier {};
barrier.sType = vk::StructureType::eBufferMemoryBarrier;
barrier.srcAccessMask = src_access;
barrier.dstAccessMask = dst_access;
barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.buffer = buffer;
barrier.size = VK_WHOLE_SIZE;
command.pipelineBarrier(src_stage, dst_stage, {}, 0, nullptr, 1, &barrier, 0, nullptr);
}
class TileCompute final {
public:
explicit TileCompute(GraphicContext& graphics): graphics(graphics) {}
void Run(bool to_tiled, const void* input, void* output, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& record);
void Release();
private:
void Prepare(bool to_tiled, uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos, std::vector<Dispatch>& dispatches) const;
void Init();
void CreatePipelines(std::span<const Dispatch> dispatches);
void CreatePipeline(uint32_t pipeline_slot);
void Resize(uint64_t staging_size, uint64_t linear_size);
void CreateBuffer(uint64_t size, bool mapped, VulkanBuffer& buffer, void** data) const;
void Execute(bool to_tiled, const void* input, void* output, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const Dispatch> dispatches,
const GpuTileRecord& record);
void Destroy(Resources& target) const;
GraphicContext& graphics;
Resources resources;
};
Common::Mutex g_tiler_mutex;
std::unique_ptr<TileCompute> g_tiler;
void TileCompute::Prepare(bool to_tiled, uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos,
std::vector<Dispatch>& dispatches) const {
EXIT_IF(infos.empty() || tiled_capacity == 0 || linear_capacity == 0);
const auto& limits = graphics.GetPhysicalDeviceProperties().limits;
EXIT_NOT_IMPLEMENTED(tiled_capacity > UINT32_MAX || linear_capacity > UINT32_MAX ||
AlignToDword(tiled_capacity) > limits.maxStorageBufferRange ||
AlignToDword(linear_capacity) > limits.maxStorageBufferRange);
dispatches.clear();
dispatches.reserve(infos.size());
for (const auto& info: infos) {
TileBlockLayout block {};
const uint32_t tiled_width = info.tiled_width != 0 ? info.tiled_width : info.pitch;
const uint32_t tiled_height = info.tiled_height != 0 ? info.tiled_height : info.height;
EXIT_NOT_IMPLEMENTED(
!TileGetBlockLayout(info.family, info.bytes_per_element, block) || info.width == 0 ||
info.height == 0 || info.depth == 0 || info.pitch < info.width ||
(!info.tail && (tiled_width < info.width || tiled_height < info.height)) ||
!IsRangeValid(info.linear_offset, info.linear_size, linear_capacity) ||
!IsRangeValid(info.tiled_offset, info.tiled_size, tiled_capacity) ||
(block.block_depth == 1 && info.depth != 1));
uint64_t elements = 0, pitch_bytes = 0;
EXIT_NOT_IMPLEMENTED(!CheckedMultiply(info.width, info.height, elements) ||
!CheckedMultiply(elements, info.depth, elements) ||
!CheckedMultiply(info.pitch, info.bytes_per_element, pitch_bytes) ||
elements > UINT32_MAX || pitch_bytes > UINT32_MAX);
uint64_t slice_bytes = info.linear_slice_stride;
EXIT_NOT_IMPLEMENTED(slice_bytes == 0 &&
!CheckedMultiply(pitch_bytes, info.height, slice_bytes));
uint64_t linear_used = 0, minimum_slice = 0;
EXIT_NOT_IMPLEMENTED(!CheckedMultiply(pitch_bytes, info.height, minimum_slice) ||
(info.depth > 1 && slice_bytes < minimum_slice) ||
!CheckedAddProduct(linear_used, info.depth - 1u, slice_bytes) ||
!CheckedAddProduct(linear_used, info.height - 1u, pitch_bytes) ||
!CheckedAddProduct(linear_used, info.width, info.bytes_per_element) ||
linear_used > info.linear_size || slice_bytes > UINT32_MAX);
const uint64_t columns =
(static_cast<uint64_t>(tiled_width) + block.block_width - 1u) / block.block_width;
const uint64_t rows =
(static_cast<uint64_t>(tiled_height) + block.block_height - 1u) / block.block_height;
uint64_t blocks_per_slice = 0;
EXIT_NOT_IMPLEMENTED(!CheckedMultiply(columns, rows, blocks_per_slice) ||
columns > UINT32_MAX || blocks_per_slice > UINT32_MAX ||
rows * block.block_height > UINT32_MAX);
if (info.tail) {
const bool supported = info.family != TileBlockFamily::Standard256B;
EXIT_NOT_IMPLEMENTED(
!supported || info.depth > block.block_depth || info.tail_x >= block.block_width ||
info.width > block.block_width - info.tail_x || info.tail_y >= block.block_height ||
info.height > block.block_height - info.tail_y ||
info.tiled_size < block.block_size);
} else {
const uint64_t slices =
(static_cast<uint64_t>(info.depth) + block.block_depth - 1u) / block.block_depth;
uint64_t tiled_used = 0;
EXIT_NOT_IMPLEMENTED(!CheckedMultiply(blocks_per_slice, slices, tiled_used) ||
!CheckedMultiply(tiled_used, block.block_size, tiled_used) ||
tiled_used > info.tiled_size);
}
const uint32_t alignment = std::min(info.bytes_per_element, 4u);
EXIT_NOT_IMPLEMENTED(((info.linear_offset | info.tiled_offset | pitch_bytes | slice_bytes) &
(alignment - 1u)) != 0);
Dispatch dispatch {};
dispatch.elements = static_cast<uint32_t>(elements);
dispatch.pipeline_slot = GetPipelineSlot(to_tiled, info.family, info.bytes_per_element);
dispatch.push.src_base =
static_cast<uint32_t>(to_tiled ? info.linear_offset : info.tiled_offset);
dispatch.push.dst_base =
static_cast<uint32_t>(to_tiled ? info.tiled_offset : info.linear_offset);
dispatch.push.width = info.width;
dispatch.push.height = info.height;
dispatch.push.depth = info.depth;
dispatch.push.surface_z = info.surface_z;
dispatch.push.pitch_bytes = static_cast<uint32_t>(pitch_bytes);
dispatch.push.slice_bytes = static_cast<uint32_t>(slice_bytes);
dispatch.push.blocks_per_row = static_cast<uint32_t>(columns);
dispatch.push.blocks_per_slice = static_cast<uint32_t>(blocks_per_slice);
dispatch.push.tail_x = info.tail_x;
dispatch.push.tail_y = info.tail_y;
dispatch.push.tail = info.tail;
dispatches.push_back(dispatch);
}
}
void TileCompute::Destroy(Resources& target) const {
if (target.mapped != nullptr) {
graphics.UnmapMemory(target.staging.memory);
}
if (target.staging.buffer != nullptr) {
graphics.DeleteBuffer(target.staging);
}
if (target.linear.buffer != nullptr) {
graphics.DeleteBuffer(target.linear);
}
for (auto pipeline: target.pipelines) {
if (pipeline != nullptr) {
graphics.device.destroyPipeline(pipeline, nullptr);
}
}
if (target.descriptor_pool != nullptr) {
graphics.device.destroyDescriptorPool(target.descriptor_pool, nullptr);
}
if (target.pipeline_layout != nullptr) {
graphics.device.destroyPipelineLayout(target.pipeline_layout, nullptr);
}
if (target.descriptor_layout != nullptr) {
graphics.device.destroyDescriptorSetLayout(target.descriptor_layout, nullptr);
}
target = {};
}
void TileCompute::Init() {
if (resources.pipeline_layout != nullptr) {
return;
}
std::array<vk::DescriptorSetLayoutBinding, 2> bindings {};
for (uint32_t i = 0; i < bindings.size(); i++) {
bindings[i] = {i, vk::DescriptorType::eStorageBuffer, 1, vk::ShaderStageFlagBits::eCompute,
nullptr};
}
vk::DescriptorSetLayoutCreateInfo descriptor_info {};
descriptor_info.sType = vk::StructureType::eDescriptorSetLayoutCreateInfo;
descriptor_info.bindingCount = static_cast<uint32_t>(bindings.size());
descriptor_info.pBindings = bindings.data();
RequireVulkanSuccess(graphics.device.createDescriptorSetLayout(&descriptor_info, nullptr,
&resources.descriptor_layout),
"create GPU tiler descriptor layout");
vk::PushConstantRange push_range {vk::ShaderStageFlagBits::eCompute, 0, sizeof(Push)};
vk::PipelineLayoutCreateInfo layout_info {};
layout_info.sType = vk::StructureType::ePipelineLayoutCreateInfo;
layout_info.setLayoutCount = 1;
layout_info.pSetLayouts = &resources.descriptor_layout;
layout_info.pushConstantRangeCount = 1;
layout_info.pPushConstantRanges = &push_range;
RequireVulkanSuccess(
graphics.device.createPipelineLayout(&layout_info, nullptr, &resources.pipeline_layout),
"create GPU tiler pipeline layout");
vk::DescriptorPoolSize pool_size {vk::DescriptorType::eStorageBuffer, 2};
vk::DescriptorPoolCreateInfo pool_info {};
pool_info.sType = vk::StructureType::eDescriptorPoolCreateInfo;
pool_info.maxSets = 1;
pool_info.poolSizeCount = 1;
pool_info.pPoolSizes = &pool_size;
RequireVulkanSuccess(
graphics.device.createDescriptorPool(&pool_info, nullptr, &resources.descriptor_pool),
"create GPU tiler descriptor pool");
vk::DescriptorSetAllocateInfo set_info {};
set_info.sType = vk::StructureType::eDescriptorSetAllocateInfo;
set_info.descriptorPool = resources.descriptor_pool;
set_info.descriptorSetCount = 1;
set_info.pSetLayouts = &resources.descriptor_layout;
RequireVulkanSuccess(
graphics.device.allocateDescriptorSets(&set_info, &resources.descriptor_set),
"allocate GPU tiler descriptor set");
}
void TileCompute::CreatePipeline(uint32_t pipeline_slot) {
const uint32_t element_size_index = pipeline_slot % BYTES_PER_ELEMENT_COUNT;
const uint32_t family_direction_index = pipeline_slot / BYTES_PER_ELEMENT_COUNT;
const uint32_t family_index = family_direction_index % FAMILY_COUNT;
const uint32_t direction_index = family_direction_index / FAMILY_COUNT;
const uint32_t specialization_values[] {1u << element_size_index, direction_index};
const vk::SpecializationMapEntry entries[] {{0, 0, 4}, {1, 4, 4}};
vk::SpecializationInfo specialization {2, entries, sizeof(specialization_values),
specialization_values};
vk::ShaderModuleCreateInfo module_info {};
module_info.sType = vk::StructureType::eShaderModuleCreateInfo;
module_info.codeSize = SHADERS[family_index].words * sizeof(uint32_t);
module_info.pCode = SHADERS[family_index].code;
vk::ShaderModule module = nullptr;
RequireVulkanSuccess(graphics.device.createShaderModule(&module_info, nullptr, &module),
"create GPU tiler shader module");
vk::PipelineShaderStageCreateInfo stage {};
stage.sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stage.stage = vk::ShaderStageFlagBits::eCompute;
stage.module = module;
stage.pName = "main";
stage.pSpecializationInfo = &specialization;
vk::ComputePipelineCreateInfo info {};
info.sType = vk::StructureType::eComputePipelineCreateInfo;
info.stage = stage;
info.layout = resources.pipeline_layout;
vk::Pipeline pipeline = nullptr;
const auto result =
graphics.device.createComputePipelines(nullptr, 1, &info, nullptr, &pipeline);
graphics.device.destroyShaderModule(module, nullptr);
RequireVulkanSuccess(result, "create GPU tiler pipeline");
resources.pipelines[pipeline_slot] = pipeline;
}
void TileCompute::CreatePipelines(std::span<const Dispatch> dispatches) {
for (const auto& dispatch: dispatches) {
if (resources.pipelines[dispatch.pipeline_slot] == nullptr) {
CreatePipeline(dispatch.pipeline_slot);
}
}
}
void TileCompute::CreateBuffer(uint64_t size, bool mapped, VulkanBuffer& buffer,
void** data) const {
buffer.usage = vk::BufferUsageFlagBits::eStorageBuffer | vk::BufferUsageFlagBits::eTransferSrc |
vk::BufferUsageFlagBits::eTransferDst;
buffer.memory.property =
mapped
? vk::MemoryPropertyFlagBits::eHostVisible | vk::MemoryPropertyFlagBits::eHostCoherent
: vk::MemoryPropertyFlags(vk::MemoryPropertyFlagBits::eDeviceLocal);
graphics.CreateBuffer(size, buffer);
if (mapped) graphics.MapMemory(buffer.memory, *data);
}
void TileCompute::Resize(uint64_t staging_size, uint64_t linear_size) {
if (resources.staging.buffer_size >= staging_size &&
resources.linear.buffer_size >= linear_size) {
return;
}
staging_size = std::max(staging_size, resources.staging.buffer_size);
linear_size = std::max(linear_size, resources.linear.buffer_size);
VulkanBuffer staging {}, linear {};
void* mapped = nullptr;
CreateBuffer(staging_size, true, staging, &mapped);
CreateBuffer(linear_size, false, linear, nullptr);
if (resources.mapped != nullptr) graphics.UnmapMemory(resources.staging.memory);
if (resources.staging.buffer != nullptr) graphics.DeleteBuffer(resources.staging);
if (resources.linear.buffer != nullptr) graphics.DeleteBuffer(resources.linear);
resources.staging = staging;
resources.linear = linear;
resources.mapped = mapped;
}
void TileCompute::Execute(bool to_tiled, const void* input, void* output, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const Dispatch> dispatches,
const GpuTileRecord& record) {
const uint64_t tiled_size = AlignToDword(tiled_capacity);
const uint64_t linear_size = AlignToDword(linear_capacity);
const uint64_t input_size = to_tiled ? linear_capacity : tiled_capacity;
if (input != nullptr) {
std::memcpy(resources.mapped, input, static_cast<size_t>(input_size));
std::memset(static_cast<uint8_t*>(resources.mapped) + input_size, 0,
static_cast<size_t>(AlignToDword(input_size) - input_size));
}
std::array<vk::DescriptorBufferInfo, 2> buffer_info {{
{to_tiled ? resources.linear.buffer : resources.staging.buffer, 0,
to_tiled ? linear_size : tiled_size},
{to_tiled ? resources.staging.buffer : resources.linear.buffer, 0,
to_tiled ? tiled_size : linear_size},
}};
std::array<vk::WriteDescriptorSet, 2> writes {};
for (uint32_t i = 0; i < writes.size(); i++) {
writes[i].sType = vk::StructureType::eWriteDescriptorSet;
writes[i].dstSet = resources.descriptor_set;
writes[i].dstBinding = i;
writes[i].descriptorCount = 1;
writes[i].descriptorType = vk::DescriptorType::eStorageBuffer;
writes[i].pBufferInfo = &buffer_info[i];
}
graphics.device.updateDescriptorSets(static_cast<uint32_t>(writes.size()), writes.data(), 0,
nullptr);
CommandBuffer command(GraphicContext::QUEUE_UTIL);
command.Begin();
auto vk_command = command.Handle();
if (input != nullptr) {
Barrier(vk_command, resources.staging.buffer, vk::AccessFlagBits::eHostWrite,
vk::AccessFlagBits::eShaderRead | vk::AccessFlagBits::eTransferRead,
vk::PipelineStageFlagBits::eHost,
vk::PipelineStageFlagBits::eComputeShader | vk::PipelineStageFlagBits::eTransfer);
}
if (to_tiled && input != nullptr) {
const vk::BufferCopy copy {0, 0, linear_size};
vk_command.copyBuffer(resources.staging.buffer, resources.linear.buffer, 1, &copy);
Barrier(vk_command, resources.linear.buffer, vk::AccessFlagBits::eTransferWrite,
vk::AccessFlagBits::eShaderRead, vk::PipelineStageFlagBits::eTransfer,
vk::PipelineStageFlagBits::eComputeShader);
Barrier(vk_command, resources.staging.buffer, vk::AccessFlagBits::eTransferRead,
vk::AccessFlagBits::eTransferWrite, vk::PipelineStageFlagBits::eTransfer,
vk::PipelineStageFlagBits::eTransfer);
}
if (to_tiled && record) {
record(command, resources.linear);
Barrier(vk_command, resources.linear.buffer,
vk::AccessFlagBits::eTransferWrite | vk::AccessFlagBits::eMemoryWrite,
vk::AccessFlagBits::eShaderRead, vk::PipelineStageFlagBits::eAllCommands,
vk::PipelineStageFlagBits::eComputeShader);
}
const auto output_buffer = to_tiled ? resources.staging.buffer : resources.linear.buffer;
const auto output_size = to_tiled ? tiled_size : linear_size;
vk_command.fillBuffer(output_buffer, 0, output_size, 0);
Barrier(vk_command, output_buffer, vk::AccessFlagBits::eTransferWrite,
vk::AccessFlagBits::eShaderRead | vk::AccessFlagBits::eShaderWrite,
vk::PipelineStageFlagBits::eTransfer, vk::PipelineStageFlagBits::eComputeShader);
vk_command.bindDescriptorSets(vk::PipelineBindPoint::eCompute, resources.pipeline_layout, 0, 1,
&resources.descriptor_set, 0, nullptr);
const uint64_t limit =
static_cast<uint64_t>(
graphics.GetPhysicalDeviceProperties().limits.maxComputeWorkGroupCount[0]) *
GROUP_SIZE;
for (const auto& dispatch: dispatches) {
vk_command.bindPipeline(vk::PipelineBindPoint::eCompute,
resources.pipelines[dispatch.pipeline_slot]);
for (uint32_t first = 0; first < dispatch.elements;) {
auto push = dispatch.push;
push.first = first;
push.count =
static_cast<uint32_t>(std::min<uint64_t>(dispatch.elements - first, limit));
vk_command.pushConstants(resources.pipeline_layout, vk::ShaderStageFlagBits::eCompute,
0, sizeof(push), &push);
vk_command.dispatch((push.count - 1u) / GROUP_SIZE + 1u, 1, 1);
first += push.count;
}
}
Barrier(vk_command, output_buffer, vk::AccessFlagBits::eShaderWrite,
vk::AccessFlagBits::eTransferRead | vk::AccessFlagBits::eHostRead,
vk::PipelineStageFlagBits::eComputeShader,
vk::PipelineStageFlagBits::eTransfer | vk::PipelineStageFlagBits::eHost);
if (!to_tiled && record) {
record(command, resources.linear);
}
if (!to_tiled && output != nullptr) {
const vk::BufferCopy copy {0, 0, linear_size};
vk_command.copyBuffer(resources.linear.buffer, resources.staging.buffer, 1, &copy);
Barrier(vk_command, resources.staging.buffer, vk::AccessFlagBits::eTransferWrite,
vk::AccessFlagBits::eHostRead, vk::PipelineStageFlagBits::eTransfer,
vk::PipelineStageFlagBits::eHost);
}
command.End();
command.Execute();
command.WaitForFence();
if (output != nullptr) {
std::memcpy(output, resources.mapped,
static_cast<size_t>(to_tiled ? tiled_capacity : linear_capacity));
}
}
void TileCompute::Run(bool to_tiled, const void* input, void* output, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& record) {
EXIT_IF((to_tiled && (output == nullptr || (input == nullptr && !record))) ||
(!to_tiled && (input == nullptr || (output == nullptr && !record))));
std::vector<Dispatch> dispatches;
Prepare(to_tiled, tiled_capacity, linear_capacity, infos, dispatches);
Init();
CreatePipelines(dispatches);
const uint64_t staging_size =
std::max(AlignToDword(tiled_capacity), AlignToDword(linear_capacity));
const uint64_t linear_size = AlignToDword(linear_capacity);
Resize(staging_size, linear_size);
Execute(to_tiled, input, output, tiled_capacity, linear_capacity, dispatches, record);
}
void TileCompute::Release() {
Destroy(resources);
}
} // namespace
void GpuDetile(const void* tiled, void* linear, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& after) {
Common::LockGuard lock(g_tiler_mutex);
if (!g_tiler) {
g_tiler = std::make_unique<TileCompute>(GetRenderContext().GetGraphics());
}
g_tiler->Run(false, tiled, linear, tiled_capacity, linear_capacity, infos, after);
}
void GpuTile(const void* linear, void* tiled, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& before) {
Common::LockGuard lock(g_tiler_mutex);
if (!g_tiler) {
g_tiler = std::make_unique<TileCompute>(GetRenderContext().GetGraphics());
}
g_tiler->Run(true, linear, tiled, tiled_capacity, linear_capacity, infos, before);
}
void GpuTileRelease() {
Common::LockGuard lock(g_tiler_mutex);
if (g_tiler) {
g_tiler->Release();
g_tiler.reset();
}
}
} // namespace Libs::Graphics
-45
View File
@@ -1,45 +0,0 @@
#pragma once
#include "graphics/guest_gpu/tile.h"
#include <cstdint>
#include <functional>
#include <span>
namespace Libs::Graphics {
struct GraphicContext;
struct VulkanBuffer;
class CommandBuffer;
struct GpuTileInfo {
TileBlockFamily family = TileBlockFamily::Count;
uint32_t bytes_per_element = 0;
uint64_t linear_offset = 0;
uint64_t linear_size = 0;
uint64_t tiled_offset = 0;
uint64_t tiled_size = 0;
uint64_t linear_slice_stride = 0;
uint32_t width = 0;
uint32_t height = 0;
uint32_t depth = 1;
uint32_t pitch = 0;
uint32_t tail_x = 0;
uint32_t tail_y = 0;
bool tail = false;
uint32_t tiled_width = 0;
uint32_t tiled_height = 0;
uint32_t surface_z = 0;
};
using GpuTileRecord = std::function<void(CommandBuffer&, VulkanBuffer&)>;
void GpuDetile(const void* tiled, void* linear, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& after = {});
void GpuTile(const void* linear, void* tiled, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
const GpuTileRecord& before = {});
void GpuTileRelease();
} // namespace Libs::Graphics
+29 -121
View File
@@ -3,13 +3,9 @@
#include "common/abi.h"
#include "common/common.h"
#include "common/threads.h"
#include "graphics/host_gpu/vulkanCommon.h" // IWYU pragma: export
#include "graphics/host_gpu/vulkanInstance.h"
#include <array>
#include <memory>
#include <mutex>
#include <vector>
#include <vk_mem_alloc.h>
@@ -20,49 +16,26 @@ struct VulkanBuffer;
struct VulkanImage;
struct VulkanMemory;
struct VulkanSwapchain {
~VulkanSwapchain();
vk::SwapchainKHR swapchain = nullptr;
vk::Format swapchain_format = vk::Format::eUndefined;
vk::Extent2D swapchain_extent = {};
std::unique_ptr<vk::Image[]> swapchain_images;
std::unique_ptr<vk::ImageView[]> swapchain_image_views;
uint32_t swapchain_images_count = 0;
std::unique_ptr<vk::Semaphore[]> image_acquired_semaphores;
std::unique_ptr<vk::Semaphore[]> render_complete_semaphores;
uint32_t current_index = 0;
uint32_t present_frame = 0;
};
struct VulkanCommandPool {
Common::Mutex mutex;
vk::CommandPool pool = nullptr;
std::unique_ptr<vk::CommandBuffer[]> buffers;
std::unique_ptr<vk::Fence[]> fences;
std::unique_ptr<vk::Semaphore[]> semaphores;
std::unique_ptr<bool[]> busy;
uint32_t buffers_count = 0;
};
struct GraphicContext: public VulkanInstance {
[[nodiscard]] bool CreateAllocator();
void DestroyAllocator();
void LogMemoryBudget() const;
void CreateBuffer(uint64_t size, VulkanBuffer& buffer);
void DeleteBuffer(VulkanBuffer& buffer);
[[nodiscard]] bool CreateImage(const vk::ImageCreateInfo& info, VulkanImage& image);
void DeleteImage(VulkanImage& image);
void MapMemory(VulkanMemory& memory, void*& data);
void UnmapMemory(VulkanMemory& memory);
void AppendHardwareRayTracingDeviceExtensions(
[[nodiscard]] bool CanReportMemoryUsage() const noexcept { return memory_budget_ext_enabled; }
[[nodiscard]] uint64_t GetDeviceMemoryUsage() const;
[[nodiscard]] uint64_t GetTotalMemoryBudget() const;
void CreateBuffer(uint64_t size, VulkanBuffer& buffer);
void DeleteBuffer(VulkanBuffer& buffer);
[[nodiscard]] bool CreateImage(const vk::ImageCreateInfo& info, VulkanImage& image);
void DeleteImage(VulkanImage& image);
void MapMemory(VulkanMemory& memory, void*& data);
void UnmapMemory(VulkanMemory& memory);
void AppendHardwareRayTracingDeviceExtensions(
const std::vector<vk::ExtensionProperties>& available_extensions,
std::vector<const char*>& device_extensions);
void LoadHardwareRayTracingFunctions() const;
uint32_t screen_width = 0;
uint32_t screen_height = 0;
std::array<Common::Mutex, QUEUES_NUM> queue_mutexes;
uint32_t screen_width = 0;
uint32_t screen_height = 0;
};
struct VulkanMemory {
@@ -77,92 +50,29 @@ struct VulkanMemory {
uint64_t unique_id = 0;
};
enum class VulkanImageType {
Unknown,
VideoOut,
DepthStencil,
Texture,
StorageTexture,
RenderTexture
};
struct ImageViewInfo {
vk::Format format = vk::Format::eUndefined;
vk::ImageViewType type = vk::ImageViewType::e2D;
vk::ImageAspectFlags aspect = {};
uint32_t base_level = 0;
uint32_t level_count = 0;
uint32_t base_layer = 0;
uint32_t layer_count = 1;
uint32_t swizzle = 0;
vk::ImageUsageFlags usage = vk::ImageUsageFlagBits::eSampled;
bool operator==(const ImageViewInfo&) const = default;
};
struct CachedImageView {
ImageViewInfo info;
vk::ImageView view = nullptr;
};
struct ImageViewCache {
std::mutex mutex;
std::vector<CachedImageView> views;
ImageViewCache() = default;
KYTY_CLASS_NO_COPY(ImageViewCache);
struct VulkanImageState {
vk::PipelineStageFlags2 pl_stage = vk::PipelineStageFlagBits2::eAllCommands;
vk::AccessFlags2 access_mask = vk::AccessFlagBits2::eNone;
vk::ImageLayout layout = vk::ImageLayout::eUndefined;
};
struct VulkanImage {
static constexpr int VIEW_MAX = 4;
static constexpr int VIEW_DEFAULT = 0;
static constexpr int VIEW_DEFAULT_ARRAY = 1;
static constexpr int VIEW_STORAGE = 2;
static constexpr int VIEW_STORAGE_ARRAY = 3;
explicit VulkanImage(VulkanImageType type): type(type) {}
VulkanImage() = default;
KYTY_CLASS_NO_COPY(VulkanImage);
VulkanImageType type = VulkanImageType::Unknown;
vk::Format format = vk::Format::eUndefined;
vk::Extent2D extent = {};
uint32_t guest_pitch = 0;
uint32_t layers = 1;
uint32_t mip_levels = 1;
uint32_t samples = 1;
vk::Image image = nullptr;
vk::ImageView image_view[VIEW_MAX] = {};
vk::ImageLayout layout = vk::ImageLayout::eUndefined;
vk::Format format = vk::Format::eUndefined;
vk::ImageType image_type = vk::ImageType::e2D;
vk::Extent3D extent = {1, 1, 1};
uint32_t guest_pitch = 0;
uint32_t layers = 1;
uint32_t mip_levels = 1;
uint32_t samples = 1;
vk::ImageUsageFlags usage = {};
vk::ImageCreateFlags flags = {};
vk::Image image = nullptr;
VulkanImageState state;
std::vector<VulkanImageState> subresource_states;
Graphics::VulkanMemory memory;
ImageViewCache view_cache;
};
struct VideoOutVulkanImage: public VulkanImage {
VideoOutVulkanImage(): VulkanImage(VulkanImageType::VideoOut) {}
};
struct DepthStencilVulkanImage: public VulkanImage {
DepthStencilVulkanImage(): VulkanImage(VulkanImageType::DepthStencil) {}
bool compressed = false;
bool initial_depth_clear_pending = false;
bool initial_stencil_clear_pending = false;
};
struct GpuTextureVulkanImage: public VulkanImage {
explicit GpuTextureVulkanImage(VulkanImageType type): VulkanImage(type) {}
};
struct TextureVulkanImage: public GpuTextureVulkanImage {
TextureVulkanImage(): GpuTextureVulkanImage(VulkanImageType::Texture) {}
};
struct StorageTextureVulkanImage: public GpuTextureVulkanImage {
StorageTextureVulkanImage(): GpuTextureVulkanImage(VulkanImageType::StorageTexture) {}
};
struct RenderTextureVulkanImage: public VulkanImage {
RenderTextureVulkanImage(): VulkanImage(VulkanImageType::RenderTexture) {}
bool initial_clear_pending = false;
};
struct VulkanBuffer {
@@ -172,8 +82,6 @@ struct VulkanBuffer {
uint64_t buffer_size = 0;
};
struct StorageVulkanBuffer: public VulkanBuffer {};
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_GRAPHICCONTEXT_H_ */
+38 -33
View File
@@ -4,16 +4,6 @@
namespace Libs::Graphics {
#if defined(KYTY_MEMORY_TRACKER_TESTS)
namespace {
std::atomic<MemoryTracker::UnmapContentionHook> g_unmap_contention_hook {nullptr};
}
void MemoryTracker::SetUnmapContentionHook(UnmapContentionHook hook) noexcept {
g_unmap_contention_hook.store(hook, std::memory_order_release);
}
#endif
static_assert(std::atomic<void*>::is_always_lock_free);
MemoryTracker::MemoryTracker(PageManager& page_manager, PageWatchMode gpu_watch_mode)
@@ -31,6 +21,44 @@ MemoryTracker::MemoryTracker(PageManager& page_manager, PageWatchMode gpu_watch_
MemoryTracker::~MemoryTracker() = default;
void MemoryTracker::ValidateGpuDirtyPages(const RangeSet& dirty, uint64_t vaddr, uint64_t size,
const char* operation) const noexcept {
if (vaddr == 0 || size == 0 || size > UINT64_MAX - vaddr ||
(vaddr & (TRACKER_PAGE_SIZE - 1)) != 0 || (size & (TRACKER_PAGE_SIZE - 1)) != 0) {
EXIT("MemoryTracker: invalid dirty-page validation range\n");
}
for (auto page = vaddr; page < vaddr + size; page += TRACKER_PAGE_SIZE) {
bool found = false;
dirty.ForEachIntersection(page, TRACKER_PAGE_SIZE,
[&found](RangeSet::Range) { found = true; });
if (!found) {
EXIT("MemoryTracker: GPU-dirty tracker page has no dirty bytes, operation=%s "
"addr=0x%016" PRIx64 "\n",
operation, page);
}
}
}
void MemoryTracker::ValidateGpuDirtyOwnership(const RangeSet& dirty, uint64_t vaddr, uint64_t size,
const char* operation) {
ValidateRange(vaddr, size);
if (vaddr + size > UINT64_MAX - (TRACKER_PAGE_SIZE - 1)) {
EXIT("MemoryTracker: dirty ownership range alignment overflow\n");
}
const auto begin = vaddr & ~(TRACKER_PAGE_SIZE - 1);
const auto end = (vaddr + size + TRACKER_PAGE_SIZE - 1) & ~(TRACKER_PAGE_SIZE - 1);
for (auto page = begin; page < end; page += TRACKER_PAGE_SIZE) {
bool has_dirty_bytes = false;
dirty.ForEachIntersection(page, TRACKER_PAGE_SIZE,
[&has_dirty_bytes](RangeSet::Range) { has_dirty_bytes = true; });
if (IsRegionGpuModified(page, TRACKER_PAGE_SIZE) != has_dirty_bytes) {
EXIT("MemoryTracker: tracker and byte ownership disagree, operation=%s "
"addr=0x%016" PRIx64 "\n",
operation, page);
}
}
}
void MemoryTracker::ValidateRange(uint64_t vaddr, uint64_t size) {
if (vaddr == 0 || size == 0 || vaddr >= TRACKER_ADDRESS_SIZE ||
size > TRACKER_ADDRESS_SIZE - vaddr) {
@@ -56,7 +84,6 @@ RegionManager* MemoryTracker::GetOrCreateRegion(uint64_t index) {
bool MemoryTracker::IsRegionCpuModified(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
return Iterate<true>(vaddr, size, [](RegionManager* manager, uint64_t offset, uint64_t bytes) {
std::scoped_lock lock(manager->lock);
return manager->IsModified<DirtySource::Cpu>(offset, bytes);
@@ -66,7 +93,6 @@ bool MemoryTracker::IsRegionCpuModified(uint64_t vaddr, uint64_t size) {
bool MemoryTracker::IsRegionGpuModified(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
return Iterate<false>(vaddr, size, [](RegionManager* manager, uint64_t offset, uint64_t bytes) {
std::scoped_lock lock(manager->lock);
return manager->IsModified<DirtySource::Gpu>(offset, bytes);
@@ -76,7 +102,6 @@ bool MemoryTracker::IsRegionGpuModified(uint64_t vaddr, uint64_t size) {
void MemoryTracker::MarkRegionAsCpuModified(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
Iterate<true>(vaddr, size, [](RegionManager* manager, uint64_t offset, uint64_t bytes) {
std::scoped_lock lock(manager->lock);
const auto changed =
@@ -88,7 +113,6 @@ void MemoryTracker::MarkRegionAsCpuModified(uint64_t vaddr, uint64_t size) {
void MemoryTracker::MarkRegionAsGpuModified(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
Iterate<true>(vaddr, size, [this](RegionManager* manager, uint64_t offset, uint64_t bytes) {
std::scoped_lock lock(manager->lock);
const auto changed =
@@ -100,7 +124,6 @@ void MemoryTracker::MarkRegionAsGpuModified(uint64_t vaddr, uint64_t size) {
void MemoryTracker::UnmarkRegionAsGpuModified(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
Iterate<true>(vaddr, size, [this](RegionManager* manager, uint64_t offset, uint64_t bytes) {
std::scoped_lock lock(manager->lock);
if (!manager->IsFullyModified<DirtySource::Gpu>(offset, bytes)) {
@@ -113,8 +136,6 @@ void MemoryTracker::UnmarkRegionAsGpuModified(uint64_t vaddr, uint64_t size) {
}
void MemoryTracker::UntrackMemoryLocked(uint64_t vaddr, uint64_t size) {
RequireMapped(vaddr, size);
std::vector<RegionManager*> managers;
managers.reserve((vaddr % TRACKER_REGION_SIZE + size + TRACKER_REGION_SIZE - 1) /
TRACKER_REGION_SIZE);
@@ -147,22 +168,6 @@ void MemoryTracker::UntrackMemory(uint64_t vaddr, uint64_t size) {
UntrackMemoryLocked(vaddr, size);
}
void MemoryTracker::UnmapMemory(uint64_t vaddr, uint64_t size) {
CheckNotInUploadCallback();
std::unique_lock access(m_access_mutex, std::try_to_lock);
if (!access.owns_lock()) {
#if defined(KYTY_MEMORY_TRACKER_TESTS)
if (const auto hook = g_unmap_contention_hook.load(std::memory_order_acquire);
hook != nullptr) {
hook();
}
#endif
access.lock();
}
UntrackMemoryLocked(vaddr, size);
m_page_manager.OnGpuUnmap(vaddr, size);
}
bool MemoryTracker::InvalidateRegion(uint64_t vaddr, uint64_t size, PageFaultPhase phase) noexcept {
switch (phase) {
case PageFaultPhase::Release: return true;
+54 -21
View File
@@ -3,6 +3,7 @@
#include "common/assert.h"
#include "graphics/host_gpu/pageManager.h"
#include "graphics/host_gpu/rangeSet.h"
#include "graphics/host_gpu/regionManager.h"
#include <algorithm>
@@ -10,6 +11,7 @@
#include <memory>
#include <mutex>
#include <type_traits>
#include <utility>
#include <vector>
namespace Libs::Graphics {
@@ -28,7 +30,6 @@ public:
void MarkRegionAsGpuModified(uint64_t vaddr, uint64_t size);
void UnmarkRegionAsGpuModified(uint64_t vaddr, uint64_t size);
void UntrackMemory(uint64_t vaddr, uint64_t size);
void UnmapMemory(uint64_t vaddr, uint64_t size);
[[nodiscard]] CpuFaultAction
BeginCpuFault(uint64_t vaddr, uint64_t size,
PageFaultAccess access = PageFaultAccess::Write) noexcept;
@@ -36,16 +37,60 @@ public:
bool downloaded) noexcept;
[[nodiscard]] bool InvalidateRegion(uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept;
template <typename Flush>
void InvalidateRegion(uint64_t vaddr, uint64_t size, Flush&& on_flush) {
static_assert(std::is_invocable_v<Flush&>);
CheckNotInUploadCallback();
ValidateRange(vaddr, size);
const auto update_cpu_state = [this, vaddr, size] {
std::lock_guard access(m_access_mutex);
std::vector<RegionManager*> managers;
Iterate<false>(vaddr, size, [&](RegionManager* manager, uint64_t, uint64_t) {
managers.push_back(manager);
});
std::vector<std::unique_lock<TrackingSpinLock>> locks;
locks.reserve(managers.size());
for (auto* manager: managers) {
locks.emplace_back(manager->lock);
}
const bool gpu_modified = Iterate<false>(
vaddr, size, [](RegionManager* manager, uint64_t offset, uint64_t bytes) {
return manager->IsModified<DirtySource::Gpu>(offset, bytes);
});
if (gpu_modified) {
return true;
}
Iterate<false>(vaddr, size,
[](RegionManager* manager, uint64_t offset, uint64_t bytes) {
const auto changed = manager->ChangeState<DirtySource::Cpu, true>(
manager->GetCpuAddr() + offset, bytes);
manager->ApplyProtection(changed, false);
});
return false;
};
if (!update_cpu_state()) {
return;
}
std::forward<Flush>(on_flush)();
if (update_cpu_state()) {
EXIT("memory invalidation retained GPU-owned pages\n");
}
}
[[nodiscard]] bool InvalidateVirtualGpuWrite(PageFaultAccess access, uint64_t vaddr,
uint64_t size, PageFaultPhase phase) noexcept;
void ValidateGpuDirtyPages(const RangeSet& dirty, uint64_t vaddr, uint64_t size,
const char* operation) const noexcept;
void ValidateGpuDirtyOwnership(const RangeSet& dirty, uint64_t vaddr, uint64_t size,
const char* operation);
template <bool clear, typename Preflight, typename Func>
void ForEachDownloadRange(uint64_t vaddr, uint64_t size, Preflight&& preflight, Func&& func) {
static_assert(std::is_nothrow_invocable_v<Preflight&, uint64_t, uint64_t>);
static_assert(std::is_nothrow_invocable_v<Func&, uint64_t, uint64_t>);
CheckNotInUploadCallback();
std::lock_guard access(m_access_mutex);
RequireMapped(vaddr, size);
std::lock_guard access(m_access_mutex);
std::vector<RegionManager*> managers;
Iterate<false>(vaddr, size, [&](RegionManager* manager, uint64_t, uint64_t) {
managers.push_back(manager);
@@ -85,11 +130,6 @@ public:
vaddr, size, [](uint64_t, uint64_t) noexcept {}, std::forward<Func>(func));
}
#if defined(KYTY_MEMORY_TRACKER_TESTS)
using UnmapContentionHook = void (*)() noexcept;
static void SetUnmapContentionHook(UnmapContentionHook hook) noexcept;
#endif
template <typename RangeFunc, typename UploadFunc>
void ForEachUploadRange(uint64_t vaddr, uint64_t size, bool is_written, RangeFunc&& range_func,
UploadFunc&& upload_func) {
@@ -97,9 +137,8 @@ public:
static_assert(std::is_nothrow_invocable_v<UploadFunc&>);
CheckNotInUploadCallback();
std::unique_lock access(m_access_mutex);
RequireMapped(vaddr, size);
Iterate<true>(vaddr, size, [](RegionManager*, uint64_t, uint64_t) {});
s_upload_owner = this;
const auto* previous_upload_owner = std::exchange(s_upload_owner, this);
Iterate<false>(vaddr, size, [&](RegionManager* manager, uint64_t offset, uint64_t bytes) {
manager->lock.lock();
manager->Track(manager->GetCpuAddr() + offset, bytes);
@@ -119,15 +158,15 @@ public:
manager->lock.unlock();
});
}
s_upload_owner = nullptr;
s_upload_owner = previous_upload_owner;
}
private:
static constexpr size_t REGION_COUNT = TRACKER_ADDRESS_SIZE / TRACKER_REGION_SIZE;
inline static thread_local const MemoryTracker* s_upload_owner = nullptr;
static void CheckNotInUploadCallback() noexcept {
if (s_upload_owner != nullptr) {
void CheckNotInUploadCallback() const noexcept {
if (s_upload_owner == this) {
EXIT("memory tracker re-entered from upload callback\n");
}
}
@@ -162,14 +201,8 @@ private:
return false;
}
static void ValidateRange(uint64_t vaddr, uint64_t size);
void UntrackMemoryLocked(uint64_t vaddr, uint64_t size);
void RequireMapped(uint64_t vaddr, uint64_t size) const {
ValidateRange(vaddr, size);
if (!m_page_manager.IsMapped(vaddr, size)) {
EXIT("memory tracker range is not mapped\n");
}
}
static void ValidateRange(uint64_t vaddr, uint64_t size);
void UntrackMemoryLocked(uint64_t vaddr, uint64_t size);
RegionManager* GetOrCreateRegion(uint64_t index);
std::unique_ptr<std::atomic<RegionManager*>[]> m_regions;
-348
View File
@@ -1,348 +0,0 @@
#include "graphics/host_gpu/objects/label.h"
#include "common/assert.h"
#include "common/threads.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <memory>
#include <vector>
namespace Libs::Graphics {
enum LabelStatus {
New,
Active,
ActiveDeleted,
NotActive,
};
struct LabelCallbacks {
LabelCallback callback_1 = nullptr;
LabelCallback callback_2 = nullptr;
uint64_t args[LABEL_ARGS_MAX] = {};
};
struct LabelEvent final {
vk::Device device = nullptr;
vk::Event event = nullptr;
~LabelEvent() {
if (event != nullptr) {
device.destroyEvent(event, nullptr);
}
}
};
struct LabelSubmission {
std::shared_ptr<LabelEvent> completion;
LabelCallbacks callbacks;
};
struct Label {
vk::Device device = nullptr;
LabelStatus status = LabelStatus::New;
LabelCallbacks callbacks;
std::vector<LabelSubmission> submissions;
};
class LabelManager {
public:
LabelManager() {
EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
Common::Thread t(ThreadRun, this);
t.Detach();
}
~LabelManager() { KYTY_NOT_IMPLEMENTED; }
KYTY_CLASS_NO_COPY(LabelManager);
Label* Create(LabelCallback callback_1, LabelCallback callback_2, const uint64_t* args);
void Delete(Label& label);
void Set(CommandBuffer& buffer, Label& label);
void Drain();
private:
static void ThreadRun(void* data);
bool Remove(Label& label);
static void Destroy(Label& label);
Common::Mutex m_mutex;
Common::CondVar m_cond_var;
std::vector<Label*> m_labels;
uint64_t m_callbacks_in_flight = 0;
};
static LabelManager* g_label_manager = nullptr;
static thread_local bool g_in_label_callback = false;
class LabelCallbackScope final {
public:
LabelCallbackScope() {
if (g_in_label_callback) {
EXIT("recursive GPU label callback\n");
}
g_in_label_callback = true;
}
~LabelCallbackScope() {
if (!g_in_label_callback) {
EXIT("GPU label callback scope is not active\n");
}
g_in_label_callback = false;
}
};
void LabelManager::ThreadRun(void* data) {
auto* manager = static_cast<LabelManager*>(data);
for (;;) {
manager->m_mutex.Lock();
uint64_t active_count = 0;
std::vector<Label*> deleted_labels;
std::vector<LabelCallbacks> fired_labels;
std::vector<LabelSubmission> finished_submissions;
deleted_labels.reserve(manager->m_labels.size());
for (auto& label: manager->m_labels) {
for (auto it = label->submissions.begin(); it != label->submissions.end();) {
active_count++;
if (it->completion == nullptr || it->completion->device == nullptr ||
it->completion->event == nullptr) {
EXIT("GPU label submission has no completion event\n");
}
const auto status = it->completion->device.getEventStatus(it->completion->event);
switch (status) {
case vk::Result::eEventSet:
fired_labels.push_back(it->callbacks);
finished_submissions.push_back(*it);
it = label->submissions.erase(it);
break;
case vk::Result::eEventReset: ++it; break;
default: EXIT("vkGetEventStatus returned an unexpected result\n");
}
}
if (label->submissions.empty()) {
switch (label->status) {
case LabelStatus::ActiveDeleted: deleted_labels.push_back(label); break;
case LabelStatus::Active: label->status = LabelStatus::NotActive; break;
default: break;
}
}
}
if (active_count == 0) {
manager->m_cond_var.Wait(&manager->m_mutex);
}
if (fired_labels.size() > UINT64_MAX - manager->m_callbacks_in_flight) {
EXIT("GPU label callback count overflow\n");
}
manager->m_callbacks_in_flight += fired_labels.size();
for (auto& label: deleted_labels) {
bool removed = manager->Remove(*label);
EXIT_NOT_IMPLEMENTED(!removed);
}
manager->m_mutex.Unlock();
// Each completion event is shared with the recording command buffer's fence retainer.
// The event is destroyed only after both the label thread observed it and that command
// buffer completed, even when either side wins the race.
(void)finished_submissions;
for (auto& label: deleted_labels) {
Destroy(*label);
}
for (auto& label: fired_labels) {
LabelCallbackScope callback_scope;
if (label.callback_1 != nullptr) {
(void)label.callback_1(label.args);
}
if (label.callback_2 != nullptr) {
label.callback_2(label.args);
}
}
if (!fired_labels.empty()) {
Common::LockGuard lock(manager->m_mutex);
if (manager->m_callbacks_in_flight < fired_labels.size()) {
EXIT("GPU label callback count underflow\n");
}
manager->m_callbacks_in_flight -= fired_labels.size();
manager->m_cond_var.SignalAll();
}
Common::Thread::SleepMicro(100);
}
}
Label* LabelManager::Create(LabelCallback callback_1, LabelCallback callback_2,
const uint64_t* args) {
Common::LockGuard lock(m_mutex);
auto* label = new Label;
label->status = LabelStatus::New;
label->device = GetRenderContext().GetGraphics().device;
label->callbacks.callback_1 = callback_1;
label->callbacks.callback_2 = callback_2;
if (args != nullptr) {
for (int i = 0; i < LABEL_ARGS_MAX; i++) {
label->callbacks.args[i] = args[i];
}
}
m_labels.push_back(label);
return label;
}
bool LabelManager::Remove(Label& label) {
EXIT_IF(label.device == nullptr);
Common::LockGuard lock(m_mutex);
const auto it = std::find(m_labels.begin(), m_labels.end(), &label);
EXIT_NOT_IMPLEMENTED(it == m_labels.end());
EXIT_NOT_IMPLEMENTED(label.status != LabelStatus::NotActive &&
label.status != LabelStatus::Active &&
label.status != LabelStatus::ActiveDeleted);
if (!label.submissions.empty()) {
label.status = LabelStatus::ActiveDeleted;
return false;
}
m_labels.erase(it);
return true;
}
void LabelManager::Destroy(Label& label) {
EXIT_IF(label.device == nullptr);
EXIT_NOT_IMPLEMENTED(!label.submissions.empty());
delete &label;
}
void LabelManager::Delete(Label& label) {
if (Remove(label)) {
Destroy(label);
}
}
void LabelManager::Set(CommandBuffer& buffer, Label& label) {
EXIT_IF(buffer.IsInvalid());
EXIT_IF(label.device == nullptr);
Common::LockGuard lock(m_mutex);
const auto it = std::find(m_labels.begin(), m_labels.end(), &label);
EXIT_NOT_IMPLEMENTED(it == m_labels.end());
EXIT_NOT_IMPLEMENTED(label.status != LabelStatus::New &&
label.status != LabelStatus::NotActive &&
label.status != LabelStatus::Active);
label.status = LabelStatus::Active;
LabelSubmission submission {};
submission.callbacks = label.callbacks;
submission.completion = std::make_shared<LabelEvent>();
submission.completion->device = label.device;
auto vk_buffer = buffer.Handle();
EXIT_NOT_IMPLEMENTED(vk_buffer == nullptr);
vk::EventCreateInfo create_info {};
create_info.sType = vk::StructureType::eEventCreateInfo;
create_info.pNext = nullptr;
create_info.flags = {};
const auto create_result =
label.device.createEvent(&create_info, nullptr, &submission.completion->event);
if (create_result != vk::Result::eSuccess || submission.completion->event == nullptr) {
EXIT("failed to create label event: %s (%d)\n", VulkanToString(create_result).c_str(),
static_cast<int>(create_result));
}
buffer.RetainResourceUntilFence(submission.completion);
// Labels can be reused before an earlier end-of-pipe event has been
// observed by the polling thread. Capture a separate Vulkan event and
// callback snapshot for each set so older writes are not lost.
const auto reset_result = label.device.resetEvent(submission.completion->event);
if (reset_result != vk::Result::eSuccess) {
EXIT("failed to reset label event: %s (%d)\n", VulkanToString(reset_result).c_str(),
static_cast<int>(reset_result));
}
vk_buffer.setEvent(submission.completion->event, vk::PipelineStageFlagBits::eBottomOfPipe);
label.submissions.push_back(submission);
m_cond_var.SignalAll();
}
void LabelManager::Drain() {
m_mutex.Lock();
for (;;) {
const bool pending = std::any_of(m_labels.begin(), m_labels.end(), [](const Label* label) {
return !label->submissions.empty();
});
if (!pending && m_callbacks_in_flight == 0) {
m_mutex.Unlock();
return;
}
m_cond_var.SignalAll();
m_cond_var.Wait(&m_mutex);
}
}
void LabelInit() {
EXIT_IF(g_label_manager != nullptr);
g_label_manager = new LabelManager;
}
Label* LabelCreate(LabelCallback callback_1, LabelCallback callback_2, const uint64_t* args) {
EXIT_IF(g_label_manager == nullptr);
return g_label_manager->Create(callback_1, callback_2, args);
}
void LabelDelete(Label& label) {
EXIT_IF(g_label_manager == nullptr);
g_label_manager->Delete(label);
}
void LabelSet(CommandBuffer& buffer, Label& label) {
EXIT_IF(g_label_manager == nullptr);
g_label_manager->Set(buffer, label);
}
void LabelDrain() {
EXIT_IF(g_label_manager == nullptr);
g_label_manager->Drain();
}
bool LabelInCallback() noexcept {
return g_in_label_callback;
}
} // namespace Libs::Graphics
-26
View File
@@ -1,26 +0,0 @@
#ifndef EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_LABEL_H_
#define EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_LABEL_H_
#include "common/abi.h"
#include "common/common.h"
namespace Libs::Graphics {
struct Label;
class CommandBuffer;
struct GraphicContext;
void LabelInit();
constexpr int LABEL_ARGS_MAX = 5;
using LabelCallback = bool (*)(const uint64_t* args);
Label* LabelCreate(LabelCallback callback_1, LabelCallback callback_2, const uint64_t* args);
void LabelDelete(Label& label);
void LabelSet(CommandBuffer& buffer, Label& label);
void LabelDrain();
[[nodiscard]] bool LabelInCallback() noexcept;
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_LABEL_H_ */
File diff suppressed because it is too large Load Diff
@@ -1,137 +0,0 @@
#ifndef EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_TEXTURECOMMON_H_
#define EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_TEXTURECOMMON_H_
#include "common/abi.h"
#include "common/common.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <vector>
namespace Libs::Graphics {
struct GraphicContext;
struct VulkanImage;
enum class TextureFormatUsage : uint32_t {
None = 0,
Sampled = 1u << 0u,
Storage = 1u << 1u,
};
constexpr TextureFormatUsage operator|(TextureFormatUsage lhs, TextureFormatUsage rhs) {
return static_cast<TextureFormatUsage>(static_cast<uint32_t>(lhs) | static_cast<uint32_t>(rhs));
}
constexpr TextureFormatUsage operator&(TextureFormatUsage lhs, TextureFormatUsage rhs) {
return static_cast<TextureFormatUsage>(static_cast<uint32_t>(lhs) & static_cast<uint32_t>(rhs));
}
constexpr TextureFormatUsage operator~(TextureFormatUsage usage) {
return static_cast<TextureFormatUsage>(~static_cast<uint32_t>(usage));
}
constexpr TextureFormatUsage& operator|=(TextureFormatUsage& lhs, TextureFormatUsage rhs) {
lhs = lhs | rhs;
return lhs;
}
constexpr bool TextureHasFormatUsage(TextureFormatUsage usage, TextureFormatUsage flag) {
return (static_cast<uint32_t>(usage & flag) == static_cast<uint32_t>(flag));
}
enum class TextureUploadDestination { MipLevels, MipAtlas };
struct RenderTargetFormatInfo {
vk::Format format = vk::Format::eUndefined;
uint32_t bytes_per_element = 0;
Prospero::ColorComponentMapping export_mapping;
};
struct TextureUploadLayout {
uint32_t tile = 0;
uint32_t pitch = 0;
uint64_t slice_stride = 0;
uint64_t source_slice_stride = 0;
uint32_t first_tail_level = 16;
TileBlockFamily tile_family = TileBlockFamily::Count;
bool volume_texture = false;
TileSizeOffset level_sizes[16] = {};
TilePaddedSize padded_sizes[16] = {};
};
struct TextureImageCreateParams {
uint32_t fmt = 0;
uint64_t width = 0;
uint64_t height = 0;
uint32_t base_level = 0;
uint64_t levels = 1;
uint32_t depth = 1;
uint64_t type = 0;
uint64_t swizzle = 0;
TextureFormatUsage format_usage = TextureFormatUsage::Sampled;
TextureFormatUsage required_format_usage = TextureFormatUsage::Sampled;
TextureFormatUsage view_usage = TextureFormatUsage::Sampled;
TextureUploadDestination image_layout = TextureUploadDestination::MipLevels;
bool allow_cube_view = false;
bool compatible_format_views = false;
bool storage_swizzle_fallback = false;
const char* owner = nullptr;
};
vk::ComponentSwizzle TextureGetComponentSwizzle(uint8_t s);
vk::ComponentMapping TextureGetComponentMapping(uint32_t swizzle);
bool TextureCheckFormat(vk::ImageCreateInfo& image_info);
bool TextureCheckStorageSwizzle(vk::ImageCreateInfo& image_info, vk::ComponentMapping& components);
vk::ImageUsageFlags TextureGetUsage(TextureFormatUsage usage);
vk::ImageUsageFlags TextureGetViewUsage(TextureFormatUsage usage);
vk::Format TextureGetFormat(uint32_t fmt);
RenderTargetFormatInfo TextureGetRenderTargetFormat(uint32_t layout, uint32_t type, uint32_t order);
uint32_t TextureGetAtlasSliceYStride(vk::Format format, uint32_t mip_height, uint32_t depth,
uint64_t levels);
uint32_t TextureCalcStackedImageHeight(vk::Format format, uint32_t height, uint32_t depth,
uint64_t levels);
uint32_t TextureCalcMipmapAtlasImageHeight(vk::Format format, uint32_t width, uint32_t height,
uint32_t depth, uint64_t levels);
bool TextureIs3DTexture(uint64_t type);
bool TextureIsCubeTexture(uint64_t type);
bool TextureIsLayeredTexture(uint64_t type);
bool TextureCanCreateCubeView(uint64_t type, uint32_t base_array, uint32_t layer_count);
vk::ComponentMapping TextureCreateImage(VulkanImage& image,
const TextureImageCreateParams& params);
void TextureCreateImageViews(VulkanImage& vk_obj,
vk::ComponentMapping components, uint64_t type, uint32_t base_array,
uint32_t base_level, uint32_t level_count, uint32_t depth,
bool allow_cube_view, TextureFormatUsage view_usage);
TextureUploadLayout TextureCalcUploadLayout(uint32_t fmt, uint64_t width, uint64_t height,
uint64_t levels, uint32_t depth, uint64_t pitch,
uint64_t tile, uint64_t upload_size,
bool allow_depth_tile, bool volume_texture,
const char* owner);
uint64_t TextureUploadSliceSourceOffset(const TextureUploadLayout& layout, uint32_t level,
uint32_t slice);
uint64_t TextureCalcUploadSize(const TextureUploadLayout& layout,
const std::vector<BufferImageCopy>& regions, uint64_t levels,
uint32_t depth);
std::vector<BufferImageCopy> TextureBuildUploadRegions(const TextureUploadLayout& layout,
vk::Format image_format, uint32_t width,
uint32_t height, uint32_t depth,
uint64_t levels, bool array_texture,
bool volume_texture,
TextureUploadDestination destination);
std::vector<ImageBufferCopy>
TextureBuildDownloadRegions(const std::vector<BufferImageCopy>& upload_regions);
bool TextureBuildGpuTileInfos(uint64_t size, const std::vector<BufferImageCopy>& regions,
const TextureUploadLayout& layout, uint32_t fmt, uint32_t depth,
uint64_t levels, std::vector<GpuTileInfo>& infos);
void TextureUploadGuestImage(VulkanImage& vk_obj, const void* src_data,
uint64_t size, const std::vector<BufferImageCopy>& regions,
const TextureUploadLayout& layout, uint32_t fmt, uint64_t width,
uint64_t height, uint32_t depth, uint64_t levels, const char* owner,
vk::ImageLayout dst_layout);
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_OBJECTS_TEXTURECOMMON_H_ */
+296 -269
View File
@@ -1,7 +1,9 @@
#include "graphics/host_gpu/pageManager.h"
#include "graphics/host_gpu/regionDefinitions.h"
#include "kernel/memory.h"
#include <algorithm>
#include <array>
#include <atomic>
#include <cstdarg>
@@ -19,6 +21,15 @@
#include <windows.h>
#undef min
#undef max
#elif defined(__APPLE__)
#include <pthread.h>
#include <sys/mman.h>
#include <unistd.h>
#else
#include <execinfo.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <unistd.h>
#endif
namespace Libs::Graphics {
@@ -28,8 +39,23 @@ constexpr uint64_t PAGE_SIZE = TRACKER_PAGE_SIZE;
constexpr uint64_t REGION_SIZE = TRACKER_REGION_SIZE;
constexpr uint64_t ADDRESS_SIZE = TRACKER_ADDRESS_SIZE;
constexpr uint64_t REGION_COUNT = ADDRESS_SIZE / REGION_SIZE;
#if KYTY_PLATFORM != KYTY_PLATFORM_WINDOWS
// The tracker reuses Win32 memory-protection tags as internal page-state values (on
// Windows they come from <windows.h> and are what VirtualQuery returns). Mirror the
// canonical Win32 numeric values so the shared state-machine logic is identical.
constexpr uint32_t PAGE_NOACCESS = 0x01;
constexpr uint32_t PAGE_READONLY = 0x02;
constexpr uint32_t PAGE_READWRITE = 0x04;
#endif
constexpr uint64_t REGION_PAGES = REGION_SIZE / PAGE_SIZE;
constexpr uint32_t NO_ACCESS_PROTECTION = PAGE_NOACCESS;
constexpr uint32_t READ_ONLY_PROTECTION = PAGE_READONLY;
constexpr uint32_t READ_WRITE_PROTECTION = PAGE_READWRITE;
// Zero is the unknown protection sentinel.
constexpr uint32_t UNKNOWN_PROTECTION = 0;
thread_local bool g_in_fault_resolution = false;
[[noreturn]] void FailFast(const char* reason = nullptr) noexcept {
@@ -46,6 +72,10 @@ thread_local bool g_in_fault_resolution = false;
std::fprintf(stderr, " frame[%u]=0x%016" PRIxPTR " image_rva=0x%016" PRIxPTR "\n", i,
address, address >= image_base ? address - image_base : 0);
}
#elif !defined(__APPLE__)
void* frames[16] {};
const int frame_count = ::backtrace(frames, static_cast<int>(std::size(frames)));
::backtrace_symbols_fd(frames, frame_count, STDERR_FILENO);
#endif
std::fflush(stderr);
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
@@ -65,11 +95,31 @@ thread_local bool g_in_fault_resolution = false;
std::_Exit(322);
}
Common::VirtualMemory::Mode ToMemoryMode(uint32_t protection) {
switch (protection) {
case NO_ACCESS_PROTECTION: return Common::VirtualMemory::Mode::NoAccess;
case READ_ONLY_PROTECTION: return Common::VirtualMemory::Mode::Read;
case READ_WRITE_PROTECTION: return Common::VirtualMemory::Mode::ReadWrite;
default: Fatal("unmappable protection 0x%08" PRIx32, protection);
}
}
uint32_t CurrentThread() noexcept {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
return GetCurrentThreadId();
#elif defined(__APPLE__)
return static_cast<uint32_t>(pthread_mach_thread_np(pthread_self()));
#elif defined(__linux__)
static thread_local const uint32_t tid = [] {
const auto raw = static_cast<uint32_t>(::syscall(SYS_gettid));
if (raw == 0) {
FailFast("gettid returned the reserved zero owner token");
}
return raw;
}();
return tid;
#else
FailFast();
FailFast("page tracking thread identity is unsupported on this platform");
#endif
}
@@ -106,27 +156,46 @@ uint64_t PageEnd(uint64_t vaddr, uint64_t size) {
struct PageManager::Impl {
struct PageState {
std::atomic_flag lock = ATOMIC_FLAG_INIT;
uint32_t mappings = 0;
uint32_t gpu_read_mappings = 0;
uint32_t gpu_write_mappings = 0;
uint32_t write_watchers = 0;
uint32_t access_watchers = 0;
uint32_t original_protection = 0;
uint32_t backing_writer = 0;
bool resolving = false;
bool resolving_read_write = false;
bool late_read_pending = false;
bool late_write_pending = false;
std::atomic_flag lock = ATOMIC_FLAG_INIT;
uint32_t write_watchers = 0;
uint32_t access_watchers = 0;
uint32_t original_protection = 0;
uint32_t backing_writer = 0;
// Shadow the protection applied through Protect().
uint32_t current_protection = UNKNOWN_PROTECTION;
bool resolving = false;
bool resolving_read_write = false;
bool late_read_pending = false;
bool late_write_pending = false;
};
struct Region {
std::array<PageState, REGION_PAGES> pages;
};
class PageRangeGuard final {
public:
explicit PageRangeGuard(std::span<PageState*> pages): m_pages(pages) {
for (auto* page: m_pages) {
while (page->lock.test_and_set(std::memory_order_acquire)) {
std::atomic_signal_fence(std::memory_order_seq_cst);
}
}
}
~PageRangeGuard() {
for (auto it = m_pages.rbegin(); it != m_pages.rend(); ++it) {
(*it)->lock.clear(std::memory_order_release);
}
}
KYTY_CLASS_NO_COPY(PageRangeGuard);
private:
std::span<PageState*> m_pages;
};
Impl(PageFaultHandler handler, void* context): fault_handler(handler), fault_context(context) {
if (fault_handler == nullptr) {
Fatal("null fault handler");
Fatal("null page-manager fault callback");
}
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
SYSTEM_INFO info {};
@@ -135,8 +204,16 @@ struct PageManager::Impl {
Fatal("unsupported host page size 0x%08" PRIx32,
static_cast<uint32_t>(info.dwPageSize));
}
#elif defined(__APPLE__)
// Under Rosetta the host page size is 4 KB, matching TRACKER_PAGE_SIZE.
if (static_cast<uint64_t>(getpagesize()) != PAGE_SIZE) {
Fatal("unsupported host page size 0x%08" PRIx32, static_cast<uint32_t>(getpagesize()));
}
#else
Fatal("page-fault invalidation is not implemented on this platform");
const auto host_page_size = ::sysconf(_SC_PAGESIZE);
if (host_page_size < 0 || static_cast<uint64_t>(host_page_size) != PAGE_SIZE) {
Fatal("unsupported host page size %ld", static_cast<long>(host_page_size));
}
#endif
regions = std::make_unique<std::atomic<Region*>[]>(REGION_COUNT);
for (uint64_t i = 0; i < REGION_COUNT; i++) {
@@ -148,9 +225,8 @@ struct PageManager::Impl {
for (const auto& region: region_storage) {
for (auto& page: region->pages) {
SpinGuard lock(page.lock);
if (page.mappings != 0 || page.gpu_read_mappings != 0 ||
page.gpu_write_mappings != 0 || page.write_watchers != 0 ||
page.access_watchers != 0 || page.backing_writer != 0 || page.resolving) {
if (page.write_watchers != 0 || page.access_watchers != 0 ||
page.backing_writer != 0 || page.resolving) {
FailFast("PageManager destroyed with live page state");
}
}
@@ -184,82 +260,78 @@ struct PageManager::Impl {
static uint32_t WatcherProtection(const PageState& page) {
if (page.access_watchers != 0) {
return PAGE_NOACCESS;
return NO_ACCESS_PROTECTION;
}
if (page.write_watchers != 0) {
return PAGE_READONLY;
return READ_ONLY_PROTECTION;
}
return page.original_protection;
}
static void PublishDelayedFaults(PageState& page, uint32_t old_protection,
uint32_t new_protection) {
if (old_protection == PAGE_NOACCESS && new_protection != PAGE_NOACCESS) {
if (old_protection == NO_ACCESS_PROTECTION && new_protection != NO_ACCESS_PROTECTION) {
page.late_read_pending = true;
}
if ((old_protection == PAGE_NOACCESS || old_protection == PAGE_READONLY) &&
new_protection == PAGE_READWRITE) {
if ((old_protection == NO_ACCESS_PROTECTION || old_protection == READ_ONLY_PROTECTION) &&
new_protection == READ_WRITE_PROTECTION) {
page.late_write_pending = true;
}
}
static uint32_t QueryProtection(uint64_t vaddr) {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
MEMORY_BASIC_INFORMATION info {};
if (VirtualQuery(reinterpret_cast<const void*>(static_cast<uintptr_t>(vaddr)), &info,
sizeof(info)) == 0 ||
info.State != MEM_COMMIT || info.Protect != PAGE_READWRITE) {
Fatal("basic path requires PAGE_READWRITE at 0x%016" PRIx64 " (state=0x%08" PRIx32
", protection=0x%08" PRIx32 ")",
vaddr, static_cast<uint32_t>(info.State), static_cast<uint32_t>(info.Protect));
static void InitializeProtection(std::span<PageState*> pages) {
for (auto* page: pages) {
page->original_protection = READ_WRITE_PROTECTION;
page->current_protection = READ_WRITE_PROTECTION;
}
return info.Protect;
#else
(void)vaddr;
Fatal("page query is unsupported on this platform");
#endif
}
static bool AllowsAccess(uint64_t vaddr, PageFaultAccess access) noexcept {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
MEMORY_BASIC_INFORMATION info {};
if (VirtualQuery(reinterpret_cast<const void*>(static_cast<uintptr_t>(vaddr)), &info,
sizeof(info)) == 0 ||
info.State != MEM_COMMIT) {
return false;
}
static bool AllowsAccess(const PageState& page, [[maybe_unused]] uint64_t vaddr,
PageFaultAccess access) noexcept {
switch (access) {
case PageFaultAccess::Read:
return info.Protect == PAGE_READONLY || info.Protect == PAGE_READWRITE;
case PageFaultAccess::Write: return info.Protect == PAGE_READWRITE;
return page.current_protection == READ_ONLY_PROTECTION ||
page.current_protection == READ_WRITE_PROTECTION;
case PageFaultAccess::Write: return page.current_protection == READ_WRITE_PROTECTION;
default: return false;
}
#else
(void)vaddr;
return false;
#endif
}
static void Protect(uint64_t vaddr, uint32_t protection, uint32_t expected_old,
bool fault_path) noexcept {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
DWORD old_protection = 0;
if (VirtualProtect(reinterpret_cast<void*>(static_cast<uintptr_t>(vaddr)), PAGE_SIZE,
protection, &old_protection) == 0 ||
old_protection != expected_old) {
if (fault_path) {
FailFast("VirtualProtect fault transition did not match expected protection");
}
Fatal("invalid protection transition at 0x%016" PRIx64 ", old=0x%08" PRIx32
", expected=0x%08" PRIx32 ", new=0x%08" PRIx32,
vaddr, static_cast<uint32_t>(old_protection), expected_old, protection);
void ProtectRange(std::span<PageState*> pages, uint64_t vaddr, uint32_t protection,
std::span<const uint32_t> expected_old, bool fault_path) noexcept {
const auto size = pages.size() * PAGE_SIZE;
if (pages.size() != expected_old.size()) {
FailFast("protection range state size mismatch");
}
#else
(void)vaddr;
(void)protection;
(void)fault_path;
FailFast("page protection is unsupported on this platform");
#endif
for (size_t i = 0; i < pages.size(); i++) {
const auto actual = pages[i]->current_protection;
if (actual != UNKNOWN_PROTECTION && actual != expected_old[i]) {
if (fault_path) {
FailFast("mprotect fault transition did not match expected protection");
}
Fatal("invalid protection transition at 0x%016" PRIx64 ", old=0x%08" PRIx32
", expected=0x%08" PRIx32 ", new=0x%08" PRIx32,
vaddr + i * PAGE_SIZE, actual, expected_old[i], protection);
}
}
if (!Libs::LibKernel::Memory::ProtectGuestHostMemory(vaddr, size,
ToMemoryMode(protection))) {
if (fault_path) {
FailFast("address-space fault protection transition failed");
}
Fatal("address-space protection failed at 0x%016" PRIx64 ", new=0x%08" PRIx32, vaddr,
protection);
}
for (auto* page: pages) {
page->current_protection = protection;
}
}
void Protect(PageState& page, uint64_t vaddr, uint32_t protection, uint32_t expected_old,
bool fault_path) noexcept {
PageState* pages[] = {&page};
uint32_t expected[] = {expected_old};
ProtectRange(pages, vaddr, protection, expected, fault_path);
}
std::unique_ptr<std::atomic<Region*>[]> regions;
@@ -296,199 +368,141 @@ bool PageManager::IsTracked(uint64_t vaddr) const noexcept {
return page.write_watchers != 0 || page.access_watchers != 0;
}
bool PageManager::IsMapped(uint64_t vaddr, uint64_t size) const noexcept {
if (g_in_fault_resolution || vaddr == 0 || size == 0 || vaddr >= ADDRESS_SIZE ||
size > ADDRESS_SIZE - vaddr) {
return false;
}
const auto end = PageStart(vaddr + size - 1) + PAGE_SIZE;
for (auto page_vaddr = PageStart(vaddr); page_vaddr < end; page_vaddr += PAGE_SIZE) {
auto* region = m_impl->FindRegion(page_vaddr);
if (region == nullptr) {
return false;
}
auto& page = m_impl->GetPage(*region, page_vaddr);
SpinGuard lock(page.lock);
if (page.mappings == 0) {
return false;
}
}
return true;
}
bool PageManager::HasAnyMapping(uint64_t vaddr, uint64_t size) const noexcept {
if (g_in_fault_resolution || vaddr == 0 || size == 0 || vaddr >= ADDRESS_SIZE ||
size > ADDRESS_SIZE - vaddr) {
return false;
}
const auto end = PageEnd(vaddr, size);
for (auto page_vaddr = PageStart(vaddr); page_vaddr < end; page_vaddr += PAGE_SIZE) {
auto* region = m_impl->FindRegion(page_vaddr);
if (region == nullptr) {
continue;
}
auto& page = m_impl->GetPage(*region, page_vaddr);
SpinGuard lock(page.lock);
if (page.mappings != 0) {
return true;
}
}
return false;
}
bool PageManager::HasGpuAccess(uint64_t vaddr, uint64_t size, GpuAccess access) const noexcept {
if (access != GpuAccess::Read && access != GpuAccess::Write && access != GpuAccess::ReadWrite) {
FailFast("HasGpuAccess received an invalid GPU access mode");
}
const bool need_read = access == GpuAccess::Read || access == GpuAccess::ReadWrite;
const bool need_write = access == GpuAccess::Write || access == GpuAccess::ReadWrite;
if (vaddr == 0 || size == 0 || vaddr >= ADDRESS_SIZE || size > ADDRESS_SIZE - vaddr) {
return false;
}
const auto end = PageEnd(vaddr, size);
for (auto addr = PageStart(vaddr); addr < end; addr += PAGE_SIZE) {
auto* region = m_impl->FindRegion(addr);
if (region == nullptr) {
return false;
}
auto& page = m_impl->GetPage(*region, addr);
SpinGuard lock(page.lock);
if ((need_read && page.gpu_read_mappings == 0) ||
(need_write && page.gpu_write_mappings == 0)) {
return false;
}
}
return true;
}
void PageManager::UpdatePageWatchers(bool track, uint64_t vaddr, uint64_t size,
PageWatchMode mode) {
if (g_in_fault_resolution) {
FailFast("page watchers changed during fault resolution");
}
if (mode != PageWatchMode::Write && mode != PageWatchMode::ReadWrite) {
Fatal("invalid watcher mode");
}
const auto end = PageEnd(vaddr, size);
for (auto page_vaddr = PageStart(vaddr); page_vaddr < end; page_vaddr += PAGE_SIZE) {
auto* region =
track ? m_impl->GetOrCreateRegion(page_vaddr) : m_impl->FindRegion(page_vaddr);
const auto begin = PageStart(vaddr);
const auto end = PageEnd(vaddr, size);
for (auto chunk_begin = begin; chunk_begin < end;) {
const auto chunk_end = std::min(end, (chunk_begin / REGION_SIZE + 1) * REGION_SIZE);
auto* region =
track ? m_impl->GetOrCreateRegion(chunk_begin) : m_impl->FindRegion(chunk_begin);
if (region == nullptr) {
Fatal("untracking unknown page 0x%016" PRIx64, page_vaddr);
Fatal("untracking unknown page 0x%016" PRIx64, chunk_begin);
}
auto& page = m_impl->GetPage(*region, page_vaddr);
SpinGuard lock(page.lock);
if (page.resolving && track) {
FailFast("new page watcher raced active fault resolution");
const auto page_count = static_cast<size_t>((chunk_end - chunk_begin) / PAGE_SIZE);
std::vector<Impl::PageState*> pages;
pages.reserve(page_count);
for (auto address = chunk_begin; address < chunk_end; address += PAGE_SIZE) {
pages.push_back(&m_impl->GetPage(*region, address));
}
if (page.mappings == 0) {
Fatal("watching unmapped page 0x%016" PRIx64, page_vaddr);
Impl::PageRangeGuard lock(pages);
std::vector<uint8_t> first_watchers(page_count);
for (size_t i = 0; i < page_count; i++) {
auto& page = *pages[i];
const auto address = chunk_begin + i * PAGE_SIZE;
if (page.resolving && track) {
FailFast("new page watcher raced active fault resolution");
}
auto& watchers =
(mode == PageWatchMode::ReadWrite ? page.access_watchers : page.write_watchers);
if (track) {
if (watchers == std::numeric_limits<uint32_t>::max()) {
Fatal("watcher overflow at 0x%016" PRIx64, address);
}
first_watchers[i] = page.write_watchers == 0 && page.access_watchers == 0;
} else {
if (watchers == 0) {
Fatal("watcher underflow at 0x%016" PRIx64, address);
}
if (page.backing_writer != 0 && page.backing_writer != CurrentThread()) {
Fatal("backing write ownership changed at 0x%016" PRIx64, address);
}
}
}
auto& watchers =
(mode == PageWatchMode::ReadWrite ? page.access_watchers : page.write_watchers);
if (track) {
if (watchers == std::numeric_limits<uint32_t>::max()) {
Fatal("watcher overflow at 0x%016" PRIx64, page_vaddr);
}
const bool first_watcher = page.write_watchers == 0 && page.access_watchers == 0;
if (first_watcher) {
page.original_protection = Impl::QueryProtection(page_vaddr);
for (size_t first = 0; first < page_count;) {
while (first < page_count && first_watchers[first] == 0) {
first++;
}
auto last = first;
while (last < page_count && first_watchers[last] != 0) {
last++;
}
if (first != last) {
Impl::InitializeProtection(std::span {pages}.subspan(first, last - first));
}
first = last;
}
}
std::vector<uint32_t> old_protections(page_count);
std::vector<uint32_t> new_protections(page_count);
std::vector<uint8_t> transitions(page_count);
for (size_t i = 0; i < page_count; i++) {
auto& page = *pages[i];
auto& watchers =
(mode == PageWatchMode::ReadWrite ? page.access_watchers : page.write_watchers);
const auto old_protection = Impl::WatcherProtection(page);
watchers++;
const auto new_protection = Impl::WatcherProtection(page);
if (new_protection != old_protection) {
Impl::Protect(page_vaddr, new_protection, old_protection, false);
if (track) {
watchers++;
} else {
watchers--;
}
switch (new_protection) {
case PAGE_NOACCESS:
page.late_read_pending = false;
page.late_write_pending = false;
const auto new_protection = Impl::WatcherProtection(page);
old_protections[i] = old_protection;
new_protections[i] = new_protection;
if (new_protection != old_protection && (track || page.backing_writer == 0)) {
transitions[i] = 1;
}
}
for (size_t first = 0; first < page_count;) {
while (first < page_count && transitions[first] == 0) {
first++;
}
if (first == page_count) {
break;
}
const auto protection = new_protections[first];
auto current = first + 1;
auto last = current;
for (; current < page_count && new_protections[current] == protection; current++) {
if (old_protections[current] != new_protections[current] &&
transitions[current] == 0) {
break;
case PAGE_READONLY: page.late_write_pending = false; break;
default: break;
}
if (transitions[current] != 0) {
last = current + 1;
}
}
} else {
if (watchers == 0) {
Fatal("watcher underflow at 0x%016" PRIx64, page_vaddr);
}
if (page.backing_writer != 0 && page.backing_writer != CurrentThread()) {
Fatal("backing write ownership changed at 0x%016" PRIx64, page_vaddr);
}
const auto old_protection = Impl::WatcherProtection(page);
watchers--;
const auto new_protection = Impl::WatcherProtection(page);
if (page.backing_writer == 0 && new_protection != old_protection) {
Impl::Protect(page_vaddr, new_protection, old_protection, false);
}
if (page.backing_writer == 0) {
Impl::PublishDelayedFaults(page, old_protection, new_protection);
}
if (page.backing_writer == 0 && page.write_watchers == 0 && page.access_watchers == 0) {
page.original_protection = 0;
m_impl->ProtectRange(std::span {pages}.subspan(first, last - first),
chunk_begin + first * PAGE_SIZE, protection,
std::span {old_protections}.subspan(first, last - first), false);
first = current;
}
for (size_t i = 0; i < page_count; i++) {
auto& page = *pages[i];
const auto protection = new_protections[i];
if (track) {
switch (protection) {
case NO_ACCESS_PROTECTION:
page.late_read_pending = false;
page.late_write_pending = false;
break;
case READ_ONLY_PROTECTION: page.late_write_pending = false; break;
default: break;
}
} else if (page.backing_writer == 0) {
Impl::PublishDelayedFaults(page, old_protections[i], protection);
if (page.write_watchers == 0 && page.access_watchers == 0) {
page.original_protection = 0;
}
}
}
chunk_begin = chunk_end;
}
}
void PageManager::OnGpuMap(uint64_t vaddr, uint64_t size, GpuAccess access) {
if (g_in_fault_resolution) {
FailFast("GPU mapping changed during fault resolution");
}
if (access != GpuAccess::Read && access != GpuAccess::Write && access != GpuAccess::ReadWrite) {
FailFast("GPU map received an invalid access mode");
}
const bool gpu_read = access == GpuAccess::Read || access == GpuAccess::ReadWrite;
const bool gpu_write = access == GpuAccess::Write || access == GpuAccess::ReadWrite;
const auto end = PageEnd(vaddr, size);
for (auto addr = PageStart(vaddr); addr < end; addr += PAGE_SIZE) {
auto& page = m_impl->GetPage(*m_impl->GetOrCreateRegion(addr), addr);
SpinGuard lock(page.lock);
if (page.resolving || page.mappings == std::numeric_limits<uint32_t>::max() ||
(gpu_read && page.gpu_read_mappings == std::numeric_limits<uint32_t>::max()) ||
(gpu_write && page.gpu_write_mappings == std::numeric_limits<uint32_t>::max())) {
Fatal("invalid map state at 0x%016" PRIx64, addr);
}
page.mappings++;
page.gpu_read_mappings += gpu_read ? 1u : 0u;
page.gpu_write_mappings += gpu_write ? 1u : 0u;
}
}
void PageManager::OnGpuMap(uint64_t, uint64_t) {}
void PageManager::OnGpuUnmap(uint64_t vaddr, uint64_t size, GpuAccess access) {
if (g_in_fault_resolution) {
FailFast("GPU unmapping changed during fault resolution");
}
if (access != GpuAccess::Read && access != GpuAccess::Write && access != GpuAccess::ReadWrite) {
FailFast("GPU unmap received an invalid access mode");
}
const bool gpu_read = access == GpuAccess::Read || access == GpuAccess::ReadWrite;
const bool gpu_write = access == GpuAccess::Write || access == GpuAccess::ReadWrite;
const auto end = PageEnd(vaddr, size);
for (auto page_vaddr = PageStart(vaddr); page_vaddr < end; page_vaddr += PAGE_SIZE) {
auto* region = m_impl->FindRegion(page_vaddr);
if (region == nullptr) {
Fatal("unmapping unknown page 0x%016" PRIx64, page_vaddr);
}
auto& page = m_impl->GetPage(*region, page_vaddr);
SpinGuard lock(page.lock);
if (page.resolving || page.mappings == 0 || (gpu_read && page.gpu_read_mappings == 0) ||
(gpu_write && page.gpu_write_mappings == 0) ||
(page.mappings == 1 && (page.write_watchers != 0 || page.access_watchers != 0))) {
Fatal("invalid unmap state at 0x%016" PRIx64, page_vaddr);
}
page.mappings--;
page.gpu_read_mappings -= gpu_read ? 1u : 0u;
page.gpu_write_mappings -= gpu_write ? 1u : 0u;
if (page.mappings == 0) {
if (page.gpu_read_mappings != 0 || page.gpu_write_mappings != 0) {
FailFast("GPU unmap left nonzero GPU mapping counts");
}
page.late_read_pending = false;
page.late_write_pending = false;
}
}
}
void PageManager::OnGpuUnmap(uint64_t, uint64_t) {}
PageManager::BackingWrite::BackingWrite(PageManager& manager, uint64_t vaddr,
uint64_t size) noexcept
@@ -500,6 +514,37 @@ PageManager::BackingWrite::~BackingWrite() {
m_manager.EndBackingWrite(m_vaddr, m_size);
}
std::vector<std::unique_ptr<PageManager::BackingWrite>>
PageManager::ReserveBackingWrites(std::span<const RangeSet::Range> ranges) {
if (ranges.empty()) {
Fatal("cannot reserve empty backing-write ranges");
}
std::vector<std::unique_ptr<BackingWrite>> writes;
writes.reserve(ranges.size());
uint64_t begin = 0;
uint64_t end = 0;
for (const auto& range: ranges) {
if (range.address == 0 || range.size == 0 || range.size > UINT64_MAX - range.address ||
range.address + range.size > UINT64_MAX - (PAGE_SIZE - 1)) {
Fatal("invalid backing-write range");
}
const auto page_begin = PageStart(range.address);
const auto page_end = PageStart(range.address + range.size + PAGE_SIZE - 1);
if (begin != 0 && page_begin > end) {
writes.push_back(std::make_unique<BackingWrite>(*this, begin, end - begin));
begin = 0;
}
if (begin == 0) {
begin = page_begin;
end = page_end;
} else {
end = std::max(end, page_end);
}
}
writes.push_back(std::make_unique<BackingWrite>(*this, begin, end - begin));
return writes;
}
void PageManager::BeginBackingWrite(uint64_t vaddr, uint64_t size) noexcept {
if (g_in_fault_resolution) {
FailFast("backing write began during fault resolution");
@@ -513,8 +558,7 @@ void PageManager::BeginBackingWrite(uint64_t vaddr, uint64_t size) noexcept {
}
auto& page = m_impl->GetPage(*region, address);
SpinGuard lock(page.lock);
if (page.mappings == 0 || page.resolving || page.backing_writer != 0 ||
page.access_watchers == 0) {
if (page.resolving || page.backing_writer != 0 || page.access_watchers == 0) {
Fatal("backing write races page resolution at 0x%016" PRIx64, address);
}
page.resolving = true;
@@ -539,10 +583,10 @@ void PageManager::EndBackingWrite(uint64_t vaddr, uint64_t size) noexcept {
if (!page.resolving || page.backing_writer != writer) {
FailFast("backing write ended without matching owner and resolving state");
}
const auto old_protection = PAGE_NOACCESS;
const auto old_protection = NO_ACCESS_PROTECTION;
const auto new_protection = Impl::WatcherProtection(page);
if (new_protection != old_protection) {
Impl::Protect(address, new_protection, old_protection, false);
m_impl->Protect(page, address, new_protection, old_protection, false);
}
Impl::PublishDelayedFaults(page, old_protection, new_protection);
if (page.write_watchers == 0 && page.access_watchers == 0) {
@@ -567,12 +611,12 @@ bool PageManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noex
while (true) {
SpinGuard lock(page.lock);
if (access == PageFaultAccess::Read && page.late_read_pending &&
Impl::AllowsAccess(fault_vaddr, access)) {
Impl::AllowsAccess(page, fault_vaddr, access)) {
page.late_read_pending = false;
return true;
}
if (access == PageFaultAccess::Write && page.late_write_pending &&
Impl::AllowsAccess(fault_vaddr, access)) {
Impl::AllowsAccess(page, fault_vaddr, access)) {
page.late_write_pending = false;
return true;
}
@@ -594,7 +638,7 @@ bool PageManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noex
}
bool& pending = (access == PageFaultAccess::Read ? page.late_read_pending
: page.late_write_pending);
const bool allowed = Impl::AllowsAccess(fault_vaddr, access);
const bool allowed = Impl::AllowsAccess(page, fault_vaddr, access);
pending = false;
if (waited && !allowed) {
FailFast("page remained inaccessible after waiting for its resolver");
@@ -643,12 +687,13 @@ bool PageManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noex
page.write_watchers = 0;
}
const auto restored_protection = Impl::WatcherProtection(page);
Impl::Protect(PageStart(fault_vaddr), restored_protection, old_protection, true);
m_impl->Protect(page, PageStart(fault_vaddr), restored_protection, old_protection,
true);
if (page.write_watchers == 0) {
page.original_protection = 0;
}
Impl::PublishDelayedFaults(page, old_protection, restored_protection);
} else if (!Impl::AllowsAccess(fault_vaddr, access)) {
} else if (!Impl::AllowsAccess(page, fault_vaddr, access)) {
FailFast("fault completion left the page inaccessible");
}
page.resolving = false;
@@ -664,22 +709,4 @@ bool PageManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noex
return true;
}
bool PageManager::HandleWriteRange(uint64_t vaddr, uint64_t size) noexcept {
if (g_in_fault_resolution || vaddr == 0 || size == 0 || vaddr >= ADDRESS_SIZE ||
size > ADDRESS_SIZE - vaddr) {
return false;
}
const auto end = PageEnd(vaddr, size);
for (auto page_vaddr = PageStart(vaddr); page_vaddr < end; page_vaddr += PAGE_SIZE) {
if (!IsMapped(page_vaddr, 1)) {
continue;
}
const auto fault_vaddr = std::max(page_vaddr, vaddr);
if (!HandleFault(PageFaultAccess::Write, fault_vaddr)) {
return false;
}
}
return true;
}
} // namespace Libs::Graphics
+7 -8
View File
@@ -2,19 +2,20 @@
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_PAGEMANAGER_H_
#include "common/common.h"
#include "graphics/host_gpu/rangeSet.h"
#include <memory>
#include <span>
#include <vector>
namespace Libs::Graphics {
enum class PageFaultAccess { Read, Write, Execute, Unknown };
enum class PageFaultPhase { Invalidate, Complete, Release };
enum class PageWatchMode { Write, ReadWrite };
enum class GpuAccess { Read, Write, ReadWrite };
using PageFaultHandler = bool (*)(void* context, PageFaultAccess access, uint64_t vaddr,
uint64_t size, PageFaultPhase phase) noexcept;
class PageManager final {
public:
class BackingWrite final {
@@ -37,17 +38,15 @@ public:
[[nodiscard]] uint64_t GetPageSize() const;
[[nodiscard]] bool IsTracked(uint64_t vaddr) const noexcept;
[[nodiscard]] bool IsMapped(uint64_t vaddr, uint64_t size) const noexcept;
[[nodiscard]] bool HasAnyMapping(uint64_t vaddr, uint64_t size) const noexcept;
[[nodiscard]] bool HasGpuAccess(uint64_t vaddr, uint64_t size, GpuAccess access) const noexcept;
void UpdatePageWatchers(bool track, uint64_t vaddr, uint64_t size,
PageWatchMode mode = PageWatchMode::Write);
void OnGpuMap(uint64_t vaddr, uint64_t size, GpuAccess access = GpuAccess::ReadWrite);
void OnGpuUnmap(uint64_t vaddr, uint64_t size, GpuAccess access = GpuAccess::ReadWrite);
void OnGpuMap(uint64_t vaddr, uint64_t size);
void OnGpuUnmap(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noexcept;
[[nodiscard]] bool HandleWriteRange(uint64_t vaddr, uint64_t size) noexcept;
[[nodiscard]] std::vector<std::unique_ptr<BackingWrite>>
ReserveBackingWrites(std::span<const RangeSet::Range> ranges);
private:
void BeginBackingWrite(uint64_t vaddr, uint64_t size) noexcept;
+10
View File
@@ -59,6 +59,16 @@ public:
return result;
}
[[nodiscard]] bool Contains(uint64_t address, uint64_t size) const {
const auto end = End(address, size);
auto it = m_ranges.upper_bound(address);
if (it == m_ranges.begin()) {
return false;
}
--it;
return it->first <= address && it->second >= end;
}
template <typename Func>
void ForEachIntersection(uint64_t address, uint64_t size, Func&& func) const {
const auto end = End(address, size);
+11
View File
@@ -16,6 +16,11 @@
#include <windows.h>
#undef min
#undef max
#elif defined(__APPLE__)
#include <pthread.h>
#elif defined(__linux__)
#include <sys/syscall.h>
#include <unistd.h>
#endif
namespace Libs::Graphics {
@@ -49,6 +54,12 @@ private:
static uint32_t CurrentThread() noexcept {
#if KYTY_PLATFORM == KYTY_PLATFORM_WINDOWS
return GetCurrentThreadId();
#elif defined(__APPLE__)
// mach thread port is a nonzero per-thread id (0 is the "no owner" sentinel).
return static_cast<uint32_t>(pthread_mach_thread_np(pthread_self()));
#elif defined(__linux__)
static thread_local const uint32_t tid = static_cast<uint32_t>(::syscall(SYS_gettid));
return tid;
#else
EXIT("region tracking thread identity is unsupported on this platform\n");
#endif
File diff suppressed because it is too large Load Diff
@@ -1,101 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
#include "common/abi.h"
#include "common/common.h"
#include "common/threads.h"
#include "graphics/host_gpu/memoryTracker.h"
#include "graphics/host_gpu/rangeSet.h"
#include <map>
#include <memory>
#include <mutex>
namespace Libs::Graphics {
struct GraphicContext;
struct VulkanBuffer;
class CommandBuffer;
class TextureCache;
class ResourceMutex;
struct BufferImageCopySource {
VulkanBuffer* buffer = nullptr;
uint64_t offset = 0;
uint64_t address = 0;
uint64_t size = 0;
bool cpu_current = false;
// True when the guest range was CPU-dirty before coherence resolution. ObtainBufferForImage
// may consume that tracker state while publishing the same bytes to a cached buffer.
bool cpu_dirty = false;
};
struct BufferCacheRange {
uint64_t address = 0;
uint64_t size = 0;
};
struct BufferBinding {
VulkanBuffer& buffer;
uint64_t offset;
};
[[nodiscard]] bool MergeOverlappingBufferCacheRange(BufferCacheRange& merged,
BufferCacheRange candidate) noexcept;
[[nodiscard]] bool CanMergeBufferCacheQueueMask(uint64_t queue_mask, uint32_t queue) noexcept;
class BufferCache {
public:
static constexpr uint64_t CACHING_PAGE_SIZE = 16ull * 1024ull;
static constexpr uint64_t GetBufferOffset(uint64_t vaddr) {
return vaddr & (CACHING_PAGE_SIZE - 1);
}
BufferCache(GraphicContext& graphics, PageManager& page_manager, ResourceMutex& resource_mutex);
~BufferCache();
KYTY_CLASS_NO_COPY(BufferCache);
[[nodiscard]] bool InvalidateMemory(PageFaultAccess access, uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept;
void UnmapMemory(uint64_t vaddr, uint64_t size);
[[nodiscard]] BufferBinding ObtainBuffer(CommandBuffer& command, uint64_t vaddr, uint64_t size,
bool is_written = false, bool is_read = true,
bool is_formatted = false);
// Emulator-owned, CPU-current scratch only. Guest ranges must use ObtainBuffer so page
// ownership is resolved before any CPU access.
[[nodiscard]] bool UploadHostData(CommandBuffer& command, const void* src, uint64_t size,
uint64_t alignment, VulkanBuffer*& out_buffer,
uint64_t& out_offset, uint64_t& out_range);
[[nodiscard]] VulkanBuffer& ObtainNullBuffer(CommandBuffer& command);
[[nodiscard]] BufferImageCopySource ObtainBufferForImage(uint64_t vaddr, uint64_t size);
void FillBuffer(CommandBuffer* command, uint64_t vaddr, uint64_t size, uint32_t value);
void CopyBuffer(CommandBuffer* command, uint64_t dst_vaddr, uint64_t src_vaddr, uint64_t size);
[[nodiscard]] bool HasPageOverlap(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool IsRegionCpuModified(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool IsRegionGpuModified(uint64_t vaddr, uint64_t size);
void PublishImageBacking(uint64_t vaddr, uint64_t size);
void ValidateGpuAccess(uint64_t vaddr, uint64_t size, bool is_read, bool is_written) const;
void SetTextureCache(TextureCache& texture_cache);
void ResetNullBuffer();
private:
struct CachedBuffer;
struct ReadbackWorker;
GraphicContext& m_graphics;
Common::Mutex m_mutex;
std::shared_ptr<VulkanBuffer> m_null_buffer;
// TODO: add LRU cache
std::map<uint64_t, std::unique_ptr<CachedBuffer>> m_buffers;
std::unique_ptr<ReadbackWorker> m_readback;
RangeSet m_gpu_modified_ranges;
MemoryTracker m_memory_tracker;
PageManager& m_page_manager;
TextureCache* m_texture_cache = nullptr;
ResourceMutex& m_resource_mutex;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
File diff suppressed because it is too large Load Diff
+144
View File
@@ -0,0 +1,144 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
#include "common/abi.h"
#include "common/common.h"
#include "common/threads.h"
#include "graphics/host_gpu/memoryTracker.h"
#include "graphics/host_gpu/rangeSet.h"
#include "graphics/host_gpu/renderer/cache/streamBuffer.h"
#include <map>
#include <memory>
#include <mutex>
#include <span>
#include <utility>
#include <vector>
namespace Libs::Graphics {
struct GraphicContext;
class CommandBuffer;
class CommandScheduler;
class TextureCache;
class ResourceMutex;
struct BufferBinding {
std::shared_ptr<void> owner;
vk::Buffer buffer = nullptr;
uint64_t offset = 0;
};
struct ImageBufferSource {
Buffer* buffer = nullptr;
uint64_t offset = 0;
bool gpu_owned = false;
};
class BufferCache {
public:
static constexpr uint64_t CACHING_PAGE_SIZE = 16ull * 1024ull;
static constexpr uint64_t GetBufferOffset(uint64_t vaddr) {
return vaddr & (CACHING_PAGE_SIZE - 1);
}
BufferCache(GraphicContext& graphics, CommandScheduler& scheduler, PageManager& page_manager,
TextureCache& texture_cache, ResourceMutex& resource_mutex);
~BufferCache();
KYTY_CLASS_NO_COPY(BufferCache);
[[nodiscard]] bool InvalidateMemory(PageFaultAccess access, uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept;
void InvalidateMemory(uint64_t vaddr, uint64_t size);
void ReadMemory(uint64_t vaddr, uint64_t size);
void UnmapMemory(uint64_t vaddr, uint64_t size);
[[nodiscard]] BufferBinding ObtainBuffer(CommandBuffer& command, uint64_t vaddr, uint64_t size,
bool is_written = false, bool is_read = true,
bool is_formatted = false);
[[nodiscard]] StreamBuffer& GetUtilityBuffer(MemoryUsage usage) noexcept;
[[nodiscard]] Buffer& GetGdsBuffer() noexcept { return m_gds_buffer; }
[[nodiscard]] const Buffer& GetGdsBuffer() const noexcept { return m_gds_buffer; }
[[nodiscard]] BufferBinding UploadTransient(const void* data, uint64_t size,
uint64_t alignment);
[[nodiscard]] std::shared_ptr<Buffer> ObtainNullBuffer();
[[nodiscard]] ImageBufferSource ObtainBufferForImage(uint64_t vaddr, uint64_t size);
[[nodiscard]] std::pair<std::shared_ptr<Buffer>, uint64_t>
ObtainBufferForImageWrite(uint64_t vaddr, uint64_t size);
void DiscardGpuDirtyBytes(uint64_t vaddr, uint64_t size);
void FillBuffer(uint64_t vaddr, uint64_t size, uint32_t value, bool is_gds = false);
void CopyBuffer(uint64_t dst_vaddr, uint64_t src_vaddr, uint64_t size, bool dst_gds = false,
bool src_gds = false);
[[nodiscard]] bool HasPageOverlap(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool HasGpuDirtyBytes(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool IsRegionCpuModified(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool IsRegionGpuModified(uint64_t vaddr, uint64_t size);
void InvalidateImageAliases(uint64_t vaddr, uint64_t size);
void BeginBackingPublication(uint64_t vaddr, uint64_t size, uint64_t tick);
void CompleteBackingPublication(uint64_t vaddr, uint64_t size, uint64_t tick);
[[nodiscard]] bool SynchronizeBacking(uint64_t vaddr, uint64_t size);
void PublishImageBuffer(uint64_t vaddr, uint64_t size);
void RunGarbageCollector();
private:
friend struct BufferCacheTestAccess;
struct CacheRange {
uint64_t address = 0;
uint64_t size = 0;
};
struct CachedBuffer;
struct DownloadCopy;
struct DownloadRange;
struct RetiredBuffer;
struct FaultReadback;
struct PendingBackingPublication;
static constexpr uint64_t DOWNLOAD_ALIGNMENT = 64;
[[nodiscard]] static uint64_t AlignDown(uint64_t value) noexcept;
[[nodiscard]] static uint64_t AlignUp(uint64_t value);
[[nodiscard]] static constexpr uint64_t AlignDownload(uint64_t size) noexcept {
return (size + DOWNLOAD_ALIGNMENT - 1) & ~(DOWNLOAD_ALIGNMENT - 1);
}
[[nodiscard]] static bool PageOverlaps(uint64_t left, uint64_t left_size, uint64_t right,
uint64_t right_size) noexcept;
[[nodiscard]] static std::pair<uint64_t, uint64_t> DownloadEnvelope(const DownloadCopy& copy);
[[nodiscard]] static bool ResolveOverlap(CacheRange& merged, CacheRange candidate) noexcept;
void Upload(CommandBuffer& command, Buffer& destination, uint64_t destination_offset,
const void* source, uint64_t size);
[[nodiscard]] CachedBuffer& GetOrCreateBuffer(CommandBuffer& command, uint64_t vaddr,
uint64_t size);
[[nodiscard]] std::vector<DownloadRange> RecordDownloads(std::span<const DownloadCopy> copies);
void PublishDownloads(std::span<const DownloadRange> downloads);
void QueueGarbageDownload(std::span<const DownloadCopy> copies, RetiredBuffer retire);
void RefreshInvalidatedRanges(CommandBuffer& command, CachedBuffer& cached, uint64_t vaddr,
uint64_t size, bool upload);
void DiscardGpuDirtyBytesLocked(uint64_t vaddr, uint64_t size, const char* operation);
void WriteHostMemory(uint64_t vaddr, std::span<const uint8_t> data);
GraphicContext& m_graphics;
CommandScheduler& m_scheduler;
Buffer m_gds_buffer;
Common::Mutex m_mutex;
std::shared_ptr<Buffer> m_null_buffer;
std::map<uint64_t, std::unique_ptr<CachedBuffer>> m_buffers;
std::unique_ptr<FaultReadback> m_fault_readback;
RangeSet m_gpu_modified_ranges;
RangeSet m_image_invalidated_ranges;
std::mutex m_publication_mutex;
std::vector<PendingBackingPublication> m_pending_backing_publications;
MemoryTracker m_memory_tracker;
StreamBuffer m_staging_buffer;
StreamBuffer m_stream_buffer;
StreamBuffer m_download_buffer;
StreamBuffer m_device_buffer;
PageManager& m_page_manager;
TextureCache& m_texture_cache;
ResourceMutex& m_resource_mutex;
uint64_t m_total_used_memory = 0;
uint64_t m_trigger_gc_memory = 1ull * 1024 * 1024 * 1024;
uint64_t m_critical_gc_memory = 2ull * 1024 * 1024 * 1024;
uint64_t m_gc_tick = 0;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BUFFERCACHE_H_
@@ -0,0 +1,150 @@
#include "graphics/host_gpu/renderer/cache/gpuResourceManager.h"
#include "common/assert.h"
#include "graphics/guest_gpu/command_processor/commandProcessor.h"
#include "graphics/guest_gpu/graphicsRun.h"
#include "graphics/host_gpu/renderer/commandScheduler.h"
namespace Libs::Graphics {
GpuResourceManager::GpuResourceManager(GraphicContext& graphics, CommandScheduler& scheduler)
: m_page_manager(FaultThunk, this),
m_buffer_cache(graphics, scheduler, m_page_manager, m_texture_cache, m_resource_mutex),
m_texture_cache(graphics, scheduler, m_page_manager, m_buffer_cache, m_resource_mutex) {}
GpuResourceManager::~GpuResourceManager() = default;
bool GpuResourceManager::FaultThunk(void* context, PageFaultAccess access, uint64_t vaddr,
uint64_t size, PageFaultPhase phase) noexcept {
return static_cast<GpuResourceManager*>(context)->InvalidateMemory(access, vaddr, size, phase);
}
bool GpuResourceManager::InvalidateMemory(PageFaultAccess access, uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept {
// Let the authoritative image materialize first. A clean overlapping buffer marks a write
// fault CPU-dirty when it begins ownership transfer; doing that before image preflight would
// make the image appear to race a real CPU write. Completion and release retain buffer-first
// ordering so its pending fault is gone before TextureCache publishes the downloaded backing.
if (phase == PageFaultPhase::Invalidate) {
const bool image_handled = m_texture_cache.InvalidateMemory(access, vaddr, size, phase);
const bool buffer_handled = m_buffer_cache.InvalidateMemory(access, vaddr, size, phase);
return buffer_handled || image_handled;
}
const bool buffer_handled = m_buffer_cache.InvalidateMemory(access, vaddr, size, phase);
const bool image_handled = m_texture_cache.InvalidateMemory(access, vaddr, size, phase);
return buffer_handled || image_handled;
}
bool GpuResourceManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noexcept {
constexpr uint64_t fault_size = 8;
if (!IsMapped(fault_vaddr, fault_size)) {
return false;
}
if (CommandScheduler::InDeferredOperation()) {
EXIT("unsupported guest-memory fault from an asynchronous GPU completion, "
"addr=0x%016" PRIx64 " access=%u\n",
fault_vaddr, static_cast<uint32_t>(access));
}
bool handled = false;
const auto resolve = [this, access, fault_vaddr, &handled](CommandProcessor& cp) {
cp.BeginReadbackTransaction();
{
ResourceMutex::FaultScope fault(m_resource_mutex);
if (access == PageFaultAccess::Write) {
m_buffer_cache.InvalidateMemory(fault_vaddr, fault_size);
m_texture_cache.InvalidateMemory(fault_vaddr, fault_size);
} else {
m_buffer_cache.ReadMemory(fault_vaddr, fault_size);
}
handled = true;
}
cp.EndReadbackTransaction();
};
if (auto* cp = Gpu::CurrentCommandProcessor(); cp != nullptr) {
resolve(*cp);
return handled;
}
if (m_resource_mutex.IsOwnedByCurrentThread()) {
EXIT("unsupported page fault from a pre-owned resource transaction, addr=0x%016" PRIx64
" access=%u\n",
fault_vaddr, static_cast<uint32_t>(access));
}
EXIT_IF(m_gpu == nullptr);
m_gpu->SendCommandSyncWithProcessor(resolve);
return handled;
}
bool GpuResourceManager::InvalidateMemory(uint64_t vaddr, uint64_t size) {
if (!IsMapped(vaddr, size)) {
return false;
}
if (CommandScheduler::InDeferredOperation()) {
EXIT("unsupported memory invalidation from an asynchronous GPU completion, "
"addr=0x%016" PRIx64 " size=0x%016" PRIx64 "\n",
vaddr, size);
}
const auto resolve = [this, vaddr, size](CommandProcessor& cp) {
cp.BeginReadbackTransaction();
{
ResourceMutex::FaultScope fault(m_resource_mutex);
m_buffer_cache.InvalidateMemory(vaddr, size);
m_texture_cache.InvalidateMemory(vaddr, size);
}
cp.EndReadbackTransaction();
};
if (auto* cp = Gpu::CurrentCommandProcessor(); cp != nullptr) {
resolve(*cp);
return true;
}
if (m_resource_mutex.IsOwnedByCurrentThread()) {
EXIT("unsupported memory invalidation from a pre-owned resource transaction, "
"addr=0x%016" PRIx64 " size=0x%016" PRIx64 "\n",
vaddr, size);
}
EXIT_IF(m_gpu == nullptr);
m_gpu->SendCommandSyncWithProcessor(resolve);
return true;
}
bool GpuResourceManager::IsMapped(uint64_t vaddr, uint64_t size) const noexcept {
if (vaddr == 0 || size == 0 || vaddr >= TRACKER_ADDRESS_SIZE ||
size > TRACKER_ADDRESS_SIZE - vaddr) {
return false;
}
std::shared_lock lock(m_mapped_ranges_mutex);
return m_mapped_ranges.Contains(vaddr, size);
}
void GpuResourceManager::MapMemory(uint64_t vaddr, uint64_t size) {
{
std::lock_guard lock(m_mapped_ranges_mutex);
m_mapped_ranges.Add(vaddr, size);
}
m_page_manager.OnGpuMap(vaddr, size);
}
void GpuResourceManager::UnmapMemory(uint64_t vaddr, uint64_t size) {
const auto unmap = [this, vaddr, size] {
m_buffer_cache.UnmapMemory(vaddr, size);
m_texture_cache.UnmapMemory(vaddr, size);
m_page_manager.OnGpuUnmap(vaddr, size);
std::lock_guard lock(m_mapped_ranges_mutex);
m_mapped_ranges.Subtract(vaddr, size);
};
if (m_gpu == nullptr) {
if (m_resource_mutex.IsOwnedByCurrentThread()) {
EXIT("cannot synchronously unmap from a resource transaction\n");
}
unmap();
return;
}
Gpu::SubmissionLock submissions(*m_gpu);
m_gpu->SendCommandSync(unmap);
}
void GpuResourceManager::RunGarbageCollector() {
m_texture_cache.ProcessDownloadImages();
m_texture_cache.RunGarbageCollector();
m_buffer_cache.RunGarbageCollector();
}
} // namespace Libs::Graphics
@@ -4,32 +4,34 @@
#include "common/abi.h"
#include "common/common.h"
#include "graphics/host_gpu/pageManager.h"
#include "graphics/host_gpu/renderer/bufferCache.h"
#include "graphics/host_gpu/renderer/resourceMutex.h"
#include "graphics/host_gpu/renderer/textureCache.h"
#include "graphics/host_gpu/renderer/cache/bufferCache.h"
#include "graphics/host_gpu/renderer/cache/resourceMutex.h"
#include "graphics/host_gpu/renderer/cache/textureCache.h"
#include <cstdint>
#include <shared_mutex>
namespace Libs::Graphics {
class CommandBuffer;
class CommandScheduler;
class Gpu;
class GpuResourceManager {
public:
explicit GpuResourceManager(GraphicContext& graphics);
GpuResourceManager(GraphicContext& graphics, CommandScheduler& scheduler);
~GpuResourceManager();
KYTY_CLASS_NO_COPY(GpuResourceManager);
[[nodiscard]] BufferCache& GetBufferCache() { return m_buffer_cache; }
[[nodiscard]] TextureCache& GetTextureCache() { return m_texture_cache; }
void SetGpu(Gpu* gpu) noexcept { m_gpu = gpu; }
[[nodiscard]] bool HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noexcept;
void PrepareHostWrite(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool InvalidateMemory(uint64_t vaddr, uint64_t size);
[[nodiscard]] bool IsMapped(uint64_t vaddr, uint64_t size) const noexcept;
void MapMemory(uint64_t vaddr, uint64_t size, GpuAccess access);
void UnmapMemory(uint64_t vaddr, uint64_t size, GpuAccess access);
void FillBuffer(CommandBuffer& command, uint64_t vaddr, uint64_t size, uint32_t value);
void CopyBuffer(CommandBuffer& command, uint64_t dst_vaddr, uint64_t src_vaddr, uint64_t size);
void MapMemory(uint64_t vaddr, uint64_t size);
void UnmapMemory(uint64_t vaddr, uint64_t size);
void RunGarbageCollector();
private:
static bool FaultThunk(void* context, PageFaultAccess access, uint64_t vaddr, uint64_t size,
@@ -37,10 +39,13 @@ private:
[[nodiscard]] bool InvalidateMemory(PageFaultAccess access, uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept;
PageManager m_page_manager;
ResourceMutex m_resource_mutex;
BufferCache m_buffer_cache;
TextureCache m_texture_cache;
PageManager m_page_manager;
ResourceMutex m_resource_mutex;
BufferCache m_buffer_cache;
TextureCache m_texture_cache;
mutable std::shared_mutex m_mapped_ranges_mutex;
RangeSet m_mapped_ranges;
Gpu* m_gpu = nullptr;
};
} // namespace Libs::Graphics
@@ -1,13 +1,14 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_MULTILEVELPAGETABLE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_MULTILEVELPAGETABLE_H_
#include "common/assert.h"
#include <algorithm>
#include <array>
#include <cstddef>
#include <cstdint>
#include <list>
#include <memory>
#include <stdexcept>
#include <utility>
#include <vector>
@@ -59,7 +60,7 @@ public:
[[nodiscard]] Entry& GetOrCreate(size_t page) {
if (!IsValidPage(page)) {
throw std::out_of_range("MultiLevelPageTable page is outside the guest address space");
EXIT("MultiLevelPageTable page is outside the guest address space");
}
auto& bucket = m_first_level[FirstLevelIndex(page)];
if (bucket == nullptr) {
@@ -1,4 +1,4 @@
#include "graphics/host_gpu/renderer/resourceMutex.h"
#include "graphics/host_gpu/renderer/cache/resourceMutex.h"
#include "common/assert.h"
@@ -9,7 +9,7 @@
namespace Libs::Graphics {
// Owner-tracked shared buffer/image transaction. External faults pause GPU submissions first;
// command-processor faults drain their queue before entering this transaction.
// command-processor faults drain pending guest processors before entering this transaction.
class ResourceMutex final {
public:
class FaultScope final {
@@ -1,4 +1,4 @@
#include "graphics/host_gpu/renderer/samplerCache.h"
#include "graphics/host_gpu/renderer/cache/samplerCache.h"
#include "common/assert.h"
#include "common/logging/log.h"
@@ -7,6 +7,13 @@
namespace Libs::Graphics {
SamplerCache::~SamplerCache() {
for (const auto& [key, sampler]: m_samplers) {
(void)key;
m_graphics.device.destroySampler(sampler, nullptr);
}
}
vk::Sampler SamplerCache::GetSampler(const ShaderSamplerResource& r) {
Common::LockGuard lock(m_mutex);
@@ -21,7 +21,7 @@ public:
explicit SamplerCache(GraphicContext& graphics): m_graphics(graphics) {
EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
}
~SamplerCache() { KYTY_NOT_IMPLEMENTED; }
~SamplerCache();
KYTY_CLASS_NO_COPY(SamplerCache);
vk::Sampler GetSampler(const ShaderSamplerResource& r);
+360
View File
@@ -0,0 +1,360 @@
#include "graphics/host_gpu/renderer/cache/streamBuffer.h"
#include "common/assert.h"
#include "common/profiler.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/commandScheduler.h"
#include "graphics/host_gpu/vma.h"
#include <cstring>
#include <limits>
#include <numeric>
#include <vk_mem_alloc.h>
namespace Libs::Graphics {
namespace {
constexpr size_t WATCHES_INITIAL_RESERVE = 0x4000;
constexpr size_t WATCHES_RESERVE_CHUNK = 0x1000;
[[nodiscard]] VmaAllocationCreateFlags AllocationFlags(MemoryUsage usage) {
switch (usage) {
case MemoryUsage::Upload:
case MemoryUsage::Stream:
return VMA_ALLOCATION_CREATE_MAPPED_BIT |
VMA_ALLOCATION_CREATE_HOST_ACCESS_SEQUENTIAL_WRITE_BIT;
case MemoryUsage::Download:
return VMA_ALLOCATION_CREATE_MAPPED_BIT | VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT;
case MemoryUsage::DeviceLocal: return {};
}
return {};
}
[[nodiscard]] VmaMemoryUsage AllocationUsage(MemoryUsage usage) {
switch (usage) {
case MemoryUsage::DeviceLocal:
case MemoryUsage::Stream: return VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE;
case MemoryUsage::Upload:
case MemoryUsage::Download: return VMA_MEMORY_USAGE_AUTO_PREFER_HOST;
}
return VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE;
}
[[nodiscard]] bool AlignUp(uint64_t value, uint64_t alignment, uint64_t& result) {
if (alignment == 0) {
result = value;
return true;
}
const auto remainder = value % alignment;
if (remainder == 0) {
result = value;
return true;
}
const auto increment = alignment - remainder;
if (value > std::numeric_limits<uint64_t>::max() - increment) {
return false;
}
result = value + increment;
return true;
}
} // namespace
Buffer::Buffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsage usage,
uint64_t cpu_address, vk::BufferUsageFlags flags, uint64_t size)
: m_graphics(&graphics), m_scheduler(&scheduler), m_usage(usage), m_cpu_address(cpu_address),
m_size(size), m_buffer(std::make_unique<VulkanBuffer>()) {
KYTY_PROFILER_FUNCTION();
EXIT_IF(graphics.allocator == nullptr || size == 0);
vk::BufferCreateInfo buffer_info {};
buffer_info.size = size;
buffer_info.usage = flags;
buffer_info.sharingMode = vk::SharingMode::eExclusive;
VmaAllocationCreateInfo allocation_info {};
allocation_info.flags = VMA_ALLOCATION_CREATE_WITHIN_BUDGET_BIT | AllocationFlags(usage);
allocation_info.usage = AllocationUsage(usage);
allocation_info.preferredFlags = usage == MemoryUsage::DeviceLocal
? VkMemoryPropertyFlags {}
: VK_MEMORY_PROPERTY_HOST_COHERENT_BIT;
VmaAllocationInfo allocation_result {};
VkBuffer native_buffer = VK_NULL_HANDLE;
const auto result = static_cast<vk::Result>(vmaCreateBuffer(
graphics.allocator, static_cast<const VkBufferCreateInfo*>(buffer_info), &allocation_info,
&native_buffer, &m_buffer->memory.allocation, &allocation_result));
if (result != vk::Result::eSuccess) {
graphics.LogMemoryBudget();
}
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
m_buffer->buffer = native_buffer;
m_buffer->usage = flags;
m_buffer->buffer_size = size;
m_buffer->memory.allocation_info = allocation_result;
m_buffer->memory.memory = allocation_result.deviceMemory;
m_buffer->memory.offset = allocation_result.offset;
m_buffer->memory.type = allocation_result.memoryType;
m_buffer->memory.unique_id = VulkanNextMemoryUniqueId();
graphics.device.getBufferMemoryRequirements(m_buffer->buffer, &m_buffer->memory.requirements);
VkMemoryPropertyFlags properties = 0;
vmaGetAllocationMemoryProperties(graphics.allocator, m_buffer->memory.allocation, &properties);
m_buffer->memory.property = vk::MemoryPropertyFlags(properties);
m_is_coherent = (properties & VK_MEMORY_PROPERTY_HOST_COHERENT_BIT) != 0;
if (allocation_result.pMappedData != nullptr) {
m_mapped = {static_cast<uint8_t*>(allocation_result.pMappedData),
static_cast<size_t>(size)};
}
VulkanTrackAllocation(m_buffer->memory);
}
Buffer::~Buffer() {
if (m_buffer->buffer != nullptr) {
VulkanUntrackAllocation(m_buffer->memory);
vmaDestroyBuffer(m_graphics->allocator, m_buffer->buffer, m_buffer->memory.allocation);
}
}
vk::Buffer Buffer::Handle() const noexcept {
return m_buffer->buffer;
}
bool Buffer::IsInBounds(uint64_t address, uint64_t size) const noexcept {
return address >= m_cpu_address && size <= m_size && address - m_cpu_address <= m_size - size;
}
void Buffer::Write(uint64_t offset, const void* source, uint64_t size) {
EXIT_IF(source == nullptr || m_mapped.empty() || offset > m_size || size > m_size - offset);
std::memcpy(m_mapped.data() + offset, source, static_cast<size_t>(size));
Flush(offset, size);
}
void Buffer::Flush(uint64_t offset, uint64_t size) {
EXIT_IF(m_mapped.empty() || offset > m_size || size > m_size - offset);
if (!m_is_coherent && size != 0) {
const auto result = vmaFlushAllocation(m_graphics->allocator, m_buffer->memory.allocation,
offset, size);
EXIT_NOT_IMPLEMENTED(static_cast<vk::Result>(result) != vk::Result::eSuccess);
}
}
vk::BufferMemoryBarrier Buffer::Barrier(uint64_t offset, uint64_t size, vk::AccessFlags source,
vk::AccessFlags destination) const {
if (Handle() == nullptr || size == 0 || offset > m_size || size > m_size - offset) {
EXIT("Buffer: invalid DMA barrier, handle=%p offset=0x%016" PRIx64
" size=0x%016" PRIx64 " capacity=0x%016" PRIx64 "\n",
static_cast<const void*>(Handle()), offset, size, m_size);
}
vk::BufferMemoryBarrier barrier {};
barrier.sType = vk::StructureType::eBufferMemoryBarrier;
barrier.srcAccessMask = source;
barrier.dstAccessMask = destination;
barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.buffer = Handle();
barrier.offset = offset;
barrier.size = size;
return barrier;
}
void Buffer::CopyFrom(CommandBuffer& command, const Buffer& source, uint64_t source_offset,
uint64_t destination_offset, uint64_t size, vk::AccessFlags source_before,
vk::AccessFlags destination_before, vk::AccessFlags source_after,
vk::AccessFlags destination_after) {
if (size == 0 || source_offset > source.m_size || size > source.m_size - source_offset ||
destination_offset > m_size || size > m_size - destination_offset) {
EXIT("Buffer: invalid copy range\n");
}
if (source.Handle() == Handle() && source_offset < destination_offset + size &&
destination_offset < source_offset + size) {
EXIT("Buffer: overlapping self-copy\n");
}
command.EndRendering();
const vk::BufferMemoryBarrier before[] = {
source.Barrier(source_offset, size, source_before, vk::AccessFlagBits::eTransferRead),
Barrier(destination_offset, size, destination_before,
vk::AccessFlagBits::eTransferWrite),
};
const auto host_access = vk::AccessFlagBits::eHostRead | vk::AccessFlagBits::eHostWrite;
auto before_stage = vk::PipelineStageFlags {vk::PipelineStageFlagBits::eAllCommands};
if (static_cast<bool>((source_before | destination_before) & host_access)) {
before_stage |= vk::PipelineStageFlagBits::eHost;
}
const auto native = command.Handle();
native.pipelineBarrier(before_stage, vk::PipelineStageFlagBits::eTransfer,
vk::DependencyFlagBits::eByRegion, 0, nullptr, 2, before, 0, nullptr);
const vk::BufferCopy copy {source_offset, destination_offset, size};
native.copyBuffer(source.Handle(), Handle(), 1, &copy);
const vk::BufferMemoryBarrier after[] = {
source.Barrier(source_offset, size, vk::AccessFlagBits::eTransferRead, source_after),
Barrier(destination_offset, size, vk::AccessFlagBits::eTransferWrite, destination_after),
};
auto after_stage = vk::PipelineStageFlags {vk::PipelineStageFlagBits::eAllCommands};
if (static_cast<bool>((source_after | destination_after) & host_access)) {
after_stage |= vk::PipelineStageFlagBits::eHost;
}
native.pipelineBarrier(vk::PipelineStageFlagBits::eTransfer, after_stage,
vk::DependencyFlagBits::eByRegion, 0, nullptr, 2, after, 0, nullptr);
}
void Buffer::Fill(uint64_t offset, uint64_t size, uint32_t value) {
if (((offset | size) & 3u) != 0) {
EXIT("Buffer: fill range must be dword aligned\n");
}
auto& command = Scheduler().Current();
command.EndRendering();
const auto before =
Barrier(offset, size, vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite,
vk::AccessFlagBits::eTransferWrite);
const auto native = command.Handle();
native.pipelineBarrier(vk::PipelineStageFlagBits::eAllCommands,
vk::PipelineStageFlagBits::eTransfer, vk::DependencyFlagBits::eByRegion,
0, nullptr, 1, &before, 0, nullptr);
native.fillBuffer(Handle(), offset, size, value);
const auto after =
Barrier(offset, size, vk::AccessFlagBits::eTransferWrite,
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite);
native.pipelineBarrier(vk::PipelineStageFlagBits::eTransfer,
vk::PipelineStageFlagBits::eAllCommands,
vk::DependencyFlagBits::eByRegion, 0, nullptr, 1, &after, 0, nullptr);
}
StreamBuffer::StreamBuffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsage usage,
uint64_t size)
: Buffer(graphics, scheduler, usage, 0, AllFlags, size) {
ReserveWatches(m_current_watches, WATCHES_INITIAL_RESERVE);
ReserveWatches(m_previous_watches, WATCHES_INITIAL_RESERVE);
}
bool StreamBuffer::NormalizeReservation(bool coherent, uint64_t atom, uint64_t& size,
uint64_t& alignment) {
if (coherent) {
return true;
}
if (!AlignUp(size, atom, size)) {
return false;
}
const auto divisor = std::gcd(alignment, atom);
if (alignment != 0 && alignment / divisor > UINT64_MAX / atom) {
return false;
}
alignment = alignment == 0 ? atom : alignment / divisor * atom;
return true;
}
std::pair<uint8_t*, uint64_t> StreamBuffer::Map(uint64_t size, uint64_t alignment,
bool allow_wait) {
if (Mapped().empty()) {
return {nullptr, 0};
}
uint64_t mapped_size = size;
const auto atom = Graphics().physical_device_properties.limits.nonCoherentAtomSize;
if (!NormalizeReservation(IsCoherent(), atom, mapped_size, alignment)) {
return {nullptr, 0};
}
if (mapped_size > Size()) {
return {nullptr, 0};
}
uint64_t aligned_offset = 0;
if (!AlignUp(m_offset, alignment, aligned_offset)) {
return {nullptr, 0};
}
const bool wrap = aligned_offset > Size() - mapped_size;
if (wrap) {
aligned_offset = 0;
}
auto wait_cursor = wrap ? size_t {0} : m_wait_cursor;
auto wait_bound = wrap ? uint64_t {0} : m_wait_bound;
auto invalidation_mark =
wrap ? std::optional<size_t> {m_current_watch_cursor} : m_invalidation_mark;
auto& pending_watches = wrap ? m_current_watches : m_previous_watches;
if (!WaitPendingOperations(pending_watches, invalidation_mark, aligned_offset + mapped_size,
allow_wait, wait_cursor, wait_bound)) {
return {nullptr, 0};
}
if (wrap) {
m_invalidation_mark = invalidation_mark;
m_current_watch_cursor = 0;
std::swap(m_previous_watches, m_current_watches);
}
m_wait_cursor = wait_cursor;
m_wait_bound = wait_bound;
m_offset = aligned_offset;
m_mapped_size = mapped_size;
return {Mapped().data() + m_offset, m_offset};
}
void StreamBuffer::Commit() {
if (!IsCoherent() && Usage() != MemoryUsage::Download && m_mapped_size != 0) {
const auto result = vmaFlushAllocation(
Graphics().allocator, NativeBuffer().memory.allocation, m_offset, m_mapped_size);
EXIT_NOT_IMPLEMENTED(static_cast<vk::Result>(result) != vk::Result::eSuccess);
}
m_offset += m_mapped_size;
const auto tick = Scheduler().CurrentTick();
if (m_current_watch_cursor != 0 && m_current_watches[m_current_watch_cursor - 1].tick == tick) {
m_current_watches[m_current_watch_cursor - 1].upper_bound = m_offset;
return;
}
if (m_current_watch_cursor + 1 >= m_current_watches.size()) {
ReserveWatches(m_current_watches, WATCHES_RESERVE_CHUNK);
}
auto& watch = m_current_watches[m_current_watch_cursor++];
watch.upper_bound = m_offset;
watch.tick = tick;
}
void StreamBuffer::Invalidate(uint64_t offset, uint64_t size) {
EXIT_IF(Usage() != MemoryUsage::Download || offset > Size() || size > Size() - offset);
if (IsCoherent() || size == 0) {
return;
}
const auto result = vmaInvalidateAllocation(Graphics().allocator,
NativeBuffer().memory.allocation, offset, size);
EXIT_NOT_IMPLEMENTED(static_cast<vk::Result>(result) != vk::Result::eSuccess);
}
uint64_t StreamBuffer::Copy(const void* source, uint64_t size, uint64_t alignment) {
EXIT_IF(source == nullptr);
const auto [data, offset] = Map(size, alignment);
EXIT_IF(data == nullptr);
std::memcpy(data, source, static_cast<size_t>(size));
Commit();
return offset;
}
void StreamBuffer::ReserveWatches(std::vector<Watch>& watches, size_t grow_size) {
watches.resize(watches.size() + grow_size);
}
bool StreamBuffer::WaitPendingOperations(const std::vector<Watch>& watches,
std::optional<size_t> invalidation_mark,
uint64_t requested_upper_bound, bool allow_wait,
size_t& wait_cursor, uint64_t& wait_bound) {
if (!invalidation_mark.has_value()) {
return true;
}
while (requested_upper_bound > wait_bound && wait_cursor < *invalidation_mark) {
const auto& watch = watches[wait_cursor];
if (!Scheduler().IsFree(watch.tick) && !allow_wait) {
return false;
}
Scheduler().Wait(watch.tick);
if (Usage() == MemoryUsage::Download) {
Scheduler().WaitPriorityOperations(watch.tick);
}
wait_bound = watch.upper_bound;
++wait_cursor;
}
return true;
}
} // namespace Libs::Graphics
+130
View File
@@ -0,0 +1,130 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_STREAMBUFFER_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_STREAMBUFFER_H_
#include "common/abi.h"
#include "common/common.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <cstdint>
#include <memory>
#include <optional>
#include <span>
#include <utility>
#include <vector>
namespace Libs::Graphics {
class CommandBuffer;
class CommandScheduler;
struct StreamBufferTestAccess;
struct GraphicContext;
struct VulkanBuffer;
enum class MemoryUsage : uint8_t {
DeviceLocal,
Upload,
Download,
Stream,
};
inline constexpr vk::BufferUsageFlags ReadFlags =
vk::BufferUsageFlagBits::eTransferSrc | vk::BufferUsageFlagBits::eUniformBuffer |
vk::BufferUsageFlagBits::eIndexBuffer | vk::BufferUsageFlagBits::eVertexBuffer |
vk::BufferUsageFlagBits::eIndirectBuffer;
inline constexpr vk::BufferUsageFlags AllFlags =
ReadFlags | vk::BufferUsageFlagBits::eTransferDst | vk::BufferUsageFlagBits::eStorageBuffer;
class Buffer {
public:
Buffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsage usage,
uint64_t cpu_address, vk::BufferUsageFlags flags, uint64_t size);
~Buffer();
KYTY_CLASS_NO_COPY(Buffer);
[[nodiscard]] vk::Buffer Handle() const noexcept;
[[nodiscard]] uint64_t Size() const noexcept { return m_size; }
[[nodiscard]] std::span<uint8_t> Mapped() const noexcept { return m_mapped; }
[[nodiscard]] bool IsCoherent() const noexcept { return m_is_coherent; }
[[nodiscard]] MemoryUsage Usage() const noexcept { return m_usage; }
[[nodiscard]] uint64_t CpuAddress() const noexcept { return m_cpu_address; }
[[nodiscard]] uint64_t Offset(uint64_t address) const noexcept {
return address - m_cpu_address;
}
[[nodiscard]] bool IsInBounds(uint64_t address, uint64_t size) const noexcept;
void Write(uint64_t offset, const void* source, uint64_t size);
void Flush(uint64_t offset, uint64_t size);
void CopyFrom(
CommandBuffer& command, const Buffer& source, uint64_t source_offset,
uint64_t destination_offset, uint64_t size,
vk::AccessFlags source_before = vk::AccessFlagBits::eMemoryWrite,
vk::AccessFlags destination_before =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite,
vk::AccessFlags source_after =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite,
vk::AccessFlags destination_after =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite);
void Fill(uint64_t offset, uint64_t size, uint32_t value);
protected:
[[nodiscard]] GraphicContext& Graphics() const noexcept { return *m_graphics; }
[[nodiscard]] CommandScheduler& Scheduler() const noexcept { return *m_scheduler; }
[[nodiscard]] VulkanBuffer& NativeBuffer() noexcept { return *m_buffer; }
private:
[[nodiscard]] vk::BufferMemoryBarrier Barrier(uint64_t offset, uint64_t size,
vk::AccessFlags source,
vk::AccessFlags destination) const;
GraphicContext* m_graphics = nullptr;
CommandScheduler* m_scheduler = nullptr;
MemoryUsage m_usage = MemoryUsage::DeviceLocal;
uint64_t m_cpu_address = 0;
uint64_t m_size = 0;
std::unique_ptr<VulkanBuffer> m_buffer;
std::span<uint8_t> m_mapped;
bool m_is_coherent = false;
};
class StreamBuffer final: public Buffer {
public:
StreamBuffer(GraphicContext& graphics, CommandScheduler& scheduler, MemoryUsage usage,
uint64_t size);
[[nodiscard]] std::pair<uint8_t*, uint64_t> Map(uint64_t size, uint64_t alignment = 0,
bool allow_wait = true);
void Commit();
// Download mappings become visible to the CPU only after their GPU completion tick is free.
// Call this from the scheduler's deferred completion operation before reading Mapped().
void Invalidate(uint64_t offset, uint64_t size);
[[nodiscard]] uint64_t Copy(const void* source, uint64_t size, uint64_t alignment = 0);
private:
friend struct StreamBufferTestAccess;
struct Watch {
uint64_t tick = 0;
uint64_t upper_bound = 0;
};
void ReserveWatches(std::vector<Watch>& watches, size_t grow_size);
[[nodiscard]] static bool NormalizeReservation(bool coherent, uint64_t atom, uint64_t& size,
uint64_t& alignment);
[[nodiscard]] bool WaitPendingOperations(const std::vector<Watch>& watches,
std::optional<size_t> invalidation_mark,
uint64_t requested_upper_bound, bool allow_wait,
size_t& wait_cursor, uint64_t& wait_bound);
uint64_t m_offset = 0;
uint64_t m_mapped_size = 0;
std::vector<Watch> m_current_watches;
size_t m_current_watch_cursor = 0;
std::optional<size_t> m_invalidation_mark;
std::vector<Watch> m_previous_watches;
size_t m_wait_cursor = 0;
uint64_t m_wait_bound = 0;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_STREAMBUFFER_H_
File diff suppressed because it is too large Load Diff
+194
View File
@@ -0,0 +1,194 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TEXTURECACHE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TEXTURECACHE_H_
#include "common/abi.h"
#include "common/common.h"
#include "common/lruCache.h"
#include "graphics/host_gpu/pageManager.h"
#include "graphics/host_gpu/regionManager.h"
#include "graphics/host_gpu/renderer/cache/multiLevelPageTable.h"
#include "graphics/host_gpu/renderer/image/blitHelper.h"
#include "graphics/host_gpu/renderer/image/image.h"
#include <compare>
#include <map>
#include <memory>
#include <optional>
#include <set>
#include <span>
#include <utility>
#include <vector>
namespace Libs::Graphics {
struct GraphicContext;
class Buffer;
class BufferCache;
class CommandBuffer;
class CommandScheduler;
class ResourceMutex;
class RenderExecutor;
class StreamBuffer;
class TileManager;
struct TextureCacheTestAccess;
class TextureCache {
public:
enum class BindingType : uint8_t { Texture, Storage, RenderTarget, DepthTarget, VideoOut };
struct ImageDesc {
ImageInfo info;
ImageViewInfo view_info;
BindingType type = BindingType::Texture;
};
struct RegionInfo {
bool image_pages = false;
bool image_bytes = false;
bool gpu_image_bytes = false;
};
TextureCache(GraphicContext& graphics, CommandScheduler& scheduler, PageManager& page_manager,
BufferCache& buffer_cache, ResourceMutex& resource_mutex);
~TextureCache();
KYTY_CLASS_NO_COPY(TextureCache);
[[nodiscard]] ImageId FindImage(ImageDesc& desc, bool exact_format = false);
[[nodiscard]] ImageId FindImageFromRange(uint64_t address, uint64_t size,
bool ensure_valid = true);
[[nodiscard]] vk::ImageView FindTexture(ImageId id, const ImageDesc& desc);
[[nodiscard]] vk::ImageView FindRenderTarget(ImageId id, const ImageDesc& desc);
[[nodiscard]] vk::ImageView FindDepthTarget(ImageId id, const ImageDesc& desc);
[[nodiscard]] Image& GetImage(ImageId id);
[[nodiscard]] const Image& GetImage(ImageId id) const;
void MarkGpuWritten(ImageId id);
[[nodiscard]] bool ClearImageFromBuffer(CommandBuffer& command, uint64_t address, uint64_t size,
uint32_t packed_clear);
void InvalidateMemory(uint64_t address, uint64_t size);
[[nodiscard]] bool SynchronizeImageToBuffer(uint64_t address, uint64_t size);
[[nodiscard]] bool InvalidateMemoryFromGPU(uint64_t address, uint64_t size,
bool formatted_buffer_write = false);
[[nodiscard]] RegionInfo QueryRegion(uint64_t address, uint64_t size);
[[nodiscard]] bool IsMeta(uint64_t address);
[[nodiscard]] bool IsMetaCleared(uint64_t address, uint32_t slice);
[[nodiscard]] bool ClearMeta(uint64_t address);
[[nodiscard]] bool TouchMeta(uint64_t address, uint32_t slice, bool is_clear);
[[nodiscard]] bool InvalidateMemory(PageFaultAccess access, uint64_t address, uint64_t size,
PageFaultPhase phase) noexcept;
void UnmapMemory(uint64_t address, uint64_t size);
void ProcessDownloadImages();
void RunGarbageCollector();
private:
enum class TransferDirection { Upload, Download };
struct ColorTransferPlan;
struct DownloadPlan;
struct Slot {
std::shared_ptr<Image> image;
uint32_t generation = 1;
};
struct MetaDataInfo {
uint32_t clear_mask = 0;
};
struct OverlapResult {
ImageId image;
int32_t mip = -1;
int32_t layer = -1;
};
using ImageOwnerIndex = MultiRangePageOwnerIndex<ImageId>;
[[nodiscard]] Image& ResolveImage(ImageId id);
[[nodiscard]] const Image& ResolveImage(ImageId id) const;
[[nodiscard]] std::shared_ptr<Image> ResolveOwner(ImageId id) const;
[[nodiscard]] ImageId InsertImage(const ImageInfo& info);
[[nodiscard]] ImageId GetNullImage(const ImageDesc& desc);
void RegisterImage(ImageId id);
void UnregisterImage(ImageId id);
void DeleteImage(ImageId id);
void DeleteImages(std::span<const ImageId> ids, std::optional<ImageId> native_source = {});
void RetainImage(CommandBuffer& command, ImageId id);
void TouchImage(Image& image);
void TrackImage(ImageId id);
void TrackImageHead(ImageId id);
void TrackImageTail(ImageId id);
void UntrackImage(ImageId id);
void UntrackImageHead(ImageId id);
void UntrackImageTail(ImageId id);
void TrackImageDownload(ImageId id);
void TrackImageDownloadLocked(ImageId id, Image& image);
[[nodiscard]] static bool SameBacking(const ImageInfo& cached, const ImageInfo& requested,
bool exact_format);
[[nodiscard]] static BindingType UploadBinding(const Image& image);
[[nodiscard]] bool SafeToDownload(const Image& image);
[[nodiscard]] std::vector<ImageId> FindImagesInRegion(uint64_t address, uint64_t size,
bool page_overlap) const;
[[nodiscard]] OverlapResult ResolveOverlap(const ImageInfo& requested, BindingType binding,
ImageId cached, ImageId merged);
[[nodiscard]] ImageId ResolveDepthOverlap(const ImageInfo& requested, BindingType binding,
ImageId cached);
[[nodiscard]] ImageId ExpandImage(const ImageInfo& info, ImageId source);
void RefreshImage(ImageId id, const ImageDesc& desc);
void InitializeImage(ImageId id, const ImageDesc& desc);
[[nodiscard]] ColorTransferPlan BuildColorTransfer(const Image& image, BindingType binding,
TransferDirection direction) const;
[[nodiscard]] DownloadPlan BuildDownload(const Image& image) const;
void UploadImage(Image& image, const ImageDesc& desc, Buffer& source, uint64_t source_offset);
void DownloadImageData(Image& image, Buffer& destination, uint64_t destination_offset,
DownloadPlan plan);
void DownloadDepth(Image& image, Buffer& destination, uint64_t destination_offset);
void CommitGpuWrite(Image& image);
void PrepareImageCopy(Image& image);
void RefreshCopySource(ImageId id);
[[nodiscard]] bool CopyD16(Image& destination, Image& source);
void CopyImage(ImageId destination, ImageId source);
void AssociateStencil(ImageId depth, GuestRange stencil);
void AssociateStencilLocked(ImageId depth, GuestRange stencil);
void CopyImageMip(ImageId destination, ImageId source, uint32_t mip, uint32_t layer);
void ValidateImageDesc(const ImageDesc& desc) const;
void InvalidateCpuAliases(uint64_t address, uint64_t size);
void ClearGpuModified(ImageId id);
[[nodiscard]] bool SynchronizeImageToBuffer(ImageId id);
void DownloadImage(ImageId id);
[[nodiscard]] bool TryDownloadImage(ImageId id);
[[nodiscard]] std::pair<uint8_t*, uint64_t> MapDownload(uint64_t size, uint64_t alignment);
void QueueDownload(GuestRange range, StreamBuffer& download, uint8_t* mapped, uint64_t offset);
GraphicContext& m_graphics;
CommandScheduler& m_scheduler;
TrackingSpinLock m_lock;
PageManager& m_page_manager;
BlitHelper m_blit_helper;
std::unique_ptr<TileManager> m_tiler;
BufferCache& m_buffer_cache;
ResourceMutex& m_resource_mutex;
std::vector<Slot> m_slots;
std::vector<uint32_t> m_free_slots;
ImageOwnerIndex m_image_owner_index;
std::map<vk::Format, ImageId> m_null_images;
Common::LeastRecentlyUsedCache<ImageId, uint64_t> m_lru_cache;
std::set<ImageId> m_download_images;
std::map<uint64_t, MetaDataInfo> m_surface_metas;
uint64_t m_total_used_memory = 0;
uint64_t m_trigger_gc_memory = 0;
uint64_t m_pressure_gc_memory = 1536ull * 1024 * 1024;
uint64_t m_critical_gc_memory = 3ull * 1024 * 1024 * 1024;
uint64_t m_gc_tick = 0;
bool m_readback_linear_images = false;
friend struct TextureCacheTestAccess;
friend class RenderExecutor;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TEXTURECACHE_H_
@@ -7,15 +7,12 @@
#include "graphics/guest_gpu/hardwareContext.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/objects/textureCommon.h"
#include "graphics/host_gpu/renderer/image/textureCommon.h"
#include "graphics/host_gpu/renderer/debug.h"
#include "graphics/host_gpu/renderer/descriptorCache.h"
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "graphics/host_gpu/renderer/pipeline/descriptorCache.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include "graphics/presentation/displayBuffer.h"
#include <algorithm>
#include <atomic>
@@ -25,9 +22,11 @@ namespace Libs::Graphics {
static std::atomic<uint32_t> g_render_color_log_count = 0;
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, RenderColorInfo& r,
uint32_t render_target_slice_offset, uint32_t render_target_slot,
bool ignore_target_mask, bool reuse_existing_render_texture) {
void RenderExecutor::ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
RenderColorInfo& r,
uint32_t render_target_slice_offset,
uint32_t render_target_slot, bool ignore_target_mask,
bool exact_format) {
KYTY_PROFILER_FUNCTION();
const auto& hw = buffer.GetRegisters();
@@ -57,14 +56,17 @@ void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
// No color output
r.type = RenderColorType::NoColorOutput;
r.desc = {};
r.base_addr = 0;
r.vulkan_buffer = nullptr;
r.vulkan_view = nullptr;
r.image_id = {};
r.image_view = nullptr;
r.format = vk::Format::eUndefined;
r.extent = {};
r.base_mip_level = 0;
r.base_array_layer = 0;
r.buffer_size = 0;
r.samples = 1;
r.export_mapping = {};
r.color_clear_enable = false;
r.color_clear_value = {};
return;
@@ -176,26 +178,33 @@ void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
pitch = width;
}
TileSizeOffset mip_sizes[16] {};
TilePaddedSize mip_padded[16] {};
if (tile) {
TileSizeAlign layout {};
bool valid_layout = false;
if (standard64) {
TileGetTextureSize(Prospero::GpuEnumValue(Prospero::BufferFormat::k32Float), width,
height, pitch, levels, rt.attrib3.tile_mode, &layout, nullptr,
nullptr);
height, pitch, levels, rt.attrib3.tile_mode, &layout, mip_sizes,
mip_padded);
valid_layout = layout.size != 0 && layout.align == 65536;
} else {
valid_layout =
levels == 1 ? TileGetRenderTargetSize(width, height, pitch, bytes_per_element,
layout, rt.attrib.num_fragments)
: TileGetRenderTargetMipLayout(width, height, pitch, bytes_per_element,
levels, layout, nullptr, nullptr);
levels, layout, mip_sizes, mip_padded);
}
if (!valid_layout) {
EXIT("unsupported render-target layout: %ux%u pitch=%u bytes=%u levels=%u\n", width,
height, pitch, bytes_per_element, levels);
}
size = layout.size;
EXIT_IF(size > UINT32_MAX);
if (levels == 1) {
mip_sizes[0] = {static_cast<uint32_t>(size), 0, 0, 0, 0, 0};
mip_padded[0] = {pitch, height};
}
if (rt.slice.slice_div64_minus1 != 0 &&
(static_cast<uint64_t>(rt.slice.slice_div64_minus1) + 1u) * 64u != size) {
EXIT("render-target slice span mismatch: encoded=0x%016" PRIx64 " derived=0x%016" PRIx64
@@ -204,6 +213,11 @@ void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
}
} else {
size = static_cast<uint64_t>(pitch) * height * bytes_per_element * samples;
if (size > UINT32_MAX) {
EXIT("linear render-target slice exceeds the supported layout size\n");
}
mip_sizes[0] = {static_cast<uint32_t>(size), 0, 0, 0, 0, 0};
mip_padded[0] = {pitch, height};
}
if (size == 0 || size > UINT64_MAX / view.image_layers) {
EXIT("render-target memory footprint is invalid\n");
@@ -213,119 +227,69 @@ void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
EXIT("render-target backing range is invalid\n");
}
auto video_image = Presentation::DisplayBufferFind(rt.base.addr, true);
if (video_image.image != nullptr &&
!IsSupportedDisplayRenderTargetTileMode(rt.attrib3.tile_mode)) {
EXIT("unsupported display render-target tile mode: tile=%u expected=%u addr=0x%010" PRIx64
" backing_size=0x%016" PRIx64 " video_size=0x%016" PRIx64 "\n",
rt.attrib3.tile_mode, Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget),
rt.base.addr, backing_size, video_image.size);
}
bool render_to_texture = view.base_layer != 0 || video_image.image == nullptr;
if (!render_to_texture && (levels != 1 || rt.view.current_mip_level != 0)) {
EXIT("mipmapped display render targets are unsupported\n");
}
const vk::Extent2D view_extent = {std::max(width >> rt.view.current_mip_level, 1u),
std::max(height >> rt.view.current_mip_level, 1u)};
auto decision_log_id = g_render_color_log_count.fetch_add(1);
if (decision_log_id < 128 || !render_to_texture) {
if (decision_log_id < 128) {
LOGF("RenderColorTarget: slot=%" PRIu32 " addr=0x%010" PRIx64 " size=0x%016" PRIx64
" extent=%ux%u view_mip=%u view_extent=%ux%u levels=%u pitch=%u"
" fmt=0x%08" PRIx32 " nfmt=0x%08" PRIx32 " order=0x%08" PRIx32
" samples=%u tile=%s target=%s video_size=0x%016" PRIx64 " video_pitch=%" PRIu64 "\n",
" fmt=0x%08" PRIx32 " nfmt=0x%08" PRIx32 " order=0x%08" PRIx32 " samples=%u tile=%s\n",
rt_slot, rt.base.addr, backing_size, width, height, rt.view.current_mip_level,
view_extent.width, view_extent.height, levels, pitch, rt.info.format,
rt.info.channel_type, rt.info.channel_order, samples, tile ? "tiled" : "linear",
render_to_texture ? "RenderTexture" : "DisplayBuffer", video_image.size,
video_image.pitch);
rt.info.channel_type, rt.info.channel_order, samples, tile ? "tiled" : "linear");
}
if (render_to_texture) {
(void)reuse_existing_render_texture;
RenderTargetInfo target {};
target.address = rt.base.addr;
target.size = backing_size;
target.format = target_format.format;
target.width = width;
target.height = height;
target.pitch = pitch;
target.bytes_per_element = target_format.bytes_per_element;
target.tile_mode = rt.attrib3.tile_mode;
target.levels = levels;
target.layers = view.image_layers;
target.samples = samples;
auto& texture_cache = GetRenderContext().GetTextureCache();
auto& buffer_vulkan = texture_cache.FindRenderTarget(buffer, target);
r.type = RenderColorType::RenderTexture;
r.base_addr = rt.base.addr;
r.vulkan_buffer = &buffer_vulkan;
r.vulkan_view = texture_cache.GetRenderTargetAttachmentView(
buffer_vulkan, target.format, rt.view.current_mip_level, view.base_layer,
view.layer_count);
r.format = target.format;
r.extent = view_extent;
r.base_mip_level = rt.view.current_mip_level;
r.buffer_size = backing_size;
r.samples = samples;
r.export_mapping = target_format.export_mapping;
r.color_clear_enable = buffer_vulkan.initial_clear_pending;
r.color_clear_value = {};
} else {
if (samples != 1) {
EXIT("multisampled display render targets are unsupported\n");
}
const auto layout = static_cast<Prospero::ChannelLayout>(rt.info.format);
const auto type = static_cast<Prospero::ChannelType>(rt.info.channel_type);
const auto order = static_cast<Prospero::ChannelOrder>(rt.info.channel_order);
bool supported_display_format =
(layout == Prospero::ChannelLayout::k8_8_8_8 &&
(type == Prospero::ChannelType::kSrgb || type == Prospero::ChannelType::kUNorm) &&
(order == Prospero::ChannelOrder::kStandard ||
order == Prospero::ChannelOrder::kAlt)) ||
(layout == Prospero::ChannelLayout::k10_10_10_2 &&
type == Prospero::ChannelType::kUNorm &&
(order == Prospero::ChannelOrder::kStandard ||
order == Prospero::ChannelOrder::kAlt)) ||
(layout == Prospero::ChannelLayout::k16_16_16_16 &&
type == Prospero::ChannelType::kFloat &&
(order == Prospero::ChannelOrder::kStandard || order == Prospero::ChannelOrder::kAlt));
EXIT_NOT_IMPLEMENTED(!supported_display_format);
// Display buffer
if (video_image.size != size) {
LOGF("RenderColorTarget: display buffer size differs from render target span, "
"video_size=0x%016" PRIx64 " render_size=0x%016" PRIx64 "\n",
video_image.size, size);
}
EXIT_NOT_IMPLEMENTED(video_image.size < size);
EXIT_NOT_IMPLEMENTED(video_image.pitch != pitch);
r.type = RenderColorType::DisplayBuffer;
r.base_addr = rt.base.addr;
r.vulkan_buffer = video_image.image;
r.vulkan_view = video_image.image->image_view[VulkanImage::VIEW_DEFAULT];
r.format = video_image.image->format;
r.extent = video_image.image->extent;
r.base_mip_level = 0;
r.buffer_size = video_image.size;
r.samples = 1;
r.export_mapping = target_format.export_mapping;
}
}
void MarkRenderTargetGpuWritten(const RenderColorInfo& target) {
const bool with_color = target.vulkan_buffer != nullptr;
if (with_color) {
if (target.type == RenderColorType::RenderTexture ||
target.type == RenderColorType::DisplayBuffer) {
GetRenderContext().GetTextureCache().MarkGpuWritten(*target.vulkan_buffer);
} else {
EXIT("unknown writable render-color resource type\n");
}
TextureCache::ImageDesc desc {};
desc.type = TextureCache::BindingType::RenderTarget;
desc.info.data = {rt.base.addr, backing_size};
desc.info.pixel_format = target_format.format;
desc.info.guest_format = ImageOps::RenderTargetTransferFormat(bytes_per_element);
desc.info.type = Prospero::ImageType::kColor2D;
desc.info.extent = {width, height, 1};
desc.info.resources = {levels, view.image_layers};
desc.info.pitch = pitch;
desc.info.bytes_per_block = bytes_per_element;
desc.info.samples = samples;
desc.info.tile_mode = rt.attrib3.tile_mode;
for (uint32_t level = 0; level < levels; level++) {
const auto level_offset =
mip_sizes[level].src_size != 0 ? mip_sizes[level].src_offset : mip_sizes[level].offset;
const auto level_size =
static_cast<uint64_t>(mip_sizes[level].src_size != 0 ? mip_sizes[level].src_size
: mip_sizes[level].size) *
view.image_layers;
desc.info.mip_layout[level] = {
level_offset,
level_size,
mip_padded[level].width,
mip_padded[level].height,
};
}
desc.view_info.format = target_format.format;
desc.view_info.type =
view.layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray;
desc.view_info.aspect = vk::ImageAspectFlagBits::eColor;
desc.view_info.base_level = rt.view.current_mip_level;
desc.view_info.level_count = 1;
desc.view_info.base_layer = view.base_layer;
desc.view_info.layer_count = view.layer_count;
desc.view_info.usage = vk::ImageUsageFlagBits::eColorAttachment;
auto& texture_cache = m_context.GetTextureCache();
r.desc = std::move(desc);
r.image_id = texture_cache.FindImage(r.desc, exact_format);
r.type = RenderColorType::RenderTexture;
r.base_addr = rt.base.addr;
r.image_view = nullptr;
r.format = r.desc.view_info.format;
r.extent = view_extent;
r.base_mip_level = rt.view.current_mip_level;
r.buffer_size = backing_size;
r.samples = samples;
r.export_mapping = target_format.export_mapping;
r.color_clear_enable = false;
r.color_clear_value = {};
BindRenderTarget(r.image_id);
}
} // namespace Libs::Graphics
@@ -3,6 +3,7 @@
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include "graphics/host_gpu/renderer/cache/textureCache.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <cstdint>
@@ -10,18 +11,17 @@
namespace Libs::Graphics {
class RenderCommandBuffer;
struct VulkanImage;
enum class RenderColorType {
NoColorOutput,
DisplayBuffer,
RenderTexture,
};
struct RenderColorInfo {
RenderColorType type = RenderColorType::NoColorOutput;
VulkanImage* vulkan_buffer = nullptr;
vk::ImageView vulkan_view = nullptr;
RenderColorType type = RenderColorType::NoColorOutput;
TextureCache::ImageDesc desc;
ImageId image_id;
vk::ImageView image_view = nullptr;
vk::Format format = vk::Format::eUndefined;
vk::Extent2D extent = {};
uint32_t base_mip_level = 0;
@@ -35,13 +35,6 @@ struct RenderColorInfo {
vk::ClearColorValue color_clear_value {};
};
void ResolveRenderColorTarget(uint64_t submit_id, RenderCommandBuffer& buffer, RenderColorInfo& r,
uint32_t render_target_slice_offset = 0,
uint32_t render_target_slot = UINT32_MAX,
bool ignore_target_mask = false,
bool reuse_existing_render_texture = false);
void MarkRenderTargetGpuWritten(const RenderColorInfo& target);
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_COLORRENDERTARGET_H_
@@ -0,0 +1,402 @@
#include "graphics/host_gpu/renderer/commandScheduler.h"
#include "common/assert.h"
#include "graphics/host_gpu/graphicContext.h"
#include <algorithm>
namespace Libs::Graphics {
static thread_local CommandScheduler* g_deferred_callback_scheduler = nullptr;
void CommandSlot::Reset() {
EXIT_IF(buffer == nullptr);
const auto result = buffer.reset(vk::CommandBufferResetFlagBits::eReleaseResources);
if (result != vk::Result::eSuccess) {
EXIT("failed to reset Vulkan command buffer: %s (%d)\n", VulkanToString(result).c_str(),
static_cast<int>(result));
}
}
CommandScheduler::CommandPool::~CommandPool() {
Destroy();
}
void CommandScheduler::CommandPool::Create(GraphicContext& graphics) {
EXIT_IF(m_pool != nullptr || m_graphics != nullptr ||
graphics.queue_family == static_cast<uint32_t>(-1));
m_graphics = &graphics;
vk::CommandPoolCreateInfo create {};
create.sType = vk::StructureType::eCommandPoolCreateInfo;
create.queueFamilyIndex = graphics.queue_family;
create.flags = vk::CommandPoolCreateFlagBits::eResetCommandBuffer;
const auto result = graphics.device.createCommandPool(&create, nullptr, &m_pool);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess || m_pool == nullptr);
}
CommandSlot* CommandScheduler::CommandPool::CreateSlot() {
EXIT_IF(m_graphics == nullptr);
auto& graphics = *m_graphics;
vk::CommandBufferAllocateInfo allocate {};
allocate.sType = vk::StructureType::eCommandBufferAllocateInfo;
allocate.commandPool = m_pool;
allocate.level = vk::CommandBufferLevel::ePrimary;
allocate.commandBufferCount = 1;
vk::CommandBuffer buffer = nullptr;
EXIT_IF(graphics.device.allocateCommandBuffers(&allocate, &buffer) != vk::Result::eSuccess);
vk::FenceCreateInfo fence_create {};
fence_create.sType = vk::StructureType::eFenceCreateInfo;
fence_create.flags = vk::FenceCreateFlagBits::eSignaled;
vk::Fence fence = nullptr;
if (graphics.device.createFence(&fence_create, nullptr, &fence) != vk::Result::eSuccess) {
graphics.device.freeCommandBuffers(m_pool, 1, &buffer);
EXIT("failed to create command-buffer fence\n");
}
auto& slot = m_slots.emplace_back();
slot.pool_mutex = &m_mutex;
slot.id = static_cast<uint32_t>(m_slots.size() - 1);
slot.buffer = buffer;
slot.fence = fence;
return &slot;
}
CommandSlot* CommandScheduler::CommandPool::Allocate(GraphicContext& graphics) {
Common::LockGuard lock(m_mutex);
if (m_pool == nullptr) {
Create(graphics);
}
EXIT_IF(m_graphics != &graphics);
auto found = std::ranges::find_if(m_slots, [](const auto& slot) { return !slot.busy; });
auto* slot = found != m_slots.end() ? &*found : CreateSlot();
slot->busy = true;
slot->Reset();
return slot;
}
void CommandScheduler::CommandPool::Destroy() {
Common::LockGuard lock(m_mutex);
if (m_pool == nullptr) {
return;
}
EXIT_IF(std::ranges::any_of(m_slots, [](const auto& slot) { return slot.busy; }));
EXIT_IF(m_graphics == nullptr);
for (const auto& slot: m_slots) {
m_graphics->device.destroyFence(slot.fence, nullptr);
}
m_graphics->device.destroyCommandPool(m_pool, nullptr);
m_slots.clear();
m_pool = nullptr;
m_graphics = nullptr;
}
bool CommandScheduler::InDeferredOperation() noexcept {
return g_deferred_callback_scheduler != nullptr;
}
CommandScheduler::CommandScheduler(RenderContext& context, GraphicContext& graphics)
: m_master(graphics), m_context(context), m_graphics(graphics),
m_priority_thread([this](std::stop_token stop) { PriorityOperationsThread(stop); }) {}
CommandScheduler::~CommandScheduler() {
Shutdown();
}
void CommandScheduler::Shutdown() {
{
std::unique_lock lock(m_operation_mutex);
if (m_operation_state == OperationState::Closed) {
return;
}
if (g_deferred_callback_scheduler == this) {
EXIT_IF(m_operation_state == OperationState::Open);
// A priority callback cannot join its own runner, while a normal callback can be
// executing inside the shutdown owner's final PopPendingOperations. The owning
// thread will finish shutdown after this callback returns.
return;
}
if (m_operation_state == OperationState::Draining) {
m_operation_available.wait(
lock, [this] { return m_operation_state == OperationState::Closed; });
return;
}
m_operation_state = OperationState::Draining;
}
if (Active() && m_recording) {
Finish();
}
DrainPriorityOperations();
m_priority_thread.request_stop();
m_operation_available.notify_all();
if (m_priority_thread.joinable()) {
m_priority_thread.join();
}
{
std::lock_guard lock(m_operation_mutex);
EXIT_IF(!m_pending_operations.empty() || !m_priority_operations.empty() ||
m_priority_active);
m_operation_state = OperationState::Closed;
}
m_operation_available.notify_all();
}
void CommandScheduler::Begin(HW::Context& registers, HW::UserConfig& user_config,
HW::Shader& shaders) {
{
std::lock_guard lock(m_operation_mutex);
EXIT_IF(m_operation_state != OperationState::Open);
}
m_registers = &registers;
m_user_config = &user_config;
m_shaders = &shaders;
if (!Active()) {
for (auto& buffer: m_buffers) {
buffer = std::make_unique<RenderCommandBuffer>(*this);
}
m_current = 0;
}
BindCurrent();
if (!m_recording) {
Current().Begin();
m_recording = true;
}
}
void CommandScheduler::BeginRendering(const RenderState& state) {
Current().BeginRendering(state);
}
void CommandScheduler::EndRendering() {
if (Active() && m_recording) {
Current().EndRendering();
}
}
void CommandScheduler::Flush() {
SubmitInfo submit;
Flush(submit);
}
void CommandScheduler::Flush(SubmitInfo& submit) {
SubmitCurrent(submit);
BeginNext();
}
CommandBuffer& CommandScheduler::FlushAndGetSubmitted() {
SubmitInfo submit;
auto& submitted = SubmitCurrent(submit);
BeginNext();
return submitted;
}
void CommandScheduler::Finish() {
CheckActive();
const auto tick = CurrentTick();
if (m_recording) {
SubmitInfo submit;
SubmitCurrent(submit);
}
for (auto& buffer: m_buffers) {
buffer->WaitForFenceAndReset();
}
m_master.Wait(tick);
PopPendingOperations();
BindCurrent();
Current().Begin();
m_recording = true;
}
void CommandScheduler::FinishCurrent() {
SubmitInfo submit;
auto& submitted = SubmitCurrent(submit);
submitted.WaitForFenceAndReset();
m_master.Refresh();
PopPendingOperations();
submitted.Begin();
m_recording = true;
}
void CommandScheduler::Wait(uint64_t tick) {
CheckActive();
EXIT_IF(tick > CurrentTick());
if (tick >= CurrentTick()) {
// A stream-buffer wrap can wait while a draw is being prepared through a reference to
// Current(). Recycle the same command object so that reference remains valid.
FinishCurrent();
return;
}
m_master.Wait(tick);
PopPendingOperations();
}
void CommandScheduler::PopPendingOperations() {
m_master.Refresh();
for (;;) {
Common::UniqueFunction<void> callback;
{
std::lock_guard lock(m_operation_mutex);
if (m_pending_operations.empty() ||
!m_master.IsFree(m_pending_operations.front().tick)) {
return;
}
callback = std::move(m_pending_operations.front().callback);
m_pending_operations.pop();
}
RunOperation(std::move(callback));
}
}
void CommandScheduler::DeferOperation(Common::UniqueFunction<void>&& operation) {
CheckActive();
EXIT_IF(!operation);
std::unique_lock lock(m_operation_mutex);
if (m_operation_state == OperationState::Open) {
m_pending_operations.push({std::move(operation), CurrentTick()});
return;
}
if (g_deferred_callback_scheduler == this) {
lock.unlock();
operation();
return;
}
m_operation_available.wait(lock,
[this] { return m_operation_state == OperationState::Closed; });
lock.unlock();
operation();
}
void CommandScheduler::DeferPriorityOperation(Common::UniqueFunction<void>&& operation) {
CheckActive();
EXIT_IF(!operation);
std::unique_lock lock(m_operation_mutex);
if (m_operation_state == OperationState::Open) {
m_priority_operations.push({std::move(operation), CurrentTick()});
lock.unlock();
m_operation_available.notify_one();
return;
}
if (g_deferred_callback_scheduler == this) {
lock.unlock();
operation();
return;
}
m_operation_available.wait(lock,
[this] { return m_operation_state == OperationState::Closed; });
lock.unlock();
operation();
}
void CommandScheduler::PriorityOperationsThread(std::stop_token stop) {
while (!stop.stop_requested()) {
PendingOperation operation;
{
std::unique_lock lock(m_operation_mutex);
m_operation_available.wait(lock, [this, &stop] {
return stop.stop_requested() || !m_priority_operations.empty();
});
if (stop.stop_requested()) {
return;
}
operation = std::move(m_priority_operations.front());
m_priority_operations.pop();
m_priority_active = true;
m_priority_active_tick = operation.tick;
}
m_master.Wait(operation.tick);
if (!stop.stop_requested()) {
RunOperation(std::move(operation.callback));
}
{
std::lock_guard lock(m_operation_mutex);
m_priority_active = false;
m_priority_active_tick = 0;
}
m_operation_available.notify_all();
}
}
void CommandScheduler::DrainPriorityOperations() {
EXIT_IF(g_deferred_callback_scheduler == this);
std::unique_lock lock(m_operation_mutex);
m_operation_available.wait(
lock, [this] { return m_priority_operations.empty() && !m_priority_active; });
}
void CommandScheduler::WaitPriorityOperations(uint64_t tick) {
EXIT_IF(g_deferred_callback_scheduler == this);
std::unique_lock lock(m_operation_mutex);
m_operation_available.wait(lock, [this, tick] {
const bool active_before_or_at =
m_priority_active && m_priority_active_tick <= tick;
const bool queued_before_or_at =
!m_priority_operations.empty() && m_priority_operations.front().tick <= tick;
return !active_before_or_at && !queued_before_or_at;
});
}
void CommandScheduler::RunOperation(Common::UniqueFunction<void>&& operation) {
auto* previous = g_deferred_callback_scheduler;
g_deferred_callback_scheduler = this;
operation();
g_deferred_callback_scheduler = previous;
}
bool CommandScheduler::IsFree(uint64_t tick) {
if (m_master.IsFree(tick)) {
return true;
}
m_master.Refresh();
return m_master.IsFree(tick);
}
CommandSlot* CommandScheduler::AllocateCommandBuffer() {
return m_command_pool.Allocate(m_graphics);
}
uint64_t CommandScheduler::NextSubmitSequence() noexcept {
return m_submit_sequence.fetch_add(1, std::memory_order_relaxed) + 1;
}
void CommandScheduler::CheckActive() const {
EXIT_IF(!Active() || m_current >= BufferCount);
}
RenderCommandBuffer& CommandScheduler::Current() const {
CheckActive();
EXIT_IF(m_buffers[m_current] == nullptr);
return *m_buffers[m_current];
}
void CommandScheduler::BindCurrent() const {
EXIT_IF(m_registers == nullptr || m_user_config == nullptr || m_shaders == nullptr);
Current().Bind(*m_registers, *m_user_config, *m_shaders);
}
CommandBuffer& CommandScheduler::SubmitCurrent(SubmitInfo& submit) {
CheckActive();
EXIT_IF(!m_recording);
auto& submitted = Current();
submitted.End();
const auto signal_tick = m_master.NextTick();
submit.AddSignal(m_master.Handle(), signal_tick);
submitted.Execute(submit);
m_recording = false;
return submitted;
}
void CommandScheduler::BeginNext() {
EXIT_IF(m_recording);
m_current = (m_current + 1) % BufferCount;
Current().WaitForFenceAndReset();
PopPendingOperations();
BindCurrent();
Current().Begin();
m_recording = true;
}
} // namespace Libs::Graphics
@@ -0,0 +1,127 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_COMMANDSCHEDULER_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_COMMANDSCHEDULER_H_
#include "common/common.h"
#include "common/uniqueFunction.h"
#include "graphics/host_gpu/renderer/masterSemaphore.h"
#include "graphics/host_gpu/renderer/render.h"
#include <array>
#include <atomic>
#include <condition_variable>
#include <deque>
#include <memory>
#include <mutex>
#include <queue>
#include <thread>
namespace Libs::Graphics {
struct CommandSlot {
Common::Mutex* pool_mutex = nullptr;
uint32_t id = 0;
vk::CommandBuffer buffer = nullptr;
vk::Fence fence = nullptr;
bool busy = false;
void Reset();
};
class CommandScheduler {
public:
static constexpr int BufferCount = 8;
CommandScheduler(RenderContext& context, GraphicContext& graphics);
~CommandScheduler();
KYTY_CLASS_NO_COPY(CommandScheduler);
void Begin(HW::Context& registers, HW::UserConfig& user_config, HW::Shader& shaders);
void BeginRendering(const RenderState& state);
void EndRendering();
void Flush();
void Flush(SubmitInfo& submit);
CommandBuffer& FlushAndGetSubmitted();
void Finish();
void FinishCurrent();
// Deferred callbacks can observe an externally owned drain, but cannot initiate shutdown:
// the priority runner cannot join itself.
void Shutdown();
void Wait(uint64_t tick);
void PopPendingOperations();
void DrainPriorityOperations();
void WaitPriorityOperations(uint64_t tick);
void DeferOperation(Common::UniqueFunction<void>&& operation);
void DeferPriorityOperation(Common::UniqueFunction<void>&& operation);
[[nodiscard]] static bool InDeferredOperation() noexcept;
[[nodiscard]] bool Active() const noexcept { return m_current >= 0; }
void CheckActive() const;
RenderCommandBuffer& Current() const;
[[nodiscard]] uint64_t CurrentTick() const noexcept { return m_master.CurrentTick(); }
[[nodiscard]] bool IsFree(uint64_t tick);
[[nodiscard]] RenderContext& Context() const noexcept { return m_context; }
[[nodiscard]] GraphicContext& Graphics() const noexcept { return m_graphics; }
private:
class CommandPool {
public:
CommandPool() = default;
~CommandPool();
KYTY_CLASS_NO_COPY(CommandPool);
CommandSlot* Allocate(GraphicContext& graphics);
private:
void Create(GraphicContext& graphics);
CommandSlot* CreateSlot();
void Destroy();
GraphicContext* m_graphics = nullptr;
Common::Mutex m_mutex;
vk::CommandPool m_pool = nullptr;
std::deque<CommandSlot> m_slots;
};
enum class OperationState { Open, Draining, Closed };
struct PendingOperation {
Common::UniqueFunction<void> callback;
uint64_t tick = 0;
};
void BindCurrent() const;
CommandBuffer& SubmitCurrent(SubmitInfo& submit);
void BeginNext();
void PriorityOperationsThread(std::stop_token stop);
void RunOperation(Common::UniqueFunction<void>&& operation);
[[nodiscard]] CommandSlot* AllocateCommandBuffer();
[[nodiscard]] uint64_t NextSubmitSequence() noexcept;
MasterSemaphore m_master;
RenderContext& m_context;
GraphicContext& m_graphics;
CommandPool m_command_pool;
std::array<std::unique_ptr<RenderCommandBuffer>, BufferCount> m_buffers;
std::queue<PendingOperation> m_pending_operations;
std::queue<PendingOperation> m_priority_operations;
std::mutex m_operation_mutex;
std::condition_variable m_operation_available;
std::jthread m_priority_thread;
bool m_priority_active = false;
uint64_t m_priority_active_tick = 0;
OperationState m_operation_state = OperationState::Open;
int m_current = -1;
bool m_recording = false;
HW::Context* m_registers = nullptr;
HW::UserConfig* m_user_config = nullptr;
HW::Shader* m_shaders = nullptr;
std::atomic<uint64_t> m_submit_sequence = 0;
friend class CommandBuffer;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_COMMANDSCHEDULER_H_
+118 -381
View File
@@ -8,68 +8,18 @@
#include "graphics/host_gpu/renderer/colorRenderTarget.h"
#include "graphics/host_gpu/renderer/debug.h"
#include "graphics/host_gpu/renderer/depthRenderTarget.h"
#include "graphics/host_gpu/renderer/descriptorCache.h"
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "graphics/host_gpu/renderer/imageView.h"
#include "graphics/host_gpu/renderer/pipeline/descriptorCache.h"
#include "graphics/host_gpu/renderer/image/imageView.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vma.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <array>
#include <atomic>
#include <bit>
#include <cstring>
#include <memory>
namespace Libs::Graphics {
static std::atomic<uint64_t> g_command_buffer_submit_seq = 0;
static void RequireValidQueueId(int queue_id) {
EXIT_IF(queue_id < 0 || queue_id >= GraphicContext::QUEUES_NUM);
}
static void ResetNativeCommandBuffer(vk::CommandBuffer buffer) {
EXIT_IF(buffer == nullptr);
const auto result = buffer.reset(vk::CommandBufferResetFlagBits::eReleaseResources);
if (result != vk::Result::eSuccess) {
EXIT("failed to reset Vulkan command buffer: %s (%d)\n", VulkanToString(result).c_str(),
static_cast<int>(result));
}
}
class CommandPool {
public:
CommandPool() = default;
~CommandPool() // NOLINT
{
// TODO(): check if destructor is called from std::_Exit()
// DeleteAll();
}
KYTY_CLASS_NO_COPY(CommandPool);
VulkanCommandPool* GetPool(int queue_id) {
RequireValidQueueId(queue_id);
if (m_pools[queue_id] == nullptr) {
Create(queue_id);
}
return m_pools[queue_id];
}
void DeleteAll();
private:
void Create(int queue_id);
std::array<VulkanCommandPool*, GraphicContext::QUEUES_NUM> m_pools {};
};
static RenderContext* g_render_ctx = nullptr;
static thread_local CommandPool g_command_pool;
RenderContext& GetRenderContext() noexcept {
return *g_render_ctx;
}
FenceResourceRetainer::~FenceResourceRetainer() {
if (!m_resources.empty()) {
@@ -92,162 +42,46 @@ void FenceResourceRetainer::ReleaseAfterFence() noexcept {
m_resources.clear();
}
void GraphicsRenderInit(GraphicContext& graphics) {
g_render_ctx = new RenderContext(graphics);
}
CommandBuffer::CommandBuffer(CommandScheduler& scheduler)
: m_context(scheduler.Context()), m_scheduler(scheduler), m_graphics(scheduler.Graphics()),
m_slot(scheduler.AllocateCommandBuffer()) {}
void GraphicsRenderReleaseThreadCommandPools() {
g_command_pool.DeleteAll();
}
CommandBuffer::CommandBuffer(int queue)
: m_graphics(GetRenderContext().GetGraphics()), m_queue(queue), m_host_stream(m_graphics) {
Allocate();
}
void CommandPool::Create(int queue_id) {
RequireValidQueueId(queue_id);
auto& graphics = GetRenderContext().GetGraphics();
auto*& pool = m_pools[queue_id];
EXIT_IF(pool != nullptr);
EXIT_IF(graphics.queues[queue_id].family == static_cast<uint32_t>(-1));
pool = new VulkanCommandPool;
vk::CommandPoolCreateInfo pool_info {};
pool_info.sType = vk::StructureType::eCommandPoolCreateInfo;
pool_info.pNext = nullptr;
pool_info.queueFamilyIndex = graphics.queues[queue_id].family;
pool_info.flags = vk::CommandPoolCreateFlagBits::eResetCommandBuffer;
const auto result = graphics.device.createCommandPool(&pool_info, nullptr, &pool->pool);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess || pool->pool == nullptr);
pool->buffers_count = 8;
pool->buffers = std::make_unique<vk::CommandBuffer[]>(pool->buffers_count);
pool->fences = std::make_unique<vk::Fence[]>(pool->buffers_count);
pool->semaphores = std::make_unique<vk::Semaphore[]>(pool->buffers_count);
pool->busy = std::make_unique<bool[]>(pool->buffers_count);
vk::CommandBufferAllocateInfo alloc_info {};
alloc_info.sType = vk::StructureType::eCommandBufferAllocateInfo;
alloc_info.commandPool = pool->pool;
alloc_info.level = vk::CommandBufferLevel::ePrimary;
alloc_info.commandBufferCount = pool->buffers_count;
if (graphics.device.allocateCommandBuffers(&alloc_info, pool->buffers.get()) !=
vk::Result::eSuccess) {
EXIT("Can't allocate command buffers");
}
for (uint32_t i = 0; i < pool->buffers_count; i++) {
pool->busy[i] = false;
vk::FenceCreateInfo fence_info {};
fence_info.sType = vk::StructureType::eFenceCreateInfo;
fence_info.pNext = nullptr;
fence_info.flags = vk::FenceCreateFlagBits::eSignaled;
if (graphics.device.createFence(&fence_info, nullptr, &pool->fences[i]) !=
vk::Result::eSuccess) {
EXIT("Can't create fence");
}
vk::SemaphoreCreateInfo semaphore_info {};
semaphore_info.sType = vk::StructureType::eSemaphoreCreateInfo;
semaphore_info.pNext = nullptr;
semaphore_info.flags = {};
if (graphics.device.createSemaphore(&semaphore_info, nullptr, &pool->semaphores[i]) !=
vk::Result::eSuccess) {
EXIT("Can't create semaphore");
}
EXIT_IF(pool->buffers[i] == nullptr);
EXIT_IF(pool->fences[i] == nullptr);
EXIT_IF(pool->semaphores[i] == nullptr);
}
}
void CommandPool::DeleteAll() {
auto& graphics = GetRenderContext().GetGraphics();
for (auto& pool: m_pools) {
if (pool != nullptr) {
for (uint32_t i = 0; i < pool->buffers_count; i++) {
graphics.device.destroySemaphore(pool->semaphores[i], nullptr);
graphics.device.destroyFence(pool->fences[i], nullptr);
}
graphics.device.freeCommandBuffers(pool->pool, pool->buffers_count,
pool->buffers.get());
graphics.device.destroyCommandPool(pool->pool, nullptr);
delete pool;
pool = nullptr;
}
}
CommandBuffer::~CommandBuffer() {
Release();
}
bool CommandBuffer::IsInvalid() const {
if (m_pool != nullptr) {
Common::LockGuard lock(m_pool->mutex);
return (m_index == static_cast<uint32_t>(-1) || m_index >= m_pool->buffers_count);
}
return true;
return m_slot == nullptr;
}
vk::CommandBuffer CommandBuffer::Handle() const {
EXIT_IF(IsInvalid());
const auto handle = m_pool->buffers[m_index];
const auto handle = m_slot->buffer;
EXIT_IF(handle == nullptr);
return handle;
}
void CommandBuffer::Allocate() {
EXIT_IF(!IsInvalid());
m_pool = g_command_pool.GetPool(m_queue);
Common::LockGuard lock(m_pool->mutex);
for (uint32_t i = 0; i < m_pool->buffers_count; i++) {
if (!m_pool->busy[i]) {
m_pool->busy[i] = true;
ResetNativeCommandBuffer(m_pool->buffers[i]);
m_index = i;
break;
}
}
EXIT_NOT_IMPLEMENTED(IsInvalid());
}
void CommandBuffer::Free() {
void CommandBuffer::Release() {
EXIT_IF(IsInvalid());
Common::LockGuard lock(m_pool->mutex);
Common::LockGuard lock(*m_slot->pool_mutex);
WaitForFence();
m_host_stream.Release();
m_pool->busy[m_index] = false;
ResetNativeCommandBuffer(m_pool->buffers[m_index]);
m_slot->busy = false;
m_slot->Reset();
ReleaseResourcesAfterFence();
m_index = static_cast<uint32_t>(-1);
m_slot = nullptr;
EXIT_NOT_IMPLEMENTED(!IsInvalid());
}
void CommandBuffer::DeleteAfterFence(VulkanBuffer& buffer) {
m_delete_after_fence.push_back(&buffer);
void CommandBuffer::RetireBufferAfterFence(std::unique_ptr<VulkanBuffer> buffer) {
if (IsInvalid() || m_execute || buffer == nullptr || buffer->buffer == nullptr) {
EXIT("cannot retire a buffer on an invalid or submitted command buffer\n");
}
m_retired_buffers.push_back(std::move(buffer));
}
void CommandBuffer::RetainResourceUntilFence(std::shared_ptr<void> resource) {
@@ -263,12 +97,13 @@ void CommandBuffer::RecycleDescriptorAfterFence(VulkanDescriptorSet& set) {
void CommandBuffer::RecycleDescriptorsAfterFence() {
for (auto* set: m_descriptor_sets_after_fence) {
GetRenderContext().GetDescriptorCache().Recycle(*set);
m_context.GetDescriptorCache().Recycle(*set);
}
m_descriptor_sets_after_fence.clear();
}
void CommandBuffer::Begin() const {
EXIT_IF(m_rendering);
auto buffer = Handle();
vk::CommandBufferBeginInfo begin_info {};
@@ -283,6 +118,7 @@ void CommandBuffer::Begin() const {
}
void CommandBuffer::End() const {
EndRendering();
auto buffer = Handle();
auto result = buffer.end();
@@ -301,45 +137,35 @@ void CommandBuffer::SetDebugInfo(uint32_t op, uint64_t submit_id, uint32_t arg0,
m_debug_arg4 = arg4;
}
void CommandBuffer::Execute() {
Submit(nullptr, {}, nullptr);
}
void CommandBuffer::ExecuteWithSemaphore(vk::Semaphore signal_semaphore) {
Submit(nullptr, {}, ResolveSignalSemaphore(signal_semaphore));
}
void CommandBuffer::ExecuteWithSemaphore(vk::Semaphore wait_semaphore,
vk::PipelineStageFlags wait_stage,
vk::Semaphore signal_semaphore) {
EXIT_IF(wait_semaphore == nullptr);
Submit(wait_semaphore, wait_stage, ResolveSignalSemaphore(signal_semaphore));
}
void CommandBuffer::Submit(vk::Semaphore wait_semaphore, vk::PipelineStageFlags wait_stage,
vk::Semaphore signal_semaphore) {
RequireValidQueueId(m_queue);
void CommandBuffer::Execute(const SubmitInfo& submit) {
EXIT_IF(IsInvalid());
EXIT_IF(m_execute);
EXIT_IF(submit.num_wait_semaphores > SubmitInfo::MaxSemaphores ||
submit.num_signal_semaphores > SubmitInfo::MaxSemaphores);
const bool has_wait = wait_semaphore != nullptr;
const bool has_signal = signal_semaphore != nullptr;
auto buffer = Handle();
auto fence = m_pool->fences[m_index];
auto buffer = Handle();
auto fence = m_slot->fence;
vk::TimelineSemaphoreSubmitInfo timeline_info {};
timeline_info.sType = vk::StructureType::eTimelineSemaphoreSubmitInfo;
timeline_info.waitSemaphoreValueCount = submit.num_wait_semaphores;
timeline_info.pWaitSemaphoreValues = submit.wait_ticks.data();
timeline_info.signalSemaphoreValueCount = submit.num_signal_semaphores;
timeline_info.pSignalSemaphoreValues = submit.signal_ticks.data();
vk::SubmitInfo submit_info {};
submit_info.sType = vk::StructureType::eSubmitInfo;
submit_info.pNext = nullptr;
submit_info.waitSemaphoreCount = has_wait ? 1u : 0u;
submit_info.pWaitSemaphores = has_wait ? &wait_semaphore : nullptr;
submit_info.pWaitDstStageMask = has_wait ? &wait_stage : nullptr;
submit_info.pNext = &timeline_info;
submit_info.waitSemaphoreCount = submit.num_wait_semaphores;
submit_info.pWaitSemaphores = submit.wait_semaphores.data();
submit_info.pWaitDstStageMask = submit.wait_stages.data();
submit_info.commandBufferCount = 1;
submit_info.pCommandBuffers = &buffer;
submit_info.signalSemaphoreCount = has_signal ? 1u : 0u;
submit_info.pSignalSemaphores = has_signal ? &signal_semaphore : nullptr;
submit_info.signalSemaphoreCount = submit.num_signal_semaphores;
submit_info.pSignalSemaphores = submit.signal_semaphores.data();
auto& graphics = GetRenderContext().GetGraphics();
const auto& queue = graphics.queues[m_queue];
auto& graphics = m_graphics;
EXIT_IF(graphics.queue == nullptr);
auto result = graphics.device.resetFences(1, &fence);
if (result != vk::Result::eSuccess) {
@@ -348,46 +174,33 @@ void CommandBuffer::Submit(vk::Semaphore wait_semaphore, vk::PipelineStageFlags
}
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
if (queue.mutex != nullptr) {
queue.mutex->Lock();
}
if (Config::GraphicsDebugDumpEnabled()) {
LOGF("vkQueueSubmit begin: queue=%d index=%u wait_semaphore=%p signal_semaphore=%p"
" debug_op=%u debug_submit=%" PRIu64 " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
m_queue, m_index, static_cast<void*>(wait_semaphore),
static_cast<void*>(signal_semaphore), m_debug_op, m_debug_submit_id, m_debug_arg0,
m_debug_arg1, m_debug_arg2, m_debug_arg3, m_debug_arg4);
LOGF("vkQueueSubmit begin: slot=%u waits=%u signals=%u debug_op=%u debug_submit=%" PRIu64
" args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
m_slot->id, submit.num_wait_semaphores, submit.num_signal_semaphores, m_debug_op,
m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
m_debug_arg4);
}
result = queue.vk_queue.submit(1, &submit_info, fence);
if (queue.mutex != nullptr) {
queue.mutex->Unlock();
{
Common::LockGuard lock(graphics.queue_mutex);
m_submit_seq = m_scheduler.NextSubmitSequence();
result = graphics.queue.submit(1, &submit_info, fence);
}
m_execute = true;
m_fence_waited = false;
m_submit_seq = g_command_buffer_submit_seq.fetch_add(1, std::memory_order_relaxed) + 1;
if (result != vk::Result::eSuccess) {
LOGF("vkQueueSubmit failed: %s (%d), queue=%d index=%u submit_seq=%" PRIu64
LOGF("vkQueueSubmit failed: %s (%d), slot=%u submit_seq=%" PRIu64
" debug_op=%u debug_submit=%" PRIu64 " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
VulkanToString(result).c_str(), static_cast<int>(result), m_queue, m_index,
m_submit_seq, m_debug_op, m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2,
m_debug_arg3, m_debug_arg4);
VulkanToString(result).c_str(), static_cast<int>(result), m_slot->id, m_submit_seq,
m_debug_op, m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
m_debug_arg4);
}
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
}
vk::Semaphore CommandBuffer::ResolveSignalSemaphore(vk::Semaphore semaphore) const {
if (semaphore != nullptr) {
return semaphore;
}
EXIT_IF(IsInvalid());
return m_pool->semaphores[m_index];
}
void CommandBuffer::WaitForFence() {
FinalizeFence(false);
}
@@ -397,14 +210,14 @@ void CommandBuffer::WaitForFenceOnly() {
if (!m_execute || m_fence_waited) {
return;
}
auto device = GetRenderContext().GetGraphics().device;
auto result = device.waitForFences(1, &m_pool->fences[m_index], VK_TRUE, UINT64_MAX);
auto device = m_graphics.device;
auto result = device.waitForFences(1, &m_slot->fence, VK_TRUE, UINT64_MAX);
if (result != vk::Result::eSuccess) {
LOGF("vkWaitForFences failed: %s (%d), queue=%d index=%u submit_seq=%" PRIu64
LOGF("vkWaitForFences failed: %s (%d), slot=%u submit_seq=%" PRIu64
" debug_op=%u debug_submit=%" PRIu64 " args=%u,%u,%u,%u,0x%016" PRIx64 "\n",
VulkanToString(result).c_str(), static_cast<int>(result), m_queue, m_index,
m_submit_seq, m_debug_op, m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2,
m_debug_arg3, m_debug_arg4);
VulkanToString(result).c_str(), static_cast<int>(result), m_slot->id, m_submit_seq,
m_debug_op, m_debug_submit_id, m_debug_arg0, m_debug_arg1, m_debug_arg2, m_debug_arg3,
m_debug_arg4);
}
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
m_fence_waited = true;
@@ -421,13 +234,10 @@ void CommandBuffer::FinalizeFence(bool reset_recording) {
m_execute = false;
m_fence_waited = false;
if (reset_recording) {
ResetNativeCommandBuffer(m_pool->buffers[m_index]);
m_recording_generation++;
Common::LockGuard lock(*m_slot->pool_mutex);
m_slot->Reset();
}
}
if (reset_recording) {
m_host_stream.Reset();
}
if (was_executed) {
ReleaseResourcesAfterFence();
}
@@ -440,144 +250,71 @@ void CommandBuffer::ReleaseResourcesAfterFence() {
}
void CommandBuffer::DeleteBuffersAfterFence() {
for (auto* buffer: m_delete_after_fence) {
GetRenderContext().GetGraphics().DeleteBuffer(*buffer);
delete buffer;
for (const auto& buffer: m_retired_buffers) {
m_graphics.DeleteBuffer(*buffer);
}
m_delete_after_fence.clear();
m_retired_buffers.clear();
}
void CommandBuffer::BeginRenderPass(VulkanFramebuffer& framebuffer, RenderColorInfo* colors,
uint32_t requested_color_count, RenderDepthInfo& depth) const {
auto buffer = Handle();
EXIT_IF(colors == nullptr);
EXIT_IF(requested_color_count > RENDER_COLOR_ATTACHMENTS_MAX);
bool with_depth = (depth.format != vk::Format::eUndefined && depth.vulkan_buffer != nullptr);
uint32_t color_count = 0;
for (uint32_t i = 0; i < requested_color_count; i++) {
if (colors[i].vulkan_buffer == nullptr) {
break;
}
color_count++;
void CommandBuffer::BeginRendering(const RenderState& state) const {
EXIT_IF(state.width == 0 || state.height == 0 || state.num_layers == 0 ||
state.num_color_attachments > RENDER_COLOR_ATTACHMENTS_MAX);
if (m_rendering && m_render_state == state) {
return;
}
bool with_color = (color_count != 0);
EndRendering();
EXIT_NOT_IMPLEMENTED(!with_depth && !with_color);
vk::ClearValue clears[RENDER_COLOR_ATTACHMENTS_MAX + 1] = {};
for (uint32_t i = 0; i < color_count; i++) {
clears[i].color = colors[i].color_clear_value;
}
clears[color_count].depthStencil = {depth.depth_clear_value, depth.stencil_clear_value};
vk::Extent2D extent = (with_color ? colors[0].extent : depth.vulkan_buffer->extent);
vk::RenderPassBeginInfo render_pass_info {};
render_pass_info.sType = vk::StructureType::eRenderPassBeginInfo;
render_pass_info.pNext = nullptr;
render_pass_info.renderPass = framebuffer.render_pass;
render_pass_info.framebuffer = framebuffer.framebuffer;
render_pass_info.renderArea.offset = {0, 0};
render_pass_info.renderArea.extent = extent;
render_pass_info.clearValueCount = color_count + (with_depth ? 1u : 0u);
render_pass_info.pClearValues = clears;
for (uint32_t i = 0; i < color_count; i++) {
const auto color_initial_layout = framebuffer.color_layout[i];
if (colors[i].vulkan_buffer->layout != color_initial_layout) {
if (graphics_debug_dump_enabled()) {
LOGF("BeginRenderPass: color%u initial barrier image=%p mem=%" PRIu64 " %s -> %s\n",
i, VulkanHandleToPointer(colors[i].vulkan_buffer->image),
colors[i].vulkan_buffer->memory.unique_id,
VulkanToString(colors[i].vulkan_buffer->layout).c_str(),
VulkanToString(color_initial_layout).c_str());
}
vk::ImageMemoryBarrier image_memory_barrier {};
image_memory_barrier.sType = vk::StructureType::eImageMemoryBarrier;
image_memory_barrier.pNext = nullptr;
image_memory_barrier.srcAccessMask = {};
image_memory_barrier.dstAccessMask = vk::AccessFlagBits::eColorAttachmentRead |
vk::AccessFlagBits::eColorAttachmentWrite;
image_memory_barrier.oldLayout = colors[i].vulkan_buffer->layout;
image_memory_barrier.newLayout = color_initial_layout;
image_memory_barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
image_memory_barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
image_memory_barrier.image = colors[i].vulkan_buffer->image;
image_memory_barrier.subresourceRange.aspectMask = vk::ImageAspectFlagBits::eColor;
image_memory_barrier.subresourceRange.baseMipLevel = 0;
image_memory_barrier.subresourceRange.levelCount = VK_REMAINING_MIP_LEVELS;
image_memory_barrier.subresourceRange.baseArrayLayer = 0;
image_memory_barrier.subresourceRange.layerCount = colors[i].vulkan_buffer->layers;
buffer.pipelineBarrier(vk::PipelineStageFlagBits::eTopOfPipe,
vk::PipelineStageFlagBits::eColorAttachmentOutput,
vk::DependencyFlags {}, 0, nullptr, 0, nullptr, 1,
&image_memory_barrier);
colors[i].vulkan_buffer->layout = image_memory_barrier.newLayout;
} else if (graphics_debug_dump_enabled()) {
LOGF("BeginRenderPass: color%u initial image=%p mem=%" PRIu64 " layout=%s\n", i,
VulkanHandleToPointer(colors[i].vulkan_buffer->image),
colors[i].vulkan_buffer->memory.unique_id,
VulkanToString(colors[i].vulkan_buffer->layout).c_str());
}
std::array<vk::RenderingAttachmentInfo, RENDER_COLOR_ATTACHMENTS_MAX> colors {};
for (uint32_t i = 0; i < state.num_color_attachments; i++) {
const auto& attachment = state.color_attachments[i];
colors[i].sType = vk::StructureType::eRenderingAttachmentInfo;
colors[i].imageView = attachment.image_view;
colors[i].imageLayout = attachment.image_layout;
colors[i].loadOp = attachment.is_clear ? vk::AttachmentLoadOp::eClear
: vk::AttachmentLoadOp::eLoad;
colors[i].storeOp = vk::AttachmentStoreOp::eStore;
colors[i].clearValue.color.uint32 = attachment.clear_value;
}
const auto depth_layout =
with_depth ? framebuffer.depth_layout : vk::ImageLayout::eDepthStencilAttachmentOptimal;
const auto& depth_stencil = state.depth_stencil_attachment;
vk::RenderingAttachmentInfo depth {};
depth.sType = vk::StructureType::eRenderingAttachmentInfo;
depth.imageView = depth_stencil.image_view;
depth.imageLayout = depth_stencil.image_layout;
depth.loadOp = depth_stencil.depth_clear ? vk::AttachmentLoadOp::eClear
: vk::AttachmentLoadOp::eLoad;
depth.storeOp = vk::AttachmentStoreOp::eStore;
depth.clearValue.depthStencil.depth = std::bit_cast<float>(depth_stencil.clear_value[0]);
if (with_depth && depth.vulkan_buffer->layout != depth_layout) {
vk::ImageMemoryBarrier image_memory_barrier {};
image_memory_barrier.sType = vk::StructureType::eImageMemoryBarrier;
image_memory_barrier.pNext = nullptr;
image_memory_barrier.srcAccessMask =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite;
image_memory_barrier.dstAccessMask =
(depth_layout == vk::ImageLayout::eDepthStencilReadOnlyOptimal
? vk::AccessFlagBits::eMemoryRead
: vk::AccessFlagBits::eMemoryWrite);
image_memory_barrier.oldLayout = depth.vulkan_buffer->layout;
image_memory_barrier.newLayout = depth_layout;
image_memory_barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
image_memory_barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
image_memory_barrier.image = depth.vulkan_buffer->image;
image_memory_barrier.subresourceRange.aspectMask =
ImageViewOps::DepthAspectMask(depth.vulkan_buffer->format);
image_memory_barrier.subresourceRange.baseMipLevel = 0;
image_memory_barrier.subresourceRange.levelCount = 1;
image_memory_barrier.subresourceRange.baseArrayLayer = 0;
image_memory_barrier.subresourceRange.layerCount = depth.vulkan_buffer->layers;
vk::RenderingAttachmentInfo stencil {};
stencil.sType = vk::StructureType::eRenderingAttachmentInfo;
stencil.imageView = depth_stencil.image_view;
stencil.imageLayout = depth_stencil.image_layout;
stencil.loadOp = depth_stencil.stencil_clear ? vk::AttachmentLoadOp::eClear
: vk::AttachmentLoadOp::eLoad;
stencil.storeOp = vk::AttachmentStoreOp::eStore;
stencil.clearValue.depthStencil.stencil = depth_stencil.clear_value[1];
buffer.pipelineBarrier(
vk::PipelineStageFlagBits::eAllGraphics | vk::PipelineStageFlagBits::eComputeShader,
vk::PipelineStageFlagBits::eAllGraphics | vk::PipelineStageFlagBits::eComputeShader,
vk::DependencyFlags {}, 0, nullptr, 0, nullptr, 1, &image_memory_barrier);
depth.vulkan_buffer->layout = image_memory_barrier.newLayout;
}
buffer.beginRenderPass(&render_pass_info, vk::SubpassContents::eInline);
for (uint32_t i = 0; i < color_count; i++) {
colors[i].vulkan_buffer->layout = RENDER_COLOR_IMAGE_LAYOUT;
if (colors[i].vulkan_buffer->type == VulkanImageType::RenderTexture) {
static_cast<RenderTextureVulkanImage*>(colors[i].vulkan_buffer)->initial_clear_pending =
false;
}
}
if (with_depth) {
depth.vulkan_buffer->initial_depth_clear_pending = false;
depth.vulkan_buffer->initial_stencil_clear_pending = false;
}
vk::RenderingInfo rendering {};
rendering.sType = vk::StructureType::eRenderingInfo;
rendering.renderArea.extent = {state.width, state.height};
rendering.layerCount = state.num_layers;
rendering.colorAttachmentCount = state.num_color_attachments;
rendering.pColorAttachments = colors.data();
rendering.pDepthAttachment = depth_stencil.has_depth ? &depth : nullptr;
rendering.pStencilAttachment = depth_stencil.has_stencil ? &stencil : nullptr;
Handle().beginRendering(rendering);
m_render_state = state;
m_rendering = true;
}
void CommandBuffer::EndRenderPass() const {
auto buffer = Handle();
buffer.endRenderPass();
void CommandBuffer::EndRendering() const {
if (!m_rendering) {
return;
}
Handle().endRendering();
m_rendering = false;
m_render_state = {};
}
} // namespace Libs::Graphics
+26 -89
View File
@@ -9,6 +9,7 @@
#include "graphics/guest_gpu/hardwareContext.h"
#include "graphics/host_gpu/renderer/render.h"
#include <algorithm>
#include <atomic>
#include <cmath>
#include <fmt/format.h>
@@ -100,8 +101,6 @@ void sh_print(const char* func, const HW::Shader& /*uc*/) {
LOGF("%s\n", func);
}
void sh_check(const HW::Shader& /*uc*/) {}
std::vector<std::string> rt_print(const char* func, const HW::RenderTarget& rt) {
std::vector<std::string> dst;
dst.reserve(53);
@@ -431,21 +430,17 @@ static void ZPrint(const char* func, const HW::DepthRenderTarget& z) {
LOGF("%s\n", func);
LOGF("\t z_info.format = 0x%08" PRIx32 "\n"
"\t z_info.tile_mode_index = 0x%08" PRIx32 "\n"
"\t z_info.num_samples = 0x%08" PRIx32 "\n"
"\t z_info.tile_surface_enable = %s\n"
"\t z_info.texture_compatibility = 0x%08" PRIx32 "\n"
"\t z_info.htile_acceleration = %s\n"
"\t z_info.expclear_enabled = %s\n"
"\t z_info.zrange_precision = 0x%08" PRIx32 "\n"
"\t z_info.embedded_sample_locations = %s\n"
"\t z_info.z_compare_base = 0x%08" PRIx32 "\n"
"\t z_info.partially_resident = %s\n"
"\t z_info.num_mip_levels = 0x%02" PRIx8 "\n"
"\t z_info.plane_compression = 0x%02" PRIx8 "\n"
"\t z_info.max_mip_level = 0x%02" PRIx8 "\n"
"\t stencil_info.format = 0x%08" PRIx32 "\n"
"\t stencil_info.tile_stencil_disable = %s\n"
"\t stencil_info.texture_compatibility = 0x%08" PRIx32 "\n"
"\t stencil_info.htile_stencil_disabled = %s\n"
"\t stencil_info.expclear_enabled = %s\n"
"\t stencil_info.tile_mode_index = 0x%08" PRIx32 "\n"
"\t stencil_info.tile_split = 0x%08" PRIx32 "\n"
"\t stencil_info.texture_compatible_stencil = %s\n"
"\t stencil_info.partially_resident = %s\n"
"\t depth_info.addr5_swizzle_mask = 0x%08" PRIx32 "\n"
"\t depth_info.array_mode = 0x%08" PRIx32 "\n"
@@ -478,15 +473,15 @@ static void ZPrint(const char* func, const HW::DepthRenderTarget& z) {
"\t height = 0x%08" PRIx32 "\n"
"\t size.x_max = 0x%04" PRIx16 "\n"
"\t size.y_max = 0x%04" PRIx16 "\n",
z.z_info.format, z.z_info.tile_mode_index, z.z_info.num_samples,
z.z_info.tile_surface_enable ? "true" : "false",
z.z_info.expclear_enabled ? "true" : "false", z.z_info.zrange_precision,
z.z_info.embedded_sample_locations ? "true" : "false",
z.z_info.partially_resident ? "true" : "false", z.z_info.num_mip_levels,
z.z_info.plane_compression, z.stencil_info.format,
z.stencil_info.tile_stencil_disable ? "true" : "false",
z.stencil_info.expclear_enabled ? "true" : "false", z.stencil_info.tile_mode_index,
z.stencil_info.tile_split, z.stencil_info.texture_compatible_stencil ? "true" : "false",
z.z_info.format, z.z_info.num_samples,
Prospero::GpuEnumValue(z.z_info.texture_compatibility),
z.z_info.htile_acceleration ? "true" : "false",
z.z_info.expclear_enabled ? "true" : "false",
Prospero::GpuEnumValue(z.z_info.z_compare_base),
z.z_info.partially_resident ? "true" : "false", z.z_info.max_mip_level,
z.stencil_info.format, Prospero::GpuEnumValue(z.stencil_info.texture_compatibility),
z.stencil_info.htile_stencil_disabled ? "true" : "false",
z.stencil_info.expclear_enabled ? "true" : "false",
z.stencil_info.partially_resident ? "true" : "false", z.depth_info.addr5_swizzle_mask,
z.depth_info.array_mode, z.depth_info.pipe_config, z.depth_info.bank_width,
z.depth_info.bank_height, z.depth_info.macro_tile_aspect, z.depth_info.num_banks,
@@ -503,36 +498,17 @@ static void ZPrint(const char* func, const HW::DepthRenderTarget& z) {
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
static void ZCheck(const HW::DepthRenderTarget& z) {
EXIT_NOT_IMPLEMENTED(!z.z_info.HasValidTextureCompatibility());
EXIT_NOT_IMPLEMENTED(!z.stencil_info.HasValidTextureCompatibility());
if (z.z_info.format == 0) {
EXIT_NOT_IMPLEMENTED(z.z_info.format != 0);
EXIT_NOT_IMPLEMENTED(z.z_info.tile_mode_index != 0);
EXIT_NOT_IMPLEMENTED(z.z_info.num_samples != 0);
EXIT_NOT_IMPLEMENTED(z.z_info.tile_surface_enable != false);
EXIT_NOT_IMPLEMENTED(z.z_info.htile_acceleration != false);
EXIT_NOT_IMPLEMENTED(z.z_info.expclear_enabled != false);
if (z.z_info.zrange_precision != 0) {
LOGF("Warning: zrange_precision != 0\n");
// z.z_info.zrange_precision = 0;
}
if (z.z_info.embedded_sample_locations) {
static bool logged = false;
if (!logged) {
LOGF("DepthTarget: temporary: ignoring embedded sample locations\n");
logged = true;
}
}
EXIT_NOT_IMPLEMENTED(z.z_info.partially_resident != false);
EXIT_NOT_IMPLEMENTED(z.z_info.num_mip_levels != 0);
if (z.z_info.plane_compression != 0) {
static bool logged = false;
if (!logged) {
LOGF("DepthTarget: temporary: ignoring PS5 plane_compression=0x%02" PRIx8 "\n",
z.z_info.plane_compression);
logged = true;
}
}
EXIT_NOT_IMPLEMENTED(z.z_info.max_mip_level != 0);
} else {
EXIT_NOT_IMPLEMENTED(z.z_info.format != 0x00000001 && z.z_info.format != 0x00000003);
// EXIT_NOT_IMPLEMENTED(z.z_info.tile_mode_index != 0x00000002);
if (z.z_info.num_samples != 0x00000000) {
static bool logged = false;
if (!logged) {
@@ -541,56 +517,17 @@ static void ZCheck(const HW::DepthRenderTarget& z) {
logged = true;
}
}
// EXIT_NOT_IMPLEMENTED(z.z_info.tile_surface_enable != true);
EXIT_NOT_IMPLEMENTED(z.z_info.expclear_enabled != false);
if (z.z_info.zrange_precision != 0x00000001) {
static bool logged = false;
if (!logged) {
LOGF("DepthTarget: temporary: ignoring zrange_precision=0x%08" PRIx32 "\n",
z.z_info.zrange_precision);
logged = true;
}
}
if (z.z_info.embedded_sample_locations) {
static bool logged = false;
if (!logged) {
LOGF("DepthTarget: temporary: ignoring embedded sample locations\n");
logged = true;
}
}
EXIT_NOT_IMPLEMENTED(z.z_info.partially_resident != false);
EXIT_NOT_IMPLEMENTED(z.z_info.num_mip_levels != 0);
if (z.z_info.plane_compression != 0) {
static bool logged = false;
if (!logged) {
LOGF("DepthTarget: temporary: ignoring PS5 plane_compression=0x%02" PRIx8 "\n",
z.z_info.plane_compression);
logged = true;
}
}
EXIT_NOT_IMPLEMENTED(z.z_info.max_mip_level != 0);
}
if (z.stencil_info.format == 0) {
// EXIT_NOT_IMPLEMENTED(z.stencil_info.format != 0);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.tile_stencil_disable != false);
EXIT_NOT_IMPLEMENTED(z.stencil_info.expclear_enabled != false);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.tile_mode_index != 0);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.tile_split != 0);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.texture_compatible_stencil != true);
EXIT_NOT_IMPLEMENTED(z.stencil_info.partially_resident != false);
} else {
// EXIT_NOT_IMPLEMENTED(z.stencil_info.format != 0x00000001);
if (z.stencil_info.tile_stencil_disable != true) {
static std::atomic<uint32_t> log_count {0};
if (log_count.fetch_add(1) < 16) {
LOGF("DepthTarget: temporary: ignoring PS5 HTILE stencil acceleration\n");
}
}
EXIT_NOT_IMPLEMENTED(z.stencil_info.format != 0x00000001);
EXIT_NOT_IMPLEMENTED(z.stencil_info.expclear_enabled != false);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.tile_mode_index != 0x00000002);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.tile_split != 0x00000002);
// EXIT_NOT_IMPLEMENTED(z.stencil_info.texture_compatible_stencil != true);
EXIT_NOT_IMPLEMENTED(z.stencil_info.partially_resident != false);
}
@@ -1135,10 +1072,10 @@ static ScissorRect ScissorRectClamp(ScissorRect r, uint32_t width, uint32_t heig
int max_right = static_cast<int>(width);
int max_bottom = static_cast<int>(height);
r.left = (r.left < 0 ? 0 : (r.left > max_right ? max_right : r.left));
r.right = (r.right < 0 ? 0 : (r.right > max_right ? max_right : r.right));
r.top = (r.top < 0 ? 0 : (r.top > max_bottom ? max_bottom : r.top));
r.bottom = (r.bottom < 0 ? 0 : (r.bottom > max_bottom ? max_bottom : r.bottom));
r.left = std::clamp(r.left, 0, max_right);
r.right = std::clamp(r.right, 0, max_right);
r.top = std::clamp(r.top, 0, max_bottom);
r.bottom = std::clamp(r.bottom, 0, max_bottom);
if (!ScissorRectValid(r)) {
r.right = r.left;
-1
View File
@@ -33,7 +33,6 @@ bool graphics_debug_dump_enabled();
void uc_print(const char* func, const HW::UserConfig& uc);
void uc_check(const HW::UserConfig& uc);
void sh_print(const char* func, const HW::Shader& uc);
void sh_check(const HW::Shader& uc);
std::vector<std::string> rt_print(const char* func, const HW::RenderTarget& rt);
bool RenderIsColorTileModeLinear(uint32_t tile_mode);
void hw_print(const RenderCommandBuffer& buffer);
@@ -10,15 +10,13 @@
#include "graphics/guest_gpu/hardwareContext.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/objects/textureCommon.h"
#include "graphics/host_gpu/renderer/image/textureCommon.h"
#include "graphics/host_gpu/renderer/debug.h"
#include "graphics/host_gpu/renderer/descriptorCache.h"
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "graphics/host_gpu/renderer/pipeline/descriptorCache.h"
#include "graphics/host_gpu/renderer/image/imageView.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include "graphics/presentation/displayBuffer.h"
#include <algorithm>
#include <atomic>
@@ -88,7 +86,8 @@ static bool UsesStencilOpValue(uint8_t fail, uint8_t pass, uint8_t depth_fail) {
}
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, RenderDepthInfo& r) {
void RenderExecutor::ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer,
RenderDepthInfo& r) {
KYTY_PROFILER_FUNCTION();
(void)submit_id;
const auto& hw = buffer.GetRegisters();
@@ -103,31 +102,34 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
if (!depth_active && !stencil_active) {
return;
}
if (!z.z_info.HasValidTextureCompatibility() ||
!z.stencil_info.HasValidTextureCompatibility()) {
DepthFatal("invalid PS5 depth texture-compatibility encoding");
}
const bool attachment_unbound =
z.z_info.format == Prospero::GpuEnumValue(Prospero::DepthFormat::kInvalid) &&
z.stencil_info.format == Prospero::GpuEnumValue(Prospero::StencilFormat::kInvalid) &&
z.z_info.tile_mode_index == 0 && z.z_info.num_samples == 0 &&
z.z_info.zrange_precision <= 1 && !z.z_info.expclear_enabled &&
!z.z_info.embedded_sample_locations && !z.z_info.partially_resident &&
z.z_info.num_mip_levels == 0 && z.z_info.plane_compression == 0 &&
z.stencil_info.tile_mode_index == 0 && z.stencil_info.tile_split == 0 &&
!z.stencil_info.expclear_enabled && !z.stencil_info.texture_compatible_stencil &&
!z.stencil_info.partially_resident && z.depth_view.slice_start == 0 &&
z.depth_view.slice_max == 0 && z.depth_view.current_mip_level == 0 &&
!z.depth_view.depth_write_disable && !z.depth_view.stencil_write_disable &&
z.depth_info.addr5_swizzle_mask == 0 && z.depth_info.array_mode == 0 &&
z.depth_info.pipe_config == 0 && z.depth_info.bank_width == 0 &&
z.depth_info.bank_height == 0 && z.depth_info.macro_tile_aspect == 0 &&
z.depth_info.num_banks == 0 && z.htile_surface.linear == 0 &&
z.htile_surface.full_cache == 0 && z.htile_surface.htile_uses_preload_win == 0 &&
z.htile_surface.preload == 0 && z.htile_surface.prefetch_width == 0 &&
z.htile_surface.prefetch_height == 0 && z.htile_surface.dst_outside_zero_to_one == 0 &&
z.z_read_base_addr == 0 && z.z_write_base_addr == 0 && z.stencil_read_base_addr == 0 &&
z.z_info.num_samples == 0 &&
z.z_info.texture_compatibility == Prospero::TextureCompatiblePlaneCompression::kDisable &&
!z.z_info.expclear_enabled && !z.z_info.partially_resident && z.z_info.max_mip_level == 0 &&
z.stencil_info.texture_compatibility == Prospero::TextureCompatibleStencil::kDisable &&
!z.stencil_info.expclear_enabled && !z.stencil_info.partially_resident &&
z.depth_view.slice_start == 0 && z.depth_view.slice_max == 0 &&
z.depth_view.current_mip_level == 0 && !z.depth_view.depth_write_disable &&
!z.depth_view.stencil_write_disable && z.depth_info.addr5_swizzle_mask == 0 &&
z.depth_info.array_mode == 0 && z.depth_info.pipe_config == 0 &&
z.depth_info.bank_width == 0 && z.depth_info.bank_height == 0 &&
z.depth_info.macro_tile_aspect == 0 && z.depth_info.num_banks == 0 &&
z.htile_surface.linear == 0 && z.htile_surface.full_cache == 0 &&
z.htile_surface.htile_uses_preload_win == 0 && z.htile_surface.preload == 0 &&
z.htile_surface.prefetch_width == 0 && z.htile_surface.prefetch_height == 0 &&
z.htile_surface.dst_outside_zero_to_one == 0 && z.z_read_base_addr == 0 &&
z.z_write_base_addr == 0 && z.stencil_read_base_addr == 0 &&
z.stencil_write_base_addr == 0 && z.htile_data_base_addr == 0 &&
// DB_DEPTH_SIZE_XY is independent state and may remain programmed after the attachment
// formats and addresses are unbound. A zero encoding is the valid 1x1 value, so its
// presence alone must not manufacture a depth attachment.
!z.z_info.tile_surface_enable && !z.width_height_valid && !z.pitch_height_valid &&
!z.z_info.htile_acceleration && !z.width_height_valid && !z.pitch_height_valid &&
z.size.x_max == 0 && z.size.y_max == 0 && z.pitch_div8_minus1 == 0 &&
z.height_div8_minus1 == 0 && z.slice_div64_minus1 == 0 && z.width == 0 && z.height == 0;
if (attachment_unbound) {
@@ -139,13 +141,13 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
}
const bool has_stencil =
z.stencil_info.format != Prospero::GpuEnumValue(Prospero::StencilFormat::kInvalid);
const bool has_htile = z.z_info.tile_surface_enable;
const bool has_htile = z.z_info.htile_acceleration;
const auto samples = render_sample_count(z.z_info.num_samples);
if (samples == 0) {
DepthFatal("unsupported depth fragment count: %u", z.z_info.num_samples);
}
const bool htile_stencil_compat = depth_htile_stencil_acceleration_compatible(
has_stencil, has_htile, z.stencil_info.tile_stencil_disable);
has_stencil, has_htile, z.stencil_info.htile_stencil_disabled);
const auto view = ResolveTargetViewInfo(z.depth_view.slice_start, z.depth_view.slice_max);
switch (view.type) {
case TargetViewType::Image2D: break;
@@ -156,14 +158,10 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
DepthFatal("invalid depth view: base=%u last=%u", z.depth_view.slice_start,
z.depth_view.slice_max);
}
// Prospero defines the compression-disable bits as tile writeback policy. Vulkan attachments
// expose the same logical depth/stencil values regardless of the driver's backing compression.
if ((stencil_active && !has_stencil) || rc.resummarize_enable || rc.copy_centroid ||
rc.copy_sample != 0 || z.z_info.expclear_enabled || z.stencil_info.expclear_enabled ||
z.z_info.embedded_sample_locations || z.z_info.partially_resident ||
z.stencil_info.partially_resident || z.z_info.plane_compression != 0 ||
z.z_info.num_mip_levels != 0 || z.z_info.tile_mode_index != 0 ||
z.z_info.zrange_precision > 1 || z.depth_view.current_mip_level != 0 ||
z.z_info.partially_resident || z.stencil_info.partially_resident ||
z.z_info.max_mip_level != 0 || z.depth_view.current_mip_level != 0 ||
z.depth_info.addr5_swizzle_mask != 0 || z.depth_info.array_mode != 0 ||
z.depth_info.pipe_config != 0 || z.depth_info.bank_width != 0 ||
z.depth_info.bank_height != 0 || z.depth_info.macro_tile_aspect != 0 ||
@@ -177,27 +175,16 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
DepthFatal("unsupported depth register state");
}
if (has_stencil) {
// Prospero defines Hi-Stencil as HTile-backed acceleration of the logical stencil plane.
// Keep the plane native in Vulkan while tracking HTile separately.
if (z.stencil_info.format != Prospero::GpuEnumValue(Prospero::StencilFormat::k8UInt) ||
z.stencil_info.tile_mode_index != 0 || z.stencil_info.tile_split != 0 ||
!htile_stencil_compat || z.stencil_info.texture_compatible_stencil ||
z.stencil_read_base_addr == 0 ||
!htile_stencil_compat || z.stencil_read_base_addr == 0 ||
z.stencil_write_base_addr != z.stencil_read_base_addr ||
(z.stencil_read_base_addr & 0xffffu) != 0 || z.depth_view.stencil_write_disable) {
DepthFatal("unsupported stencil attachment state");
}
if (!z.stencil_info.tile_stencil_disable) {
static std::atomic_bool logged = false;
if (!logged.load(std::memory_order_relaxed) &&
!logged.exchange(true, std::memory_order_relaxed)) {
LOGF("DepthTarget: compatibility: using native stencil with PS5 HTILE "
"acceleration\n");
}
}
} else if (z.stencil_read_base_addr != 0 || z.stencil_write_base_addr != 0 ||
z.stencil_info.tile_mode_index != 0 || z.stencil_info.tile_split != 0 ||
!htile_stencil_compat || z.stencil_info.texture_compatible_stencil) {
!htile_stencil_compat ||
z.stencil_info.texture_compatibility !=
Prospero::TextureCompatibleStencil::kDisable) {
DepthFatal("stencil state without an active stencil attachment");
}
if (has_htile) {
@@ -272,28 +259,19 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
(has_htile && htile_backing_size > TRACKER_ADDRESS_SIZE - z.htile_data_base_addr)) {
DepthFatal("layered depth backing range is invalid");
}
r.htile = has_htile;
r.width = width;
r.height = height;
r.samples = samples;
r.depth_buffer_size = depth_backing_size;
r.depth_buffer_vaddr = z.z_read_base_addr;
r.stencil_buffer_size = has_stencil ? stencil_backing_size : 0;
r.stencil_buffer_vaddr = has_stencil ? z.stencil_read_base_addr : 0;
r.htile_buffer_size = has_htile ? htile_backing_size : 0;
r.htile_buffer_vaddr = has_htile ? z.htile_data_base_addr : 0;
auto& cache = GetRenderContext().GetTextureCache();
if (has_htile) {
cache.RegisterMeta(r.htile_buffer_vaddr, r.htile_buffer_size, view.image_layers);
}
if (has_htile && rc.depth_clear_enable && !cache.ClearMeta(z.htile_data_base_addr)) {
DepthFatal("failed to acquire HTile metadata for a depth clear");
}
const bool meta_clear =
has_htile && cache.IsMetaCleared(z.htile_data_base_addr, z.depth_view.slice_start);
r.htile = has_htile;
r.width = width;
r.height = height;
r.samples = samples;
r.depth_buffer_size = depth_backing_size;
r.depth_buffer_vaddr = z.z_read_base_addr;
r.stencil_buffer_size = has_stencil ? stencil_backing_size : 0;
r.stencil_buffer_vaddr = has_stencil ? z.stencil_read_base_addr : 0;
r.htile_buffer_size = has_htile ? htile_backing_size : 0;
r.htile_buffer_vaddr = has_htile ? z.htile_data_base_addr : 0;
r.depth_clear_enable = rc.depth_clear_enable;
r.depth_meta_clear_enable = meta_clear;
r.depth_load_clear_enable = r.depth_clear_enable || r.depth_meta_clear_enable;
r.depth_meta_clear_enable = false;
r.depth_load_clear_enable = r.depth_clear_enable;
r.depth_clear_value = hw.GetDepthClearValue();
r.depth_test_enable = dc.z_enable;
r.depth_write_enable = dc.z_write_enable && !z.depth_view.depth_write_disable;
@@ -343,55 +321,93 @@ void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, R
r.vaddr[1] = r.stencil_buffer_vaddr;
r.size[1] = r.stencil_buffer_size;
}
DepthTargetInfo info {};
info.address = r.depth_buffer_vaddr;
info.size = r.depth_buffer_size;
info.stencil_address = r.stencil_buffer_vaddr;
info.stencil_size = r.stencil_buffer_size;
info.htile_address = r.htile_buffer_vaddr;
info.htile_size = r.htile_buffer_size;
info.format = r.format;
info.guest_format = guest_format;
info.width = width;
info.height = height;
info.pitch = pitch;
info.bytes_per_element = bytes;
info.tile_mode = Prospero::GpuEnumValue(Prospero::TileMode::kDepth);
info.layers = view.image_layers;
info.samples = samples;
info.depth_load_clear = r.depth_load_clear_enable;
info.depth_access = depth_active;
info.stencil_load_clear = rc.stencil_clear_enable;
info.stencil_access =
r.stencil_clear_enable ||
(r.stencil_test_enable &&
(stencil_face_accesses_attachment(r.stencil_static_front, r.stencil_dynamic_front) ||
stencil_face_accesses_attachment(r.stencil_static_back, r.stencil_dynamic_back)));
info.stencil_htile_compressed =
has_stencil && has_htile && !z.stencil_info.tile_stencil_disable;
r.vulkan_buffer = &cache.FindDepthTarget(buffer, info);
r.vulkan_view =
cache.GetDepthTargetAttachmentView(*r.vulkan_buffer, view.base_layer, view.layer_count);
if (r.vulkan_buffer->initial_depth_clear_pending) {
r.depth_load_clear_enable = true;
r.depth_clear_value = 0.0f;
}
if (r.vulkan_buffer->initial_stencil_clear_pending) {
r.stencil_clear_enable = true;
r.stencil_clear_value = 0;
}
if (meta_clear && !cache.TouchMeta(z.htile_data_base_addr, z.depth_view.slice_start, false)) {
DepthFatal("failed to consume HTile clear state");
}
TextureCache::ImageDesc desc {};
desc.type = TextureCache::BindingType::DepthTarget;
desc.info.data = {r.depth_buffer_vaddr, r.depth_buffer_size};
desc.info.stencil = {r.stencil_buffer_vaddr, r.stencil_buffer_size};
desc.info.pixel_format = r.format;
desc.info.guest_format = guest_format;
desc.info.type = Prospero::ImageType::kColor2D;
desc.info.extent = {width, height, 1};
desc.info.resources = {1, view.image_layers};
desc.info.pitch = pitch;
desc.info.bytes_per_block = bytes;
desc.info.samples = samples;
desc.info.tile_mode = Prospero::GpuEnumValue(Prospero::TileMode::kDepth);
desc.info.mip_layout[0] = {0, r.depth_buffer_size, pitch, height};
desc.info.metadata.range = {r.htile_buffer_vaddr, r.htile_buffer_size};
desc.info.metadata.kind = has_htile ? ImageMetadataKind::Htile : ImageMetadataKind::None;
desc.info.metadata.stencil_compressed =
has_stencil && has_htile && !z.stencil_info.htile_stencil_disabled;
desc.view_info.format = r.format;
desc.view_info.type =
view.layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray;
desc.view_info.aspect = ImageViewOps::DepthAspectMask(r.format);
desc.view_info.base_level = 0;
desc.view_info.level_count = 1;
desc.view_info.base_layer = view.base_layer;
desc.view_info.layer_count = view.layer_count;
desc.view_info.usage = vk::ImageUsageFlagBits::eDepthStencilAttachment;
r.desc = std::move(desc);
auto& cache = m_context.GetTextureCache();
r.image_id = cache.FindImage(r.desc);
r.image_view = nullptr;
BindRenderTarget(r.image_id);
}
void MarkRenderTargetGpuWritten(const RenderDepthInfo& target) {
const bool with_depth =
target.format != vk::Format::eUndefined && target.vulkan_buffer != nullptr;
if (with_depth && !depth_attachment_read_only(target)) {
GetRenderContext().GetTextureCache().MarkGpuWritten(*target.vulkan_buffer);
vk::ImageAspectFlags RenderDepthInfo::AttachmentWriteAspects() const {
if (format == vk::Format::eUndefined) {
return {};
}
const auto available = ImageViewOps::DepthAspectMask(format);
vk::ImageAspectFlags writes {};
if ((available & vk::ImageAspectFlagBits::eDepth) &&
(depth_load_clear_enable || (depth_test_enable && depth_write_enable))) {
writes |= vk::ImageAspectFlagBits::eDepth;
}
if (!(available & vk::ImageAspectFlagBits::eStencil)) {
return writes;
}
const auto face_writes = [&](const PipelineStencilStaticState& state,
const PipelineStencilDynamicState& dynamic) {
if (dynamic.writeMask == 0) {
return false;
}
bool can_pass = state.compareOp != vk::CompareOp::eNever;
bool can_fail = state.compareOp != vk::CompareOp::eAlways;
if (dynamic.compareMask == 0) {
switch (state.compareOp) {
case vk::CompareOp::eEqual:
case vk::CompareOp::eLessOrEqual:
case vk::CompareOp::eGreaterOrEqual:
case vk::CompareOp::eAlways:
can_pass = true;
can_fail = false;
break;
case vk::CompareOp::eNever:
case vk::CompareOp::eLess:
case vk::CompareOp::eGreater:
case vk::CompareOp::eNotEqual:
can_pass = false;
can_fail = true;
break;
default: break;
}
}
const bool depth_pass = !depth_test_enable || depth_compare_op != vk::CompareOp::eNever;
const bool depth_fail = depth_test_enable && depth_compare_op != vk::CompareOp::eAlways;
return (can_fail && state.failOp != vk::StencilOp::eKeep) ||
(can_pass && depth_pass && state.passOp != vk::StencilOp::eKeep) ||
(can_pass && depth_fail && state.depthFailOp != vk::StencilOp::eKeep);
};
if (stencil_clear_enable ||
(stencil_test_enable && (face_writes(stencil_static_front, stencil_dynamic_front) ||
face_writes(stencil_static_back, stencil_dynamic_back)))) {
writes |= vk::ImageAspectFlagBits::eStencil;
}
return writes;
}
} // namespace Libs::Graphics
@@ -2,7 +2,9 @@
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_DEPTHRENDERTARGET_H_
#include "common/assert.h"
#include "graphics/host_gpu/renderer/image/imageView.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include "graphics/host_gpu/renderer/cache/textureCache.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <cstdint>
@@ -10,14 +12,14 @@
namespace Libs::Graphics {
class RenderCommandBuffer;
struct DepthStencilVulkanImage;
inline constexpr bool depth_htile_stencil_acceleration_compatible(bool has_stencil, bool has_htile,
bool acceleration_disabled) {
return acceleration_disabled || (has_stencil && has_htile);
bool htile_stencil_disabled) {
return htile_stencil_disabled || (has_stencil && has_htile);
}
struct RenderDepthInfo {
TextureCache::ImageDesc desc;
vk::Format format = vk::Format::eUndefined;
uint32_t width = 0;
uint32_t height = 0;
@@ -49,29 +51,46 @@ struct RenderDepthInfo {
PipelineStencilStaticState stencil_static_back;
PipelineStencilDynamicState stencil_dynamic_front;
PipelineStencilDynamicState stencil_dynamic_back;
DepthStencilVulkanImage* vulkan_buffer = nullptr;
vk::ImageView vulkan_view = nullptr;
uint64_t vaddr[3] = {};
uint64_t size[3] = {};
int vaddr_num = 0;
ImageId image_id;
vk::ImageView image_view = nullptr;
uint64_t vaddr[3] = {};
uint64_t size[3] = {};
int vaddr_num = 0;
[[nodiscard]] vk::ImageAspectFlags AttachmentWriteAspects() const;
};
inline bool depth_attachment_read_only(const RenderDepthInfo& depth) {
const bool stencil_write =
depth.stencil_test_enable &&
(depth.stencil_dynamic_front.writeMask != 0 || depth.stencil_dynamic_back.writeMask != 0);
return !depth.depth_load_clear_enable && !depth.stencil_clear_enable &&
!depth.depth_write_enable && !stencil_write;
return !depth.AttachmentWriteAspects();
}
inline vk::ImageLayout depth_attachment_layout(const RenderDepthInfo& depth) {
return depth_attachment_read_only(depth) ? vk::ImageLayout::eDepthStencilReadOnlyOptimal
: vk::ImageLayout::eDepthStencilAttachmentOptimal;
const auto available = ImageViewOps::DepthAspectMask(depth.format);
const auto writes = depth.AttachmentWriteAspects();
const bool has_depth = static_cast<bool>(available & vk::ImageAspectFlagBits::eDepth);
const bool has_stencil = static_cast<bool>(available & vk::ImageAspectFlagBits::eStencil);
const bool depth_write = static_cast<bool>(writes & vk::ImageAspectFlagBits::eDepth);
const bool stencil_write = static_cast<bool>(writes & vk::ImageAspectFlagBits::eStencil);
if (!has_stencil) {
return depth_write ? vk::ImageLayout::eDepthAttachmentOptimal
: vk::ImageLayout::eDepthReadOnlyOptimal;
}
if (!has_depth) {
return stencil_write ? vk::ImageLayout::eStencilAttachmentOptimal
: vk::ImageLayout::eStencilReadOnlyOptimal;
}
if (depth_write && stencil_write) {
return vk::ImageLayout::eDepthStencilAttachmentOptimal;
}
if (depth_write) {
return vk::ImageLayout::eDepthAttachmentStencilReadOnlyOptimal;
}
if (stencil_write) {
return vk::ImageLayout::eDepthReadOnlyStencilAttachmentOptimal;
}
return vk::ImageLayout::eDepthStencilReadOnlyOptimal;
}
void ResolveRenderDepthTarget(uint64_t submit_id, RenderCommandBuffer& buffer, RenderDepthInfo& r);
void MarkRenderTargetGpuWritten(const RenderDepthInfo& target);
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_DEPTHRENDERTARGET_H_
@@ -1,854 +0,0 @@
#include "graphics/host_gpu/renderer/descriptors.h"
#include "common/assert.h"
#include "common/common.h"
#include "common/file.h"
#include "common/logging/log.h"
#include "common/profiler.h"
#include "common/stringUtils.h"
#include "common/threads.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/gpu_format.h"
#include "graphics/guest_gpu/graphicsRun.h"
#include "graphics/guest_gpu/hardwareContext.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/hostMemory.h"
#include "graphics/host_gpu/objects/textureCommon.h"
#include "graphics/host_gpu/renderer/debug.h"
#include "graphics/host_gpu/renderer/descriptorCache.h"
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "graphics/host_gpu/renderer/imageView.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/renderer/renderTargetBarriers.h"
#include "graphics/host_gpu/renderer/shaderResourceBarrier.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vma.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include "graphics/presentation/displayBuffer.h"
#include "graphics/shader/recompiler/BindingLayout.h"
#include "graphics/shader/recompiler/ResourceMaterialization.h"
#include "graphics/shader/recompiler/ShaderIR.h"
#include "graphics/shader/shader.h"
#include <algorithm>
#include <atomic>
#include <fmt/format.h>
#include <limits>
#include <span>
#ifdef min
#undef min
#endif
#ifdef max
#undef max
#endif
namespace Libs::Graphics {
using TextureVariant = DescriptorCache::TextureVariant;
static void BindNullStorageBuffer(CommandBuffer& cmd_buffer, BufferView& dst) {
dst.buffer = &GetRenderContext().GetBufferCache().ObtainNullBuffer(cmd_buffer);
dst.offset = 0;
dst.range = 16;
}
// NOLINTNEXTLINE(readability-function-cognitive-complexity)
static const char* VulkanImageTypeName(VulkanImageType type) {
switch (type) {
case VulkanImageType::VideoOut: return "VideoOut";
case VulkanImageType::DepthStencil: return "DepthStencil";
case VulkanImageType::Texture: return "Texture";
case VulkanImageType::StorageTexture: return "StorageTexture";
case VulkanImageType::RenderTexture: return "RenderTexture";
case VulkanImageType::Unknown:
default: return "Unknown";
}
}
static int SampledArrayViewIndex(const VulkanImage& image, int view_index) {
switch (view_index) {
case VulkanImage::VIEW_DEFAULT: return VulkanImage::VIEW_DEFAULT_ARRAY;
default: return view_index;
}
}
static int SelectSampledTextureArrayView(const VulkanImage& image, int base_view) {
const int array_view = SampledArrayViewIndex(image, base_view);
if (image.image_view[array_view] == nullptr) {
EXIT("missing sampled array image view: image_type=%s base_view=%d array_view=%d "
"layers=%u\n",
VulkanImageTypeName(image.type), base_view, array_view, image.layers);
}
return array_view;
}
static bool TextureVariantIsUint(TextureVariant variant) {
return variant == TextureVariant::Uint2D || variant == TextureVariant::UintArray ||
variant == TextureVariant::Uint3D;
}
static bool TextureVariantIsArray(TextureVariant variant) {
return variant == TextureVariant::FloatArray || variant == TextureVariant::UintArray;
}
static bool TextureVariantIs3D(TextureVariant variant) {
return variant == TextureVariant::Float3D || variant == TextureVariant::Uint3D;
}
static int TextureVariantDefaultView(TextureVariant variant) {
return TextureVariantIsArray(variant) ? VulkanImage::VIEW_DEFAULT_ARRAY
: VulkanImage::VIEW_DEFAULT;
}
static VulkanImage& GetDummySampledTexture(TextureVariant variant) {
return GetRenderContext().GetTextureCache().GetDummySampledTexture(
TextureVariantIsUint(variant), TextureVariantIs3D(variant));
}
static VulkanImage& GetDummyStorageTexture(TextureVariant variant) {
return GetRenderContext().GetTextureCache().GetDummyStorageTexture(
TextureVariantIsUint(variant), TextureVariantIs3D(variant));
}
static void CopyNativeDescriptor(const ShaderRecompiler::IR::DescriptorValue& source,
std::span<uint32_t> destination) {
EXIT_IF(source.dword_count != destination.size());
std::copy_n(source.dwords.begin(), destination.size(), destination.begin());
}
static BufferView NativeStorageBuffer(uint64_t submit_id, CommandBuffer& command_buffer,
const ShaderBufferResource& descriptor,
const ShaderRecompiler::IR::BufferResource& resource) {
BufferView result;
const auto address = descriptor.Base48();
const auto stride = descriptor.Stride();
const auto records = descriptor.NumRecords();
if (stride != 0 && records > UINT64_MAX / stride) {
EXIT("storage buffer descriptor footprint overflow\n");
}
const auto size = stride != 0 ? static_cast<uint64_t>(stride) * records : records;
if (address == 0 || size == 0) {
BindNullStorageBuffer(command_buffer, result);
return result;
}
const auto& graphics = GetRenderContext().GetGraphics();
const auto alignment = graphics.StorageMinAlignment();
if (alignment == 0 ||
size > graphics.GetPhysicalDeviceProperties().limits.maxStorageBufferRange ||
BufferCache::CACHING_PAGE_SIZE % alignment != 0) {
EXIT("storage buffer range or device alignment is unsupported\n");
}
(void)submit_id;
auto binding = GetRenderContext().GetBufferCache().ObtainBuffer(
command_buffer, address, size, resource.written, resource.read, resource.formatted);
if (binding.offset % alignment != 0) {
EXIT("storage buffer binding is not device-aligned\n");
}
result.buffer = &binding.buffer;
result.offset = binding.offset;
result.range = static_cast<vk::DeviceSize>(size);
return result;
}
static BufferView
NativeAddressBuffer(uint64_t submit_id, CommandBuffer& command_buffer,
const ShaderRecompiler::IR::AddressResource& resource,
const ShaderRecompiler::IR::ResourceSnapshot::Address& address) {
BufferView result;
if (address.binding_base == 0) {
BindNullStorageBuffer(command_buffer, result);
return result;
}
if (resource.written) {
EXIT("writable address resources are unsupported\n");
}
const auto limit = resource.kind == ShaderRecompiler::IR::ResourceKind::Flat
? ShaderRecompiler::IR::FlatAddressWindowSize
: static_cast<uint64_t>(GetRenderContext()
.GetGraphics()
.GetPhysicalDeviceProperties()
.limits.maxStorageBufferRange);
uint64_t size = 0;
const auto access = HostMemoryAccess::Mapped;
if (!HostMemoryQueryRange(address.binding_base, limit, access, size)) {
EXIT("address resource is not host-accessible: base=0x%016" PRIx64 "\n",
address.binding_base);
}
const auto& graphics = GetRenderContext().GetGraphics();
const auto alignment = graphics.StorageMinAlignment();
if (alignment == 0 ||
size > graphics.GetPhysicalDeviceProperties().limits.maxStorageBufferRange ||
BufferCache::GetBufferOffset(address.binding_base) % alignment != 0) {
EXIT("address resource range or alignment is unsupported\n");
}
(void)submit_id;
auto binding = GetRenderContext().GetBufferCache().ObtainBuffer(command_buffer,
address.binding_base, size);
result.buffer = &binding.buffer;
result.offset = binding.offset;
result.range = static_cast<vk::DeviceSize>(size);
return result;
}
static TextureVariant NativeTextureVariant(const ShaderRecompiler::IR::ImageResource& resource) {
const bool uint_image = resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint ||
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint;
switch (resource.dimension) {
case ShaderRecompiler::Decoder::ImageDimension::Dim3D:
return uint_image ? TextureVariant::Uint3D : TextureVariant::Float3D;
case ShaderRecompiler::Decoder::ImageDimension::Dim2DArray:
return uint_image ? TextureVariant::UintArray : TextureVariant::FloatArray;
default: return uint_image ? TextureVariant::Uint2D : TextureVariant::Float2D;
}
}
static bool IsSupportedSampledColorResource(const ShaderRecompiler::IR::ImageResource& resource) {
bool supported_dimension = false;
switch (resource.dimension) {
case ShaderRecompiler::Decoder::ImageDimension::Dim2D:
case ShaderRecompiler::Decoder::ImageDimension::Dim2DArray:
supported_dimension = true;
break;
default: break;
}
const bool sampled_kind = resource.kind == ShaderRecompiler::IR::ResourceKind::Image ||
resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint;
return sampled_kind && supported_dimension &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.read &&
!resource.written && !resource.atomic && !resource.depth_compare;
}
TargetTextureViewInfo ResolveTargetTextureView(const ShaderRecompiler::IR::ImageResource& resource,
Prospero::ImageType type, uint32_t base_layer,
uint32_t image_layers) {
switch (type) {
case Prospero::ImageType::kColor2D:
return resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
base_layer == 0 && image_layers == 1
? TargetTextureViewInfo {vk::ImageViewType::e2D, 0, 1}
: TargetTextureViewInfo {};
case Prospero::ImageType::kCube:
if (resource.dimension != ShaderRecompiler::Decoder::ImageDimension::Dim2DArray ||
base_layer >= image_layers || (image_layers - base_layer) % 6u != 0) {
return {};
}
return {vk::ImageViewType::e2DArray, base_layer, image_layers - base_layer};
case Prospero::ImageType::kColor2DArray:
if (resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
base_layer == 0 && image_layers == 1) {
return {vk::ImageViewType::e2D, 0, 1};
}
return resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray &&
base_layer < image_layers
? TargetTextureViewInfo {vk::ImageViewType::e2DArray, base_layer,
image_layers - base_layer}
: TargetTextureViewInfo {};
default: return {};
}
}
bool IsSupportedSampledVideoOutView(const ShaderRecompiler::IR::ImageResource& resource,
const ShaderTextureResource& descriptor,
const VulkanImage& image) {
return image.type == VulkanImageType::VideoOut && image.layers == 1 &&
IsSupportedSampledColorResource(resource) &&
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
descriptor.Type() == Prospero::GpuEnumValue(Prospero::ImageType::kColor2D) &&
descriptor.Depth() == 0 && descriptor.BaseArray5() == 0;
}
bool IsSupportedDepthTargetDescriptor(const ShaderTextureResource& descriptor,
const VulkanImage& image) {
const auto width = static_cast<uint32_t>(descriptor.Width5()) + 1u;
const auto height = static_cast<uint32_t>(descriptor.Height5()) + 1u;
const auto pitch = TileGetTexturePitch(descriptor.Format(), width, 1, descriptor.TileMode());
const auto type = static_cast<Prospero::ImageType>(descriptor.Type());
const bool supported_single_layer =
image.layers == 1 && descriptor.Depth() == 0 && descriptor.BaseArray5() == 0 &&
(type == Prospero::ImageType::kColor2D || type == Prospero::ImageType::kColor2DArray);
const bool supported_cube = type == Prospero::ImageType::kCube && width == height &&
image.layers >= 6 && image.layers % 6u == 0 &&
static_cast<uint32_t>(descriptor.Depth()) + 1u == image.layers &&
descriptor.BaseArray5() == 0;
return image.type == VulkanImageType::DepthStencil && width == image.extent.width &&
height == image.extent.height && (supported_single_layer || supported_cube) &&
descriptor.BaseLevel() == 0 && descriptor.LastLevel() == 0 && descriptor.MaxMip() == 0 &&
descriptor.MinLod() == 0 && descriptor.BaseArray5() == 0 &&
descriptor.TileMode() == Prospero::GpuEnumValue(Prospero::TileMode::kDepth) &&
descriptor.BCSwizzle() == 0 && !descriptor.MsaaDepth() && pitch >= width &&
pitch == image.guest_pitch;
}
bool IsSupportedDepthTextureEncoding(const ShaderTextureResource& descriptor) {
constexpr uint32_t field1_reserved_mask = 0x200fff00u;
constexpr uint32_t field2_reserved_mask = 0xf0003000u;
constexpr uint32_t field3_common = 0x01800000u;
constexpr uint32_t field5_expected = 0x00700000u;
const uint32_t field3_expected =
(descriptor.Type() << 28u) | field3_common | descriptor.DstSelXYZW();
const uint32_t field4_expected = descriptor.Depth() | (descriptor.BaseArray5() << 16u);
return (descriptor.fields[1] & field1_reserved_mask) == 0 &&
(descriptor.fields[2] & field2_reserved_mask) == 0 &&
descriptor.fields[3] == field3_expected && descriptor.fields[4] == field4_expected &&
descriptor.fields[5] == field5_expected && descriptor.fields[6] == 0 &&
descriptor.fields[7] == 0;
}
static void ValidateDepthTargetBinding(const ShaderRecompiler::IR::ImageResource& resource,
const ShaderTextureResource& descriptor,
const VulkanImage* image, vk::Format view_format,
uint64_t size) {
const bool resource_ok = IsSupportedSampledDepthResource(resource);
const bool descriptor_ok =
image != nullptr && IsSupportedDepthTargetDescriptor(descriptor, *image);
const bool encoding_ok = IsSupportedDepthTextureEncoding(descriptor);
const bool format_ok = image != nullptr && IsSupportedSampledDepthFormat(
image->format, descriptor.Format(), view_format);
if (resource_ok && descriptor_ok && encoding_ok && format_ok && size != 0) {
return;
}
const auto descriptor_pitch =
TileGetTexturePitch(descriptor.Format(), static_cast<uint32_t>(descriptor.Width5()) + 1u, 1,
descriptor.TileMode());
EXIT("unsupported sampled depth target: resource=%d descriptor=%d encoding=%d format=%d "
"kind=%u dimension=%u mip_mode=%u read=%d written=%d atomic=%d compare=%d "
"guest_format=%u swizzle=0x%03x image_format=%d view_format=%d image_layers=%u "
"descriptor_type=%u base_array=%u depth=%u descriptor_pitch=%u target_pitch=%u "
"addr=0x%016" PRIx64 " size=0x%016" PRIx64
" dwords=%08x,%08x,%08x,%08x,%08x,%08x,%08x,%08x\n",
resource_ok, descriptor_ok, encoding_ok, format_ok, static_cast<uint32_t>(resource.kind),
static_cast<uint32_t>(resource.dimension), static_cast<uint32_t>(resource.mip_mode),
resource.read, resource.written, resource.atomic, resource.depth_compare,
descriptor.Format(), descriptor.DstSelXYZW(),
image == nullptr ? static_cast<int>(vk::Format::eUndefined)
: static_cast<int>(image->format),
static_cast<int>(view_format), image == nullptr ? 0u : image->layers, descriptor.Type(),
descriptor.BaseArray5(), descriptor.Depth(), descriptor_pitch,
image == nullptr ? 0u : image->guest_pitch, descriptor.Base40(), size,
descriptor.fields[0], descriptor.fields[1], descriptor.fields[2], descriptor.fields[3],
descriptor.fields[4], descriptor.fields[5], descriptor.fields[6], descriptor.fields[7]);
}
static bool IsSupportedStorageTextureDescriptor(const ShaderRecompiler::IR::ImageResource& resource,
const ShaderTextureResource& descriptor) {
const auto tile = descriptor.TileMode();
const auto width = static_cast<uint32_t>(descriptor.Width5()) + 1u;
const auto height = static_cast<uint32_t>(descriptor.Height5()) + 1u;
const auto depth = static_cast<uint32_t>(descriptor.Depth()) + 1u;
const bool is_2d = resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
descriptor.Type() == Prospero::GpuEnumValue(Prospero::ImageType::kColor2D) &&
descriptor.Depth() == 0;
const bool is_2d_array =
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray &&
descriptor.Type() == Prospero::GpuEnumValue(Prospero::ImageType::kColor2DArray) &&
descriptor.BaseArray5() <= descriptor.Depth();
const bool is_3d = resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim3D &&
descriptor.Type() == Prospero::GpuEnumValue(Prospero::ImageType::kColor3D);
const bool supported_depth_tile =
tile == Prospero::GpuEnumValue(Prospero::TileMode::kDepth) && !resource.read &&
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint &&
IsSupportedStorageDepthTile(descriptor.Format(), descriptor.Type(), width, height, depth);
const bool supported_tile = tile == Prospero::GpuEnumValue(Prospero::TileMode::kLinear) ||
tile == Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget) ||
supported_depth_tile;
const bool supported_swizzle =
IsSupportedStorageSwizzle(descriptor.Format(), descriptor.DstSelXYZW()) &&
(descriptor.DstSelXYZW() == DstSel(4, 5, 6, 7) || !resource.read);
const bool supported_mip_view = descriptor.BaseLevel() == 0 || is_2d;
return (is_2d || is_2d_array || is_3d) && supported_tile && supported_mip_view &&
descriptor.BaseLevel() == descriptor.LastLevel() &&
descriptor.LastLevel() <= descriptor.MaxMip() && descriptor.MinLod() == 0 &&
descriptor.BaseArray5() == 0 && supported_swizzle && descriptor.BCSwizzle() == 0 &&
!descriptor.MsaaDepth();
}
static bool IsSupportedStorageTextureEncoding(const ShaderTextureResource& descriptor) {
constexpr uint32_t field1_reserved_mask = 0x200fff00u;
constexpr uint32_t field2_reserved_mask = 0xf0003000u;
constexpr uint32_t field5_expected = 0x00700000u;
constexpr uint32_t field5_max_mip_mask = 0x000000f0u;
const uint32_t expected_field3 = descriptor.DstSelXYZW() |
(static_cast<uint32_t>(descriptor.BaseLevel()) << 12u) |
(static_cast<uint32_t>(descriptor.LastLevel()) << 16u) |
(static_cast<uint32_t>(descriptor.TileMode()) << 20u) |
(static_cast<uint32_t>(descriptor.Type()) << 28u);
return (descriptor.fields[1] & field1_reserved_mask) == 0 &&
(descriptor.fields[2] & field2_reserved_mask) == 0 &&
descriptor.fields[3] == expected_field3 && descriptor.fields[4] == descriptor.Depth() &&
(descriptor.fields[5] & ~field5_max_mip_mask) == field5_expected &&
descriptor.fields[6] == 0 && descriptor.fields[7] == 0;
}
void ValidateStorageTexture(const ShaderRecompiler::IR::ImageResource& resource,
const ShaderTextureResource& descriptor, uint64_t size) {
const auto format = descriptor.Format();
const bool resource_ok = IsSupportedStorageImageResource(resource);
const bool descriptor_ok = IsSupportedStorageTextureDescriptor(resource, descriptor);
const bool encoding_ok = IsSupportedStorageTextureEncoding(descriptor);
const bool uint_resource =
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint;
const bool format_ok = Prospero::IsSupportedTextureFormat(format) &&
uint_resource == Prospero::IsUintTextureFormat(format);
if (resource_ok && descriptor_ok && encoding_ok && format_ok && size != 0) {
return;
}
EXIT("unsupported storage texture: resource=%d descriptor=%d encoding=%d format=%d "
"kind=%u dimension=%u mip_mode=%u atomic=%d compare=%d "
"base_level=%u last_level=%u max_mip=%u min_lod=%u base_array=%u bc=%u msaa=%d "
"depth_tile_shape=%d swizzle_ok=%d "
"addr=0x%016" PRIx64 " size=0x%016" PRIx64
" extent=%ux%ux%u type=%u format=%u tile=%u swizzle=0x%03x read=%d written=%d "
"dwords=%08x,%08x,%08x,%08x,%08x,%08x,%08x,%08x\n",
resource_ok, descriptor_ok, encoding_ok, format_ok, static_cast<uint32_t>(resource.kind),
static_cast<uint32_t>(resource.dimension), static_cast<uint32_t>(resource.mip_mode),
resource.atomic, resource.depth_compare, descriptor.BaseLevel(), descriptor.LastLevel(),
descriptor.MaxMip(), descriptor.MinLod(), descriptor.BaseArray5(), descriptor.BCSwizzle(),
descriptor.MsaaDepth(),
IsSupportedStorageDepthTile(descriptor.Format(), descriptor.Type(),
static_cast<uint32_t>(descriptor.Width5()) + 1u,
static_cast<uint32_t>(descriptor.Height5()) + 1u,
static_cast<uint32_t>(descriptor.Depth()) + 1u),
IsSupportedStorageSwizzle(descriptor.Format(), descriptor.DstSelXYZW()),
descriptor.Base40(), size, static_cast<uint32_t>(descriptor.Width5()) + 1u,
static_cast<uint32_t>(descriptor.Height5()) + 1u,
static_cast<uint32_t>(descriptor.Depth()) + 1u, descriptor.Type(), format,
descriptor.TileMode(), descriptor.DstSelXYZW(), resource.read, resource.written,
descriptor.fields[0], descriptor.fields[1], descriptor.fields[2], descriptor.fields[3],
descriptor.fields[4], descriptor.fields[5], descriptor.fields[6], descriptor.fields[7]);
}
void ValidateMetadataReuseTexture(const ShaderRecompiler::IR::ImageResource& resource,
const ShaderTextureResource& descriptor, uint64_t size) {
constexpr uint32_t field1_reserved = 0x200fff00u;
constexpr uint32_t field2_reserved = 0xf0003000u;
constexpr uint32_t field5_common = 0x00700000u;
const auto format = descriptor.Format();
const bool resource_ok = IsSupportedSampledColorResource(resource);
const bool swizzle_ok = IsValidSampledColorSwizzle(descriptor.DstSelXYZW());
const bool descriptor_ok =
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
descriptor.Type() == Prospero::GpuEnumValue(Prospero::ImageType::kColor2D) &&
descriptor.Depth() == 0 && descriptor.BaseArray5() == 0 &&
descriptor.BaseLevel() <= descriptor.LastLevel() &&
descriptor.LastLevel() <= descriptor.MaxMip() && descriptor.MaxMip() < 15 &&
descriptor.TileMode() == Prospero::GpuEnumValue(Prospero::TileMode::kStandard4KB) &&
swizzle_ok;
const uint32_t field3_expected = descriptor.DstSelXYZW() |
(static_cast<uint32_t>(descriptor.BaseLevel()) << 12u) |
(static_cast<uint32_t>(descriptor.LastLevel()) << 16u) |
(static_cast<uint32_t>(descriptor.TileMode()) << 20u) |
(static_cast<uint32_t>(descriptor.Type()) << 28u);
const uint32_t field4_expected = descriptor.Depth() | (descriptor.BaseArray5() << 16u);
const uint32_t field5_expected =
field5_common | (static_cast<uint32_t>(descriptor.MaxMip()) << 4u);
const bool encoding_ok =
(descriptor.fields[1] & field1_reserved) == 0 &&
(descriptor.fields[2] & field2_reserved) == 0 && descriptor.fields[3] == field3_expected &&
descriptor.fields[4] == field4_expected && descriptor.fields[5] == field5_expected &&
descriptor.fields[6] == 0 && descriptor.fields[7] == 0;
const bool format_ok =
Prospero::IsSupportedTextureFormat(format) && !Prospero::IsUintTextureFormat(format);
if (!resource_ok || !descriptor_ok || !encoding_ok || !format_ok || size == 0) {
EXIT("unsupported metadata-reuse sampled texture: resource=%d descriptor=%d encoding=%d "
"format=%d "
"kind=%u dimension=%u mip_mode=%u read=%d written=%d atomic=%d compare=%d "
"base_level=%u last_level=%u max_mip=%u base_array=%u swizzle_ok=%d "
"addr=0x%016" PRIx64 " size=0x%016" PRIx64
" extent=%ux%ux%u type=%u format=%u tile=%u swizzle=0x%03x "
"dwords=%08x,%08x,%08x,%08x,%08x,%08x,%08x,%08x\n",
resource_ok, descriptor_ok, encoding_ok, format_ok,
static_cast<uint32_t>(resource.kind), static_cast<uint32_t>(resource.dimension),
static_cast<uint32_t>(resource.mip_mode), resource.read, resource.written,
resource.atomic, resource.depth_compare, descriptor.BaseLevel(),
descriptor.LastLevel(), descriptor.MaxMip(), descriptor.BaseArray5(), swizzle_ok,
descriptor.Base40(), size, static_cast<uint32_t>(descriptor.Width5()) + 1u,
static_cast<uint32_t>(descriptor.Height5()) + 1u,
static_cast<uint32_t>(descriptor.Depth()) + 1u, descriptor.Type(), format,
descriptor.TileMode(), descriptor.DstSelXYZW(), descriptor.fields[0],
descriptor.fields[1], descriptor.fields[2], descriptor.fields[3], descriptor.fields[4],
descriptor.fields[5], descriptor.fields[6], descriptor.fields[7]);
}
}
static DescriptorCache::TextureBinding
NativeTexture(uint64_t submit_id, CommandBuffer& command_buffer,
const ShaderRecompiler::IR::ImageResource& resource,
const ShaderRecompiler::IR::DescriptorValue& value) {
ShaderTextureResource descriptor;
CopyNativeDescriptor(value, descriptor.fields);
const bool storage = resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImage ||
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint;
const auto variant = NativeTextureVariant(resource);
if (storage) {
ValidateStorageImageResource(resource);
}
if (descriptor.IsNull()) {
return storage ? DescriptorCache::TextureBinding {&GetDummyStorageTexture(variant),
TextureVariantDefaultView(variant)}
: DescriptorCache::TextureBinding {&GetDummySampledTexture(variant),
TextureVariantDefaultView(variant)};
}
const auto address = descriptor.Base40();
const auto width = static_cast<uint32_t>(descriptor.Width5()) + 1u;
const auto height = static_cast<uint32_t>(descriptor.Height5()) + 1u;
const auto base_level = descriptor.BaseLevel();
const auto last_level = descriptor.LastLevel();
const auto levels = static_cast<uint32_t>(descriptor.MaxMip()) + 1u;
if (base_level > last_level || last_level >= levels) {
EXIT("unsupported texture mip view: base=%u last=%u levels=%u\n", base_level, last_level,
levels);
}
const auto view_levels = last_level - base_level + 1u;
const auto depth = static_cast<uint32_t>(descriptor.Depth()) + 1u;
const auto tile = descriptor.TileMode();
const auto format = descriptor.Format();
const bool sampled_numeric_class =
storage || ((resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint) ==
Prospero::IsUintTextureFormat(format));
if (!storage &&
(resource.kind == ShaderRecompiler::IR::ResourceKind::Image ||
resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint) &&
!sampled_numeric_class) {
EXIT("sampled image numeric class mismatch: kind=%u format=%u addr=0x%016" PRIx64 "\n",
static_cast<uint32_t>(resource.kind), format, address);
}
const auto view_format = TextureGetFormat(format);
const auto type = static_cast<Prospero::ImageType>(descriptor.Type());
const auto target_view =
ResolveTargetTextureView(resource, type, descriptor.BaseArray5(), depth);
const auto pitch = TileGetTexturePitch(format, width, levels, tile);
const auto swizzle = descriptor.DstSelXYZW();
TileSizeAlign size;
TileGetTextureTotalSize(format, width, height, depth, pitch, levels, tile,
type == Prospero::ImageType::kColor3D, size);
EXIT_NOT_IMPLEMENTED(size.size == 0 ||
(address & (static_cast<uint64_t>(size.align) - 1u)) != 0);
if (storage) {
ValidateStorageTexture(resource, descriptor, size.size);
GetRenderContext().GetBufferCache().ValidateGpuAccess(address, size.size, resource.read,
resource.written);
}
VulkanImage* image = nullptr;
int view = VulkanImage::VIEW_DEFAULT;
vk::ImageView image_view = nullptr;
const bool check_depth = static_cast<Prospero::TileMode>(tile) == Prospero::TileMode::kDepth ||
descriptor.MsaaDepth();
if (image == nullptr) {
if (check_depth) {
image = GetRenderContext().GetTextureCache().FindDepthTargetByRange(
command_buffer, address, size.size, true);
} else {
image = GetRenderContext().GetTextureCache().FindRenderTargetByRange(
command_buffer, address, size.size);
}
if (image != nullptr) {
if (check_depth) {
const bool uint_reinterpret =
IsSupportedSampledDepthUintResource(resource) &&
IsSupportedDepthTargetDescriptor(descriptor, *image) &&
IsSupportedDepthTextureEncoding(descriptor) &&
IsDepthUintTextureReinterpretation(image->format, descriptor.Format(),
view_format);
const bool uint_storage_reinterpret =
storage && IsSupportedStorageImageResource(resource) &&
IsSupportedStorageTextureDescriptor(resource, descriptor) &&
IsSupportedStorageTextureEncoding(descriptor) &&
IsDepthUintTextureReinterpretation(image->format, descriptor.Format(),
view_format);
if (uint_reinterpret || uint_storage_reinterpret) {
image = nullptr;
} else {
const auto depth_view = ResolveTargetTextureView(
resource, type, descriptor.BaseArray5(), image->layers);
ValidateDepthTargetBinding(resource, descriptor, image, view_format, size.size);
if (depth_view.type ==
static_cast<vk::ImageViewType>(VK_IMAGE_VIEW_TYPE_MAX_ENUM)) {
EXIT("unsupported sampled depth target view: dimension=%u "
"descriptor_type=%u "
"base_array=%u image_layers=%u\n",
static_cast<uint32_t>(resource.dimension), descriptor.Type(),
descriptor.BaseArray5(), image->layers);
}
image_view = GetRenderContext().GetTextureCache().GetDepthTargetSampledView(
*static_cast<DepthStencilVulkanImage*>(image), view_format, swizzle, 0, 1,
depth_view.type, depth_view.base_layer, depth_view.layer_count);
}
} else {
if (!(storage ? IsSupportedStorageImageResource(resource)
: IsSupportedSampledColorResource(resource)) ||
image->type != VulkanImageType::RenderTexture || width != image->extent.width ||
height != image->extent.height ||
(storage ? levels != image->mip_levels || base_level != 0
: levels != image->mip_levels || base_level >= levels) ||
target_view.type ==
static_cast<vk::ImageViewType>(VK_IMAGE_VIEW_TYPE_MAX_ENUM) ||
target_view.base_layer >= image->layers ||
target_view.layer_count > image->layers - target_view.base_layer) {
EXIT("unsupported cached render-target image view: storage=%d resource=%u "
"dimension=%u"
" image_type=%u layers=%u extent=%ux%u/%ux%u depth=%u"
" levels=%u/%u base_level=%u base_array=%u descriptor_type=%u\n",
storage, static_cast<uint32_t>(resource.kind),
static_cast<uint32_t>(resource.dimension),
static_cast<uint32_t>(image->type), image->layers, width, height,
image->extent.width, image->extent.height, depth, levels,
image->mip_levels, base_level, descriptor.BaseArray5(),
static_cast<uint32_t>(type));
}
if (storage) {
view = SelectStorageColorView(image->format, view_format, swizzle);
image_view = GetRenderContext().GetTextureCache().GetRenderTargetStorageView(
*static_cast<RenderTextureVulkanImage*>(image), view_format, base_level,
view_levels, target_view.type, target_view.base_layer,
target_view.layer_count);
} else {
image_view = GetRenderContext().GetTextureCache().GetSampledColorView(
*image, view_format, swizzle, base_level, view_levels, target_view.type,
target_view.base_layer, target_view.layer_count);
}
}
if (image != nullptr && image_view == nullptr && image->image_view[view] == nullptr) {
EXIT("required cached texture image view is missing\n");
}
if (storage && image != nullptr) {
GetRenderContext().GetTextureCache().MarkGpuWritten(*image);
}
}
}
if (image == nullptr) {
const auto video = Presentation::DisplayBufferFind(address);
if (video.image != nullptr) {
if (storage) {
const bool exact =
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint &&
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
!resource.read && resource.written && !resource.atomic &&
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8_8_8_8UInt) &&
view_format == vk::Format::eR8G8B8A8Uint && swizzle == DstSel(6, 5, 4, 7) &&
width == video.image->extent.width && height == video.image->extent.height &&
depth == 1 && levels == 1 && base_level == 0 && view_levels == 1 &&
type == Prospero::ImageType::kColor2D && video.size == size.size &&
video.pitch == pitch &&
(video.image->format == vk::Format::eR8G8B8A8Srgb ||
video.image->format == vk::Format::eB8G8R8A8Srgb) &&
video.image->image_view[VulkanImage::VIEW_STORAGE] != nullptr;
if (!exact) {
EXIT("unsupported storage access to video-out surface: format=%u view=%d"
" extent=%ux%u size=0x%016" PRIx64 " pitch=%u\n",
format, static_cast<int>(view_format), width, height, size.size, pitch);
}
image = video.image;
view = VulkanImage::VIEW_STORAGE;
GetRenderContext().GetTextureCache().MarkGpuWritten(*image);
} else {
const bool exact =
IsSupportedSampledVideoOutView(resource, descriptor, *video.image) &&
width == video.image->extent.width && height == video.image->extent.height &&
levels == 1 && base_level == 0 && view_levels == 1 && video.size == size.size &&
video.pitch == pitch;
if (!exact) {
EXIT("unsupported sampled access to video-out surface: resource=%u dimension=%u"
" image_format=%d view_format=%d swizzle=0x%03x extent=%ux%u/%ux%u"
" depth=%u levels=%u base=%u count=%u type=%u size=0x%016" PRIx64
"/0x%016" PRIx64 " pitch=%u/%u\n",
static_cast<uint32_t>(resource.kind),
static_cast<uint32_t>(resource.dimension),
static_cast<int>(video.image->format), static_cast<int>(view_format),
swizzle, width, height, video.image->extent.width,
video.image->extent.height, depth, levels, base_level, view_levels,
static_cast<uint32_t>(type), size.size, video.size, pitch, video.pitch);
}
image = video.image;
image_view = GetRenderContext().GetTextureCache().GetSampledColorView(
*video.image, view_format, swizzle, 0, 1, vk::ImageViewType::e2D, 0, 1);
}
}
}
if (image == nullptr) {
auto& texture_cache = GetRenderContext().GetTextureCache();
const bool metadata_read = texture_cache.QueryRegion(address, size.size).metadata_pages;
if (storage && metadata_read) {
EXIT("storage texture overlaps surface metadata\n");
}
if (!storage && metadata_read) {
ValidateMetadataReuseTexture(resource, descriptor, size.size);
}
(void)submit_id;
(void)command_buffer;
ImageInfo info {};
info.address = address;
info.size = size.size;
info.format = format;
info.width = width;
info.height = height;
info.pitch = pitch;
info.base_level = base_level;
info.levels = levels;
info.view_levels = view_levels;
info.tile = tile;
info.swizzle = swizzle;
info.depth = depth;
info.type = descriptor.Type();
info.base_array = descriptor.BaseArray5();
if (storage) {
image = &texture_cache.FindStorageTexture(command_buffer, info);
view = VulkanImage::VIEW_DEFAULT;
image_view = texture_cache.GetStorageTextureStorageView(
*static_cast<StorageTextureVulkanImage*>(image), base_level);
} else {
image = &texture_cache.FindTexture(command_buffer, info, metadata_read);
if (image->type == VulkanImageType::StorageTexture) {
image_view = texture_cache.GetStorageTextureSampledView(
*static_cast<StorageTextureVulkanImage*>(image), info);
}
}
}
EXIT_NOT_IMPLEMENTED(image == nullptr);
if (NeedsStaticSampledArrayView(resource.dimension ==
ShaderRecompiler::Decoder::ImageDimension::Dim2DArray,
image_view != nullptr)) {
view = SelectSampledTextureArrayView(*image, view);
}
return {image, view, image_view};
}
static vk::Sampler NativeSampler(const ShaderRecompiler::IR::Program& program, uint32_t index,
const ShaderRecompiler::IR::DescriptorValue& value) {
ShaderSamplerResource descriptor;
CopyNativeDescriptor(value, descriptor.fields);
const bool depth_compare = std::any_of(program.info.sampled_pairs.begin(),
program.info.sampled_pairs.end(), [&](const auto& pair) {
return pair.sampler == index &&
pair.image < program.info.images.size() &&
program.info.images[pair.image].depth_compare;
});
if (!depth_compare) {
descriptor.fields[0] &= ~(0x7u << 12u);
}
return GetRenderContext().GetSamplerCache().GetSampler(descriptor);
}
static BufferView NativeUpload(CommandBuffer& command_buffer, std::span<const uint32_t> data) {
EXIT_IF(data.empty());
BufferView result;
EXIT_IF(!GetRenderContext().GetBufferCache().UploadHostData(
command_buffer, data.data(), data.size_bytes(), 256, result.buffer, result.offset,
result.range));
return result;
}
void BindDescriptors(uint64_t submit_id, CommandBuffer& buffer,
vk::PipelineBindPoint pipeline_bind_point, vk::PipelineLayout layout,
const ShaderStageRuntime& runtime, vk::ShaderStageFlags vk_stage,
DescriptorCache::Stage stage) {
KYTY_PROFILER_FUNCTION();
EXIT_IF(!runtime);
const auto& program = *runtime.program;
const auto& snapshot = *runtime.resources;
std::string error;
if (!ShaderRecompiler::IR::ValidateResourceSpecialization(program, snapshot, &error)) {
EXIT("invalid native shader runtime snapshot: %s\n", error.c_str());
}
auto vk_buffer = buffer.Handle();
const auto shader_stages = ShaderPipelineStages(vk_stage);
DescriptorCache::NativeDescriptors descriptors;
descriptors.buffers.reserve(program.info.buffers.size());
for (uint32_t i = 0; i < program.info.buffers.size(); i++) {
ShaderBufferResource descriptor;
CopyNativeDescriptor(snapshot.buffers[i], descriptor.fields);
descriptors.buffers.push_back(
NativeStorageBuffer(submit_id, buffer, descriptor, program.info.buffers[i]));
}
descriptors.images.reserve(program.info.images.size());
for (uint32_t i = 0; i < program.info.images.size(); i++) {
const auto kind = program.info.images[i].kind;
if ((kind == ShaderRecompiler::IR::ResourceKind::StorageImage ||
kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint) &&
vk_stage != vk::ShaderStageFlagBits::eCompute) {
EXIT("storage images are unsupported outside compute shaders\n");
}
descriptors.images.push_back(
NativeTexture(submit_id, buffer, program.info.images[i], snapshot.images[i]));
}
descriptors.samplers.reserve(program.info.samplers.size());
for (uint32_t i = 0; i < program.info.samplers.size(); i++) {
descriptors.samplers.push_back(NativeSampler(program, i, snapshot.samplers[i]));
}
descriptors.addresses.reserve(program.info.addresses.size());
for (uint32_t i = 0; i < program.info.addresses.size(); i++) {
descriptors.addresses.push_back(NativeAddressBuffer(
submit_id, buffer, program.info.addresses[i], snapshot.addresses[i]));
}
if (ShaderRecompiler::IR::FindBinding(
program.bindings, ShaderRecompiler::IR::DescriptorBindingKind::FlattenedSrt) !=
nullptr) {
descriptors.flattened_srt = NativeUpload(buffer, snapshot.flattened_srt);
}
std::vector<uint32_t> user_data;
user_data.reserve(program.bindings.user_data_registers.size());
for (const auto reg: program.bindings.user_data_registers) {
user_data.push_back(snapshot.user_data[reg - program.user_data_base]);
}
if (ShaderRecompiler::IR::FindBinding(
program.bindings, ShaderRecompiler::IR::DescriptorBindingKind::UserData) != nullptr) {
descriptors.user_data = NativeUpload(buffer, user_data);
}
if (ShaderRecompiler::IR::FindBinding(
program.bindings, ShaderRecompiler::IR::DescriptorBindingKind::Gds) != nullptr) {
descriptors.gds.buffer = &GetRenderContext().GetGdsBuffer().GetBuffer();
const auto barrier = MakeGdsDependency(*descriptors.gds.buffer);
vk_buffer.pipelineBarrier(
vk::PipelineStageFlagBits::eHost | vk::PipelineStageFlagBits::eTransfer |
vk::PipelineStageFlagBits::eAllGraphics | vk::PipelineStageFlagBits::eComputeShader,
shader_stages, vk::DependencyFlags {}, 0, nullptr, 1, &barrier, 0, nullptr);
}
for (uint32_t i = 0; i < program.info.images.size(); i++) {
auto* image = descriptors.images[i].image;
const auto& resource = program.info.images[i];
if (resource.kind == ShaderRecompiler::IR::ResourceKind::Image ||
resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint) {
switch (image->type) {
case VulkanImageType::DepthStencil:
GraphicsRenderDepthStencilBarrier(vk_buffer, *image);
break;
case VulkanImageType::RenderTexture:
case VulkanImageType::StorageTexture:
GraphicsRenderTextureBarrier(vk_buffer, *image);
break;
case VulkanImageType::VideoOut:
GraphicsRenderColorImageBarrier(vk_buffer, *image, RENDER_COLOR_IMAGE_LAYOUT);
break;
default: break;
}
} else {
const auto barrier =
MakeStorageImageDependency(*image, resource.read, resource.written);
vk_buffer.pipelineBarrier(vk::PipelineStageFlagBits::eAllCommands, shader_stages,
vk::DependencyFlags {}, 0, nullptr, 0, nullptr, 1, &barrier);
image->layout = vk::ImageLayout::eGeneral;
}
}
if (!program.bindings.descriptors.empty()) {
auto& set =
GetRenderContext().GetDescriptorCache().GetDescriptor(stage, program, descriptors);
vk_buffer.bindDescriptorSets(pipeline_bind_point, layout, program.bindings.descriptor_set,
1, &set.set, 0, nullptr);
buffer.RecycleDescriptorAfterFence(set);
}
if (program.bindings.push_constant_size != 0) {
EXIT_IF(program.bindings.push_constant_size != user_data.size() * sizeof(uint32_t));
vk_buffer.pushConstants(layout, vk_stage, program.bindings.push_constant_offset,
program.bindings.push_constant_size, user_data.data());
}
}
} // namespace Libs::Graphics
@@ -1,51 +0,0 @@
#include "graphics/host_gpu/renderer/dummyTextureCache.h"
#include "common/assert.h"
#include "graphics/host_gpu/renderer/image.h"
#include "graphics/host_gpu/transfer.h"
#include <algorithm>
namespace Libs::Graphics {
namespace {
[[nodiscard]] constexpr size_t DummyTextureIndex(bool uint_format, bool image_3d) noexcept {
return (image_3d ? 2u : 0u) + (uint_format ? 1u : 0u);
}
} // namespace
DummyTextureCache::~DummyTextureCache() {
Common::LockGuard lock(m_mutex);
const auto populated = [](const auto& slots) {
return std::ranges::any_of(slots, [](const auto& slot) { return slot.image != nullptr; });
};
if (!populated(m_sampled) && !populated(m_storage)) {
return;
}
Transfer::WaitForGraphicsIdle();
const auto destroy = [](auto& slots) {
for (auto& slot: slots) {
if (slot.image != nullptr) {
ImageOps::Destroy(*slot.image);
slot.image = nullptr;
}
}
};
destroy(m_sampled);
destroy(m_storage);
}
VulkanImage& DummyTextureCache::Get(Usage usage, bool uint_format, bool image_3d) {
Common::LockGuard lock(m_mutex);
auto& slots = usage == Usage::Storage ? m_storage : m_sampled;
auto& slot = slots[DummyTextureIndex(uint_format, image_3d)];
if (slot.image == nullptr) {
slot.image = ImageOps::CreateDummyTexture(uint_format, image_3d,
usage == Usage::Storage);
}
return *slot.image;
}
} // namespace Libs::Graphics
@@ -1,35 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_DUMMYTEXTURECACHE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_DUMMYTEXTURECACHE_H_
#include "common/abi.h"
#include "common/threads.h"
#include "graphics/host_gpu/graphicContext.h"
#include <array>
#include <cstdint>
namespace Libs::Graphics {
class DummyTextureCache final {
public:
enum class Usage : uint8_t { Sampled, Storage };
DummyTextureCache() = default;
~DummyTextureCache();
KYTY_CLASS_NO_COPY(DummyTextureCache);
[[nodiscard]] VulkanImage& Get(Usage usage, bool uint_format, bool image_3d);
private:
struct Slot {
GpuTextureVulkanImage* image = nullptr;
};
Common::Mutex m_mutex;
std::array<Slot, 4> m_sampled {};
std::array<Slot, 4> m_storage {};
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_DUMMYTEXTURECACHE_H_
@@ -1,357 +0,0 @@
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "common/assert.h"
#include "common/logging/log.h"
#include "common/profiler.h"
#include "graphics/host_gpu/renderer/colorRenderTarget.h"
#include "graphics/host_gpu/renderer/depthRenderTarget.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <atomic>
namespace Libs::Graphics {
VulkanFramebuffer* FramebufferCache::CreateFramebuffer(RenderColorInfo* colors,
uint32_t requested_color_count,
RenderDepthInfo& depth) {
KYTY_PROFILER_FUNCTION();
Common::LockGuard lock(m_mutex);
EXIT_IF(colors == nullptr);
EXIT_IF(requested_color_count > RENDER_COLOR_ATTACHMENTS_MAX);
bool with_depth = (depth.format != vk::Format::eUndefined && depth.vulkan_buffer != nullptr);
bool with_color[RENDER_COLOR_ATTACHMENTS_MAX] = {};
uint32_t color_count = 0;
VulkanImage* first_color = nullptr;
vk::Extent2D first_color_extent = {};
uint32_t attachment_samples = 0;
for (uint32_t i = 0; i < requested_color_count; i++) {
with_color[i] = (colors[i].vulkan_buffer != nullptr);
if (!with_color[i]) {
break;
}
if (first_color == nullptr) {
first_color = colors[i].vulkan_buffer;
first_color_extent = colors[i].extent;
attachment_samples = colors[i].samples;
} else if (colors[i].extent.width != first_color_extent.width ||
colors[i].extent.height != first_color_extent.height) {
LOGF("Framebuffer: temporary: dropping mismatched MRT%u attachment color0=%ux%u "
"color%u=%ux%u\n",
i, first_color_extent.width, first_color_extent.height, i, colors[i].extent.width,
colors[i].extent.height);
with_color[i] = false;
break;
}
if (colors[i].samples != attachment_samples ||
colors[i].vulkan_buffer->samples != colors[i].samples) {
EXIT("Framebuffer: mismatched color attachment samples at slot %u, expected=%u "
"requested=%u image=%u\n",
i, attachment_samples, colors[i].samples, colors[i].vulkan_buffer->samples);
}
color_count++;
}
if (with_depth) {
if (depth.samples != depth.vulkan_buffer->samples) {
EXIT("Framebuffer: depth attachment sample identity mismatch, requested=%u image=%u\n",
depth.samples, depth.vulkan_buffer->samples);
}
if (attachment_samples == 0) {
attachment_samples = depth.samples;
} else if (attachment_samples != depth.samples) {
EXIT(
"Framebuffer: mixed color/depth sample counts are unsupported, color=%u depth=%u\n",
attachment_samples, depth.samples);
}
}
if (!with_depth && color_count == 0) {
LOGF("Framebuffer: warning: no color or depth attachment\n");
return nullptr;
}
if (vulkan_sample_count(attachment_samples) == vk::SampleCountFlagBits {}) {
EXIT("Framebuffer: invalid attachment sample count %u\n", attachment_samples);
}
vk::ImageLayout color_layout[RENDER_COLOR_ATTACHMENTS_MAX] = {};
for (auto& layout: color_layout) {
layout = RENDER_COLOR_IMAGE_LAYOUT;
}
auto depth_layout = (with_depth ? depth_attachment_layout(depth)
: vk::ImageLayout::eDepthStencilAttachmentOptimal);
auto depth_read_only =
(with_depth && depth_layout == vk::ImageLayout::eDepthStencilReadOnlyOptimal);
if (with_depth && first_color != nullptr &&
(first_color_extent.width != depth.vulkan_buffer->extent.width ||
first_color_extent.height != depth.vulkan_buffer->extent.height)) {
static std::atomic<uint32_t> log_count {0};
if (log_count.fetch_add(1, std::memory_order_relaxed) < 16) {
LOGF("Framebuffer: temporary: dropping mismatched PS5 depth attachment color=%ux%u "
"depth=%ux%u format=%s\n",
first_color_extent.width, first_color_extent.height,
depth.vulkan_buffer->extent.width, depth.vulkan_buffer->extent.height,
VulkanToString(depth.format).c_str());
}
depth.format = vk::Format::eUndefined;
depth.vulkan_buffer = nullptr;
depth.vulkan_view = nullptr;
depth.depth_test_enable = false;
depth.depth_write_enable = false;
depth.depth_bounds_test_enable = false;
depth.stencil_test_enable = false;
depth.depth_clear_enable = false;
depth.depth_load_clear_enable = false;
depth.stencil_clear_enable = false;
with_depth = false;
depth_layout = vk::ImageLayout::eDepthStencilAttachmentOptimal;
depth_read_only = false;
}
for (auto& f: m_framebuffers) {
bool color_match = (f.framebuffer != nullptr);
for (uint32_t i = 0; color_match && i < RENDER_COLOR_ATTACHMENTS_MAX; i++) {
const uint64_t image_id =
(i < color_count && with_color[i] ? colors[i].vulkan_buffer->memory.unique_id : 0);
color_match =
color_match && f.image_id[i] == image_id &&
f.color_view[i] ==
(i < color_count && with_color[i] ? colors[i].vulkan_view : nullptr) &&
f.color_clear_enable[i] ==
(i < color_count && with_color[i] && colors[i].color_clear_enable) &&
f.color_layout[i] == color_layout[i];
}
if (color_match && f.depth_id == (with_depth ? depth.vulkan_buffer->memory.unique_id : 0) &&
f.depth_view == (with_depth ? depth.vulkan_view : nullptr) &&
f.depth_clear_enable == depth.depth_load_clear_enable &&
f.stencil_clear_enable == depth.stencil_clear_enable &&
f.depth_read_only == depth_read_only) {
return f.framebuffer;
}
}
EXIT_NOT_IMPLEMENTED(with_depth && first_color != nullptr &&
(first_color_extent.width != depth.vulkan_buffer->extent.width ||
first_color_extent.height != depth.vulkan_buffer->extent.height));
if (first_color == nullptr) {
first_color_extent = depth.vulkan_buffer->extent;
}
auto* framebuffer = new VulkanFramebuffer;
framebuffer->render_pass = nullptr;
framebuffer->framebuffer = nullptr;
framebuffer->samples = attachment_samples;
for (uint32_t i = 0; i < RENDER_COLOR_ATTACHMENTS_MAX; i++) {
framebuffer->color_layout[i] = color_layout[i];
}
framebuffer->depth_layout = depth_layout;
vk::AttachmentDescription attachments[RENDER_COLOR_ATTACHMENTS_MAX + 1] = {};
for (uint32_t i = 0; i < color_count; i++) {
attachments[i].flags = {};
attachments[i].format = colors[i].format;
attachments[i].samples = vulkan_sample_count(attachment_samples);
attachments[i].loadOp = (colors[i].color_clear_enable ? vk::AttachmentLoadOp::eClear
: vk::AttachmentLoadOp::eLoad);
attachments[i].storeOp = vk::AttachmentStoreOp::eStore;
attachments[i].stencilLoadOp = vk::AttachmentLoadOp::eDontCare;
attachments[i].stencilStoreOp = vk::AttachmentStoreOp::eDontCare;
attachments[i].initialLayout = color_layout[i];
attachments[i].finalLayout = RENDER_COLOR_IMAGE_LAYOUT;
}
const uint32_t depth_attachment = color_count;
attachments[depth_attachment].flags = {};
attachments[depth_attachment].format = depth.format;
attachments[depth_attachment].samples = vulkan_sample_count(attachment_samples);
attachments[depth_attachment].loadOp =
(depth.depth_load_clear_enable ? vk::AttachmentLoadOp::eClear
: vk::AttachmentLoadOp::eLoad);
attachments[depth_attachment].storeOp = vk::AttachmentStoreOp::eStore;
attachments[depth_attachment].stencilLoadOp =
(depth.stencil_clear_enable ? vk::AttachmentLoadOp::eClear : vk::AttachmentLoadOp::eLoad);
attachments[depth_attachment].stencilStoreOp = vk::AttachmentStoreOp::eStore;
attachments[depth_attachment].initialLayout = depth_layout;
attachments[depth_attachment].finalLayout = depth_layout;
vk::AttachmentReference color_attachment_ref[RENDER_COLOR_ATTACHMENTS_MAX] = {};
for (uint32_t i = 0; i < RENDER_COLOR_ATTACHMENTS_MAX; i++) {
color_attachment_ref[i].attachment = (i < color_count ? i : VK_ATTACHMENT_UNUSED);
color_attachment_ref[i].layout = RENDER_COLOR_IMAGE_LAYOUT;
}
vk::AttachmentReference depth_attachment_ref {};
depth_attachment_ref.attachment = depth_attachment;
depth_attachment_ref.layout = depth_layout;
vk::SubpassDescription subpass {};
subpass.flags = {};
subpass.pipelineBindPoint = vk::PipelineBindPoint::eGraphics;
subpass.inputAttachmentCount = 0;
subpass.pInputAttachments = nullptr;
subpass.colorAttachmentCount = color_count;
subpass.pColorAttachments = (color_count > 0 ? color_attachment_ref : nullptr);
subpass.pResolveAttachments = nullptr;
subpass.pDepthStencilAttachment = (with_depth ? &depth_attachment_ref : nullptr);
subpass.preserveAttachmentCount = 0;
subpass.pPreserveAttachments = nullptr;
const auto attachment_stage_mask =
static_cast<vk::PipelineStageFlags>(vk::PipelineStageFlagBits::eColorAttachmentOutput |
vk::PipelineStageFlagBits::eEarlyFragmentTests |
vk::PipelineStageFlagBits::eLateFragmentTests);
const auto attachment_access_mask = static_cast<vk::AccessFlags>(
vk::AccessFlagBits::eColorAttachmentRead | vk::AccessFlagBits::eColorAttachmentWrite |
vk::AccessFlagBits::eDepthStencilAttachmentRead |
vk::AccessFlagBits::eDepthStencilAttachmentWrite);
vk::SubpassDependency dependencies[2] = {};
dependencies[0].srcSubpass = VK_SUBPASS_EXTERNAL;
dependencies[0].dstSubpass = 0;
dependencies[0].srcStageMask = vk::PipelineStageFlagBits::eAllCommands;
dependencies[0].dstStageMask = attachment_stage_mask;
dependencies[0].srcAccessMask =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite;
dependencies[0].dstAccessMask = attachment_access_mask;
dependencies[0].dependencyFlags = vk::DependencyFlagBits::eByRegion;
dependencies[1].srcSubpass = 0;
dependencies[1].dstSubpass = VK_SUBPASS_EXTERNAL;
dependencies[1].srcStageMask = attachment_stage_mask;
dependencies[1].dstStageMask = vk::PipelineStageFlagBits::eAllCommands;
dependencies[1].srcAccessMask = attachment_access_mask;
dependencies[1].dstAccessMask =
vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite;
dependencies[1].dependencyFlags = vk::DependencyFlagBits::eByRegion;
vk::RenderPassCreateInfo render_pass_info {};
render_pass_info.sType = vk::StructureType::eRenderPassCreateInfo;
render_pass_info.pNext = nullptr;
render_pass_info.flags = {};
render_pass_info.attachmentCount = color_count + (with_depth ? 1u : 0u);
render_pass_info.pAttachments = attachments;
render_pass_info.subpassCount = 1;
render_pass_info.pSubpasses = &subpass;
render_pass_info.dependencyCount = 2;
render_pass_info.pDependencies = dependencies;
auto result =
m_graphics.device.createRenderPass(&render_pass_info, nullptr, &framebuffer->render_pass);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
vk::Format color_formats[RENDER_COLOR_ATTACHMENTS_MAX] = {};
for (uint32_t i = 0; i < color_count; i++) {
color_formats[i] = colors[i].format;
}
framebuffer->render_pass_id = render_pass_compat_id(
color_count, color_formats, with_depth, depth.format, depth_layout, attachment_samples);
EXIT_NOT_IMPLEMENTED(framebuffer->render_pass == nullptr);
vk::ImageView views[RENDER_COLOR_ATTACHMENTS_MAX + 1] = {};
for (uint32_t i = 0; i < color_count; i++) {
if (colors[i].vulkan_view == nullptr) {
EXIT("Framebuffer: color attachment view is missing at slot %u\n", i);
}
views[i] = colors[i].vulkan_view;
}
if (with_depth) {
if (depth.vulkan_view == nullptr) {
EXIT("Framebuffer: depth attachment view is missing\n");
}
views[color_count] = depth.vulkan_view;
}
vk::FramebufferCreateInfo framebuffer_info {};
framebuffer_info.sType = vk::StructureType::eFramebufferCreateInfo;
framebuffer_info.pNext = nullptr;
framebuffer_info.flags = {};
framebuffer_info.renderPass = framebuffer->render_pass;
framebuffer_info.attachmentCount = color_count + (with_depth ? 1u : 0u);
framebuffer_info.pAttachments = views;
framebuffer_info.width = first_color_extent.width;
framebuffer_info.height = first_color_extent.height;
framebuffer_info.layers = 1;
result =
m_graphics.device.createFramebuffer(&framebuffer_info, nullptr, &framebuffer->framebuffer);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
EXIT_NOT_IMPLEMENTED(framebuffer->framebuffer == nullptr);
Framebuffer fnew;
fnew.framebuffer = framebuffer;
for (uint32_t i = 0; i < RENDER_COLOR_ATTACHMENTS_MAX; i++) {
fnew.image_id[i] =
(i < color_count && with_color[i] ? colors[i].vulkan_buffer->memory.unique_id : 0);
fnew.color_view[i] = (i < color_count && with_color[i] ? colors[i].vulkan_view : nullptr);
fnew.color_clear_enable[i] =
(i < color_count && with_color[i] && colors[i].color_clear_enable);
fnew.color_layout[i] = color_layout[i];
}
fnew.depth_id = (with_depth ? depth.vulkan_buffer->memory.unique_id : 0);
fnew.depth_view = (with_depth ? depth.vulkan_view : nullptr);
fnew.depth_clear_enable = depth.depth_load_clear_enable;
fnew.stencil_clear_enable = depth.stencil_clear_enable;
fnew.depth_read_only = depth_read_only;
bool updated = false;
for (auto& f: m_framebuffers) {
if (f.framebuffer == nullptr) {
f = fnew;
updated = true;
break;
}
}
if (!updated) {
m_framebuffers.push_back(fnew);
}
return framebuffer;
}
void FramebufferCache::FreeFramebufferByColor(VulkanImage& image) {
Common::LockGuard lock(m_mutex);
for (auto& f: m_framebuffers) {
bool uses_image = false;
for (auto image_id: f.image_id) {
if (image_id == image.memory.unique_id) {
uses_image = true;
break;
}
}
if (f.framebuffer != nullptr && uses_image) {
m_graphics.device.destroyFramebuffer(f.framebuffer->framebuffer, nullptr);
m_graphics.device.destroyRenderPass(f.framebuffer->render_pass, nullptr);
delete f.framebuffer;
f.framebuffer = nullptr;
}
}
}
void FramebufferCache::FreeFramebufferByDepth(DepthStencilVulkanImage& image) {
Common::LockGuard lock(m_mutex);
for (auto& f: m_framebuffers) {
if (f.framebuffer != nullptr && f.depth_id == image.memory.unique_id) {
m_graphics.device.destroyFramebuffer(f.framebuffer->framebuffer, nullptr);
m_graphics.device.destroyRenderPass(f.framebuffer->render_pass, nullptr);
delete f.framebuffer;
f.framebuffer = nullptr;
}
}
}
} // namespace Libs::Graphics
@@ -1,85 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_FRAMEBUFFERCACHE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_FRAMEBUFFERCACHE_H_
#include "common/abi.h"
#include "common/assert.h"
#include "common/common.h"
#include "common/threads.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <vector>
namespace Libs::Graphics {
struct RenderColorInfo;
struct RenderDepthInfo;
static constexpr vk::ImageLayout RENDER_COLOR_IMAGE_LAYOUT = vk::ImageLayout::eGeneral;
struct VulkanFramebuffer {
vk::RenderPass render_pass = nullptr;
uint64_t render_pass_id = 0;
vk::Framebuffer framebuffer = nullptr;
uint32_t samples = 1;
vk::ImageLayout color_layout[RENDER_COLOR_ATTACHMENTS_MAX] = {};
vk::ImageLayout depth_layout = vk::ImageLayout::eDepthStencilAttachmentOptimal;
};
inline uint64_t render_pass_compat_id(uint32_t color_count, const vk::Format* color_formats,
bool with_depth, vk::Format depth_format,
vk::ImageLayout depth_layout, uint32_t samples) {
uint64_t id = 0xcbf29ce484222325ull;
auto mix = [&id](uint64_t v) {
id ^= v;
id *= 0x100000001b3ull;
};
mix(color_count);
for (uint32_t i = 0; i < color_count; i++) {
mix(static_cast<uint32_t>(color_formats[i]));
}
mix(with_depth ? 1u : 0u);
mix(static_cast<uint32_t>(depth_format));
mix(static_cast<uint32_t>(depth_layout));
mix(samples);
return id;
}
class FramebufferCache {
public:
explicit FramebufferCache(GraphicContext& graphics): m_graphics(graphics) {
EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
}
~FramebufferCache() { KYTY_NOT_IMPLEMENTED; }
KYTY_CLASS_NO_COPY(FramebufferCache);
VulkanFramebuffer* CreateFramebuffer(RenderColorInfo* colors, uint32_t color_count,
RenderDepthInfo& depth);
void FreeFramebufferByColor(VulkanImage& image);
void FreeFramebufferByDepth(DepthStencilVulkanImage& image);
private:
struct Framebuffer {
VulkanFramebuffer* framebuffer = nullptr;
uint64_t image_id[RENDER_COLOR_ATTACHMENTS_MAX] = {};
vk::ImageView color_view[RENDER_COLOR_ATTACHMENTS_MAX] = {};
uint64_t depth_id = 0;
vk::ImageView depth_view = nullptr;
bool color_clear_enable[RENDER_COLOR_ATTACHMENTS_MAX] = {};
vk::ImageLayout color_layout[RENDER_COLOR_ATTACHMENTS_MAX] = {};
bool depth_clear_enable = false;
bool stencil_clear_enable = false;
bool depth_read_only = false;
};
GraphicContext& m_graphics;
Common::Mutex m_mutex;
std::vector<Framebuffer> m_framebuffers;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_FRAMEBUFFERCACHE_H_
@@ -1,67 +0,0 @@
#include "graphics/host_gpu/renderer/gdsBuffer.h"
#include "common/assert.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vma.h"
namespace Libs::Graphics {
void GdsBuffer::Init() {
if (m_buffer == nullptr) {
m_buffer = std::make_unique<VulkanBuffer>();
m_buffer->usage = vk::BufferUsageFlagBits::eStorageBuffer;
m_buffer->memory.property = vk::MemoryPropertyFlagBits::eHostVisible |
vk::MemoryPropertyFlagBits::eHostCoherent |
vk::MemoryPropertyFlagBits::eHostCached;
m_graphics.CreateBuffer(DW_SIZE * 4, *m_buffer);
}
}
void GdsBuffer::Clear(uint64_t dw_offset, uint32_t dw_num, uint32_t clear_value) {
Common::LockGuard lock(m_mutex);
Init();
EXIT_NOT_IMPLEMENTED(dw_offset >= DW_SIZE);
EXIT_NOT_IMPLEMENTED(dw_offset + dw_num > DW_SIZE);
void* data = nullptr;
m_graphics.MapMemory(m_buffer->memory, data);
for (uint32_t i = 0; i < dw_num; i++) {
static_cast<uint32_t*>(data)[dw_offset + i] = clear_value;
}
m_graphics.UnmapMemory(m_buffer->memory);
}
void GdsBuffer::Read(uint32_t* dst, uint32_t dw_offset, uint32_t dw_size) {
EXIT_IF(dst == nullptr);
Common::LockGuard lock(m_mutex);
Init();
EXIT_NOT_IMPLEMENTED(dw_offset >= DW_SIZE);
EXIT_NOT_IMPLEMENTED(dw_offset + dw_size > DW_SIZE);
void* data = nullptr;
m_graphics.MapMemory(m_buffer->memory, data);
for (uint32_t i = 0; i < dw_size; i++) {
dst[i] = static_cast<uint32_t*>(data)[dw_offset + i];
}
m_graphics.UnmapMemory(m_buffer->memory);
}
VulkanBuffer& GdsBuffer::GetBuffer() {
Common::LockGuard lock(m_mutex);
Init();
return *m_buffer;
}
} // namespace Libs::Graphics
@@ -1,39 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_GDSBUFFER_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_GDSBUFFER_H_
#include "common/abi.h"
#include "common/assert.h"
#include "common/common.h"
#include "common/threads.h"
#include "graphics/host_gpu/graphicContext.h"
#include <memory>
namespace Libs::Graphics {
class GdsBuffer {
public:
explicit GdsBuffer(GraphicContext& graphics): m_graphics(graphics) {
EXIT_NOT_IMPLEMENTED(!Common::Thread::IsMainThread());
}
~GdsBuffer() { KYTY_NOT_IMPLEMENTED; }
KYTY_CLASS_NO_COPY(GdsBuffer);
void Clear(uint64_t dw_offset, uint32_t dw_num, uint32_t clear_value);
void Read(uint32_t* dst, uint32_t dw_offset, uint32_t dw_size);
VulkanBuffer& GetBuffer();
private:
static constexpr uint64_t DW_SIZE = 0x3000;
void Init();
GraphicContext& m_graphics;
Common::Mutex m_mutex;
std::unique_ptr<VulkanBuffer> m_buffer;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_GDSBUFFER_H_
@@ -1,133 +0,0 @@
#include "graphics/host_gpu/renderer/gpuResourceManager.h"
#include "common/assert.h"
#include "graphics/guest_gpu/command_processor/commandProcessor.h"
#include "graphics/guest_gpu/graphicsRun.h"
#include "graphics/host_gpu/objects/label.h"
#include "graphics/host_gpu/renderer/render.h"
#include "graphics/host_gpu/renderer/renderContext.h"
namespace Libs::Graphics {
GpuResourceManager::GpuResourceManager(GraphicContext& graphics)
: m_page_manager(FaultThunk, this), m_buffer_cache(graphics, m_page_manager, m_resource_mutex),
m_texture_cache(graphics, m_page_manager, m_buffer_cache, m_resource_mutex) {
m_buffer_cache.SetTextureCache(m_texture_cache);
}
GpuResourceManager::~GpuResourceManager() = default;
bool GpuResourceManager::FaultThunk(void* context, PageFaultAccess access, uint64_t vaddr,
uint64_t size, PageFaultPhase phase) noexcept {
return static_cast<GpuResourceManager*>(context)->InvalidateMemory(access, vaddr, size, phase);
}
bool GpuResourceManager::InvalidateMemory(PageFaultAccess access, uint64_t vaddr, uint64_t size,
PageFaultPhase phase) noexcept {
const bool buffer_handled = m_buffer_cache.InvalidateMemory(access, vaddr, size, phase);
const bool image_handled = m_texture_cache.InvalidateMemory(access, vaddr, size, phase);
return buffer_handled || image_handled;
}
bool GpuResourceManager::HandleFault(PageFaultAccess access, uint64_t fault_vaddr) noexcept {
if (!m_page_manager.IsMapped(fault_vaddr, 1)) {
return false;
}
if (LabelInCallback()) {
EXIT("unsupported guest-memory fault from an asynchronous GPU label callback, "
"addr=0x%016" PRIx64 " access=%u\n",
fault_vaddr, static_cast<uint32_t>(access));
}
if (auto* cp = GraphicsRunCurrentCommandProcessor(); cp != nullptr) {
cp->BeginReadbackTransaction();
bool handled = false;
{
ResourceMutex::FaultScope fault(m_resource_mutex);
handled = m_page_manager.HandleFault(access, fault_vaddr);
}
cp->EndReadbackTransaction();
return handled;
}
if (m_resource_mutex.IsOwnedByCurrentThread()) {
EXIT("unsupported page fault from a pre-owned resource transaction, addr=0x%016" PRIx64
" access=%u\n",
fault_vaddr, static_cast<uint32_t>(access));
}
// Stop command-processor jobs before taking the shared cache transaction. External readback
// workers inherit this paused state and therefore never form resource -> submission lock
// inversion.
GraphicsRunSubmissionLock submissions;
ResourceMutex::FaultScope fault(m_resource_mutex);
return m_page_manager.HandleFault(access, fault_vaddr);
}
void GpuResourceManager::PrepareHostWrite(uint64_t vaddr, uint64_t size) {
if (!m_page_manager.HasAnyMapping(vaddr, size)) {
return;
}
if (LabelInCallback()) {
EXIT("unsupported host write from an asynchronous GPU label callback, addr=0x%016" PRIx64
" size=0x%016" PRIx64 "\n",
vaddr, size);
}
const auto handle_range = [this, vaddr, size]() {
if (!m_page_manager.HandleWriteRange(vaddr, size)) {
EXIT("failed to prepare host write, addr=0x%016" PRIx64 " size=0x%016" PRIx64 "\n",
vaddr, size);
}
};
if (auto* cp = GraphicsRunCurrentCommandProcessor(); cp != nullptr) {
cp->BeginReadbackTransaction();
{
ResourceMutex::FaultScope fault(m_resource_mutex);
handle_range();
}
cp->EndReadbackTransaction();
return;
}
if (m_resource_mutex.IsOwnedByCurrentThread()) {
EXIT("unsupported host write from a pre-owned resource transaction, addr=0x%016" PRIx64
" size=0x%016" PRIx64 "\n",
vaddr, size);
}
GraphicsRunSubmissionLock submissions;
ResourceMutex::FaultScope fault(m_resource_mutex);
handle_range();
}
bool GpuResourceManager::IsMapped(uint64_t vaddr, uint64_t size) const noexcept {
return m_page_manager.IsMapped(vaddr, size);
}
void GpuResourceManager::MapMemory(uint64_t vaddr, uint64_t size, GpuAccess access) {
m_page_manager.OnGpuMap(vaddr, size, access);
}
void GpuResourceManager::UnmapMemory(uint64_t vaddr, uint64_t size, GpuAccess access) {
if (!IsMapped(vaddr, size)) {
EXIT("cannot unmap an unmapped GPU resource range\n");
}
m_texture_cache.UnmapMemory(vaddr, size);
m_buffer_cache.UnmapMemory(vaddr, size);
m_page_manager.OnGpuUnmap(vaddr, size, access);
}
void GpuResourceManager::FillBuffer(CommandBuffer& command, uint64_t vaddr, uint64_t size,
uint32_t value) {
if (command.IsInvalid()) {
EXIT("cannot fill a buffer without a valid render command context\n");
}
Common::LockGuard lock(GetRenderContext().GetMutex());
m_buffer_cache.FillBuffer(&command, vaddr, size, value);
}
void GpuResourceManager::CopyBuffer(CommandBuffer& command, uint64_t dst_vaddr, uint64_t src_vaddr,
uint64_t size) {
if (command.IsInvalid()) {
EXIT("cannot copy a buffer without a valid render command context\n");
}
Common::LockGuard lock(GetRenderContext().GetMutex());
m_buffer_cache.CopyBuffer(&command, dst_vaddr, src_vaddr, size);
}
} // namespace Libs::Graphics
-489
View File
@@ -1,489 +0,0 @@
#include "graphics/host_gpu/renderer/image.h"
#include "common/assert.h"
#include "common/profiler.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/gpuTiler.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/objects/textureCommon.h"
#include "graphics/host_gpu/regionDefinitions.h"
#include "graphics/host_gpu/renderer/framebufferCache.h"
#include "graphics/host_gpu/renderer/imageView.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include "graphics/host_gpu/transfer.h"
#include "graphics/host_gpu/vma.h"
#include "graphics/shader/shader.h"
#include <algorithm>
namespace Libs::Graphics {
namespace {
TextureImageCreateParams MakeImageParams(const ImageInfo& info, bool storage) {
TextureImageCreateParams params {};
params.fmt = info.format;
params.width = info.width;
params.height = info.height;
params.base_level = SelectImageBackingBaseLevel(storage, info.base_level);
params.levels = info.levels;
params.depth = info.depth;
params.type = info.type;
// Storage image views use identity component mapping. The guest storage write mapping is
// validated before this point and intentionally does not become a Vulkan view swizzle.
params.swizzle = storage ? DstSel(4, 5, 6, 7) : info.swizzle;
params.format_usage = TextureFormatUsage::Sampled | TextureFormatUsage::Storage;
params.required_format_usage = storage
? TextureFormatUsage::Sampled | TextureFormatUsage::Storage
: TextureFormatUsage::Sampled;
params.view_usage = storage ? TextureFormatUsage::Sampled | TextureFormatUsage::Storage
: TextureFormatUsage::Sampled;
params.image_layout = TextureUploadDestination::MipLevels;
params.allow_cube_view = !storage;
params.compatible_format_views =
storage && (IsRgba8SrgbViewFormat(TextureGetFormat(info.format)) ||
info.format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32UInt) ||
info.format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32Float));
params.owner = storage ? "StorageTextureCache" : "TextureCache";
return params;
}
bool RenderTargetSupportsStorage(vk::Format format,
vk::ImageCreateFlags flags) {
const auto compatible = SrgbStorageViewFormat(format);
const auto required_flags =
vk::ImageCreateFlagBits::eMutableFormat | vk::ImageCreateFlagBits::eExtendedUsage;
const bool compatible_views = (flags & required_flags) == required_flags;
return ImageViewOps::FormatSupportsStorage(format) ||
(compatible_views && compatible != vk::Format::eUndefined &&
ImageViewOps::FormatSupportsStorage(compatible));
}
vk::ImageCreateFlags RenderTargetCreateFlags(vk::Format format) {
const bool compatible_format_view =
IsRgba8SrgbViewFormat(format) ||
BgraToRgbaSampledViewFormat(format) != vk::Format::eUndefined ||
format == vk::Format::eR8G8B8A8Uint || format == vk::Format::eR16G16B16A16Sfloat ||
format == vk::Format::eR16G16B16A16Uint;
return compatible_format_view
? vk::ImageCreateFlagBits::eMutableFormat | vk::ImageCreateFlagBits::eExtendedUsage
: vk::ImageCreateFlags {0};
}
vk::ImageUsageFlags RenderTargetUsage(vk::Format format,
vk::ImageCreateFlags flags, uint32_t samples) {
auto& graphics = GetRenderContext().GetGraphics();
auto usage = static_cast<vk::ImageUsageFlags>(vk::ImageUsageFlagBits::eColorAttachment) |
static_cast<vk::ImageUsageFlags>(vk::ImageUsageFlagBits::eTransferSrc) |
static_cast<vk::ImageUsageFlags>(vk::ImageUsageFlagBits::eTransferDst);
if (samples == 1) {
usage |= vk::ImageUsageFlagBits::eSampled;
if (RenderTargetSupportsStorage(format, flags)) {
usage |= vk::ImageUsageFlagBits::eStorage;
}
}
vk::ImageFormatProperties properties {};
if (graphics.GetImageFormatProperties(format, vk::ImageType::e2D, vk::ImageTiling::eOptimal,
usage, flags, &properties) != vk::Result::eSuccess ||
!static_cast<bool>(properties.sampleCounts & vulkan_sample_count(samples))) {
EXIT("TextureCache: render-target format does not support required usage, format=%d "
"usage=0x%x samples=%u supported=0x%x\n",
static_cast<int>(format), static_cast<vk::ImageUsageFlags::MaskType>(usage), samples,
static_cast<vk::SampleCountFlags::MaskType>(properties.sampleCounts));
}
return usage;
}
[[nodiscard]] uint32_t RenderTargetTransferFormatImpl(uint32_t bytes_per_element) {
switch (bytes_per_element) {
case 1: return Prospero::GpuEnumValue(Prospero::BufferFormat::k8UNorm);
case 2: return Prospero::GpuEnumValue(Prospero::BufferFormat::k16UNorm);
case 4: return Prospero::GpuEnumValue(Prospero::BufferFormat::k32Float);
case 8: return Prospero::GpuEnumValue(Prospero::BufferFormat::k16_16_16_16Float);
case 16: return Prospero::GpuEnumValue(Prospero::BufferFormat::k32_32_32_32Float);
default:
EXIT("TextureCache: unsupported render-target element size: %u\n", bytes_per_element);
}
}
static constexpr uint32_t DummyTextureSwizzle() {
return Prospero::GpuEnumValue(Prospero::CompSwizzle::kRed) |
(Prospero::GpuEnumValue(Prospero::CompSwizzle::kGreen) << 3u) |
(Prospero::GpuEnumValue(Prospero::CompSwizzle::kBlue) << 6u) |
(Prospero::GpuEnumValue(Prospero::CompSwizzle::kAlpha) << 9u);
}
TextureImageCreateParams MakeDummyTextureParams(bool uint_format, bool image_3d,
TextureFormatUsage usage, const char* owner) {
TextureImageCreateParams params {};
params.fmt = static_cast<uint32_t>(
Prospero::GpuEnumValue(uint_format ? Prospero::BufferFormat::k8_8_8_8UInt
: Prospero::BufferFormat::k8_8_8_8UNorm));
params.width = 1;
params.height = 1;
params.base_level = 0;
params.levels = 1;
params.depth = 1;
params.type = Prospero::GpuEnumValue(image_3d ? Prospero::ImageType::kColor3D
: Prospero::ImageType::kColor2D);
params.swizzle = DummyTextureSwizzle();
params.format_usage = usage;
params.required_format_usage = usage;
params.view_usage = usage;
params.image_layout = TextureUploadDestination::MipLevels;
params.allow_cube_view = true;
params.storage_swizzle_fallback = TextureHasFormatUsage(usage, TextureFormatUsage::Storage);
params.owner = owner;
return params;
}
} // namespace
namespace ImageOps {
uint32_t RenderTargetTransferFormat(uint32_t bytes_per_element) {
return RenderTargetTransferFormatImpl(bytes_per_element);
}
GpuTextureVulkanImage* CreateTexture(const ImageInfo& info, bool storage,
vk::ComponentMapping& components) {
auto* image = storage ? static_cast<GpuTextureVulkanImage*>(new StorageTextureVulkanImage)
: new TextureVulkanImage;
components = TextureCreateImage(*image, MakeImageParams(info, storage));
return image;
}
void CreateTextureViews(GpuTextureVulkanImage& image,
const ImageInfo& info, bool storage, vk::ComponentMapping components) {
if (storage) {
TextureCreateImageViews(image, components, info.type, 0, 0, 1, info.depth, false,
TextureFormatUsage::Sampled | TextureFormatUsage::Storage);
} else {
TextureCreateImageViews(image, components, info.type, info.base_array,
info.base_level, info.view_levels, info.depth, true,
TextureFormatUsage::Sampled);
}
}
void UploadRenderTargetLayers(RenderTextureVulkanImage& image,
const RenderTargetInfo& info, uint32_t base_layer,
uint32_t layer_count, bool refresh) {
if (info.layers == 0 || info.size % info.layers != 0 || layer_count == 0 ||
base_layer >= info.layers || layer_count > info.layers - base_layer ||
base_layer >= image.layers || layer_count > image.layers - base_layer) {
EXIT("TextureCache: invalid render-target layer upload, base=%u count=%u "
"info_layers=%u image_layers=%u size=0x%016" PRIx64 "\n",
base_layer, layer_count, info.layers, image.layers, info.size);
}
if (info.samples != 1 || image.samples != 1) {
EXIT("TextureCache: multisampled render-target upload is unsupported, samples=%u/%u\n",
info.samples, image.samples);
}
if (refresh) {
Transfer::WaitForGraphicsIdle();
}
const auto slice_size = info.size / info.layers;
const auto upload_size = slice_size * layer_count;
const bool standard64 = IsSupportedStandard64RenderTarget(info);
if (standard64 || info.levels > 1 || info.layers > 1) {
const auto format = RenderTargetTransferFormat(info.bytes_per_element);
auto layout = TextureCalcUploadLayout(format, info.width, info.height, info.levels,
layer_count, info.pitch, info.tile_mode, upload_size,
false, false, "TextureCache render target");
const bool render_target_tiled =
info.tile_mode == Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget);
if (!standard64 &&
((render_target_tiled && layout.tile_family != TileBlockFamily::RenderTarget64KB) ||
layout.pitch != info.pitch)) {
EXIT("TextureCache: unsupported render-target mip upload layout, pitch=%u/%u tile=%u\n",
info.pitch, layout.pitch, info.tile_mode);
}
auto regions = TextureBuildUploadRegions(layout, info.format, info.width, info.height,
layer_count, info.levels, true, false,
TextureUploadDestination::MipLevels);
for (auto& region: regions) {
region.dst_layer += base_layer;
}
const auto source_address = info.address + slice_size * base_layer;
TextureUploadGuestImage(image, reinterpret_cast<const void*>(source_address),
upload_size, regions, layout, format, info.width, info.height,
layer_count, info.levels, "TextureCache render target",
vk::ImageLayout::eGeneral);
return;
}
if (info.tile_mode == Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget) &&
Transfer::GuestBufferIsTiled(info.address, slice_size)) {
const auto format = RenderTargetTransferFormat(info.bytes_per_element);
auto layout = TextureCalcUploadLayout(format, info.width, info.height, 1, 1, info.pitch,
info.tile_mode, slice_size, false, false,
"TextureCache render target");
auto regions = TextureBuildUploadRegions(layout, info.format, info.width, info.height, 1, 1,
true, false, TextureUploadDestination::MipLevels);
TextureUploadGuestImage(image, reinterpret_cast<const void*>(info.address),
slice_size, regions, layout, format, info.width, info.height, 1, 1,
"TextureCache render target", vk::ImageLayout::eGeneral);
} else {
Transfer::UploadImage(image, reinterpret_cast<const void*>(info.address),
slice_size, info.pitch, vk::ImageLayout::eGeneral);
}
}
void UploadRenderTarget(RenderTextureVulkanImage& image,
const RenderTargetInfo& info, bool refresh) {
UploadRenderTargetLayers(image, info, 0, info.layers, refresh);
}
RenderTextureVulkanImage* CreateRenderTarget(
const RenderTargetInfo& info) {
auto& graphics = GetRenderContext().GetGraphics();
auto* image = new RenderTextureVulkanImage;
image->extent.width = info.width;
image->extent.height = info.height;
image->format = info.format;
image->mip_levels = info.levels;
image->layers = info.layers;
image->samples = info.samples;
image->layout = vk::ImageLayout::eUndefined;
vk::ImageCreateInfo create {};
create.sType = vk::StructureType::eImageCreateInfo;
create.flags = RenderTargetCreateFlags(info.format);
create.imageType = vk::ImageType::e2D;
create.extent = {info.width, info.height, 1};
create.mipLevels = info.levels;
create.arrayLayers = info.layers;
create.format = info.format;
create.tiling = vk::ImageTiling::eOptimal;
create.initialLayout = vk::ImageLayout::eUndefined;
create.usage = RenderTargetUsage(info.format, create.flags, info.samples);
create.sharingMode = vk::SharingMode::eExclusive;
create.samples = vulkan_sample_count(info.samples);
image->memory.property = vk::MemoryPropertyFlagBits::eDeviceLocal;
if (!graphics.CreateImage(create, *image)) {
EXIT("TextureCache: failed to create render target, addr=0x%016" PRIx64
" extent=%ux%u format=%d\n",
info.address, info.width, info.height, static_cast<int>(info.format));
}
ImageViewOps::CreateRenderTargetViews(*image);
return image;
}
DepthStencilVulkanImage* CreateDepthTarget(const DepthTargetInfo& info) {
auto& graphics = GetRenderContext().GetGraphics();
vk::ImageCreateInfo create {};
create.sType = vk::StructureType::eImageCreateInfo;
create.imageType = vk::ImageType::e2D;
create.extent = {info.width, info.height, 1};
create.mipLevels = 1;
create.arrayLayers = info.layers;
create.format = info.format;
create.tiling = vk::ImageTiling::eOptimal;
create.initialLayout = vk::ImageLayout::eUndefined;
create.usage = DepthTargetImageUsage();
create.sharingMode = vk::SharingMode::eExclusive;
create.samples = vulkan_sample_count(info.samples);
vk::ImageFormatProperties properties {};
if (graphics.GetImageFormatProperties(
info.format, vk::ImageType::e2D, vk::ImageTiling::eOptimal, create.usage,
vk::ImageCreateFlags {}, &properties) != vk::Result::eSuccess ||
!static_cast<bool>(properties.sampleCounts & create.samples)) {
EXIT("TextureCache: depth format does not support required usage, format=%d usage=0x%x "
"samples=%u supported=0x%x\n",
static_cast<int>(info.format),
static_cast<vk::ImageUsageFlags::MaskType>(create.usage), info.samples,
static_cast<vk::SampleCountFlags::MaskType>(properties.sampleCounts));
}
auto* image = new DepthStencilVulkanImage;
image->extent.width = info.width;
image->extent.height = info.height;
image->guest_pitch = info.pitch;
image->layers = info.layers;
image->samples = info.samples;
image->format = info.format;
image->layout = vk::ImageLayout::eUndefined;
image->compressed = false;
image->memory.property = vk::MemoryPropertyFlagBits::eDeviceLocal;
if (!graphics.CreateImage(create, *image)) {
EXIT("TextureCache: failed to create depth target, addr=0x%016" PRIx64
" extent=%ux%u format=%d\n",
info.address, info.width, info.height, static_cast<int>(info.format));
}
ImageViewOps::CreateDepthViews(*image);
return image;
}
void ValidateVideoOut(const VideoOutInfo& info) {
const auto compression =
ClassifyVideoOutCompression(info.compression != VideoOutCompression::Uncompressed,
info.metadata_address, info.dcc_control, 0);
const bool metadata_invalid = compression != VideoOutCompression::Uncompressed &&
compression != VideoOutCompression::Unsupported &&
(info.metadata_address >= TRACKER_ADDRESS_SIZE ||
(info.metadata_address >= info.address &&
info.metadata_address < info.address + info.size));
if (info.address == 0 || info.size == 0 || info.address >= TRACKER_ADDRESS_SIZE ||
info.size > TRACKER_ADDRESS_SIZE - info.address || (info.address & 0xffffu) != 0 ||
info.width == 0 || info.height == 0 || info.width > 16384 || info.height > 16384 ||
info.pitch < info.width ||
info.tile_mode != Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget) ||
compression == VideoOutCompression::Unsupported || compression != info.compression ||
metadata_invalid || !IsSupportedVideoOutFormat(info)) {
EXIT("TextureCache: unsupported video-out surface, addr=0x%016" PRIx64 " size=0x%016" PRIx64
" metadata=0x%016" PRIx64 " dcc=0x%08" PRIx32
" extent=%ux%u pitch=%u tile=%u guest_format=%u bpe=%u vk_format=%d\n",
info.address, info.size, info.metadata_address, info.dcc_control, info.width,
info.height, info.pitch, info.tile_mode, info.guest_format, info.bytes_per_element,
static_cast<int>(info.format));
}
TileSizeAlign exact {};
TileGetTextureTotalSize(info.guest_format, info.width, info.height, 1, info.pitch, 1,
info.tile_mode, false, exact);
if (exact.align != 65536 || exact.size != info.size ||
TileGetTexturePitch(info.guest_format, info.width, 1, info.tile_mode) != info.pitch) {
EXIT("TextureCache: video-out tile layout mismatch, addr=0x%016" PRIx64
" size=0x%016" PRIx64 " expected_size=0x%016" PRIx64 " align=0x%016" PRIx64
" pitch=%u\n",
info.address, info.size, exact.size, exact.align, info.pitch);
}
(void)RenderTargetUsage(info.format, vk::ImageCreateFlags {}, 1);
}
VideoOutVulkanImage* CreateVideoOut(const VideoOutInfo& info) {
auto& graphics = GetRenderContext().GetGraphics();
auto* image = new VideoOutVulkanImage;
image->extent.width = info.width;
image->extent.height = info.height;
image->format = info.format;
image->layout = vk::ImageLayout::eUndefined;
vk::ImageCreateInfo create {};
create.sType = vk::StructureType::eImageCreateInfo;
create.imageType = vk::ImageType::e2D;
create.extent = {info.width, info.height, 1};
create.mipLevels = 1;
create.arrayLayers = 1;
create.format = info.format;
create.tiling = vk::ImageTiling::eOptimal;
create.initialLayout = vk::ImageLayout::eUndefined;
create.flags = RenderTargetCreateFlags(info.format);
create.usage = RenderTargetUsage(info.format, create.flags, 1);
create.sharingMode = vk::SharingMode::eExclusive;
create.samples = vk::SampleCountFlagBits::e1;
image->memory.property = vk::MemoryPropertyFlagBits::eDeviceLocal;
if (!graphics.CreateImage(create, *image)) {
EXIT("TextureCache: failed to create video-out image, addr=0x%016" PRIx64
" extent=%ux%u format=%d\n",
info.address, info.width, info.height, static_cast<int>(info.format));
}
ImageViewOps::CreateVideoOutViews(*image);
return image;
}
void UploadVideoOut(VideoOutVulkanImage& image, const VideoOutInfo& info,
bool refresh) {
if (info.compression != VideoOutCompression::Uncompressed) {
EXIT("TextureCache: compressed video-out guest upload is unsupported, "
"addr=0x%016" PRIx64 " metadata=0x%016" PRIx64 " dcc=0x%08" PRIx32 "\n",
info.address, info.metadata_address, info.dcc_control);
}
if (refresh) {
Transfer::WaitForGraphicsIdle();
}
image.layout = vk::ImageLayout::eUndefined;
if (!info.bgra16) {
auto layout =
TextureCalcUploadLayout(info.guest_format, info.width, info.height, 1, 1, info.pitch,
info.tile_mode, info.size, false, false, "VideoOut");
auto regions = TextureBuildUploadRegions(layout, info.format, info.width, info.height, 1, 1,
false, false, TextureUploadDestination::MipLevels);
TextureUploadGuestImage(image, reinterpret_cast<const void*>(info.address),
info.size, regions, layout, info.guest_format, info.width,
info.height, 1, 1, "VideoOut", vk::ImageLayout::eGeneral);
return;
}
Transfer::ScratchBuffer scratch(info.size);
TileBlockLayout block {};
EXIT_NOT_IMPLEMENTED(
!TileGetBlockLayout(TileBlockFamily::RenderTarget64KB, info.bytes_per_element, block));
const GpuTileInfo tile_info {block.family,
block.bytes_per_element,
0,
info.size,
0,
info.size,
0,
info.width,
info.height,
1,
info.pitch};
GpuDetile(reinterpret_cast<const void*>(info.address), scratch.Data(), info.size,
info.size, std::span<const GpuTileInfo>(&tile_info, 1));
SwapVideoOutBgra16(scratch.Data(), info.size);
Transfer::UploadImage(image, scratch.Data(), info.size, info.pitch,
vk::ImageLayout::eGeneral);
}
void SwapVideoOutBgra16(void* data, uint64_t size) {
auto* pixels = static_cast<uint16_t*>(data);
for (uint64_t i = 0; i < size / sizeof(uint16_t); i += 4) {
std::swap(pixels[i], pixels[i + 2]);
}
}
GpuTextureVulkanImage* CreateDummyTexture(bool uint_format, bool image_3d,
bool storage) {
auto* image = storage ? static_cast<GpuTextureVulkanImage*>(new StorageTextureVulkanImage)
: new TextureVulkanImage;
auto usage = storage ? TextureFormatUsage::Storage : TextureFormatUsage::Sampled;
auto layout = storage ? vk::ImageLayout::eGeneral : vk::ImageLayout::eShaderReadOnlyOptimal;
auto owner = storage ? "DummyStorageTexture" : "DummySampledTexture";
auto params = MakeDummyTextureParams(uint_format, image_3d, usage, owner);
auto components = TextureCreateImage(*image, params);
static constexpr uint32_t zero = 0;
Transfer::UploadImage(*image, &zero, sizeof(zero), 1, layout);
TextureCreateImageViews(*image, components, params.type, 0, params.base_level,
params.levels, params.depth, params.allow_cube_view, params.view_usage);
return image;
}
void Destroy(VulkanImage& image) {
auto& graphics = GetRenderContext().GetGraphics();
KYTY_PROFILER_BLOCK("TextureCache::DeleteImage");
switch (image.type) {
case VulkanImageType::RenderTexture:
case VulkanImageType::VideoOut:
GetRenderContext().GetFramebufferCache().FreeFramebufferByColor(image);
break;
case VulkanImageType::DepthStencil:
GetRenderContext().GetFramebufferCache().FreeFramebufferByDepth(
static_cast<DepthStencilVulkanImage&>(image));
break;
case VulkanImageType::Texture:
case VulkanImageType::StorageTexture: break;
case VulkanImageType::Unknown: EXIT("cannot destroy an untyped Vulkan image\n");
}
ImageViewOps::DestroyViews(image);
graphics.DeleteImage(image);
switch (image.type) {
case VulkanImageType::Texture: delete &static_cast<TextureVulkanImage&>(image); break;
case VulkanImageType::StorageTexture:
delete &static_cast<StorageTextureVulkanImage&>(image);
break;
case VulkanImageType::RenderTexture:
delete &static_cast<RenderTextureVulkanImage&>(image);
break;
case VulkanImageType::DepthStencil:
delete &static_cast<DepthStencilVulkanImage&>(image);
break;
case VulkanImageType::VideoOut: delete &static_cast<VideoOutVulkanImage&>(image); break;
case VulkanImageType::Unknown: EXIT("cannot delete an untyped Vulkan image\n");
}
}
} // namespace ImageOps
} // namespace Libs::Graphics
-168
View File
@@ -1,168 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
#include "common/assert.h"
#include "graphics/host_gpu/renderer/imageInfo.h"
#include <algorithm>
#include <cstddef>
#include <span>
namespace Libs::Graphics {
struct DepthStencilVulkanImage;
struct GpuTextureVulkanImage;
struct GraphicContext;
struct RenderTextureVulkanImage;
struct VideoOutVulkanImage;
struct VulkanImage;
struct Image final: ImageInfo {
Image& operator=(const ImageInfo& value) {
if (IsCpuDirty()) {
EXIT("dirty sampled image cannot be reassigned\n");
}
static_cast<ImageInfo&>(*this) = value;
m_track_begin = address;
m_track_end = address + size;
m_maybe_cpu_hash_valid = false;
return *this;
}
void InvalidateCpuWrite(uint64_t vaddr, uint64_t size) {
if (ImageRangeOverlaps(address, this->size, vaddr, size)) {
m_cpu_dirty = true;
m_maybe_cpu_dirty = false;
m_maybe_cpu_hash_valid = false;
m_track_begin = m_track_end;
} else if (ImagePageRangesOverlap(address, this->size, vaddr, size)) {
constexpr uint64_t page_mask = 4096 - 1;
if (vaddr + size <= address) {
const auto next_page = (address + page_mask) & ~page_mask;
m_track_begin = std::min(m_track_end, std::max(m_track_begin, next_page));
} else if (vaddr >= address + this->size) {
const auto page = (address + this->size) & ~page_mask;
m_track_end = std::max(m_track_begin, std::min(m_track_end, page));
}
m_maybe_cpu_dirty = m_track_begin == m_track_end;
}
}
[[nodiscard]] bool IsCpuDirty() const { return m_cpu_dirty || m_maybe_cpu_dirty; }
[[nodiscard]] bool IsDefinitelyCpuDirty() const { return m_cpu_dirty; }
[[nodiscard]] bool IsMaybeCpuDirty() const { return m_maybe_cpu_dirty; }
[[nodiscard]] bool NeedsMaybeCpuHash() const {
return m_maybe_cpu_dirty && !m_maybe_cpu_hash_valid;
}
[[nodiscard]] bool IsCpuTrackingComplete() const {
return m_track_begin == address && m_track_end == address + size;
}
void SetMaybeCpuHash(uint64_t hash) {
if (!NeedsMaybeCpuHash()) {
EXIT("sampled image cannot initialize maybe-dirty hash\n");
}
m_maybe_cpu_hash = hash;
m_maybe_cpu_hash_valid = true;
}
[[nodiscard]] bool ResolveMaybeCpuHash(uint64_t hash) {
if (!m_maybe_cpu_dirty || !m_maybe_cpu_hash_valid || m_cpu_dirty) {
EXIT("sampled image cannot resolve maybe-dirty hash\n");
}
m_maybe_cpu_dirty = false;
m_maybe_cpu_hash_valid = false;
m_cpu_dirty = hash != m_maybe_cpu_hash;
if (!m_cpu_dirty) {
m_track_begin = address;
m_track_end = address + size;
}
return m_cpu_dirty;
}
void RefreshComplete() {
if (!IsCpuDirty()) {
EXIT("clean sampled image cannot complete a refresh\n");
}
m_cpu_dirty = false;
m_maybe_cpu_dirty = false;
m_maybe_cpu_hash_valid = false;
m_track_begin = address;
m_track_end = address + size;
}
private:
bool m_cpu_dirty = false;
bool m_maybe_cpu_dirty = false;
bool m_maybe_cpu_hash_valid = false;
uint64_t m_track_begin = 0;
uint64_t m_track_end = 0;
uint64_t m_maybe_cpu_hash = 0;
};
namespace ImageOps {
[[nodiscard]] GpuTextureVulkanImage* CreateTexture(const ImageInfo& info,
bool storage, vk::ComponentMapping& components);
void CreateTextureViews(GpuTextureVulkanImage& image,
const ImageInfo& info, bool storage, vk::ComponentMapping components);
[[nodiscard]] RenderTextureVulkanImage* CreateRenderTarget(
const RenderTargetInfo& info);
[[nodiscard]] uint32_t RenderTargetTransferFormat(uint32_t bytes_per_element);
void UploadRenderTargetLayers(RenderTextureVulkanImage& image,
const RenderTargetInfo& info, uint32_t base_layer,
uint32_t layer_count, bool refresh);
void UploadRenderTarget(RenderTextureVulkanImage& image,
const RenderTargetInfo& info, bool refresh);
[[nodiscard]] DepthStencilVulkanImage* CreateDepthTarget(
const DepthTargetInfo& info);
void ValidateVideoOut(const VideoOutInfo& info);
[[nodiscard]] VideoOutVulkanImage* CreateVideoOut(
const VideoOutInfo& info);
void SwapVideoOutBgra16(void* data, uint64_t size);
void UploadVideoOut(VideoOutVulkanImage& image, const VideoOutInfo& info,
bool refresh);
[[nodiscard]] GpuTextureVulkanImage* CreateDummyTexture(bool uint_format,
bool image_3d, bool storage);
void Destroy(VulkanImage& image);
} // namespace ImageOps
struct ImageRetirementRange {
uint64_t address = 0;
uint64_t size = 0;
bool retire = false;
};
struct ImageRetirementConflict {
size_t retired = SIZE_MAX;
size_t retained = SIZE_MAX;
[[nodiscard]] bool Exists() const { return retired != SIZE_MAX; }
};
[[nodiscard]] inline ImageRetirementConflict
FindImageRetirementConflict(std::span<const ImageRetirementRange> ranges) {
for (size_t retired = 0; retired < ranges.size(); retired++) {
if (!ranges[retired].retire) {
continue;
}
for (size_t retained = 0; retained < ranges.size(); retained++) {
if (ranges[retained].retire) {
continue;
}
if (ImageRangeOverlaps(ranges[retired].address, ranges[retired].size,
ranges[retained].address, ranges[retained].size)) {
return {retired, retained};
}
}
}
return {};
}
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
@@ -0,0 +1,231 @@
#include "graphics/host_gpu/renderer/image/blitHelper.h"
#include "common/assert.h"
#include "gpu_blit_shaders/gpu_blit_color_to_ms_depth_spv.h"
#include "gpu_blit_shaders/gpu_blit_fs_triangle_spv.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/commandScheduler.h"
#include "graphics/host_gpu/renderer/image/image.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include <algorithm>
#include <array>
#include <iterator>
namespace Libs::Graphics {
BlitHelper::BlitHelper(GraphicContext& graphics, CommandScheduler& scheduler)
: m_graphics(graphics), m_scheduler(scheduler) {
vk::DescriptorSetLayoutBinding texture_binding {};
texture_binding.binding = 0;
texture_binding.descriptorType = vk::DescriptorType::eSampledImage;
texture_binding.descriptorCount = 1;
texture_binding.stageFlags = vk::ShaderStageFlagBits::eFragment;
vk::DescriptorSetLayoutCreateInfo descriptor_info {};
descriptor_info.sType = vk::StructureType::eDescriptorSetLayoutCreateInfo;
descriptor_info.flags = vk::DescriptorSetLayoutCreateFlagBits::ePushDescriptorKHR;
descriptor_info.bindingCount = 1;
descriptor_info.pBindings = &texture_binding;
RequireVulkanSuccess(m_graphics.device.createDescriptorSetLayout(&descriptor_info, nullptr,
&m_descriptor_layout),
"create BlitHelper descriptor layout");
vk::PipelineLayoutCreateInfo layout_info {};
layout_info.sType = vk::StructureType::ePipelineLayoutCreateInfo;
layout_info.setLayoutCount = 1;
layout_info.pSetLayouts = &m_descriptor_layout;
RequireVulkanSuccess(
m_graphics.device.createPipelineLayout(&layout_info, nullptr, &m_pipeline_layout),
"create BlitHelper pipeline layout");
m_vertex_shader = CreateShader(GPU_BLIT_FS_TRIANGLE_SPV, std::size(GPU_BLIT_FS_TRIANGLE_SPV));
m_fragment_shader =
CreateShader(GPU_BLIT_COLOR_TO_MS_DEPTH_SPV, std::size(GPU_BLIT_COLOR_TO_MS_DEPTH_SPV));
}
BlitHelper::~BlitHelper() {
for (const auto& pipeline: m_pipelines) {
m_graphics.device.destroyPipeline(pipeline.handle, nullptr);
}
if (m_fragment_shader != nullptr) {
m_graphics.device.destroyShaderModule(m_fragment_shader, nullptr);
}
if (m_vertex_shader != nullptr) {
m_graphics.device.destroyShaderModule(m_vertex_shader, nullptr);
}
if (m_pipeline_layout != nullptr) {
m_graphics.device.destroyPipelineLayout(m_pipeline_layout, nullptr);
}
if (m_descriptor_layout != nullptr) {
m_graphics.device.destroyDescriptorSetLayout(m_descriptor_layout, nullptr);
}
}
vk::ShaderModule BlitHelper::CreateShader(const uint32_t* code, size_t words) const {
EXIT_IF(code == nullptr || words == 0);
vk::ShaderModuleCreateInfo create {};
create.sType = vk::StructureType::eShaderModuleCreateInfo;
create.codeSize = words * sizeof(uint32_t);
create.pCode = code;
vk::ShaderModule module = nullptr;
RequireVulkanSuccess(m_graphics.device.createShaderModule(&create, nullptr, &module),
"create BlitHelper shader module");
return module;
}
vk::Pipeline BlitHelper::GetPipeline(PipelineKey key) {
const auto cached = std::ranges::find(m_pipelines, key, &Pipeline::key);
if (cached != m_pipelines.end()) {
return cached->handle;
}
const auto samples = vulkan_sample_count(key.samples);
EXIT_IF(samples == vk::SampleCountFlagBits {} || key.format == vk::Format::eUndefined);
std::array<vk::PipelineShaderStageCreateInfo, 2> stages {};
stages[0].sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stages[0].stage = vk::ShaderStageFlagBits::eVertex;
stages[0].module = m_vertex_shader;
stages[0].pName = "main";
stages[1].sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stages[1].stage = vk::ShaderStageFlagBits::eFragment;
stages[1].module = m_fragment_shader;
stages[1].pName = "main";
vk::PipelineVertexInputStateCreateInfo vertex_input {};
vertex_input.sType = vk::StructureType::ePipelineVertexInputStateCreateInfo;
vk::PipelineInputAssemblyStateCreateInfo input_assembly {};
input_assembly.sType = vk::StructureType::ePipelineInputAssemblyStateCreateInfo;
input_assembly.topology = vk::PrimitiveTopology::eTriangleList;
vk::PipelineViewportStateCreateInfo viewport {};
viewport.sType = vk::StructureType::ePipelineViewportStateCreateInfo;
viewport.viewportCount = 1;
viewport.scissorCount = 1;
vk::PipelineRasterizationStateCreateInfo rasterization {};
rasterization.sType = vk::StructureType::ePipelineRasterizationStateCreateInfo;
rasterization.polygonMode = vk::PolygonMode::eFill;
rasterization.cullMode = vk::CullModeFlagBits::eNone;
rasterization.lineWidth = 1.0f;
vk::PipelineMultisampleStateCreateInfo multisample {};
multisample.sType = vk::StructureType::ePipelineMultisampleStateCreateInfo;
multisample.rasterizationSamples = samples;
vk::PipelineDepthStencilStateCreateInfo depth {};
depth.sType = vk::StructureType::ePipelineDepthStencilStateCreateInfo;
depth.depthTestEnable = VK_TRUE;
depth.depthWriteEnable = VK_TRUE;
depth.depthCompareOp = vk::CompareOp::eAlways;
vk::PipelineColorBlendStateCreateInfo color_blend {};
color_blend.sType = vk::StructureType::ePipelineColorBlendStateCreateInfo;
const std::array dynamic_states {vk::DynamicState::eViewport, vk::DynamicState::eScissor};
vk::PipelineDynamicStateCreateInfo dynamic {};
dynamic.sType = vk::StructureType::ePipelineDynamicStateCreateInfo;
dynamic.dynamicStateCount = static_cast<uint32_t>(dynamic_states.size());
dynamic.pDynamicStates = dynamic_states.data();
vk::PipelineRenderingCreateInfo rendering {};
rendering.sType = vk::StructureType::ePipelineRenderingCreateInfo;
rendering.depthAttachmentFormat = key.format;
vk::GraphicsPipelineCreateInfo create {};
create.sType = vk::StructureType::eGraphicsPipelineCreateInfo;
create.pNext = &rendering;
create.stageCount = static_cast<uint32_t>(stages.size());
create.pStages = stages.data();
create.pVertexInputState = &vertex_input;
create.pInputAssemblyState = &input_assembly;
create.pViewportState = &viewport;
create.pRasterizationState = &rasterization;
create.pMultisampleState = &multisample;
create.pDepthStencilState = &depth;
create.pColorBlendState = &color_blend;
create.pDynamicState = &dynamic;
create.layout = m_pipeline_layout;
vk::Pipeline pipeline = nullptr;
RequireVulkanSuccess(
m_graphics.device.createGraphicsPipelines(nullptr, 1, &create, nullptr, &pipeline),
"create color-to-MS-depth pipeline");
m_pipelines.push_back({key, pipeline});
return pipeline;
}
void BlitHelper::ReinterpretColorAsMsDepth(Image& source, Image& destination) {
const auto& source_info = source.info;
const auto& destination_info = destination.info;
EXIT_IF(DepthAspectTransferFormat(source_info.pixel_format) != vk::Format::eUndefined ||
DepthAspectTransferFormat(destination_info.pixel_format) == vk::Format::eUndefined ||
source_info.samples != 1 || destination_info.samples <= 1 ||
destination_info.samples > 4 || source.backing.image_type != vk::ImageType::e2D ||
destination.backing.image_type != vk::ImageType::e2D ||
source_info.extent.width != destination_info.extent.width ||
source_info.extent.height != destination_info.extent.height ||
source_info.extent.depth != 1 || destination_info.extent.depth != 1 ||
source.backing.image == nullptr || destination.backing.image == nullptr);
m_scheduler.EndRendering();
ImageViewInfo source_view_info {};
source_view_info.format = source_info.pixel_format;
source_view_info.type = vk::ImageViewType::e2D;
source_view_info.aspect = vk::ImageAspectFlagBits::eColor;
source_view_info.usage = vk::ImageUsageFlagBits::eSampled;
const auto source_view = source.FindView(source_view_info);
ImageViewInfo destination_view_info {};
destination_view_info.format = destination_info.pixel_format;
destination_view_info.type = vk::ImageViewType::e2D;
destination_view_info.aspect = vk::ImageAspectFlagBits::eDepth;
destination_view_info.usage = vk::ImageUsageFlagBits::eDepthStencilAttachment;
const auto destination_view = destination.FindView(destination_view_info);
auto& command_buffer = m_scheduler.Current();
auto command = command_buffer.Handle();
source.Transit(vk::ImageLayout::eShaderReadOnlyOptimal, vk::AccessFlagBits2::eShaderRead, {},
command);
destination.Transit(ColorToMsDepthLayout,
vk::AccessFlagBits2::eDepthStencilAttachmentWrite, {}, command);
vk::RenderingAttachmentInfo depth_attachment {};
depth_attachment.sType = vk::StructureType::eRenderingAttachmentInfo;
depth_attachment.imageView = destination_view;
depth_attachment.imageLayout = ColorToMsDepthLayout;
depth_attachment.loadOp = vk::AttachmentLoadOp::eClear;
depth_attachment.storeOp = vk::AttachmentStoreOp::eStore;
depth_attachment.clearValue.depthStencil = {0.0f, 0};
vk::RenderingInfo rendering {};
rendering.sType = vk::StructureType::eRenderingInfo;
rendering.renderArea.extent = {destination_info.extent.width, destination_info.extent.height};
rendering.layerCount = 1;
rendering.pDepthAttachment = &depth_attachment;
command.beginRendering(&rendering);
vk::DescriptorImageInfo descriptor_image {};
descriptor_image.imageView = source_view;
descriptor_image.imageLayout = vk::ImageLayout::eShaderReadOnlyOptimal;
vk::WriteDescriptorSet descriptor_write {};
descriptor_write.sType = vk::StructureType::eWriteDescriptorSet;
descriptor_write.dstBinding = 0;
descriptor_write.descriptorCount = 1;
descriptor_write.descriptorType = vk::DescriptorType::eSampledImage;
descriptor_write.pImageInfo = &descriptor_image;
command.pushDescriptorSetKHR(vk::PipelineBindPoint::eGraphics, m_pipeline_layout, 0, 1,
&descriptor_write);
command.bindPipeline(vk::PipelineBindPoint::eGraphics,
GetPipeline({destination_info.samples, destination_info.pixel_format}));
const vk::Viewport viewport {0.0f,
0.0f,
static_cast<float>(destination_info.extent.width),
static_cast<float>(destination_info.extent.height),
0.0f,
1.0f};
const vk::Rect2D scissor {{0, 0},
{destination_info.extent.width, destination_info.extent.height}};
command.setViewport(0, 1, &viewport);
command.setScissor(0, 1, &scissor);
command.draw(3, 1, 0, 0);
command.endRendering();
}
} // namespace Libs::Graphics
@@ -0,0 +1,54 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BLITHELPER_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BLITHELPER_H_
#include "common/common.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <compare>
#include <vector>
namespace Libs::Graphics {
class CommandScheduler;
class Image;
struct GraphicContext;
class BlitHelper final {
public:
inline static constexpr auto ColorToMsDepthLayout =
vk::ImageLayout::eDepthStencilAttachmentOptimal;
BlitHelper(GraphicContext& graphics, CommandScheduler& scheduler);
~BlitHelper();
KYTY_CLASS_NO_COPY(BlitHelper);
void ReinterpretColorAsMsDepth(Image& source, Image& destination);
private:
struct PipelineKey {
uint32_t samples = 1;
vk::Format format = vk::Format::eUndefined;
auto operator<=>(const PipelineKey&) const = default;
};
struct Pipeline {
PipelineKey key;
vk::Pipeline handle = nullptr;
};
[[nodiscard]] vk::ShaderModule CreateShader(const uint32_t* code, size_t words) const;
[[nodiscard]] vk::Pipeline GetPipeline(PipelineKey key);
GraphicContext& m_graphics;
CommandScheduler& m_scheduler;
vk::DescriptorSetLayout m_descriptor_layout = nullptr;
vk::PipelineLayout m_pipeline_layout = nullptr;
vk::ShaderModule m_vertex_shader = nullptr;
vk::ShaderModule m_fragment_shader = nullptr;
std::vector<Pipeline> m_pipelines;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_BLITHELPER_H_
@@ -0,0 +1,749 @@
#include "graphics/host_gpu/renderer/image/image.h"
#include "common/assert.h"
#include "common/profiler.h"
#include "graphics/host_gpu/renderer/cache/streamBuffer.h"
#include "graphics/host_gpu/renderer/commandScheduler.h"
#include "graphics/host_gpu/renderer/image/imageView.h"
#include "graphics/host_gpu/renderer/renderTarget.h"
#include "kernel/memory.h"
#include <algorithm>
#include <array>
#include <cstdint>
#include <xxhash.h>
namespace Libs::Graphics {
namespace {
[[nodiscard]] vk::ImageType HostImageType(Prospero::ImageType type) {
switch (type) {
case Prospero::ImageType::kColor1D: return vk::ImageType::e1D;
case Prospero::ImageType::kColor3D: return vk::ImageType::e3D;
case Prospero::ImageType::kColor2D: return vk::ImageType::e2D;
default: EXIT("non-base image type: %u\n", static_cast<uint32_t>(type));
}
}
[[nodiscard]] vk::ImageCreateFlags ImageCreateFlags(const ImageInfo& info) {
vk::ImageCreateFlags flags {};
if (DepthAspectTransferFormat(info.pixel_format) == vk::Format::eUndefined) {
flags |= vk::ImageCreateFlagBits::eMutableFormat;
flags |= vk::ImageCreateFlagBits::eExtendedUsage;
if (Prospero::BlockCompressedBytesPerBlock(info.guest_format) != 0) {
flags |= vk::ImageCreateFlagBits::eBlockTexelViewCompatible;
}
}
if (info.IsVolume()) {
flags |= vk::ImageCreateFlagBits::e2DArrayCompatible;
}
return flags;
}
[[nodiscard]] bool HasFormatFeature(vk::FormatProperties properties,
vk::FormatFeatureFlagBits feature) {
return static_cast<bool>(properties.optimalTilingFeatures & feature);
}
[[nodiscard]] vk::ImageUsageFlags ImageUsageFlags(GraphicContext& graphics, const ImageInfo& info) {
const auto properties = graphics.GetFormatProperties(info.pixel_format);
auto usage = vk::ImageUsageFlagBits::eTransferSrc | vk::ImageUsageFlagBits::eTransferDst;
if (HasFormatFeature(properties, vk::FormatFeatureFlagBits::eSampledImage)) {
usage |= vk::ImageUsageFlagBits::eSampled;
}
if (DepthAspectTransferFormat(info.pixel_format) != vk::Format::eUndefined) {
usage |= vk::ImageUsageFlagBits::eDepthStencilAttachment;
return usage;
}
if (HasFormatFeature(properties, vk::FormatFeatureFlagBits::eColorAttachment)) {
usage |= vk::ImageUsageFlagBits::eColorAttachment;
}
if (info.samples == 1 &&
HasFormatFeature(properties, vk::FormatFeatureFlagBits::eStorageImage)) {
usage |= vk::ImageUsageFlagBits::eStorage;
} else if (info.samples == 1) {
const auto compatible = SrgbStorageViewFormat(info.pixel_format);
if (compatible != vk::Format::eUndefined &&
HasFormatFeature(graphics.GetFormatProperties(compatible),
vk::FormatFeatureFlagBits::eStorageImage)) {
usage |= vk::ImageUsageFlagBits::eStorage;
}
}
return usage;
}
void ValidateRange(GuestRange range, const char* name) {
if ((range.address == 0) != (range.size == 0) ||
(range.address != 0 && (range.address >= TRACKER_ADDRESS_SIZE ||
range.size > TRACKER_ADDRESS_SIZE - range.address))) {
EXIT("invalid %s image range: address=0x%016llx size=0x%016llx\n", name,
static_cast<unsigned long long>(range.address),
static_cast<unsigned long long>(range.size));
}
}
} // namespace
vk::ImageAspectFlags Image::FullAspectMask(vk::Format format) noexcept {
switch (format) {
case vk::Format::eD16Unorm:
case vk::Format::eX8D24UnormPack32:
case vk::Format::eD32Sfloat: return vk::ImageAspectFlagBits::eDepth;
case vk::Format::eS8Uint: return vk::ImageAspectFlagBits::eStencil;
case vk::Format::eD16UnormS8Uint:
case vk::Format::eD24UnormS8Uint:
case vk::Format::eD32SfloatS8Uint:
return vk::ImageAspectFlagBits::eDepth | vk::ImageAspectFlagBits::eStencil;
default: return vk::ImageAspectFlagBits::eColor;
}
}
Image::Barriers Image::GetBarriers(vk::ImageLayout destination_layout,
vk::AccessFlags2 destination_access,
vk::PipelineStageFlags2 destination_stage,
std::optional<ImageSubresourceRange> range) {
auto& state = backing.state;
auto& subresource_states = backing.subresource_states;
const bool partial =
range && (range->base_level != 0 || range->level_count != info.resources.levels ||
range->base_layer != 0 || range->layer_count != info.resources.layers);
const bool has_subresource_states = !subresource_states.empty();
Barriers barriers;
if (partial || has_subresource_states) {
if (!has_subresource_states) {
subresource_states.resize(info.resources.levels * info.resources.layers, state);
}
const uint32_t base_level = partial ? range->base_level : 0;
const uint32_t level_count = partial ? range->level_count : info.resources.levels;
const uint32_t base_layer = partial ? range->base_layer : 0;
const uint32_t layer_count = partial ? range->layer_count : info.resources.layers;
for (uint32_t level = base_level; level < base_level + level_count; level++) {
for (uint32_t layer = base_layer; layer < base_layer + layer_count; layer++) {
const auto index = level * info.resources.layers + layer;
EXIT_IF(index >= subresource_states.size());
auto& subresource_state = subresource_states[index];
constexpr auto write_access = vk::AccessFlagBits2::eTransferWrite |
vk::AccessFlagBits2::eShaderWrite |
vk::AccessFlagBits2::eMemoryWrite;
const bool repeated_write =
static_cast<bool>(subresource_state.access_mask & write_access);
if (subresource_state.layout != destination_layout ||
subresource_state.access_mask != destination_access || repeated_write) {
vk::ImageMemoryBarrier2 barrier {};
barrier.srcStageMask = subresource_state.pl_stage;
barrier.srcAccessMask = subresource_state.access_mask;
barrier.dstStageMask = destination_stage;
barrier.dstAccessMask = destination_access;
barrier.oldLayout = subresource_state.layout;
barrier.newLayout = destination_layout;
barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.image = backing.image;
barrier.subresourceRange.aspectMask = FullAspectMask(backing.format);
barrier.subresourceRange.baseMipLevel = level;
barrier.subresourceRange.levelCount = 1;
barrier.subresourceRange.baseArrayLayer = layer;
barrier.subresourceRange.layerCount = 1;
barriers.push_back(barrier);
subresource_state = {destination_stage, destination_access, destination_layout};
}
}
}
if (!partial) {
subresource_states.clear();
}
} else {
constexpr auto write_access = vk::AccessFlagBits2::eTransferWrite |
vk::AccessFlagBits2::eShaderWrite |
vk::AccessFlagBits2::eMemoryWrite;
const bool repeated_write = static_cast<bool>(state.access_mask & write_access);
if (state.layout == destination_layout && state.access_mask == destination_access &&
!repeated_write) {
return {};
}
vk::ImageMemoryBarrier2 barrier {};
barrier.srcStageMask = state.pl_stage;
barrier.srcAccessMask = state.access_mask;
barrier.dstStageMask = destination_stage;
barrier.dstAccessMask = destination_access;
barrier.oldLayout = state.layout;
barrier.newLayout = destination_layout;
barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.image = backing.image;
barrier.subresourceRange.aspectMask = FullAspectMask(backing.format);
barrier.subresourceRange.baseMipLevel = 0;
barrier.subresourceRange.levelCount = VK_REMAINING_MIP_LEVELS;
barrier.subresourceRange.baseArrayLayer = 0;
barrier.subresourceRange.layerCount = VK_REMAINING_ARRAY_LAYERS;
barriers.push_back(barrier);
}
state = {destination_stage, destination_access, destination_layout};
return barriers;
}
void Image::Transit(vk::ImageLayout destination_layout, vk::AccessFlags2 destination_access,
std::optional<ImageSubresourceRange> range, vk::CommandBuffer command_buffer) {
const auto transfer_access =
vk::AccessFlagBits2::eTransferRead | vk::AccessFlagBits2::eTransferWrite;
vk::PipelineStageFlags2 destination_stage {};
if (static_cast<bool>(destination_access & transfer_access)) {
destination_stage |= vk::PipelineStageFlagBits2::eTransfer;
}
if (!destination_access ||
static_cast<bool>(destination_access & ~vk::AccessFlags2 {transfer_access})) {
destination_stage |=
vk::PipelineStageFlagBits2::eAllGraphics | vk::PipelineStageFlagBits2::eComputeShader;
}
const auto barriers =
GetBarriers(destination_layout, destination_access, destination_stage, range);
if (barriers.empty()) {
return;
}
if (m_scheduler != nullptr) {
m_scheduler->EndRendering();
}
vk::DependencyInfo dependency {};
dependency.imageMemoryBarrierCount = static_cast<uint32_t>(barriers.size());
dependency.pImageMemoryBarriers = barriers.data();
command_buffer.pipelineBarrier2(dependency);
}
void Image::Upload(std::span<const vk::BufferImageCopy> copies, vk::Buffer buffer, uint64_t offset,
uint64_t size) {
EXIT_IF(m_scheduler == nullptr || copies.empty() || buffer == nullptr || size == 0);
m_scheduler->EndRendering();
vk::BufferMemoryBarrier2 buffer_barrier {};
buffer_barrier.srcStageMask = vk::PipelineStageFlagBits2::eAllCommands;
buffer_barrier.srcAccessMask = vk::AccessFlagBits2::eMemoryWrite;
buffer_barrier.dstStageMask = vk::PipelineStageFlagBits2::eTransfer;
buffer_barrier.dstAccessMask = vk::AccessFlagBits2::eTransferRead;
buffer_barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
buffer_barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
buffer_barrier.buffer = buffer;
buffer_barrier.offset = offset;
buffer_barrier.size = size;
const auto image_barriers =
GetBarriers(vk::ImageLayout::eTransferDstOptimal, vk::AccessFlagBits2::eTransferWrite,
vk::PipelineStageFlagBits2::eCopy, {});
vk::DependencyInfo dependency {};
dependency.dependencyFlags = vk::DependencyFlagBits::eByRegion;
dependency.bufferMemoryBarrierCount = 1;
dependency.pBufferMemoryBarriers = &buffer_barrier;
dependency.imageMemoryBarrierCount = static_cast<uint32_t>(image_barriers.size());
dependency.pImageMemoryBarriers = image_barriers.data();
auto command = m_scheduler->Current().Handle();
command.pipelineBarrier2(dependency);
command.copyBufferToImage(buffer, backing.image, vk::ImageLayout::eTransferDstOptimal,
static_cast<uint32_t>(copies.size()), copies.data());
buffer_barrier.srcStageMask = vk::PipelineStageFlagBits2::eTransfer;
buffer_barrier.srcAccessMask = vk::AccessFlagBits2::eTransferRead;
buffer_barrier.dstStageMask = vk::PipelineStageFlagBits2::eAllCommands;
buffer_barrier.dstAccessMask =
vk::AccessFlagBits2::eMemoryRead | vk::AccessFlagBits2::eMemoryWrite;
dependency.imageMemoryBarrierCount = 0;
dependency.pImageMemoryBarriers = nullptr;
command.pipelineBarrier2(dependency);
Transit(vk::ImageLayout::eGeneral,
vk::AccessFlagBits2::eShaderRead | vk::AccessFlagBits2::eTransferRead, {}, command);
}
void Image::Download(std::span<const vk::BufferImageCopy> copies, vk::Buffer buffer,
uint64_t offset, uint64_t size) {
EXIT_IF(m_scheduler == nullptr || copies.empty() || buffer == nullptr || size == 0);
m_scheduler->EndRendering();
vk::BufferMemoryBarrier2 buffer_barrier {};
buffer_barrier.srcStageMask = vk::PipelineStageFlagBits2::eAllCommands;
buffer_barrier.srcAccessMask =
vk::AccessFlagBits2::eMemoryRead | vk::AccessFlagBits2::eMemoryWrite;
buffer_barrier.dstStageMask = vk::PipelineStageFlagBits2::eCopy;
buffer_barrier.dstAccessMask = vk::AccessFlagBits2::eTransferWrite;
buffer_barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
buffer_barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
buffer_barrier.buffer = buffer;
buffer_barrier.offset = offset;
buffer_barrier.size = size;
const auto image_barriers =
GetBarriers(vk::ImageLayout::eTransferSrcOptimal, vk::AccessFlagBits2::eTransferRead,
vk::PipelineStageFlagBits2::eCopy, {});
vk::DependencyInfo dependency {};
dependency.dependencyFlags = vk::DependencyFlagBits::eByRegion;
dependency.bufferMemoryBarrierCount = 1;
dependency.pBufferMemoryBarriers = &buffer_barrier;
dependency.imageMemoryBarrierCount = static_cast<uint32_t>(image_barriers.size());
dependency.pImageMemoryBarriers = image_barriers.data();
auto command = m_scheduler->Current().Handle();
command.pipelineBarrier2(dependency);
command.copyImageToBuffer(backing.image, vk::ImageLayout::eTransferSrcOptimal, buffer,
static_cast<uint32_t>(copies.size()), copies.data());
buffer_barrier.srcStageMask = vk::PipelineStageFlagBits2::eCopy;
buffer_barrier.srcAccessMask = vk::AccessFlagBits2::eTransferWrite;
buffer_barrier.dstStageMask = vk::PipelineStageFlagBits2::eAllCommands;
buffer_barrier.dstAccessMask =
vk::AccessFlagBits2::eMemoryRead | vk::AccessFlagBits2::eMemoryWrite;
dependency.imageMemoryBarrierCount = 0;
dependency.pImageMemoryBarriers = nullptr;
command.pipelineBarrier2(dependency);
}
std::pair<uint32_t, uint32_t> Image::SanitizeCopyLayers(const Image& source,
const Image& destination, uint32_t depth) {
const auto source_type = source.backing.image_type;
const auto destination_type = destination.backing.image_type;
uint32_t source_layers = source.backing.layers;
uint32_t destination_layers = destination.backing.layers;
if (source_type == vk::ImageType::e3D) {
source_layers = 1;
}
if (destination_type == vk::ImageType::e3D) {
destination_layers = 1;
}
if (source_type == destination_type) {
source_layers = destination_layers = std::min(source_layers, destination_layers);
} else if (source_type == vk::ImageType::e2D && destination_type == vk::ImageType::e3D) {
source_layers = depth;
} else if (source_type == vk::ImageType::e3D && destination_type == vk::ImageType::e2D) {
destination_layers = depth;
}
return {source_layers, destination_layers};
}
void Image::CopyImage(Image& source) {
EXIT_IF(m_scheduler == nullptr || source.backing.samples != backing.samples);
m_scheduler->EndRendering();
const uint32_t levels = std::min(source.backing.mip_levels, backing.mip_levels);
const uint32_t base_depth = backing.image_type == vk::ImageType::e3D
? backing.extent.depth
: source.backing.extent.depth;
const auto source_aspect =
FullAspectMask(source.backing.format) & ~vk::ImageAspectFlagBits::eStencil;
const auto destination_aspect =
FullAspectMask(backing.format) & ~vk::ImageAspectFlagBits::eStencil;
std::vector<vk::ImageCopy> copies;
copies.reserve(levels);
for (uint32_t level = 0; level < levels; level++) {
const auto width = std::max(source.backing.extent.width >> level, 1u);
const auto height = std::max(source.backing.extent.height >> level, 1u);
const auto depth = std::max(base_depth >> level, 1u);
const auto [source_layers, destination_layers] = SanitizeCopyLayers(source, *this, depth);
vk::ImageCopy copy {};
copy.srcSubresource = {source_aspect, level, 0, 1};
copy.dstSubresource = {destination_aspect, level, 0, 1};
if (source.backing.image_type == backing.image_type) {
if (source.backing.image_type == vk::ImageType::e3D) {
copy.extent = {width, height, depth};
} else {
copy.srcSubresource.layerCount = std::min(source_layers, destination_layers);
copy.dstSubresource.layerCount = copy.srcSubresource.layerCount;
copy.extent = {width, height, 1};
}
} else if (source.backing.image_type == vk::ImageType::e2D) {
copy.srcSubresource.layerCount = source_layers;
copy.extent = {width, height, source_layers};
} else {
copy.dstSubresource.layerCount = destination_layers;
copy.extent = {width, height, destination_layers};
}
copies.push_back(copy);
}
if (copies.empty()) {
return;
}
auto command = m_scheduler->Current().Handle();
source.Transit(vk::ImageLayout::eTransferSrcOptimal, vk::AccessFlagBits2::eTransferRead, {},
command);
Transit(vk::ImageLayout::eTransferDstOptimal, vk::AccessFlagBits2::eTransferWrite, {}, command);
command.copyImage(source.backing.image, vk::ImageLayout::eTransferSrcOptimal, backing.image,
vk::ImageLayout::eTransferDstOptimal, static_cast<uint32_t>(copies.size()),
copies.data());
Transit(vk::ImageLayout::eGeneral,
vk::AccessFlagBits2::eShaderRead | vk::AccessFlagBits2::eTransferRead, {}, command);
}
void Image::Resolve(Image& source, const ImageSubresourceRange& source_range,
const ImageSubresourceRange& destination_range) {
EXIT_IF(m_scheduler == nullptr || backing.samples != 1 ||
source.backing.image_type != vk::ImageType::e2D ||
backing.image_type != vk::ImageType::e2D || source_range.level_count != 1 ||
destination_range.level_count != 1 ||
source_range.base_level >= source.backing.mip_levels ||
destination_range.base_level >= backing.mip_levels ||
source_range.base_layer >= source.backing.layers ||
destination_range.base_layer >= backing.layers);
const auto layers = std::min({source_range.layer_count, destination_range.layer_count,
source.backing.layers - source_range.base_layer,
backing.layers - destination_range.base_layer});
const auto source_width = std::max(source.backing.extent.width >> source_range.base_level, 1u);
const auto source_height =
std::max(source.backing.extent.height >> source_range.base_level, 1u);
const auto destination_width =
std::max(backing.extent.width >> destination_range.base_level, 1u);
const auto destination_height =
std::max(backing.extent.height >> destination_range.base_level, 1u);
const bool copy = source.backing.samples == 1;
EXIT_IF(layers == 0 || info.extent.width > source_width || info.extent.height > source_height ||
info.extent.width > destination_width || info.extent.height > destination_height ||
(copy ? !ImageViewOps::FormatsCompatible(source.backing.format, backing.format)
: source.backing.format != backing.format));
auto resolved_source_range = source_range;
auto resolved_destination_range = destination_range;
resolved_source_range.layer_count = layers;
resolved_destination_range.layer_count = layers;
const vk::Extent3D resolve_extent {info.extent.width, info.extent.height, 1};
m_scheduler->EndRendering();
auto command = m_scheduler->Current().Handle();
source.Transit(vk::ImageLayout::eTransferSrcOptimal, vk::AccessFlagBits2::eTransferRead,
resolved_source_range, command);
Transit(vk::ImageLayout::eTransferDstOptimal, vk::AccessFlagBits2::eTransferWrite,
resolved_destination_range, command);
if (copy) {
vk::ImageCopy region {};
region.srcSubresource = {vk::ImageAspectFlagBits::eColor, resolved_source_range.base_level,
resolved_source_range.base_layer, layers};
region.dstSubresource = {vk::ImageAspectFlagBits::eColor,
resolved_destination_range.base_level,
resolved_destination_range.base_layer, layers};
region.extent = resolve_extent;
command.copyImage(source.backing.image, vk::ImageLayout::eTransferSrcOptimal, backing.image,
vk::ImageLayout::eTransferDstOptimal, region);
} else {
vk::ImageResolve region {};
region.srcSubresource = {vk::ImageAspectFlagBits::eColor, resolved_source_range.base_level,
resolved_source_range.base_layer, layers};
region.dstSubresource = {vk::ImageAspectFlagBits::eColor,
resolved_destination_range.base_level,
resolved_destination_range.base_layer, layers};
region.extent = resolve_extent;
command.resolveImage(source.backing.image, vk::ImageLayout::eTransferSrcOptimal,
backing.image, vk::ImageLayout::eTransferDstOptimal, region);
}
}
uint32_t Image::CopyRows(uint64_t row_size, uint32_t rows, uint64_t capacity) noexcept {
if (row_size == 0 || rows == 0 || row_size > capacity) {
return 0;
}
return static_cast<uint32_t>(std::min<uint64_t>(rows, capacity / row_size));
}
void Image::CopyImageWithBuffer(Image& source, Buffer& buffer) {
EXIT_IF(m_scheduler == nullptr || buffer.Handle() == nullptr || source.backing.samples != 1 ||
backing.samples != 1);
m_scheduler->EndRendering();
const uint32_t levels = std::min(source.backing.mip_levels, backing.mip_levels);
const auto source_aspect =
FullAspectMask(source.backing.format) & ~vk::ImageAspectFlagBits::eStencil;
const auto destination_aspect =
FullAspectMask(backing.format) & ~vk::ImageAspectFlagBits::eStencil;
const auto source_bytes = DepthAspectTransferBytes(source.backing.format) != 0
? DepthAspectTransferBytes(source.backing.format)
: source.info.bytes_per_block;
const auto destination_bytes = DepthAspectTransferBytes(backing.format) != 0
? DepthAspectTransferBytes(backing.format)
: info.bytes_per_block;
const uint32_t source_block = source.info.IsBlock() ? 4u : 1u;
const uint32_t destination_block = info.IsBlock() ? 4u : 1u;
EXIT_IF(levels == 0 || source_bytes == 0 || source_bytes != destination_bytes ||
source_block != destination_block);
vk::BufferMemoryBarrier2 barrier {};
barrier.srcStageMask = vk::PipelineStageFlagBits2::eTransfer;
barrier.srcAccessMask = vk::AccessFlagBits2::eTransferRead;
barrier.dstStageMask = vk::PipelineStageFlagBits2::eTransfer;
barrier.dstAccessMask = vk::AccessFlagBits2::eTransferWrite;
barrier.srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barrier.buffer = buffer.Handle();
barrier.offset = 0;
vk::DependencyInfo dependency {};
dependency.dependencyFlags = vk::DependencyFlagBits::eByRegion;
dependency.bufferMemoryBarrierCount = 1;
dependency.pBufferMemoryBarriers = &barrier;
auto command = m_scheduler->Current().Handle();
source.Transit(vk::ImageLayout::eTransferSrcOptimal, vk::AccessFlagBits2::eTransferRead, {},
command);
Transit(vk::ImageLayout::eTransferDstOptimal, vk::AccessFlagBits2::eTransferWrite, {}, command);
for (uint32_t level = 0; level < levels; level++) {
const auto width = std::max(source.backing.extent.width >> level, 1u);
const auto height = std::max(source.backing.extent.height >> level, 1u);
const auto source_depth = source.backing.image_type == vk::ImageType::e3D
? std::max(source.backing.extent.depth >> level, 1u)
: source.backing.layers;
const auto destination_depth = backing.image_type == vk::ImageType::e3D
? std::max(backing.extent.depth >> level, 1u)
: backing.layers;
const auto slices = std::min(source_depth, destination_depth);
const auto block_rows = (height + source_block - 1) / source_block;
const auto row_size =
static_cast<uint64_t>((width + source_block - 1) / source_block) * source_bytes;
const auto rows_per_copy = CopyRows(row_size, block_rows, buffer.Size());
EXIT_IF(slices == 0 || rows_per_copy == 0);
for (uint32_t slice = 0; slice < slices; slice++) {
for (uint32_t block_row = 0; block_row < block_rows; block_row += rows_per_copy) {
const auto copy_rows = std::min(rows_per_copy, block_rows - block_row);
const auto y = block_row * source_block;
const auto copy_height = std::min(copy_rows * source_block, height - y);
const auto copy_size = row_size * copy_rows;
vk::BufferImageCopy source_copy {};
source_copy.imageSubresource = {
source_aspect, level,
source.backing.image_type == vk::ImageType::e3D ? 0u : slice, 1};
source_copy.imageOffset = {0, static_cast<int32_t>(y),
source.backing.image_type == vk::ImageType::e3D
? static_cast<int32_t>(slice)
: 0};
source_copy.imageExtent = {width, copy_height, 1};
auto destination_copy = source_copy;
destination_copy.imageSubresource = {
destination_aspect, level,
backing.image_type == vk::ImageType::e3D ? 0u : slice, 1};
destination_copy.imageOffset.z =
backing.image_type == vk::ImageType::e3D ? static_cast<int32_t>(slice) : 0;
barrier.size = copy_size;
barrier.srcAccessMask = vk::AccessFlagBits2::eTransferRead;
barrier.dstAccessMask = vk::AccessFlagBits2::eTransferWrite;
command.pipelineBarrier2(dependency);
command.copyImageToBuffer(source.backing.image,
vk::ImageLayout::eTransferSrcOptimal, buffer.Handle(),
source_copy);
barrier.srcAccessMask = vk::AccessFlagBits2::eTransferWrite;
barrier.dstAccessMask = vk::AccessFlagBits2::eTransferRead;
command.pipelineBarrier2(dependency);
command.copyBufferToImage(buffer.Handle(), backing.image,
vk::ImageLayout::eTransferDstOptimal, destination_copy);
}
}
}
Transit(vk::ImageLayout::eGeneral,
vk::AccessFlagBits2::eShaderRead | vk::AccessFlagBits2::eTransferRead, {}, command);
}
void Image::CopyMip(Image& source, uint32_t mip, uint32_t layer) {
EXIT_IF(m_scheduler == nullptr || source.backing.samples != backing.samples ||
mip >= backing.mip_levels || layer >= backing.layers);
m_scheduler->EndRendering();
const auto width = std::max(backing.extent.width >> mip, 1u);
const auto height = std::max(backing.extent.height >> mip, 1u);
const auto depth = std::max(backing.extent.depth >> mip, 1u);
EXIT_IF(width != source.backing.extent.width || height != source.backing.extent.height);
const auto [source_layers, destination_layers] = SanitizeCopyLayers(source, *this, depth);
const auto aspects = FullAspectMask(source.backing.format);
EXIT_IF(aspects != FullAspectMask(backing.format));
std::array<vk::ImageCopy, 2> copies {};
uint32_t copy_count = 0;
for (const auto aspect: {vk::ImageAspectFlagBits::eColor, vk::ImageAspectFlagBits::eDepth,
vk::ImageAspectFlagBits::eStencil}) {
if (!static_cast<bool>(aspects & aspect)) {
continue;
}
auto& copy = copies[copy_count++];
copy.srcSubresource = {aspect, 0, 0, source_layers};
copy.dstSubresource = {aspect, mip, layer, destination_layers};
copy.extent = {width, height, depth};
}
auto command = m_scheduler->Current().Handle();
Transit(vk::ImageLayout::eTransferDstOptimal, vk::AccessFlagBits2::eTransferWrite, {}, command);
source.Transit(vk::ImageLayout::eTransferSrcOptimal, vk::AccessFlagBits2::eTransferRead, {},
command);
command.copyImage(source.backing.image, vk::ImageLayout::eTransferSrcOptimal, backing.image,
vk::ImageLayout::eTransferDstOptimal, copy_count, copies.data());
Transit(vk::ImageLayout::eGeneral,
vk::AccessFlagBits2::eShaderRead | vk::AccessFlagBits2::eTransferRead, {}, command);
}
namespace ImageOps {
void Validate(const ImageInfo& info) {
ValidateRange(info.data, "data");
ValidateRange(info.stencil, "stencil");
if (info.pixel_format == vk::Format::eUndefined) {
const bool metadata_empty =
info.metadata.range.address == 0 && info.metadata.range.size == 0 &&
info.metadata.kind == ImageMetadataKind::None && info.metadata.control == 0 &&
info.metadata.compression == VideoOutCompression::Uncompressed &&
!info.metadata.stencil_compressed;
if (info.data.Empty() || info.HasStencil() || !metadata_empty || info.extent.width == 0 ||
info.extent.height == 0 || info.extent.depth == 0 || info.resources.levels != 1 ||
info.resources.layers != 1 || info.samples != 1 || info.pitch != 0 ||
info.bytes_per_block != 0) {
EXIT("invalid stencil association image\n");
}
return;
}
if (info.extent.width == 0 || info.extent.height == 0 || info.extent.depth == 0 ||
info.resources.levels == 0 || info.resources.levels > info.mip_layout.size() ||
info.resources.layers == 0 || info.samples == 0 ||
vulkan_sample_count(info.samples) == vk::SampleCountFlagBits {} ||
info.bytes_per_block == 0 || (info.data.address != 0 && info.pitch == 0)) {
EXIT("invalid image geometry or format\n");
}
switch (info.type) {
case Prospero::ImageType::kColor1D:
if (info.extent.height != 1 || info.extent.depth != 1) {
EXIT("invalid 1D image shape\n");
}
break;
case Prospero::ImageType::kColor3D:
if (info.resources.layers != 1) {
EXIT("3D images cannot have array layers\n");
}
break;
case Prospero::ImageType::kColor2D:
if (info.extent.depth != 1) {
EXIT("invalid 2D image shape\n");
}
break;
default: EXIT("non-base image type: %u\n", static_cast<uint32_t>(info.type));
}
if (info.samples > 1 && info.resources.levels != 1) {
EXIT("multisampled images cannot have mip levels\n");
}
if (info.metadata.stencil_compressed && !info.HasStencil()) {
EXIT("compressed stencil metadata requires a stencil plane\n");
}
switch (info.metadata.kind) {
case ImageMetadataKind::None:
if (info.metadata.range.address != 0 || info.metadata.range.size != 0 ||
info.metadata.control != 0 ||
info.metadata.compression != VideoOutCompression::Uncompressed ||
info.metadata.stencil_compressed) {
EXIT("metadata-free image has metadata state\n");
}
break;
case ImageMetadataKind::Htile:
ValidateRange(info.metadata.range, "HTILE");
if (info.metadata.range.Empty() ||
info.metadata.compression != VideoOutCompression::Uncompressed) {
EXIT("invalid HTILE metadata\n");
}
break;
case ImageMetadataKind::Dcc:
if (info.metadata.range.address == 0 ||
info.metadata.range.address >= TRACKER_ADDRESS_SIZE ||
(info.metadata.range.size != 0 &&
info.metadata.range.size > TRACKER_ADDRESS_SIZE - info.metadata.range.address) ||
info.metadata.compression == VideoOutCompression::Unsupported) {
EXIT("invalid DCC metadata\n");
}
break;
}
}
uint32_t RenderTargetTransferFormat(uint32_t bytes_per_element) {
switch (bytes_per_element) {
case 1: return Prospero::GpuEnumValue(Prospero::BufferFormat::k8UNorm);
case 2: return Prospero::GpuEnumValue(Prospero::BufferFormat::k16UNorm);
case 4: return Prospero::GpuEnumValue(Prospero::BufferFormat::k32Float);
case 8: return Prospero::GpuEnumValue(Prospero::BufferFormat::k16_16_16_16Float);
case 16: return Prospero::GpuEnumValue(Prospero::BufferFormat::k32_32_32_32Float);
default: EXIT("unsupported render-target element size: %u\n", bytes_per_element);
}
}
} // namespace ImageOps
Image::Image(GraphicContext& graphics, CommandScheduler& scheduler, const ImageInfo& image_info)
: info(image_info), m_graphics(&graphics), m_scheduler(&scheduler) {
KYTY_PROFILER_FUNCTION();
ImageOps::Validate(info);
m_cpu_dirty = !info.data.Empty();
if (info.pixel_format == vk::Format::eUndefined) {
return;
}
backing.format = info.pixel_format;
backing.image_type = HostImageType(info.type);
backing.extent = info.extent;
backing.guest_pitch = info.pitch;
backing.layers = info.IsVolume() ? 1u : info.resources.layers;
backing.mip_levels = info.resources.levels;
backing.samples = info.samples;
backing.flags = ImageCreateFlags(info);
backing.usage = ImageUsageFlags(graphics, info);
vk::ImageCreateInfo create {};
create.sType = vk::StructureType::eImageCreateInfo;
create.flags = backing.flags;
create.imageType = backing.image_type;
create.extent = backing.extent;
create.mipLevels = backing.mip_levels;
create.arrayLayers = backing.layers;
create.format = backing.format;
create.tiling = vk::ImageTiling::eOptimal;
create.initialLayout = backing.state.layout;
create.usage = backing.usage;
create.sharingMode = vk::SharingMode::eExclusive;
create.samples = vulkan_sample_count(backing.samples);
vk::ImageFormatProperties properties {};
if (graphics.GetImageFormatProperties(create.format, create.imageType, create.tiling,
create.usage, create.flags,
&properties) != vk::Result::eSuccess ||
!static_cast<bool>(properties.sampleCounts & create.samples)) {
EXIT("image format does not support required usage: format=%d type=%d usage=0x%x "
"flags=0x%x samples=%u\n",
static_cast<int>(create.format), static_cast<int>(create.imageType),
static_cast<vk::ImageUsageFlags::MaskType>(create.usage),
static_cast<vk::ImageCreateFlags::MaskType>(create.flags), backing.samples);
}
backing.memory.property = vk::MemoryPropertyFlagBits::eDeviceLocal;
if (!graphics.CreateImage(create, backing)) {
EXIT("failed to create image: extent=%ux%ux%u format=%d layers=%u levels=%u\n",
create.extent.width, create.extent.height, create.extent.depth,
static_cast<int>(create.format), create.arrayLayers, create.mipLevels);
}
}
uint64_t Image::HashGuestEdges() const {
constexpr uint64_t page_mask = TRACKER_PAGE_SIZE - 1;
std::array<uint8_t, TRACKER_PAGE_SIZE * 2> bytes {};
const auto range = info.data;
const uint64_t head_end = std::min(range.End(), (range.address + page_mask) & ~page_mask);
const uint64_t tail_begin = std::max(range.address, range.End() & ~page_mask);
const uint64_t head_size = head_end - range.address;
const uint64_t tail_address = tail_begin < head_end ? head_end : tail_begin;
const uint64_t tail_size = range.End() - tail_address;
if ((head_size != 0 &&
!LibKernel::Memory::TryReadBacking(range.address, bytes.data(), head_size)) ||
(tail_size != 0 &&
!LibKernel::Memory::TryReadBacking(tail_address, bytes.data() + head_size, tail_size))) {
EXIT("Image: failed to hash guest backing\n");
}
return XXH3_64bits(bytes.data(), static_cast<size_t>(head_size + tail_size));
}
Image::~Image() {
KYTY_PROFILER_FUNCTION();
if (m_graphics == nullptr) {
return;
}
{
std::lock_guard lock(views.mutex);
for (auto& cached: views.views) {
if (cached.view != nullptr) {
m_graphics->device.destroyImageView(cached.view, nullptr);
cached.view = nullptr;
}
}
views.views.clear();
}
if (backing.image != nullptr) {
m_graphics->DeleteImage(backing);
}
}
} // namespace Libs::Graphics
@@ -0,0 +1,197 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
#include "common/assert.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/image/imageInfo.h"
#include <compare>
#include <limits>
#include <mutex>
#include <optional>
#include <span>
#include <utility>
#include <vector>
namespace Libs::Graphics {
class Buffer;
class CommandScheduler;
struct ImageTestAccess;
struct ImageId {
uint32_t index = std::numeric_limits<uint32_t>::max();
uint32_t generation = 0;
[[nodiscard]] explicit operator bool() const noexcept {
return index != std::numeric_limits<uint32_t>::max();
}
auto operator<=>(const ImageId&) const = default;
};
struct CachedImageView {
ImageViewInfo info;
vk::ImageView view = nullptr;
};
struct ImageViewCache {
ImageViewCache() = default;
KYTY_CLASS_NO_COPY(ImageViewCache);
std::mutex mutex;
std::vector<CachedImageView> views;
};
struct ImageUsage {
bool texture = false;
bool storage = false;
bool render_target = false;
bool depth_target = false;
bool video_out = false;
};
struct ImageBinding {
bool is_bound = false;
bool is_target = false;
bool needs_rebind = false;
bool force_general = false;
};
class Image final {
public:
Image(GraphicContext& graphics, CommandScheduler& scheduler, const ImageInfo& info);
~Image();
KYTY_CLASS_NO_COPY(Image);
[[nodiscard]] vk::ImageView FindView(const ImageViewInfo& view_info);
void AssociateDepth(ImageId image_id) { depth_id = image_id; }
using Barriers = std::vector<vk::ImageMemoryBarrier2>;
[[nodiscard]] Barriers GetBarriers(vk::ImageLayout destination_layout,
vk::AccessFlags2 destination_access,
vk::PipelineStageFlags2 destination_stage,
std::optional<ImageSubresourceRange> range);
void Transit(vk::ImageLayout destination_layout, vk::AccessFlags2 destination_access,
std::optional<ImageSubresourceRange> range, vk::CommandBuffer command_buffer);
void Upload(std::span<const vk::BufferImageCopy> copies, vk::Buffer buffer, uint64_t offset,
uint64_t size);
void Download(std::span<const vk::BufferImageCopy> copies, vk::Buffer buffer, uint64_t offset,
uint64_t size);
void CopyImage(Image& source);
void Resolve(Image& source, const ImageSubresourceRange& source_range,
const ImageSubresourceRange& destination_range);
void CopyImageWithBuffer(Image& source, Buffer& buffer);
void CopyMip(Image& source, uint32_t mip, uint32_t layer);
void InvalidateCpuWrite(uint64_t vaddr, uint64_t size) {
if (ImageRangeOverlaps(info.data.address, info.data.size, vaddr, size)) {
m_cpu_dirty = true;
m_maybe_cpu_dirty = false;
m_maybe_hash_valid = false;
} else if (ImagePageRangesOverlap(info.data.address, info.data.size, vaddr, size)) {
m_maybe_cpu_dirty = true;
}
}
[[nodiscard]] bool IsCpuDirty() const { return m_cpu_dirty || m_maybe_cpu_dirty; }
[[nodiscard]] bool IsDefinitelyCpuDirty() const { return m_cpu_dirty; }
[[nodiscard]] bool IsMaybeCpuDirty() const { return m_maybe_cpu_dirty; }
void MarkMaybeCpuDirty() {
if (!m_cpu_dirty) {
m_maybe_cpu_dirty = true;
}
}
[[nodiscard]] bool NeedsMaybeCpuHash() const {
return m_maybe_cpu_dirty && !m_maybe_hash_valid;
}
void SetMaybeCpuHash(uint64_t hash) {
if (!NeedsMaybeCpuHash()) {
EXIT("image cannot initialize maybe-dirty hash\n");
}
m_maybe_cpu_hash = hash;
m_maybe_hash_valid = true;
}
[[nodiscard]] bool ResolveMaybeCpuHash(uint64_t hash) {
if (!m_maybe_cpu_dirty || !m_maybe_hash_valid || m_cpu_dirty) {
EXIT("image cannot resolve maybe-dirty hash\n");
}
m_maybe_cpu_dirty = false;
m_maybe_hash_valid = false;
m_cpu_dirty |= hash != m_maybe_cpu_hash;
return m_cpu_dirty;
}
void RefreshComplete() {
if (!IsCpuDirty()) {
EXIT("clean image cannot complete a refresh\n");
}
m_cpu_dirty = false;
m_maybe_cpu_dirty = false;
m_maybe_hash_valid = false;
}
[[nodiscard]] bool IsGpuModified() const noexcept { return m_gpu_modified; }
void MarkGpuModified() noexcept { m_gpu_modified = true; }
void ClearGpuModified() noexcept { m_gpu_modified = false; }
[[nodiscard]] bool IsBufferModified() const noexcept { return m_buffer_modified; }
void MarkBufferModified() noexcept { m_buffer_modified = true; }
void ClearBufferModified() noexcept { m_buffer_modified = false; }
[[nodiscard]] bool Overlaps(uint64_t address, uint64_t size,
bool pages = false) const noexcept {
return pages ? ImagePageRangesOverlap(info.data.address, info.data.size, address, size)
: ImageRangeOverlaps(info.data.address, info.data.size, address, size);
}
[[nodiscard]] bool GpuOverlaps(uint64_t address, uint64_t size) const noexcept {
return IsGpuModified() && Overlaps(address, size);
}
[[nodiscard]] bool SafeToDownload() const noexcept {
return IsGpuModified() && !IsBufferModified() && !IsCpuDirty();
}
[[nodiscard]] bool IsTracked() const noexcept { return track_addr != 0 && track_addr_end != 0; }
[[nodiscard]] uint64_t AccountedSize() const noexcept {
return backing.image == nullptr ? 0 : (info.data.size + 1023) & ~uint64_t {1023};
}
[[nodiscard]] uint64_t HashGuestEdges() const;
ImageInfo info;
VulkanImage backing;
ImageViewCache views;
ImageUsage usage;
ImageBinding binding;
bool registered = false;
uint64_t track_addr = 0;
uint64_t track_addr_end = 0;
ImageId depth_id {};
uint64_t tick_accessed_last = 0;
size_t lru_id = 0;
private:
friend struct ImageTestAccess;
[[nodiscard]] static vk::ImageAspectFlags FullAspectMask(vk::Format format) noexcept;
[[nodiscard]] static uint32_t CopyRows(uint64_t row_size, uint32_t rows,
uint64_t capacity) noexcept;
[[nodiscard]] static std::pair<uint32_t, uint32_t>
SanitizeCopyLayers(const Image& source, const Image& destination, uint32_t depth);
GraphicContext* m_graphics = nullptr;
CommandScheduler* m_scheduler = nullptr;
uint64_t m_maybe_cpu_hash = 0;
bool m_cpu_dirty = false;
bool m_maybe_cpu_dirty = false;
bool m_maybe_hash_valid = false;
bool m_gpu_modified = false;
bool m_buffer_modified = false;
};
namespace ImageOps {
void Validate(const ImageInfo& info);
[[nodiscard]] uint32_t RenderTargetTransferFormat(uint32_t bytes_per_element);
} // namespace ImageOps
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGE_H_
@@ -0,0 +1,580 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEINFO_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEINFO_H_
#include "common/assert.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/gpu_format.h"
#include "graphics/host_gpu/regionDefinitions.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <array>
#include <bit>
#include <cmath>
#include <cstdint>
namespace Libs::Graphics {
struct GuestRange {
uint64_t address = 0;
uint64_t size = 0;
[[nodiscard]] constexpr bool Empty() const noexcept { return address == 0 || size == 0; }
[[nodiscard]] constexpr bool Valid() const noexcept {
return !Empty() && address < TRACKER_ADDRESS_SIZE &&
size <= TRACKER_ADDRESS_SIZE - address;
}
[[nodiscard]] constexpr uint64_t End() const noexcept { return address + size; }
auto operator<=>(const GuestRange&) const = default;
};
enum class VideoOutCompression : uint8_t { Uncompressed, Dcc256_256_0, Dcc256_64_64, Unsupported };
enum class ImageMetadataKind : uint8_t { None, Htile, Dcc };
struct ImageMetadataInfo {
GuestRange range;
ImageMetadataKind kind = ImageMetadataKind::None;
uint32_t control = 0;
VideoOutCompression compression = VideoOutCompression::Uncompressed;
bool stencil_compressed = false;
};
struct ImageSubresources {
uint32_t levels = 1;
uint32_t layers = 1;
auto operator<=>(const ImageSubresources&) const = default;
};
struct ImageSubresourceRange {
uint32_t base_level = 0;
uint32_t level_count = 1;
uint32_t base_layer = 0;
uint32_t layer_count = 1;
auto operator<=>(const ImageSubresourceRange&) const = default;
};
struct ImageMipInfo {
uint64_t offset = 0;
uint64_t size = 0;
uint32_t pitch = 0;
uint32_t height = 0;
auto operator<=>(const ImageMipInfo&) const = default;
};
struct ImageInfo {
GuestRange data;
GuestRange stencil;
ImageMetadataInfo metadata;
uint32_t htile_clear_mask = UINT32_MAX;
vk::Format pixel_format = vk::Format::eUndefined;
uint32_t guest_format = 0;
Prospero::ImageType type = Prospero::ImageType::kColor2D;
vk::Extent3D extent = {1, 1, 1};
ImageSubresources resources;
uint32_t pitch = 0;
uint32_t bytes_per_block = 0;
uint32_t samples = 1;
uint32_t tile_mode = 0;
bool bgra16 = false;
std::array<ImageMipInfo, 16> mip_layout {};
[[nodiscard]] constexpr bool HasStencil() const noexcept { return !stencil.Empty(); }
[[nodiscard]] constexpr bool HasMetadata() const noexcept {
return metadata.kind != ImageMetadataKind::None;
}
[[nodiscard]] bool IsDepth() const noexcept;
[[nodiscard]] bool IsBlock() const noexcept {
return Prospero::BlockCompressedBytesPerBlock(guest_format) != 0;
}
[[nodiscard]] bool IsTiled() const noexcept {
return tile_mode != Prospero::GpuEnumValue(Prospero::TileMode::kLinear);
}
[[nodiscard]] constexpr bool IsVolume() const noexcept {
return type == Prospero::ImageType::kColor3D;
}
[[nodiscard]] constexpr bool IsLayered() const noexcept {
return !IsVolume() && resources.layers > 1;
}
[[nodiscard]] constexpr uint32_t TransferLayers() const noexcept {
return IsVolume() ? extent.depth : resources.layers;
}
[[nodiscard]] vk::Extent2D BlockExtent() const noexcept {
const auto shift = Prospero::BlockCompressedBytesPerBlock(guest_format) != 0 ? 2u : 0u;
return {pitch >> shift, extent.height >> shift};
}
[[nodiscard]] bool IsCompatible(const ImageInfo& other) const noexcept {
return pixel_format == other.pixel_format && samples == other.samples &&
bytes_per_block == other.bytes_per_block;
}
[[nodiscard]] int32_t MipOf(const ImageInfo& container) const noexcept {
if (!IsCompatible(container) || tile_mode != container.tile_mode || resources.levels != 1 ||
container.resources.layers == 0 ||
container.resources.levels > container.mip_layout.size()) {
return -1;
}
if (HasStencil() != container.HasStencil() ||
(HasStencil() && (stencil.address < container.stencil.address ||
stencil.End() > container.stencil.End()))) {
return -1;
}
int32_t mip = -1;
for (uint32_t level = 0; level < container.resources.levels; level++) {
const auto& layout = container.mip_layout[level];
if (layout.size == 0 || layout.size % container.resources.layers != 0 ||
container.data.address > UINT64_MAX - layout.offset) {
continue;
}
const auto mip_base = container.data.address + layout.offset;
const auto slice_size = layout.size / container.resources.layers;
if (slice_size == 0 || mip_base > UINT64_MAX - layout.size) {
continue;
}
const auto mip_end = mip_base + layout.size;
if (data.address >= mip_base && data.address < mip_end &&
(data.address - mip_base) % slice_size == 0) {
mip = static_cast<int32_t>(level);
break;
}
}
if (mip < 0) {
return -1;
}
const auto level = static_cast<uint32_t>(mip);
if (extent.width != std::max(container.extent.width >> level, 1u) ||
extent.height != std::max(container.extent.height >> level, 1u)) {
return -1;
}
const auto mip_depth = std::max(container.extent.depth >> level, 1u);
if (container.type == Prospero::ImageType::kColor3D &&
type == Prospero::ImageType::kColor2D) {
if (resources.layers != mip_depth) {
return -1;
}
} else if (type != container.type) {
return -1;
}
return mip;
}
[[nodiscard]] int32_t SliceOf(const ImageInfo& container, int32_t mip) const noexcept {
if (!IsCompatible(container) || type != container.type || mip < 0 ||
static_cast<uint32_t>(mip) >= container.resources.levels ||
container.resources.levels > container.mip_layout.size() ||
container.resources.layers == 0 || data.size == 0) {
return -1;
}
const auto level = static_cast<uint32_t>(mip);
if (extent.width != std::max(container.extent.width >> level, 1u) ||
extent.height != std::max(container.extent.height >> level, 1u)) {
return -1;
}
const auto& layout = container.mip_layout[level];
if (layout.size == 0 || layout.size % container.resources.layers != 0 ||
container.data.address > UINT64_MAX - layout.offset) {
return -1;
}
const auto slice_size = layout.size / container.resources.layers;
if (slice_size == 0 || data.size % slice_size != 0) {
return -1;
}
const auto mip_base = container.data.address + layout.offset;
if (data.address < mip_base) {
return -1;
}
const auto address_delta = data.address - mip_base;
if (address_delta % data.size != 0 || address_delta / data.size > INT32_MAX) {
return -1;
}
return static_cast<int32_t>(address_delta / data.size);
}
};
struct ImageViewInfo {
vk::Format format = vk::Format::eUndefined;
vk::ImageViewType type = vk::ImageViewType::e2D;
vk::ImageAspectFlags aspect = vk::ImageAspectFlagBits::eColor;
uint32_t base_level = 0;
uint32_t level_count = 1;
uint32_t base_layer = 0;
uint32_t layer_count = 1;
vk::ComponentMapping mapping = {};
vk::ImageUsageFlags usage = vk::ImageUsageFlagBits::eSampled;
[[nodiscard]] bool operator==(const ImageViewInfo& rhs) const noexcept {
return format == rhs.format && type == rhs.type && aspect == rhs.aspect &&
base_level == rhs.base_level && level_count == rhs.level_count &&
base_layer == rhs.base_layer && layer_count == rhs.layer_count &&
mapping.r == rhs.mapping.r && mapping.g == rhs.mapping.g &&
mapping.b == rhs.mapping.b && mapping.a == rhs.mapping.a && usage == rhs.usage;
}
};
struct DepthFormatPolicy {
Prospero::DepthFormat depth_format;
Prospero::BufferFormat guest_format;
uint32_t bytes_per_element;
vk::Format sampled_view_format;
vk::Format depth_attachment_format;
std::array<vk::Format, 3> stencil_attachment_formats;
};
inline constexpr std::array<DepthFormatPolicy, 2> DEPTH_FORMAT_POLICIES {{
{Prospero::DepthFormat::kZ16,
Prospero::BufferFormat::k16UNorm,
2,
vk::Format::eR16Unorm,
vk::Format::eD16Unorm,
{vk::Format::eD16UnormS8Uint, vk::Format::eD24UnormS8Uint, vk::Format::eD32SfloatS8Uint}},
{Prospero::DepthFormat::kZ32F,
Prospero::BufferFormat::k32Float,
4,
vk::Format::eR32Sfloat,
vk::Format::eD32Sfloat,
{vk::Format::eD32SfloatS8Uint, vk::Format::eUndefined, vk::Format::eUndefined}},
}};
[[nodiscard]] inline constexpr const DepthFormatPolicy*
FindDepthFormatPolicy(uint32_t depth_format) noexcept {
for (const auto& policy: DEPTH_FORMAT_POLICIES) {
if (Prospero::GpuEnumValue(policy.depth_format) == depth_format) {
return &policy;
}
}
return nullptr;
}
[[nodiscard]] inline constexpr const DepthFormatPolicy*
FindGuestDepthFormatPolicy(uint32_t guest_format) noexcept {
for (const auto& policy: DEPTH_FORMAT_POLICIES) {
if (Prospero::GpuEnumValue(policy.guest_format) == guest_format) {
return &policy;
}
}
return nullptr;
}
[[nodiscard]] inline constexpr bool IsStencilAttachmentFormat(const DepthFormatPolicy& policy,
vk::Format format) noexcept {
for (const auto candidate: policy.stencil_attachment_formats) {
if (candidate != vk::Format::eUndefined && candidate == format) {
return true;
}
}
return false;
}
[[nodiscard]] inline constexpr vk::Format DepthAttachmentFormat(const DepthFormatPolicy& policy,
bool has_stencil) noexcept {
return has_stencil ? policy.stencil_attachment_formats.front() : policy.depth_attachment_format;
}
[[nodiscard]] inline constexpr vk::Format DepthAttachmentFormat(uint32_t depth_format,
uint32_t stencil_format) noexcept {
bool has_stencil = false;
switch (static_cast<Prospero::StencilFormat>(stencil_format)) {
case Prospero::StencilFormat::kInvalid: break;
case Prospero::StencilFormat::k8UInt: has_stencil = true; break;
default: return vk::Format::eUndefined;
}
const auto* policy = FindDepthFormatPolicy(depth_format);
return policy == nullptr ? vk::Format::eUndefined : DepthAttachmentFormat(*policy, has_stencil);
}
[[nodiscard]] inline constexpr vk::ImageUsageFlags DepthTargetImageUsage() noexcept {
return vk::ImageUsageFlagBits::eDepthStencilAttachment | vk::ImageUsageFlagBits::eSampled |
vk::ImageUsageFlagBits::eTransferSrc | vk::ImageUsageFlagBits::eTransferDst;
}
[[nodiscard]] inline constexpr vk::Format DepthAspectTransferFormat(vk::Format format) noexcept {
switch (format) {
case vk::Format::eD16Unorm:
case vk::Format::eD16UnormS8Uint: return vk::Format::eD16Unorm;
case vk::Format::eD24UnormS8Uint: return vk::Format::eX8D24UnormPack32;
case vk::Format::eD32Sfloat:
case vk::Format::eD32SfloatS8Uint: return vk::Format::eD32Sfloat;
default: return vk::Format::eUndefined;
}
}
inline bool ImageInfo::IsDepth() const noexcept {
return DepthAspectTransferFormat(pixel_format) != vk::Format::eUndefined;
}
[[nodiscard]] inline constexpr uint32_t DepthAspectTransferBytes(vk::Format format) noexcept {
switch (DepthAspectTransferFormat(format)) {
case vk::Format::eD16Unorm: return 2;
case vk::Format::eX8D24UnormPack32:
case vk::Format::eD32Sfloat: return 4;
default: return 0;
}
}
[[nodiscard]] inline constexpr uint32_t EncodeD16AsD24(uint16_t value) noexcept {
return static_cast<uint32_t>((static_cast<uint64_t>(value) * 0x00ffffffu + 0x7fffu) / 0xffffu);
}
[[nodiscard]] inline uint32_t EncodeD16AsD32(uint16_t value) noexcept {
return std::bit_cast<uint32_t>(static_cast<float>(value) / 65535.0f);
}
[[nodiscard]] inline constexpr bool IsSupportedSampledDepthFormat(vk::Format image_format,
uint32_t guest_format,
vk::Format view_format) noexcept {
const auto* policy = FindGuestDepthFormatPolicy(guest_format);
return policy != nullptr && view_format == policy->sampled_view_format &&
(image_format == policy->depth_attachment_format ||
IsStencilAttachmentFormat(*policy, image_format));
}
[[nodiscard]] inline constexpr bool IsSupportedSampledDepthFormat(vk::Format image_format,
vk::Format view_format) noexcept {
for (const auto& policy: DEPTH_FORMAT_POLICIES) {
if (IsSupportedSampledDepthFormat(image_format, Prospero::GpuEnumValue(policy.guest_format),
view_format)) {
return true;
}
}
return false;
}
[[nodiscard]] inline constexpr bool IsSupportedDepthTargetFormat(const ImageInfo& info) {
const auto* policy = FindGuestDepthFormatPolicy(info.guest_format);
return policy != nullptr && info.bytes_per_block == policy->bytes_per_element &&
(info.HasStencil() ? IsStencilAttachmentFormat(*policy, info.pixel_format)
: info.pixel_format == policy->depth_attachment_format);
}
[[nodiscard]] inline constexpr bool IsSupportedDepthPlaneReadback(const ImageInfo& info) {
if (!IsSupportedDepthTargetFormat(info)) {
return false;
}
const auto transfer_bytes = DepthAspectTransferBytes(info.pixel_format);
return transfer_bytes == info.bytes_per_block ||
(info.bytes_per_block == sizeof(uint16_t) &&
transfer_bytes == sizeof(uint32_t));
}
[[nodiscard]] inline VideoOutCompression
ClassifyVideoOutCompression(bool compressed, uint64_t metadata_address, uint32_t dcc_control,
uint64_t dcc_clear_color) noexcept {
constexpr uint32_t DCC_256_256_0 = 0x00000048u;
constexpr uint32_t DCC_256_64_64 = 0x00000208u;
if (!compressed) {
return metadata_address == 0 && dcc_control == 0 && dcc_clear_color == 0
? VideoOutCompression::Uncompressed
: VideoOutCompression::Unsupported;
}
if (metadata_address == 0 || (metadata_address & 0xffu) != 0 || dcc_clear_color != 0) {
return VideoOutCompression::Unsupported;
}
switch (dcc_control) {
case DCC_256_256_0: return VideoOutCompression::Dcc256_256_0;
case DCC_256_64_64: return VideoOutCompression::Dcc256_64_64;
default: return VideoOutCompression::Unsupported;
}
}
[[nodiscard]] inline constexpr bool
CanUseVideoOutNativeWithoutUpload(VideoOutCompression compression, bool render_target,
bool gpu_modified, bool guest_modified) noexcept {
return compression != VideoOutCompression::Uncompressed &&
compression != VideoOutCompression::Unsupported && !guest_modified &&
(render_target || gpu_modified);
}
struct VideoOutPixelFormatInfo {
vk::Format format = vk::Format::eUndefined;
uint32_t guest_format = 0;
uint32_t bytes_per_element = 0;
bool bgra16 = false;
};
struct VideoOutFormatPolicy {
uint64_t pixel_format;
VideoOutPixelFormatInfo info;
};
inline constexpr std::array<VideoOutFormatPolicy, 6> VIDEO_OUT_FORMAT_POLICIES {{
{0x8000000022000000ull,
{vk::Format::eR8G8B8A8Srgb, Prospero::GpuEnumValue(Prospero::BufferFormat::k8_8_8_8Srgb), 4,
false}},
{0x8000000000000000ull,
{vk::Format::eB8G8R8A8Srgb, Prospero::GpuEnumValue(Prospero::BufferFormat::k8_8_8_8Srgb), 4,
false}},
{0x8100000022000000ull,
{vk::Format::eA2B10G10R10UnormPack32,
Prospero::GpuEnumValue(Prospero::BufferFormat::k10_10_10_2UNorm), 4, false}},
{0x8100000000000000ull,
{vk::Format::eA2R10G10B10UnormPack32,
Prospero::GpuEnumValue(Prospero::BufferFormat::k10_10_10_2UNorm), 4, false}},
{0xc001000622000000ull,
{vk::Format::eR16G16B16A16Sfloat,
Prospero::GpuEnumValue(Prospero::BufferFormat::k16_16_16_16Float), 8, false}},
{0xc001000600000000ull,
{vk::Format::eR16G16B16A16Sfloat,
Prospero::GpuEnumValue(Prospero::BufferFormat::k16_16_16_16Float), 8, true}},
}};
[[nodiscard]] inline bool DecodeVideoOutPixelFormat(uint64_t pixel_format,
VideoOutPixelFormatInfo& info) {
for (const auto& policy: VIDEO_OUT_FORMAT_POLICIES) {
if (policy.pixel_format == pixel_format) {
info = policy.info;
return true;
}
}
return false;
}
[[nodiscard]] inline bool IsSupportedVideoOutFormat(const ImageInfo& info) {
for (const auto& policy: VIDEO_OUT_FORMAT_POLICIES) {
if (info.pixel_format == policy.info.format &&
info.guest_format == policy.info.guest_format &&
info.bytes_per_block == policy.info.bytes_per_element &&
info.bgra16 == policy.info.bgra16) {
return true;
}
}
return false;
}
[[nodiscard]] inline constexpr bool
IsSupportedDisplayRenderTargetTileMode(uint32_t tile_mode) noexcept {
return tile_mode == Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget);
}
[[nodiscard]] inline constexpr bool IsSupportedStandard64RenderTarget(const ImageInfo& info) {
if (info.tile_mode != Prospero::GpuEnumValue(Prospero::TileMode::kStandard64KB) ||
info.data.address == 0 || (info.data.address & 0xffffu) != 0 || info.extent.width == 0 ||
info.extent.height == 0 || info.bytes_per_block != 4 || info.resources.levels != 1 ||
info.resources.layers != 1 || info.samples != 1) {
return false;
}
const auto expected_pitch =
(static_cast<uint64_t>(info.extent.width) + 127u) & ~uint64_t {127u};
const auto padded_height =
(static_cast<uint64_t>(info.extent.height) + 127u) & ~uint64_t {127u};
return expected_pitch <= UINT32_MAX && info.pitch == expected_pitch &&
expected_pitch <= UINT64_MAX / padded_height / info.bytes_per_block &&
info.data.size == expected_pitch * padded_height * info.bytes_per_block;
}
[[nodiscard]] inline constexpr bool IsTiledRenderTarget(const ImageInfo& info) noexcept {
return info.tile_mode == Prospero::GpuEnumValue(Prospero::TileMode::kRenderTarget) ||
IsSupportedStandard64RenderTarget(info);
}
[[nodiscard]] inline bool DecodePackedColorClear(vk::Format format, uint32_t packed,
vk::ClearColorValue& clear) {
vk::ClearColorValue next {};
const auto unorm8 = [](uint32_t value) { return static_cast<float>(value & 0xffu) / 255.0f; };
const auto srgb8 = [](uint32_t value) {
const auto encoded = static_cast<float>(value & 0xffu) / 255.0f;
return encoded <= 0.04045f ? encoded / 12.92f
: std::pow((encoded + 0.055f) / 1.055f, 2.4f);
};
switch (format) {
case vk::Format::eR32Uint:
next.uint32[0] = packed;
break;
case vk::Format::eR32Sint:
next.int32[0] = static_cast<int32_t>(packed);
break;
case vk::Format::eR8G8B8A8Srgb:
next.float32[0] = srgb8(packed);
next.float32[1] = srgb8(packed >> 8u);
next.float32[2] = srgb8(packed >> 16u);
next.float32[3] = unorm8(packed >> 24u);
break;
case vk::Format::eB8G8R8A8Srgb:
next.float32[0] = srgb8(packed >> 16u);
next.float32[1] = srgb8(packed >> 8u);
next.float32[2] = srgb8(packed);
next.float32[3] = unorm8(packed >> 24u);
break;
case vk::Format::eR8G8B8A8Unorm:
next.float32[0] = unorm8(packed);
next.float32[1] = unorm8(packed >> 8u);
next.float32[2] = unorm8(packed >> 16u);
next.float32[3] = unorm8(packed >> 24u);
break;
case vk::Format::eB8G8R8A8Unorm:
next.float32[0] = unorm8(packed >> 16u);
next.float32[1] = unorm8(packed >> 8u);
next.float32[2] = unorm8(packed);
next.float32[3] = unorm8(packed >> 24u);
break;
case vk::Format::eA2B10G10R10UnormPack32:
next.float32[0] = static_cast<float>(packed & 0x3ffu) / 1023.0f;
next.float32[1] = static_cast<float>((packed >> 10u) & 0x3ffu) / 1023.0f;
next.float32[2] = static_cast<float>((packed >> 20u) & 0x3ffu) / 1023.0f;
next.float32[3] = static_cast<float>((packed >> 30u) & 0x3u) / 3.0f;
break;
case vk::Format::eA2R10G10B10UnormPack32:
next.float32[0] = static_cast<float>((packed >> 20u) & 0x3ffu) / 1023.0f;
next.float32[1] = static_cast<float>((packed >> 10u) & 0x3ffu) / 1023.0f;
next.float32[2] = static_cast<float>(packed & 0x3ffu) / 1023.0f;
next.float32[3] = static_cast<float>((packed >> 30u) & 0x3u) / 3.0f;
break;
default: return false;
}
clear = next;
return true;
}
[[nodiscard]] inline bool DecodePackedStencilClear(uint32_t packed, uint8_t& clear) {
const auto value = static_cast<uint8_t>(packed);
if (packed != static_cast<uint32_t>(value) * 0x01010101u) {
return false;
}
clear = value;
return true;
}
[[nodiscard]] inline bool DecodePackedDepthClear(vk::Format format, uint32_t packed, float& clear) {
if (format != vk::Format::eD32Sfloat && format != vk::Format::eD32SfloatS8Uint) {
return false;
}
const auto value = std::bit_cast<float>(packed);
if (!std::isfinite(value) || value < 0.0f || value > 1.0f) {
return false;
}
clear = value;
return true;
}
[[nodiscard]] inline bool ImageRangeOverlaps(uint64_t left, uint64_t left_size, uint64_t right,
uint64_t right_size) {
if (left_size == 0 || right_size == 0 || left > UINT64_MAX - left_size ||
right > UINT64_MAX - right_size) {
EXIT("invalid image overlap range\n");
}
return left < right + right_size && right < left + left_size;
}
[[nodiscard]] inline bool ImageRangeOverlaps(GuestRange left, GuestRange right) {
return ImageRangeOverlaps(left.address, left.size, right.address, right.size);
}
[[nodiscard]] inline bool ImagePageRangesOverlap(uint64_t left, uint64_t left_size, uint64_t right,
uint64_t right_size) {
if (left_size == 0 || right_size == 0 || left > UINT64_MAX - left_size ||
right > UINT64_MAX - right_size) {
EXIT("invalid image page-overlap range\n");
}
const auto left_first = left / TRACKER_PAGE_SIZE;
const auto left_last = (left + left_size - 1) / TRACKER_PAGE_SIZE;
const auto right_first = right / TRACKER_PAGE_SIZE;
const auto right_last = (right + right_size - 1) / TRACKER_PAGE_SIZE;
return left_first <= right_last && right_first <= left_last;
}
[[nodiscard]] inline bool ImagePageRangesOverlap(GuestRange left, GuestRange right) {
return ImagePageRangesOverlap(left.address, left.size, right.address, right.size);
}
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEINFO_H_
@@ -0,0 +1,416 @@
#include "graphics/host_gpu/renderer/image/imageView.h"
#include "common/assert.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/image/image.h"
#include <mutex>
namespace Libs::Graphics {
namespace {
[[nodiscard]] bool IsComponentSwizzle(vk::ComponentSwizzle swizzle) {
switch (swizzle) {
case vk::ComponentSwizzle::eIdentity:
case vk::ComponentSwizzle::eZero:
case vk::ComponentSwizzle::eOne:
case vk::ComponentSwizzle::eR:
case vk::ComponentSwizzle::eG:
case vk::ComponentSwizzle::eB:
case vk::ComponentSwizzle::eA: return true;
default: return false;
}
}
[[nodiscard]] bool IsCompatibleViewFormat(vk::Format image_format, vk::Format view_format) {
return ImageViewOps::FormatsCompatible(image_format, view_format);
}
[[nodiscard]] bool IsStencilViewFormat(vk::Format format) {
switch (format) {
case vk::Format::eS8Uint:
case vk::Format::eR8Uint:
case vk::Format::eR8Unorm: return true;
default: return false;
}
}
[[nodiscard]] bool IsDepthViewFormat(vk::Format format) {
switch (format) {
case vk::Format::eD16Unorm:
case vk::Format::eR16Unorm:
case vk::Format::eD32Sfloat:
case vk::Format::eR32Sfloat:
case vk::Format::eR32Uint: return true;
default: return false;
}
}
[[nodiscard]] bool IsValidViewType(const VulkanImage& image, const ImageViewInfo& info) {
switch (image.image_type) {
case vk::ImageType::e1D:
if (info.type != vk::ImageViewType::e1D && info.type != vk::ImageViewType::e1DArray) {
return false;
}
return info.type != vk::ImageViewType::e1D || info.layer_count == 1;
case vk::ImageType::e2D:
switch (info.type) {
case vk::ImageViewType::e2D: return info.layer_count == 1;
case vk::ImageViewType::e2DArray: return true;
case vk::ImageViewType::eCube:
return static_cast<bool>(image.flags &
vk::ImageCreateFlagBits::eCubeCompatible) &&
info.base_layer % 6 == 0 && info.layer_count == 6;
case vk::ImageViewType::eCubeArray:
return static_cast<bool>(image.flags &
vk::ImageCreateFlagBits::eCubeCompatible) &&
info.base_layer % 6 == 0 && info.layer_count % 6 == 0;
default: return false;
}
case vk::ImageType::e3D:
switch (info.type) {
case vk::ImageViewType::e3D:
return info.base_layer == 0 && info.layer_count == 1;
case vk::ImageViewType::e2D:
return static_cast<bool>(
image.flags & vk::ImageCreateFlagBits::e2DArrayCompatible) &&
info.level_count == 1 && info.layer_count == 1;
case vk::ImageViewType::e2DArray:
return static_cast<bool>(
image.flags & vk::ImageCreateFlagBits::e2DArrayCompatible) &&
info.level_count == 1;
default: return false;
}
default: return false;
}
}
[[nodiscard]] bool IsValidAspect(const VulkanImage& image, vk::ImageAspectFlags aspect) {
const auto depth_format = DepthAspectTransferFormat(image.format);
if (depth_format == vk::Format::eUndefined) {
return aspect == vk::ImageAspectFlagBits::eColor;
}
const auto supported = ImageViewOps::DepthAspectMask(image.format);
return static_cast<bool>(aspect) && !(aspect & ~supported);
}
} // namespace
namespace ImageViewOps {
namespace {
enum CompatibilityClass : uint32_t {
None = 0,
Bit8 = 1u << 0,
Bit16 = 1u << 1,
Bit24 = 1u << 2,
Bit32 = 1u << 3,
Bit48 = 1u << 4,
Bit64 = 1u << 5,
Bit96 = 1u << 6,
Bit128 = 1u << 7,
Bit192 = 1u << 8,
Bit256 = 1u << 9,
Bc1Rgb = 1u << 10,
Bc1Rgba = 1u << 11,
Bc2 = 1u << 12,
Bc3 = 1u << 13,
Bc4 = 1u << 14,
Bc5 = 1u << 15,
Bc6h = 1u << 16,
Bc7 = 1u << 17,
D16 = 1u << 18,
D16S8 = 1u << 19,
D24 = 1u << 20,
D24S8 = 1u << 21,
D32 = 1u << 22,
D32S8 = 1u << 23,
S8 = 1u << 24,
};
[[nodiscard]] uint32_t FormatClass(vk::Format format) noexcept {
switch (format) {
case vk::Format::eR4G4UnormPack8:
case vk::Format::eR8Sint:
case vk::Format::eR8Snorm:
case vk::Format::eR8Srgb:
case vk::Format::eR8Sscaled:
case vk::Format::eR8Uint:
case vk::Format::eR8Unorm:
case vk::Format::eR8Uscaled: return Bit8;
case vk::Format::eA1R5G5B5UnormPack16:
case vk::Format::eA4B4G4R4UnormPack16:
case vk::Format::eA4R4G4B4UnormPack16:
case vk::Format::eB4G4R4A4UnormPack16:
case vk::Format::eB5G5R5A1UnormPack16:
case vk::Format::eB5G6R5UnormPack16:
case vk::Format::eR10X6UnormPack16:
case vk::Format::eR12X4UnormPack16:
case vk::Format::eR16Sfloat:
case vk::Format::eR16Sint:
case vk::Format::eR16Snorm:
case vk::Format::eR16Sscaled:
case vk::Format::eR16Uint:
case vk::Format::eR16Unorm:
case vk::Format::eR16Uscaled:
case vk::Format::eR4G4B4A4UnormPack16:
case vk::Format::eR5G5B5A1UnormPack16:
case vk::Format::eR5G6B5UnormPack16:
case vk::Format::eR8G8Sint:
case vk::Format::eR8G8Snorm:
case vk::Format::eR8G8Srgb:
case vk::Format::eR8G8Sscaled:
case vk::Format::eR8G8Uint:
case vk::Format::eR8G8Unorm:
case vk::Format::eR8G8Uscaled: return Bit16;
case vk::Format::eB8G8R8Sint:
case vk::Format::eB8G8R8Snorm:
case vk::Format::eB8G8R8Srgb:
case vk::Format::eB8G8R8Sscaled:
case vk::Format::eB8G8R8Uint:
case vk::Format::eB8G8R8Unorm:
case vk::Format::eB8G8R8Uscaled:
case vk::Format::eR8G8B8Sint:
case vk::Format::eR8G8B8Snorm:
case vk::Format::eR8G8B8Srgb:
case vk::Format::eR8G8B8Sscaled:
case vk::Format::eR8G8B8Uint:
case vk::Format::eR8G8B8Unorm:
case vk::Format::eR8G8B8Uscaled: return Bit24;
case vk::Format::eA2B10G10R10SintPack32:
case vk::Format::eA2B10G10R10SnormPack32:
case vk::Format::eA2B10G10R10SscaledPack32:
case vk::Format::eA2B10G10R10UintPack32:
case vk::Format::eA2B10G10R10UnormPack32:
case vk::Format::eA2B10G10R10UscaledPack32:
case vk::Format::eA2R10G10B10SintPack32:
case vk::Format::eA2R10G10B10SnormPack32:
case vk::Format::eA2R10G10B10SscaledPack32:
case vk::Format::eA2R10G10B10UintPack32:
case vk::Format::eA2R10G10B10UnormPack32:
case vk::Format::eA2R10G10B10UscaledPack32:
case vk::Format::eA8B8G8R8SintPack32:
case vk::Format::eA8B8G8R8SnormPack32:
case vk::Format::eA8B8G8R8SrgbPack32:
case vk::Format::eA8B8G8R8SscaledPack32:
case vk::Format::eA8B8G8R8UintPack32:
case vk::Format::eA8B8G8R8UnormPack32:
case vk::Format::eA8B8G8R8UscaledPack32:
case vk::Format::eB10G11R11UfloatPack32:
case vk::Format::eB8G8R8A8Sint:
case vk::Format::eB8G8R8A8Snorm:
case vk::Format::eB8G8R8A8Srgb:
case vk::Format::eB8G8R8A8Sscaled:
case vk::Format::eB8G8R8A8Uint:
case vk::Format::eB8G8R8A8Unorm:
case vk::Format::eB8G8R8A8Uscaled:
case vk::Format::eE5B9G9R9UfloatPack32:
case vk::Format::eR10X6G10X6Unorm2Pack16:
case vk::Format::eR12X4G12X4Unorm2Pack16:
case vk::Format::eR16G16Sfloat:
case vk::Format::eR16G16Sint:
case vk::Format::eR16G16Snorm:
case vk::Format::eR16G16Sscaled:
case vk::Format::eR16G16Uint:
case vk::Format::eR16G16Unorm:
case vk::Format::eR16G16Uscaled:
case vk::Format::eR32Sfloat:
case vk::Format::eR32Sint:
case vk::Format::eR32Uint:
case vk::Format::eR8G8B8A8Sint:
case vk::Format::eR8G8B8A8Snorm:
case vk::Format::eR8G8B8A8Srgb:
case vk::Format::eR8G8B8A8Sscaled:
case vk::Format::eR8G8B8A8Uint:
case vk::Format::eR8G8B8A8Unorm:
case vk::Format::eR8G8B8A8Uscaled: return Bit32;
case vk::Format::eR16G16B16Sfloat:
case vk::Format::eR16G16B16Sint:
case vk::Format::eR16G16B16Snorm:
case vk::Format::eR16G16B16Sscaled:
case vk::Format::eR16G16B16Uint:
case vk::Format::eR16G16B16Unorm:
case vk::Format::eR16G16B16Uscaled: return Bit48;
case vk::Format::eR16G16B16A16Sfloat:
case vk::Format::eR16G16B16A16Sint:
case vk::Format::eR16G16B16A16Snorm:
case vk::Format::eR16G16B16A16Sscaled:
case vk::Format::eR16G16B16A16Uint:
case vk::Format::eR16G16B16A16Unorm:
case vk::Format::eR16G16B16A16Uscaled:
case vk::Format::eR32G32Sfloat:
case vk::Format::eR32G32Sint:
case vk::Format::eR32G32Uint:
case vk::Format::eR64Sfloat:
case vk::Format::eR64Sint:
case vk::Format::eR64Uint: return Bit64;
case vk::Format::eR32G32B32Sfloat:
case vk::Format::eR32G32B32Sint:
case vk::Format::eR32G32B32Uint: return Bit96;
case vk::Format::eR32G32B32A32Sfloat:
case vk::Format::eR32G32B32A32Sint:
case vk::Format::eR32G32B32A32Uint:
case vk::Format::eR64G64Sfloat:
case vk::Format::eR64G64Sint:
case vk::Format::eR64G64Uint: return Bit128;
case vk::Format::eR64G64B64Sfloat:
case vk::Format::eR64G64B64Sint:
case vk::Format::eR64G64B64Uint: return Bit192;
case vk::Format::eR64G64B64A64Sfloat:
case vk::Format::eR64G64B64A64Sint:
case vk::Format::eR64G64B64A64Uint: return Bit256;
case vk::Format::eBc1RgbSrgbBlock:
case vk::Format::eBc1RgbUnormBlock: return Bc1Rgb | Bit64;
case vk::Format::eBc1RgbaSrgbBlock:
case vk::Format::eBc1RgbaUnormBlock: return Bc1Rgba | Bit64;
case vk::Format::eBc2SrgbBlock:
case vk::Format::eBc2UnormBlock: return Bc2 | Bit128;
case vk::Format::eBc3SrgbBlock:
case vk::Format::eBc3UnormBlock: return Bc3 | Bit128;
case vk::Format::eBc4SnormBlock:
case vk::Format::eBc4UnormBlock: return Bc4 | Bit64;
case vk::Format::eBc5SnormBlock:
case vk::Format::eBc5UnormBlock: return Bc5 | Bit128;
case vk::Format::eBc6HSfloatBlock:
case vk::Format::eBc6HUfloatBlock: return Bc6h | Bit128;
case vk::Format::eBc7SrgbBlock:
case vk::Format::eBc7UnormBlock: return Bc7 | Bit128;
case vk::Format::eD16Unorm: return D16;
case vk::Format::eD16UnormS8Uint: return D16S8;
case vk::Format::eX8D24UnormPack32: return D24;
case vk::Format::eD24UnormS8Uint: return D24S8;
case vk::Format::eD32Sfloat: return D32;
case vk::Format::eD32SfloatS8Uint: return D32S8;
case vk::Format::eS8Uint: return S8;
default: return None;
}
}
} // namespace
vk::ImageAspectFlags DepthAspectMask(vk::Format format) {
switch (format) {
case vk::Format::eD16Unorm:
case vk::Format::eD32Sfloat: return vk::ImageAspectFlagBits::eDepth;
case vk::Format::eD16UnormS8Uint:
case vk::Format::eD24UnormS8Uint:
case vk::Format::eD32SfloatS8Uint:
return vk::ImageAspectFlagBits::eDepth | vk::ImageAspectFlagBits::eStencil;
default: EXIT("unsupported depth/stencil image format: %d\n", static_cast<int>(format));
}
}
bool FormatsCompatible(vk::Format base, vk::Format view) noexcept {
if (base == view) {
return true;
}
const auto base_class = FormatClass(base);
const auto view_class = FormatClass(view);
return view_class != None && (base_class & view_class) == view_class;
}
} // namespace ImageViewOps
vk::ImageView Image::FindView(const ImageViewInfo& view_info) {
const auto& image = backing;
auto normalized = view_info;
const bool is_storage =
static_cast<bool>(normalized.usage & vk::ImageUsageFlagBits::eStorage);
normalized.aspect = FullAspectMask(image.format);
if (normalized.aspect & vk::ImageAspectFlagBits::eDepth &&
IsDepthViewFormat(normalized.format)) {
normalized.format = image.format;
normalized.aspect = vk::ImageAspectFlagBits::eDepth;
}
if (normalized.aspect & vk::ImageAspectFlagBits::eStencil &&
IsStencilViewFormat(normalized.format)) {
normalized.format = image.format;
normalized.aspect = vk::ImageAspectFlagBits::eStencil;
}
normalized.usage =
is_storage ? vk::ImageUsageFlagBits::eStorage : vk::ImageUsageFlags {};
const bool format_compatible = normalized.format != vk::Format::eUndefined &&
IsCompatibleViewFormat(image.format, normalized.format);
const bool slice_view = image.image_type == vk::ImageType::e3D &&
(normalized.type == vk::ImageViewType::e2D ||
normalized.type == vk::ImageViewType::e2DArray);
const bool levels_valid = normalized.level_count != 0 &&
normalized.base_level < image.mip_levels &&
normalized.level_count <= image.mip_levels - normalized.base_level;
const auto view_layers = slice_view && levels_valid
? std::max(image.extent.depth >> normalized.base_level, 1u)
: image.layers;
const bool ranges_valid = levels_valid &&
normalized.layer_count != 0 && normalized.base_layer < view_layers &&
normalized.layer_count <= view_layers - normalized.base_layer;
const bool mapping_valid =
IsComponentSwizzle(normalized.mapping.r) && IsComponentSwizzle(normalized.mapping.g) &&
IsComponentSwizzle(normalized.mapping.b) && IsComponentSwizzle(normalized.mapping.a);
if (image.image == nullptr || !format_compatible || !ranges_valid || !mapping_valid ||
!IsValidViewType(image, normalized) ||
!IsValidAspect(image, normalized.aspect)) {
EXIT("invalid image view: image_format=%d view_format=%d type=%d aspect=0x%x "
"mip=%u+%u layer=%u+%u usage=0x%x image_levels=%u image_layers=%u\n",
static_cast<int>(image.format), static_cast<int>(normalized.format),
static_cast<int>(normalized.type),
static_cast<vk::ImageAspectFlags::MaskType>(normalized.aspect), normalized.base_level,
normalized.level_count, normalized.base_layer, normalized.layer_count,
static_cast<vk::ImageUsageFlags::MaskType>(normalized.usage), image.mip_levels,
image.layers);
}
std::lock_guard lock(views.mutex);
for (const auto& cached: views.views) {
if (cached.info == normalized) {
return cached.view;
}
}
vk::ImageViewUsageCreateInfo usage {};
usage.sType = vk::StructureType::eImageViewUsageCreateInfo;
usage.usage = image.usage;
if (!is_storage) {
usage.usage &= ~vk::ImageUsageFlagBits::eStorage;
}
vk::ImageViewCreateInfo create {};
create.sType = vk::StructureType::eImageViewCreateInfo;
create.pNext = &usage;
create.image = image.image;
create.viewType = normalized.type;
create.format = normalized.format;
create.components = normalized.mapping;
create.subresourceRange.aspectMask = normalized.aspect;
create.subresourceRange.baseMipLevel = normalized.base_level;
create.subresourceRange.levelCount = normalized.level_count;
create.subresourceRange.baseArrayLayer = normalized.base_layer;
create.subresourceRange.layerCount = normalized.layer_count;
vk::ImageView view = nullptr;
const auto result = m_graphics->device.createImageView(&create, nullptr, &view);
if (result != vk::Result::eSuccess || view == nullptr) {
EXIT("failed to create image view: result=%d image_format=%d view_format=%d type=%d "
"aspect=0x%x mip=%u+%u layer=%u+%u usage=0x%x\n",
static_cast<int>(result), static_cast<int>(image.format),
static_cast<int>(view_info.format), static_cast<int>(view_info.type),
static_cast<vk::ImageAspectFlags::MaskType>(view_info.aspect), view_info.base_level,
view_info.level_count, view_info.base_layer, view_info.layer_count,
static_cast<vk::ImageUsageFlags::MaskType>(view_info.usage));
}
views.views.push_back({normalized, view});
return view;
}
} // namespace Libs::Graphics
@@ -0,0 +1,141 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
#include "common/assert.h"
#include "graphics/host_gpu/renderer/image/imageInfo.h"
#include "graphics/shader/recompiler/ir/ShaderIR.h"
#include "graphics/shader/shader.h"
namespace Libs::Graphics {
namespace ImageViewOps {
[[nodiscard]] vk::ImageAspectFlags DepthAspectMask(vk::Format format);
[[nodiscard]] bool FormatsCompatible(vk::Format base, vk::Format view) noexcept;
} // namespace ImageViewOps
[[nodiscard]] inline bool IsValidImageSwizzle(uint32_t swizzle) noexcept {
if ((swizzle & ~0xfffu) != 0) {
return false;
}
for (uint32_t channel = 0; channel < 4; channel++) {
switch (GetDstSel(swizzle, channel)) {
case 0:
case 1:
case 4:
case 5:
case 6:
case 7: break;
default: return false;
}
}
return true;
}
[[noreturn]] inline void UnsupportedColorView(const char* usage, vk::Format image_format,
vk::Format view_format, uint32_t swizzle) noexcept {
EXIT("unsupported %s color image view: image_format=%d view_format=%d swizzle=0x%03x\n", usage,
static_cast<int>(image_format), static_cast<int>(view_format), swizzle);
}
[[nodiscard]] inline vk::Format SrgbStorageViewFormat(vk::Format image_format) noexcept {
switch (image_format) {
case vk::Format::eR8G8B8A8Srgb:
case vk::Format::eB8G8R8A8Srgb: return vk::Format::eR8G8B8A8Unorm;
default: return vk::Format::eUndefined;
}
}
[[nodiscard]] inline bool IsSupportedSampledColorView(vk::Format image_format,
vk::Format view_format,
uint32_t swizzle) noexcept {
return IsValidImageSwizzle(swizzle) &&
ImageViewOps::FormatsCompatible(image_format, view_format);
}
[[nodiscard]] inline uint32_t
SelectSampledColorView(vk::Format image_format, vk::Format view_format, uint32_t swizzle) noexcept {
if (IsSupportedSampledColorView(image_format, view_format, swizzle)) {
return swizzle;
}
UnsupportedColorView("sampled", image_format, view_format, swizzle);
}
[[nodiscard]] inline bool IsSupportedSampledDepthView(vk::Format image_format,
vk::Format view_format,
uint32_t swizzle) noexcept {
if (!IsSupportedSampledDepthFormat(image_format, view_format)) {
return false;
}
switch (swizzle) {
case DstSel(4, 4, 4, 4):
case DstSel(4, 0, 0, 0):
case DstSel(4, 0, 0, 1): return true;
default: return false;
}
}
[[nodiscard]] inline uint32_t
SelectSampledDepthView(vk::Format image_format, vk::Format view_format, uint32_t swizzle) noexcept {
if (IsSupportedSampledDepthView(image_format, view_format, swizzle)) {
return swizzle;
}
EXIT("unsupported sampled depth image view: image_format=%d view_format=%d swizzle=0x%03x\n",
static_cast<int>(image_format), static_cast<int>(view_format), swizzle);
}
[[nodiscard]] inline bool
IsSupportedSampledDepthResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return resource.kind == ShaderRecompiler::IR::ResourceKind::Image &&
(resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray) &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.read &&
!resource.written && !resource.atomic;
}
[[nodiscard]] inline bool
IsSupportedSampledDepthUintResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint &&
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.read &&
!resource.written && !resource.atomic && !resource.depth_compare;
}
inline void ValidateStorageColorView(vk::Format image_format, vk::Format view_format,
uint32_t swizzle) noexcept {
if (!ImageViewOps::FormatsCompatible(image_format, view_format) ||
!IsValidImageSwizzle(swizzle)) {
UnsupportedColorView("storage", image_format, view_format, swizzle);
}
}
[[nodiscard]] inline bool
IsSupportedStorageImageResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return (resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImage ||
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint) &&
(resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim1D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim1DArray ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim3D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray) &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.written &&
(!resource.atomic ||
(resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint &&
resource.read)) &&
!resource.depth_compare;
}
inline void
ValidateStorageImageResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
if (!IsSupportedStorageImageResource(resource)) {
EXIT("unsupported storage color image resource: kind=%u dimension=%u mip=%u "
"read=%d written=%d atomic=%d depth_compare=%d\n",
static_cast<uint32_t>(resource.kind), static_cast<uint32_t>(resource.dimension),
static_cast<uint32_t>(resource.mip_mode), resource.read, resource.written,
resource.atomic, resource.depth_compare);
}
}
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
@@ -0,0 +1,637 @@
#include "graphics/host_gpu/renderer/image/textureCommon.h"
#include "common/assert.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/gpu_format.h"
#include "graphics/host_gpu/renderer/image/tiler.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <algorithm>
#include <bit>
#include <cinttypes>
#include <cstring>
namespace Libs::Graphics {
namespace {
struct RenderTargetFormatMapping {
Prospero::ChannelLayout layout;
Prospero::ChannelType type;
Prospero::ChannelOrder order;
RenderTargetFormatInfo info;
};
constexpr RenderTargetFormatMapping kRenderTargetFormats[] = {
{Prospero::ChannelLayout::k8_8,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR8G8Unorm, 2}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR8G8B8A8Unorm, 4}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kSNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR8G8B8A8Snorm, 4}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kSrgb,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR8G8B8A8Srgb, 4}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kAlt,
{vk::Format::eB8G8R8A8Unorm, 4}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kSNorm,
Prospero::ChannelOrder::kAlt,
{vk::Format::eB8G8R8A8Snorm, 4}},
{Prospero::ChannelLayout::k8_8_8_8,
Prospero::ChannelType::kSrgb,
Prospero::ChannelOrder::kAlt,
{vk::Format::eB8G8R8A8Srgb, 4}},
{Prospero::ChannelLayout::k5_5_5_1,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR5G5B5A1UnormPack16, 2}},
{Prospero::ChannelLayout::k4_4_4_4,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kReversed,
{vk::Format::eB4G4R4A4UnormPack16, 2}},
{Prospero::ChannelLayout::k10_10_10_2,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eA2B10G10R10UnormPack32, 4}},
{Prospero::ChannelLayout::k10_10_10_2,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kAlt,
{vk::Format::eA2R10G10B10UnormPack32, 4}},
{Prospero::ChannelLayout::k11_11_10,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eB10G11R11UfloatPack32, 4}},
{Prospero::ChannelLayout::k16,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16Unorm, 2}},
{Prospero::ChannelLayout::k16,
Prospero::ChannelType::kUInt,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16Uint, 2}},
{Prospero::ChannelLayout::k16,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16Sfloat, 2}},
{Prospero::ChannelLayout::k16_16,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16Unorm, 4}},
{Prospero::ChannelLayout::k16_16,
Prospero::ChannelType::kSNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16Snorm, 4}},
{Prospero::ChannelLayout::k16_16,
Prospero::ChannelType::kUInt,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16Uint, 4}},
{Prospero::ChannelLayout::k16_16,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16Sfloat, 4}},
{Prospero::ChannelLayout::k16_16_16_16,
Prospero::ChannelType::kUNorm,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16B16A16Unorm, 8}},
{Prospero::ChannelLayout::k16_16_16_16,
Prospero::ChannelType::kUInt,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16B16A16Uint, 8}},
{Prospero::ChannelLayout::k16_16_16_16,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR16G16B16A16Sfloat, 8}},
{Prospero::ChannelLayout::k16_16_16_16,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kAlt,
{vk::Format::eR16G16B16A16Sfloat, 8, Prospero::ColorMappingBgra}},
{Prospero::ChannelLayout::k16_16_16_16,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kReversed,
{vk::Format::eR16G16B16A16Sfloat, 8, Prospero::ColorMappingAbgr}},
{Prospero::ChannelLayout::k32,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR32Sfloat, 4}},
{Prospero::ChannelLayout::k32_32,
Prospero::ChannelType::kUInt,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR32G32Uint, 8}},
{Prospero::ChannelLayout::k32_32,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR32G32Sfloat, 8}},
{Prospero::ChannelLayout::k32_32_32_32,
Prospero::ChannelType::kFloat,
Prospero::ChannelOrder::kStandard,
{vk::Format::eR32G32B32A32Sfloat, 16}},
};
} // namespace
// TODO: cleanup!
RenderTargetFormatInfo TextureGetRenderTargetFormat(uint32_t raw_layout, uint32_t raw_type,
uint32_t raw_order) {
const auto layout = static_cast<Prospero::ChannelLayout>(raw_layout);
const auto type = static_cast<Prospero::ChannelType>(raw_type);
const auto order = static_cast<Prospero::ChannelOrder>(raw_order);
if (layout == Prospero::ChannelLayout::k8 && type == Prospero::ChannelType::kUNorm &&
raw_order <= Prospero::GpuEnumValue(Prospero::ChannelOrder::kAltReversed)) {
return {vk::Format::eR8Unorm, 1};
}
for (const auto& mapping: kRenderTargetFormats) {
if (mapping.layout == layout && mapping.type == type && mapping.order == order) {
return mapping.info;
}
}
EXIT("unsupported render-target format combination: layout=%u type=%u order=%u\n", raw_layout,
raw_type, raw_order);
}
namespace {
static uint64_t GetLevelSrcOffset(const TileSizeOffset& level_size) {
return (level_size.src_size != 0 ? level_size.src_offset : level_size.offset);
}
static uint64_t GetLevelSrcSize(const TileSizeOffset& level_size) {
return (level_size.src_size != 0 ? level_size.src_size : level_size.size);
}
static uint32_t GetTextureLevelDepth(uint32_t depth, uint32_t level, bool volume_texture) {
return volume_texture ? std::max(depth >> level, 1u) : depth;
}
static size_t GetTextureRegionCount(uint32_t depth, uint64_t levels, bool volume_texture) {
size_t count = 0;
for (uint32_t level = 0; level < levels; level++) {
count += GetTextureLevelDepth(depth, level, volume_texture);
}
return count;
}
uint64_t TextureUploadSliceSourceOffset(const TextureUploadLayout& layout, uint32_t level,
uint32_t slice) {
if (level >= 16 || layout.level_sizes[level].size == 0) {
EXIT("invalid texture upload slice source, level=%u slice=%u\n", level, slice);
}
const auto level_offset = GetLevelSrcOffset(layout.level_sizes[level]);
const auto slice_stride =
layout.source_slice_stride != 0 ? layout.source_slice_stride : layout.slice_stride;
if (slice_stride != 0 && slice > (UINT64_MAX - level_offset) / slice_stride) {
EXIT("texture upload slice source offset overflow, level=%u slice=%u\n", level, slice);
}
return level_offset + static_cast<uint64_t>(slice) * slice_stride;
}
vk::ComponentSwizzle TextureGetComponentSwizzle(uint8_t s) {
switch (static_cast<Prospero::CompSwizzle>(s)) {
case Prospero::CompSwizzle::kZero: return vk::ComponentSwizzle::eZero;
case Prospero::CompSwizzle::kOne: return vk::ComponentSwizzle::eOne;
case Prospero::CompSwizzle::kRed: return vk::ComponentSwizzle::eR;
case Prospero::CompSwizzle::kGreen: return vk::ComponentSwizzle::eG;
case Prospero::CompSwizzle::kBlue: return vk::ComponentSwizzle::eB;
case Prospero::CompSwizzle::kAlpha: return vk::ComponentSwizzle::eA;
default: EXIT("unknown swizzle: %d\n", static_cast<int>(s));
}
return vk::ComponentSwizzle::eIdentity;
}
static uint32_t TextureGetDstSel(uint32_t swizzle, uint32_t channel) {
return (swizzle >> (channel * 3u)) & 0x7u;
}
} // namespace
vk::ComponentMapping TextureGetComponentMapping(uint32_t swizzle) {
vk::ComponentMapping components {};
components.r = TextureGetComponentSwizzle(static_cast<uint8_t>(TextureGetDstSel(swizzle, 0)));
components.g = TextureGetComponentSwizzle(static_cast<uint8_t>(TextureGetDstSel(swizzle, 1)));
components.b = TextureGetComponentSwizzle(static_cast<uint8_t>(TextureGetDstSel(swizzle, 2)));
components.a = TextureGetComponentSwizzle(static_cast<uint8_t>(TextureGetDstSel(swizzle, 3)));
return components;
}
vk::Format TextureGetFormat(uint32_t fmt) {
const auto vk_format = VulkanFormat(fmt);
if (vk_format != vk::Format::eUndefined) {
return vk_format;
}
EXIT("unknown format: fmt = %u\n", fmt);
return vk::Format::eUndefined;
}
namespace {
uint64_t CalcTextureSliceStride(const TileSizeOffset* level_sizes, uint64_t levels,
uint64_t total_size, uint32_t depth) {
uint64_t stride = 0;
for (uint32_t i = 0; i < levels; i++) {
stride =
std::max(stride, static_cast<uint64_t>(level_sizes[i].offset) + level_sizes[i].size);
}
if (depth > 1 && total_size != 0 && total_size % depth == 0) {
const auto guest_stride = total_size / depth;
if (guest_stride >= stride) {
stride = guest_stride;
}
}
return stride;
}
uint64_t CalcLinearUploadLevelSize(uint32_t fmt, uint32_t pitch, uint32_t height) {
if (const uint32_t bytes_per_element = Prospero::NumBytesPerElement(fmt);
bytes_per_element != 0) {
return static_cast<uint64_t>(pitch) * height * bytes_per_element;
}
if (const uint32_t bytes_per_block = Prospero::BlockCompressedBytesPerBlock(fmt);
bytes_per_block != 0) {
const uint32_t blocks_w = std::max((pitch + 3u) / 4u, 1u);
const uint32_t blocks_h = std::max((height + 3u) / 4u, 1u);
return static_cast<uint64_t>(blocks_w) * blocks_h * bytes_per_block;
}
return 0;
}
uint64_t SetLinearUploadLevels(TileSizeOffset* level_sizes, uint32_t fmt, uint64_t height,
uint64_t levels, uint32_t base_pitch) {
uint64_t offset = 0;
auto pitch = base_pitch;
auto h = static_cast<uint32_t>(height);
for (uint32_t i = 0; i < levels; i++) {
const auto size = CalcLinearUploadLevelSize(fmt, pitch, h);
EXIT_NOT_IMPLEMENTED(size > 0xffffffffull);
EXIT_NOT_IMPLEMENTED(offset > 0xffffffffull);
level_sizes[i].size = static_cast<uint32_t>(size);
level_sizes[i].offset = static_cast<uint32_t>(offset);
level_sizes[i].src_size = 0;
level_sizes[i].src_offset = 0;
level_sizes[i].x = 0;
level_sizes[i].y = 0;
offset += size;
if (pitch > 1) {
pitch /= 2;
}
if (h > 1) {
h /= 2;
}
}
return offset;
}
} // namespace
TextureUploadLayout TextureCalcUploadLayout(uint32_t fmt, uint64_t width, uint64_t height,
uint64_t levels, uint32_t depth, uint64_t pitch,
uint64_t tile, uint64_t upload_size,
bool allow_depth_tile, bool volume_texture,
const char* owner) {
TextureUploadLayout layout {};
layout.tile = static_cast<uint32_t>(tile);
layout.pitch = static_cast<uint32_t>(pitch);
layout.texel_block = Prospero::BlockCompressedBytesPerBlock(fmt) != 0 ? 4u : 1u;
layout.volume_texture = volume_texture;
if (fmt != 0) {
if (layout.tile != 0) {
const auto tile_mode = static_cast<Prospero::TileMode>(layout.tile);
switch (tile_mode) {
case Prospero::TileMode::kStandard256B:
if (TileIsStandard256BTextureSupported(fmt)) {
layout.tile_family = TileBlockFamily::Standard256B;
}
break;
case Prospero::TileMode::kStandard4KB:
if (TileIsStandard4KBTextureSupported(fmt)) {
layout.tile_family = TileBlockFamily::Standard4KB;
}
break;
case Prospero::TileMode::kStandard64KB:
if (TileIsStandard64KBTextureSupported(fmt)) {
layout.tile_family = TileBlockFamily::Standard64KB;
}
break;
case Prospero::TileMode::kPrt:
if (TileIsStandard64KBTextureSupported(fmt)) {
layout.tile_family = TileBlockFamily::Prt64KB;
}
break;
case Prospero::TileMode::kRenderTarget:
if (Prospero::RenderTargetBytesPerElement(fmt) != 0) {
layout.tile_family = TileBlockFamily::RenderTarget64KB;
}
break;
case Prospero::TileMode::kDepth:
if (allow_depth_tile && Prospero::RenderTargetBytesPerElement(fmt) != 0) {
layout.tile_family = TileBlockFamily::Depth64KB;
}
break;
default: break;
}
if (layout.tile_family == TileBlockFamily::Count) {
EXIT("%s: unsupported typed tiled upload: fmt=%u tile=%u "
"size=%" PRIu64 " extent=%" PRIu64 "x%" PRIu64 " pitch=%" PRIu64
" levels=%" PRIu64 "\n",
owner, static_cast<uint32_t>(fmt), layout.tile, upload_size, width, height,
pitch, levels);
}
}
layout.pitch = TileGetTexturePitch(fmt, width, levels, layout.tile);
TileGetTextureSize(fmt, width, height, layout.pitch, levels, layout.tile, nullptr,
layout.level_sizes, layout.padded_sizes);
if (static_cast<Prospero::TileMode>(layout.tile) != Prospero::TileMode::kLinear) {
if (layout.volume_texture) {
layout.slice_stride = SetLinearUploadLevels(layout.level_sizes, fmt, height, levels,
static_cast<uint32_t>(width));
} else {
TileSizeOffset tiled_levels[16] {};
std::copy_n(layout.level_sizes, levels, tiled_levels);
layout.source_slice_stride =
CalcTextureSliceStride(tiled_levels, levels, upload_size, depth);
SetLinearUploadLevels(layout.level_sizes, fmt, height, levels, layout.pitch);
for (uint32_t i = 0; i < levels; ++i) {
if (tiled_levels[i].src_size > tiled_levels[i].size) {
layout.first_tail_level = std::min(layout.first_tail_level, i);
}
layout.level_sizes[i].src_offset = GetLevelSrcOffset(tiled_levels[i]);
layout.level_sizes[i].src_size = GetLevelSrcSize(tiled_levels[i]);
layout.level_sizes[i].x = tiled_levels[i].x;
layout.level_sizes[i].y = tiled_levels[i].y;
}
}
} else if (layout.volume_texture) {
layout.slice_stride =
CalcTextureSliceStride(layout.level_sizes, levels, upload_size, depth);
}
} else {
EXIT("%s: legacy texture upload format unsupported: fmt=0 tile=%u size=%" PRIu64
" extent=%" PRIu64 "x%" PRIu64 " pitch=%" PRIu64 " levels=%" PRIu64 "\n",
owner, layout.tile, upload_size, width, height, pitch, levels);
}
if (!layout.volume_texture) {
layout.slice_stride =
CalcTextureSliceStride(layout.level_sizes, levels, upload_size, depth);
}
return layout;
}
std::vector<vk::BufferImageCopy>
TextureBuildImageCopies(const TextureUploadLayout& layout, uint32_t width, uint32_t height,
uint32_t depth, uint64_t levels, bool array_texture,
bool volume_texture) {
uint32_t mip_width = width;
uint32_t mip_height = height;
uint32_t mip_pitch = volume_texture && static_cast<Prospero::TileMode>(layout.tile) !=
Prospero::TileMode::kLinear
? width
: layout.pitch;
std::vector<vk::BufferImageCopy> regions;
regions.reserve(GetTextureRegionCount(depth, levels, volume_texture));
for (uint32_t i = 0; i < levels; i++) {
EXIT_NOT_IMPLEMENTED(layout.level_sizes[i].size == 0);
const auto mip_depth = GetTextureLevelDepth(depth, i, volume_texture);
for (uint32_t z = 0; z < mip_depth; z++) {
const auto slice_offset = z * layout.slice_stride;
vk::BufferImageCopy region {};
region.bufferOffset =
layout.level_sizes[i].offset + slice_offset;
region.imageSubresource = {vk::ImageAspectFlagBits::eColor, i,
array_texture ? z : 0, 1};
region.imageOffset.z = volume_texture ? static_cast<int>(z) : 0;
region.imageExtent = {mip_width, mip_height, 1};
const bool linear =
static_cast<Prospero::TileMode>(layout.tile) == Prospero::TileMode::kLinear;
if (linear) {
region.bufferRowLength = layout.padded_sizes[i].width;
region.bufferImageHeight = layout.padded_sizes[i].height;
} else {
const auto align = [](uint32_t value, uint32_t block) {
return ((value + block - 1u) / block) * block;
};
const auto pitch = align(mip_pitch, layout.texel_block);
region.bufferRowLength =
pitch > align(mip_width, layout.texel_block) ? pitch : 0;
}
regions.push_back(region);
}
if (mip_width > 1) {
mip_width /= 2;
}
if (mip_height > 1) {
mip_height /= 2;
}
if (mip_pitch > 1) {
mip_pitch /= 2;
}
}
return regions;
}
struct GpuTileElementLayout {
uint32_t bytes = 0;
uint32_t wide = 1;
uint32_t tall = 1;
};
static bool GetGpuTileElementLayout(uint32_t fmt, GpuTileElementLayout& out) {
if (const auto bytes = Prospero::NumBytesPerElement(fmt); bytes != 0) {
out = {bytes, 1, 1};
return true;
}
if (const auto bytes = Prospero::BlockCompressedBytesPerBlock(fmt); bytes != 0) {
out = {bytes, 4, 4};
return true;
}
return false;
}
static bool SetGpuTileSize(uint64_t offset, uint64_t length, uint64_t capacity, uint64_t& size) {
if (offset > capacity || length > capacity - offset) {
return false;
}
size = length;
return true;
}
bool TextureBuildGpuTileInfos(uint64_t size,
const std::vector<vk::BufferImageCopy>& regions,
const TextureUploadLayout& layout, uint32_t fmt, uint32_t depth,
uint64_t levels, std::vector<GpuTileInfo>& out_infos) {
if (size == 0 || levels == 0 || levels > 16 || depth == 0 ||
regions.size() != GetTextureRegionCount(depth, levels, layout.volume_texture) ||
Prospero::IsFmaskTextureFormat(fmt)) {
return false;
}
GpuTileElementLayout element {};
if (layout.tile_family == TileBlockFamily::RenderTarget64KB ||
layout.tile_family == TileBlockFamily::Depth64KB) {
element.bytes = Prospero::RenderTargetBytesPerElement(fmt);
} else if (!GetGpuTileElementLayout(fmt, element)) {
return false;
}
if (element.bytes == 0) {
return false;
}
std::vector<GpuTileInfo> infos;
infos.reserve(regions.size());
if (layout.volume_texture) {
TileVolumeLayout volume {};
if (!TileGetTextureVolumeLayout(fmt, regions[0].imageExtent.width,
regions[0].imageExtent.height, depth,
static_cast<uint32_t>(levels), layout.tile, volume)) {
return false;
}
element = {volume.bytes_per_element, volume.texel_width, volume.texel_height};
TileBlockLayout block {};
if (!TileGetBlockLayout(volume.family, element.bytes, block)) return false;
size_t region_base = 0;
for (uint32_t level = 0; level < levels; ++level) {
const uint32_t mip_depth = GetTextureLevelDepth(depth, level, true);
const bool tail = level >= volume.first_tail_level;
const uint64_t linear_stride = layout.slice_stride;
for (uint32_t z = 0; z < mip_depth; z += block.block_depth) {
const uint32_t copy_depth = std::min(block.block_depth, mip_depth - z);
const auto& region = regions[region_base + z];
const auto pitch = region.bufferRowLength != 0
? region.bufferRowLength
: region.imageExtent.width;
const auto logical_height = region.bufferImageHeight != 0
? region.bufferImageHeight
: region.imageExtent.height;
GpuTileInfo info {};
info.family = block.family;
info.bytes_per_element = block.bytes_per_element;
info.linear_offset = region.bufferOffset;
info.tiled_offset =
static_cast<uint64_t>(z / block.block_depth) * volume.block_slice_size +
volume.level_offsets[level];
const uint64_t linear_span =
static_cast<uint64_t>(copy_depth - 1u) * linear_stride +
layout.level_sizes[level].size;
if (!SetGpuTileSize(info.linear_offset, linear_span, size, info.linear_size) ||
!SetGpuTileSize(info.tiled_offset, volume.level_sizes[level], size,
info.tiled_size)) {
return false;
}
info.linear_slice_stride = linear_stride;
info.width = std::max(
(region.imageExtent.width + element.wide - 1u) / element.wide, 1u);
info.height = std::max(
(logical_height + element.tall - 1u) / element.tall, 1u);
info.depth = copy_depth;
info.surface_z = block.block_depth == 1
? static_cast<uint32_t>(region.imageOffset.z)
: 0;
info.pitch =
std::max((pitch + element.wide - 1u) / element.wide, 1u);
info.tail_x = tail ? volume.tail_x[level] : 0;
info.tail_y = tail ? volume.tail_y[level] : 0;
info.tail = tail;
info.tiled_width = volume.level_widths[level];
info.tiled_height = volume.level_heights[level];
infos.push_back(info);
}
region_base += mip_depth;
}
} else {
const auto base_family = layout.tile_family;
if (base_family == TileBlockFamily::Count) {
return false;
}
size_t region_index = 0;
for (uint32_t level = 0; level < levels; level++) {
const auto& level_size = layout.level_sizes[level];
const bool tail = level >= layout.first_tail_level;
const auto family = base_family;
TileBlockLayout block {};
if (!TileGetBlockLayout(family, element.bytes, block)) {
return false;
}
const auto level_depth = GetTextureLevelDepth(depth, level, layout.volume_texture);
for (uint32_t z = 0; z < level_depth; z++) {
const auto& region = regions[region_index++];
const auto pitch = region.bufferRowLength != 0
? region.bufferRowLength
: region.imageExtent.width;
const auto logical_height = region.bufferImageHeight != 0
? region.bufferImageHeight
: region.imageExtent.height;
GpuTileInfo info {};
info.family = block.family;
info.bytes_per_element = block.bytes_per_element;
info.linear_offset = region.bufferOffset;
info.tiled_offset = TextureUploadSliceSourceOffset(layout, level, z);
if (!SetGpuTileSize(info.linear_offset, level_size.size, size, info.linear_size) ||
!SetGpuTileSize(info.tiled_offset, GetLevelSrcSize(level_size), size,
info.tiled_size)) {
return false;
}
info.width = std::max(
(region.imageExtent.width + element.wide - 1u) / element.wide, 1u);
info.height = std::max(
(logical_height + element.tall - 1u) / element.tall, 1u);
info.surface_z = base_family == TileBlockFamily::RenderTarget64KB ||
base_family == TileBlockFamily::Depth64KB
? region.imageSubresource.baseArrayLayer
: 0;
info.pitch =
std::max((pitch + element.wide - 1u) / element.wide, 1u);
info.tail = tail;
info.tail_x = tail ? level_size.x : 0;
info.tail_y = tail ? level_size.y : 0;
info.tiled_width =
layout.padded_sizes[level].width != 0
? std::max((layout.padded_sizes[level].width + element.wide - 1u) /
element.wide,
1u)
: info.pitch;
info.tiled_height =
layout.padded_sizes[level].height != 0
? std::max((layout.padded_sizes[level].height + element.tall - 1u) /
element.tall,
1u)
: info.height;
infos.push_back(info);
}
}
}
if (infos.empty()) {
return false;
}
out_infos = std::move(infos);
return true;
}
} // namespace Libs::Graphics
@@ -0,0 +1,54 @@
#ifndef EMULATOR_INCLUDE_EMULATOR_GRAPHICS_HOST_GPU_RENDERER_IMAGE_TEXTURECOMMON_H_
#define EMULATOR_INCLUDE_EMULATOR_GRAPHICS_HOST_GPU_RENDERER_IMAGE_TEXTURECOMMON_H_
#include "common/abi.h"
#include "common/common.h"
#include "graphics/guest_gpu/gpu_defs.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <vector>
namespace Libs::Graphics {
struct GpuTileInfo;
struct RenderTargetFormatInfo {
vk::Format format = vk::Format::eUndefined;
uint32_t bytes_per_element = 0;
Prospero::ColorComponentMapping export_mapping;
};
struct TextureUploadLayout {
uint32_t tile = 0;
uint32_t pitch = 0;
uint32_t texel_block = 1;
uint64_t slice_stride = 0;
uint64_t source_slice_stride = 0;
uint32_t first_tail_level = 16;
TileBlockFamily tile_family = TileBlockFamily::Count;
bool volume_texture = false;
TileSizeOffset level_sizes[16] = {};
TilePaddedSize padded_sizes[16] = {};
};
vk::ComponentMapping TextureGetComponentMapping(uint32_t swizzle);
vk::Format TextureGetFormat(uint32_t fmt);
RenderTargetFormatInfo TextureGetRenderTargetFormat(uint32_t layout, uint32_t type, uint32_t order);
TextureUploadLayout TextureCalcUploadLayout(uint32_t fmt, uint64_t width, uint64_t height,
uint64_t levels, uint32_t depth, uint64_t pitch,
uint64_t tile, uint64_t upload_size,
bool allow_depth_tile, bool volume_texture,
const char* owner);
std::vector<vk::BufferImageCopy>
TextureBuildImageCopies(const TextureUploadLayout& layout, uint32_t width, uint32_t height,
uint32_t depth, uint64_t levels, bool array_texture,
bool volume_texture);
bool TextureBuildGpuTileInfos(uint64_t size,
const std::vector<vk::BufferImageCopy>& regions,
const TextureUploadLayout& layout, uint32_t fmt, uint32_t depth,
uint64_t levels, std::vector<GpuTileInfo>& infos);
} // namespace Libs::Graphics
#endif /* EMULATOR_INCLUDE_EMULATOR_GRAPHICS_HOST_GPU_RENDERER_IMAGE_TEXTURECOMMON_H_ */
@@ -0,0 +1,718 @@
#include "graphics/host_gpu/renderer/image/tiler.h"
#include "common/assert.h"
#include "gpu_tiler_shaders/gpu_tiler_demote_d16_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_depth_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_promote_d16_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_prt_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_prt_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_render_target_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard256_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard4_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard4_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard64_3d_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_standard64_spv.h"
#include "gpu_tiler_shaders/gpu_tiler_swap_bgra16_spv.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/commandScheduler.h"
#include "graphics/host_gpu/renderer/image/image.h"
#include "graphics/host_gpu/renderer/cache/streamBuffer.h"
#include <algorithm>
#include <array>
#include <bit>
#include <cstring>
#include <limits>
namespace Libs::Graphics {
TileManager::TileManager(GraphicContext& graphics, CommandScheduler& scheduler,
StreamBuffer& stream_buffer)
: m_graphics(graphics), m_scheduler(scheduler), m_stream_buffer(stream_buffer) {
static_assert(FamilyCount == 9);
static_assert(sizeof(Push) == 52);
std::array<vk::DescriptorSetLayoutBinding, 3> bindings {};
for (uint32_t index = 0; index < 2; index++) {
bindings[index] = {index, vk::DescriptorType::eStorageBuffer, 1,
vk::ShaderStageFlagBits::eCompute, nullptr};
}
bindings[2] = {2, vk::DescriptorType::eUniformBuffer, 1, vk::ShaderStageFlagBits::eCompute,
nullptr};
vk::DescriptorSetLayoutCreateInfo descriptor_info {};
descriptor_info.sType = vk::StructureType::eDescriptorSetLayoutCreateInfo;
descriptor_info.flags = vk::DescriptorSetLayoutCreateFlagBits::ePushDescriptorKHR;
descriptor_info.bindingCount = static_cast<uint32_t>(bindings.size());
descriptor_info.pBindings = bindings.data();
RequireVulkanSuccess(m_graphics.device.createDescriptorSetLayout(&descriptor_info, nullptr,
&m_descriptor_layout),
"create TileManager descriptor layout");
const vk::PushConstantRange push_range {vk::ShaderStageFlagBits::eCompute, 0, sizeof(Push)};
vk::PipelineLayoutCreateInfo layout_info {};
layout_info.sType = vk::StructureType::ePipelineLayoutCreateInfo;
layout_info.setLayoutCount = 1;
layout_info.pSetLayouts = &m_descriptor_layout;
layout_info.pushConstantRangeCount = 1;
layout_info.pPushConstantRanges = &push_range;
RequireVulkanSuccess(
m_graphics.device.createPipelineLayout(&layout_info, nullptr, &m_pipeline_layout),
"create TileManager pipeline layout");
}
TileManager::~TileManager() {
for (auto pipeline: m_pipelines) {
if (pipeline != nullptr) {
m_graphics.device.destroyPipeline(pipeline, nullptr);
}
}
if (m_d16_to_d24 != nullptr) {
m_graphics.device.destroyPipeline(m_d16_to_d24, nullptr);
}
if (m_d16_to_d32 != nullptr) {
m_graphics.device.destroyPipeline(m_d16_to_d32, nullptr);
}
if (m_d24_to_d16 != nullptr) {
m_graphics.device.destroyPipeline(m_d24_to_d16, nullptr);
}
if (m_d32_to_d16 != nullptr) {
m_graphics.device.destroyPipeline(m_d32_to_d16, nullptr);
}
if (m_swap_bgra16 != nullptr) {
m_graphics.device.destroyPipeline(m_swap_bgra16, nullptr);
}
if (m_pipeline_layout != nullptr) {
m_graphics.device.destroyPipelineLayout(m_pipeline_layout, nullptr);
}
if (m_descriptor_layout != nullptr) {
m_graphics.device.destroyDescriptorSetLayout(m_descriptor_layout, nullptr);
}
}
TileManager::Scratch TileManager::AllocateScratch(uint64_t size) {
EXIT_IF(size == 0);
vk::BufferCreateInfo create {};
create.sType = vk::StructureType::eBufferCreateInfo;
create.size = size;
create.usage = vk::BufferUsageFlagBits::eStorageBuffer | vk::BufferUsageFlagBits::eTransferSrc |
vk::BufferUsageFlagBits::eTransferDst;
create.sharingMode = vk::SharingMode::eExclusive;
VmaAllocationCreateInfo allocate {};
allocate.usage = VMA_MEMORY_USAGE_AUTO_PREFER_DEVICE;
VkBuffer buffer = VK_NULL_HANDLE;
VmaAllocation memory = nullptr;
const auto raw = static_cast<VkBufferCreateInfo>(create);
RequireVulkanSuccess(static_cast<vk::Result>(vmaCreateBuffer(
m_graphics.allocator, &raw, &allocate, &buffer, &memory, nullptr)),
"allocate TileManager scratch buffer");
return {buffer, memory, size};
}
void TileManager::DeferDestroy(Scratch scratch) {
auto allocator = m_graphics.allocator;
m_scheduler.DeferOperation(
[allocator, scratch] { vmaDestroyBuffer(allocator, scratch.buffer, scratch.allocation); });
}
void TileManager::Prepare(bool tile, uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos, uint64_t source_base,
uint64_t target_base, std::vector<Dispatch>& dispatches) {
EXIT_IF(infos.empty() || tiled_capacity == 0 || linear_capacity == 0);
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
EXIT_NOT_IMPLEMENTED(tiled_capacity > UINT32_MAX || linear_capacity > UINT32_MAX);
const auto checked_multiply = [](uint64_t left, uint64_t right, uint64_t& result) {
return (left == 0 || right <= UINT64_MAX / left) && (result = left * right, true);
};
const auto checked_add = [](uint64_t left, uint64_t right, uint64_t& result) {
return right <= UINT64_MAX - left && (result = left + right, true);
};
const auto valid_range = [](uint64_t offset, uint64_t size, uint64_t capacity) {
return size != 0 && offset <= capacity && size <= capacity - offset;
};
dispatches.clear();
dispatches.reserve(infos.size());
for (const auto& info: infos) {
TileBlockLayout block {};
const uint32_t tiled_width = info.tiled_width != 0 ? info.tiled_width : info.pitch;
const uint32_t tiled_height = info.tiled_height != 0 ? info.tiled_height : info.height;
const uint64_t groups_x = (static_cast<uint64_t>(info.width) + 7u) / 8u;
const uint64_t groups_y = (static_cast<uint64_t>(info.height) + 7u) / 8u;
EXIT_NOT_IMPLEMENTED(
!TileGetBlockLayout(info.family, info.bytes_per_element, block) || info.width == 0 ||
info.height == 0 || info.depth == 0 || info.pitch < info.width ||
groups_x > limits.maxComputeWorkGroupCount[0] ||
groups_y > limits.maxComputeWorkGroupCount[1] ||
info.depth > limits.maxComputeWorkGroupCount[2] ||
(!info.tail && (tiled_width < info.width || tiled_height < info.height)) ||
!valid_range(info.linear_offset, info.linear_size, linear_capacity) ||
!valid_range(info.tiled_offset, info.tiled_size, tiled_capacity) ||
(block.block_depth == 1 && info.depth != 1));
uint64_t pitch_bytes = 0;
EXIT_NOT_IMPLEMENTED(!checked_multiply(info.pitch, info.bytes_per_element, pitch_bytes) ||
pitch_bytes > UINT32_MAX);
uint64_t slice_bytes = info.linear_slice_stride;
uint64_t minimum_slice = 0;
EXIT_NOT_IMPLEMENTED(!checked_multiply(pitch_bytes, info.height, minimum_slice));
if (slice_bytes == 0) {
slice_bytes = minimum_slice;
}
uint64_t linear_used = 0;
uint64_t bytes = 0;
EXIT_NOT_IMPLEMENTED((info.depth > 1 && slice_bytes < minimum_slice) ||
!checked_multiply(info.depth - 1u, slice_bytes, bytes) ||
!checked_add(linear_used, bytes, linear_used) ||
!checked_multiply(info.height - 1u, pitch_bytes, bytes) ||
!checked_add(linear_used, bytes, linear_used) ||
!checked_multiply(info.width, info.bytes_per_element, bytes) ||
!checked_add(linear_used, bytes, linear_used) ||
linear_used > info.linear_size || slice_bytes > UINT32_MAX);
const uint64_t columns =
(static_cast<uint64_t>(tiled_width) + block.block_width - 1u) / block.block_width;
const uint64_t rows =
(static_cast<uint64_t>(tiled_height) + block.block_height - 1u) / block.block_height;
uint64_t blocks_per_slice = 0;
EXIT_NOT_IMPLEMENTED(!checked_multiply(columns, rows, blocks_per_slice) ||
columns > UINT32_MAX || blocks_per_slice > UINT32_MAX);
if (info.tail) {
EXIT_NOT_IMPLEMENTED(
info.family == TileBlockFamily::Standard256B || info.depth > block.block_depth ||
info.tail_x >= block.block_width || info.width > block.block_width - info.tail_x ||
info.tail_y >= block.block_height ||
info.height > block.block_height - info.tail_y ||
info.tiled_size < block.block_size);
} else {
const uint64_t slices =
(static_cast<uint64_t>(info.depth) + block.block_depth - 1u) / block.block_depth;
uint64_t tiled_used = 0;
EXIT_NOT_IMPLEMENTED(!checked_multiply(blocks_per_slice, slices, tiled_used) ||
!checked_multiply(tiled_used, block.block_size, tiled_used) ||
tiled_used > info.tiled_size);
}
const uint32_t alignment = std::min(info.bytes_per_element, 4u);
EXIT_NOT_IMPLEMENTED(((info.linear_offset | info.tiled_offset | pitch_bytes | slice_bytes) &
(alignment - 1u)) != 0);
const uint64_t src = source_base + (tile ? info.linear_offset : info.tiled_offset);
const uint64_t dst = target_base + (tile ? info.tiled_offset : info.linear_offset);
EXIT_NOT_IMPLEMENTED(src > UINT32_MAX || dst > UINT32_MAX);
const uint32_t family_index = static_cast<uint32_t>(info.family);
const uint32_t element_index = std::countr_zero(info.bytes_per_element);
EXIT_NOT_IMPLEMENTED(family_index >= FamilyCount || element_index >= BytesPerElementCount);
Dispatch dispatch {};
dispatch.pipeline_slot =
((tile ? FamilyCount : 0u) + family_index) * BytesPerElementCount + element_index;
dispatch.push.src_base = static_cast<uint32_t>(src);
dispatch.push.dst_base = static_cast<uint32_t>(dst);
dispatch.push.width = info.width;
dispatch.push.height = info.height;
dispatch.push.depth = info.depth;
dispatch.push.surface_z = info.surface_z;
dispatch.push.pitch_bytes = static_cast<uint32_t>(pitch_bytes);
dispatch.push.slice_bytes = static_cast<uint32_t>(slice_bytes);
dispatch.push.blocks_per_row = static_cast<uint32_t>(columns);
dispatch.push.blocks_per_slice = static_cast<uint32_t>(blocks_per_slice);
dispatch.push.tail_x = info.tail_x;
dispatch.push.tail_y = info.tail_y;
dispatch.push.tail = info.tail;
dispatches.push_back(dispatch);
}
const uint64_t uniform_alignment =
std::max<uint64_t>(limits.minUniformBufferOffsetAlignment, 1);
const uint64_t stride = (sizeof(Push) + uniform_alignment - 1) & ~(uniform_alignment - 1);
EXIT_NOT_IMPLEMENTED(dispatches.size() > UINT64_MAX / stride);
const uint64_t bytes = dispatches.size() * stride;
auto [mapped, offset] = m_stream_buffer.Map(bytes, uniform_alignment);
EXIT_IF(mapped == nullptr);
for (size_t index = 0; index < dispatches.size(); index++) {
std::memcpy(mapped + index * stride, &dispatches[index].push, sizeof(Push));
dispatches[index].params_offset = offset + index * stride;
}
m_stream_buffer.Commit();
}
vk::Pipeline TileManager::GetPipeline(uint32_t slot) {
EXIT_IF(slot >= m_pipelines.size());
if (m_pipelines[slot] != nullptr) {
return m_pipelines[slot];
}
struct Shader {
const uint32_t* code;
size_t words;
};
static constexpr std::array<Shader, FamilyCount> shaders {{
{GPU_TILER_STANDARD256_SPV, std::size(GPU_TILER_STANDARD256_SPV)},
{GPU_TILER_STANDARD4_SPV, std::size(GPU_TILER_STANDARD4_SPV)},
{GPU_TILER_STANDARD4_3D_SPV, std::size(GPU_TILER_STANDARD4_3D_SPV)},
{GPU_TILER_STANDARD64_SPV, std::size(GPU_TILER_STANDARD64_SPV)},
{GPU_TILER_STANDARD64_3D_SPV, std::size(GPU_TILER_STANDARD64_3D_SPV)},
{GPU_TILER_PRT_SPV, std::size(GPU_TILER_PRT_SPV)},
{GPU_TILER_PRT_3D_SPV, std::size(GPU_TILER_PRT_3D_SPV)},
{GPU_TILER_RENDER_TARGET_SPV, std::size(GPU_TILER_RENDER_TARGET_SPV)},
{GPU_TILER_DEPTH_SPV, std::size(GPU_TILER_DEPTH_SPV)},
}};
const uint32_t element_index = slot % BytesPerElementCount;
const uint32_t direction_index = slot / (FamilyCount * BytesPerElementCount);
const uint32_t family_index = (slot / BytesPerElementCount) % FamilyCount;
const uint32_t values[] {1u << element_index, direction_index};
const vk::SpecializationMapEntry entries[] {{0, 0, 4}, {1, 4, 4}};
const vk::SpecializationInfo specialization {2, entries, sizeof(values), values};
vk::ShaderModuleCreateInfo module_info {};
module_info.sType = vk::StructureType::eShaderModuleCreateInfo;
module_info.codeSize = shaders[family_index].words * sizeof(uint32_t);
module_info.pCode = shaders[family_index].code;
vk::ShaderModule module = nullptr;
RequireVulkanSuccess(m_graphics.device.createShaderModule(&module_info, nullptr, &module),
"create TileManager shader module");
vk::PipelineShaderStageCreateInfo stage {};
stage.sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stage.stage = vk::ShaderStageFlagBits::eCompute;
stage.module = module;
stage.pName = "main";
stage.pSpecializationInfo = &specialization;
vk::ComputePipelineCreateInfo create {};
create.sType = vk::StructureType::eComputePipelineCreateInfo;
create.stage = stage;
create.layout = m_pipeline_layout;
const auto result =
m_graphics.device.createComputePipelines(nullptr, 1, &create, nullptr, &m_pipelines[slot]);
m_graphics.device.destroyShaderModule(module, nullptr);
RequireVulkanSuccess(result, "create TileManager pipeline");
return m_pipelines[slot];
}
void TileManager::Record(bool tile, vk::Buffer source, uint64_t source_offset,
uint64_t source_capacity, vk::Buffer target, uint64_t target_offset,
uint64_t target_capacity, std::span<Dispatch> dispatches,
bool clear_target) {
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const uint64_t descriptor_alignment =
std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const uint64_t source_descriptor_offset = source_offset & ~(descriptor_alignment - 1);
const uint64_t target_descriptor_offset = target_offset & ~(descriptor_alignment - 1);
const uint64_t source_base = source_offset - source_descriptor_offset;
const uint64_t target_base = target_offset - target_descriptor_offset;
const uint64_t source_range = (source_base + source_capacity + 3u) & ~uint64_t {3};
const uint64_t target_range = (target_base + target_capacity + 3u) & ~uint64_t {3};
EXIT_NOT_IMPLEMENTED(source_range > limits.maxStorageBufferRange ||
target_range > limits.maxStorageBufferRange || target_offset % 4 != 0 ||
target_capacity % 4 != 0);
m_scheduler.EndRendering();
auto command = m_scheduler.Current().Handle();
vk::BufferMemoryBarrier barriers[3] {};
barriers[0].sType = vk::StructureType::eBufferMemoryBarrier;
barriers[0].srcAccessMask = vk::AccessFlagBits::eMemoryWrite | vk::AccessFlagBits::eHostWrite;
barriers[0].dstAccessMask = vk::AccessFlagBits::eShaderRead;
barriers[0].srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[0].dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[0].buffer = source;
barriers[0].offset = source_offset;
barriers[0].size = source_capacity;
barriers[1] = barriers[0];
barriers[1].srcAccessMask = vk::AccessFlagBits::eMemoryWrite | vk::AccessFlagBits::eHostWrite;
barriers[1].dstAccessMask =
clear_target ? vk::AccessFlagBits::eTransferWrite
: vk::AccessFlagBits::eShaderRead | vk::AccessFlagBits::eShaderWrite;
barriers[1].buffer = target;
barriers[1].offset = target_offset;
barriers[1].size = target_capacity;
barriers[2] = barriers[0];
barriers[2].srcAccessMask = vk::AccessFlagBits::eHostWrite;
barriers[2].dstAccessMask = vk::AccessFlagBits::eUniformRead;
barriers[2].buffer = m_stream_buffer.Handle();
barriers[2].offset = dispatches.front().params_offset;
barriers[2].size =
dispatches.back().params_offset - dispatches.front().params_offset + sizeof(Push);
command.pipelineBarrier(
vk::PipelineStageFlagBits::eAllCommands | vk::PipelineStageFlagBits::eHost,
vk::PipelineStageFlagBits::eComputeShader | vk::PipelineStageFlagBits::eTransfer, {}, 0,
nullptr, 3, barriers, 0, nullptr);
if (clear_target) {
command.fillBuffer(target, target_offset, target_capacity, 0);
barriers[1].srcAccessMask = vk::AccessFlagBits::eTransferWrite;
barriers[1].dstAccessMask =
vk::AccessFlagBits::eShaderRead | vk::AccessFlagBits::eShaderWrite;
command.pipelineBarrier(vk::PipelineStageFlagBits::eTransfer,
vk::PipelineStageFlagBits::eComputeShader, {}, 0, nullptr, 1,
&barriers[1], 0, nullptr);
}
const vk::DescriptorBufferInfo source_info {source, source_descriptor_offset, source_range};
const vk::DescriptorBufferInfo target_info {target, target_descriptor_offset, target_range};
for (auto& dispatch: dispatches) {
const vk::DescriptorBufferInfo params_info {m_stream_buffer.Handle(),
dispatch.params_offset, sizeof(Push)};
const vk::DescriptorBufferInfo infos[] {source_info, target_info, params_info};
std::array<vk::WriteDescriptorSet, 3> writes {};
for (uint32_t index = 0; index < writes.size(); index++) {
writes[index].sType = vk::StructureType::eWriteDescriptorSet;
writes[index].dstBinding = index;
writes[index].descriptorCount = 1;
writes[index].descriptorType = index == 2 ? vk::DescriptorType::eUniformBuffer
: vk::DescriptorType::eStorageBuffer;
writes[index].pBufferInfo = &infos[index];
}
command.pushDescriptorSetKHR(vk::PipelineBindPoint::eCompute, m_pipeline_layout, 0,
static_cast<uint32_t>(writes.size()), writes.data());
command.bindPipeline(vk::PipelineBindPoint::eCompute, GetPipeline(dispatch.pipeline_slot));
command.dispatch((dispatch.push.width + 7u) / 8u, (dispatch.push.height + 7u) / 8u,
dispatch.push.depth);
}
barriers[1].srcAccessMask = vk::AccessFlagBits::eShaderWrite;
barriers[1].dstAccessMask = vk::AccessFlagBits::eTransferRead | vk::AccessFlagBits::eMemoryRead;
command.pipelineBarrier(vk::PipelineStageFlagBits::eComputeShader,
vk::PipelineStageFlagBits::eAllCommands, {}, 0, nullptr, 1,
&barriers[1], 0, nullptr);
}
TileManager::Result TileManager::Detile(vk::Buffer tiled, uint64_t tiled_offset,
uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos) {
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const uint64_t descriptor_alignment =
std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const uint64_t source_base = tiled_offset & (descriptor_alignment - 1);
std::vector<Dispatch> dispatches;
Prepare(false, tiled_capacity, linear_capacity, infos, source_base, 0, dispatches);
auto scratch = AllocateScratch((linear_capacity + 3u) & ~uint64_t {3});
DeferDestroy(scratch);
Record(false, tiled, tiled_offset, tiled_capacity, scratch.buffer, 0, scratch.size, dispatches,
true);
return {scratch.buffer, 0, linear_capacity};
}
void TileManager::Tile(vk::Buffer linear, uint64_t linear_offset, uint64_t linear_capacity,
vk::Buffer tiled, uint64_t tiled_offset, uint64_t tiled_capacity,
std::span<const GpuTileInfo> infos) {
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const uint64_t descriptor_alignment =
std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const uint64_t source_base = linear_offset & (descriptor_alignment - 1);
const uint64_t target_base = tiled_offset & (descriptor_alignment - 1);
std::vector<Dispatch> dispatches;
Prepare(true, tiled_capacity, linear_capacity, infos, source_base, target_base, dispatches);
Record(true, linear, linear_offset, linear_capacity, tiled, tiled_offset, tiled_capacity,
dispatches, false);
}
void TileManager::TileImage(Image& image, std::span<const vk::BufferImageCopy> regions,
vk::Buffer tiled, uint64_t tiled_offset, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos,
ColorTransform transform) {
EXIT_IF(regions.empty());
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const uint64_t descriptor_alignment =
std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const uint64_t target_base = tiled_offset & (descriptor_alignment - 1);
std::vector<Dispatch> dispatches;
// Reserve all stream parameters before creating a scheduler-lived scratch dependency:
// StreamBuffer::Map is allowed to submit the current tick when it wraps.
Prepare(true, tiled_capacity, linear_capacity, infos, 0, target_base, dispatches);
auto linear = AllocateScratch((linear_capacity + 3u) & ~uint64_t {3});
DeferDestroy(linear);
image.Download(regions, linear.buffer, 0, linear.size);
Result source {linear.buffer, 0, linear.size};
if (transform == ColorTransform::SwapBgra16) {
source = SwapBgra16(source);
}
Record(true, source.buffer, source.offset, linear_capacity, tiled, tiled_offset, tiled_capacity,
dispatches, false);
}
TileManager::Result TileManager::GetScratchBuffer(uint64_t size) {
auto scratch = AllocateScratch((size + 3u) & ~uint64_t {3});
DeferDestroy(scratch);
return {scratch.buffer, 0, scratch.size};
}
TileManager::StorageBinding TileManager::BindStorage(Result buffer, uint64_t size) const {
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const auto alignment = std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const auto descriptor_offset = buffer.offset - buffer.offset % alignment;
const auto base = buffer.offset - descriptor_offset;
EXIT_IF(buffer.buffer == nullptr || size == 0 || buffer.size < size || base > UINT32_MAX ||
size > UINT64_MAX - base || base + size > UINT64_MAX - 3);
const auto range = (base + size + 3) & ~uint64_t {3};
EXIT_IF(range > limits.maxStorageBufferRange || range > UINT32_MAX);
return {{buffer.buffer, descriptor_offset, range}, static_cast<uint32_t>(base)};
}
uint32_t TileManager::ConversionRows(uint64_t offset, uint64_t row_stride, uint64_t active,
uint32_t remaining, uint64_t alignment, uint64_t max_range,
uint32_t max_groups) noexcept {
if (row_stride == 0 || active == 0 || remaining == 0 || alignment == 0 || max_groups == 0) {
return 0;
}
const auto prefix = offset % alignment;
if (prefix >= max_range || active > max_range - prefix) {
return 0;
}
const auto descriptor_rows = 1 + (max_range - prefix - active) / row_stride;
return static_cast<uint32_t>(std::min<uint64_t>({remaining, descriptor_rows, max_groups}));
}
void TileManager::ConvertD16(Result source, Result target, D16Direction direction, bool d32,
const D16Layout& layout) {
vk::Pipeline* pipeline_pointer = nullptr;
if (direction == D16Direction::Promote) {
pipeline_pointer = d32 ? &m_d16_to_d32 : &m_d16_to_d24;
} else {
pipeline_pointer = d32 ? &m_d32_to_d16 : &m_d24_to_d16;
}
auto& pipeline = *pipeline_pointer;
if (pipeline == nullptr) {
const uint32_t value = d32 ? 1u : 0u;
const vk::SpecializationMapEntry entry {0, 0, sizeof(value)};
const vk::SpecializationInfo specialization {1, &entry, sizeof(value), &value};
const uint32_t* code = nullptr;
size_t words = 0;
if (direction == D16Direction::Promote) {
code = GPU_TILER_PROMOTE_D16_SPV;
words = std::size(GPU_TILER_PROMOTE_D16_SPV);
} else {
code = GPU_TILER_DEMOTE_D16_SPV;
words = std::size(GPU_TILER_DEMOTE_D16_SPV);
}
vk::ShaderModuleCreateInfo module_info {};
module_info.sType = vk::StructureType::eShaderModuleCreateInfo;
module_info.codeSize = words * sizeof(uint32_t);
module_info.pCode = code;
vk::ShaderModule module = nullptr;
RequireVulkanSuccess(m_graphics.device.createShaderModule(&module_info, nullptr, &module),
"create D16 conversion shader module");
vk::PipelineShaderStageCreateInfo stage {};
stage.sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stage.stage = vk::ShaderStageFlagBits::eCompute;
stage.module = module;
stage.pName = "main";
stage.pSpecializationInfo = &specialization;
vk::ComputePipelineCreateInfo create {};
create.sType = vk::StructureType::eComputePipelineCreateInfo;
create.stage = stage;
create.layout = m_pipeline_layout;
const auto result =
m_graphics.device.createComputePipelines(nullptr, 1, &create, nullptr, &pipeline);
m_graphics.device.destroyShaderModule(module, nullptr);
RequireVulkanSuccess(result, "create D16 conversion pipeline");
}
const uint64_t source_element =
direction == D16Direction::Promote ? sizeof(uint16_t) : sizeof(uint32_t);
const uint64_t target_element =
direction == D16Direction::Promote ? sizeof(uint32_t) : sizeof(uint16_t);
const uint64_t source_active = static_cast<uint64_t>(layout.width) * source_element;
const uint64_t target_active = static_cast<uint64_t>(layout.width) * target_element;
const auto required = [](uint32_t height, uint32_t layers, uint64_t row_stride,
uint64_t slice_stride, uint64_t active) {
EXIT_IF(height == 0 || layers == 0 || row_stride < active ||
(height - 1) > (UINT64_MAX - active) / row_stride);
const auto slice = static_cast<uint64_t>(height - 1) * row_stride + active;
EXIT_IF(slice_stride < slice || (layers - 1) > (UINT64_MAX - slice) / slice_stride);
return static_cast<uint64_t>(layers - 1) * slice_stride + slice;
};
EXIT_IF(layout.width == 0 || layout.source_row_stride > UINT32_MAX ||
layout.target_row_stride > UINT32_MAX);
const auto source_required = required(layout.height, layout.layers, layout.source_row_stride,
layout.source_slice_stride, source_active);
const auto target_required = required(layout.height, layout.layers, layout.target_row_stride,
layout.target_slice_stride, target_active);
EXIT_IF(source_required > UINT64_MAX - 3 || target_required > UINT64_MAX - 3);
const auto source_barrier_size = (source_required + 3) & ~uint64_t {3};
const auto target_barrier_size = (target_required + 3) & ~uint64_t {3};
EXIT_IF(source.size < source_barrier_size || target.size < target_barrier_size);
m_scheduler.EndRendering();
auto command = m_scheduler.Current().Handle();
vk::BufferMemoryBarrier barriers[2] {};
barriers[0].sType = vk::StructureType::eBufferMemoryBarrier;
barriers[0].srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[0].dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[0].buffer = source.buffer;
barriers[0].offset = source.offset;
barriers[0].size = source_barrier_size;
barriers[0].srcAccessMask = vk::AccessFlagBits::eMemoryWrite | vk::AccessFlagBits::eHostWrite |
vk::AccessFlagBits::eTransferWrite |
vk::AccessFlagBits::eShaderWrite;
barriers[0].dstAccessMask = vk::AccessFlagBits::eShaderRead;
barriers[1].sType = vk::StructureType::eBufferMemoryBarrier;
barriers[1].srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[1].dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[1].buffer = target.buffer;
barriers[1].offset = target.offset;
barriers[1].size = target_barrier_size;
barriers[1].srcAccessMask = vk::AccessFlagBits::eMemoryRead | vk::AccessFlagBits::eMemoryWrite |
vk::AccessFlagBits::eHostWrite |
vk::AccessFlagBits::eTransferWrite |
vk::AccessFlagBits::eShaderWrite;
barriers[1].dstAccessMask = vk::AccessFlagBits::eShaderRead | vk::AccessFlagBits::eShaderWrite;
command.pipelineBarrier(
vk::PipelineStageFlagBits::eAllCommands | vk::PipelineStageFlagBits::eHost,
vk::PipelineStageFlagBits::eComputeShader, {}, 0, nullptr, 2, barriers, 0, nullptr);
command.bindPipeline(vk::PipelineBindPoint::eCompute, pipeline);
const auto& limits = m_graphics.GetPhysicalDeviceProperties().limits;
const auto descriptor_alignment = std::max<uint64_t>(limits.minStorageBufferOffsetAlignment, 4);
const auto rows_for = [&](Result buffer, uint64_t relative, uint64_t stride, uint64_t active,
uint32_t remaining) {
EXIT_IF(relative > buffer.size || buffer.offset > UINT64_MAX - relative);
const auto offset = buffer.offset + relative;
return ConversionRows(offset, stride, active, remaining, descriptor_alignment,
limits.maxStorageBufferRange, limits.maxComputeWorkGroupCount[1]);
};
const auto groups_x = (static_cast<uint64_t>(layout.width) + 63u) / 64u;
EXIT_IF(groups_x == 0 || groups_x > limits.maxComputeWorkGroupCount[0]);
for (uint32_t layer = 0; layer < layout.layers; layer++) {
for (uint32_t row = 0; row < layout.height;) {
const auto source_relative =
layout.source_slice_stride * layer + layout.source_row_stride * row;
const auto target_relative =
layout.target_slice_stride * layer + layout.target_row_stride * row;
const auto remaining = layout.height - row;
const auto rows = std::min(rows_for(source, source_relative, layout.source_row_stride,
source_active, remaining),
rows_for(target, target_relative, layout.target_row_stride,
target_active, remaining));
EXIT_IF(rows == 0);
const auto source_span =
static_cast<uint64_t>(rows - 1) * layout.source_row_stride + source_active;
const auto target_span =
static_cast<uint64_t>(rows - 1) * layout.target_row_stride + target_active;
const auto source_binding = BindStorage(
{source.buffer, source.offset + source_relative, source.size - source_relative},
source_span);
const auto target_binding = BindStorage(
{target.buffer, target.offset + target_relative, target.size - target_relative},
target_span);
const vk::DescriptorBufferInfo infos[] {
source_binding.info,
target_binding.info,
};
std::array<vk::WriteDescriptorSet, 2> writes {};
for (uint32_t index = 0; index < writes.size(); index++) {
writes[index].sType = vk::StructureType::eWriteDescriptorSet;
writes[index].dstBinding = index;
writes[index].descriptorCount = 1;
writes[index].descriptorType = vk::DescriptorType::eStorageBuffer;
writes[index].pBufferInfo = &infos[index];
}
command.pushDescriptorSetKHR(vk::PipelineBindPoint::eCompute, m_pipeline_layout, 0,
static_cast<uint32_t>(writes.size()), writes.data());
Push push {};
push.src_base = source_binding.base;
push.dst_base = target_binding.base;
push.width = layout.width;
push.height = rows;
push.pitch_bytes = static_cast<uint32_t>(layout.source_row_stride);
push.slice_bytes = static_cast<uint32_t>(layout.target_row_stride);
command.pushConstants(m_pipeline_layout, vk::ShaderStageFlagBits::eCompute, 0,
sizeof(push), &push);
command.dispatch(static_cast<uint32_t>(groups_x), rows, 1);
row += rows;
}
}
barriers[1].srcAccessMask = vk::AccessFlagBits::eShaderWrite;
barriers[1].dstAccessMask = vk::AccessFlagBits::eTransferRead | vk::AccessFlagBits::eMemoryRead;
command.pipelineBarrier(vk::PipelineStageFlagBits::eComputeShader,
vk::PipelineStageFlagBits::eAllCommands, {}, 0, nullptr, 1,
&barriers[1], 0, nullptr);
}
void TileManager::SwapBgra16(Result input, Result output, uint32_t pixels) {
if (m_swap_bgra16 == nullptr) {
vk::ShaderModuleCreateInfo module_info {};
module_info.sType = vk::StructureType::eShaderModuleCreateInfo;
module_info.codeSize = std::size(GPU_TILER_SWAP_BGRA16_SPV) * sizeof(uint32_t);
module_info.pCode = GPU_TILER_SWAP_BGRA16_SPV;
vk::ShaderModule module = nullptr;
RequireVulkanSuccess(m_graphics.device.createShaderModule(&module_info, nullptr, &module),
"create BGRA16 swap shader module");
vk::PipelineShaderStageCreateInfo stage {};
stage.sType = vk::StructureType::ePipelineShaderStageCreateInfo;
stage.stage = vk::ShaderStageFlagBits::eCompute;
stage.module = module;
stage.pName = "main";
vk::ComputePipelineCreateInfo create {};
create.sType = vk::StructureType::eComputePipelineCreateInfo;
create.stage = stage;
create.layout = m_pipeline_layout;
const auto result =
m_graphics.device.createComputePipelines(nullptr, 1, &create, nullptr, &m_swap_bgra16);
m_graphics.device.destroyShaderModule(module, nullptr);
RequireVulkanSuccess(result, "create BGRA16 swap pipeline");
}
const uint64_t bytes = static_cast<uint64_t>(pixels) * 8u;
EXIT_IF(pixels == 0);
const auto input_binding = BindStorage(input, bytes);
const auto output_binding = BindStorage(output, bytes);
const vk::DescriptorBufferInfo infos[] {
input_binding.info,
output_binding.info,
};
std::array<vk::WriteDescriptorSet, 2> writes {};
for (uint32_t index = 0; index < writes.size(); index++) {
writes[index].sType = vk::StructureType::eWriteDescriptorSet;
writes[index].dstBinding = index;
writes[index].descriptorCount = 1;
writes[index].descriptorType = vk::DescriptorType::eStorageBuffer;
writes[index].pBufferInfo = &infos[index];
}
vk::BufferMemoryBarrier barriers[2] {};
for (uint32_t index = 0; index < 2; index++) {
barriers[index].sType = vk::StructureType::eBufferMemoryBarrier;
barriers[index].srcQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[index].dstQueueFamilyIndex = VK_QUEUE_FAMILY_IGNORED;
barriers[index].buffer = infos[index].buffer;
barriers[index].offset = infos[index].offset;
barriers[index].size = infos[index].range;
}
barriers[0].srcAccessMask = vk::AccessFlagBits::eMemoryWrite | vk::AccessFlagBits::eHostWrite |
vk::AccessFlagBits::eShaderWrite;
barriers[0].dstAccessMask = vk::AccessFlagBits::eShaderRead;
barriers[1].srcAccessMask = vk::AccessFlagBits::eMemoryRead;
barriers[1].dstAccessMask = vk::AccessFlagBits::eShaderWrite;
m_scheduler.EndRendering();
auto command = m_scheduler.Current().Handle();
command.pipelineBarrier(
vk::PipelineStageFlagBits::eAllCommands | vk::PipelineStageFlagBits::eHost,
vk::PipelineStageFlagBits::eComputeShader, {}, 0, nullptr, 2, barriers, 0, nullptr);
command.bindPipeline(vk::PipelineBindPoint::eCompute, m_swap_bgra16);
command.pushDescriptorSetKHR(vk::PipelineBindPoint::eCompute, m_pipeline_layout, 0,
static_cast<uint32_t>(writes.size()), writes.data());
Push push {};
push.src_base = input_binding.base;
push.dst_base = output_binding.base;
push.width = pixels;
command.pushConstants(m_pipeline_layout, vk::ShaderStageFlagBits::eCompute, 0, sizeof(push),
&push);
command.dispatch((pixels + 63u) / 64u, 1, 1);
barriers[1].srcAccessMask = vk::AccessFlagBits::eShaderWrite;
barriers[1].dstAccessMask = vk::AccessFlagBits::eTransferRead;
command.pipelineBarrier(vk::PipelineStageFlagBits::eComputeShader,
vk::PipelineStageFlagBits::eTransfer, {}, 0, nullptr, 1, &barriers[1],
0, nullptr);
}
TileManager::Result TileManager::SwapBgra16(Result input) {
EXIT_NOT_IMPLEMENTED(input.size == 0 || input.size % 8u != 0 || input.size / 8u > UINT32_MAX);
auto output = AllocateScratch(input.size);
DeferDestroy(output);
Result result {output.buffer, 0, output.size};
SwapBgra16(input, result, static_cast<uint32_t>(input.size / 8u));
return result;
}
void TileManager::SwapBgra16(Result input, Result output) {
EXIT_NOT_IMPLEMENTED(input.size == 0 || input.size % 8u != 0 || input.size / 8u > UINT32_MAX ||
output.size < input.size);
SwapBgra16(input, output, static_cast<uint32_t>(input.size / 8u));
}
} // namespace Libs::Graphics
@@ -0,0 +1,149 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TILER_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TILER_H_
#include "common/common.h"
#include "graphics/guest_gpu/tile.h"
#include "graphics/host_gpu/vulkanCommon.h"
#include <array>
#include <span>
#include <vector>
#include <vk_mem_alloc.h>
namespace Libs::Graphics {
class CommandScheduler;
class Image;
class StreamBuffer;
struct GraphicContext;
struct TileManagerTestAccess;
struct GpuTileInfo {
TileBlockFamily family = TileBlockFamily::Count;
uint32_t bytes_per_element = 0;
uint64_t linear_offset = 0;
uint64_t linear_size = 0;
uint64_t tiled_offset = 0;
uint64_t tiled_size = 0;
uint64_t linear_slice_stride = 0;
uint32_t width = 0;
uint32_t height = 0;
uint32_t depth = 1;
uint32_t pitch = 0;
uint32_t tail_x = 0;
uint32_t tail_y = 0;
bool tail = false;
uint32_t tiled_width = 0;
uint32_t tiled_height = 0;
uint32_t surface_z = 0;
};
class TileManager final {
public:
enum class D16Direction { Promote, Demote };
enum class ColorTransform { None, SwapBgra16 };
struct Result {
vk::Buffer buffer = nullptr;
uint64_t offset = 0;
uint64_t size = 0;
};
struct D16Layout {
uint32_t width = 0;
uint32_t height = 0;
uint32_t layers = 0;
uint64_t source_row_stride = 0;
uint64_t target_row_stride = 0;
uint64_t source_slice_stride = 0;
uint64_t target_slice_stride = 0;
};
TileManager(GraphicContext& graphics, CommandScheduler& scheduler, StreamBuffer& stream_buffer);
~TileManager();
KYTY_CLASS_NO_COPY(TileManager);
// The returned device-local buffer remains alive through the current scheduler tick.
[[nodiscard]] Result Detile(vk::Buffer tiled, uint64_t tiled_offset, uint64_t tiled_capacity,
uint64_t linear_capacity, std::span<const GpuTileInfo> infos);
void Tile(vk::Buffer linear, uint64_t linear_offset, uint64_t linear_capacity, vk::Buffer tiled,
uint64_t tiled_offset, uint64_t tiled_capacity, std::span<const GpuTileInfo> infos);
void TileImage(Image& image, std::span<const vk::BufferImageCopy> regions, vk::Buffer tiled,
uint64_t tiled_offset, uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos,
ColorTransform transform = ColorTransform::None);
[[nodiscard]] Result GetScratchBuffer(uint64_t size);
void ConvertD16(Result source, Result target, D16Direction direction, bool d32,
const D16Layout& layout);
[[nodiscard]] Result SwapBgra16(Result input);
void SwapBgra16(Result input, Result output);
private:
friend struct TileManagerTestAccess;
static constexpr uint32_t FamilyCount = static_cast<uint32_t>(TileBlockFamily::Count);
static constexpr uint32_t BytesPerElementCount = 5;
static constexpr uint32_t DirectionCount = 2;
static constexpr uint32_t PipelineCount = FamilyCount * BytesPerElementCount * DirectionCount;
struct Push {
uint32_t src_base;
uint32_t dst_base;
uint32_t width;
uint32_t height;
uint32_t depth;
uint32_t surface_z;
uint32_t pitch_bytes;
uint32_t slice_bytes;
uint32_t blocks_per_row;
uint32_t blocks_per_slice;
uint32_t tail_x;
uint32_t tail_y;
uint32_t tail;
};
struct Dispatch {
Push push {};
uint32_t pipeline_slot = 0;
uint64_t params_offset = 0;
};
struct Scratch {
vk::Buffer buffer = nullptr;
VmaAllocation allocation = nullptr;
uint64_t size = 0;
};
struct StorageBinding {
vk::DescriptorBufferInfo info;
uint32_t base = 0;
};
[[nodiscard]] Scratch AllocateScratch(uint64_t size);
[[nodiscard]] StorageBinding BindStorage(Result buffer, uint64_t size) const;
[[nodiscard]] static uint32_t ConversionRows(uint64_t offset, uint64_t row_stride,
uint64_t active, uint32_t remaining,
uint64_t alignment, uint64_t max_range,
uint32_t max_groups) noexcept;
void DeferDestroy(Scratch scratch);
void Prepare(bool tile, uint64_t tiled_capacity, uint64_t linear_capacity,
std::span<const GpuTileInfo> infos, uint64_t source_base, uint64_t target_base,
std::vector<Dispatch>& dispatches);
void Record(bool tile, vk::Buffer source, uint64_t source_offset, uint64_t source_capacity,
vk::Buffer target, uint64_t target_offset, uint64_t target_capacity,
std::span<Dispatch> dispatches, bool clear_target);
[[nodiscard]] vk::Pipeline GetPipeline(uint32_t slot);
void SwapBgra16(Result input, Result output, uint32_t pixels);
GraphicContext& m_graphics;
CommandScheduler& m_scheduler;
StreamBuffer& m_stream_buffer;
vk::DescriptorSetLayout m_descriptor_layout = nullptr;
vk::PipelineLayout m_pipeline_layout = nullptr;
std::array<vk::Pipeline, PipelineCount> m_pipelines {};
vk::Pipeline m_d16_to_d24 = nullptr;
vk::Pipeline m_d16_to_d32 = nullptr;
vk::Pipeline m_d24_to_d16 = nullptr;
vk::Pipeline m_d32_to_d16 = nullptr;
vk::Pipeline m_swap_bgra16 = nullptr;
};
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_TILER_H_
File diff suppressed because it is too large Load Diff
@@ -1,392 +0,0 @@
#include "graphics/host_gpu/renderer/imageView.h"
#include "common/assert.h"
#include "graphics/host_gpu/objects/textureCommon.h"
#include "graphics/host_gpu/renderer/renderContext.h"
#include "graphics/host_gpu/renderer/textureCache.h"
#include <mutex>
namespace Libs::Graphics {
namespace {
void CreateView(VulkanImage& image, int view_index,
vk::ImageViewType view_type, vk::ImageAspectFlags aspect_mask,
vk::ComponentMapping components, uint32_t base_array_layer, uint32_t base_mip_level,
uint32_t layer_count, uint32_t level_count,
vk::Format view_format = vk::Format::eUndefined,
vk::ImageUsageFlags view_usage = {}) {
auto& graphics = GetRenderContext().GetGraphics();
if (view_index < 0 || view_index >= VulkanImage::VIEW_MAX ||
image.image_view[view_index] != nullptr) {
EXIT("invalid image-view creation target: image=%p index=%d current_view=%d\n",
static_cast<const void*>(&image), view_index,
view_index >= 0 && view_index < VulkanImage::VIEW_MAX &&
image.image_view[view_index] != nullptr);
}
vk::ImageViewUsageCreateInfo usage_info {};
usage_info.sType = vk::StructureType::eImageViewUsageCreateInfo;
usage_info.usage = view_usage;
vk::ImageViewCreateInfo create_info {};
create_info.sType = vk::StructureType::eImageViewCreateInfo;
create_info.pNext = view_usage ? &usage_info : nullptr;
create_info.image = image.image;
create_info.viewType = view_type;
create_info.format = view_format != vk::Format::eUndefined ? view_format : image.format;
create_info.components = components;
create_info.subresourceRange.aspectMask = aspect_mask;
create_info.subresourceRange.baseArrayLayer = base_array_layer;
create_info.subresourceRange.baseMipLevel = base_mip_level;
create_info.subresourceRange.layerCount = layer_count;
create_info.subresourceRange.levelCount = level_count;
const auto result =
graphics.device.createImageView(&create_info, nullptr, &image.image_view[view_index]);
if (result != vk::Result::eSuccess || image.image_view[view_index] == nullptr) {
EXIT("failed to create image view: result=%d image_format=%d view_format=%d index=%d\n",
static_cast<int>(result), static_cast<int>(image.format),
static_cast<int>(create_info.format), view_index);
}
}
void CreateRenderTargetView(VulkanImage& image, int index,
vk::ComponentSwizzle r, vk::ComponentSwizzle g, vk::ComponentSwizzle b,
vk::ComponentSwizzle a, vk::ImageViewType type = vk::ImageViewType::e2D,
vk::Format view_format = vk::Format::eUndefined,
vk::ImageUsageFlags view_usage = {}, uint32_t level_count = 0) {
const auto layer_count = type == vk::ImageViewType::e2DArray ? image.layers : 1u;
CreateView(image, index, type, vk::ImageAspectFlagBits::eColor, {r, g, b, a}, 0, 0,
layer_count, level_count == 0 ? image.mip_levels : level_count, view_format,
view_usage);
}
} // namespace
namespace ImageViewOps {
vk::ImageAspectFlags DepthAspectMask(vk::Format format) {
switch (format) {
case vk::Format::eD16Unorm:
case vk::Format::eD32Sfloat: return vk::ImageAspectFlagBits::eDepth;
case vk::Format::eD16UnormS8Uint:
case vk::Format::eD24UnormS8Uint:
case vk::Format::eD32SfloatS8Uint:
return vk::ImageAspectFlagBits::eDepth | vk::ImageAspectFlagBits::eStencil;
default: EXIT("unsupported depth/stencil image format: %d\n", static_cast<int>(format));
}
}
bool FormatSupportsStorage(vk::Format format) {
auto& graphics = GetRenderContext().GetGraphics();
const auto properties = graphics.GetFormatProperties(format);
return static_cast<bool>(properties.optimalTilingFeatures &
vk::FormatFeatureFlagBits::eStorageImage);
}
void CreateRenderTargetViews(RenderTextureVulkanImage& image) {
CreateRenderTargetView(image, VulkanImage::VIEW_DEFAULT,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity);
if (image.layers > 1) {
CreateRenderTargetView(image, VulkanImage::VIEW_DEFAULT_ARRAY,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ImageViewType::e2DArray);
}
if (image.samples == 1 && FormatSupportsStorage(image.format)) {
CreateRenderTargetView(image, VulkanImage::VIEW_STORAGE,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ImageViewType::e2D, vk::Format::eUndefined,
vk::ImageUsageFlags {}, 1);
if (image.layers > 1) {
CreateRenderTargetView(image, VulkanImage::VIEW_STORAGE_ARRAY,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ImageViewType::e2DArray, vk::Format::eUndefined,
vk::ImageUsageFlags {}, 1);
}
}
}
void CreateDepthViews(DepthStencilVulkanImage& image) {
CreateView(image, VulkanImage::VIEW_DEFAULT, vk::ImageViewType::e2D,
DepthAspectMask(image.format),
{vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity},
0, 0, 1, 1);
}
void CreateVideoOutViews(VideoOutVulkanImage& image) {
CreateRenderTargetView(image, VulkanImage::VIEW_DEFAULT,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity);
if ((image.format == vk::Format::eR8G8B8A8Srgb || image.format == vk::Format::eB8G8R8A8Srgb) &&
FormatSupportsStorage(vk::Format::eR8G8B8A8Uint)) {
CreateRenderTargetView(image, VulkanImage::VIEW_STORAGE,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ComponentSwizzle::eIdentity, vk::ComponentSwizzle::eIdentity,
vk::ImageViewType::e2D, vk::Format::eR8G8B8A8Uint,
vk::ImageUsageFlagBits::eStorage, 1);
}
}
void DestroyViews(VulkanImage& image) {
auto& graphics = GetRenderContext().GetGraphics();
for (auto& cached: image.view_cache.views) {
if (cached.view != nullptr) {
graphics.device.destroyImageView(cached.view, nullptr);
cached.view = nullptr;
}
}
image.view_cache.views.clear();
for (auto& view: image.image_view) {
if (view != nullptr) {
graphics.device.destroyImageView(view, nullptr);
view = nullptr;
}
}
}
} // namespace ImageViewOps
vk::ImageView TextureCache::GetRenderTargetAttachmentView(RenderTextureVulkanImage& image,
vk::Format format, uint32_t level,
uint32_t base_layer,
uint32_t layer_count) {
if (image.format == vk::Format::eUndefined || level >= image.mip_levels || level >= 16 ||
layer_count == 0 || base_layer >= image.layers || layer_count > image.layers - base_layer) {
EXIT("TextureCache: invalid render-target attachment view, image=%p format=%d"
" level=%u image_levels=%u base_layer=%u layer_count=%u image_layers=%u\n",
static_cast<const void*>(&image), static_cast<int>(format), level, image.mip_levels,
base_layer, layer_count, image.layers);
}
if (format != image.format && !IsRgba8SrgbReinterpretation(image.format, format)) {
EXIT("TextureCache: incompatible render-target attachment view, image_format=%d"
" view_format=%d level=%u\n",
static_cast<int>(image.format), static_cast<int>(format), level);
}
return GetImageView(
image, {format, layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray,
vk::ImageAspectFlagBits::eColor, level, 1, base_layer, layer_count,
DstSel(4, 5, 6, 7), vk::ImageUsageFlagBits::eColorAttachment});
}
vk::ImageView TextureCache::GetDepthTargetAttachmentView(DepthStencilVulkanImage& image,
uint32_t base_layer,
uint32_t layer_count) {
if (layer_count == 0 || base_layer >= image.layers || layer_count > image.layers - base_layer) {
EXIT("TextureCache: invalid depth-target attachment view, image=%p base_layer=%u "
"layer_count=%u image_layers=%u\n",
static_cast<const void*>(&image), base_layer, layer_count, image.layers);
}
return GetImageView(image,
{image.format,
layer_count == 1 ? vk::ImageViewType::e2D : vk::ImageViewType::e2DArray,
ImageViewOps::DepthAspectMask(image.format), 0, 1, base_layer, layer_count,
DstSel(4, 5, 6, 7), vk::ImageUsageFlagBits::eDepthStencilAttachment});
}
vk::ImageView TextureCache::GetImageView(VulkanImage& image, const ImageViewInfo& info) {
const bool supported_type = info.type == vk::ImageViewType::e2D ||
info.type == vk::ImageViewType::e2DArray ||
info.type == vk::ImageViewType::e3D;
const bool supported_usage = info.usage == vk::ImageUsageFlagBits::eSampled ||
info.usage == vk::ImageUsageFlagBits::eStorage ||
info.usage == vk::ImageUsageFlagBits::eColorAttachment ||
info.usage == vk::ImageUsageFlagBits::eDepthStencilAttachment;
const bool valid_shape =
(info.type == vk::ImageViewType::e2D && info.layer_count == 1) ||
info.type == vk::ImageViewType::e2DArray ||
(info.type == vk::ImageViewType::e3D && info.base_layer == 0 && info.layer_count == 1);
if (info.format == vk::Format::eUndefined || !info.aspect || info.level_count == 0 ||
info.base_level >= (image.mip_levels) ||
info.level_count > image.mip_levels - info.base_level || info.layer_count == 0 ||
info.base_layer >= image.layers || info.layer_count > image.layers - info.base_layer ||
!supported_type || !valid_shape || !supported_usage) {
EXIT("TextureCache: invalid dynamic image view, image=%p format=%d aspect=0x%x"
" swizzle=0x%03x mip=%u+%u layer=%u+%u type=%d usage=0x%x"
" image_levels=%u image_layers=%u\n",
static_cast<const void*>(&image), static_cast<int>(info.format),
static_cast<vk::ImageAspectFlags::MaskType>(info.aspect), info.swizzle,
info.base_level, info.level_count, info.base_layer, info.layer_count,
static_cast<int>(info.type), static_cast<vk::ImageUsageFlags::MaskType>(info.usage),
image.mip_levels, image.layers);
}
auto& cache = image.view_cache;
std::lock_guard lock(cache.mutex);
for (const auto& cached: cache.views) {
if (cached.info == info) {
return cached.view;
}
}
vk::ImageViewUsageCreateInfo usage {};
usage.sType = vk::StructureType::eImageViewUsageCreateInfo;
usage.usage = info.usage;
vk::ImageViewCreateInfo create {};
create.sType = vk::StructureType::eImageViewCreateInfo;
create.pNext = &usage;
create.image = image.image;
create.viewType = info.type;
create.format = info.format;
create.components = info.usage == vk::ImageUsageFlagBits::eSampled
? TextureGetComponentMapping(info.swizzle)
: vk::ComponentMapping {};
create.subresourceRange.aspectMask = info.aspect;
create.subresourceRange.baseMipLevel = info.base_level;
create.subresourceRange.levelCount = info.level_count;
create.subresourceRange.baseArrayLayer = info.base_layer;
create.subresourceRange.layerCount = info.layer_count;
vk::ImageView view = nullptr;
const auto result = m_graphics.device.createImageView(&create, nullptr, &view);
if (result != vk::Result::eSuccess || view == nullptr) {
EXIT("TextureCache: failed to create dynamic image view, result=%d format=%d"
" aspect=0x%x swizzle=0x%03x mip=%u+%u layer=%u+%u type=%d usage=0x%x\n",
static_cast<int>(result), static_cast<int>(info.format),
static_cast<vk::ImageAspectFlags::MaskType>(info.aspect), info.swizzle,
info.base_level, info.level_count, info.base_layer, info.layer_count,
static_cast<int>(info.type), static_cast<vk::ImageUsageFlags::MaskType>(info.usage));
}
cache.views.push_back({info, view});
return view;
}
vk::ImageView TextureCache::GetDepthTargetSampledView(DepthStencilVulkanImage& image,
vk::Format view_format, uint32_t swizzle,
uint32_t base_level, uint32_t level_count,
vk::ImageViewType type, uint32_t base_layer,
uint32_t layer_count) {
if (view_format == vk::Format::eUndefined ||
!IsSupportedSampledDepthView(image.format, view_format, swizzle)) {
EXIT("TextureCache: invalid sampled depth-target view, image=%p image_format=%d"
" view_format=%d swizzle=0x%03x mip=%u+%u layer=%u+%u type=%d"
" image_levels=%u image_layers=%u\n",
static_cast<const void*>(&image), static_cast<int>(image.format),
static_cast<int>(view_format), swizzle, base_level, level_count, base_layer,
layer_count, static_cast<int>(type), image.mip_levels, image.layers);
}
return GetImageView(image, {image.format, type, vk::ImageAspectFlagBits::eDepth, base_level,
level_count, base_layer, layer_count, swizzle});
}
vk::ImageView TextureCache::GetSampledColorView(VulkanImage& image, vk::Format view_format,
uint32_t swizzle, uint32_t base_level,
uint32_t level_count, vk::ImageViewType type,
uint32_t base_layer, uint32_t layer_count) {
if (view_format == vk::Format::eUndefined || base_level >= 16 ||
(type != vk::ImageViewType::e2D && type != vk::ImageViewType::e2DArray) ||
!IsSupportedSampledColorView(image.format, view_format, swizzle)) {
EXIT("TextureCache: invalid sampled color view, image=%p swizzle=0x%03x"
" view_format=%d mip=%u+%u layer=%u+%u type=%d image_levels=%u image_layers=%u\n",
static_cast<const void*>(&image), swizzle, static_cast<int>(view_format), base_level,
level_count, base_layer, layer_count, static_cast<int>(type), image.mip_levels,
image.layers);
}
const auto precreated_view = type == vk::ImageViewType::e2DArray
? VulkanImage::VIEW_DEFAULT_ARRAY
: VulkanImage::VIEW_DEFAULT;
const bool full_view = base_level == 0 && level_count == image.mip_levels && base_layer == 0 &&
layer_count == (type == vk::ImageViewType::e2DArray ? image.layers : 1u);
if (view_format == image.format && swizzle == DstSel(4, 5, 6, 7) && full_view &&
image.image_view[precreated_view] != nullptr) {
return image.image_view[precreated_view];
}
return GetImageView(image, {view_format, type, vk::ImageAspectFlagBits::eColor, base_level,
level_count, base_layer, layer_count, swizzle});
}
vk::ImageView TextureCache::GetRenderTargetStorageView(RenderTextureVulkanImage& image,
vk::Format view_format, uint32_t base_level,
uint32_t level_count, vk::ImageViewType type,
uint32_t base_layer, uint32_t layer_count) {
if (view_format == vk::Format::eUndefined ||
(type != vk::ImageViewType::e2D && type != vk::ImageViewType::e2DArray)) {
EXIT("TextureCache: invalid render-target storage view, image=%p view_format=%d"
" mip=%u+%u layer=%u+%u type=%d image_levels=%u image_layers=%u\n",
static_cast<const void*>(&image), static_cast<int>(view_format), base_level,
level_count, base_layer, layer_count, static_cast<int>(type), image.mip_levels,
image.layers);
}
const bool exact = view_format == image.format;
const bool compatible = view_format == BgraSrgbStorageViewFormat(image.format);
if (!exact && !compatible) {
EXIT("TextureCache: incompatible render-target storage view, image_format=%d"
" view_format=%d base=%u count=%u\n",
static_cast<int>(image.format), static_cast<int>(view_format), base_level,
level_count);
}
if (exact) {
const auto index = type == vk::ImageViewType::e2DArray ? VulkanImage::VIEW_STORAGE_ARRAY
: VulkanImage::VIEW_STORAGE;
const bool full_view =
base_level == 0 && level_count == 1 && base_layer == 0 &&
layer_count == (type == vk::ImageViewType::e2DArray ? image.layers : 1u);
if (full_view && image.image_view[index] != nullptr) {
return image.image_view[index];
}
}
if (compatible && !ImageViewOps::FormatSupportsStorage(view_format)) {
EXIT("TextureCache: compatible render-target storage format lacks storage support,"
" image_format=%d view_format=%d base=%u count=%u\n",
static_cast<int>(image.format), static_cast<int>(view_format), base_level,
level_count);
}
return GetImageView(image, {view_format, type, vk::ImageAspectFlagBits::eColor, base_level,
level_count, base_layer, layer_count, DstSel(4, 5, 6, 7),
vk::ImageUsageFlagBits::eStorage});
}
vk::ImageView TextureCache::GetStorageTextureSampledView(StorageTextureVulkanImage& image,
const ImageInfo& info) {
const auto shape = SelectStorageSampledViewShape(info.type, info.depth, image.layers);
if (image.image == nullptr || shape == StorageSampledViewShape::Unsupported ||
info.base_array != 0 || info.levels != image.mip_levels || info.base_level >= info.levels ||
info.view_levels == 0 || info.base_level + info.view_levels > info.levels) {
EXIT("TextureCache: invalid sampled view of storage texture, image=%p type=%u depth=%u"
" base=%u levels=%u view_levels=%u image_levels=%u base_array=%u\n",
static_cast<const void*>(&image), info.type, info.depth, info.base_level, info.levels,
info.view_levels, image.mip_levels, info.base_array);
}
const auto view_format = TextureGetFormat(info.format);
if (view_format != image.format && !IsRgba8SrgbReinterpretation(image.format, view_format) &&
!IsR32UintFloatReinterpretation(image.format, view_format)) {
EXIT("TextureCache: incompatible sampled view of storage texture, image_format=%d"
" view_format=%d swizzle=0x%03x\n",
static_cast<int>(image.format), static_cast<int>(view_format), info.swizzle);
}
vk::ImageViewType type = static_cast<vk::ImageViewType>(VK_IMAGE_VIEW_TYPE_MAX_ENUM);
switch (shape) {
case StorageSampledViewShape::Image2D: type = vk::ImageViewType::e2D; break;
case StorageSampledViewShape::Image2DArray: type = vk::ImageViewType::e2DArray; break;
case StorageSampledViewShape::Image3D: type = vk::ImageViewType::e3D; break;
case StorageSampledViewShape::Unsupported:
EXIT("TextureCache: unsupported sampled storage-image view shape\n");
}
const auto layer_count = shape == StorageSampledViewShape::Image2DArray ? info.depth : 1u;
return GetImageView(image, {view_format, type, vk::ImageAspectFlagBits::eColor, info.base_level,
info.view_levels, 0, layer_count, info.swizzle});
}
vk::ImageView TextureCache::GetStorageTextureStorageView(StorageTextureVulkanImage& image,
uint32_t base_level) {
if (image.image == nullptr || base_level >= (image.mip_levels)) {
EXIT("TextureCache: invalid storage-texture mip view, image=%p level=%u levels=%u\n",
static_cast<const void*>(&image), base_level, image.mip_levels);
}
if (base_level == 0) {
return image.image_view[VulkanImage::VIEW_DEFAULT];
}
return GetImageView(image, {image.format, vk::ImageViewType::e2D,
vk::ImageAspectFlagBits::eColor, base_level, 1, 0, 1,
DstSel(4, 5, 6, 7), vk::ImageUsageFlagBits::eStorage});
}
} // namespace Libs::Graphics
-230
View File
@@ -1,230 +0,0 @@
#ifndef EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
#define EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
#include "common/assert.h"
#include "graphics/host_gpu/graphicContext.h"
#include "graphics/host_gpu/renderer/imageInfo.h"
#include "graphics/shader/recompiler/ShaderIR.h"
#include "graphics/shader/shader.h"
namespace Libs::Graphics {
[[nodiscard]] inline bool IsSupportedStorageSwizzle(uint32_t format, uint32_t swizzle) noexcept {
const bool single_channel =
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8UNorm) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8UInt) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k16UInt) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32UInt) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k16Float) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32Float);
return swizzle == DstSel(4, 5, 6, 7) ||
(single_channel && (swizzle == DstSel(4, 0, 0, 0) || swizzle == DstSel(4, 0, 0, 1) ||
swizzle == DstSel(4, 4, 4, 4))) ||
(format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32_32UInt) &&
swizzle == DstSel(4, 5, 0, 1)) ||
((format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8_8_8_8UNorm) ||
format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8_8_8_8UInt)) &&
(swizzle == DstSel(4, 5, 6, 1) || swizzle == DstSel(6, 5, 4, 7))) ||
(format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32_32_32_32Float) &&
swizzle == DstSel(5, 6, 7, 4));
}
[[nodiscard]] inline bool IsSupportedStorageDepthTile(uint32_t format, uint32_t type,
uint32_t width, uint32_t height,
uint32_t depth) noexcept {
return (format == Prospero::GpuEnumValue(Prospero::BufferFormat::k8UInt) &&
type == Prospero::GpuEnumValue(Prospero::ImageType::kColor2DArray) && width != 0 &&
height != 0 && depth == 1) ||
(format == Prospero::GpuEnumValue(Prospero::BufferFormat::k32UInt) &&
type == Prospero::GpuEnumValue(Prospero::ImageType::kColor2D) && width != 0 &&
height != 0 && depth == 1);
}
[[noreturn]] inline void UnsupportedColorView(const char* usage, vk::Format image_format,
vk::Format view_format, uint32_t swizzle) noexcept {
EXIT("unsupported %s color image view: image_format=%d view_format=%d swizzle=0x%03x\n", usage,
static_cast<int>(image_format), static_cast<int>(view_format), swizzle);
}
[[nodiscard]] inline vk::Format BgraToRgbaSampledViewFormat(vk::Format image_format) noexcept {
switch (image_format) {
case vk::Format::eB8G8R8A8Unorm: return vk::Format::eR8G8B8A8Unorm;
case vk::Format::eB8G8R8A8Srgb: return vk::Format::eR8G8B8A8Srgb;
case vk::Format::eA2R10G10B10UnormPack32: return vk::Format::eA2B10G10R10UnormPack32;
default: return vk::Format::eUndefined;
}
}
[[nodiscard]] inline bool IsBgraToRgbaSampledView(vk::Format image_format,
vk::Format view_format) noexcept {
switch (image_format) {
case vk::Format::eB8G8R8A8Unorm:
case vk::Format::eB8G8R8A8Srgb:
switch (view_format) {
case vk::Format::eR8G8B8A8Unorm:
case vk::Format::eR8G8B8A8Srgb: return true;
default: return false;
}
case vk::Format::eA2R10G10B10UnormPack32:
return view_format == vk::Format::eA2B10G10R10UnormPack32;
default: return false;
}
}
[[nodiscard]] inline vk::Format BgraSrgbStorageViewFormat(vk::Format image_format) noexcept {
return image_format == vk::Format::eB8G8R8A8Srgb ? vk::Format::eR8G8B8A8Unorm
: vk::Format::eUndefined;
}
[[nodiscard]] inline vk::Format SrgbStorageViewFormat(vk::Format image_format) noexcept {
return image_format == vk::Format::eR8G8B8A8Srgb ? vk::Format::eR8G8B8A8Unorm
: BgraSrgbStorageViewFormat(image_format);
}
[[nodiscard]] inline bool IsBgraSrgbStorageView(vk::Format image_format, vk::Format view_format,
uint32_t swizzle) noexcept {
return view_format == BgraSrgbStorageViewFormat(image_format) && swizzle == DstSel(6, 5, 4, 7);
}
[[nodiscard]] inline bool IsValidSampledColorSwizzle(uint32_t swizzle) noexcept {
if ((swizzle & ~0xfffu) != 0) {
return false;
}
for (uint32_t channel = 0; channel < 4; channel++) {
switch (GetDstSel(swizzle, channel)) {
case 0:
case 1:
case 4:
case 5:
case 6:
case 7: break;
default: return false;
}
}
return true;
}
[[nodiscard]] inline bool IsSupportedSampledColorView(vk::Format image_format,
vk::Format view_format,
uint32_t swizzle) noexcept {
if (!IsValidSampledColorSwizzle(swizzle)) {
return false;
}
if (image_format == view_format || IsRgba8SrgbReinterpretation(image_format, view_format)) {
return true;
}
if ((IsRgba16UintFloatReinterpretation(image_format, view_format) ||
IsRgba8UnormUintReinterpretation(image_format, view_format)) &&
swizzle == DstSel(4, 5, 6, 7)) {
return true;
}
return IsBgraToRgbaSampledView(image_format, view_format) && swizzle == DstSel(6, 5, 4, 7);
}
[[nodiscard]] inline uint32_t
SelectSampledColorView(vk::Format image_format, vk::Format view_format, uint32_t swizzle) noexcept {
if (IsSupportedSampledColorView(image_format, view_format, swizzle)) {
return swizzle;
}
UnsupportedColorView("sampled", image_format, view_format, swizzle);
}
[[nodiscard]] inline bool IsSupportedSampledDepthView(vk::Format image_format,
vk::Format view_format,
uint32_t swizzle) noexcept {
if (!IsSupportedSampledDepthFormat(image_format, view_format)) {
return false;
}
switch (swizzle) {
case DstSel(4, 4, 4, 4):
case DstSel(4, 0, 0, 0):
case DstSel(4, 0, 0, 1): return true;
default: return false;
}
}
[[nodiscard]] inline uint32_t
SelectSampledDepthView(vk::Format image_format, vk::Format view_format, uint32_t swizzle) noexcept {
if (IsSupportedSampledDepthView(image_format, view_format, swizzle)) {
return swizzle;
}
EXIT("unsupported sampled depth image view: image_format=%d view_format=%d swizzle=0x%03x\n",
static_cast<int>(image_format), static_cast<int>(view_format), swizzle);
}
[[nodiscard]] inline bool
IsSupportedSampledDepthResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return resource.kind == ShaderRecompiler::IR::ResourceKind::Image &&
(resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray) &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.read &&
!resource.written && !resource.atomic;
}
[[nodiscard]] inline bool
IsSupportedSampledDepthUintResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return resource.kind == ShaderRecompiler::IR::ResourceKind::ImageUint &&
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.read &&
!resource.written && !resource.atomic && !resource.depth_compare;
}
[[nodiscard]] inline int SelectStorageColorView(vk::Format image_format, vk::Format view_format,
uint32_t swizzle) noexcept {
const bool single_channel =
view_format == vk::Format::eR8Unorm || view_format == vk::Format::eR8Uint ||
view_format == vk::Format::eR16Uint || view_format == vk::Format::eR32Uint ||
view_format == vk::Format::eR16Sfloat || view_format == vk::Format::eR32Sfloat;
const bool swizzle_ok =
swizzle == DstSel(4, 5, 6, 7) ||
(single_channel && (swizzle == DstSel(4, 0, 0, 0) || swizzle == DstSel(4, 0, 0, 1) ||
swizzle == DstSel(4, 4, 4, 4))) ||
(view_format == vk::Format::eR32G32Uint && swizzle == DstSel(4, 5, 0, 1)) ||
((view_format == vk::Format::eR8G8B8A8Unorm || view_format == vk::Format::eR8G8B8A8Uint) &&
(swizzle == DstSel(4, 5, 6, 1) || swizzle == DstSel(6, 5, 4, 7))) ||
(view_format == vk::Format::eR32G32B32A32Sfloat && swizzle == DstSel(5, 6, 7, 4));
if ((image_format != view_format &&
!IsBgraSrgbStorageView(image_format, view_format, swizzle)) ||
!swizzle_ok) {
UnsupportedColorView("storage", image_format, view_format, swizzle);
}
return VulkanImage::VIEW_STORAGE;
}
[[nodiscard]] inline bool
IsSupportedStorageImageResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
return (resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImage ||
resource.kind == ShaderRecompiler::IR::ResourceKind::StorageImageUint) &&
(resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim3D ||
resource.dimension == ShaderRecompiler::Decoder::ImageDimension::Dim2DArray) &&
resource.mip_mode == ShaderRecompiler::IR::ImageMipMode::None && resource.written &&
!resource.atomic && !resource.depth_compare;
}
inline void
ValidateStorageImageResource(const ShaderRecompiler::IR::ImageResource& resource) noexcept {
if (!IsSupportedStorageImageResource(resource)) {
EXIT("unsupported storage color image resource: kind=%u dimension=%u mip=%u "
"read=%d written=%d atomic=%d depth_compare=%d\n",
static_cast<uint32_t>(resource.kind), static_cast<uint32_t>(resource.dimension),
static_cast<uint32_t>(resource.mip_mode), resource.read, resource.written,
resource.atomic, resource.depth_compare);
}
}
namespace ImageViewOps {
[[nodiscard]] vk::ImageAspectFlags DepthAspectMask(vk::Format format);
[[nodiscard]] bool FormatSupportsStorage(vk::Format format);
void CreateRenderTargetViews(RenderTextureVulkanImage& image);
void CreateDepthViews(DepthStencilVulkanImage& image);
void CreateVideoOutViews(VideoOutVulkanImage& image);
void DestroyViews(VulkanImage& image);
} // namespace ImageViewOps
} // namespace Libs::Graphics
#endif // EMULATOR_SRC_GRAPHICS_HOST_GPU_RENDERER_IMAGEVIEW_H_
@@ -0,0 +1,60 @@
#include "graphics/host_gpu/renderer/masterSemaphore.h"
#include "common/assert.h"
#include "graphics/host_gpu/graphicContext.h"
namespace Libs::Graphics {
MasterSemaphore::MasterSemaphore(GraphicContext& graphics): m_graphics(graphics) {
vk::SemaphoreTypeCreateInfo type_info {};
type_info.sType = vk::StructureType::eSemaphoreTypeCreateInfo;
type_info.semaphoreType = vk::SemaphoreType::eTimeline;
type_info.initialValue = 0;
vk::SemaphoreCreateInfo create_info {};
create_info.sType = vk::StructureType::eSemaphoreCreateInfo;
create_info.pNext = &type_info;
const auto result = m_graphics.device.createSemaphore(&create_info, nullptr, &m_semaphore);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess || m_semaphore == nullptr);
}
MasterSemaphore::~MasterSemaphore() {
if (m_semaphore != nullptr) {
m_graphics.device.destroySemaphore(m_semaphore, nullptr);
}
}
void MasterSemaphore::Refresh() {
uint64_t counter = 0;
const auto result = m_graphics.device.getSemaphoreCounterValue(m_semaphore, &counter);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
auto known = m_gpu_tick.load(std::memory_order_acquire);
while (known < counter &&
!m_gpu_tick.compare_exchange_weak(known, counter, std::memory_order_release,
std::memory_order_relaxed)) {
}
}
void MasterSemaphore::Wait(uint64_t tick) {
if (IsFree(tick)) {
return;
}
Refresh();
if (IsFree(tick)) {
return;
}
vk::SemaphoreWaitInfo wait_info {};
wait_info.sType = vk::StructureType::eSemaphoreWaitInfo;
wait_info.semaphoreCount = 1;
wait_info.pSemaphores = &m_semaphore;
wait_info.pValues = &tick;
const auto result = m_graphics.device.waitSemaphores(&wait_info, UINT64_MAX);
EXIT_NOT_IMPLEMENTED(result != vk::Result::eSuccess);
Refresh();
}
} // namespace Libs::Graphics

Some files were not shown because too many files have changed in this diff Show More