fix(linux/vulkan): harden encoder fallback

[codeafridi]

Description

Fixes a segmentation fault on Ubuntu 26.04 Wayland when Sunshine starts with the Vulkan encoder path, but the GPU/driver does not support Vulkan H.264 encoding.

The failed Vulkan encoder state is now handled safely, and KMS monitor detection no longer continues with stale or empty monitor data.

Screenshot

N/A

Issues Fixed or Closed

• Fixes Segmentation fault #5168

Roadmap Issues

N/A

Type of Change

• fix: Bug fix (non-breaking change which fixes an issue)

Checklist

• Code has been self-reviewed

AI Usage

• Light: AI provided minor assistance (formatting, simple suggestions)

[codeafridi]

@ReenigneArcher please review

[ReenigneArcher]

Seems fine to me, but would like some users facing the issue to test.

[Kishi85]

Seems fine to me, but would like some users facing the issue to test.

+1 also applied this to my local system and no regressions using kms-/portal-/kwingrab. I'm not affected by the issue so can't comment on that.

@ReenigneArcher You might want to kick off the CI pipeline so affected users can use the build artifacts to easily test without having to self compile.

[codecov]

Bundle Report

Bundle size has no change ✅

[codecov]

Codecov Report

❌ Patch coverage is 11.00000% with 89 lines in your changes missing coverage. Please review.
✅ Project coverage is 17.74%. Comparing base (3c7952b) to head (b6c8fec).
⚠️ Report is 161 commits behind head on master.

Files with missing lines	Patch %	Lines
src/platform/linux/vulkan_encode.cpp	0.00%	43 Missing ⚠️
src/video.cpp	13.63%	20 Missing and 18 partials ⚠️
src/platform/linux/kmsgrab.cpp	38.46%	4 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5174      +/-   ##
==========================================
- Coverage   17.78%   17.74%   -0.05%     
==========================================
  Files         111      111              
  Lines       24154    24226      +72     
  Branches    10688    10732      +44     
==========================================
+ Hits         4296     4299       +3     
- Misses      14682    15549     +867     
+ Partials     5176     4378     -798     

Flag	Coverage Δ
Archlinux	11.15% <5.55%> (-0.04%)	⬇️
FreeBSD-aarch64	?
FreeBSD-amd64	13.25% <1.14%> (-0.06%)	⬇️
Homebrew-ubuntu-22.04	13.45% <6.66%> (-0.08%)	⬇️
Linux-AppImage	12.04% <6.66%> (-0.07%)	⬇️
Windows-AMD64	14.83% <2.77%> (-0.02%)	⬇️
Windows-ARM64	13.17% <2.56%> (-0.04%)	⬇️
macOS-arm64	18.82% <2.56%> (-0.04%)	⬇️
macOS-x86_64	18.30% <12.82%> (-0.04%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/platform/linux/kmsgrab.cpp	3.69% <38.46%> (-0.15%)	⬇️
src/video.cpp	31.96% <13.63%> (-0.47%)	⬇️
src/platform/linux/vulkan_encode.cpp	0.30% <0.00%> (-0.03%)	⬇️

... and 31 files with indirect coverage changes

[codeafridi]

this issue is solved let me know if any further changes are needed.

[codeafridi]

I updated the pr so sunshine checks vulkan H.264 encode support before going into the vulkan encoder setup. If it is not supported thenit skips vulkan and falls back to other encoders vaapi.

[ReenigneArcher]

@codeafridi We seem to fighting each other on the rebases... Looks like you forced push without rebasing... I'll let you rebase before I run CI then (everything needs to be up to date for CI to run properly).

[codeafridi]

Sorry @ReenigneArcher i think i pushed over your rebase by mistake ill rebase my branch on the latest master now and push it back

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Kishi85]

So here's something interesting: I'm unsure if this is related but I seem to hit the conditions for #4943 to occur a lot easier with this PR applied (and also not only when trying to force the issue but on a simple one-time button press display switch as well). I'll have to do more testing but just a heads up to be aware of.