CHANGE: Migrate ci to recipe engine #2226

windxu88 · 2025-08-26T15:28:44Z

Description

This PR migrates CI to use Recipe Engine.

In Input CI, tests fall into two categories: functional tests and performance tests.
For each category, tests run on the following Runtime platforms:

Editor on all three desktop OS platforms which are Windows, Ubuntu and MacOS
Standalone on all three desktop OS platforms
Standalone with il2cpp backend on all three desktop OS platforms
Mobile platforms including TvOS, iOS and Android. For Android, we cover both mono and il2cpp backends.

In the migration, I created four recipes for both Functional and Performance tests.
For functional tests, they are:

EditorFunctionalTests
StandaloneFunctionalTests
StandaloneIl2CppFunctionalTests
MobileFunctionalTests

For performance tests, there are four equivalents:

EditorPerformanceTests
StandalonePerformanceTests
StandaloneIl2CppPerformanceTests
MobilePerformanceTests

For editor and standalone tests recipes, the tests will run on WrenchPackage.DefaultEditorPlatforms which are Windows, Mac and Ubuntu. That's what we want.
For mobile tests recipe, I added the following platforms in InputSystemSettings.cs:

MobileBuildPlatforms
MobileTestPlatforms
Mobile platform settings are read from mobile_config.json file in .yamato folder.

Another difference between editor/standalone and mobile tests recipes is:

Editor/Standalone tests recipes inherit from BaseRecipe
Mobile tests recipes inherit from MobileBaseRecipe which inherits BaseRecipe. That's because I need to add support for both mono and il2cpp backends for Android. I did some customization for Android about how to create jobs for it. The details can be seen in MobileBaseRecipe.cs.

The major differences between functional tests recipes and performance tests recipes are in UtrCommand.

Functional tests recipes use WithCategory("!Performance") and --enable-code-coverage.
Performance tests recipes use .WithCategory("Performance") and WithPerformanceDataReporting.
Other than that, functional tests recipes and performance tests recipes are very similar.

Please note that: In the old Build & Run jobs on Desktop OS platforms, upm-ci package test can be replaced by Wrench validation jobs. That's why I didn't add it in the recipes. For example, the old PVS tests have been replaced by upm-pvp test in Wrench validation jobs. Wrench validation jobs also run tests inside the package, so there is no need to run package tests in other jobs.

Testing status & QA

Please describe the testing already done by you and what testing you request/recommend QA to execute. If you used or created any testing project please link them here too for QA.

Overall Product Risks

Please rate the potential complexity and halo effect from low to high for the reviewers. Note down potential risks to specific Editor branches if any.

Complexity:
Halo Effect:

Comments to reviewers

Please describe any additional information such as what to focus on, or historical info for the reviewers.

Checklist

Before review:

Changelog entry added.
- Explains the change in Changed, Fixed, Added sections.
- For API change contains an example snippet and/or migration example.
- JIRA ticket linked, example (case %%). If it is a private issue, just add the case ID without a link.
- Jira port for the next release set as "Resolved".
Tests added/changed, if applicable.
- Functional tests Area_CanDoX, Area_CanDoX_EvenIfYIsTheCase, Area_WhenIDoX_AndYHappens_ThisIsTheResult.
- Performance tests.
- Integration tests.
Docs for new/changed API's.
- Xmldoc cross references are set correctly.
- Added explanation how the API works.
- Usage code examples added.
- The manual is updated, if needed.

During merge:

Commit message for squash-merge is prefixed with one of the list:
- NEW: ___.
- FIX: ___.
- DOCS: ___.
- CHANGE: ___.
- RELEASE: 1.1.0-preview.3.

After merge:

Create forward/backward port if needed. If you are blocked from creating a forward port now please add a task to ISX-1444.

codecov-github-com · 2025-08-26T17:43:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

@@             Coverage Diff             @@
##           develop    #2226      +/-   ##
===========================================
+ Coverage    68.14%   76.70%   +8.56%     
===========================================
  Files          367      465      +98     
  Lines        53685    87919   +34234     
===========================================
+ Hits         36585    67442   +30857     
- Misses       17100    20477    +3377

Flag	Coverage Δ
inputsystem_MacOS_2021.3	`5.91% <ø> (?)`
inputsystem_MacOS_2021.3_project	`78.05% <ø> (?)`
inputsystem_MacOS_2022.3	`5.37% <ø> (?)`
inputsystem_MacOS_2022.3_project	`74.58% <ø> (?)`
inputsystem_MacOS_6000.0	`5.19% <ø> (?)`
inputsystem_MacOS_6000.0_project	`76.50% <ø> (?)`
inputsystem_MacOS_6000.2	`5.19% <ø> (?)`
inputsystem_MacOS_6000.2_project	`76.50% <ø> (?)`
inputsystem_MacOS_6000.3	`5.19% <ø> (?)`
inputsystem_MacOS_6000.3_project	`76.50% <ø> (?)`
inputsystem_MacOS_6000.4	`5.19% <ø> (?)`
inputsystem_MacOS_6000.4_project	`76.50% <ø> (?)`
inputsystem_Ubuntu_2021.3	`5.91% <ø> (?)`
inputsystem_Ubuntu_2021.3_project	`77.96% <ø> (?)`
inputsystem_Ubuntu_2022.3	`5.38% <ø> (?)`
inputsystem_Ubuntu_2022.3_project	`74.39% <ø> (?)`
inputsystem_Ubuntu_6000.0	`5.19% <ø> (?)`
inputsystem_Ubuntu_6000.0_project	`76.32% <ø> (?)`
inputsystem_Ubuntu_6000.2	`5.19% <ø> (?)`
inputsystem_Ubuntu_6000.2_project	`76.32% <ø> (?)`
inputsystem_Ubuntu_6000.3	`5.19% <ø> (?)`
inputsystem_Ubuntu_6000.3_project	`76.31% <ø> (?)`
inputsystem_Ubuntu_6000.4	`5.19% <ø> (?)`
inputsystem_Ubuntu_6000.4_project	`76.31% <ø> (?)`
inputsystem_Windows_2021.3	`5.91% <ø> (?)`
inputsystem_Windows_2021.3_project	`78.20% <ø> (?)`
inputsystem_Windows_2022.3	`5.37% <ø> (?)`
inputsystem_Windows_2022.3_project	`74.72% <ø> (?)`
inputsystem_Windows_6000.0	`5.19% <ø> (?)`
inputsystem_Windows_6000.0_project	`76.64% <ø> (?)`
inputsystem_Windows_6000.2	`5.19% <ø> (?)`
inputsystem_Windows_6000.2_project	`76.64% <ø> (?)`
inputsystem_Windows_6000.3	`5.19% <ø> (?)`
inputsystem_Windows_6000.3_project	`76.64% <ø> (?)`
inputsystem_Windows_6000.4	`5.19% <ø> (?)`
inputsystem_Windows_6000.4_project	`76.64% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 100 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stefanunity

Thanks for doing this! While this is now a bunch more files flying around than before, it is going to be easier to keep alive without everyone bolting every new thing onto that one yml file :D
Only two small comments from my side.

stefanunity · 2025-09-03T09:17:43Z

Tools/CI/Recipes/EditorPerformanceTests.cs

+                    .WithRerun(1, true)
+                    .WithPerformanceDataReporting(true)
+                    .WithPerformanceProject("InputSystem")
+                    .WithExtraArgs("--report-performance-data --performance-project-id=InputSystem")


This line produces "--report-performance-data --performance-project-id=InputSystem" in the yml cmd line a 2nd time, since the two .Withs above it already say the same thing.
Looks fine for the other performance test recipes.

Good catch!

stefanunity · 2025-09-03T09:25:16Z

Tools/CI/Recipes/Triggers.cs

+                .WithDependencies(allMobileFunctionalTests)
+                .WithDependencies(allTvOSFunctionalBuildJobs),
+
+            JobBuilder.Create("All Functional Tests")


Maybe a trigger could be added for "all functional editor jobs for trunk" to run during development to save some clicks and have a quick CI turnaround before the PR stage? But I'll leave that up to @jfreire-unity to decide. Just a thought.

… and regenerate CI

jfreire-unity

Looks like a good improvement, so thanks for this!

I added some comments to avoid code repetition and questions regarding matching the current functionality.

Also, I think this area would benefit from having a README in case someone in the team needs to do a CI fix. I think a lot of us are not aware of how these CI Recipes so I'll try to book a meeting next week.

I'm not sure how we will "deploy" this. Are we able to test this on demand before landing the PR to make sure we have 1:1 matching with the current jobs that run?

My last comment/nitpick is about the size of the PR. We could have probably done this in smaller chunks so that we wouldn't have so many changes landing. Also would have been easier to review.

Nevertheless, this is great work, and thanks for improving our CI workflow! 🚀 🙏

jfreire-unity · 2025-09-05T07:33:05Z

Tools/CI/Recipes/InputBaseRecipe.cs

+
+namespace InputSystem.Cookbook.Recipes;
+
+public abstract class InputBaseRecipe: RecipeBase


Nit pick: We're in the InputSystem.Cookbook.Recipes so I would think we don't need Input prefix for inputBaseRecipe, InputMobileBaseRecipe. But this is just my preference and I don't know if without those prefixes, those classes exist elsewhere. You will likely be more into this area so I trust your judgement.

jfreire-unity · 2025-09-05T07:36:47Z

Tools/CI/Recipes/MobileFunctionalTests.cs

+        if (platform.System == SystemType.Android && jobName.Contains("il2cpp"))
+        {
+            job.WithCommands(UtrCommand.Run(platform.System, b => b
+                .WithTestProject($"{ProjectPath}")
+                .WithEditor(".Editor")
+                .WithSuite(UtrTestSuiteType.Playmode)
+                .WithPlatform(platform.System)
+                .WithCategory("!Performance")
+                .WithScriptingBackend(ScriptingBackendType.Il2Cpp)
+                .WithExtraArgs("--clean-library")
+                .WithRerun(1, true)
+                .WithBuildOnly()
+                .WithPlayerSavePath("build/players")
+                .WithArtifacts("build/logs")));
+        }
+        else if(platform.System == SystemType.TvOS)
+        {
+            job.WithCommands(UtrCommand.Run(platform.System, b => b
+                .WithTestProject($"{ProjectPath}")
+                .WithEditor(".Editor")
+                .WithSuite(UtrTestSuiteType.Playmode)
+                .WithCategory("!Performance")
+                .WithExtraArgs("--platform=tvOS --clean-library")
+                .WithRerun(1, true)
+                .WithBuildOnly()
+                .WithPlayerSavePath("build/players")
+                .WithArtifacts("build/logs")));
+        }
+        else
+        {
+            job.WithCommands(UtrCommand.Run(platform.System, b => b
+                .WithTestProject($"{ProjectPath}")
+                .WithEditor(".Editor")
+                .WithSuite(UtrTestSuiteType.Playmode)
+                .WithPlatform(platform.System)
+                .WithCategory("!Performance")
+                .WithExtraArgs("--clean-library")
+                .WithRerun(1, true)
+                .WithBuildOnly()
+                .WithPlayerSavePath("build/players")
+                .WithArtifacts("build/logs")));
+        }


I see some repeated method calls such as WithArtifacts , WithPlayersSavePath, etc. I wonder if we can make all of the repetition go away so that it's clearer of what's different bewteeen the jobs.

Also, you can take this comment to all the other places we are creating jobs. I don't know it the builder pattern needs to follow a specific order as well. Still, I think at least adding the methods that are different depending on the condition we check for would be beneficial to all maintainers.

I have done some refactoring to reduce the duplication. Please re-review.

jfreire-unity · 2025-09-05T07:37:19Z

Tools/CI/Recipes/MobileFunctionalTests.cs

+            job.WithCommands(UtrCommand.Run(platform.System, b => b
+                .WithTestProject($"{ProjectPath}")
+                .WithEditor(".Editor")
+                .WithSuite(UtrTestSuiteType.Playmode)
+                .WithCategory("!Performance")
+                .WithExtraArgs("--platform=tvOS --clean-library")
+                .WithRerun(1, true)
+                .WithBuildOnly()
+                .WithPlayerSavePath("build/players")
+                .WithArtifacts("build/logs")));


I don't see a WithPlatform here. Is it expected?

"WithPlatform" doesn't work with tvOS. Instead I used --platform=tvOS directly.

jfreire-unity · 2025-09-05T07:38:27Z

Tools/CI/Recipes/MobileFunctionalTests.cs

+        if (platform.System == SystemType.IOS && float.Parse(unityVersion) > 6000.2f)
+            job.WithEnvironmentVariable("UNITY_HANDLEUIINTERRUPTIONS", 1);


Why is this only done for 6.2 and forward? Maybe some comments explaining this would help understand its need.

Have added some comments

Tools/CI/regenerate.bat

jfreire-unity · 2025-09-05T07:47:52Z

Tools/CI/regenerate.sh

When should this be run? Could we actually add a README.md to Tools/CI for us to know how it is setup and what to run when we make changes?

We need to run regenerate.sh[bat] when any of following things happen:

Changes made to InputSystem.Cookbook project

An unity version reaches EOL

Trunk version has changed which also means a new version has branched off from Trunk

Minimum Editor version of Input package has changed

Wrench version is updated

Tools/CI/Recipes/InputMobileBaseRecipe.cs

jfreire-unity

Thanks for doing this! Approving to not leave this lingering longer. No blockers for me.

windxu88 added 23 commits August 7, 2025 13:28

Add first recipe for Input tests on desktop platforms

f020635

Only install netfx on Windows

545e75e

Add more flags

cf15599

Add infrastructure_instabilities_detection

89c197b

Some refactoring

9df440f

Add two standalone test recipes

d2c3783

Add performance tests recipes and code coverage options

53206a5

Download StandaloneSupport-IL2CPP component for IL2CPP tests

fab79f6

Fix instability detection script path

9e3857d

Update Wrench to 0.12.3

0158874

Fix YAMATO_SOURCE_DIR variable

9cfc571

Add more recipes

cd986d2

Add PR trigger

032923d

Add backend support for Android jobs

0456bef

Refactoring to mobile jobs

22c4d5b

Add nightly trigger

7f6d4fb

Add publish_dryrun job to PR trigger

9382d56

Update Wrench to v1.0.1 and update Ubuntu image to 22.04

9dfc7a3

Merge develop branch

683767b

Fix ios and tvos jobs

77b19df

Add a trigger for all mobile tests

68ab89b

Fix tvOS test job

2325971

Remove legacy test jobs

1415be6

windxu88 changed the title ~~Migrate ci to recipe engine~~ CHANGE: Migrate ci to recipe engine Aug 26, 2025

windxu88 added 3 commits September 2, 2025 09:42

Remove more legacy jobs

0a77838

Remove upm-ci.yml

57ac36c

Merge from develop branch

95f1719

windxu88 requested a review from stefanunity September 2, 2025 15:35

windxu88 marked this pull request as ready for review September 2, 2025 15:35

windxu88 requested a review from jfreire-unity September 2, 2025 15:49

stefanunity approved these changes Sep 3, 2025

View reviewed changes

Remove duplicated performance test flags in Editor Performance recipe…

a1c4450

… and regenerate CI

jfreire-unity reviewed Sep 5, 2025

View reviewed changes

windxu88 and others added 4 commits September 5, 2025 10:59

Reduce duplication in mobile recipes

90ece43

Rename base recipe classes

ec7256b

More refactoring

a519e93

Merge branch 'develop' into migrate_ci_to_recipe_engine

b36bf50

jfreire-unity approved these changes Sep 12, 2025

View reviewed changes


		namespace InputSystem.Cookbook.Recipes;

		public abstract class InputBaseRecipe: RecipeBase

		if (platform.System == SystemType.IOS && float.Parse(unityVersion) > 6000.2f)
		job.WithEnvironmentVariable("UNITY_HANDLEUIINTERRUPTIONS", 1);

CHANGE: Migrate ci to recipe engine #2226

Are you sure you want to change the base?

CHANGE: Migrate ci to recipe engine #2226

Uh oh!

Conversation

windxu88 commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing status & QA

Overall Product Risks

Comments to reviewers

Checklist

Uh oh!

codecov-github-com bot commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

stefanunity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stefanunity Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jfreire-unity left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jfreire-unity left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

windxu88 commented Aug 26, 2025 •

edited

Loading

codecov-github-com bot commented Aug 26, 2025 •

edited

Loading

stefanunity Sep 3, 2025 •

edited

Loading