Implement the `join_inputs_design` into the current SDA workflows. #3634

DongchenZ · 2025-10-03T13:47:40Z

Description

Motivation and Context

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I agree that PEcAn Project may distribute my contribution under any or all of
- the same license as the existing code,
- and/or the BSD 3-clause license.
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

mdietze

FYI, this provides an immediate fix, but really you should be passing in the new input_design object too, not just the new ensemble.size variable. That change is needed to get the desired result of sampling jointly once at the start of the SDA and then keeping that design as the SDA iterates.

mdietze

Just checked are was reminded that input_design is now the first argument to write.ensemble.configs and it's not optional, so adding that fix is actually required in this PR, not just a good next step for a future PR.

pecan/modules/uncertainty/R/ensemble.R

Line 228 in ccc5135

    
           write.ensemble.configs <- function(input_design , ensemble.size, defaults, ensemble.samples, settings, model,

I'd recommend that you add a call to modules/uncertainty/R/generate_joint_ensemble_design once in your code early on, that you save the design to restart, and that you read that design during restart rather than generating a new one (i.e., that if you have an "if restart" option, then the input_design is generated in the
"else")

DongchenZ · 2025-10-06T20:51:58Z

modules/assim.sequential/R/sda.enkf_MultiSite.R

-                                            TimeseriesPlot = FALSE,
-                                            debug = FALSE,
-                                            pause = FALSE,
-                                            Profiling = FALSE,


Removed unnecessary flags.

DongchenZ · 2025-10-06T21:02:05Z

modules/assim.sequential/R/sda.enkf_MultiSite.R

+                               control=list(TimeseriesPlot = FALSE,
                                            OutlierDetection=FALSE,
-                                            parallel_qsub = TRUE,
+                                            parallel_qsub = FALSE,


Set the parallel_qsub as FALSE because we can't use both local and remote qsub for the model execution.

DongchenZ · 2025-10-06T21:03:02Z

modules/assim.sequential/R/sda.enkf_MultiSite.R

                                            run_parallel = TRUE,
-                                            MCMC.args = NULL),
+                                            MCMC.args = NULL,
+                                            local.execution = TRUE),


Add the local.execution when we just want to run the jobs locally.

…acters.

DongchenZ · 2025-10-19T23:15:11Z

base/logger/R/logger.R

      "latin-ascii"
    )
-    if (nchar(long.msg) > 20 && wrap) {
+    if (wrap) {


The hard-coded length in the if control should be removed. The wrap argument should fully define it.

DongchenZ · 2025-10-19T23:15:54Z

models/stics/NAMESPACE


 export(met2model.STICS)
 export(model2netcdf.STICS)
+export(pecan2stics)


This is generated by building the existing pecan files.

DongchenZ

The newest commits provide the new debias workflow (which has already been implemented in the current SDA workflows), and the old script has been moved to the inst folder. This is cleaner and easier to understand, which could also benefit any future development of the debiasing algorithm.

DongchenZ · 2025-10-19T23:27:06Z

modules/assim.sequential/R/Analysis_sda_block.R

  }
+  cores <- cores - 1
+  # if we only have one CPU.
+  if (cores < 1) cores <- 1


Update the parallel registration.

DongchenZ · 2025-10-19T23:27:55Z

modules/assim.sequential/R/sda.enkf_MultiSite.R

+#' `qsub_parallel`, where we first combine jobs and submit them into the SCC.
+#' @param debias List: R list containing the covariance directory and the start year.
+#' covariance directory should include GeoTIFF files named by year.
+#' start year is numeric input which decide when to start the debiasing feature.


Removed the previous debias workflow and replaced it with the new workflow.

This reverts commit 77bda48.

…A_data

DongchenZ · 2025-10-21T15:49:44Z

modules/data.remote/R/SMAP_SMP_prep.R

  }
  PEcAn.logger::logger.info("SMAP SMP Prep Completed!")
-  list(SMP_Output = SMAP_Output, time_points = time_points, var = "SoilMoist")
+  list(SMP_Output = SMAP_Output, time_points = time_points, var = "SoilMoistFrac")


Correct the variable name from SoilMoist to SoilMoistFrac.

infotroph · 2025-10-29T21:43:31Z

modules/uncertainty/R/generate_joint_ensemble_design.R


-
-
  return(list(X = design_matrix))


Is it important that this be a list? All the uses I've seen so far immediately subset the result to only store the X component, so would it make sense to return design_matrix unwrapped in the first place?

It is important so that the sobol version works correctly. This ensures that regardless of whether the sobol or non-sobol version is called that the output is a list that includes the design as X. In the sobol version the list includes additional info beyond just X that's required by the function that does the sobol index calculations, but not required to do the runs themselves

👍 Can you add this to the function docs, please? I think it'd be enough to list in the @return block the possible items for the Sobol case and mention that in the non-sobol case the list only contains X

Bug fixes for the missing ensemble.size error.

8276e71

DongchenZ requested review from infotroph and mdietze and removed request for mdietze October 3, 2025 13:47

github-actions bot added the Modules label Oct 3, 2025

DongchenZ requested a review from mdietze October 3, 2025 13:47

mdietze approved these changes Oct 3, 2025

View reviewed changes

mdietze enabled auto-merge October 3, 2025 14:00

mdietze added this pull request to the merge queue Oct 3, 2025

mdietze reviewed Oct 3, 2025

View reviewed changes

mdietze removed this pull request from the merge queue due to a manual request Oct 3, 2025

mdietze requested changes Oct 3, 2025

View reviewed changes

Dongchen Zhang and others added 13 commits October 3, 2025 14:20

Implement joint input sampling in the SDA workflows.

05f82da

Update the joint inputs sampling function.

884a69d

Merge branch 'PecanProject:develop' into SDA_data

f7f7736

Update the ensemble configs function.

447b08d

Clean the multisite sda function.

92cd3f1

Revert the changes.

f2a96d1

Revert changes.

40701fe

revert changes.

4893c11

Revert changes.

f6d473a

Add documentation and bug fixes.

f787a17

Update document.

c82acac

Update the function to work with the newest functions.

f6f2232

Remove the samples argument from the function.

a30c9e8

DongchenZ changed the title ~~Bug fixes for the missing ensemble.size error.~~ Implement the join_inputs_design into the current SDA workflows. Oct 6, 2025

DongchenZ commented Oct 6, 2025

View reviewed changes

Dongchen Zhang added 2 commits October 19, 2025 12:50

Bug fix

e6ef100

Remove the condition for the wrapping where we have more than 20 char…

087f18a

…acters.

github-actions bot added the Base label Oct 19, 2025

DongchenZ commented Oct 19, 2025

View reviewed changes

Update the runner script.

5a04563

DongchenZ commented Oct 19, 2025

View reviewed changes

Dongchen Zhang and others added 16 commits October 20, 2025 12:21

remove arg.

7ba598e

Update script.

19db026

Revert "Move out of the if control."

2f14acc

This reverts commit 77bda48.

Resolve conflicts.

caefddf

Revert back.

6da0cd6

Revert to using the new debias workflow.

cb2379a

Update document.

668c4b1

Update document.

e543c23

Update the runner script.

a44399d

Update the bias correction function.

4701b41

Update the usage of the bias correction workflow.

2754b91

Merge branch 'PecanProject:develop' into SDA_data

459c945

Update observation paths.

0e08b79

Update namespace.

af5c081

Merge branch 'SDA_data' of https://github.com/DongchenZ/pecan into SD…

300dfae

…A_data

Modify the test to match the change in the logger.message function.

c8a8206

github-actions bot added the Tests label Oct 21, 2025

Dongchen Zhang added 2 commits October 21, 2025 11:35

Change the default value of wrap in the logger message function.

441f732

Revert back.

9099aa0

DongchenZ commented Oct 21, 2025

View reviewed changes

Merge branch 'PecanProject:develop' into SDA_data

986930b

infotroph reviewed Oct 29, 2025

View reviewed changes

infotroph mentioned this pull request Oct 29, 2025

Partial fixes for multisite ensembles #3654

Open

14 tasks

Implement the join_inputs_design into the current SDA workflows. #3634

Are you sure you want to change the base?

Implement the join_inputs_design into the current SDA workflows. #3634

Uh oh!

Conversation

DongchenZ commented Oct 3, 2025

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

Uh oh!

mdietze left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mdietze left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DongchenZ left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Implement the `join_inputs_design` into the current SDA workflows. #3634

Implement the `join_inputs_design` into the current SDA workflows. #3634