Capacity shortages during release preview

[mshanemc]

It's release preview time. See this known issue for more information and updates on a resolution: Intermittent performance issues on a subset of North American sandbox instances

What happens during preview

Reduced capacity on scratch org infrastructure leads to longer times for things like

• scratch org creation
• metadata api (deploy/retrieve/push/push/delete)
• package create/install
• test execution

Why it happens

There's a fixed amount of non-prod infrastructure (where scratch orgs and sandboxes live). When the release preview starts, a portion of the infrastructure is updated to the new release.

At first, all y'all's dev hubs are all on the current (summer 22) release, so will by default put their scratch orgs on the same release as their hub, which is causing the traffic jam.

What we're doing

Additional capacity (more than double) has been added to non-prod infrastructure and more is being added. A technology team meets daily to review performance measures.

What you can do

Tell us! We want to know when you're being affected by performance issues. Click "This Issue Affects Me" on the Known Issue.

If you can, use scratch orgs on the newer release (Winter '23), which is typically under less load.

Try release: preview (needs to be lowercase p on preview) on your scratch definition files or CLI commands until your hub updates to the new release, and then remove that fix. See docs for Create a Scratch Org for a Specific Release

[Julian88Tex]

@mshanemc This makes sense, but is there any reason this seems to be especially true for Winter '23? We've seen our deploy times go from ~20 minutes to ~140 minutes. I don't remember it ever being this bad in the past. Also, Winter '23 has its first release on 9/9/22. I'm curious why we are experiencing this so early and if there is any way to anticipate it in the future.

Finally, while we can move our builds to the preview release, we still need to continue running builds on Summer '22 since all (and most for some time) will be on this release and we want to be sure and not introduce any bugs that would only be discoverable on Summer '22.

Appreciate any guidance on this!

[mshanemc]

I'm curious why we are experiencing this so early and if there is any way to anticipate it in the future.

It happens when sandbox preview starts. There's usually something like this for every release. Sandbox preview started 8/26.

is there any reason this seems to be especially true for Winter '23?

I don't have data, but it has seemed to get worse the last few release cycles.

we still need to continue running builds on Summer '22

There's definitely a use cases for non-preview orgs. I'm hoping that if more orgs that can move to the preview infrastructure do, then the supply/demand imbalance improves for those that don't/can't.

[gdman]

Thanks for providing information. In my experience it's not limited to scratch orgs - sandboxes in general are slow during maintenance release previews.

I've been working with Salesforce for about 10 years and most release previews bring performance issues. Some releases feel worse than others. Anecdotally, it appears the last few releases have been much worse.

Given there are 3 releases a year, that's at least 3 months of release previews per year where sandbox performance is poor. The org performance significantly impacts team productivity. Last I checked, a full copy sandbox is listed at a whopping 30% of total spend, partial 20%, etc.

Is Salesforce doing anything to improve infrastructure?

[gfarb]

Thanks for providing information. In my experience it's not limited to scratch orgs - sandboxes in general are slow during maintenance release previews.

@mshanemc we at GitHub have opened a Salesforce support case about this (43205655), but practically all of our new Sandbox requests have been hanging on pending, processing or activating. I concur with @gdman that this is not exclusive to scratch org infra.

[daveespo]

I want to know why Salesforce hasn't put any priority against this problem too. Given that core members of the CLI/SFDX team know about this pain, it's not like this productivity impact is a surprise to anyone at Salesforce and Andrew's analysis is spot on. It's really 18 weeks out of the year -- that's more than 25% of the year we're living in a paralyzed state.

Salesforce is trotting out Hyperforce and new Production pods in every corner of the earth ... it seems incongruous that Sandbox infrastructure is limited in capacity.

(And FWIW, we also opened a case about this #43210316 too)

[mshanemc]

for shared understanding: scratch orgs and sandboxes are the same infra. When all scratch org traffic hits the "current release" infrastructure, that's gonna impact sandboxes on the current release, unfortunately.

[scoleman-redhat]

Thanks for the information. This issue is crushing our CI/CD process, which relies on the availability of a pool of scratch orgs with managed package dependencies pre-installed. Preparation time for a single scratch org has gone from 2 hours to 12+. I'd like to try your suggestion to start using preview scratch orgs, but receive this message when I add release: Preview to my scratch org definition:

ERROR running force:org:create: At this time, we are outside of the preview period. You can create only current release scratch orgs.

Any thoughts?

[mshanemc]

ERROR running force:org:create: At this time, we are outside of the preview period. You can create only current release scratch orgs.

My fault. Use lowercase preview. It doesn't like Preview. I changed the issue. 🤷🏻

[dribb-sprout]

Use lowercase preview. It doesn't like Preview. I changed the issue. 🤷🏻

FWIW - I'm encountering the same behavior as @scoleman-redhat but with lowercase preview in my definition file. I'm also receiving the same error when trying to override the value via the command-line like so:

sfdx force:org:create --setalias development -f config/project-scratch-def.json release=Preview

I've confirmed that I'm on the latest version of sfdx:

sfdx-cli/7.165.0 darwin-x64 node-v16.17.0

[mshanemc]

use release=preview on the command line.

[dribb-sprout]

use release=preview on the command line.

I'm not sure what changed but I just tried the lowercase version again in my definition file and now it's working... And I confirmed that release=preview on the command-line also seems to work.

[Julian88Tex]

Linking to a few other threads incase anyone else provides some relevant info there:

CumulusCI Community Group: https://trailhead.salesforce.com/trailblazer-community/feed/0D54S00000JdFmaSAF SFDX Community Group: https://trailhead.salesforce.com/trailblazer-community/feed/0D54S00000JdEyjSAF

[scoleman-redhat]

FYI - Across numerous runs of sfdx force:org:create with "release": "preview" (lower case p) in the scratch org definition, I'm still intermittently observing the "At this time, we are outside of the preview period. You can create only current release scratch orgs." error. I haven't done the math but eyeballing my logs it looks like it happens maybe 25% of the time. All other things are equal with these runs, it only fails sometimes. If I can create a reproducible scenario outside my environment I'll file a bug.

[aozomaro]

Does this suggestion work? What do I do about installing the preview version in a non preview org? Will I be allowed to install the preview version? If not I do not see how the issue is solved at all

[gsbasso]

Modifying the scratch org config file is not the right answer either because package:version:create commands will fail.

For example, if you have package A with no dependencies and package B that depends on package A, creating a new version of package B (after creating a new version of package A) will fail with error:

An error occurred while trying to install a package dependency, ID <id of package A>

Further, you cannot deploy package A to a scratch org created with the preview option (which partially explains why the creation of dependent packages fails). The install command will fail with error:

Encountered errors installing the package!

Finally, if you attempt to view a newly created package in packagingSetupUI, it will not recognize the installation key (assuming your package has one).

I don't know if the above are normal or symptoms of other problems. But it seems the best option is to use release=preview on the command line and leave everything else unchanged.

[jeffreyazevedo]

FYI - Across numerous runs of sfdx force:org:create with "release": "preview" (lower case p) in the scratch org definition, I'm still intermittently observing the "At this time, we are outside of the preview period. You can create only current release scratch orgs." error. I haven't done the math but eyeballing my logs it looks like it happens maybe 25% of the time. All other things are equal with these runs, it only fails sometimes. If I can create a reproducible scenario outside my environment I'll file a bug.

@scoleman-redhat I'm also experiencing this behavior intermittently.

[Charles-Revelle]

This conversation is helpful, so thanks to all. We did update our scratch.def file to include "release": "preview" which greatly improved the performance (timing) of scratch org create, package version create, deploys, and installs. We too received the "At this time, we are outside of the preview period" which did not happen on a re-run. The issue now is that package versions created with preview in the def file can not be installed in to orgs that are not on the preview instance (like one of our test sandboxes). We received error: Mismatching versions. So had to back the preview out. A case is open with Salesforce support, but I'm not confident it will be help.

[Julian88Tex]

@mshanemc just wanted to follow up on the comparison and expectations. I took a look at our builds last preview (Summer '22) period and I did see what appeared to be a 50-100% increase for a short period but it was relatively infrequent at about 10-20% of the time. This time (Winter '23) around we are seeing on average a 500-600% increase occurring 40-50% of the time.

I realize this is "expected" to a certain degree and that you mentioned it has been getting worse, but it appears at first glance that this is exponentially worse.

[shahzebrizvi16]

Thanks everyone, your responses have been very helpful but I am still seeing errors in my org. At first I saw the following error At this time, we are outside of the preview period

Now after re-running, every time I run the command I am consistently seeing the error ERROR running force:org:create: The request to create a scratch org failed with error code: C-9998

I am running the following command as suggested: sfdx force:org:create -f config/project-scratch-def.json release=preview -v devHub -w 30

I even tried by adding release: preview to the scratch org definition file but I still saw the same failure with error code: C-9998.

[rob-baillie-ortoo]

Have been experimenting with this today, and am seeing the following:

• Yes, in general the performance of the new release scratch orgs is generally significantly better
• Periodically see the error 'At this time, we are outside of the preview period. You can create only current release scratch orgs' on attempting to create a scratch org.
• Periodically still get long wait times on creation of a scratch org (over 5 minutes)
• Periodically get a long wait followed by 'The org cannot be found' error

Conclusion - using "release":"preview" in the definition is a help, but it's not a solution.

The only actual solution is for Salesforce to ensure that the infrastructure they they put in place for development environments is up to the job. And since it fails to support the requirements for up to 18 weeks of the year it simply isn't up to the job.

[ipavlic]

What you can do

Invest into your fixed amount of non-prod infrastructure.

[Julian88Tex]

Tentatively some good news. Looks like our build times are starting to go back down. The last 5x job was 21 hours ago for us.

[vnehess]

This has caused several problems for our CI build process. Normally we wait -w 37 min to create a scratch org, these org-create operations are timing out.
Normally we wait -w 25 min to deploy code, these are timing out at 50m today.

we are also seeing the following error when running deploy, no clue what this could be :

ERROR running force:source:deploy: Metadata API request failed: UNKNOWN_EXCEPTION: common.exception.SfdcSqlException ORA-06550: line 1, column 13: PLS-00306: wrong number or types of arguments in call to 'MARK_USED_AUTO' ORA-06550: line 1, column 7: PL/SQL: Statement ignored

This almost certainly means your PL/SQL is out of sync with your Java. Rebuild and then run ./ant plsql.

SQLException while executing plsql statement: {?=call cApiCursor.mark_used_auto(?)}(01g5400000qmdu6)

I would like to voice my opinion that the Scratch Org infrastructure is critical to ISV developers and more important than the Preview program (IMO). Scratch Org infrastructure should be resourced in alignment with the Salesforce Value:: Trust.

[azlam-abdulsalam]

Does this suggestion work? What do I do about installing the preview version in a non preview org? Will I be allowed to install the preview version? If not I do not see how the issue is solved at all

You could set the sourceApiVersion to a defined version in your sfdx project.json, even though the scratch org will be in preview, the package created will be compatible on non preview orgs

[imTachu]

Thanks for the workaround @mshanemc, as others already did, I would like to press the point in the slowness of Scratch orgs generation but even more relevant when generating package versions. Here's the plot I have attached to my ongoing support case (43086260) in which it can be seen that at the beginning of 2021 it was taking around 10 mins to generate a package version and this year (different days) it has gone up to 3 hours and a half! ⚠️

Separately and as others have pointed out, having three major releases in a year that impact everyone's development process is really suboptimal.

[tfink-cc]

By the risk of repeating the finger pointing done already in this thread, I can only agree that it is difficult to understand that Salesforce is not learning from this and running (us!) into the similar situation at each major release.

Assuming that you @mshanemc are an employee of Salesforce and the only one (?) in this thread, maybe you could speak up internally and bring attention to this?

We ISVs work day in day out for our products running on Salesforce and in the end making (a lot of) money for Salesforce. I think it might be fair to say that we would appreciate, if Salesforce would look out for us a little more here and there. And this topic at hand is a perfect place to start that. :)

Thank you!

[mshanemc]

Assuming that you @mshanemc are an employee of Salesforce and the only one (?) in this thread, maybe you could speak up internally and bring attention to this?

Oh, that's definitely been done. 😄

[Julian88Tex]

I cannot speak for @mshanemc but as a former Salesforce employee, I suspect he's likely as frustrated as we are :-(.

For this particular issue one strategy to get some more attention is to speak with your Partner Account Manager and make it your number one request (assuming that's what it is to you). Another thing I'm thinking about is looking for a current Idea Exchange issue or creating one myself. This is a longer-term strategy but will likely get a more diverse set of Salesforce PM eyes on it than a GitHub Issue.

[veaux]

So when upgrades to orgs used to mean the org was inaccessible for hours, but has now changed to it only being read only for 10 minutes, we need to do similar for ISVs, especially during preview window, in this area. This doesn't work.

Many of us are actually creating more activity during this period either trying to beat the release or do updates and release something shortly after it comes out.

[mikekozub]

This is also negatively impacting our CICD processes. All sfdx commands across the board are excruciatingly slow, most of the time requiring restarting our jobs due to timeout errors. An example being that creating new package versions using force:package:version:create has gone from 15 minutes to 3+ hours. Our force:source:push commands have gone from 2 minutes to 30+ minutes. Also echoing what everybody else here has said about times with scratch org creation. We only recently adopted and introduced sfdx packaging into our org so this is our first time experiencing this, but does this truly happen every release, 3 months out of the year?