mina icon indicating copy to clipboard operation
mina copied to clipboard
verified publisher icon MinaProtocol

Devnet failures investigation (June 2025)

Open georgeee opened this issue 9 months ago • 3 comments

During our latest release testing on devnet, we've identified two critical issues that are currently blocking the release of the compatible branch:

  1. Node halt issue: A devnet node abruptly stops producing logs and any other observable activity, requiring a manual restart to resume operations.

  2. Prover failure: We're experiencing rare prover failures that may or may not be connected to the node halt issue.

This issue will serve as the central tracking point for both problems as we work to resolve these blockers.

georgeee avatar Jun 11 '25 10:06 georgeee

List of PRs in between the last successful release and the failing release

  • PR #17253
  • PR #17229
  • PR #16463
  • PR #17207
  • PR #17012
  • PR #17210
  • PR #17147
  • PR #17219
  • PR #17208
  • PR #17159
  • PR #17204
  • PR #17205
  • PR #17203
  • PR #17198
  • PR #17195
  • PR #17196
  • PR #17200
  • PR #17201
  • PR #17193
  • PR #17156
  • PR #17187
  • PR #17171
  • PR #17176
  • PR #17184
  • PR #17174
  • PR #17175
  • PR #17173
  • PR #17172
  • PR #17169
  • PR #17124
  • PR #17153
  • PR #17165
  • PR #17154
  • PR #16966
  • PR #17116
  • PR #17161
  • PR #16580
  • PR #17115
  • PR #17150
  • PR #17095
  • PR #17141
  • PR #17075
  • PR #17066
  • PR #17064
  • PR #17060
  • PR #17135
  • PR #17119
  • PR #17097
  • PR #17118
  • PR #17128
  • PR #17127
  • PR #17122
  • PR #17123
  • PR #16770
  • PR #17065
  • PR #17068
  • PR #17067
  • PR #17114
  • PR #17106
  • PR #17094
  • PR #17070
  • PR #17069
  • PR #17098
  • PR #17054
  • PR #17053
  • PR #17050
  • PR #17036
  • PR #17105
  • PR #17104
  • PR #16713
  • PR #17102
  • PR #17099
  • PR #17093
  • PR #16661
  • PR #17051
  • PR #17085
  • PR #17091
  • PR #17090
  • PR #17039
  • PR #16925
  • PR #17073
  • PR #17061
  • PR #17071
  • PR #17057
  • PR #17058
  • PR #17056
  • PR #17078
  • PR #17031
  • PR #16965
  • PR #17048
  • PR #16968
  • PR #17034
  • PR #17033
  • PR #17035
  • PR #17030
  • PR #17043
  • PR #17016
  • PR #16972
  • PR #16974
  • PR #17023

georgeee avatar Jun 11 '25 11:06 georgeee

Related patches:

  • https://github.com/MinaProtocol/mina/pull/17392
  • https://github.com/MinaProtocol/mina/pull/17395
  • https://github.com/MinaProtocol/mina/pull/17399
  • https://github.com/MinaProtocol/mina/pull/17391

dannywillems avatar Jun 12 '25 17:06 dannywillems

By looking at the impact of individual patches, we reduced the list to

  • PR #17207
  • PR #17012
  • PR #17210
  • PR #17147
  • PR #17204
  • PR #17203
  • PR #17196
  • PR #17200
  • PR #17201
  • PR #17187
  • PR #17171
  • PR #17169
  • PR #17153
  • PR #16966
  • PR #17115
  • PR #17141
  • PR #16713
  • PR #17099
  • PR #17091
  • PR #17039
  • PR #16965
  • PR #16968
  • PR #17035
  • PR #17016
  • PR #16974

dannywillems avatar Jun 13 '25 15:06 dannywillems