edgetpu icon indicating copy to clipboard operation
edgetpu copied to clipboard

Published 20 hours ago • verified publisher icon google-coral

fastboot does not work on Coral DevBoard Mini, be frozen with a message "musb-hdrc: peripheral reset irq lost!"

Open in-die-nibelungen opened this issue 2 years ago • 9 comments

Description

After an illegal power off, my Coral Dev Board Mini have not been able to boot up. The attached CoralDevBoardMini_Boot_Failed.txt is the log message when booting got from serial console. Found some "failed" and "Could not" among this log such as:

  • SELinux: Could not open policy file <= /etc/sel[ 6.261053] systemd[1]: System time before build time, advancing clock.
  • [ 6.301122] systemd[1]: Failed to find module 'autofs4'
  • [ 6.372352] [MT6620][nvram_read] : failed to open!!

But, I actually don't know what's the root cause of this problem among this log. I really appreciate it if someone could tell it to me :-)

Anyway, I found the board could not boot up, so I tried to reflash the board using fastboot method, but it failed.

I followed the steps of "force-boot-into-fastboot-mode" after finishing first-time-setup on Ubuntu. In step.7, that prompt never returned after powering on the board. The log output got from serial console is shown in "Relevant Log Output".

The followings are the last 5 lines of the log.

gpio: pin 42 (gpio 42) value is 0 Saving Environment to MMC... Writing to MMC(0)... OK gpio: pin 19 (gpio 19) value is 0 gpio: pin 18 (gpio 18) value is 1 musb-hdrc: peripheral reset irq lost!

It seemed that the board was getting into 'fastboot' mode, but failed somehow. I really appreciate it if someone could tell me how I can fix this.

The host OS is Ubuntu 16.04 with python 3.7.3.

Many thanks in advance

Click to expand!

Issue Type

Support

Operating System

No response

Coral Device

Dev Board Mini

Other Devices

No response

Programming Language

No response

Relevant Log Output

NOTICE:  BL2: v2.3():c1984dc174fa
NOTICE:  BL2: Built : 08:07:51, Oct  2 2020
NOTICE:  CHIPSET: MT8167
[PWRAP] pmic ID: 2092.
[EMI] MDL number = 0
[EMI] LPDDR3
Vcore HV NV LV, Vdram HV NV LV
0x5f 0x48 0x3f, 0xb 0x0 0x3
[EMI] Use default emigen emi settings
[EMI] Config emi settings:
EMI_CONA=0x2a052,   EMI_CONH=0x3
EMI_RAMK0=0x40000000, EMI_RAMK1=0x40000000
EMI_CONA=0x2a052
EMI_CONF=0x4210000
EMI_CONH=0x3

[DramcSwImpedanceCal] FINAL: DRVP=8, DRVN=7

LPDDR3 Pinmux 2
SSC OFF
DRAM Clock: 1584MHz

[Write Leveling]
WriteLevelingMoveDQSInsteadOfCLK
===============================================================================
Dram Type= 3, Freqency= 1600, rank 1
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
WL Clk delay = 0, CA CLK delay = 0
No need to update CA/CS delay because the CLK delay is small than CA training.
Final Clk output delay = 0
R1 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS0 delay =  26
R1 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS1 delay =  26
R1 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS2 delay =  25
R1 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS3 delay =  25
[DramcWriteLeveling] ====Done====
[DramRankNumberDetection] 2, 0x0
LPDDR3 Pinmux 2
SSC OFF
DRAM Clock: 1584MHz

[CATrainingLP3]
FINAL: CA0  (1~31) 16
FINAL: CA1  (0~30) 15
FINAL: CA2  (1~31) 16
FINAL: CA3  (0~30) 15
FINAL: CA5  (0~31) 15
FINAL: CA6  (0~30) 15
FINAL: CA7  (0~30) 15
FINAL: CA8  (2~32) 17
FINAL: CA4  (1~31) 16
FINAL: CA9  (2~31) 16
=========================================
u4GoldenPattern 0xrank0:
Macro0 Clk Dealy is 0, CA delay is 15
rank1:
Macro0 Clk Dealy is 0, CA delay is 15
[CATrainingLP3] ====Done====


[Write Leveling]
WriteLevelingMoveDQSInsteadOfCLK
===============================================================================
Dram Type= 3, Freqency= 1600, rank 0
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
WL Clk delay = 0, CA CLK delay = 0
No need to update CA/CS delay because the CLK delay is small than CA training.
Final Clk output delay = 0
R0 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS0 delay =  27
R0 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS1 delay =  26
R0 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS2 delay =  27
R0 FINAL: WriteLeveling DQS:(2, 2) OEN:(2, 0) DQS3 delay =  26
[DramcWriteLeveling] ====Done====

[Gating]
===============================================================================
Dram Type= 3, Freqency= 1600, rank 0
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
[Byte 0]First pass (1, 6, 12)
[Byte 1]First pass (1, 6, 12)
[Byte 2]First pass (1, 6, 12)
[Byte 3]First pass (1, 6, 12)
[Byte 0]Bigger pass win(2, 0, 5)  Pass tap=57
[Byte 1]Bigger pass win(2, 0, 5)  Pass tap=57
[Byte 3]Bigger pass win(2, 0, 5)  Pass tap=57
[Byte 2]Bigger pass win(2, 0, 6)  Pass tap=58
===============================================================================
    dqs input gating widnow, final delay value
    Frequency=1600  rank=0
===============================================================================
R0 FINAL: GW best DQS0 P0 delay(2T, 0.5T, PI) = (1, 7, 8) [tap = 57]
R0 FINAL: GW best DQS1 P0 delay(2T, 0.5T, PI) = (1, 7, 8) [tap = 57]
R0 FINAL: GW best DQS2 P0 delay(2T, 0.5T, PI) = (1, 7, 9) [tap = 58]
R0 FINAL: GW best DQS3 P0 delay(2T, 0.5T, PI) = (1, 7, 8) [tap = 57]
R0 FINAL: GW best DQS0 P1 delay(2T, 0.5T, PI) = (2, 1, 8)
R0 FINAL: GW best DQS1 P1 delay(2T, 0.5T, PI) = (2, 1, 8)
R0 FINAL: GW best DQS2 P1 delay(2T, 0.5T, PI) = (2, 1, 9)
R0 FINAL: GW best DQS3 P1 delay(2T, 0.5T, PI) = (2, 1, 8)
[DramcRxdqsGatingCal] ====Done====

[DATLAT]
DATLAT Default value = 0x10
5, 0x6, 0x7, 0x8, 0x9, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0xpattern=5 first_step=15 total pass=6 best
R0 FINAL: DATLAT = 17 [15 ~ 20]
[DramcRxdatlatCal] ====Done====

[RX]
===============================================================================
Dram Type= 3, Freqency= 1600, rank 0
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
RX Window Sum 851
R0 FINAL: RX Bit 0, 15 (3 ~ 27) 25
R0 FINAL: RX Bit 1, 13 (1 ~ 25) 25
R0 FINAL: RX Bit 2, 12 (-1 ~ 26) 28
R0 FINAL: RX Bit 3, 13 (1 ~ 25) 25
R0 FINAL: RX Bit 4, 17 (5 ~ 29) 25
R0 FINAL: RX Bit 5, 16 (1 ~ 31) 31
R0 FINAL: RX Bit 6, 16 (3 ~ 30) 28
R0 FINAL: RX Bit 7, 16 (3 ~ 30) 28
R0 FINAL: RX Bit 8, 17 (4 ~ 30) 27
R0 FINAL: RX Bit 9, 16 (4 ~ 29) 26
R0 FINAL: RX Bit 10, 15 (1 ~ 30) 30
R0 FINAL: RX Bit 11, 17 (5 ~ 30) 26
R0 FINAL: RX Bit 12, 17 (2 ~ 32) 31
R0 FINAL: RX Bit 13, 18 (6 ~ 30) 25
R0 FINAL: RX Bit 14, 18 (5 ~ 31) 27
R0 FINAL: RX Bit 15, 17 (2 ~ 32) 31
R0 FINAL: RX Bit 16, 16 (2 ~ 31) 30
R0 FINAL: RX Bit 17, 17 (2 ~ 33) 32
R0 FINAL: RX Bit 18, 15 (1 ~ 30) 30
R0 FINAL: RX Bit 19, 17 (3 ~ 31) 29
R0 FINAL: RX Bit 20, 13 (1 ~ 25) 25
R0 FINAL: RX Bit 21, 14 (1 ~ 27) 27
R0 FINAL: RX Bit 22, 13 (1 ~ 25) 25
R0 FINAL: RX Bit 23, 13 (0 ~ 26) 27
R0 FINAL: RX Bit 24, 17 (4 ~ 30) 27
R0 FINAL: RX Bit 25, 17 (6 ~ 29) 24
R0 FINAL: RX Bit 26, 17 (3 ~ 32) 30
R0 FINAL: RX Bit 27, 18 (4 ~ 32) 29
R0 FINAL: RX Bit 28, 17 (4 ~ 30) 27
R0 FINAL: RX Bit 29, 16 (2 ~ 31) 30
R0 FINAL: RX Bit 30, 17 (5 ~ 29) 25
R0 FINAL: RX Bit 31, 15 (2 ~ 29) 28
===============================================================================
Dram Type= 3, Freqency= 1600, rank 0
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
DQS Delay :
DQS0 = 0, DQS1 = 0, DQS2 = 0, DQS3 = 0
DQM Delay :
DQM0 = 14, DQM1 = 16, DQM2 = 14, DQM3 = 16
DQ Delay :
DQ0 =15, DQ1 =13, DQ2 =12, DQ3 =13
DQ4 =17, DQ5 =16, DQ6 =16, DQ7 =16
DQ8 =17, DQ9 =16, DQ10 =15, DQ11 =17
DQ12 =17, DQ13 =18, DQ14 =18, DQ15 =17
DQ16 =16, DQ17 =17, DQ18 =15, DQ19 =17
DQ20 =13, DQ21 =14, DQ22 =13, DQ23 =13
DQ24 =17, DQ25 =17, DQ26 =17, DQ27 =18
DQ28 =17, DQ29 =16, DQ30 =17, DQ31 =15
________________________________________________________________________
[DramcRxWindowPerbitCal] ====Done====

[TX]
[DramcTxWindowPerbitCal] Frequency=1600, Rank=0, calType=2
[DramcTxWindowPerbitCal] Begin, TX DQ(2, 2),  DQ OEN(2, 0)
TX Window Sum 728
===============================================================================
Dram Type= 3, Freqency= 1600, rank 0
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
R0 FINAL: TX Bit0 (32~55) 24 43,   Bit8 (34~56) 23 45,  Bit16 (33~53) 21 43,   Bit24 (34~55) 22 44
R0 FINAL: TX Bit1 (28~51) 24 39,   Bit9 (32~56) 25 44,  Bit17 (32~53) 22 42,   Bit25 (33~55) 23 44
R0 FINAL: TX Bit2 (31~54) 24 42,   Bit10 (32~56) 25 44,  Bit18 (31~53) 23 42,   Bit26 (34~55) 22 44
R0 FINAL: TX Bit3 (32~52) 21 42,   Bit11 (35~56) 22 45,  Bit19 (31~53) 23 42,   Bit27 (34~55) 22 44
R0 FINAL: TX Bit4 (35~55) 21 45,   Bit12 (34~57) 24 45,  Bit20 (31~50) 20 40,   Bit28 (33~55) 23 44
R0 FINAL: TX Bit5 (34~55) 22 44,   Bit13 (34~58) 25 46,  Bit21 (31~53) 23 42,   Bit29 (32~55) 24 43
R0 FINAL: TX Bit6 (34~55) 22 44,   Bit14 (34~56) 23 45,  Bit22 (31~51) 21 41,   Bit30 (32~55) 24 43
R0 FINAL: TX Bit7 (34~55) 22 44,   Bit15 (35~56) 22 45,  Bit23 (31~52) 22 41,   Bit31 (32~55) 24 43

==================================================================
Byte0, PI DQ Delay 42 Delay2 43
Final DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,2, 43)
OEN DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,0, 43)

Byte1, PI DQ Delay 45 Delay2 45
Final DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,2, 45)
OEN DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,0, 45)

Byte2, PI DQ Delay 41 Delay2 41
Final DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,2, 41)
OEN DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,0, 41)

Byte3, PI DQ Delay 43 Delay2 44
Final DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,2, 44)
OEN DQ PI Delay(LargeUI, SmallUI, PI) =(2 ,0, 44)

[DramcTxWindowPerbitCal] ====Done====

[Gating]
===============================================================================
Dram Type= 3, Freqency= 1600, rank 1
odt_onoff= 0, Byte mode= 0, Read_DBI= 0, Write DBI= 0
===============================================================================
[Byte 0]First pass (1, 6, 12)
[Byte 1]First pass (1, 6, 12)
[Byte 2]First pass (1, 6, 12)
[Byte 3]First pass (1, 6, 12)
[Byte 1]Bigger pass win(2, 0, 5)  Pass tap=57
[Byte 3]Bigger pass win(2, 0, 5)  Pass tap=57
[Byte 0]Bigger pass win(2, 0, 6)  Pass tap=58
[Byte 2]Bigger pass win(2, 0, 6)  Pass tap=58
===============================================================================
    dqs input gating widnow, final delay value
    Frequency=1600  rank=1
===============================================================================
R1 FINAL: GW best DQS0 P0 delay(2T, 0.5T, PI) = (1, 7, 9) [tap = 58]
R1 FINAL: GW best DQS1 P0 delay(2T, 0.5T, PI) = (1, 7, 8) [tap = 57]
R1 FINAL: GW best DQS2 P0 delay(2T, 0.5T, PI) = (1, 7, 9) [tap = 58]
R1 FINAL: GW best DQS3 P0 delay(2T, 0.5T, PI) = (1, 7, 8) [tap = 57]
R1 FINAL: GW best DQS0 P1 delay(2T, 0.5T, PI) = (2, 1, 9)
R1 FINAL: GW best DQS1 P1 delay(2T, 0.5T, PI) = (2, 1, 8)
R1 FINAL: GW best DQS2 P1 delay(2T, 0.5T, PI) = (2, 1, 9)
R1 FINAL: GW best DQS3 P1 delay(2T, 0.5T, PI) = (2, 1, 8)
[DramcRxdqsGatingCal] ====Done====

[DATLAT]
DATLAT Default value = 0x11
5, 0x6, 0x7, 0x8, 0x9, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0xpattern=5 first_step=15 total pass=6 best
R1 FINAL: DATLAT = 17 [15 ~ 20]
[DramcRxdatlatCal] ====Done====
[DramcRxdqsGatingPostProcess] p->frequency 1600
[DramcRxdqsGatingPostProcess] s1ChangeDQSINCTL 0, reg_TX_dly_DQSgated_min 3, u1TXDLY_Cal_min 3
[DramcDualRankRxdatlatCal] RANK0: 17, RANK1: 17, Final_Datlat 17
[Get MRR] Vendor 5.
[Get MRR] MR8 1f1f.
[Get MRR] Rank 0, u1DieNumber 1, Desity 0x40000000.
[Get MRR] MR8 1f1f.
[Get MRR] Rank 1, u1DieNumber 1, Desity 0x40000000.
[vDramcUpdateEmiSetting] EMI_CONA 0x2a052, EMI_CONH 0x3
[vDramcACTimingOptimize] Density 7, u1TRFC 73, u1TRFCpb 25, u1TRFC_05T 0, u1TXREFCNT 86

Settings after calibration ...
=== [DramcRunTimeConfig] ===
HW_GATING: ON
DUMMY_READ_FOR_TRACKING: ON
DFS_HW_SYNC_GATING_TRACKING: OFF
ZQCS_ENABLE: ON
LOWPOWER_GOLDEN_SETTINGS(DCM): ON
SPM_CONTROL_AFTERK: ON
TEMP_SENSOR_ENABLE: OFF
ENABLE_PER_BANK_REFRESH: OFF
HW_SAVE_FOR_SR: OFF
=========================
[MEM] complex R/W mem test pass
NOTICE:  BL2: Booting BL31


U-Boot 2019.10 (Dec 09 2020 - 23:51:30 +0000), Build: jenkins-excelsior.excelsior-bootloader-2

CPU:   MediaTek MT8516
DRAM:  512 MiB
WDT:   Started with servicing (60s timeout)
MMC:   mmc@11120000: 0
Loading Environment from MMC... OK
In:    serial@11005000
Out:   serial@11005000
Err:   serial@11005000
8GB mmc detected
Hit any key to stop autoboot:  0
gpio: pin 42 (gpio 42) value is 0
Saving Environment to MMC... Writing to MMC(0)... OK
gpio: pin 19 (gpio 19) value is 0
gpio: pin 18 (gpio 18) value is 1
musb-hdrc: peripheral reset irq lost!

in-die-nibelungen avatar Jul 01 '22 06:07 in-die-nibelungen

Can you please try to flash the board with Ubutnu 18.04 or Ubuntu 20.04. Thanks!

hjonnala avatar Jul 01 '22 14:07 hjonnala

@hjonnala , Thanks for your comment!

I tried it now on Ubuntu 20.04 with python 3.8.10, but almost no change on both of host PC terminal and the serial console outputs.

One change on the host PC's terminal is that the following texts appears after running "bash enable_lk_fastboot.sh"

$ bash enable_lk_fastboot.sh
********************************************************************************
Unable to connect to debug board, check connection or manually toggle GPIO.
********************************************************************************
.
.
.

I've never seen this on Ubuntu 16.04. What does this mean? Especially, I'm concerned about "manually toggle GPIO". Do I have to do something before fastboot or connect DevBoard Mini and the host PC with a serial cable?

in-die-nibelungen avatar Jul 01 '22 22:07 in-die-nibelungen

Those logs are expected. Here is the complete reflash log: https://github.com/hjonnala/snippets/blob/main/devboardmini/flashlogs.md

hjonnala avatar Jul 01 '22 22:07 hjonnala

@hjonnala , Thanks for the prompt replay.

Those logs are expected. Here is the complete reflash log: https://github.com/hjonnala/snippets/blob/main/devboardmini/flashlogs.md That's very helpful. Could you tell me how long it will take for "INFO: Got /dev/ttyACM0" to appear after "INFO: Waiting to connect platform..." show up? As for me, waited for more than 10 seconds after "INFO: Waiting to connect platform..." showed up (but nothing appeared on the terminal :( ).

The text outputs of "bash enable_lk_fastboot.sh" are as follows (seeming no problem):

$ bash enable_lk_fastboot.sh
**********************************************************************
Unable to connect to debug board, check connection or manually toggle GPIO.
**********************************************************************
/tmp/tmp.Hva3P9HpR1 ~/Work/mdt/excelsior-eagle-20201210233645
INFO: pySerial version: (3.5)
INFO: Use config file: /tmp/tmp.Hva3P9HpR1/dl_addr.ini
INFO: Waiting to connect platform...

@hjonnala , have you got the log outputs of the serial console in fastboot mode? Here are some last lines of the log.

U-Boot 2019.10 (Dec 09 2020 - 23:51:30 +0000), Build: jenkins-excelsior.excelsior-bootloader-2

CPU:   MediaTek MT8516
DRAM:  512 MiB
WDT:   Started with servicing (60s timeout)
MMC:   mmc@11120000: 0
Loading Environment from MMC... OK
In:    serial@11005000
Out:   serial@11005000
Err:   serial@11005000
8GB mmc detected
Hit any key to stop autoboot:  0
gpio: pin 42 (gpio 42) value is 0
Saving Environment to MMC... Writing to MMC(0)... OK
gpio: pin 19 (gpio 19) value is 0
gpio: pin 18 (gpio 18) value is 1
musb-hdrc: peripheral reset irq lost!

I'd like to know if the log shown above is an expected one or not, and how to fix if it's not expected.

Many thanks in advance.

in-die-nibelungen avatar Jul 03 '22 23:07 in-die-nibelungen

musb-hdrc: peripheral reset irq lost!

This is not expected. I think re flashing the board would fix this issue. Please refer to the serial console logs at: https://github.com/hjonnala/snippets/blob/main/devboardmini/serailconsole_logs.txt#L285

hjonnala avatar Jul 05 '22 16:07 hjonnala

@hjonnala , Thanks for the respond.

musb-hdrc: peripheral reset irq lost!

This is not expected. I think re flashing the board would fix this issue.

I understood this is not expected. Since the boot process is frozen after showing this message in the serial console when I enter it into 'fastboot' mode, I can't run re-flashing. So, I'm having trouble on what to do next.

Please refer to the serial console logs at: https://github.com/hjonnala/snippets/blob/main/devboardmini/serailconsole_logs.txt#L285 Thanks. I looked through it and found it's the log in normal boot mode, right? I'd like to see one in fastboot mode, if possible.

Thanks,

in-die-nibelungen avatar Jul 05 '22 23:07 in-die-nibelungen

Hello @in-die-nibelungen I don't see any logs on serial console while doing fastboot and It is recommended to connect Dev board mini only with usb cable while doing fastboot.

$ bash enable_lk_fastboot.sh

Unable to connect to debug board, check connection or manually toggle GPIO.

/tmp/tmp.Hva3P9HpR1 ~/Work/mdt/excelsior-eagle-20201210233645 INFO: pySerial version: (3.5) INFO: Use config file: /tmp/tmp.Hva3P9HpR1/dl_addr.ini INFO: Waiting to connect platform...

please check these comments for the next steps after these steps: https://github.com/google-coral/edgetpu/issues/505#issuecomment-972943815, https://github.com/google-coral/edgetpu/issues/505#issuecomment-973698267

hjonnala avatar Jul 06 '22 02:07 hjonnala

I have the same issue. Did you manage to fix it @in-die-nibelungen ?

reynhardtstef avatar Jul 12 '22 17:07 reynhardtstef

@reynhardtstef , not yet.

Followed @hjonnala 's advice below, but didn't fix it.

Hello @in-die-nibelungen I don't see any logs on serial console while doing fastboot and It is recommended to connect Dev board mini only with usb cable while doing fastboot.

However, my board can boot up OS now after several trials of booting (without fastboot), where tried more than 40 or 50 times. Let me know if you would like to know what I did more details. I don't think it's recommended way and don't know it's helpful for your situation.

in-die-nibelungen avatar Jul 14 '22 02:07 in-die-nibelungen