Various Issues with Sipeed Tang Primer 20k

[bacintom]

Hi,

I'm getting started on FPGAs (apologies in advance for noob behavior).

I've bought a Sipeed Tang Primer 20k ( https://wiki.sipeed.com/hardware/en/tang/tang-primer-20k/primer-20k.html ) and I'm trying to use LiteX on it. The board is already available in litex-boards.

I have a few issues, also found some workarounds (no 'nice' solution though, still I hope it helps), I'll report about them here. I'm open to any suggestions if I could help improve support for this board.

Sorry it is quite long, I'll try to split into separate parts. Summary :

1. The reported amount of RAM is 256MB, but it should be 128?
2. Memory initialization fails (workaround: change latency settings)
3. SD card init fails (workaround: use spisdcard)
4. Building with --with-video-terminal fails (workaround: disable frequency check).
5. Trying to boot linux-on-litex-vexriscv: so far, stuck at "liftoff!"

---

Environment

• OS: ubuntu 23.04
• python 3.11.4
• litex updated today (Aug 20 2023) using ./litex_setup.py --update
• toolchain: Gowin_V1.9.8.11_Education_linux

---

1. The reported amount of RAM is 256MB, but it should be 128?

Building litex for my board, with the most default settings:

python3 -m litex_boards.targets.sipeed_tang_primer_20k --build --load

I get the following output from the serial:

        __   _ __      _  __
       / /  (_) /____ | |/_/
      / /__/ / __/ -_)>  <
     /____/_/\__/\__/_/|_|
   Build your hardware, easily!

 (c) Copyright 2012-2023 Enjoy-Digital
 (c) Copyright 2007-2015 M-Labs

 BIOS built on Aug 20 2023 13:33:28
 BIOS CRC passed (45f9f79d)

 LiteX git sha1: 1520d0f3

--=============== SoC ==================--
CPU:            VexRiscv @ 48MHz
BUS:            WISHBONE 32-bit @ 4GiB
CSR:            32-bit data
ROM:            128.0KiB
SRAM:           8.0KiB
SDRAM:          256.0MiB 16-bit @ 192MT/s (CL-6 CWL-5)
MAIN-RAM:       256.0MiB

According to the specs ( https://wiki.sipeed.com/hardware/en/tang/tang-primer-20k/primer-20k.html ), my board is supposed to have 1Gbit / 128MB of DDR3. Why does it say 256MB...?

2. Memory initialization fails

This is the second part of the serial output:

--========== Initialization ============--
Initializing SDRAM @0x40000000...
Switching SDRAM to software control.
Read leveling:
  m0, b00: |00000000| delays: -
  m0, b01: |00000000| delays: -
  m0, b02: |00000000| delays: -
  m0, b03: |00000000| delays: -
  best: m0, b02 delays: -
  m1, b00: |00000000| delays: -
  m1, b01: |00000000| delays: -
  m1, b02: |00000000| delays: -
  m1, b03: |00000000| delays: -
  best: m1, b02 delays: -
Switching SDRAM to hardware control.
Memtest at 0x40000000 (2.0MiB)...
  Write: 0x40000000-0x40200000 2.0MiB
   Read: 0x40000000-0x40200000 2.0MiB
  bus errors:  241/256
  addr errors: 0/8192
  data errors: 421599/524288
Memtest KO
Memory initialization failed

--============= Console ================--

I looked at similar github issues and attempted to fix it like this: try different latency values, calibrate, test. The tighest timing I found to work is cl=8, wcl=7. Commands below :

cd litedram/bench
python3 ./ddr3_mr_gen.py --cl=8 --cwl=7

The script generates the following commands to write on the LiteX BIOS:

sdram_mr_write 0 2624
sdram_mr_write 1 2054
sdram_mr_write 2 528

I then do

sdram_cal
sdram_test

It seems to work:

litex> sdram_mr_write 0 2624
Switching SDRAM to software control.
Writing 0x0a40 to MR0
Switching SDRAM to hardware control.

litex> sdram_mr_write 1 2054
Switching SDRAM to software control.
Writing 0x0806 to MR1
Switching SDRAM to hardware control.

litex> sdram_mr_write 2 528
Switching SDRAM to software control.
Writing 0x0210 to MR2
Switching SDRAM to hardware control.

litex> sdram_cal
Switching SDRAM to software control.
Read leveling:
  m0, b00: |00000000| delays: -
  m0, b01: |00000000| delays: -
  m0, b02: |01110000| delays: 02+-01
  m0, b03: |00000000| delays: -
  best: m0, b02 delays: 02+-01
  m1, b00: |00000000| delays: -
  m1, b01: |00000000| delays: -
  m1, b02: |01110000| delays: 02+-01
  m1, b03: |00000000| delays: -
  best: m1, b02 delays: 02+-01
Switching SDRAM to hardware control.

litex> sdram_test
Memtest at 0x40000000 (8.0MiB)...
  Write: 0x40000000-0x40800000 8.0MiB
   Read: 0x40000000-0x40800000 8.0MiB
Memtest OK

Next, I tried to go further and see if I could adjust the code so I don't have to type these lines every time I load litex on the board. But I failed to find where I'm supposed to specify the cl/cwl timings in litex-boards or litedram. Any suggestions?

3. SD card init fails

Now, building with:

python3 -m litex_boards.targets.sipeed_tang_primer_20k --with-sdcard --build --load

It seems the SD card doesn't work. For the record, I'm using a 32GB Sandisk Extreme Pro from ~2017.

litex> sdcard_detect
SDCard inserted.

litex> sdcard_init
Initialize SDCard... Failed.

Obviously I cannot boot from SDCard but using --with-spi-sdcard seems to work, at least more than --with-sdcard ( see 5 below)

4. Building with --with-video-terminal fails

The following command:

python3 -m litex_boards.targets.sipeed_tang_primer_20k --with-video-terminal --build

gives the following error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "litex-boards/litex_boards/targets/sipeed_tang_primer_20k.py", line 257, in <module>
    main()
  File "litex-boards/litex_boards/targets/sipeed_tang_primer_20k.py", line 246, in main
    builder.build(**parser.toolchain_argdict)
  File "litex/litex/soc/integration/builder.py", line 332, in build
    self.soc.finalize()
  File "litex/litex/soc/integration/soc.py", line 1304, in finalize
    Module.finalize(self)
  File "migen/migen/fhdl/module.py", line 156, in finalize
    subfragments = self._collect_submodules()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "migen/migen/fhdl/module.py", line 149, in _collect_submodules
    r.append((name, submodule.get_fragment()))
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "migen/migen/fhdl/module.py", line 102, in get_fragment
    self.finalize()
  File "migen/migen/fhdl/module.py", line 156, in finalize
    subfragments = self._collect_submodules()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "migen/migen/fhdl/module.py", line 149, in _collect_submodules
    r.append((name, submodule.get_fragment()))
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "migen/migen/fhdl/module.py", line 102, in get_fragment
    self.finalize()
  File "migen/migen/fhdl/module.py", line 157, in finalize
    self.do_finalize(*args, **kwargs)
  File "litex/litex/soc/cores/clock/gowin_gw1n.py", line 211, in do_finalize
    config = self.compute_config()
             ^^^^^^^^^^^^^^^^^^^^^
  File "litex/litex/soc/cores/clock/gowin_gw1n.py", line 188, in compute_config
    raise ValueError(f"Can't obtain requested frequency {diff_f} > {r_freq * margin}")
ValueError: Can't obtain requested frequency 125000.0 > 124875.0

I tried to disable the check, by commenting out the lines in litex/litex/soc/cores/clock/gowin_gw1n.py, around line 186:

# check if value fit criterion
#    if diff_f > r_freq*margin:
#        raise ValueError(f"Can't obtain requested frequency {diff_f} > {r_freq * margin}")

It seems to work on my display and I can see the text output by the LiteX BIOS. I don't know how to fix that "correctly" though. I guess I need to adjust a clock frequency somewhere?... I don't know.

5. Trying to boot linux-on-litex-vexriscv: so far, stuck at "liftoff!"

Call me optimistic, the tang primer 20k is not a supported board in linux-on-litex-vexriscv. But the similar tang nano 20k is, so I tried.

Mainly 2 methods:

Method 1:

I tried to add the following in linux-on-litex-vexriscv/make.py:

class Sipeed_tang_primer_20k(Board):
    soc_kwargs = {"l2_size" : 2048} # Use Wishbone and L2 for memory accesses.
    def __init__(self):
        from litex_boards.targets import sipeed_tang_primer_20k
        Board.__init__(self, sipeed_tang_primer_20k.BaseSoC, soc_capabilities={
            # Communication
            "serial",
            "spisdcard",
            "video_terminal",
        })

Method 2: python3 -m litex_boards.targets.sipeed_tang_primer_20k --with-sdcard --cpu-type vexriscv --cpu-variant linux --with-video-terminal --build --load

Method 1 & 2 so far give the same result (note: LiteX BIOS's behavior with arrow keys is weird with method 1). I get the following:

litex> sdcardboot
Booting from SDCard in SPI-Mode...
Booting from boot.json...
Copying Image to 0x40000000 (7531468 bytes)...
[########################################]
Copying rv32.dtb to 0x40ef0000 (2195 bytes)...
[########################################]
Copying rootfs.cpio to 0x41000000 (3781632 bytes)...
[########################################]
Copying opensbi.bin to 0x40f00000 (53640 bytes)...
[########################################]
Executing booted program at 0x40f00000

--============= Liftoff! ===============--

Then it is stuck forever at this point. I also tried without video-terminal. But video-terminal is useful to confirm I didn't just loose the serial connection. Anyway, always the same result.

---

Note: SDCard (not spi) gives:

litex> sdcardboot
Booting from SDCard in SD-Mode...
Booting from boot.json...
Booting from boot.bin...
SDCard boot failed.

... as expected because sdcard_init fails.

[enjoy-digital]

Hi @bacintom,

thanks a lot for the detailed report. The Tang Primer is currently probably not the best board to start with since all features haven't been validated yes (especially the DDR3, SDCard), but that's really interesting that you got the DDR3 working by adjusting cl/cwl since with this, I think you are the first to get no errors with the memtest: I'll have a closer look, try to understand and do the proper fixes in upstream. (Note: This is also related to https://github.com/enjoy-digital/litex/issues/1649).

I'll also look at the other issues/workaround while looking at this.

[trabucayre]

Hi @bacintom We have tried your fix for DDR3 with two tang Primer 20k variants:

• one with SKHynix chip: KO
• one with ICMax chip: OK

We have to find why there is a different behaviour between these two chips and how integrate your modification/update (at litedram phy level).

For RAM: size parameters are good and matchs RAM datasheet we have to investigate why a factor of two is introduces when size is computed.

Thanks!

[enjoy-digital]

Thanks @trabucayre, @bacintom: Can you tell us which DDR3 variant is mounted on your Primer (SKHynix or ICMax)?

[bacintom]

Hi! Thanks for yout replies! In my case, it is a SK Hynix. The mystery thickens. I somehow managed to take a photo with a readable part number, let me attach it here.

[bacintom]

@trabucayre to be honest I don't understand much about RAM but here's my take about the size issue:

I found a datasheet of H5TQ1Gx3EFR-xxx modules

In my case the exact number is H5TQ1G63EFR-PBC the "63" (unlike "83") means it is a 64M x 16 configuration

There is a table called "Row and column address table"

The right column should be used -There we have "row address" ranging from A0 to A12, I guess those are pin numbers... That's 13 bits, so the field 'nrows' of the DDR3Module should be 2^13 = 8192 (not 16384 as in the master) -Column address is A0-A9 -> 10bits -> 2^10=1024 OK

What do you think?

[e-yes]

Hi @bacintom could you try this (once again, not sure about proper timings): https://github.com/enjoy-digital/litex/issues/1649#issuecomment-1656509253

[bacintom]

@e-yes As long as I set nrows=8192 (the quickest way to test is, take an unmodified master and set the value in MT41J128M16) it will report the correct amount 128MiB, no other change needed.

So your change works and reports 128MiB. It will also fail the test in BIOS, and succeed after calling the sdram_mr_write/sdram_cal commands.

[nickoe]

Thanks to the fix in from @trabucayre in https://github.com/enjoy-digital/litedram/commit/6dadc11b21a8253aa20305f1a54c1a2f1f4bbfd6 I can make it pass and this configuration for the ddr3 module:

class H5TQ1G63EFR(DDR3Module):
    # geometry
    nbanks = 8
    nrows  = 8192
    ncols  = 1024
    # timings
    technology_timings = _TechnologyTimings(tREFI=64e6/8192, tWTR=(4, 7.5), tCCD=(4, None), tRRD=(4, 6), tZQCS=(64, 80))
    speedgrade_timings = {
        "800": _SpeedgradeTimings(tRP=15, tRCD=15, tWR=15, tRFC=(120, None), tFAW=(None, 50), tRAS=37.5),
        "1066": _SpeedgradeTimings(tRP=13.125, tRCD=13.125, tWR=15, tRFC=(160, None), tFAW=(None, 50), tRAS=37.5),
        "1600": _SpeedgradeTimings(tRP=13.75, tRCD=13.75, tWR=15, tRFC=(160, None), tFAW=(None, 40), tRAS=35),
    }
    speedgrade_timings["default"] = speedgrade_timings["800"]

Note the nrows as e-yes is also mentioning.

I am not sure how I got to those numbers as such, but some of them are either taken by searching for tRP numbers from other modules and probably https://github.com/enjoy-digital/litex/issues/1649#issuecomment-1656509253 -- but it is what I tested.

I get highest speed with the 800 timing, ... I don't know if that is expected.

  Write speed: 23.4MiB/s
   Read speed: 8.3MiB/s

For my 1066 and 1600 configration:

  Write speed: 22.2MiB/s
   Read speed: 7.8MiB/s

@bacintom after I got working sdram, the sdcard also worked for me. At least it would boot the demo.bin from a boot.json.

[bacintom]

@nickoe thanks I hadn't noticed something was pushed to litedram master.

With @trabucayre 's commit, the bios passes the sdram check successfully. No need to tweak with mode register commands anymore and that's nice

If I use @nickoe 's timings, sdcard (non-spi) also works. Well, at least I can go as far as spisdcard: it shows "liftoff!" and freezes.

So, issues 1,2,3 are pretty much fixed. Thanks everyone!

[verilogzhou]

@bacintom Regarding the liftoff issue, it can be seen from the following link that linux-on-litex-vexriscv supports vexriscv_smp,so it should be --cpu-type vexriscv_smp, but it will compile and prompt an error. https://github.com/litex-hub/linux-on-litex-vexriscv

[e-yes]

I have tried to compile the gateware for vexriscv_smp, but it doesn't fit and the compilation failed.

[bacintom]

Hi,

I got over my liftoff issue. I did the following :

-update all litex. Patches about supporting tang primer 20k have been pushed recently.

-check the files in the SD card using md5sum. One of my files had a wrong signature so I had to copy it again. I would recommend to double-check like this: unmount, unplug sd card, plug sd card, remount, verify checksum. This is to avoid being tricked by the OS, it could keep files in cache and skip reading from the actual SD card.

-building with linux-on-litex-vexriscv: ./make.py --board=sipeed_tang_primer_20k --toolchain=gowin --build --load (a patch added sipeed_tang_primer_20k on August 29th)

With this the board can boot linux!

I could also build with video_terminal, but it's unstable. Only part of the text appears on screen and the board will freeze before linux is fully booted.

[Sujunyou]

@bacintom hi , i firstly use litex to build tang-primer-20k linux , but in this linux , i find commands can not be uesd such as cd ,ls and so on. How do you know about these commands? Is there a corresponding instruction manual?

[kevinsu20]

hi , i use same command:./make.py --board=sipeed_tang_primer_20k --toolchain=gowin --build --load ,but i got same error like 4 : how do you solve this problem

[bacintom]

@Sujunyou cd and ls work fine in this linux, so I'm not sure what your problem is...

Are you sure you set up a linux?... If you just build litex it will bring you to the litex bios, only a few bios-level commands are available.

There are some explanations here: https://github.com/litex-hub/linux-on-litex-vexriscv in particular, you should download linux_2022_03_23.zip from https://github.com/litex-hub/linux-on-litex-vexriscv/issues/164 , format an SD card with a FAT16 or FAT32 partition and extract the files there. However, about the file rv32.dtb I use the one built by make.py in the linux-on-litex-vexriscv/images because it has a different signature, I'm not sure whether it is important.

[kevinsu20]

@Sujunyou cd and ls work fine in this linux, so I'm not sure what your problem is...

Are you sure you set up a linux?... If you just build litex it will bring you to the litex bios, only a few bios-level commands are available.

There are some explanations here: https://github.com/litex-hub/linux-on-litex-vexriscv in particular, you should download linux_2022_03_23.zip from litex-hub/linux-on-litex-vexriscv#164 , format an SD card with a FAT16 or FAT32 partition and extract the files there. However, about the file rv32.dtb I use the one built by make.py in the linux-on-litex-vexriscv/images because it has a different signature, I'm not sure whether it is important.

thanks fo your reply.i follow your steps to build linux(use command:python3 -m litex_boards.targets.sipeed_tang_primer_20k --with-sdcard --cpu-type vexriscv --cpu-variant linux --with-video-terminal --build --load )but i got following error: `ERROR (RP0006) : The number(61480(56055 LUTs, 1537 ALUs, 0 ROM16s, 648 SSRAMs)) of logic in the design exceeds the resource limit(20736) of current device ERROR (RP0001) : The number(76249) of DFF(DL) in the design exceeds the resource limit(16173) of current device(GW2A-LV18PG256C8/I7) GowinSynthesis finish

while executing

"run all" (file "run.tcl" line 11) Traceback (most recent call last): File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None, File "/usr/lib/python3.8/runpy.py", line 87, in _run_code exec(code, run_globals) File "/root/python-litex/litex-boards/litex_boards/targets/sipeed_tang_primer_20k.py", line 239, in main() File "/root/python-litex/litex-boards/litex_boards/targets/sipeed_tang_primer_20k.py", line 228, in main builder.build(**parser.toolchain_argdict) File "/root/python-litex/litex/litex/soc/integration/builder.py", line 367, in build vns = self.soc.build(build_dir=self.gateware_dir, **kwargs) File "/root/python-litex/litex/litex/soc/integration/soc.py", line 1332, in build return self.platform.build(self, *args, **kwargs) File "/root/python-litex/litex/litex/build/gowin/platform.py", line 43, in build return self.toolchain.build(self, *args, **kwargs) File "/root/python-litex/litex/litex/build/generic_toolchain.py", line 123, in build self.run_script(script) File "/root/python-litex/litex/litex/build/gowin/gowin.py", line 150, in run_script raise OSError("Error occured during Gowin's script execution.") OSError: Error occured during Gowin's script execution. `it seems resources is not available.

[bacintom]

@sususjysjy The issue 4 is not solved, I just disable the check that fails by editing gowin_gw1n.py, it's also fully explained how to do it in my first post...

[bacintom]

@sususjysjy I'm confused, I was replying to @Sujunyou there, are you the same person?

I get the same error as you nowadays if I use litex_boards, the error means the design won't fit in the FPGA. But it works if I use linux-on-litex-vexriscv/make.py instead... I don't know why 🤷

[kevinsu20]

@sususjysjy I'm confused, I was replying to @Sujunyou there, are you the same person?

I get the same error as you nowadays if I use litex_boards, the error means the design won't fit in the FPGA. But it works if I use linux-on-litex-vexriscv/make.py instead... I don't know why 🤷

@sususjysjy I'm confused, I was replying to @Sujunyou there, are you the same person?

I get the same error as you nowadays if I use litex_boards, the error means the design won't fit in the FPGA. But it works if I use linux-on-litex-vexriscv/make.py instead... I don't know why 🤷

sorry,these all my account. I find reasons following: in litex,if use above command,generate all the sdcard; but if you use linux-on-litex-vexriscv , it generates spisdcard, it's smaller than litex

[bacintom]

@sususjysjy I can build with make.py and enable sdcard instead of spisdcard. It will build successfully. The size difference must be somewhere else?

But sdcard+linux-on-litex-vexriscv doesn't work on master with the tang primer 20k... I've just tried with @nickoe 's RAM timings, but I could not boot, sdcard_init fails again. 🤷🤷🤷

So, I think it's better to use spisdcard for now.

[kevinsu20]

yeah,i use make.py, spi sdcard can be built succeessfully ; but in litex, i use command python3 -m litex_boards.targets.sipeed_tang_primer_20k --with-spi-sdcard --build , it will show me resources are not available。 i'm so confused for this

[kevinsu20]

@sususjysjy I can build with make.py and enable sdcard instead of spisdcard. It will build successfully. The size difference must be somewhere else?

But sdcard+linux-on-litex-vexriscv doesn't work on master with the tang primer 20k... I've just tried with @nickoe 's RAM timings, but I could not boot, sdcard_init fails again. 🤷🤷🤷

So, I think it's better to use spisdcard for now.

hi , you said "you can use make.py and enable sdcard",in make.py,it builds spisdcard , should i convert spisdcard to sdcard? The following is the default version：

if i convert to sdcard , it will occurr error May I take a look at your project？

[e-yes]

@sususjysjy what version of GOWIN IDE do you use?

[kevinsu20]

@sususjysjy what version of GOWIN IDE do you use?

thanks for your reply i use the latest version 1.9.8.11_education now i meet the new problem , i build .fs and load to tang-primer-20k , and bios work fine but when i want to boot linux(provided by (linux-on-litex-vexriscv) from sdcard , terminal shows following error: i don't know what's wrong

[e-yes]

You are on the right path I believe. Can you give us more details? I am even jealous, that I didn't come so close to solving this puzzle)

[kevinsu20]

You are on the right path I believe. Can you give us more details? I am even jealous, that I didn't come so close to solving this puzzle)

i use linux-on-litex-vexriscv , use the command ./make.py --board=sipeed_tang_primer_20k --toolchain=gowin --build ,then i load the .fs to board . and then , i put five files in sdcard ,and boot the board finally , i see the following terminal:

[e-yes]

Almost there, bro https://codebrowser.dev/linux/linux/drivers/soc/litex/litex_soc_ctrl.c.html#48

[bentwire]

Ok this is amazing. I had no idea this board had the FPGA resources to boot linux on! Is the vexriscv core the only one to use for this?

[e-yes]

I suspect vexriscv_smp is required at least for these prebuilt images.

[kevinsu20]

i have solve this problem.thanks all