porting to a ZCU102 and ZCU111?

[sirjansel0t]

Hello,

I am curious how to make the 10 GbE core work on the ZCU102 and ZCU111.

The ZCU111 have these extra signals:

SFP_TX_FAULT
SFP_TX_DISABLE J
SFP_MOD_DETECT
SFP_RS0
SFP_RS1
SFP_LOS

While the ZCU102 only has:

SFP0_TX_P
SFP0_TX_N
SFP0_RX_P
SFP0_RX_N
SFP0_TX_DISABLE(1)

While you have these signals on the VCU118 example:
qsfp2_modsell
qsfp2_resetl
qsfp2_modprsl
qsfp2_intl
qsfp2_lpmode

Question is do I need the extra VCU118 and ZCU111 signals? Also, it looks like you are using 2 10 GbE on the VCU118 is that correct? Sorry I am trying to port this into those two boards and I don't have a VCU118 in hand. Also, can I use a 1000Base-T with this? Thanks.

[alexforencich]

The only signals you need for data are the TX/RX_P/N connections. The rest of the signals are module control and status pins. You can ignore the status pins, but you may need to tie some of the control pins to specific levels so the transceiver will work correctly. For example, you have to tie tx_disable correctly so that the laser turns on in an optical module. You may also have to tie the rs pins. If you're using a direct attach copper cable, the levels on those pins might not make any difference.

You only need to drive what signals are present on your board. The VCU118 has dual QSFP28 connectors, so it has control signals for those. You can think of the VCU118 has having 8x 10G ethernet interfaces. Main thing to do is regenerate the GTH/GTY core with the appropriate configuration for your device, and then connect that to the PHY cores.

I think you can also use the gigabit port, you'll just have to figure out how to access that from the PL side and then connect it to the appropriate gigabit MAC core, and possibly a gigabit PCS/PMA core if it's an SGMII PHY chip.

Also, my standing offer for anyone who wants a design ported to a new board: donate a copy of the hardware, and I'll port it and maintain it.

[sirjansel0t]

Thank you for the swift response. I started porting the project onto the ZCU102. I will let you know if i was successful.

Also, the qsfp1_gt_txdata_# have bit widths of 128 but only 64 of them are actually useful? I looked at the GTH/GTY core and it seems like they take an input of 128 per channel but your core is specified to have a user data width of 64. is it just being 0 padded and only 64 is actually useful?

I'm referring to these signals:

https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/rtl/fpga.v#L470

wire [127:0] qsfp1_gt_txdata_1;
wire [127:0] qsfp1_gt_txdata_2;
wire [127:0] qsfp1_gt_txdata_3;
wire [127:0] qsfp1_gt_txdata_4;

wire [127:0] qsfp2_gt_txdata_1;
wire [127:0] qsfp2_gt_txdata_2;
wire [127:0] qsfp2_gt_txdata_3;
wire [127:0] qsfp2_gt_txdata_4;

I'm wondering because I am modifying the GTH/GTY core and it has a different userdata_tx_data_in size where it's 64 bits wide per GTH channel. Yours generate 128 per GTY channel.

Also, does comment: // Divide by 8 to get output frequency of 125 MHz
https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/rtl/fpga.v#L172

What does that exactly mean? You take in a 125 MHz clk and divide by 8 to get 125?

Sorry for the noobie questions. I'm new to all this stuff.

[alexforencich]

So the native GTY primitive has a bit width of 128, but not all of those bits are used. So there is an option to generate the core with or without the width change. I generated it without that, so I have to concatenate 128 bit signals and then use only the lower 64 bits. If you enable that option, then everything will match.

The divide by 8 is the MMCM output divider setting, there is also a milultiplier, which is also set to 8. So it multiplies and then divides the clock by 8. You could remove that if you want, but it's useful in case you need to generate some other clocks.

[sirjansel0t]

Do you have SFP and QSFP transceiver recommendations that you use to test this with? I just realized I only have 1000BASE-T so I have to go out and buy something.

I'm still working on this but I am getting close. For now I'm just going to implement 10G on the ZCU-102, not the 1G but I should prepare to somehow test it. Also, would I be able to test the 10G mac with just a 1000BASE-T to see if i can get 1G with it? Let me know. Thank you!

[alexforencich]

I usually use direct attach copper cables unless I have a need for a transceiver. I can't recommend any specific transceivers, there are a wide variety depending on your requirements.

You won't be able to test the 10G MAC by connecting it to a 1G interface. However, some of the example designs can switch the 1G port into the 10G path so you can test the 10G design with a loopback module and 1G NIC.

[sirjansel0t]

Thanks for the direct copper cable advice. I didn't think of that. Looks cheaper than a transceiver.

I was looking at the timing constraints on your code between lines 4 to 11 and I see those in the VCU118 and VCU 108 manual but not in the ZCU102 manual. Are those necessary? I'm wondering bc they make it clear that if you don't put those on the VCU118/108 you can potentially damage the board. I'm not sure if I need something similar on the ZCU102

[sirjansel0t]

These are the timing constraints I am referring to.

https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/fpga.xdc#L4

FYI, which example are you referring to that switches the 1G interface?

Thank you.

[alexforencich]

Any board that has both a 1G and a 10G interface (VCU108 and VCU118 at least) should be set up that way - gigabit MAC, 10G MAC, 10G UDP stack, AXI stream switch.

The first two of those general parameters are required at least for non Zynq parts, not sure if you need them for Zynq. I think the tools will complain if you leave them out. Compress is generally a good idea, it makes the output bit file smaller and speeds up the configuration sequence. But it's not required. The other parameters are for loading the design from the flash on the dev board, these will need to match the board configuration. They are not required when loading via JTAG. They may not be applicable for Zynq designs, again not sure about that.

[sirjansel0t]

This is my only final critical warning. It won't route the first 2 SFP ports and I don't know what is going on.

[Vivado 12-1411] Cannot set LOC property of ports, Terminal sfp0_rx_p cannot be placed on E4 (GTHE4_CHANNEL_X1Y12) because the pad is already occupied by terminal sfp1_rx_p possibly due to user constraint ["/home/serf203/verilog-ethernet/example/ZCU102/fpga_10g/fpga.xdc":60]

[Vivado 12-1411] Cannot set LOC property of ports, Instance sfp_gty_inst/inst/gen_gtwizard_gthe4_top.gtwizard_ultrascale_0_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[27].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[0].GTHE4_CHANNEL_PRIM_INST can not be placed in GTHE4_CHANNEL of site GTHE4_CHANNEL_X1Y13 because the bel is occupied by sfp_gty_inst/inst/gen_gtwizard_gthe4_top.gtwizard_ultrascale_0_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[27].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[1].GTHE4_CHANNEL_PRIM_INST(port:). This could be caused by bel constraint conflict ["/home/serf203/verilog-ethernet/example/ZCU102/fpga_10g/fpga.xdc":61]

There seems to be a conflict between the xdc files that I made and the one generated by the transceiver core.

The only constraints i have on those ports are:

set_property PACKAGE_PIN D2 [get_ports "sfp0_rx_p"] ;# Bank 230 - MGTHRXP0_230
set_property PACKAGE_PIN C4 [get_ports "sfp1_rx_p"] ;# Bank 230 - MGTHRXP1_230
set_property PACKAGE_PIN B2 [get_ports "sfp2_rx_p"] ;# Bank 230 - MGTHRXP2_230
set_property PACKAGE_PIN A4 [get_ports "sfp3_rx_p"] ;# Bank 230 - MGTHRXP3_230

set_property PACKAGE_PIN E4 [get_ports "sfp0_tx_p"] ;# Bank 230 - MGTHTXP0_230
set_property PACKAGE_PIN D6 [get_ports "sfp1_tx_p"] ;# Bank 230 - MGTHTXP1_230
set_property PACKAGE_PIN B6 [get_ports "sfp2_tx_p"] ;# Bank 230 - MGTHTXP2_230
set_property PACKAGE_PIN A8 [get_ports "sfp3_tx_p"] ;# Bank 230 - MGTHTXP3_230

any idea what should I do?

[sirjansel0t]

i figured it out. I had some mislabels but now it's fixed. Unfortunately i cannot test anything since I don't have sfp connectors =(

[sirjansel0t]

Can I use an sfp connector to test a 10G MAC with a 1G interface? I have 1000BASE-T. I would just like to make sure. Sorry for all the dumb questions.

[alexforencich]

No, you cannot use a 1G SFP module to test a 10G connection, the line protocols are totally different. You could set up the interface in gigabit mode, though - use a gigabit MAC and gigabit PCS/PMA from Xilinx - but I'm not sure if this would show you anything useful.

[sirjansel0t]

Yes I was wondering if I can setup one of the SFP connectors in a 1 gigabit configuration. I didn't see that in your examples. I know you set up all boards using a dedicated 1 gigabit ethernet port. Right now I have all SFP ports set up for 10G but I can change them to 1 gigabit. I don't need all 4 SFP ports in a 10G config. Do I just need to use the PCS/PMA? Do i need to use the transceiver wizard at all? Thanks a lot.