[VPP-1578] VPP crashes when attempting to add 50k routes

[vvalderrv]

Description

I'm not able to add more than approximately 38k routes into VPP before it crashes with the following traceback. I'm using VPP 19.01 with dpdk.

1. adding 50k routes

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: received signal SIGABRT, PC 0x7fa5ecb7253f

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: #0 0x00007fa5ed6d6ae8 0x7fa5ed6d6ae8

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Fixed: ip6_fib_dump #1 0x00007fa5ed41f030 0x7fa5ed41f030

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: add .gitlab-ci.yml in order to use ci/cd in gitlab #2 0x00007fa5ecb7253f gsignal + 0x10f

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: fix issue with missing sample_main in sample plugin #3 0x00007fa5ecb5c895 abort + 0x127

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: fix issue with missing sample_main in sample plugin #4 0x0000000000407223 0x407223

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: fix issue with missing sample_main in sample plugin #5 0x00007fa5ed1db775 vec_resize_allocate_memory + 0x395

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: fix packets redirect ineffective on af-packet interface #6 0x00007fa5ed692b65 vlib_validate_combined_counter + 0xc5

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Decrement n_retry in tx_burst_vector_internal #7 0x00007fa5edfb163f load_balance_create + 0x15f

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Stable/1810 #8 0x00007fa5edf8d5dd fib_entry_src_mk_lb + 0x10d

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Update stats.md #9 0x00007fa5edf8d956 fib_entry_src_action_install + 0x136

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Merge pull request #1 from FDio/master #10 0x00007fa5edf8ac21 fib_entry_create + 0x71

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: pci_dev in dpdk/device/cli.c for RTE_SCHED_COLLECT_STATS #11 0x00007fa5edf81965 fib_table_entry_path_add2 + 0x135

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: pci_dev in dpdk/device/cli.c for RTE_SCHED_COLLECT_STATS #12 0x00007fa5edc55ae0 0x7fa5edc55ae0

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: vpp crashes when attempting to run in kubernetes Pod  #13 0x00007fa5ed690d96 0x7fa5ed690d96

   Mar 04 17:26:30 nfvpe-02.oot.lab.eng.bos.redhat.com vnet[25931]: Merge pull request #1 from FDio/master #14 0x00007fa5ed6913d7 0x7fa5ed6913d7

Assignee

Neale Ranns

Reporter

Tim Rozet

Comments

• nranns (Tue, 28 May 2019 16:21:08 +0000): Hi Billy,

in:

 https://gerrit.fd.io/r/#/c/19855/

if added some detail on how the respective heaps are used and some examples of how they scale as routes are added.

The separate heaps can be scaled individually.  More than 4G is probably not required.

/neale

• billym (Mon, 11 Mar 2019 13:34:02 +0000): Neale,

Do you have recommended values for each field? Probably max'd out without having to recompile for CLIB_VEC64. For example, does 'ip {heap-size}' need to be less than the over all 'heapsize'? Also, do you think we need to be greater than 4G and recompile for CLIB_VEC64 with 50k routes?

Thanks,

Billy

• nranns (Mon, 11 Mar 2019 09:11:54 +0000): and this one:

statseg { size 1g }

• trozet (Tue, 5 Mar 2019 17:56:54 +0000): I set the heapsize to 2GB and set the ip heap to 512MB:

heapsize 2G

ip {

heap-size 512M

}

and I still hit the crash at the 38210 mark of route addition.

• nranns (Tue, 5 Mar 2019 16:15:42 +0000): try this too:

https://wiki.fd.io/view/VPP/Command-line_Arguments#heapsize_.7B_..._.7D

• trozet (Tue, 5 Mar 2019 16:13:58 +0000): Well before I had the default which is 32MB, then used 256MB and still have the same limit on routes. I would think I would have been able to achieve more routes by that initial bump. Am I mistaken? What value do you think I should try?
• nranns (Tue, 5 Mar 2019 15:53:25 +0000): and if you assign more memory?
• trozet (Tue, 5 Mar 2019 15:51:19 +0000): Thanks Neale Ranns. I added this config:

 

ip {

heap-size 256M

}

but I still see the same crash after adding 38210 routes.

• nranns (Tue, 5 Mar 2019 14:59:59 +0000): ip {heap-size }

 

to increase the amount of memory available for FIB

Original issue: https://jira.fd.io/browse/VPP-1578

[vvalderrv]

Hi Billy,

in:

 https://gerrit.fd.io/r/#/c/19855/

if added some detail on how the respective heaps are used and some examples of how they scale as routes are added.

The separate heaps can be scaled individually.  More than 4G is probably not required.

/neale

[vvalderrv]

Neale,

Do you have recommended values for each field? Probably max'd out without having to recompile for CLIB_VEC64. For example, does 'ip {heap-size}' need to be less than the over all 'heapsize'? Also, do you think we need to be greater than 4G and recompile for CLIB_VEC64 with 50k routes?

Thanks,

Billy

[vvalderrv]

and this one:

statseg { size 1g }

[vvalderrv]

I set the heapsize to 2GB and set the ip heap to 512MB:

heapsize 2G

ip {
heap-size 512M
}

and I still hit the crash at the 38210 mark of route addition.

 

[vvalderrv]

try this too:

https://wiki.fd.io/view/VPP/Command-line_Arguments#heapsize_.7B_..._.7D

 

[vvalderrv]

Well before I had the default which is 32MB, then used 256MB and still have the same limit on routes. I would think I would have been able to achieve more routes by that initial bump. Am I mistaken? What value do you think I should try?

[vvalderrv]

and if you assign more memory?

[vvalderrv]

Thanks Neale Ranns. I added this config:

 

ip {
heap-size 256M
}

but I still see the same crash after adding 38210 routes.

[vvalderrv]

ip {heap-size <value>}

 

to increase the amount of memory available for FIB