address_lists_parser.rb requires a lot of memory (~30 mb)

[schneems]

Loading the address_lists_parser.rb is the cause for about 1/3 of the startup memory for CodeTriage.

TOP: 105.7266 MiB
  mail/parsers: 31.9609 MiB
    mail/parsers/address_lists_parser: 29.7266 MiB

Using the derailed_benchmarks gem on codetriage/codetriage.

It looks like it is generated by a parser generator Is there some way we can reduce that memory overhead?

[ahorek]

There's an option to recompile ragel sources for lower memory consumption, but it does hurt performance, see #1215

@jeremy any plans to support ragel 7?

[schneems]

The Ragel Bitmap PR looks pretty promising. Doesn’t look like it affects performance.

I tested with derailed and instead of 30 mb I’m seeing about 5mb. Which is a pretty significant improvement.

Edit, it totally affects performance

[jeremy]

Remember #812 → #815 → #912? 😅

Lazy-loading the parsers is desirable, considering that most usage is building and delivering messages, not parsing them. But the lazy-loading has to work consistently.

[ahorek]

Well, it depends. For our helpdesk we use the mail gem for parsing incoming messages a lot.
Ragel never used to generate a performant code in ruby, but the question is if 30MB per instance really matters these days...

[schneems]

😅 Sorry for the duplicate issue spam. I was working with someone on derailed benchmarks and mis-remembered that somehow I had gotten away without the memory increase for https://www.codetriage.com.

While it doesn't look like there is an obvious replacement (that satisfies all memory and speed requirements) it did generate some good discussion and eyeballs in #1343 whether any changes end up getting merged or not, there's some new techniques in there that i've not seen before.

the question is if 30MB per instance really matters these days...

It matters more for memory constrained implementations, FaaS and PaaS. Usually, there's a line somewhere and if you're over even a few mb, you'll still start to swap and incur enormous perf penalties.

I certainly understand that for some absolute performance is important. I also understand that one of the goals of the project is to not have a c-extension dependency.

Thanks for your time and maintenance!

[gatopanx]

any workarounds for Rails , seeing the same:

derailed output:

TOP: 175.1016 MiB
  rails/all: 58.0781 MiB
    action_mailbox/engine: 31.4609 MiB
      action_mailbox: 31.4609 MiB
        action_mailbox/mail_ext: 31.457 MiB
          action_mailbox/mail_ext/address_equality.rb: 29.1094 MiB
            mail/elements/address: 29.1016 MiB
              mail/parsers/address_lists_parser: 29.0703 MiB
          mail: 2.3359 MiB
            mail/field: 0.4805 MiB
            mail/message: 0.3711 MiB
            mail/multibyte: 0.3086 MiB

lockfile fragments:

    rails (6.0.6.1)
      actioncable (= 6.0.6.1)
      actionmailbox (= 6.0.6.1)
      actionmailer (= 6.0.6.1)
      actionpack (= 6.0.6.1)
      actiontext (= 6.0.6.1)
      actionview (= 6.0.6.1)
      activejob (= 6.0.6.1)
      activemodel (= 6.0.6.1)
      activerecord (= 6.0.6.1)
      activestorage (= 6.0.6.1)
      activesupport (= 6.0.6.1)
      bundler (>= 1.3.0)
      railties (= 6.0.6.1)
      sprockets-rails (>= 2.0.0)

    actionmailbox (6.0.6.1)
      actionpack (= 6.0.6.1)
      activejob (= 6.0.6.1)
      activerecord (= 6.0.6.1)
      activestorage (= 6.0.6.1)
      activesupport (= 6.0.6.1)
      mail (>= 2.7.1)

    mail (2.8.1)
      mini_mime (>= 0.1.1)
      net-imap
      net-pop
      net-smtp