add importer for log files #75

rohitner · 2018-05-10T10:32:43Z

This PR integrates the gem request-log-analyzer to convert rails logs to Daru::DataFrame.

zverok

Adding to what said in comments, changes without tests are unverifiable :)

zverok · 2018-05-10T17:07:39Z

daru-io.gemspec


  spec.add_development_dependency 'guard-rspec' if RUBY_VERSION >= '2.2.5'
+
+  spec.add_runtime_dependency 'request-log-analyzer', '~> 1.13.4'


Note that no other file format gem is required unconditionally: daru-io relies on user installing the proper gems. Otherwise, anybody who wants daru-io just for CSV import, will install TON of other gems for pdf, excel and whatnot. For example, look here: https://github.com/SciRuby/daru-io/blob/master/lib/daru/io/importers/excelx.rb#L12 — we have optional_gem construct for requiring something ONLY when it is used.

zverok · 2018-05-10T17:08:14Z

lib/daru/io/importers/rails.rb

+  module IO
+    module Importers
+      class Rails < Base
+        Daru::DataFrame.register_io_module :read_rails, self


read_rails_log is better name, read_rails is too vague.

zverok · 2018-05-10T17:09:58Z

lib/daru/io/importers/rails.rb

+        def initialize
+          require 'request_log_analyzer'
+          RequestLogAnalyzer::Source::LogParser.class_eval do
+            def initialize(format, options = {})


Better extract it all into importers/rails_logs/request_log_analyzer_patch.rb, create there module RailsLogAnalyzerPatch, and here just include that into RailsLogAnalyzer.

The current initialize,parse and read methods are going to be the same for all the importers that will be added. Shall I instead create module RequestLogAnalyzerPatch in importers/request_log_analyzer.rb? Then I will pass :rails3, :apache and :amazon_s3 to the module in rails.rb, apache.rb and amazon_s3.rb.

zverok · 2018-05-10T17:13:22Z

lib/daru/io/importers/rails.rb

+        end
+      end
+    end
+  end


In fact, here you are copy-pasting and slightly changing code from other gem. It is totally unreliable: if the gem changes a slightest bit, you'll never follow this with your code. Why do you need this rewriting?

I will try to come up with a single method instead of changing the code.

rohitner · 2018-05-12T06:18:43Z

Please review the changes so that I can start the documentation and fix rubocop tests.

zverok

Also, please don't ignore Rubocop's suggestions.

zverok · 2018-05-12T10:39:10Z

lib/daru/io/importers/base.rb

          new.read(path)
        end
+
+        def self.parse_log(file,format)


Why do you think it is appropriate to add method parse_log, related to a very specific task, to Base importer?.. All code related to some importer should be totally local to this importer: either in importers/rails_log.rb, or in importers/rails_log/request_log_analyzer_patch.rb (the latter is preferable, as well as extracting patch to a module, instead of doing class_eval)

parse_log is a helper method as apache and amazon_s3 importers will also use it. Making this part local will later require adding the same code for the remaining two importers. I feel the method is not that specific to be ineligible for the base class.

I feel the method is not that specific to be ineligible for the base class.

In fact, it is. The base class provides the common base all importers need, not "some 3 of 20 importers may need that method, so we'll just drop it here". It requires specific external gem. If several different importers will need it, the module with this "patch" could be stored in, like importers/shared/request_log_analyzer_patch.rb, and required/included where it is necessary.

zverok · 2018-05-12T10:39:31Z

lib/daru/io/importers/rails_log.rb

+        def call()
+          ind = [:method, :path, :ip, :timestamp, :line_type, :lineno, :source,
+                 :controller, :action, :format, :params, :rendered_file,
+                 :partial_duration, :status, :duration, :view, :db]


That definitely should be a constant.

zverok · 2018-05-12T10:45:31Z

lib/daru/io/importers/rails_log.rb

+            row = []
+            ind.each do |attr|
+              row << hash[attr]
+            end


That should be map, at least.
But if I am reading it correctly, Hash#values_at should be totally enough.

zverok · 2018-05-15T11:58:45Z

lib/daru/io/importers/rails_log.rb

+        # initializes the instance variables @path and @file_data to nil
+        def initialize
+          @path = nil
+          @file_data = nil


Why do you need this vars here?

zverok · 2018-05-15T11:58:50Z

lib/daru/io/importers/rails_log.rb

+        include RequestLogAnalyzerPatch
+        Daru::DataFrame.register_io_module :read_rails_log, self
+
+        # initializes the instance variables @path and @file_data to nil


Comments should not just retell the code, code should speak for itself.
For such simple methods, comments aren't necessary at all.

zverok · 2018-05-15T11:59:16Z

lib/daru/io/importers/rails_log.rb

+        # @example Reading from plaintext file
+        #   instance = Daru::IO::Importers::RailsLog.read("rails_test.log")
+        def read(path)
+          @path = path


zverok · 2018-05-15T11:59:36Z

lib/daru/io/importers/rails_log.rb

+        #   instance = Daru::IO::Importers::RailsLog.read("rails_test.log")
+        def read(path)
+          @path = path
+          @file_data = RequestLogAnalyzerPatch.parse_log(@path,:rails3)


Why :rails3, couldn't it be another log format?..

Having the log format as another argument (with maybe :rails3 as default) would be a more generic way to do this.

zverok · 2018-05-15T12:13:56Z

lib/daru/io/importers/shared/request_log_analyzer_patch.rb

@@ -0,0 +1,73 @@
+module RequestLogAnalyzerPatch
+  require 'request_log_analyzer'


This file now is required always when daru-io is required. It should be made optional_gem in importer's initialize, like other importers do.

zverok · 2018-05-15T12:28:40Z

lib/daru/io/importers/shared/request_log_analyzer_patch.rb

@@ -0,0 +1,73 @@
+module RequestLogAnalyzerPatch


When I proposed this module name, I've meant the following: You define ONLY parse_hash in that module, and then, in importer (not here in separate method) you do the following:

parser = RequestLogAnalyzer::Source::LogParser.new(RequestLogAnalyzer::FileFormat.load(format)) parser.extend(RequestLogAnalyzerPatch) # only this instance has method replaced parser.parse_hash(path)

This way it is easy to see what you "inserting" into the third-party gem, and keep it local.

zverok · 2018-05-15T12:32:46Z

lib/daru/io/importers/shared/request_log_analyzer_patch.rb

+      # @example Reading from rails log file
+      #   format   = RequestLogAnalyzer::FileFormat.load(:rails3)
+      #   instance = RequestLogAnalyzer::Source::LogParser.new(format)
+      #   list     = instance.parse_hash('test_rails.log')


This is internal to implementation, so does NOT need any comment about "how it can be used".
Instead, the comment that is absolutely necessary here is: explain that this method only copy-pasted (from where exactly; from which version exactly) and this and that lines changed (with "how it was, why it was changed" in comments). This way, it would be possible to follow request_log_analyzer's development, if it would change the method significantly.

zverok · 2018-05-15T12:35:02Z

lib/daru/io/importers/sql.rb


        private

-        def attempt_sqlite3_connection(db)


db is totally good name. Please add this to Naming/UncommunicativeMethodParamName: cop's AllowedNames: settings.

In general, if Rubocop is starting to fail on code unrelated to your PR, it is better to notify maintainer (@athityakumar) than judge yourself.

zverok · 2018-05-15T12:37:14Z

spec/daru/io/importers/rails_log_spec.rb

+      order: %i[method path ip timestamp line_type lineno source
+                controller action format params rendered_file
+                partial_duration status duration view db],
+      :'timestamp.to_a' => [20180312174118], # rubocop:disable Style/NumericLiterals


Don't disable here, just do what Rubocop suggests.

athityakumar · 2018-05-15T15:34:01Z

lib/daru/io/importers/rails_log.rb

+          @file_data.each do |hash|
+            data.add_row(INDEX.map { |attr| hash[attr] })
+          end
+          data


I think df is more readable than data, and has either been consistently named so in other Importers or rather just returned a Daru::DataFrame directly.

athityakumar · 2018-05-15T15:36:08Z

lib/daru/io/importers/rails_log.rb

+        #   #   1        GET          /  127.0.0.1 2018022716  completed         12  /home/roh Rails...
+        #   # ...        ...        ...        ...        ...        ...        ...        ...      ...
+        def call
+          data = Daru::DataFrame.new({},index: INDEX).transpose


Um, INDEX is actually supposed to be the order of the DataFrame and has been weirdly named so just due to transpose operation? 🤔

athityakumar · 2018-05-15T15:37:16Z

lib/daru/io/importers/rails_log.rb

+          data = Daru::DataFrame.new({},index: INDEX).transpose
+          @file_data.each do |hash|
+            data.add_row(INDEX.map { |attr| hash[attr] })
+          end


Can't the whole transpose + add_row logic be replaced by Daru::DataFrame.rows(array_of_arrays, order: ORDER)?

It can be, but then the array_of_arrays will need further processing as all vectors don't have the same length due to missing attributes in some of them.

rohitner · 2018-05-16T13:26:18Z

@zverok @athityakumar please review and suggest changes for rubocop failure.

rohitner · 2018-05-17T05:18:33Z

How about adding importers of all three formats in a single file? It would eliminate the additional separate module for the patch. The log.rb file will look like this

module Importers
  class Log < Base
    Daru::Dataframe.register_io_module :read_log, self
    ORDER_RAILS  = %i[...].freeze
    ORDER_APACHE = %i[...].freeze
    ORDER_S3     = %i[...].freeze
    ...
    def read(path, format: :rails3) # default format is :rails3
      case format
      when :rails3    then order = ORDER_RAILS
      when :apache    then order = ORDER_APACHE
      when :amazon_s3 then order = ORDER_S3
      else raise ArgumentError, 'Invalid format provided.'
      ...
      @file_data = parser.parse_hash(path, order)
      self
    end
    ...
    private
    module RequestLogAnalyzerPatch
    ...
      def parse_hash(file, order)
      ...
          parsed_list[i] = order.map ...
      ...
      end
    end
  end
end

In the above method, I will have need to modify the line containing register_io_module to take one more argument specifying the format. WDYT?

zverok · 2018-05-17T10:22:16Z

How about adding importers of all three formats in a single file? It would eliminate the additional separate module for the patch. The log.rb file will look like this

That may work. Just the code can be simpler:

ORDERS = {
  rails: %i[...].freeze,
  apache: %i[...].freeze,
  s3: %i[...].freeze
}
# and then...
order = ORDERS.fetch(format)

zverok · 2018-05-16T18:00:57Z

lib/daru/io/importers/rails_log.rb

+        Daru::DataFrame.register_io_module :read_rails_log, self
+
+        # Checks for required gem dependencies of RailsLog importer
+        # and requires the patch for request-log-analyzer gem


Just remove the comment. Please don't repeat what method code says clearly (even if other importers do that :)).

zverok · 2018-05-16T18:02:18Z

lib/daru/io/importers/rails_log.rb

+
+        # Reads data from a rails log file
+        #
+        # @!method self.read(path)


Please document format parameter too.

zverok · 2018-05-16T18:05:13Z

lib/daru/io/importers/rails_log.rb

+        #   instance = Daru::IO::Importers::RailsLog.read("rails_test.log")
+        def read(path, format: :rails3)
+          parser = RequestLogAnalyzer::Source::LogParser.new(RequestLogAnalyzer::FileFormat.load(format))
+          parser.extend(RequestLogAnalyzerPatch)


This means, that RequestLogAnalyzerPatch should look like this:

module RequestLogAnalyzerPatch # look, no other nested modules! def parse_hash(file) end end

This way, when you do this extend, parse_hash becomes redefined.

zverok · 2018-05-17T10:55:15Z

lib/daru/io/importers/shared/request_log_analyzer_patch.rb

+      # parse_io and parse_line of the LogParser class. Assigns all the necessary instance
+      # variables defined in the above specified methods. Creates a request for each line of
+      # the file stream and stores the hash of parsed information in raw_list. Each element of
+      # parsed_list is an array of one parsed entry in log file.


Ugh, now, I see a problem here (I thought that LogParser already has method parse_hash, you just fixing something in it):

combine the methods parse_file, parse_io and parse_line of the LogParser class

Why???? What's the point of combining three methods in one 40-lines mess?.. Why can't you use original methods as they were, just calling them?

parse_file returns nil and so does parse_io. That's why I have tried fixing these methods in the earlier commits instead of making a new single method. But then it was a lot of rewriting. It was a hard time finding points to tap out the parsed hash from the gem as it is only meant for CLI.

Ugh. You should have spend a bit more time on understanding LogParser logic. From reading the code, I believe what you need is (without ANY patching) just call LogParser#each_request:

Reads the input, which can either be a file, sequence of files or STDIN to parse lines specified in the FileFormat. This lines will be combined into Request instances, that will be yielded.

...and, as it is aliased as each, and Enumerable is included, ALL you really need is:

RequestLogAnalyzer::Source::LogParser .new(RequestLogAnalyzer::FileFormat.load(format)) .map { converting Request into line for dataframe }

...and that would be completely it.

rohitner · 2018-05-27T13:34:25Z

@zverok please review

zverok

👍

zverok · 2018-05-28T17:16:25Z

lib/daru/io/importers/log.rb

+        #   # ...        ...        ...        ...        ...        ...        ...        ...      ...
+        def call
+          Daru::DataFrame.rows(@file_data, order: ORDERS.fetch(@format)
+            .map { |attr| attr == :path ? :resource_path : attr })


I'd say RENAME_FIELDS.fetch(attr, attr) with

RENAME_FIELDS = { path: :resource_path }

It would be more regular (even if for now used only for one field).

zverok · 2018-05-28T18:39:11Z

(Just turn off the Lint/SplatKeywordArguments, it is honestly meaningless and already removed in the master).

zverok · 2018-05-30T09:04:24Z

Merged! Congrats 🎉

add importer for rails log

c731a3a

zverok requested changes May 10, 2018

View reviewed changes

rohitner added 3 commits May 11, 2018 17:06

relocate parsing method to base class

cb95d52

changed naming

14f459c

correct parse_hash method

16b0221

rohitner force-pushed the rails_logs branch from 0b16aff to ef50d50 Compare May 11, 2018 21:15

add test

ef50d50

zverok requested changes May 12, 2018

View reviewed changes

rohitner added 3 commits May 12, 2018 22:30

move patch to separate module

91ce356

fix rubocop

049e262

add documentation

162946c

zverok requested changes May 15, 2018

View reviewed changes

athityakumar reviewed May 15, 2018

View reviewed changes

refine code

c6c6047

zverok requested changes May 17, 2018

View reviewed changes

rohitner force-pushed the rails_logs branch from 1b0e783 to ecc7f41 Compare May 17, 2018 22:56

add importers for apache and amazon_s3 format

ecc7f41

rohitner changed the title ~~add importer for rails log~~ add importer for log files May 19, 2018

rohitner force-pushed the rails_logs branch from 7c1b62d to b1d77bc Compare May 24, 2018 21:19

change order to fix iruby error

b1d77bc

zverok approved these changes May 28, 2018

View reviewed changes

disable a lint cop and change field renaming

b557e4b

zverok merged commit dce4be1 into SciRuby:master May 30, 2018

rohitner mentioned this pull request Jun 6, 2018

Fix rubocop offences #74

Closed


		spec.add_development_dependency 'guard-rspec' if RUBY_VERSION >= '2.2.5'

		spec.add_runtime_dependency 'request-log-analyzer', '~> 1.13.4'

		@@ -0,0 +1,73 @@
		module RequestLogAnalyzerPatch
		require 'request_log_analyzer'

add importer for log files #75

add importer for log files #75

Uh oh!

Conversation

rohitner commented May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zverok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohitner May 10, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohitner commented May 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zverok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohitner May 12, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohitner May 16, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rohitner commented May 16, 2018

Uh oh!

rohitner commented May 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zverok commented May 17, 2018

Uh oh!

Choose a reason for hiding this comment

rohitner commented May 10, 2018 •

edited

Loading

rohitner May 10, 2018 •

edited

Loading

rohitner commented May 12, 2018 •

edited

Loading

rohitner May 12, 2018 •

edited

Loading

rohitner May 16, 2018 •

edited

Loading

rohitner commented May 17, 2018 •

edited

Loading

rohitner May 17, 2018 •

edited

Loading

zverok commented May 28, 2018 •

edited

Loading