Cache test pack compilation (#539)

* Cache test pack compilation

Update changelog

Fix test

Update webpacker

Fix wordings

Change method name for clarity

Update readme

Remove webpack-dev-server bit for now

Update toc

Add tests for new configuration

Move configuration to compiler class and make it configurable programmatically

Fix readme

Only run compiler in test env

Add compiler test

Change method name

Remove Rails.root

Use stub

Rename to compile and update syntax

Update changelog

Revert escaping

* Update changelog

* Setup source path from config file

Author: gauravtiwari

CHANGELOG.md

@@ -4,14 +4,32 @@
 - Update `webpack-dev-server.tt` to respect RAILS_ENV and NODE_ENV values [#502](https://github.com/rails/webpacker/issues/502)
 
 ### Breaking changes
-- Add `compile_missing_packs` option to `config/webpacker.yml` for configuring lazy compilation of packs when not found [#503](https://github.com/rails/webpacker/pull/503). To enable expected behavior for defaults and production, update `config/webpacker.yml`:
+- Add `compile` option to `config/webpacker.yml` for configuring lazy compilation of packs when a file under tracked paths is changed [#503](https://github.com/rails/webpacker/pull/503). To enable expected behavior, update `config/webpacker.yml`:
 
   ```yaml
     default: &default
-      compile_missing_packs: true
+      compile: false
 
-    production:
-      compile_missing_packs: false
+    test:
+      compile: true
+
+    development:
+      compile: true
+  ```
+
+- Make test compilation cacheable and configurable so that the lazy compilation
+only triggers if files are changed under tracked paths.
+Following paths are watched by default -
+
+  ```rb
+    ["app/javascript/**/*", "yarn.lock", "package.json", "config/webpack/**/*"]
+  ```
+
+  To add more paths:
+
+  ```rb
+  # config/initializers/webpacker.rb or config/application.rb
+  Webpacker::Compiler.watched_paths << 'bower_components'
   ```
 
 ## [2.0] - 2017-05-24

Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    webpacker (1.2)
+    webpacker (2.0)
       activesupport (>= 4.2)
       multi_json (~> 1.2)
       railties (>= 4.2)

README.md

@@ -24,6 +24,7 @@ in which case you may not even need the asset pipeline. This is mostly relevant
   - [React](#react)
   - [Angular with TypeScript](#angular-with-typescript)
   - [Vue](#vue)
+    - [Using Rails helpers in .vue files](#using-rails-helpers-in-vue-files)
   - [Elm](#elm)
 - [Binstubs](#binstubs)
   - [Webpack dev server](#webpack-dev-server)
@@ -63,6 +64,8 @@ in which case you may not even need the asset pipeline. This is mostly relevant
 - [Deployment](#deployment)
   - [Heroku](#heroku)
 - [Testing](#testing)
+  - [Lazy compilation](#lazy-compilation)
+    - [Caching](#caching)
 - [Troubleshooting](#troubleshooting)
     - [ENOENT: no such file or directory - node-sass](#enoent-no-such-file-or-directory---node-sass)
     - [Can't find hello_react.js in manifest.json](#cant-find-hello_reactjs-in-manifestjson)
@@ -1060,9 +1063,12 @@ git push heroku master
 
 ## Testing
 
+### Lazy compilation
+
 Webpacker lazily compiles assets in test env so you can write your tests without any extra
 setup and everything will just work out of the box.
 
+
 Here is a sample system test case with hello_react example component:
 
 ```js
@@ -1101,6 +1107,25 @@ class HomesTest < ApplicationSystemTestCase
 end
 ```
 
+#### Caching
+
+By default, the lazy compilation is cached until a file is changed under
+tracked paths. You can configure the paths tracked
+by adding new paths to `watched_paths` array, much like rails `autoload_paths`:
+
+```rb
+# config/initializers/webpacker.rb
+# or config/application.rb
+Webpacker::Compiler.watched_paths << 'bower_components'
+```
+
+Compiler stores a timestamp under `tmp/webpacker/` directory to keep track of
+changes and you can configure that by overriding compiler `cache_dir`:
+
+```rb
+Webpacker::Compiler.cache_dir = "tmp/foo"
+```
+
 ## Troubleshooting
 
 ##### ENOENT: no such file or directory - node-sass

lib/install/config/webpacker.yml

@@ -4,7 +4,7 @@ default: &default
   source_path: app/javascript
   source_entry_path: packs
   public_output_path: packs
-  compile_missing_packs: true
+  compile: false
 
   extensions:
     - .coffee
@@ -34,8 +34,8 @@ test:
   <<: *default
 
   public_output_path: packs-test
+  # Compile webpack assets during lookup when running tests
+  compile: true
 
 production:
   <<: *default
-
-  compile_missing_packs: false

lib/webpacker/compiler.rb

@@ -3,12 +3,47 @@
 module Webpacker::Compiler
   extend self
 
+  # Additional paths that test compiler needs to watch
+  # Webpacker::Compiler.watched_paths << 'bower_components'
+  mattr_accessor(:watched_paths) { [] }
+
+  # Compiler cache directory
+  # Webpacker::Compiler.cache_dir = 'tmp/cache'
+  mattr_accessor(:cache_dir) { "tmp/webpacker" }
+
   def compile
+    return unless compile?
+    cache_source_timestamp
     compile_task.invoke
     compile_task.reenable
   end
 
+  def compile?
+    return true unless File.exist?(cached_timestamp_path)
+    return true unless File.exist?(Webpacker::Configuration.output_path)
+
+    File.read(cached_timestamp_path) != current_source_timestamp
+  end
+
+  def default_watched_paths
+    ["#{Webpacker::Configuration.source}/**/*", "yarn.lock", "package.json", "config/webpack/**/*"].freeze
+  end
+
   private
+    def current_source_timestamp
+      files = Dir[*default_watched_paths, *watched_paths].reject { |f| File.directory?(f) }
+      files.map { |f| File.mtime(f).utc.to_i }.max.to_s
+    end
+
+    def cache_source_timestamp
+      File.write(cached_timestamp_path, current_source_timestamp)
+    end
+
+    def cached_timestamp_path
+      FileUtils.mkdir_p(cache_dir) unless File.directory?(cache_dir)
+      Rails.root.join(cache_dir, ".compiler-timestamp")
+    end
+
     def compile_task
       @compile_task ||= load_rake_task("webpacker:compile")
     end

lib/webpacker/configuration.rb

@@ -36,8 +36,8 @@ def source
       fetch(:source_path)
     end
 
-    def compile_missing_packs?
-      fetch(:compile_missing_packs)
+    def compile?
+      fetch(:compile)
     end
 
     def fetch(key)
@@ -51,7 +51,7 @@ def data
     end
 
     def defaults
-      @defaults ||= HashWithIndifferentAccess.new(YAML.load(default_file_path.read)["default"])
+      @defaults ||= HashWithIndifferentAccess.new(YAML.load(default_file_path.read)[Webpacker.env])
     end
   end
 

lib/webpacker/manifest.rb

@@ -14,10 +14,8 @@ def file_path
     end
 
     def lookup(name)
-      load if Webpacker.env.development?
-
-      if Webpacker::Configuration.compile_missing_packs?
-        find(name) || compile_and_find!(name)
+      if Webpacker::Configuration.compile?
+        compile_and_find!(name)
       else
         find!(name)
       end
@@ -28,10 +26,6 @@ def lookup_path(name)
     end
 
     private
-      def find(name)
-        instance.data[name.to_s] if instance
-      end
-
       def find!(name)
         raise Webpacker::FileLoader::FileLoaderError.new("Webpacker::Manifest.load must be called first") unless instance
         instance.data[name.to_s] || raise(Webpacker::FileLoader::NotFoundError.new("Can't find #{name} in #{file_path}. Is webpack still compiling?"))

test/compiler_test.rb

@@ -0,0 +1,25 @@
+require "webpacker_test"
+
+class CompilerTest < Minitest::Test
+  def test_default_watched_paths
+    assert_equal Webpacker::Compiler.default_watched_paths, ["app/javascript/**/*", "yarn.lock", "package.json", "config/webpack/**/*"]
+  end
+
+  def test_empty_watched_paths
+    assert_equal Webpacker::Compiler.watched_paths, []
+  end
+
+  def test_watched_paths
+    Webpacker::Compiler.stub :watched_paths, ["Gemfile"] do
+      assert_equal Webpacker::Compiler.watched_paths, ["Gemfile"]
+    end
+  end
+
+  def test_cache_dir
+    assert_equal Webpacker::Compiler.cache_dir, "tmp/webpacker"
+  end
+
+  def test_compile?
+    assert Webpacker::Compiler.compile?
+  end
+end

test/configuration_test.rb

@@ -30,7 +30,7 @@ def test_source_path
     assert_equal Webpacker::Configuration.source_path.to_s, source_path
   end
 
-  def test_compile_missing_packs
-    assert Webpacker::Configuration.compile_missing_packs?
+  def test_compile?
+    refute Webpacker::Configuration.compile?
   end
 end

test/manifest_test.rb

@@ -10,10 +10,11 @@ def test_lookup_exception
     manifest_path = File.join(File.dirname(__FILE__), "test_app/public/packs", "manifest.json").to_s
     asset_file = "calendar.js"
 
-    Webpacker::Configuration.stub :compile_missing_packs?, false do
+    Webpacker::Configuration.stub :compile?, false do
       error = assert_raises Webpacker::FileLoader::NotFoundError do
         Webpacker::Manifest.lookup(asset_file)
       end
+
       assert_equal error.message, "Can't find #{asset_file} in #{manifest_path}. Is webpack still compiling?"
     end
   end