Update: Since this article was published, the Rails team announced the inclusion of both webpack and Yarn into Rails 5.1. I have a new article covering these latest changes: Embracing Change: Rails 5.1 Adopts Yarn, Webpack, and the JS Ecosystem.
Ruby on Rails has a pretty sweet ecosystem. We have libraries for everything
from printing a table flipping emojicon on an exception to natural language processing. Just add the desired libraries to your Gemfile, type bundle, and you’re on your way (ahem. Usually).
When it comes to dealing with JavaScript libraries in a Rails project, however,
things aren’t as straightforward. Should you download the Javascript code and
manually store it in the /vendor directory or look for a wrapper gem that’ll
just make the bad Javascript man go away? If you install the libraries manually,
it’s up to you to ensure all the dependent libraries are also installed. If you
rely on a gem, you must rely on the gem maintainer to keep things current. It’s
a real problem, and we’re not even considering the minor detail of version
compatibility amongst the various javascript libraries in both scenarios.
The solution is to let each ecosystem, Ruby and Javascript, be awesome at what they do best. The idea of the asset pipeline was a great idea when it was first conceived. It provided a solution to breaking cached assets with fingerprints, it concatenated and minified both CSS and Javascript, and it even accounted for fonts, images, and whatever else we wanted to throw in there. What it didn’t account for was how dominant Node.js and all the frontend frameworks would become.
Today, when you look at any Javascript library, the installation
instructions can be summed up with this: npm install libraryname. There are no
instructions for how to install the library manually, or if there are, it’s with
an incredulous tone. It’s just not done that way anymore.
Rails no longer needs to control the entire ecosystem, and arguably it shouldn’t because tools have been developed which do a better job. Letting go of this one area doesn’t reduce the value or importance of Rails, and it doesn’t mean Node.js won. It just means we’re choosing the right tools for the job.
Assuming you agree with me, let’s start the process of replacing the Asset Pipeline.
We’ll first need a Rails project to use as a our lab for this experiment. You can switch to a new branch of an existing project, but it may result in more headaches until you have a better grasp of the technologies. It’s your call.
rails new antipipeline
With our Rails application created, we can start replacing the Pipeline.
To chisel out the asset pipeline from Rails, we’re going to need some tools. Those tools are Node.js, the Javascript Runtime, and Yarn, a dependecy management tool (think Bundler). Node allows us to run Javascript executables from the commandline, and Yarn enables us to install all the other libraries we need.
Both tools are super easy to install if you’re running MacOS.
brew install node
brew install yarn
You’ll want to refer to the documentation of both tools if you’re running another OS.
You’re probably already familiar with Node, but Yarn is a newcomer to the Javascript scene. It is a replacement for npm, and was developed as “a collaboration between Facebook, Exponent, Google, and Tilde”. It still pulls libraries from the “npm registry, but can install packages more quickly and manage dependencies consistently across machines or in secure offline environments.”
Sprockets does a lot of work, and it’s work our new toolchain will now need to handle, such as concatenating the Javascript and CSS files, minifying them, transpiling CoffeScript to Javascript, etc. To do that we need a new tool. We’re going to use Webpack.
Webpack is a lot like Grunt and Gulp, but is quickly gaining more and more mindshare. If you’re not familiar with any of these tools, the idea is really simple: they take the asset files (CSS, Javascript, SaSS, images, etc.), and process them for use.
Webpack is the tool which will handle transpiling CoffeeScript to Javascript, including all the dependencies, minifying the output, exporting the files, and everything else we once relied on Sprockets to perform.
We can install Webpack with two commands. The first installs webpack into the
global space, allowing us to call the webpack command from the command line.
yarn global add webpack
The second command adds Webpack to our development environment so we can use it
from within our webpack.config.js file.
yarn add --dev webpack
Webpack needs to know which directories to read from, what transformations
it needs to apply to what files, and where to put everything once it’s completed
its run. To provide it direction, we need to create a webpack.config.js file
which will live in the Rails app’s root directory:
'use strict';
const webpack = require("webpack");
module.exports = {
context: __dirname + "/app/assets/javascripts",
entry: {
application: "./application.js",
},
output: {
path: __dirname + "/public",
filename: "javascripts/[name].js",
},
};
The configuration file can be boiled down to two actions: take an input (the
“entry” block) and producing an output (the “output” block). More specifically,
it’s instructing Webpack to read the application.js file from
/app/assets/javascripts, perform any actions required by that file such as
including other libraries, giving it the name “application”, and then output
the resulting file(s) to /public/javascripts/application.js.
Try it for yourself: run webpack from the command line.
With Webpack providing our new foundation, we can start removing the old asset pipeline.
To begin cutting our ties to the past, we need to first remove some Ruby dependencies.
Comment out or delete the following gems from your Gemfile and run bundle
from the command line:
These gems are no longer needed, being replaced or deprecated by Webpack and the Javascript libraries we’ll install later..
Our next task will be to tell Rails to ignore assets in both the
development.rb and production.rb files of config/environments (test.rb
will be left untouched.)
In your config/environments/development.rb file:
config.assets.debug = false
config.assets.compile = false
config.assets.quiet = true
In the config/environments/production.rb file:
config.public_file_server.enabled = true
# config.assets.js_compressor = :uglifier
# config.assets.css_compressor = :sass
We enable config.public_file_server in order to serve static files out of the
/public directory.
By removing jquery-rails, coffee-rails, and sass-rails we’ve put ourselves
in a bit of a pickle: we’re stuck with raw CSS and Javascript. Let’s fix that.
First, the bad news: This tutorial doesn’t cover adding CoffeeScript. Now the good news: ES6 more than makes up for it.
CoffeeScript made Javascript nicer by cleaning up the syntax, simplifying class creation, perform string interpolation, and call functions. ES6 is doing the same thing, albeit with a more Javascript feel, and it will be the language browsers eventually switch too.
Babel is the library we’ll use to transpile ES6 to pure Javascript. We can install it with the following command:
yarn add --dev babel-core babel-preset-es2015 babel-polyfill
With babel installed, we then need to add a “loader” to enable webpack to use it:
yarn add --dev babel-loader # loader for webpack
The last thing we must do is configure Webpack to use it. We’ll do that by adding the loader to the “module” section:
'use strict';
const webpack = require("webpack");
module.exports = {
...
module: {
loaders: [
{
test: /\.js$/,
exclude: /node_modules/,
loader: 'babel',
query: {
presets: ['es2015']
}
},
]
},
};
If you add the following line to your app/assets/javascripts/application.js
file, and run webpack, you’ll see it output into pure Javascript in the
public/javascripts directory:
let odds = [0,2,4,6,8].map(v => v + 1);
Adding CSS and Sass isn’t much different from adding
Babel. We need to install the libraries and the loaders, and then modify the
webpack.config.js file.
Install the CSS and Sass libraries with the following commands:
yarn add --dev css-loader style-loader extract-text-webpack-plugin
yarn add --dev sass-loader node-sass
To update your config file, first add the following line near the top of the file:
const ExtractTextPlugin = require("extract-text-webpack-plugin");
Next, you’ll want to add the “loader” underneath the Babel loader:
{
test: /\.css$/,
loader: ExtractTextPlugin.extract("css!sass")
}
Finally, we need to add a “plugins” section underneath the “module” section and
initialize the ExtractTextPlugin:
plugins: [
new ExtractTextPlugin("stylesheets/[name].css"),
]
The ExtractTextPlugin will handle moving our CSS files to their final resting
place.
If we run webpack now, we’re not going to see our CSS files output. I’ll get
to that in a minute, but first I’d like to talk about the importance of
fingerprinting.
When the Asset Pipeline was introduced, a new method of “fingerprinting” assets was also introduced:
Fingerprinting is a technique that makes the name of a file dependent on the contents of the file. When the file contents change, the filename is also changed. For content that is static or infrequently changed, this provides an easy way to tell whether two versions of a file are identical, even across different servers or deployment dates. - The Rails Guides
Fingerprinting allows us to make changes to the assets, compile and deploy the
changes, and in the process, break the caching of the previous versions. Webpack
allows us to continue fingerprinting our assets, by simply adding [hash] to
the filename attibute.
In the “output” section, replace the “filename” line with the following:
filename: "[name]-[hash].js",
And in the “plugins” section, replace the only line with this:
new ExtractTextPlugin("stylesheets/[name]-[hash].css"),
Running webpack now creates the files with the added “hash” signature. There’s
just two more issues we need to tackle: 1) getting Rails to read the correct
asset file; 2) deleting the previously compiled files.
The easiest way to provide Rails the fingerprint is to dump the hash into an configuration file. We can do that by dumping it into an initializer.
To work with files, we’ll first need to include the File System (fs) module at
the top of your webpack.config.js:
const fs = require('fs');
Then we can create the plugin to perform the write:
plugins: [
...
function() {
// output the fingerprint
this.plugin("done", function(stats) {
let output = "ASSET_FINGERPRINT = \"" + stats.hash + "\""
fs.writeFileSync("config/initializers/fingerprint.rb", output, "utf8");
});
}
]
This function opens the config/initializers/fingerprint.rb file for
writing and then adds the ASSET_FINGERPRINT constant and hash to the file. If
the file already exists, it just overwrites it.
Unfortunately, if we only rely on retrieving the fingerprint from the initializer, it means we’ll need to restart the Rails server each time we change and recompile an asset. That won’t do. To get around that, we’ll want to change some behavior depending on whether or not we’re running in production.
Add the following lines near the top of your webpack.config.js file:
const prod = process.argv.indexOf('-p') !== -1;
const css_output_template = prod ? "stylesheets/[name]-[hash].css" : "stylesheets/[name].css";
const js_output_template = prod ? "javascripts/[name]-[hash].js" : "javascripts/[name].js";
When you pass the -p flag to the webpack command, it prepares the assets as
if it were in production. We can use that to our advantage, structuring the CSS
and Javascript output files accordingly. If we’re in production, we’ll create
asset files with fingerprints. When we’re in development, we won’t.
You’ll need to change the filename key of the output section to
js_output_template, and the argument to ExtractTextPlugin to
css_output_template in the plugins directory.
Finally, we can use a helper method to allow us to include the asset files in our views:
module ApplicationHelper
def fingerprinted_asset(name)
Rails.env.production? ? "#{name}-#{ASSET_FINGERPRINT}" : name
end
end
And in our view:
<%= stylesheet_link_tag fingerprinted_asset('application'), media: 'all' %>
<%= javascript_include_tag fingerprinted_asset('application') %>
In the same way that fingerprinting breaks the cache by creating a new filename,
it also leaves behind the old files. If you look in your public/javascripts
and public/stylesheets folders, you’ll see them littered with previously
created assets. We need to get rid of those. We can do that by adding function
to the plugins section to handle it:
plugins: [
...
function() {
// delete previous outputs
this.plugin("compile", function() {
let basepath = __dirname + "/public";
let paths = ["/javascripts", "/stylesheets"];
for (let x = 0; x < paths.length; x++) {
const asset_path = basepath + paths[x];
fs.readdir(asset_path, function(err, files) {
if (files === undefined) {
return;
}
for (let i = 0; i < files.length; i++) {
fs.unlinkSync(asset_path + "/" + files[i]);
}
});
}
});
}
]
This function just loops through all the files in both the javascripts and
stylesheets folders, deleting them as it goes, or breaking out if there are no
files. Don’t worry, the only files which should be in those directories will
have been put there by Webpack.
At this point, when you run webpack from the command line, the only thing
which is output is the application.js file. That’s cool, but we also need the
CSS file. The solution is super simple, we just turn the “entry” into an array
and provide the CSS file:
entry: {
application: ["./javascripts/application.js", "./stylesheets/application.css"]
},
Now, let’s say we had Javascript and CSS we only wanted to use in a specific view like a “Contact us” page. We could continue to dump all the things into “application”, or we could break those assets out into their own files:
entry: {
application: ["./javascripts/application.js", "./stylesheets/application.css"]
contact: ["./javascripts/contact.js", "./stylesheets/contact.css"]
},
Doing this – assuming you have a contact.js and contact.css file - will
produce the application.js and application.css, and contact.js and
contact.css files in the public folder which you can then use in the
appropriate views.
For the sake of completeness, let’s make sure we can deploy to Heroku. I’m going
to assume you have a Heroku account, know how to create a new app, and also how
to type git push heroku.
When we initially commented out gems related to the pipeline, we didn’t touch
the sqlite3 gem. Heroku doesn’t support SQLite, so you’ll need to replace the
sqlite3 gem with the pg gem.
Next, we’ve done everything necessary to host assets from the /public
directory, but Heroku’s still going to try to precompile the assets. To get
around that, we’re going to override (i.e. monkeypatch) the assets:precompile
rake task with one of our own (HT: Josh Becker)
# /lib/tasks/assets.rake
Rake::Task["assets:precompile"].clear
namespace :assets do
task 'precompile' do
puts "#----- Skip asset precompilation -----#"
end
end
Finally, and this may be the most irritating piece of the entire puzzle, before you deploy to Heroku, you’ll need to remember to both compile webpack for production and commit it.
webpack -p
Seriously, you think it’s hard to remember to run heroku run rake db:migrate?
Yeah. This is so much more easily forgettable.
We now have a complete replacement for the Rails Asset Pipeline. You can continue developing Javascript and CSS in your normal directories, and even better, when you’re looking at how to install new Javascript files, you no longer need to worry about installing them manually or hoping for a Ruby gem to do it for you. Now you can just type:
yarn add <packagename>
Ruby on Rails is still awesome and it’s still helping companies launch new products every day, but Node.js and its ecosystem are pretty awesome too. For a while we really did need Rails to handle compiling and bundling our assets into a manageable output, but things have changed and it’s in our best interest to start using the tools which are best suited to the task.
webpack.config.jsconst fs = require('fs');
const webpack = require("webpack");
const ExtractTextPlugin = require("extract-text-webpack-plugin");
const prod = process.argv.indexOf('-p') !== -1;
const css_output_template = prod ? "stylesheets/[name]-[hash].css" : "stylesheets/[name].css";
const js_output_template = prod ? "javascripts/[name]-[hash].js" : "javascripts/[name].js";
module.exports = {
context: __dirname + "/app/assets",
entry: {
application: ["./javascripts/application.js", "./stylesheets/application.css"]
},
output: {
path: __dirname + "/public",
filename: js_output_template,
},
module: {
loaders: [
{
test: /\.js$/,
exclude: /node_modules/,
loader: 'babel',
query: {
presets: ['es2015']
}
},
{
test: /\.css$/,
loader: ExtractTextPlugin.extract("css!sass")
},
]
},
plugins: [
new ExtractTextPlugin(css_output_template),
function() {
// delete previous outputs
this.plugin("compile", function() {
let basepath = __dirname + "/public";
let paths = ["/javascripts", "/stylesheets"];
for (let x = 0; x < paths.length; x++) {
const asset_path = basepath + paths[x];
fs.readdir(asset_path, function(err, files) {
if (files === undefined) {
return;
}
for (let i = 0; i < files.length; i++) {
fs.unlinkSync(asset_path + "/" + files[i]);
}
});
}
});
// output the fingerprint
this.plugin("done", function(stats) {
let output = "ASSET_FINGERPRINT = \"" + stats.hash + "\""
fs.writeFileSync("config/initializers/fingerprint.rb", output, "utf8");
});
}
]
};
const fs = require('fs'); under “Giving Rails the Fingerprint”