For What It's Worth

Sean's article was bleeding edge and about as simple as it could get when it was written. The solutions described in this article weren't available until five months after he wrote his.

The three great virtues of a programmer are laziness, impatience, and hubris. I lean toward laziness more than the other two, often thinking, “There’s got to be an easier way.” A recent example of this was when I was experimenting with streaming LLM output in LiveView. While researching, I found that the most popular article on the topic, Streaming ChatGPT Responses With Phoenix LiveView by Sean Moriarity, to be quite complex. I knew there had to be a simpler solution; that’s when I stumbled across LiveView’s asynchronous functions: assign_async/4, stream_async/4, start_async/4, and handle_async/3.

These functions leverage Elixir’s lightweight processes to perform work off the main LiveView process. When the work is completed, your application can deal with the results without disrupting your user’s experience.

Use cases

There are any number of reasons and situations where you might want to run processes asynchronously in LiveView: presenting a dashboard immediately while allowing charts and graphs to eventually load, performing multiple data fetches concurrently, isolate the UI from high-latency or failure-prone operations, presenting data as it becomes available, etc. Every use case boils down into three general categories:

• Streaming data: Examples of this include LLM-style data output, displaying log output, and live data feeds.
• Long running processes: Running reports, progressive rendering, and fire-and-forget-it style processes are all potential use cases.
• Resilience and lifecycle management: Allows you to separate the retrieval of data from its presentation.

To demonstrate these use cases and how each of the four “async” functions can be used to address the use case, we’ll use a simple LiveView module, rewriting it for each example.

assign_async/4

assign_async/4 is as straightforward as it gets, and once you’ve seen how to use it, you’ll start using it all the time. It’s just like using assign/3, but instead of assigning a value, you provide it with a function. Upon completion, the function updates the assigned key with the result. In the code below (line 28), we assign async_output the simulate_work/0 function which, when called, “sleeps” for two seconds, and then returns a success tuple with key set to “Well that took a long time!”.

defmodule MyAppWeb.PageLive do
  use MyAppWeb, :live_view

  alias Phoenix.LiveView.AsyncResult

  @impl true
  def mount(_params, _session, socket) do
    socket = assign(socket, :async_output, AsyncResult.loading(false))

    {:ok, socket}
  end

  @impl true
  def render(assigns) do
    ~H"""
    <button phx-click="output" class="bg-blue-600 text-white py-2 px-4">
      Async output!
    </button>

    <%= if @async_output.ok? do %>
      <p><%= @async_output.result %></p>
    <% end %>
    """
  end

  @impl true
  def handle_event("output", _params, socket) do
    {:noreply, assign_async(socket, :async_output, &simulate_work/0)}
  end

  defp simulate_work() do
    Process.sleep(2000)
    {:ok, %{async_output: "Well that took a long time!"}}
  end
end

If you were to run the code above, you could click a button and then see “Well that took a long time!” displayed below the button two seconds later.

One thing to note, which will also come into play in the other functions, is the use of Phoenix.LiveView.AsyncResult. We use this module to keep track of a key’s status. We initialize AsyncResult with loading: false to represent an idle state. When the simulate_work/0 function completes, it automatically updates AsyncResult with an ok status.

Each key passed to assign_async/3; will be assigned to an Phoenix.LiveView.AsyncResult struct holding the status of the operation and the result when the function completes.

stream_async/4

In all honesty, I’ve struggled to figure out why stream_async/4 exists. I understand that it’s supposed to help with reducing the boilerplate of start_async/4 and handle_async/3, but it also appears to only work in the mount/3 function and it requires a specific data structure to integrate with Phoenix streams. But for the sake of completeness…

Below is an example of using it to display a list of ten random words from the Lorem Ipsum after a one-second delay.

defmodule MyAppWeb.PageLive do
  use MyAppWeb, :live_view

  alias Phoenix.LiveView.AsyncResult

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> stream_async(:async_output, &stream_response/0)

    {:ok, socket}
  end

  @impl true
  def render(assigns) do
    ~H"""
    <.async_result :let={async_output} assign={@async_output}>
      <:loading>Loading output...</:loading>
      <:failed :let={_failure}>there was an error loading the output</:failed>
      <%= if async_output do %>
        <ul>
        <%= for {x, item} <- @streams.async_output do %>
          <li>ID: <%= x %> - <%= item.id %> - <%= item.word %></li>
        <% end %>
        </ul>
      <% else %>
        You don't have output yet.
      <% end %>
    </.async_result>
    """
  end

  defp stream_response() do
    Process.sleep(1000)

    words =
      "Lorem ipsum dolor sit amet, ullum phaedrum in est, sit viris dissentiunt eu. Ad qui aperiri senserit necessitatibus. In ferri persius vel, te option saperet pertinacia sit. At duis nulla zril per. Alienum accumsan qui ei, at quem constituto pri, ei facer libris cum. Doctus integre blandit pri an, quas intellegam quaerendum eu per."
      |> String.split()
      |> Enum.take_random(10)
      |> Enum.with_index(fn word, idx -> %{id: idx, word: word} end)

    {:ok, words}
  end
end

It looks more complicated than what it is. The bulk of the work is in the stream response which sleeps for one second (i.e. 1,000 milliseconds), and then pulls 10 random words from the Lorem Ipsum string which it returns as an :ok tuple. The list of words is then rendered in the async_result/1 block. We use async_result/1 for rendering, because it handles the potential errors that stream_async/4 might receive. If, for example, stream_response/0 returns an :error tuple, async_result/1 would have rendered “there was an error loading the output”.

This is an example what the above LiveView module will present after waiting one second:

ID: async_output-0 - 0 - In
ID: async_output-1 - 1 - necessitatibus.
ID: async_output-2 - 2 - pri
ID: async_output-3 - 3 - aperiri
ID: async_output-4 - 4 - Ad
ID: async_output-5 - 5 - ullum
ID: async_output-6 - 6 - quaerendum
ID: async_output-7 - 7 - dissentiunt
ID: async_output-8 - 8 - intellegam
ID: async_output-9 - 9 - ei,

In my personal opinion, you’re better off skipping this function and using to the next two. They’re much more flexible and provide finer control over what and when you can perform asynchronous work.

start_async/4 and handle_async/4

Anything you can do with assign_async/4 and stream_async/4, you can do with start_async/4 and handle_async/3. In the example below, we’ll use the two functions to stream random words to the page in the same way an LLM might. The module creates a very simple LiveView page. It has a “Go!” button, which triggers the word stream. A “Cancel” button which cancels the stream and is only visible while words are streaming. Lastly, it has a “Reset” button to clear out previously streamed words.

defmodule MyAppWeb.PageLive do
  use MyAppWeb, :live_view

  alias Phoenix.LiveView.AsyncResult

  @impl true
  def mount(_params, _session, socket) do
    socket =
      socket
      |> assign(:lorem, "")
      |> assign(:async_state, AsyncResult.loading(false))

    {:ok, socket}
  end

  @impl true
  def render(assigns) do
    ~H"""
    <%= if @async_state.loading do %>
      <button phx-click="cancel" class="bg-red-800 text-white py-2 px-4">
        Cancel
      </button>
    <% else %>
      <button phx-click="go" class="bg-blue-600 text-white py-2 px-4">
        Go!
      </button>
      <button phx-click="reset" class="bg-blue-600 text-white py-2 px-4">
        Reset
      </button>
    <% end %>

    <br />

    {@lorem}
    """
  end

  @impl true
  def handle_event("go", _params, socket) do
    pid = self()

    socket =
      socket
      |> assign(:async_state, AsyncResult.loading())
      |> start_async(:data_stream, fn ->
        stream_response(pid)
      end)

    {:noreply, socket}
  end

  def handle_event("reset", _params, socket) do
    socket =
      socket
      |> assign(:async_state, AsyncResult.ok("Reset"))
      |> assign(:lorem, "")

    {:noreply, socket}
  end

  def handle_event("cancel", _params, socket) do
    {:noreply, cancel_async(socket, :data_stream)}
  end

  @impl true
  def handle_async(:data_stream, {:exit, {:shutdown, :cancel}}, socket) do
    {:noreply, assign(socket, :async_state, AsyncResult.ok("cancelled"))}
  end

  @impl true
  def handle_info({:render_lorem, word}, socket) do
    lorem = socket.assigns.lorem <> " " <> word

    {:noreply, assign(socket, lorem: lorem)}
  end

  defp stream_response(pid) do
    word =
      "Lorem ipsum dolor sit amet, ullum phaedrum in est, sit viris dissentiunt eu. Ad qui aperiri senserit necessitatibus. In ferri persius vel, te option saperet pertinacia sit. At duis nulla zril per. Alienum accumsan qui ei, at quem constituto pri, ei facer libris cum. Doctus integre blandit pri an, quas intellegam quaerendum eu per."
      |> String.split()
      |> Enum.random()

    Process.sleep(150)

    send(pid, {:render_lorem, word})
    stream_response(pid)
  end
end

Go!

Clicking the “Go!” button starts everything off. The handle_event/3 function on line 39 first sets the pid variable to the current process. Next, it assigns the :async_state to “loading”, and finally starts stream_response/1 function by using the start_async/4 function.

stream_response/1 might look a little complicated, but all it does is grab a random word from the Lorem Ipsum string, sleeps for 150 milliseconds, sends the word to the parent process, and then call itself again (i.e. recursion). We pass the pid to stream_response/1, because start_async/4 starts it as a child process and we use that child process to return the data to the parent. By passing the parent PID, the child process can utilize message passing to update the UI.

The final piece of the puzzle is handle_info/2 on line 71. This function matches messages sent with {:render_lorem, word}, (e.g. sent from stream_response/1.) It then appends the word to the :lorem assigns variable. At that point, the page is updated with the new string.

Most applications won’t need streaming data. Instead, you’ll use start_async/4/handle_async/3 to perform multiple, related tasks and assign the results as needed. An example might be to fetch stock data from an external source, store the results to the database, and then pull the list of stock data from the database to recalculate and display on the page.

Cancel

When you click the “Cancel” button, it triggers the “cancel” event on line 61. This event first sets the :async_state to “cancelled”, and then uses the cancel_async/3 function to terminate functions spawned by start_async/4. This also has the side effect of executing the handle_async/3 function with the {:exit, {:shutdown, :cancel}} tuple, which just sets :async_state to cancelled.

Reset

The last feature is the “Reset” button. When clicked, this sends the “reset” event to the handle_event/3 function on line 52, which then sets :async_state to “Reset” using AsyncResult.ok/1, and sets :lorem to an empty string.

In Summary

The evolution of Phoenix LiveView has turned what used to be a complex orchestration of manual process management into a streamlined, declarative developer experience. By embracing the “async” suite of functions, you can adhere to the programmer’s virtue of laziness—writing less boilerplate while achieving more robust results.

Choosing the Right Tool

To help you decide which function fits your specific needs, here is a quick reference:

While Sean Moriarity’s original approach was an excellent solution when written, the introduction of these native async utilities means we no longer have to “fight” the framework to handle long-running tasks. Whether you are building dashboards or a real-time LLM interface, these tools allow you to keep the UI responsive and your code maintainable.

In my previous article, Crawling the Web with Elixir’s Broadway and Wallaby, I suggested that it might be “beneficial to use either a Protocol or Behaviour to reduce code duplication,” but when considering which to use for what I’m working on, it left me second-guessing myself yet again. What’s the difference? When would you use one over the other? How do they actually help?

Because I don’t think I’m alone in my confusion, I’m going to try to answer these questions by first explaining what I thought I understood about both, talking about what I got wrong, showing the differences, and then giving a real-world—if contrived and incomplete—example for when you might use both.

What I Knew About Protocols

I thought I had a pretty good handle on protocols; I even wrote an article about them. I knew they provided, “a mechanism to achieve polymorphism in Elixir.” That is,

by implementing functions specific to a protocol, we make sure our structs and built-in data types can take advantage of everything the implemented library has to offer.

I also knew they provided a way to extend a module’s functionality without having access to the module’s source code.

What I Thought I Knew About Behaviours

I’m embarrassed to admit it, but I thought behaviours were only used to define what functions a module had to have. That’s not entirely wrong, but it misses the “why?”. I came to this conclusion after reading José’s article, Mocks and Explicit Contracts. In that article he shows how to create mocks by switching out one module for another in a test environment, but it only worked if both modules, the original and the mock, conformed to the same contract, i.e. Behaviour. Because the Behaviour example from the article did nothing other than define callbacks, I concluded that that’s all they were used for.

This conclusion was further solidified after hearing an Elixir podcast host state that he thought Behaviours were best used in libraries rather than in your project, because you have control over the modules in your own project. That’s probably not what he meant, but it’s what I heard and it conformed to what I was already thinking: Behaviours are kind of useless.

What I Got Wrong

There were three things I got wrong about the Protocols and Behaviours.

Protocols are Just Specialized Behaviours

The first misunderstanding I had was thinking Protocols were Behaviours with added functionality. I came to that conclusion from the following quote:

Protocol is a behaviour with the dispatching logic so you don’t need to hand roll it nor impose a particular implementation in the user module.

– José Valim - Google Group discussion

It’s true that “[a] protocol is indeed a behaviour + dispatching logic,” but what I misunderstood was thinking a Protocol was interchangeable with a Behaviour. They are not.

Behaviours Don’t Provide Functionality

Maybe it’s because I knew about behaviors in C# or because I had heard “contract” used in relation to Behaviours, but for whatever reason, I had it in my mind that Behaviours only defined a module’s functionality, but didn’t provide any itself. I can’t defend this misunderstanding. If I had taken five minutes to think about how GenServer or Plug worked, I would have quickly abandoned that idea: both Behaviours obviously provide lots of functionality.

Behaviours Were Better Suited in Libraries

The idea that Behaviours are better suited in libraries than in your projects came from a couple places. I first heard the argument on a podcast, (which shall remain nameless) but I don’t think it’s what the speaker meant. And because I already had the idea that Protocols were just specialized Behaviours, and Behaviours didn’t provide functionality, it was an easy, if erroneous, conclusion to reach.

What are the Differences

Protocols and Behaviours differ primarily by what they execute against. Protocols work with data types, while Behaviours execute against modules. Every other difference is based off these fundamental concepts.

…a behaviour is internal to a module–the module implements the behaviour. Protocols are different–you can place a protocol’s implementation completely outside the module. This means you can extend modules functionality without having to add code to them…”

– Dave Thomas, Programming Elixir

Protocols Don’t Provide Functionality

The first thing to notice when working with Protocols—that is, if you are creating one—is that they don’t provide functionality, they define it. Consider the Enumerable Protocol:

defprotocol Enumerable do
  # documentation, type definitions, and specs have been removed
  def reduce(enumerable, acc, fun)

  def count(enumerable)

  def member?(enumerable, element)

  def slice(enumerable)
end

It’s not until you implement the Protocol, that you gain functionality.

Example:

defmodule User do
  defstruct [:id, :name, :age]

  defimpl Inspect do
    def inspect(user, _opts_) do
      "#{user.name} (#{user.age})"
    end
  end
end

# iEX

iex :: 1 > IO.inspect %User{id: '10-289", name: "John Galt", age: 38}
John Galt (38)

Protocols Work on Data Types

Unlike Behaviours which work at the Module level, Protocols work on data types. In the previous example, where we implemented the Inspect protocol on User, a %User{} struct is passed to IO.inspect/2. The inspect/2 function is then able to do something with it because User defines how Inspect should handle it.

Protocol is type/data based polymorphism. When I call Enum.each(foo, …), the concrete enumeration is determined from the type of foo.

– Sasa Juric StackOverflow answer

At runtime, Protocols allow us to execute the appropriate logic against the specific datatype. This is dynamic dispatching.

Protocols Allow Polymorphism

In OOP, polymorphism allows objects of different classes to be treated as instances of a common superclass. You get a similar behavior in Elixir when you implement a Protocol in your datatypes.

Polymorphism is a runtime decision about which code to execute, based on the nature of the input data. In Elixir, the basic (but not the only) way of doing this is by using the language feature called protocols.

– Sasa Juric

As we saw when we implemented the Inspect Protocol in the %User{} struct above, any datatype that implements the Inspect protocol can be passed to functions like Kernel.inspect/2 and IO.inspect/2 and Elixir figures out how to handle each at runtime based on the datatype implementation.

Protocols Extend Module Behavior

If you’ve come to Elixir from another language like Ruby or JavaScript, you might be familiar with the term, “monkey patching.” Monkey patching allows us to “open up” a class or object and add or overwrite functionality. While you can’t overwrite functions in Elixir, you can extend functionality through the use of Protocols.

protocols allow us to extend the original behavior for as many data types as we need. That’s because dispatching on a protocol is available to any data type that has implemented the protocol and a protocol can be implemented by anyone, at any time.

– Elixir Lang

As an example, if you had the following Emptiness protocol in your codebase…

defprotocol Emptiness do
  @doc "Returns a boolean value based on the 'emptiness' of the term"
  @spec empty?(term) :: boolean()
  def empty?(t)
end

…you could extend any Elixir type with Emptiness regardless of whether or not it was a 1st-party type or if it was included as a library.

defimpl Emptiness, for: Plug.Conn do
  def empty?(%Plug.Conn{resp_body: nil}), do: true
  def empty?(%Plug.Conn{resp_body: body}), do: length(body) == 0
end

In this example, even though we didn’t create Plug.Conn, we can still add new functionality with the Emptiness Protocol.

Behaviours Require Conformity

Protocols don’t require you to implement every function definition, but if you don’t you may not get every available feature. For example, if you only implement the Enumerable.reduce/2 function in your module, you’ll only be able to pass your module type to some of Enum’s functions. With Behaviours, on the other hand, you either implement every associated function or your application doesn’t run.

A module that declares that it implements a particular behaviour must implement all of the associated functions. If it doesn’t, Elixir will generate a compilation warning.

– Dave Thomas, Programming Elixir

And again…

By declaring that our module implements that behaviour, we let the compiler validate that we have actually supplied the necessary interface. This reduces the chance of an unexpected runtime error.”

– Dave Thomas, Programming Elixir

Behaviours Execute Modules

As stated under What are the Differences, “Protocols and Behaviours differ primarily by what they are executed against. Protocols work with data types, while Behaviours execute against modules.” In order for this to work, modules must conform to the Behaviour by implementing the required functions.

Behaviour is a typeless plug-in mechanism. When I call GenServer.start(MyModule), I explicitly pass MyModule as a plug-in, and the generic code from GenServer will call into this module when needed.

– Sasa Juric StackOverflow answer

Behaviours Define and Implement Common Logic

Behaviours provide a way of abstracting away functionality that is common across every module that would implement them. For example, when you create a module using the GenServer Behaviour and implement init/1 and handle_cast/2, you don’t think about the loop it runs in, or handling state. The GenServer Behaviour provides that functionality. All you need to worry about is implementing the required functions.

A behaviour is a way to say: give me a module as argument and I will invoke the following callbacks on it, which these argument and so on. A more complex example for behaviours besides a GenServer are the Ecto adapters.

– José Valim Google Group discussion

Usually the provided functionality is accomplished through meta programming, and you would use the specific Behaviour, but it’s not required and there’s no reason you couldn’t populate the Behaviour module with “normal” functions for added utility.

Conclusion

We get things wrong all the time. We miss meetings, include or exclude the wrong features, work on the wrong tasks, and the list goes on. Sometimes these mistakes are due to a misunderstanding, sometimes it’s poor communication, and sometimes it’s failing to check your premises. With regard to Protocols and Behaviours, for me it was a little bit of everything.

What I discovered, however, is that Behaviours and Protocols, while superficially similar, serve very different roles in the Elixir ecosystem. Protocols provide extensibility and polymorphism to your types, while Behaviours provide functionality to and demand conformity from your modules. Behaviours are about plugging in modules (GenServer, Plug, Ecto.Repo, etc.). Protocols are about plugging in data types (Enumerable, Jason.Encoder, String.Chars, etc.). Once you see it that way, you’ll never second-guess yourself again.

Example

What follows is a completely contrived and incomplete example of how one might use Behaviours and Protocols together. The idea is that you might want to update a local User’s identity with information from a social network, and also update the social network’s information from that changed locally.

Here is both an example Behaviour and Protocol:

# SocialProfile Behaviour

defmodule SocialProfile do
  @callback get_profile(Identity.t) :: {:ok, term} | {:error, atom}

  @callback update_profile(Profile.t) :: {:ok, term} | {:error, atom}

  @callback to_profile(Identity.t) :: term | {:error, atom}
end

# Identifier Protocol

defprotocol Identifier do
  def to_identity(profile)
end

Modules implementing the SocialProfile Behaviour are required to implement the functions: get_profile/1, update_profile/1, to_profile/1, and follow the TypeSpecs provided.

The Identifier Protocol requires that any module implementing it create the to_identity/1 function. You would use this to transform social media profiles into local Identities.

Here are two Profiles you might create:

# x_profile.ex

defmodule XProfile do
  @behaviour SocialProfile

  @impl SocialProfile
  def get_profile(identity) do
    # logic to retrieve profile based on provided identity
  end

  @impl SocialProfile
  def update_profile(profile) do
    # logic to update profile
  end

  @impl SocialProfile
  def to_profile(identity) do
    [first_name, last_name] = String.split(identity.name)
    %{
      first_name: first_name,
      last_name: last_name,
      email: identity.email,
      bio: identity.bio
    }
  end
end

# Protocol implemented outside of module
defimpl Identifier, for: XProfile do
  def to_identity(profile) do
    %Identity{
      name: "#{profile.first_name} #{profile.last_name}",
      email: profile.email,
      bio: profile.bio
    }
  end
end

# github_profile.ex

defmodule GitHubProfile do
  @behaviour SocialProfile

  @impl SocialProfile
  def get_profile(identity) do
    # logic to retrieve profile based on provided identity
  end

  @impl SocialProfile
  def update_profile(profile) do
    # logic to update profile
  end

  @impl SocialProfile
  def to_profile(identity) do
    %{
      name: identity.name,
      email: identity.email,
      description: identity.bio
    }
  end

  # Protocol implemented inside module
  defimpl Identifier do
    def to_identity(profile) do
      %Identity{
        name: profile.name,
        email: profile.email,
        bio: profile.description
      }
    end
  end
end

The main difference between the two is the use of first_name and last_name on the XProfile versus just having a name in GitHub. Because of that, we need a little extra logic in to_identity/1 and to_profile/1 in XProfile.

I didn’t add example logic to either get_profile/1 or update_profile/1 because it seemed like unnecessary effort. The purpose of both is clear.

Lastly, we have an example of how the above might be used:

# Example Behaviour usage

identity = Users.get_identity_by_email("john@galtsgulch.co")

[GitHubProfile, XProfile]
|> Enum.each(fn social_profile ->
  identity
  |> social_profile.to_profile()
  |> social_profile.update_profile()
end)

In this usage example, we are retrieving the User’s identity based on their email. With that in hand, we are then able to update all of their social profiles after first transforming the identity into the required profile.

In the next example, we retrieve the User’s profile from GitHub, and then transform it into an Identity in order to update our local copy.

# Example Protocol usage

identity = Users.get_identity_by_email("john@galtsgulch.co")

profile = GitHubProfile.get_profile(identity)

identity = Identifier.to_identity(profile)

Users.update_identity(identity)

Thankfully, Elixir’s Broadway and Wallaby libraries provide an excellent solution: Broadway to manage concurrency and load, and Wallaby—while normally used for User Acceptance Testing—to handle visiting and processing webpages; even JS-heavy sites.

In this article, we’ll quickly review where we’ve been with custom Broadway producers, look at the architecture of where we want to go, and then go over the additions and changes we need to make to get to that final solution.

Broadway and Custom Producers

As you probably know, Broadway is a sort of mini-framework built on top of the GenStage library. It’s concurrent and robust, and has built-in batching, rate limiting, and back-pressure. This means that instead of accepting data “pushed” to it, it “pulls” the data from its data sources via its producers. These data sources can be anything: databases, Kafka streams, SQS queues, or, as we’ll see in this article, websites.

To be able to pull data from websites, we’ll need a custom producer that’s able to provide the Broadway consumer with URLs from which to fetch data. We’ve already seen how to do that in my previous article, and in this article we’ll tweak it slightly to customize how each site is crawled.

Let’s look at what the architecture of our app will look like when completed.

Architecture

Here’s the basic architecture of the crawler. The URLQueue is where everything starts. Once a URL is added to it, the URLProducer provides it to the Pipeline, which tells the Crawlers to process it. The Crawlers then find links on the page and adds them to the URLQueue to keep the cycle going.

         +------------+
         |            |             +-------------+
         |  Pipeline  |<------------| URLProducer |
         |            |             +-------------+
         +------------+                    ^
                |                          |
                v                          |
 + - - - - - - - - - - - - - -+            |
   Processors                              |
 |                            |      +----------+
    +---------+  +---------+   ----->| URLQueue |
 |  | Crawler |  | Crawler |  |      +----------+
    +---------+  +---------+
 + - - - - - - - - - - - - - -+
        |           |
        v           |
 +-------------+    |        *----*----*
 | URLRegistry |    -------->|  Magic  |
 +-------------+             *----*----*

The Pipeline is the Broadway consumer. This module defines what producers to use, how many processors to run, how to transform the data provided by the producer, and finally how to process it.

The URLProducer is a GenStage producer that pulls data from the URLQueue and provides data to the Pipeline consumer. It remains unchanged from the previous article.

The URLQueue is a GenServer used to keep track of what URLs to crawl next. As mentioned previously, this is just a List, but could easily be a :queue or even outsourced to SQS, Kafka, or some other datastore.

Crawlers are the workhorses of this architecture. The way we’re building it, each site we want to crawl will have its own Crawler, which is responsible for finding and storing links in the URLQueue, processing the webpage (i.e. Magic), and registering the URL in the URLRegistry so it’s not crawled again.

Finally we have the URLRegistry. This GenServer is responsible for keeping a list of URLs which have been crawled. If a URL is in the list, it should be skipped by the Crawler. In this version, we’re using a MapSet, but for serious work you should use a datastore like ETS or Redis.

Pipeline v2

Pipeline is the Broadway consumer which provides configuration for the producer and processors. For the sake of simplicity, we’re forgoing the use of batching processes.

defmodule MyApp.Pipeline do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {MyApp.URLProducer, []},
        transformer: {__MODULE__, :transform, []},
        concurrency: 1
      ],
      processors: [
        default: [
          concurrency: 2,
          min_demand: 1,
          max_demand: 2
        ]
      ]
    )
  end

  def handle_message(:default, %Message{data: %{module: module, url: url}} = message, _context) do
    module.crawl(url)
    message
  end

  def transform(url_item, _opts) do
    %Broadway.Message{
      data: url_item,
      acknowledger: Broadway.NoopAcknowledger.init()
    }
  end

  def ack(_ref, _successes, _failures) do
    :ok
  end
end

With the exception of the handle_message/3 function, the module remains unchanged from my previous article. You’ll note the data field in %Message{} now expects a Map with keys of module and url instead of the previous URL string. Also note that we’re calling the crawl/1 function of the module that’s passed in. We’ll get to that next.

Crawler

Unless you’re building a crawler for a search engine, LLM, or internet archiving service—and if you are, this is not the article you should be reading—you’ll only need to crawl a handful of sites. If that’s the case, you’ll likely also have specific requirements you’re looking for: links to follow, data to retrieve, and pages to skip. With that in mind, we’ll build our Crawler to limit itself to a single site, Internet Assigned Numbers Authority by starting the crawl at https://example.com.

Before we begin, you’ll need to install Elixir’s Wallaby library (as you would any other library) and Google’s chromedriver (On a mac: brew install chromedriver).

Note: You may also need to install google-chrome as well (brew install google-chrome).

Below is our crawler, MyApp.Crawlers.Example, in its entirety. It looks like a lot, but there’s only two public API functions—init/0 and crawl/1—both of which are described in more detail below. The module itself is responsible for starting and stopping the Wallaby session, visiting pages, capturing links to follow, and processing pages. We’ll break it down into smaller chunks below.

defmodule MyApp.Crawlers.Example do
  alias MyApp.{URLQueue, URLRegistry}
  alias Wallaby.{Browser, Element, Query}

  def init() do
    URLQueue.push(%{
      module: __MODULE__,
      url: "https://example.com"
    })
  end

  def crawl(url) do
    session()
    |> Browser.visit(url)
    |> process_links()
    |> process_page(url)
    |> Wallaby.end_session()
  end

  defp session() do
    {:ok, session} = Wallaby.start_session(
      capabilities: %{
        chromeOptions: %{
          args: [
                "--headless",
                "--no-sandbox",
                "window-size=1280,800",
                "--fullscreen",
                "--disable-gpu",
                "--disable-dev-shm-usage"
          ]
        }
      }
    )

    session
  end

  defp process_links(session) do
    css_selectors = "header a, footer a"

    session
    |> Browser.all(Query.css(css_selectors, minimum: 0))
    |> Enum.each(&store_link/1)

    session
  end

  defp process_page(session, url) do
    IO.inspect "Processing page: #{url}"
    session
  end

  defp store_link(element) do
    element
    |> Element.attr("href")
    |> _store_link()
  end

  defp _store_link("https://www.iana.org/" <> _rest = url) do
    _store_good_link(url)
  end

  defp _store_link("https://iana.org/" <> _rest = url) do
    _store_good_link(url)
  end

  defp _store_link(url) do
    IO.inspect url, label: "Some other link"
    :ok
  end

  defp _store_good_link(url) do
    url_item = %{
      url: url,
      module: __MODULE__
    }

    case URLRegistry.registered?(url_item) do
      true ->
        :ok
      false ->
        IO.inspect url_item
        URLRegistry.register(url_item)
        URLQueue.push(url_item)
    end
  end
end

Fun Fact: setting css_selectors to "a", and changing _store_link/1 to accept any URL, will crawl the entire internet, but not before you’ve consumed all your computer’s resources. I don’t recommend it.

init/0

As its name implies, init/0 is responsible for initializing the Crawler into action. It does so by “pushing” a Map containing the module and starting url, in this case MyApp.Crawlers.Example and “https://example.com” respectively. With this Map in the URLQueue, the URLProducer can retrieve it and pass it to the Pipeline.

def init() do
  URLQueue.push(%{
    module: __MODULE__,
    url: "https://example.com"
  })
end

crawl/1

The crawl/1 function accepts a URL in the form of a string and creates a pipeline for processing that URL. It’s a very readable function, but there are a couple things to take note of: 1) it starts by creating a Wallaby session token and passes that along through the whole pipeline; 2) it finishes by ending the Wallaby session. This last piece is very important. If you don’t end the Wallaby session, you will end up with lots of chromedriver instances running on your computer.

def crawl(url) do
  session()
  |> Browser.visit(url)
  |> process_links()
  |> process_page(url)
  |> Wallaby.end_session()
end

session/0

The pipeline described in crawl/1 passes a Wallaby session token to each function. The session is created using the start_session/1 function. The chromeOptions used are what I found to be the minimum required options to work. A full list of options can be found on the Chromium Command Line Switches page.

defp session() do
  {:ok, session} = Wallaby.start_session(
    capabilities: %{
      chromeOptions: %{
        args: [
              "--headless",
              "--no-sandbox",
              "window-size=1280,800",
              "--fullscreen",
              "--disable-gpu",
              "--disable-dev-shm-usage"
        ]
      }
    }
  )

  session
end

I should also add that Wallaby itself can be configured from a Config file. A basic example is provided below:

# my_app/config/config.exs

import Config

config :wallaby,
  js_errors: false,
  driver: Wallaby.Chrome,
  hackney_options: [timeout: 5_000, recv_timeout: :infinity, pool: :wallaby_pool]

process_links/1

With the session token in hand, we can now start processing our page. The first step is to find all the links on the page and store them. We can do that with Wallaby’s Query.css/2 function. Below you can see that we’re only looking for links in the header and footer elements, which once found, we iterate through each to store them using store_link/1.

defp process_links(session) do
  css_selectors = "header a, footer a"

  session
  |> Browser.all(Query.css(css_selectors, minimum: 0))
  |> Enum.each(&store_link/1)

  session
end

store_link/1

Even though the store_link/1 function and its associated private functions take up more space than anything else in the module, it’s relatively simple and can be broken down into the following steps:

1. Retrieve the URL from the anchor element passed in
2. Compare it against allowed domains
3. Push it onto the URLQueue and URLRegistry if it’s an allowed domain
4. Ignore it if it’s not an allowed domain

The store_link/1 function is what starts things off. It accepts an Element struct, retrieves the URL from the href attribute, and passes that value to _store_link/1.

_store_link/1 then uses binary matching to filter out URLs that aren’t allowed. In our case, we’re only looking for URLs under the “iana.org” domain. If it matches we call _store_good_link/1, otherwise we print the URL to stdout with the label of “Some other link.”

The final step is to call the _store_good_link/1 function. This function builds a Map containing two items: a url and the module used to crawl that URL (module.crawl is used on line 25 of the Pipeline module.) Next, it checks to see if that url_item has already been registered in the URLRegistry. If it has, it returns :ok, otherwise it prints out the contents, registers the item, and pushes it onto the queue to be crawled.

defp store_link(element) do
  element
  |> Element.attr("href")
  |> _store_link()
end

defp _store_link("https://www.iana.org/" <> _rest = url) do
  _store_good_link(url)
end

defp _store_link("https://iana.org/" <> _rest = url) do
  _store_good_link(url)
end

defp _store_link(url) do
  IO.inspect url, label: "Some other link"
  :ok
end

defp _store_good_link(url) do
  url_item = %{
    url: url,
    module: __MODULE__
  }

  case URLRegistry.registered?(url_item) do
    true ->
      :ok
    false ->
      IO.inspect url_item
      URLRegistry.register(url_item)
      URLQueue.push(url_item)
  end
end

process_page/2

The last step in the Crawler pipeline before we end the session is process_page/2. This is where the magic happens. In our example, all we do is print out that we’re processing the page, but in a “real world” scenario, this is where you would retrieve data from the page, submit forms, follow links, etc. I’ll leave it up to you to determine how best to complete this function. You will, however, need to return the session object in order to successfully end the session.

defp process_page(session, url) do
  IO.inspect "Processing page: #{url}"
  session
end

URLRegistry

The last module to consider from the architecture section, and perhaps a bit anti-climactic, is the URLRegistry. It’s a GenServer that uses a MapSet to store a unique list of url_items. The two functions in the API you will use are register/1, to register a url_item, and registered? to determine if a url_item has already been crawled.

defmodule MyApp.URLRegistry do
  use GenServer

  def start_link(args) do
    GenServer.start_link(__MODULE__, args, name: __MODULE__)
  end

  def register(url_item) do
    GenServer.cast(__MODULE__, {:register, url_item})
  end

  def registered?(url_item) do
    GenServer.call(__MODULE__, {:registered?, url_item})
  end

  @impl GenServer
  def init(_args), do: {:ok, MapSet.new()}

  @impl GenServer
  def handle_cast({:register, url_item}, state) do
    {:noreply, MapSet.put(state, url_item)}
  end

  @impl GenServer
  def handle_call({:registered?, url_item}, _from, state) do
    {:reply, MapSet.member?(state, url_item), state}
  end
end

This is a very naive approach to keeping track of crawled URLs, but is useful for demonstration purposes. In a production environment, you might consider an ETS table or a Redis store with TTL (Time to Live) settings so pages can be re-crawled after a set period of time.

Running the Crawler

To run the example Crawler, you need to start the URLQueue, Pipeline, and URLRegistry processes, and then execute the init/0 function for the desired Crawler. Here’s an example of doing that and the resulting output.

iex1> MyApp.URLQueue.start_link([])
{:ok, #PID<0.252.0>}

iex2> MyApp.Pipeline.start_link([])
{:ok, #PID<0.253.0>}

iex3> MyApp.URLRegistry.start_link([])
{:ok, #PID<0.254.0>}

iex4> MyApp.Crawlers.Example.init()
:ok

...after a few moments of waiting...

%{
  module: MyApp.Crawlers.Example,
  url: "https://www.iana.org/"
}
%{
  module: MyApp.Crawlers.Example,
  url: "https://www.iana.org/domains"
}
%{
  module: MyApp.Crawlers.Example,
  url: "https://www.iana.org/protocols"
}

...

Some other link: "https://pti.icann.org/"
Some other link: "https://www.icann.org/"
Some other link: "https://www.icann.org/privacy/policy"
Some other link: "https://www.icann.org/privacy/tos"
"Processing page: https://www.iana.org/help/example-domains"
Some other link: "https://pti.icann.org/"
Some other link: "https://www.icann.org/"
Some other link: "https://www.icann.org/privacy/policy"
Some other link: "https://www.icann.org/privacy/tos"
"Processing page: https://www.iana.org/about/excellence"
Some other link: "https://pti.icann.org/"

In a supervised app, you’d just need to add those processes to your application.ex file.

def start(_type, _args) do
  children = [
    {MyApp.URLQueue, []},
    {MyApp.Pipeline, []},
    {MyApp.URLRegistry, []}
  ]

  opts = [strategy: :one_for_one, name: Glutton.Supervisor]
  Supervisor.start_link(children, opts)
end

Consider Also…

Before we wrap this up there are a few things you should also consider before building your own crawler(s).

Be a Good Netizen

Crawling a webpage requires resources from the hosting site and you’ll want to make sure you don’t overload that site consuming their data. To that end, the authors of the excellent Crawly library have provided four items which define a polite crawler:

1. A polite crawler respects robots.txt.
2. A polite crawler never degrades a website’s performance.
3. A polite crawler identifies its creator with contact information.
4. A polite crawler is not a pain in the buttocks of system administrators.

Poolboy

In the Example crawler we initiate and end a Wallaby session in the crawl/1 pipeline. But what happens if an error occurs before the pipeline finishes? Well, you end up with an unfinished session which can eat up resources. A better solution would be to use poolboy to manage sessions.

For brevity’s sake, I didn’t include poolboy’s use in this article, but I have another article tackling its use: Elixir, Poolboy, and Little’s Law.

Build Crawlers with Protocols or Behaviours

If you end up with multiple Crawlers, you might find it beneficial to use either a Protocol or Behaviour to reduce code duplication. Again, I didn’t include that here for the sake of brevity, but what a great idea for an article.

Using Floki

Wallaby’s good at retrieving HTML from a page, but if you need more control over parsing it, you can output the page source with Wallaby.Browser.page_source/1 and use Floki to get at the content you want.

Wrapping Up

Crawling the web, like traversing a file system, is a kind of recursive process: you start from a single page, collect all the links from it, crawl the next page, collect the links, and so on until you’ve crawled everything you need. It’s a perfect problem to solve with Broadway, thanks to its concurrent processing, use of back-pressure, and rate-limiting. When combined with Wallaby and its use of headless browsers, you’re no longer limited to traditional HTML-only pages, but can crawl SPA sites as well.

All the code above and from my Building Custom Producers with Elixir’s Broadway article can be found in my Glutton GitHub repo.

The thing about Elixir’s Broadway is that once you’ve used it, you start seeing opportunities to use everywhere. We used it heavily at my previous job, and it was my go-to tool for any continuous stream of data that required concurrent processing.

Recently I decided to build a web crawler/scraper to pull in data for my side project, Makerplans. I’d been using Crawly with some success, but so many sites are SPAs that it makes libraries like Crawly, which rely on full page reloads, less useful.

I’ll write more about building my crawler/scraper, Glutton, in a future article. For now, I want to focus on the first problem I ran into: getting custom Broadway producers to work.

v1

The Dashbit team did a great job documenting Broadway, and they even have a page describing how to build Custom Producers. It’s what I referenced while building the first version of Glutton’s Producer (shown below). On the surface, this appears to be doing everything the custom Producer example is doing, but as we’ll see, there’s a problem.

defmodule MyApp.URLProducer do
  use GenStage

  alias MyApp.URLQueue

  def start_link(args) do
    GenStage.start_link(__MODULE__, args)
  end

  @impl GenStage
  def init(_args), do: {:producer, []}

  @impl GenStage
  def handle_demand(demand, state) when demand > 0 do
    urls = URLQueue.pop(demand)

    {:noreply, urls, []}
  end
end

You’ll notice first that the URLProducer uses the GenStage behavior. It’s very similar to the GenServer behavior with the exception of init/1’s return value and the handle_demand/2 functions. Whereas a GenServer’s init/1 will return an {:ok, state} tuple, Broadway expects its producers to return {:producer, state}, which is what we do above.

handle_demand/2 is very similar to GenServer’s handle_call/3 or handle_cast/2 functions, except it, uh…handles…demand from consumers, which is what your main Broadway module is.

Speaking of the Broadway consumer, let’s make that:

defmodule MyApp.Pipeline do
  use Broadway

  alias Broadway.Message

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module: {MyApp.URLProducer, []},
        transformer: {__MODULE__, :transform, []},
        concurrency: 1
      ],
      processors: [
        default: [
          concurrency: 2,
          min_demand: 1,
          max_demand: 2
        ]
      ]
    )
  end

  def handle_message(:default, %Message{data: url} = message, _context) do
    message
    |> IO.inspect
  end

  def transform(url, _opts) do
    %Broadway.Message{
      data: url,
      acknowledger: Broadway.NoopAcknowledger.init()
    }
  end

  def ack(_ref, _successes, _failures) do
    :ok
  end
end

This is a standard Broadway consumer, so if you’ve used Broadway before, this should look familiar. I kept the concurrency and demand low because each Glutton process runs a headless Chrome browser, and processing is already slow enough that higher demand isn’t necessary.

Lastly, we need a “queue” from which our custom producer can draw. We can do that with a simple GenServer.

defmodule MyApp.URLQueue do
  use GenServer

  def start_link(args) do
    GenServer.start_link(__MODULE__, args, name: __MODULE__)
  end

  def push(url) do
    GenServer.cast(__MODULE__, {:push, url})
  end

  def pop(count) do
    GenServer.call(__MODULE__, {:pop, count})
  end

  def list() do
    GenServer.call(__MODULE__, :list)
  end

  @impl GenServer
  def init(_args), do: {:ok, []}

  @impl GenServer
  def handle_cast({:push, url}, state) do
    {:noreply, [url | state]}
  end

  @impl GenServer
  def handle_call({:pop, count}, _from, state) do
    {urls, new_state} = Enum.split(state, count)
    {:reply, urls, new_state}
  end

  @impl GenServer
  def handle_call(:list, _from, state) do
    {:reply, state, state}
  end
end

It’s a straightforward GenServer. state is just a list to which we can add URLs (push/1), remove the next one (pop/0), or get the current contents of the queue (list/0).

Running v1

If you were to start these processes up…

iex(1)> MyApp.URLQueue.start_link([])
{:ok, #PID<0.196.0>}
iex(2)> MyApp.Pipeline.start_link([])
{:ok, #PID<0.198.0>}

…and add items to the queue…

iex(3)> MyApp.URLQueue.push("http://example.com")
:ok
iex(4)> MyApp.URLQueue.push("http://example.com")
:ok
iex(5)> MyApp.URLQueue.push("http://example.com")
:ok
iex(6)> MyApp.URLQueue.push("http://example.com")
:ok
iex(7)> MyApp.URLQueue.push("http://example.com")
:ok

…you’d notice that…uh…nothing happens.

Hmmm. Okay. What happens if you add items to the queue before starting the Broadway consumer?

iex(1)> MyApp.URLQueue.start_link([])
{:ok, #PID<0.196.0>}
iex(2)> MyApp.URLQueue.push("http://example.com")
:ok
iex(3)> MyApp.URLQueue.push("http://example.com")
:ok
iex(4)> MyApp.Pipeline.start_link([])
{:ok, #PID<0.198.0>}

Now you’ll see…

%Broadway.Message{
  data: "http://example.com",
  metadata: %{},
  acknowledger: {Broadway.NoopAcknowledger, nil, nil},
  batcher: :default,
  batch_key: :default,
  batch_mode: :bulk,
  status: :ok
}
%Broadway.Message{
  data: "http://example.com",
  metadata: %{},
  acknowledger: {Broadway.NoopAcknowledger, nil, nil},
  batcher: :default,
  batch_key: :default,
  batch_mode: :bulk,
  status: :ok
}

…but if you were to add more URLs to the queue, nothing else happens. What gives?

The Problem in v1

As mentioned at the beginning of this section, there was a problem in the way I initially wrote the producer: I followed the documentation. While the Broadway documentation for Custom Producers is useful, it falls short in describing how to handle instances where data isn’t always available to retrieve (i.e. what happens when you’ve processed everything?) In their Counter example, they provided the following function for handling demand from the Broadway Consumer:

def handle_demand(demand, counter) when demand > 0 do
  events = Enum.to_list(counter..counter+demand-1)
  {:noreply, events, counter + demand}
end

Notice that when there is demand, this function returns both a list of integers—which is always available since it generates the list itself—and updates state. There’s never a scenario when this function won’t be able to handle demand. But in our app, it’s likely, and even expected, to run out of URLs to crawl. What’s going on?

In response to this very problem brought up in a GenStage issue, José explained that if a consumer (our Pipeline) sends demand and the producer (URLProducer) can’t serve that demand immediately, the producer should “store” that demand until it’s able to fulfill it.

The mistake I made was thinking the consumer would continually ask its producers for data. It doesn’t. It asks once and expects its producers to fulfill that demand whenever they’re able. Because our handle_demand/2 function only runs when a request is made from the consumer, and the consumer is only going to ask once until demand is fulfilled, our app stalled out.

The way to fix that is to make our URLProducer keep checking the URLQueue for data.

v2

Here’s the second version of URLProducer.

defmodule MyApp.URLProducer do
  use GenStage

  alias MyApp.URLQueue

  @receive_interval 5_000

  def start_link(args) do
    GenStage.start_link(__MODULE__, args)
  end

  @impl GenStage
  def init(_args), do: {:producer, %{demand: 0}}

  @impl GenStage
  def handle_demand(incoming_demand, state) when incoming_demand > 0 do
    schedule_receive_messages(0)

    {:noreply, [], %{state | demand: state.demand + incoming_demand}}
  end

  @impl GenStage
  def handle_info(:receive_messages, state) do
    case URLQueue.pop(state.demand) do
      urls when is_list(urls) and length(urls) > 0 ->
        {:noreply, urls, %{demand: state.demand - length(urls)}}

      [] ->
        schedule_receive_messages(@receive_interval)
        {:noreply, [], state}
    end

  end

  defp schedule_receive_messages(interval) do
    Process.send_after(self(), :receive_messages, interval)
  end
end

There are several changes of which to take note. The first is state. Previously, we were just passing around an empty list, because we were completely handling demand on each request. Now we want to keep track of demand and to do so, we’ll store it in a map. We could keep track of it as an integer, but in most cases, you’ll also want to keep track of a buffer of data. For example:

%{urls: [], demand: 2}

The next thing to notice is that instead of retrieving URLs from the queue in handle_demand/2, we’re calling schedule_receive_messages/1. This function— along with handle_info/2 which handles the sent message, :receive_messages— sets up an asynchronous loop to continually pull data from URLQueue. It responds with data if it’s able to fulfill demand; otherwise, it schedules itself to check again in five seconds (@receive_interval.) Without schedule_receive_messages/1, and Process.send_after/3, our producer would stay idle after the first time it was unable to address demand. Using it allows the producer to continuously check for new work asynchronously.

Everything else remains the same. If we were to launch our app again and add URLs to the queue, we’d see that our Pipeline consumer outputs data within five seconds of data getting added to the URLQueue.

iex(1)> MyApp.URLQueue.start_link([])
{:ok, #PID<0.196.0>}
iex(2)> MyApp.Pipeline.start_link([])
{:ok, #PID<0.198.0>}
iex(3)> MyApp.URLQueue.push("http://example.com")
:ok
iex(4)> MyApp.URLQueue.push("http://example.com")
:ok

%Broadway.Message{
  data: "http://example.com",
  metadata: %{},
  acknowledger: {Broadway.NoopAcknowledger, nil, nil},
  batcher: :default,
  batch_key: :default,
  batch_mode: :bulk,
  status: :ok
}
%Broadway.Message{
  data: "http://example.com",
  metadata: %{},
  acknowledger: {Broadway.NoopAcknowledger, nil, nil},
  batcher: :default,
  batch_key: :default,
  batch_mode: :bulk,
  status: :ok
}

The Broadway library is one of Elixir’s unsung heroes. It’s lightweight, easy to use, and its processing capabilities are unparalleled (Because they run concurrently. Get it? Ugh, that was so dumb.) Once you’ve figured out how to use it to solve one problem, you start seeing lots of places where Broadway is a good fit, but some of those problems are going to require a custom Producer.

As we’ve seen, custom producers are easy to build, but unless they themselves are the data generators, it can be a little tricky to keep them providing data to the Broadway consumer. To get around that, we used a combination of Process.send_after/3 and GenStage’s handle_info/2.

I kept the code in this article as simple as possible. To see examples of more robust solutions, I encourage you to look at the code for both the “Official” and “Off-Broadway” producers.

In the next article, we’ll build on this foundation and use Wallaby to start crawling websites.

This idea isn’t original. In fact it’s over 100 years old. It’s a productivity method created by a man named Ivy Lee when he was asked by Charles M. Schwab to help him “increase the efficiency of his team and discover better ways to get things done.” (The Ivy Lee Method - James Clear)

What it is

The Ivy Lee Method can be summarized like this:

1. At the end of each workday, write down the most important tasks you need to complete the next day (no more than six).
2. Prioritize those tasks in order of importance.
3. The next day, start with the first task and work on it until it’s finished before moving to the next.
4. Continue down the list, moving on only after completing each task.
5. At the end of the day, move any unfinished tasks to the next day’s list.
6. Repeat daily.

Why it works:

• It allows you to focus on what matters most and ignore everything else
• It keeps your to-do list manageable (no more than six tasks)
• It gives you something to look forward to, because you know exactly what to work on the next day
• It builds daily momentum through clarity and discipline

How to use it

Until recently, my mistake in using this method was failing to clearly define my goals and making sure they were achievable. This led to ambiguous tasks that never seemed to end, and a list of tasks that kept growing. The solution was to start making SMART goals.

SMART is a framework for setting clear, actionable goals. Each goal should be:

• Specific: Clearly defined, focused, and unambiguous
• Measurable: Quantifiable so progress can be tracked
• Achievable: Realistic and attainable given resources and constraints
• Relevant: Aligned with broader objectives and meaningful to you
• Time-bound: Has a clear deadline or time frame

Another surprisingly effective habit is visualizing myself working on the next day’s tasks. For whatever reason, going through the tasks in my head during the course of the evening helps me look forward to starting my work the following day.

Lastly, while it’s all well and good that you use this method to improve your own productivity, it’s even better to involve your team. If you’re a team lead, manager, or above, it’s a simple matter to add a Slack reminder to a channel to remind the team to set their goals for the next day. Why should you be the only one benefiting from this productivity boost?

Conclusion

When Ivy Lee suggested this method to Charles M. Schwab, he only asked that Schwab send him a check based on the value he thought it brought to his organization. “After three months, Schwab was so delighted with the progress his company had made that he called Lee into his office and wrote him a check for $25,000.” That’s the equivalent of more than $500,000 in 2025. If Charles M. Schwab found that much value in such a simple change, doesn’t it make sense that you would too?

20 minutes. That’s how long I sat staring at a blank page while the interviewer waited, patiently observing. 20 minutes of considering the different possibilities; 20 minutes of talking myself down from panic; 20 minutes of painful inactivity. The longer I took the more I worried about what my interviewer was thinking. I finally gave up trying to figure out the perfect solution and just started coding something. The mental blocks shattered, and everything began to fall into place.

20 minutes. That’s how long I sat patiently waiting for her to start solving the challenge. 20 minutes of wondering if I needed to give her a hint; 20 minutes of watching her try to calm down; 20 minutes of me trying to stay engaged. The longer she took, the more I worried this was a waste of both of our time. She finally gave up, and in tears stopped the interview.

These two moments — separated by perspective, but still similar — underscored for me that the problem wasn’t just with the candidates. It was with the interview process itself.

There had to be a better way.

The Purpose of the Coding Interview

Coding assessments have become notorious in our industry for presenting problems you would never face in everyday work, conducted by interviewers more focused on intellectual sparring than genuine evaluation, and take-home challenges you spend your weekend working on only to receive a curt followup email thanking you for your time, but they’ve already filled the position. They are intense and stressful, they vary from company to company, and they don’t seem to achieve their stated goal.

Even so, they aren’t without merit.

There are two reasons for the coding interview: to filter out the unqualified, and to see how candidates think and work. The question is, “Do traditional technical interviews address these two reasons?” I argue that they do not.

Where the Traditional Coding Interview Falls Short

First, consider the reason to filter out the unqualified. Yes, you want to eliminate candidates who lack the competency for the role, but you don’t want to eliminate those who are competent, but who don’t perform well under high-pressure and time-constrained situations.

Consider these findings from Microsoft:

The impact on performance was drastic: nearly twice as many participants failed to solve the problem correctly, and the median correctness score was cut more than half, when simply being watched by an interviewer.

—Does Stress Impact Technical Interview Performance?

This means traditional interviews can systematically misrepresent candidate ability. The results for women were even worse:

In the public setting, no women (n = 5) successfully solved their task; however, in the private setting, all women (n = 4) successfully solved their task—even providing the most optimal solution in two cases.

—Does Stress Impact Technical Interview Performance?

Second, I remain unconvinced that testing candidates’ abilities to solve LeetCode challenges, (easy or hard,) shows anything beyond a person’s ability to memorize challenges and handle high-stress, test-taking scenarios. Homegrown challenges are often equally unfruitful. When hiring, I’m looking for a candidate’s ability to assess trade-offs, that they understand best practices, that they produce quality code, and have a good philosophy of software design. Traditional technical interviews don’t address these concerns.

A Turning Point

The turning point for me came after reading an article about replacing the technical challenge with code reviews. In this scenario, the interview transforms from a challenge to a conversation, wherein the candidate is required to evaluate the code, explain what it does, and provide feedback about how it might be improved. This style is effective because it reduces the anxiety of the challenge, reveals problem-solving approaches in a realistic context, and mirrors the kinds of technical conversations they would have on the job.

After using this on a handful of interviews, however, I found it fell short in two areas: 1) reading code isn’t the same as writing code; 2) you don’t get to see how the candidate works. This led me to come up with, what I believe is a better solution.

A Better Way

I finally settled on the following solution:

• Provide at least one file and associated tests from existing production code (these are run from the interviewing environment, not in production)
• Modify the code and tests to introduce bugs or inefficiencies as desired and remove features you want the candidate to add
• Provide a prompt to fix any failing tests, add a new function/method to do x, and perform any refactorings on the file or tests that make sense.

An example prompt might look like the following:

The /ui/version endpoint, represented in the app.js file, is used by the AcmeApp to ensure the version used by the backend matches the version running on the client’s machine. Each time the UI is deployed, it uses the API to set the version in the database.

Because this endpoint has been around since the beginning of AcmeApp, it has gone through several iterations. The current version is a bit of a mess, and we would like to refactor it.

It has three endpoints:

• GET /ui/version - returns the current version
• POST /ui/version - sets the current version
• DELETE /ui/version - deletes the current version

Things to note:

There should only ever be one version in the database.

What we need you to do:

• Refactor the code to make it more readable, maintainable, and modern.
• Add tests to ensure the code works as expected
• Some initial tests have been written, but more may be needed
• Fix any bugs you find
• Identify security issues, if any

Files

• main.js - the test file
• app.js - the actual app file

— Don’t modify these files —

• support/semver.js - simplified version of the semver library
• support/db.js - a fake database.
• support/http.js - a dumbed down HTTP mocking library for the response object

Why it’s Better

What follows are just some of the reasons why performing technical interviews this way is better:

See how candidates work in a realistic setting

One of the reasons this is a better solution to the technical interview is that it better reflects a day-in-the-life of what candidates will do: reading code, fixing bugs, adding features, writing tests, and reading product requirements. By using this method, you get insight into how the candidate works and at what level.

• Do they use tests to validate their code?
  • Do they even run the tests? (Don’t get me started)
• What’s important to them when refactoring?
  • An inexperienced engineer might waste time reformatting a file to fit their preferences while more experienced engineers will focus on efficiencies, good naming, and design.
• Do they follow instructions or make assumptions?
  • There have been many instances where candidates change files they were explicitly told not to modify.
• Can they do the work, and at what level?

Avoid the blank-page problem

One of the biggest problems with traditional coding challenges is facing the blank editor. How should I start this? How should I organize it? What’s the right solution? You can imagine a dozen different ways to solve it, but you’re not sure which way is optimal. When you are given an existing page of code, on the other hand, you’re able to focus on what the code is currently doing, and sidestep some of the anxiety brought on by facing the Paradox of Choice.

Encourages a two-way technical discussion

The whole point of the interview process is to determine if a candidate is a good fit for your team: Are they qualified? Are they a good culture fit? Are they better than the rest of the candidates? Portions of these questions will be answered in other interviews, but additional information can be provided in the technical interview as well. As the candidate “solves” the challenge, it’s important to keep dialogue open to understand why they’re making the choices they are:

• Why was it important to do x or y?
• Did you consider x?
• Why did you break out that block into a new function?
• Why did you make that a private method?
• Is that change optimal? Why? Is it important that it is?

These are just a few of the many types of questions you can ask during the process, the purpose of which is to better understand why the make the choices they do, the trade-offs they consider, and if this is someone you or your team would want to work with.

Adjusts for your company and your project

As stated, this type of technical interview should use code from an existing production code base. This allows your interviews to be specific to your company and project. By doing it this way it allows you to see if candidates have the background and experience to perform the role to which they are applying. For example, a database company might use a challenge that only those who have worked with database would be able to answer. Likewise, companies focused on AI, web applications, search, or anything else could tailor the challenge for their specific industry, and not settle for a cookie-cutter coding challenge.

An Imperfect Solution

There isn’t a perfect technical interview. Candidates’ anxiety will still run higher than in Q&A style interviews, interviewers will still occasionally be detached or condescending, and the wrong candidate will still get through when the better one gets left behind. We can’t attain perfect, but we can and should improve our process so we can get closer.

Traditional interviews fall short in helping candidates overcome anxiety and accurately measuring competence and cultural fit. Attempts have been made to improve technical interviews with take-home challenges, code review interviews, and Bug Squash interviews. Each iteration has made improvements, but have also introduced other problems.

The solution presented in this article seeks to address the issues in the traditional coding interview as well as the problems in other types as well. It affords the candidate the opportunity to show how they would work against a real-world problem, provides greater opportunities for interviewers to ask questions and get a feel for culture fit, it helps alleviate anxiety by starting candidates off with existing code, and it can be adjusted for any company and project.

It may not be perfect, but it’s a proven step toward more accurate, fair, and effective hiring — and worth serious consideration by any team that wants to improve the quality of its hires.

It should come as no surprise that managers talk about their teams with one another: who’s high-maintenance and who’s easy, who’s performing well and who’s not, and who’s adding positive energy to the team and who’s a cancer. I say it should come as no surprise, because team members have similar conversations. What might surprise you are the conversations we have about people we know need to be promoted. Often, these conversations begin with a manager discussing their own team members, but plenty of conversations are started by managers who recognize the value of individuals on other teams. We want to see effort and value rewarded.

If you’ve struggled to advance from a junior position to a senior one, or from individual contributor to management, the question you might be asking is, “How can I become that person management wants to promote?”

In short: mind the gap.

A gap, as you already know, is the space between two things. In the case of the London Underground – where the term “Mind the Gap” originated – it’s the space between the landing and the train car. In business it could be the gap between your experience and what’s required for the next level, the amount of work left undone, opportunities that are missed, or the knowledge and experience that is lacking in the organization. Or, to put it another way, it’s what you’re lacking (internal gaps) or what the organization is lacking (external gaps).

Internal Gaps

Inscribed upon the Temple of Apollo in the ancient Greek precinct of Delphi, was the phrase, “Know thyself.” It was a phrase instructing the reader to understand what they were capable of, where they fell short, what their potential was, and above all, to be honest with themselves about it all. After all, no one is easier to deceive than oneself.

To that end, before you can advance to the next level, you need to understand where you are. The easiest way to do that is to look at the job ladder for your position: are you meeting all the requirements for your current position? What are you lacking for the next position up? What comes naturally, and what doesn’t? There are going to be areas you don’t want to focus on, because they don’t come easily, but those are likely the areas you need to work on most.

If your company doesn’t have a career ladder designed for your position, you have a couple options. The first is to look outside your company to see what other companies have set up. The second is to look at people within your company who have the position you’re aiming for and compare yourself against them. You might even schedule a conversation with them about how they advanced into the position.

Although I’m putting this last in the section, it’s the most important: talk to your boss about their perspective on where you fall short. They should know better than anyone what they’re looking for and what’s expected, and they should also know where you are in relation to that next step. Be warned, however, they may point out that you still have work to do to fulfill your current position. Regardless of what they say—and potentially how they say it—listen to them and consider their feedback. They’re the one who will be recommending you for promotion, and ignoring their feedback will only set you up for failure.

External Gaps

If dealing with your internal gaps is about knowing yourself, then external gaps are all about knowing your team, your boss, and your organization. It’s about recognizing what they’re missing or are unable to get to, and stepping up to support them. Addressing your internal gaps is like providing an argument against passing you over due to a lack of competency. Addressing external gaps, on the other hand, argues for recognizing the position you’re already fulfilling.

If, as they say, “all politics is local,” then it makes sense to first focus on your local environment, i.e. your current team. The first thing to look for is what’s going undone or what would help improve productivity. Other things to consider are becoming the “go-to person” for information about your project, or becoming the most helpful person on the team. How you go about that will be unique to your team, but one thing is for certain: you’ll need to outwork everyone else.

Expanding your perspective from looking at the gaps in your team, it’s time to look at the gaps your boss is either overlooking or unable to address. The easiest thing you can do here is to ask if there’s anything you can take of his or her plate—It goes without saying that you should do this only when all your other responsibilities are taken care of. Beyond that, listen for what they complain about in meetings, or areas of improvement they often “wish” for. Consider also what you would do when faced with the same decisions and be—respectfully—sound counsel for your manager. Having said all of this, remember, your goal shouldn’t be to replace your manager prematurely, but by being the obvious choice to replace them when they are promoted or leave.

The last area to look at is your organization. This can be your department, region, or even the entire company. Regardless of the size, all organizations have gaps. They tend to be more easily seen in small companiies, but larger companies have gaps too—often around their ability to maneuver.

Miscellaneous

Not everything fits neatly in the categories of internal and external gaps, but they warrant brief discussions.

Goals

Before angling for that next promotion, you need to know what it is you’re aiming for, because that will dictate how you fill in the gaps. If you want promoted on the IC (Individual Contributor) path (e.g. from Software Engineer to Senior Software Engineer) then the two things you need to do are performing the work of that next role and consistently outworking your peers. If you are on the management track, however, doing more IC work may not cut it. In those cases, your leaders may be looking for your ability to delegate and manage projects, not just do all the work for the project.

Capacity

Unfortunately, you could be doing all the right things: working harder, helping your boss, being the go-to person on your team, and it still may not be enough. There are any number of reasons for this, but it usually comes down to capacity. If there are no openings or if there isn’t the budget for the promotion, there’s not much you can do. But, there is something you can do.

Hiring freezes and resource constraints are temporary. If you are serious about moving up to that next rung of the career ladder, use this time to prepare yourself for that next position; it will open up eventually. Focus on your growth and do what you can to make sure you’re the obvious choice when constraints ease up.

Reality

Here’s the ice-cold bucket of reality: you are not owed or guaranteed a promotion, a raise, or even kudos. You may be doing the work of five people and most of your boss’ job, but your company doesn’t owe you anything other than a paycheck and not violate certain laws. It sucks, and maybe it seems unfair, but that’s the way it is. If you’re in this sort of position, you should talk to your manager to find out why.

“But I’m the victim or racism, sexism, ageism, homophobia, religiophobia, or some other -ism or -phobia.” You might be. It happens, and you would be surprised at how frequently it happens. It could also be that someone doesn’t like you in particular. This also happens. Everyone has obstacles they’re trying to overcome. Everyone. Some people have larger obstacles than others. Some people are smarter, prettier, or better socially than others. It’s reality. The way to get around that is to “be so good they can’t ignore you.”

I’ve mentioned “career ladder” a couple times in this article, but the reality is that oftentimes the “ladder” is more like a jungle gym, where you move horizontally or downward in order to move up to the position you really want. In both cases above, the way forward may be to move horizontally or (what feels like) backwards. Leaving your role for another, even if it’s inside your company, can feel like a setback, but is it worse than waiting on something that’s out of your control or actively working against you?

Final Thoughts

“Mind the gap.” It’s not only good advice for stepping off a subway car, it’s also good for your career. The gaps you face can be internal: skills, experience, knowledge, or even your network. Gaps can also be found externally: the work your boss isn’t able to get to or isn’t aware of, opportunities on your team, and also areas in your organization which go unnoticed. Sometimes gaps are too big to overcome because of limited resources, discrimination, or something else entirely. Some gaps can only be overcome by going around or even backwards.

Promotions rarely go to the person who simply wants the job—they go to the person who’s ready to fill it. Look around, where are the gaps? Fill them. Quietly. Consistently. Continuously. When the time comes, no one will question whether you’re ready; they’ll assume the job is yours already.

“Huh. That’s weird.” That was my first thought when I received a calendar invite from our CPO (Chief Product Officer)/head of engineering (which is also weird). It shouldn’t have been weird since there was only a single layer (my director) between him and myself, but I’d only met with him once since he started eight months ago, so yeah, it was weird.

Within moments of joining the meeting the CEO and head of HR also joined. I immediately knew what was happening and said, “Oh. I’m getting laid off.” And that’s what was happening. After the CPO read his script he bounced and the CEO and HR guy explained the “why” and “what for”, and told me I would be able to work for another month and then receive a decent severance package.

It’s better than most people get.

What now?

No one tells you this, but when you lose your job you immediately get a new job; it just takes a while to figure that out. My new job is finding a new job. It’s obvious from the outside, but when you’re blindsided by getting tossed aside it can take a while for reality to settle in; even when you know what you’re supposed to do from the very start.

My days now consist of doing everything I can to find the next opportunity: applying for jobs, writing articles to get noticed, researching and experimenting with new technology, and [shudder] network. There’s, of course, no way I can spend 40 hours a week doing all of that, so I intend to practice bass guitar, spend time woodworking, and help out more around the house.

What the hell?

As of this writing, it’s been a month and I think I’m at a good place now. I’m less anxious about the future in spite of only having one interview in that time, and in spite of hearing more and more stories about layoffs. I won’t bore you with my reasoning, but I think this happened at the right time and I think something better is around the corner. I’m hopeful for the future.

Lest anyone think I’ve risen above my baser emotions like anger and resentment, I haven’t. I still have moments—usually while working out—where those emotions rise to the surface. It’s just part of the process, which I’m trying to embrace. Do I have moments where I want to lash out and leave a scathing GlassDoor review or whine about it on social media? Of course, but I’m not going to.

2 Consider it all joy, my brethren, when you encounter various trials, 3 knowing that the testing of your faith produces endurance. 4 And let endurance have its perfect result, so that you may be perfect and complete, lacking in nothing.

—James 1:2-4

There’s also the matter of embarrassment. It’s embarrassing to lose your job. It’s embarrassing to have to tell your wife and kids you were laid off. It’s embarrassing to add that “green circle of shame” to your LinkedIn profile. But so what? I know what I accomplished while at ActiveProspect, I know I have the respect of my peers and the teams I managed while working there. Everything else is ego and emotional noise. The truth is, sometimes “bad” things happen to you regardless of the efforts you put in.

When we are guests at a dinner party, we content ourselves with the food on offer; if anyone were to tell the host to put out fish or cake, he would seem rude. In real life, however, we ask the gods for what they do not give, and this though they have provided us with plenty. – Epictetus, Fragments, 17

Or as Job put it:

Shall we indeed accept good from God and not accept adversity?

—Job 2:10

I’m genuinely grateful for the six years I had at ActiveProspect. I started as a Senior Elixir Engineer and ended as a Senior Engineering Manager. I hired a lot of great talent (most of whom are still there), made some great friends, learned a lot about scaling, startups, and management, and I have no regrets about the work I did there. ActiveProspect was a blessing to me and my family for these past six years, and I’m grateful for every moment of it; even the layoff.

What’s next?

I have no idea where I’ll end up next. Ideally I’ll end up managing another software development team for a small to medium company and I’ve applied for a number of roles along those lines, but I’m open to doing something completely different too. I have some ideas of where I’d like to take my career, but that’s three to five years out. In the meantime I’ll be exploring AI quite a bit in conjunction with working on Makerplans.

What I do know is this is not a setback, and it’s only temporary.

The 1-1 isn’t intended to be a time to update your manager about what you’re doing, or whether or not you’ll get your tasks done on time. Instead, as Andy Grove points out in High Output Management, it’s about “…mutual teaching and exchange of information. By talking about specific problems and situations, the supervisor teaches the subordinate his skills and know-how, and suggest ways to approach things.” The 1-1 is for you and your professional development.

The Purpose of the 1-1

In her book, The Manager’s Path, Camille Fournier identifies the two main purposes of the 1-1:

1. “[T]hey create connection between you and your manager.”
2. They provide “a regular opportunity for you to speak privately with your manager about whatever needs discussing.”

The second item is obvious and most people might be satisfied accepting that as the sole purpose of the 1-1. What’s less obvious is the value inherent in item one. Connecting with your manager is valuable for the following reasons: 1) you feel less like an island on your team; 2) you get reassurance that you belong; and 3) you learn where you stand with your boss and within your team.

If you’ve ever gone for an extended period of time without connecting with your manager you know how isolating it can be, and it’s this understanding that argues that 1-1s are for you, not your manager.

1-1s Are For You, Not Your Manager

Every manager has his or her own way of running 1-1s, and they’re different for each of their reports, and change over time. In spite of that, Andy Grove argues the 1-1 “should be regarded as the reports’s meeting, with its agenda and tone set by him.” (High Output Management) To that end, Grove continues, the manager “should facilitate the subordinate’s expression of what’s going on and what’s bothering him.” So even though a manager may have a particular way of running it, the 1-1 is still for you.

The fact that 1-1s are for you, but facilitated by your manager, doesn’t absolve you from your role and responsibilities therein. You still need to show up with questions and items to discuss, be able to provide answers to your manager’s questions, and have goals for what you want to accomplish in the meeting. Your manager will facilitate the 1-1, but they shouldn’t be the main driver.

However, there will be times when your manager drives and directs the 1-1. It may be that they have news or information they need to share with you, they need to have a performance discussion with you, they want to focus on your career growth, or they have something else they need to talk about. This type of 1-1s will generally be more infrequent, and you should still come prepared.

Lastly, consider what it would be like if you allowed your manager to always direct the 1-1. Do you really trust someone else to know what is best for you, to say what path you should take, or make decisions for you? But every time you show up to a 1-1 without preparation, it’s doing this very thing. The more frequently you abdicate your responsibility, the more control you give to someone else. The choices they make for you will be the ones that are easiest for them, not the ones that are best for you.

Come Prepared

By this point we should be agreed that 1-1s are about the team member, that they should be driving them, and to do so, the team member needs to come prepared. If that’s so, what do you need to do?

Come with items to discuss

This should be obvious, but make sure you come to your 1-1 with a few things to discuss with your manager. How many you bring will depend on the nature of the topics. I advise my teams to add items to their list as things come up through the week. Here are some potential topics to discuss:

• Status update: Limit this to just a few minutes. Most status updates occur during standups or as comments in your project management tool, and you waste valuable time with your manager telling them things they should already know from other meetings.
• Accomplishments: Share your accomplishments with your manager: Course completions, finishing projects, writing blog posts, giving conference talks, etc. are all things they should know about
• Problems: Inform your manager about problems (technical, interpersonal, etc) you run across. If possible, come with solutions, or at least show you’ve researched how to solve the problem. No manager wants yet another problem dropped in their lap.
• Ideas: If you have ideas to make things better for your project, team, company, or anything else, bring them to the 1-1 for discussion.
• Research topics: If you’ve been researching something to improve yourself of the team, a 1-1 is great time to share what you’ve uncovered.
• Team needs: Let your manager know about the needs of your team. New technologies, missing skill sets, tools, access, etc. are all things you should take to them.
• Goal check-in: Let your manager know about the progress you’ve made on your goals.
• Ask about your manager’s challenges: It’s okay to ask about what your manager needs. It shows that you’re thinking outside of yourself, and are looking to expand your responsibilities.
• Feedback: Ask for feedback, but be prepared to hear both positive and constructive feedback. Furthermore, only ask for it if you’re really looking to improve.

Be prepared to answer questions

Managers are going to ask questions: about you, your work, your aspirations, and more. You can also expect them to ask probing questions, trying to dig deeper on certain topics such as key projects or interpersonal relationships. If your manager is really good, they’ll use questions to help you gain a greater understanding about the business, your role, and yourself.

Your responsibility is to come prepared to answer these questions. You should be able to provide thorough answers about your work and career aspirations. There will also be questions which will blindside you, against which you really can’t prepare; for those, you just have to roll with the punches and do the best you can. In most cases, however, you should expect to be able to provide reasonable answers for all of your manager’s questions.

Show you’ve worked on action items

You should have at least one “action item” you take away from every 1-1; something you’ve learned or can apply. These action items aren’t always explicit, and could be something as simple as researching a paper or book your manager mentioned off-handedly. You should be able to show that you made progress on that action item the next time you meet with them.

Take notes

This shouldn’t need to be said, but I know it does: take notes. Organize them by date so your manager can corroborate with them. It won’t seem useful in the beginning, but eventually your notes will reach a critical mass and you’ll rely on them more and more to answer questions. Also, if your manager acts unbecomingly or if a sensitive topic is discussed, you’ll want to have your notes, especially if you’re the one in the hot seat.

If you don’t have 1-1s with your manager, or have only recently started them, don’t worry, you’re not alone. In an informal poll I conducted on LinkedIn, over 33% of the respondents said they’ve only started having 1-1s with their manager in the past two years, and I had a number of people lament there was no option for zero years. Whether you’ve had 1-1s for years or just started, there’s no time like the present to start working to make them better. On the other hand, if your manager doesn’t have regular 1-1s with you, there’s no reason you can’t schedule one with them.

Now that I’m in management, I have more 1-1s than ever. Not just with my team and director, but our VP, another engineering manager, the director of product, and others, and the frequencies varies. Some are to seek alignment, such as my 1-1 with the head of Product, others are for advice and council, while at least one is just to connect, but they’re all valuable and make me a better manager and leader.

Refactoring is the process of changing a software system in such a way that it does not alter the external behavior of the code yet improves its internal structure.

– Martin Fowler

Restructuring differs from refactoring in that it alters the external behavior. Argument types and arity, function names, classes, and modules can all be dramatically changed with a restructure. So can the return value.

I’m managing teams now, and aside from code reviews and the occasional one or two point story, I rarely get a chance to write code, let alone refactor anything. Nonetheless, I’m still captivated by refactoring; for the chance to improve processes and code.

I first stumbled on the idea of the team refactor meeting–as I most often do–after meeting with one of my team. In that meeting he made an offhanded remark about getting together with the team to consider which areas of the project needed the most work. Something about the way he said it gave me the idea for an ongoing meeting to make everything about the way we work better.

What is the Team Refactor?

Unlike refactoring code, the team refactor is a meeting used to review and rethink the way your team and project operates. It’s a time to address the pain points we experience–but often ignore–in our project, processes, and the code base. It gives us the chance to ask questions such as:

• Where are the problem areas in our code base?
• Where are the bottlenecks in the system?
• Which processes are we struggling with the most?
• What can we do to improve our communication?
• What would make the development experience better?
• Which process is the most difficult to understand?

It’s tempting to confuse this with a retrospective, but the two are altogether different. A retrospective focuses on the successes, failures, and lessons drawn from a sprint or feature release, whereas the team refactor takes a step back to look at the bigger picture, concentrating more on what can be done to make the team more effective.

How Does it Work?

The team refactor is a meeting, and should be held no more than once a month, and no less than once a quarter (we hold ours on the last Friday of every month). If it’s held weekly or bi-weekly, it’s difficult to see progress and you risk it devolving into a gripe session. Anything less than once a quarter and you’ll lose engagement. I recommend basing the frequency of the meeting on the amount of the team’s technical or organizational debt.

To make certain there are topics to discuss, regularly remind the team about it leading up to the meeting (such as in daily standups). It keeps the meeting “top of mind”, giving everyone time to watch for areas to improve. Make sure to keep a list of your own topics in case things stall out.

You can run the meeting any way you prefer. What I’ve done so far is break it up into three parts: Review, Propose, and Prioritize.

Review

During the “review” section of the meeting, your job is to remind the team what was covered in the last team refactor, highlighting which items the team completed. If no progress was made, own up to it; it’s better for morale if you don’t hide from it.

This part usually takes less than a third of the meeting.

Note: The “review” only works if you keep notes from one session to another.

Propose

The bulk of the meeting should be dedicated to sharing and talking about new ideas and topics. During this time, each team member should have something they’d like to see improved, with the team lead or manager offering ideas after everyone else or if no one is willing to open up.

Prioritize

Use the last segment of the meeting to prioritize what the team plans to do. Which of the ideas discussed this time and in prior meetings should be tackled, and in what order? This part of the meeting should take the least amount of time.

What Now?

Once the meeting concludes, it’s imperative you create stories to address the problems the team brought up. There’s no point in holding a team refactor unless the team can execute on those items. If the team isn’t allowed to act, you’ll deliver a gut punch to their morale. Don’t do that. Instead, make sure to schedule one or two items for work before the next team refactor. The stories don’t have to be completed, but they should at least be started.

So What?

Refactoring attracts developers, because it’s an opportunity to improve a piece of code; a way of proving we can solve the problem better. It’s the same thing for the team refactor. It’s coming together as a group, recognizing we can improve every area of our project, and then setting out to do it.

For the engineers, this means making the work environment–be it the system, processes, or codebase–more enjoyable to work in. They get to focus on solving problems instead of fighting the system or struggling to untangle spaghetti.

For the manager or the lead, it means seeing the team more engaged by proving that things can get better and are getting better. It also shows that you hear and understand them.

I’ve run team refactors on two different teams for a year, and I can honestly say it works. There may be griping in the first meeting or two about “yet another meeting,” but it settles down once everyone sees things improve. Once they see they’re the ones responsible for that change–that they’re the ones driving it–they only want more. That’s a good thing.