Ember Octane: Pulling Out Data

This post is the third in a series on building an Ember application HTML-first. In this series, we're going to build the EmberConf schedule application from the ground up.

1. Let's Go
2. Components
3. Pulling Out Data ← This post
4. Airtable Time
5. Cleaning Things Up
6. Adding More Pages
7. Polishing: Server-Side Rendering, Prerendering and Code Splitting

In the last post, we created an Event component so that we don't have to repeat the HTML for talks in our schedule over and over.

But we still have to list out all of our talks directly inside of the HTML.

We can improve things by pulling the data out of the HTML and moving it into a separate file. One benefit of moving it into a separate file is that it makes it easy to maintain the website by just changing a data file.

1. The Data File

I've created a JSON file based on the original data actually used for the EmberConf 2020 website. Here's what an event looks like in that data.

{
  "id": "recQMxkGUjYnHqoDr",
  "created_at": "2016-10-30T21:17:58.000Z",
  "updated_keys": [],
  "fields": {
    "name": "Opening Keynote",
    "speakers": ["Yehuda Katz", "Jen Weber", "Godfrey Chan"],
    "description": "Join us for a State of the Ember Union by Ember Creator Yehuda Katz and Core Team Members Jen Weber and Godfrey Chan.",
    "day": ["Tuesday"],
    "end_time": "10:30am",
    "start_time": "9:30am"
  }
}

If you're not already familiar with JSON, it's a data format that has strings, numbers, true, false,the special value null, and collections of these. The official web page for JSON has a diagram of what's allowed inside of JSON. Don't worry if you can't figure it out. That's exactly how I felt the first time I saw it. Just follow along and you'll get the hang of it.

The whole document has a bunch of events nested under an events key.

{
  "events": [
    {
      "id": "recQMxkGUjYnHqoDr",
      "created_at": "2016-10-30T21:17:58.000Z",
      "updated_keys": [],
      "fields": {
        "name": "Opening Keynote",
        "speakers": ["Yehuda Katz", "Jen Weber", "Godfrey Chan"],
        "description": "Join us for a State of the Ember Union by Ember Creator Yehuda Katz and Core Team Members Jen Weber and Godfrey Chan.",
        "day": ["Tuesday"],
        "end_time": "10:30am",
        "start_time": "9:30am"
      }
    },
    {
      "id": "rec1P9lgPCyoAf30T",
      "created_at": "2016-10-30T21:17:58.000Z",
      "updated_keys": [],
      "fields": {
        "name": "FastFlood: The Story of a Massive Memory Leak in FastBoot Land",
        "speakers": ["Sergio Arbeo"],
        "description": "What would you do if you found a memory leak so big, that most of the data of your requests are leaking? What if everyone on your team was distributed? What if no one on the team shared a timezone?\n\nThis talk presents the typical techniques to find a memory leak and a few unusual ones when dealing with a significant leak.\n\nWe will also discuss how to organize a distributed team to find the leak faster.",
        "day": ["Tuesday"],
        "end_time": "11:15am",
        "start_time": "10:45am",
        "slides_url": "https://www.dropbox.com/s/jbdn54fc32n7qlf/FastFlood.pptx?dl=0",
        "transcript_url": "https://thisten.co/rr8x4"
      }
    },
    // ...
  ]
}

I posted the whole document as a gist. Your next step is to copy the contents of the gist into public/events.json in your application.

2. A JavaScript Pep Talk

We're about to start working with the first JavaScript code in this tutorial, which will allow us to dynamically fetch some data and use it in our page. Later, we'll also use JavaScript to add interactivity.

JavaScript may seem intimidating, but don't worry about understanding everything right away. The way I learned to code was to start with bits of code that I didn't fully understand, and then I'd tweak them a little bit to see what changed. Sometimes the tweak worked, and I learned something. Sometimes it didn't, and I learned what not to do.

Feel free to mess around as much as you want. You can always use the undo button to get back the original code, or copy it again from the tutorial.

Our goal right now is to do a little bit more with our application. By the time we're done with this step, we'll have made some progress, and we'll deploy the updated website on the Internet. Always be deploying.

With the pep talk out of the way, we're ready to write our first JavaScript. Buckle in!

3. Getting The Data Into Your App

To get the data into your application, you're going to write your first JavaScript.

So far, our application looks like this.

We have an application template, and the application template uses a bunch of Event components. Right now, that means we need to create a new <Event> tag for every event.

Instead, we're going to get the data dynamically. We're going to get an array of events from our JSON file and use that array in our template. But how do we get the data from a separate file?

To accomplish that, we need to learn about another piece of the Ember architecture: routes.

The route is responsible for getting the model and giving it to the template. A "model" is a fancy name for data that the template uses to create the final result, or render it.

If you're wondering about the name, we'll see in a later post that each page in your application can have its own "route".

Let's create a route for our application template by using the Ember route generator.

$ ember g route application
? Overwrite app\templates\application.hbs? No

Since we already have an application.hbs, there is no need to overwrite it. What we're looking for is the route that Ember generated in app/routes/application.js.

import Route from '@ember/routing/route';

export default class ApplicationRoute extends Route {
}

What you're looking at is a JavaScript module, which allows you to import code from other places and export your own code.

In this case, you're importing Ember's Route class, and exporting a special version of that class that we will use to tell Ember how to get the model for application.hbs.

If you're not familiar with object oriented programming or JavaScript classes, don't worry. At the moment, it's just a small amount of boilerplate that tells Ember what to do. As you learn more JavaScript, you'll see how import, export default, class and extends work.

To get events.json into our application, we'll need to fetch it.

import Route from "@ember/routing/route";
import fetch from "fetch";

export default class ApplicationRoute extends Route {
  async model() {
    let response = await fetch("/events.json");
    let data = await response.json();
    return data.events;
  }
}

Let's break down what we added here.

1. import fetch from "fetch". This brings the standard fetch library into your route module.
2. async model. This creates a new method called model inside of your route class. Ember expects to find a method named model in your route, which it will call to get the model for your template. The async prefix allows us to use await to wait for slow network requests inside this method.
3. await fetch("/events.json"). This tells JavaScript to fetch the events.json file, and then wait until it's downloaded. The fetch method is "asynchronous" (aka slow), so we need to await it.
4. await response.json() says that we want to interpret the events.json file as JSON, so that we're giving a JavaScript object to our template, and not a string. The process of converting a raw response into JSON is also "asynchronous", so we need to await it.

You might be thinking that all of this asynchronous stuff is a little weird, especially if you have some experience with Ruby or Python. The reason comes down to the fact that we're building user experiences with JavaScript, and we want to allow the user to continue scrolling and interacting with the page while we do slow things in the background. When we say await, we're telling JavaScript: "it's ok to let the user interact with the page and to continue painting the page now".

When writing a method like async model, I find it useful to use the built-in JavaScript debugger to play around with what I'm working with.

import Route from "@ember/routing/route";
import fetch from "fetch";

export default class ApplicationRoute extends Route {
  async model() {
+   debugger;
    let response = await fetch("/events.json");
    let data = await response.json();
    return data;
  }
}

Open up the developer tools in your browser and reload the page.

When you reload the page, you should see something like this.

The code in the developer tools is a little different from the code in your editor. This is because Ember automatically combines all of your JavaScript modules so that they will work in today's browsers. It also compiles new JavaScript features that are not yet implemented in today's browsers so that you can use them now.

We can step a few steps forward (until line 378), and then take a peek at the response object.

The response object represents the low-level details of fetching something over HTTP. That isn't really what we want, although it is interesting!

Let's take another step forward.

Cool! The data variable contains all of the data that we stuck in events.json.

4. Using The Data In Your Template

The data that we saw in the previous step is going to be available in application.hbs as @model. We can take a peek at @model is by using {{log}}.

<ul class="events">

+ {{log @model}}

  <Event @title="Opening Keynote" @start="9:30am" @end="10:30am"
    @speakers={{array "Yehuda Katz" "Jen Weber" "Godfrey Chan"}}
    @images={{array "yehuda-katz" "jen-weber" "godfrey-chan"}} />

  <Event @title="FastFlood: The Story of a Massive Memory Leak in FastBoot Land" @start="10:45am" @end="11:15am"
    @speakers={{array "Sergio Arbeo"}} @images={{array "sergio-arbeo"}} />

  <Event @title="Octane: A Paradigm shift in EmberJS" @start="11:30am" @end="12:00pm"
    @speakers={{array "Suchita Doshi"}} @images={{array "suchita-doshi"}} />

  <Event @title="Break" @start="12:00pm" @end="1:30pm" />

  <Event @title="Storytime: Georgia's Terrific Colorific Experiment" @start="12:00pm" @end="12:20pm" />

  <Event @title="Welcome Back!" @start="1:25pm" @end="1:30pm" />

  <Event @title="AST: the Secret Weapon to Transform a Codebase" @start="1:30pm" @end="2:00pm"
    @speakers={{array "Sophia Wang"}} @images={{array "sophia-wang"}} />

</ul>

Reload the page and take a peek at the developer console.

It looks like what we returned from the model method that we wrote earlier. There are a handful of properties that Ember added for its own bookkeeping, but you can ignore those.

We already learned how to use #each in the previous post. What we're going to do now is replace our hardcoded <Event> tags and replace them with a loop.

<ul class="events">

+ {{#each @model.events as |event|}}
+   <Event
+     @title={{event.fields.name}}
+     @start={{event.fields.start_time}}
+     @end={{event.fields.end_time}}
+     @speakers={{event.fields.speakers}}
+   />
+ {{/each}}

- <Event
-   @title="Opening Keynote"
-   @start="9:30am"
-   @end="10:30am"
-   @speakers={{array "Yehuda Katz" "Jen Weber" "Godfrey Chan"}}
-   @images={{array "yehuda-katz" "jen-weber" "godfrey-chan"}}
- />
-
- <Event
-   @title="FastFlood: The Story of a Massive Memory Leak in FastBoot Land" 
-   @start="10:45am"
-   @end="11:15am"
-   @speakers={{array "Sergio Arbeo"}}
-   @images={{array "sergio-arbeo"}}
- />
-
- <Event
-   @title="Octane: A Paradigm shift in EmberJS"
-   @start="11:30am"
-   @end="12:00pm"
-   @speakers={{array "Suchita Doshi"}}
-   @images={{array "suchita-doshi"}}
- />
-
- <Event
-   @title="Break"
-   @start="12:00pm"
-   @end="1:30pm"
- />
-
- <Event
-   @title="Storytime: Georgia's Terrific Colorific Experiment"
-   @start="12:00pm"
-   @end="12:20pm"
- />
-
- <Event
-   @title="Welcome Back!"
-   @start="1:25pm"
-   @end="1:30pm"
- />
-
- <Event
-   @title="AST: the Secret Weapon to Transform a Codebase"
-   @start="1:30pm"
-   @end="2:00pm"
-   @speakers={{array "Sophia Wang"}}
-   @images={{array "sophia-wang"}}
- />

</ul>

If we reload the page, we get:

Hm. We've lost our images.

We could add an array of images to the items in events.json, but

1. In real life, you can't always just change the API
2. If we look closely, we can see that the URL for the speaker images is just a small transformation of the speaker names.

5. Helpers

Let's take a look at our event.hbs component file again.

<li class="event">
  <div class="time">
    <p>{{@start}}</p>
    <p>{{@end}}</p>
  </div>
  <h1>{{@title}}</h1>
  <h2>
    <ul>
      {{#each @speakers as |speaker|}}
      <li>{{speaker}}</li>
      {{/each}}
    </ul>
  </h2>
  <ul class="images">
    {{#each @images as |image|}}
    <li>
      <img src="https://emberconf.com/assets/images/people/{{image}}.jpg">
    </li>
    {{/each}}
  </ul>
</li>

First, we loop over the speakers to stick the text inside of the lis, and then we loop over the images to create the correct URL for the img tag. But as we observed, we could get the image anchor by transforming the speaker's name.

In Ember, we use helpers in situations like this, which is basically a JavaScript function that you stick in a module. Here's how we would update our template to use an anchorize helper on our list of speakers.

<li class="event">
  <div class="time">
    <p>{{@start}}</p>
    <p>{{@end}}</p>
  </div>
  <h1>{{@title}}</h1>
  <h2>
    <ul>
      {{#each @speakers as |speaker|}}
      <li>{{speaker}}</li>
      {{/each}}
    </ul>
  </h2>
  <ul class="images">
-   {{#each @images as |image|}}
+   {{#each @speakers as |speaker|}}
    <li>
-     <img src="https://emberconf.com/assets/images/people/{{image}}.jpg">
+     <img src="https://emberconf.com/assets/images/people/{{anchorize speaker}}.jpg">
    </li>
    {{/each}}
  </ul>
</li>

Now we need to write the anchorize helper, which will take names like "Yehuda Katz" and turn them into "yehuda-katz".

Let's generate the helper.

$ ember g helper anchorize

This created app/helpers/anchorize.js.

import { helper } from '@ember/component/helper';

export default helper(function anchorize(params/*, hash*/) {
  return params;
});

There's a little bit of boilerplate here, but the main thing to pay attention to is the anchorize function. Helpers take their parameters as an array, so that means that params[0] (the first element of the params array) will be the speaker that you pass in to anchorize.

Let's use a debugger to take a peek.

import { helper } from "@ember/component/helper";

export default helper(function anchorize(params /*, hash*/) {
+ debugger;
  return params;
});

Reload the page, and here's what we get.

Ok, cool. We're going to get the parameter that we passed to anchorize in an array.

If we play around a bit, we can figure out how to convert our names into image anchors.

import { helper } from "@ember/component/helper";

export default helper(function anchorize(params /*, hash*/) {
- return params;
+ return params[0].toLowerCase().replace(" ", "-");
});

Reload the page, and we're very close, but not quite:

If we look closely, we'll see that only the first space character in "Samanta de Barros" was replaced with a -.

We could solve the problem with a regex, but there's an even easier solution.

import { helper } from "@ember/component/helper";

export default helper(function anchorize(params /*, hash*/) {
- return params[0].toLowerCase().replace(" ", "-");
+ return params[0].toLowerCase().split(" ").join("-");
});

Hm, we still didn't get the job done.

The two files we didn't match correctly are:

1. annegreeth-van-herwijnen.jpg
2. james-c-davis.jpg

In the first case, it looks like any dashes that were in the original name are removed in the final image name. In the second, we also don't want to include .s in the name.

No problem, we'll just remove them before we convert the " " into "-".

Looks great! Let's update our helper.

import { helper } from "@ember/component/helper";

export default helper(function anchorize(params /*, hash*/) {
- return params[0].toLowerCase().split(" ").join("-");
+ return params[0]
+   .toLowerCase()
+   .replace(".", "")
+   .replace("-", "")
+   .split(" ")
+   .join("-");
});

And it worked!

6. Tests

Ok, that was a pretty finicky thing to get right. To make sure we don't mess it up again in the future if we need to make more changes to it, let's write some tests.

The basic idea of testing is that we're going to write down what we expect to get for different inputs.

When we generated our helper, Ember automatically generated a test for us. Let's take a look at it.

import { module, test } from 'qunit';
import { setupRenderingTest } from 'ember-qunit';
import { render } from '@ember/test-helpers';
import { hbs } from 'ember-cli-htmlbars';

module('Integration | Helper | anchorize', function(hooks) {
  setupRenderingTest(hooks);

  // Replace this with your real tests.
  test('it renders', async function(assert) {
    this.set('inputValue', '1234');

    await render(hbs`{{anchorize inputValue}}`);

    assert.equal(this.element.textContent.trim(), '1234');
  });
});

There's some boilerplate here, but the important part is the test that starts on line 11.

There are three parts to the test:

1. Specifying the input value
2. Rendering the helper
3. checking that the output looks like what we expect

The easiest way to run our tests is to go to http://localhost:4200/tests in the browser.

Coincidentally, our tests pass, because after all of the replacement rules, 1234 is still 1234.

Let's write a test that we know will fail, to see what happens.

import { module, test } from 'qunit';
import { setupRenderingTest } from 'ember-qunit';
import { render } from '@ember/test-helpers';
import { hbs } from 'ember-cli-htmlbars';

module('Integration | Helper | anchorize', function(hooks) {
  setupRenderingTest(hooks);

- // Replace this with your real tests.
- test('it renders', async function(assert) {
-   this.set('inputValue', '1234');
-
-   await render(hbs`{{anchorize inputValue}}`);
-
-   assert.equal(this.element.textContent.trim(), '1234');
- });
+ test("it anchorizes two-part names", async function(assert) {
+   this.set("speaker", "Yehuda Katz");
+ 
+   await render(hbs`{{anchorize speaker}}`);
+ 
+   assert.equal(this.element.textContent, "yuda-katz");
+ });
});

When we reload the tests, we get a failing test, as we would expect.

Fix it up and the test will pass.

import { module, test } from "qunit";
import { setupRenderingTest } from "ember-qunit";
import { render } from "@ember/test-helpers";
import { hbs } from "ember-cli-htmlbars";

module("Integration | Helper | anchorize", function(hooks) {
  setupRenderingTest(hooks);

  test("it anchorizes two-part names", async function(assert) {
    this.set("speaker", "Yehuda Katz");

    await render(hbs`{{anchorize speaker}}`);
    
-   assert.equal(this.element.textContent, "yuda-katz");
+   assert.equal(this.element.textContent, "yehuda-katz");
  });
});

Let's write tests for the rest of our table of cases.

import { module, test } from "qunit";
import { setupRenderingTest } from "ember-qunit";
import { render } from "@ember/test-helpers";
import { hbs } from "ember-cli-htmlbars";

module("Integration | Helper | anchorize", function(hooks) {
  setupRenderingTest(hooks);

  test("two-part names", async function(assert) {
    this.set("speaker", "Yehuda Katz");

    await render(hbs`{{anchorize speaker}}`);

    assert.equal(this.element.textContent, "yehuda-katz");
  });

+ test("three-part names", async function(assert) {
+   this.set("speaker", "Samanta de Barros");
+
+    await render(hbs`{{anchorize speaker}}`);
+
+   assert.equal(this.element.textContent, "samanta-de-barros");
+ });
+
+ test("names with periods", async function(assert) {
+   this.set("speaker", "James C. Davis");
+
+   await render(hbs`{{anchorize speaker}}`);
+
+   assert.equal(this.element.textContent, "james-c-davis");
+ });
+
+ test("names with hyphens", async function(assert) {
+   this.set("speaker", "Anne-Greeth van Herwijnen");
+
+   await render(hbs`{{anchorize speaker}}`);
+
+   assert.equal(this.element.textContent, "annegreeth-van-herwijnen");
+ });
});

And it passes!

Just for good measure, let's break the anchorize method and see what happens to our tests. We'll remove the step that we added for "James C. Davis", which removed .s.

import { helper } from "@ember/component/helper";

export default helper(function anchorize(params /*, hash*/) {
  return params[0]
    .toLowerCase()
-   .replace(".", "")
    .replace("-", "")
    .split(" ")
    .join("-");
});

What do you think is going to happen to our tests?

If you said "the names with periods test will fail", you're right of course.

A good tip when you have a failing test is to throw a debugger into the test so you can take a look at all of the variables.

You sometimes hear people say that there's no point in writing tests, because it takes forever to write them and you can the job done through manual testing.

My feeling is that, at minimum, it's worth writing tests when you get tired of manual testing, especially for finicky code. When I'm awake and thinking clearly, I try to set myself up for success later on, when I'll be tired and cranky.

7. Deploy Again

When your tests are passing, you're ready to deploy your app to production.

Just like in the first post, you can either run ember build -prod and drag the dist folder to Netlify, or you can set up Netlify to deploy automatically every time you push to Github.

If you didn't get Netlify hooked up to Github yet, first deploy using the drag-and-drop method. After that, now would be a good time to work on getting set up to automatically deploy. It's much nicer!

8. Take a Deep Breath

Look at what you got done today!

We started with an application with no JavaScript, and by the end of the post, we dynamically fetched the data instead of hardcoding it, used a small JavaScript function to avoid unnecessary repetition, and wrote some tests so we wouldn't have to keep checking our code by hand.

And along the way, we learned how to use the JavaScript debugger to poke around at the code and data as we figure out what to write.

Savor the moment! You did great!

Before moving on to the next post, you might want to tinker with the code we wrote. Try changing the data and see how it affects the code. Try moving code around and see if it still works. If you're really ambitious, you can even try to make more helpers!

In the next post, we'll move the data to Airtable so you can update your schedule without needing to re-deploy your app.