Blog

- August 12, 2015

For the Summer of APIs hackathon, following up my first post on visualizing Open Data sets and on storing an Angular.JS application in APISpark, I’m now focusing on explaining how to define the various data layers we’ll display later on thanks to D3.js.

When browsing the suggested list of open datasets, I was curious to see what NASA provides. Of course, there are a lot of things about space, but we can also find out some data about earth itself, specially about meteorites hitting our planet! It should be fun to see on a map if some meteorites landed around and to play with such data, right?

All along the post, we will show how a Web API (and APISpark) can be useful to build the map and its components. We dont focus on the structure of the application itself yet. But don’t worry, it will be the subject of a following post!

For a better understanding, here is the big picture of the application.

00-overview

Let’s start with the map itself.

Preparing the map

As you can see on the previous figure, there are two dedicated resources for maps:

  • One to list and create them
  • Another one to manage them (update, delete…)

To create our map, we will leverage the first one with the following content:

POST /maps/
Content-Type: application/json
{
    "id": "1",
    "name": "World",
    "type: "d3js",
    "projection": "orthographic",
    "scale": 250,
    "interactions": {
        "moving": "mouseMove",
        "zooming": "mouseWheel"
    }
}

This content gives some hints about the map we want to create like its projection, its initial scale and the way to interact wth it. We choose here to enable both zooming (with the mouse wheel) and moving (with drag’n drop).

Although we have our map created, nothing appears since there is no content in it yet. Within our application, content corresponds to layers that can be map data like continents boundaries and raw data like rates or geographical positions (longitude, latitude).

We will now create all these layers.

Defining background layers

The orthographic projection is really great since it allows to see earth as a globe. To make this more beautiful, we will add some blue background for oceans and draw the shapes of continents.

To visualize the globe, we need first to create a graticule layer — a graticule is a grid of horizontal and vertical lines. This is the term for this in D3.js, the underlying library we use.

Like for maps, we have similar resources for map layers. We have separate resources since layers can be used against several maps.

To create our layer, we will leverage the POST method of the list resource with the following content. We specify in some hints to configure what we want to display (background, lines and globe border) and some styles to define colors and line properties. Don’t forget to set the identifier of the map as reference.

POST /layers/
Content-Type: application/json
{
    "id": "graticuleLayer",
    "type": "graticule",
    "name": "Graticule",
    "applyOn": "layers",
    "applied": true,
    "visible": true,
    "maps": [
        "067e6162-3b6f-4ae2-a171-2470b63dff00"
    ],
    "display": {
        "background": true,
        "lines": true,
        "border": true
    },
    "styles": {
        "background": {
            "fill": "#a4bac7"
        },
        "border": {
            "stroke": "#000",
            "strokeWidth": "3px"
        },
        "lines": {
            "stroke": "#777",
            "strokeWidth": ".5px",
            "strokeOpacity": ".5"
        }
    }
}

Now, with this first layer created, our map looks like this:

01-layer-graticule-s

Looks promising but there are still layers to define! The next step is to configure shapes for continents. D3.js comes with a lot of samples of maps and especially, one for continents. The corresponding file provides data defined with the TopoJSON format.

Based on this file (that we imported within a folder of our Web API), we can create a new layer for GeoData. This layer is quite simple since we only need to reference the file and tell the root object within the file we want to use (here countries).

POST /layers/
Content-Type: application/json
{
    "id": "worldLayer",
    "type": "geodata",
    "name": "World",
    "applyOn": "layers",
    "applied": true,
    "visible": true,
    "maps": [
        "067e6162-3b6f-4ae2-a171-2470b63dff00"
    ],
    "data": {
        "url": "http://mapapi.apispark-dev.com:8182/files/continent.json",
        "rootObject": "countries",
        "type": "topojson"
    }
}

With this second layer created, we can see continents displayed on our map:

02-layer-continents-s

We have the foundations of our map created. Let’s now dive into our dataset.

Defining the layer for meteorites

We downloaded the file from the NASA web site. The file contains 34 513 meteorites and its structure is provided below:

  • name: the place where the meteorite fell
  • id: the identifier
  • nametype: is for most meteorites and are for objects that were once meteorites but are now highly altered by weathering on Earth
  • recclass: the type of the meteorite
  • mass: the mass of the meteorite (in grammes)
  • year: the year of fall
  • reclat: the latitude
  • reclong: the longiturde
  • GeoLocation: the complete geo location (latitude and longitude)

For simplicity sake, we directly uploaded the file of the dataset within a folder of our Web API with the path /data (for example). We could go further and create a dedicated entity to store such data and then upload them using the bulk import of the entity browser. This will the subject of another post.

In attribute values, expressions can be provided. In this context, d corresponds to the current data of elements and contains all hints regarding this current row.

The first section of the layer configuration consists of data. We specify here the data file to load for this layer and some hints to filter and sort data. As a matter of fact, since there is an important gap within mass of meteorites, we need to display subset of data to make the map readable.

POST /layers/
(Content-Type: application/json)
{
    "id": "meteorites",
    "type": "data",
    "mode": "objects",
    "name": "Meteorites",
    "maps": [
        "067e6162-3b6f-4ae2-a171-2470b63dff00"
    ],
    "data": {
        "url": "http://mapapi.apispark-dev.com:8182/files/Meteorite_Landings.csv",
        "type": "csv",
        "source": "meteoritesSource",
        "where": "d.mass > 10000",
        "order": {
            "field": "mass",
            "ascending": false
        }
    },
    (...)
    "applyOn": "layers",
    "applied": true,
    "visible": true
}

The second section is about what we want to display and how. We want to draw a circle for each meteorite at the place where it fell. The radius of the circle is proportional with its mass and the background color dependent on the year of landing. The more the date is close from now, the more the color is close from red.

Color transition is handled by the threshold feature of D3 configured using a palette. Some beautiful palettes are provided by the link: http://bl.ocks.org/mbostock/5577023.

Following snippet provides the complete configuration of this shape:

{
    "id": "meteorites",
    "type": "data",
    "mode": "objects",
    "name": "Meteorites",
    "maps": [
        "067e6162-3b6f-4ae2-a171-2470b63dff00"
    ],
    (...)
    "display": {
        "shape": {
            "type": "circle",
            "radius": "d.mass / 5000000",
            "origin": "[ d.reclong, d.reclat ]",
            "opacity": 0.75,
            "threshold": {
                "code": "YlOrRd",
                "values": [ 1800, 1900, 1950, 2000, 2015 ],
                "colors": [ "#ffffb2", "#fed976", "#feb24c",
                                   "#fd8d3c", "#f03b20", "#bd0026" ]
            },
            "value": "parseDate(d.year).getFullYear()"
        }
    },
    (...)
    "applyOn": "layers",
    "applied": true,
    "visible": true
}

Now this third layer created. We can see where big meteorites fell on our map.

03-meteorites-africa-asia-all-s

We can add more hints about the displayed data like the color legend and tooltip providing the complete data for the meteorite landing. To add this to the map, we simply need to add legend and tooltip sections, as described below. In this case, we need to leverage the PUT method of the single resource with the layer content. We can notice that all the content of the layer needs to be specified as payload of the request.

PUT /layers/meteorites
(Content-Type: application/json)
{
    "id": "meteorites",
    "type": "data",
    "mode": "objects",
    "name": "Meteorites",
    "maps": [
        "067e6162-3b6f-4ae2-a171-2470b63dff00"
    ],
    (...)
    "display": {
        (...)
        "legend": {
            "enabled": true,
            "label": "d"
         },
         "tooltip": {
            "enabled": true,
            "text": "'Name: '+d.name+'<br/>Year: '+d.year+'<br/>Mass (g): '+d.mass"
        }
    }
    (...)
}

To finish, let’s configure the interaction with the layer. It simply corresponds to specifying that we want to display the tooltip area when clicking on circles.

{
    "id": "meteorites",
    "type": "data",
    "mode": "objects",
    "name": "Meteorites",
    (...)
    "behavior": {
        "tooltip": {
            "display": "click"
        }
    },
    (...)
}

Let’s have another look at our map.

04-meteorites-america-all-s

The big advantage of such approach is that we can easily configure the data to display and the way to display them to be relevant. We will show now how to play with subsets of data.

Playing with data

As we can see, there is an important gap between masses of meteorites. In previous sections, we mainly displayed the big ones. Restricting the data to display small ones allows to show a different representation of meteorites data.

We will display here meteorites that have a mass inferior to 50 kg. We simply need to update the attribute where in the section data section. In this case, we need to increase a bit the radius of displayed circles.

Below are the two updated sections within the map definition:

{
    (...)
    "data": {
        "url": "http://mapapi.apispark-dev.com:8182/files/Meteorite_Landings.csv",
        "type": "csv",
        "where": "d.mass < 50000",
        "order": {
            "field": "mass",
            "ascending": false
        }
    },
    "display": {
        "shape": {
            "type": "circle",
            "radius": "d.mass / 50000",
            "origin": "[ d.reclong, d.reclat ]",
            "opacity": 0.75,
            "threshold": {
                "code": "YlOrRd",
                "values": [ 1800, 1900, 1950, 2000, 2015 ],
                "colors": [ "#ffffb2", "#fed976", "#feb24c",
                                  "#fd8d3c", "#f03b20", "#bd0026" ]
            },
            "value": "parseDate(d.year).getFullYear()"
        },
        (...)
    },
    (...)
}

By reloading data, we have now the following maps:

05-meteorites-america-small-s
06-meteorites-asia-australia-small-s
08-meteorites-europa-africa-small-tooltip-s

In this case, the map is less reactive since there are much more data to display. Some optimizations would be interesting like only applying data on the map for the displayed area.

In a future post, we will describe the APISpark Web API itself, its structure and the way to interact with it.

Blog post originally posted on Thierry’s blog.