Provider Specification - Model class

Every provider must implement a Model class. Its primary job is to fetch data from a remote source like an API or database and return GeoJSON to Koop for further processing. This class is usually placed in a file called model.js and then referenced in the registration object. The follow snippet shows an example implementation:

// Example Model class
function Model () {}

Model.prototype.getData(request, callback) {

  // use the npm package `request` fetch geojson from Socrata API
  request('https://data.ct.gov/resource/y6p2-px98.geojson', (err, res, geojson) => {

    // if the http request fails, return and callback with error
    if (err) return callback(err)

    // Set metadata used by Koop
    geojson.metadata = { geometryType: 'Point' }

    callback(null, geojson)
  })
}

module.exports = Model

Method: getData(request, callback)

The getData method is a requirement of all providers. It will fetch data from a remote API and convert it to GeoJSON. It may also add a metadata property to the GeoJSON object and populate it appropriately. Using the callback argument, it should handle any errors, or pass the GeoJSON object as its second argument.

Arguments

Fetching data from the remote API

Route parameters

The details of how you fetch data from the remote API depends on how responsive you want the provider to be. The example in figure 1 is extremely simple, but also not very responsive; it will only fetch data for the resource at https://data.ct.gov/resource/y6p2-px98.geojson. Instead, we can use Koop’s route parameters, found in the request object, to build the URL to the remote API.

Model.prototype.getData(request, callback) {

  // Get 'host' and 'id' params from request object
  const { params: { host, id } } = request

  // Use 'host' and 'id' to build URL to remote API
  request(`https://${host}/resource/${id}.geojson`, (err, res, geojson) => {

    if (err) return callback(err)

    callback(null, geojson)
  })
}

In figure 2, we take the host and id parameters from the incoming Koop request, and use them to dynamically build the remote API request. Thus, parameters in the Koop request determine which resource is returned. Using the code example in figure 2, a Koop request like http://localhost:8080/koop-provider-socrata/my-host/my-id/FeatureServer/0/query would construct a remote API URL like https://my-host/resource/my-id.geojson.

Note that you need to configure the provider to use the host andid parameters using the hosts and disableIdParam settings, otherwise routes will be built without them and they will be undefined in getData. See provider registion docs for more details.

In addition to host and id, there may be other route parameters available in getData. It all depends on how the output-plugin defines its routes. For example, the Geoservices output defines the following route /FeatureServer/:layer/:method. On a request like http://localhost:8080/koop-provider-socrata/my-host/my-id/FeatureServer/0/query, getData could also access the layer and method parameters, which would have values of 0 and query, respectively.

Query parameters

Any query parameters that are part of the Koop request can be accessed in the getData function and may be useful for building the request to the remote API.

Model.prototype.getData(request, callback) {

  // Get 'host', 'id', 'where' params from request object
  const { params: { host, id }, query: { where } } = request

  // Use 'host' and 'id' to build URL to remote API
  request(`https://${host}/resource/${id}.geojson?where=${where}`, (err, res, geojson) => {

    if (err) return callback(err)

    geojson.filtersApplied = { where: true }

    callback(null, geojson)
  })
}

In figure 3, we attach the where query parameter from the Koop request to the remote URL. This allows the remote API to executing some of the data filtering, which reduces the amount of data coming over the wire and the amount of post-processing for Koop. Provider that pass on such instructions to remote API are termed pass-through providers.

If you do end up using query parameters to offload operations, you can add a filtersApplied object to your GeoJSON. The object should have a boolean flag for any Geoservices operation that has already been executed on the GeoJSON. See the example in figure 3, and the FeatureServer documentation for addition information on filtersApplied flags.

Translating fetched data to GeoJSON

In the code snippets above, the remote API returned GeoJSON, so no translation was necessary. Your remote data source may not offer GeoJSON as a format. In such a case, there must be a translation function to convert the fetch data to GeoJSON.

GeoJSON passed to the callback function should be valid with respect to the GeoJSON specification. Operations in some output-plugins may expect standard GeoJSON properties and / or values. In some cases, having data that conforms to the GeoJSON spec’s righthand rule is esstential for generating expected results (e.g., features crossing the antimeridian). Koop includes a GeoJSON validation that is suitable for non-production environments and is active when NODE_ENV is not production and KOOP_WARNINGS is not set to suppress. In this mode, invalid GeoJSON from getData will trigger informative console warnings.

Adding provider metadata to the GeoJSON

You should add a metadata object to the GeoJSON returned from getData (see Figure 1). Many Koop output plugins will expect metadata to be defined and use certain properties therein. For example, Koop’s default Geoservices output-plugin uses metadata.geometry to determine if the data is spatial or tabular. There are many other optional metadata settings (shown below) that may be useful for customizing your Geoservices output.

Special considerations for idField

As noted above idField designates which feature attribute should be used as the feature’s unique identifier. For ArcGIS clients, having a unique identifier is required, and it must be of type integer and in the range of 0 - 2,147,483,647, otherwise some functionality cannot be supported. If you chose not to set idField or don’t have a feature attribute that fits its requirements, Koop will create an OBJECTID field in its Geoservices output by default which can be used as a unique identifier. Its value is calculated as the numeric hash of each feature. While you can let Koop handle this, define an idField in your metadata is overall more reliable.

Method: getStream(request)

The optional getStream method should return a Readable-Stream of GeoJSON and is leveraged by any output-plugins that invoke the pullStream model method. Note that as of this writing few output-plugins leverage this method.

Arguments

Method: authorize(request)

The optional authorize method may be defined on a provider’s model class. This method will invoked before any invocations of getData or getStream. It overrides any authorize method of a registered authorization-plugin. This methods spec is identical to the authorize method of a authorization-plugin.

Method: authenticate(request)

The optional authenticate method may be defined on a provider’s model class. This method may be use by some output-plugins that have routes that allow token generation. It overrides any authenticate method of a registered authorization-plugin. This methods spec is identical to the authenticate method of a authorization-plugin.

Method: createKey(request)

Koop uses a an internal createKey function to generate a string for use as a key in the cache-plugin’s key-value store. Koop’s createKey uses the provider name and route parameters to define a key. This allows all requests with the same provider name and route parameters to leverage cached data.

Models can optionally implement a function called createKey. If defined, the Model’s createKey overrides Koop’s internal function. This can be useful if the cache key should be composed with additional parameters.

Arguments

In the example below, the createKey method uses the query parameter start to make a more specific cache key:

Model.prototype.createKey = function (request) {

  const { url, params: { host, id }, query: { start } } = request

  const keyFragments = []

  // Add provider name
  keyFragments.push(url.split('/')[1])

  if (host) keyFragments.push(host)

  if (id) keyFragments.push(id)

  if (start) keyFragments.push(start)

  return keyFragments.join('::')
}