Sync datacontent to Algolia with webhooks

In this guide we will explore how to sync specific GraphCMS data models to Algolia using a Zeit Now serverless function. ๐Ÿš€

To get started, create accounts at GraphCMS, Algolia & Zeit. They all offer incredibly generous free tiers.

Your GraphCMS data model

Once inside the GraphCMS Dashboard, you'll be presented the option to create a new project, go ahead and create a new project "From Scratch" and give it a name (I'll use GraphEvents throughout the tutorial for my application names.), pick the Basic plan & click Start in trial. The trial gives you access to the paid webhooks feature for a few weeks.

Next, inside your project we'll go ahead and complete our onboarding checklist. Select 'Create a model'.

Create Field with GraphCMS

First give the model a name, in this case we'll call it Event. Once we have the model created, we can go ahead and add our fields.

We need the following fields;

  • Title: Single line text, required
  • Description: Multi line text, required
  • Starts: Date and Time, required
  • Ends: Date and Time, required
  • Location: Map, required

You can use the GraphCMS UI to create these by clicking "Fields" from the right drawer and dragging them onto your data model.

In the above, GraphCMS is defining our data model under the hood when we save each fields. We'll later explore this inside the API Explorer to see our newly defined schema.

Repeat the above step for all fields, not forgetting to set them as required.

Create some content

Now we've completed the above model, we can begin to add some example entries.

Create content with GraphCMS

Make sure to set the status to PUBLISHED before saving.

Add as many example events as you like!

Query your data

Now we can use the API Explorer to query our data and explore the schema we created above.

Inside the GraphCMS, select the API Explorer from the navigation and once inside, this will look familiar if you've used GraphQL before.

Go ahead and run the following query:

{
  events {
    id
    title
    description
    starts
    ends
    location {
      latitude
      longitude
    }
  }
}

You'll get the data back from GraphCMS on the right, if you created more than one Event, this will appear inside response array events.

GraphCMS API Explorer

Next, if you open the Docs inside the API Explorer, you'll be able to search for Event and see the schema we created above.

API Explorer Docs

Setup Algolia

Once inside the Algolia Dashboard, create a new Application, giving it a name and selecting the free Community plan and selecting a region to get started. Make sure to select the same or closest region to your GraphCMS region.

Algolia create application modal

Next go ahead and create your first Index and call it Events. We'll not worry about prefixing it with development or production, but you'll want to if you're running this in multiple stages.

Algolia create index modal

Now our Index Events has been created, we'll head to the API Keys section and grab a copy of the Application ID and Admin API Key.

Setup dev tools

Next, you'll want to go ahead and install the Zeit Now:

npm install -g now

It's important important we install ngrok too, so we can expose localhost to the outside world (in our case, GraphCMS).

If you have Homebrew installed, you can do this via the command line too, otherwise you can download from their website.

brew cask install ngrok

Setup initial webhook

Now we've the tools in place to create our webhook, let's go ahead and create a directory for our project and install the Algolia NPM module.

mkdir graphcms-algolia-webhook
cd graphcms-algolia-webhook
yarn add algoliasearch # npm install algoliasearch

Now open the newly created directory inside your favourite code editor. If you're using Visual Studio Code, simply type code . to open the current directory.

You should see two files package.json and yarn.lock (or package-lock.json if you used NPM above).

Go ahead and create two new files:

index.js

module.exports = async (req, res) => {
  res.status(200).json(req.body)
}

now.json

{
  "version": 2,
  "builds": [{ "src": "index.js", "use": "@now/node" }],
  "routes": [{ "src": "/", "dest": "/index.js" }]
}

All we're doing above is creating a basic function that returns the body of the request that it receives and configuring Zeit Now so it knows how to execute and bundle our code.

Now we should be able to run now dev via the command line and get back a URL for our local now environment. This will by default be http://localhost:3000.

Next, let's start ngrok to expose the URL:

ngrok http 3000

Once ngrok is running, open http://localhost:4040 and make sure the "Inspect" tab is selected.

Now copy the https address (it should look something like https://744b9b48.ngrok.io) and head over to the GraphCMS Dashboard.

Head to the "Webhooks" section from the sidebar, and paste the ngrok URL as the webhook URL. Give the webhook a name, description and make sure to set the scope to EXTENDED.

Create a GraphCMS webhook modal

Now once saved, head to the content area and edit an Event you created earlier, change any of the fields and click Save.

Now back at http://localhost:4040 you should see a new request containing the webhook payload, it'll look a little something like this.

We'll use the info.args.operation and info.args.responseData payloads inside our serverless function to sync to Algolia. Let's do that next.

Sync data to Algolia

Before we go ahead and start coding our serverless function, it's important we take a look at the data structure for Algolia.

Algolia recommend you structure your data in a certain when indexing, if you expand on our data model above and add relations such as categories, Algolia would expect you to send an array of category names for faster filtering.

We'll be tracking our indexes by the Event id. Algolia want you to reference each record by a unique ID named objectID.

If we later wanted to make use of Algolia's geographical filtering and ranking, such as searching around a radius or polygonal area, it's important we explicitly tell Algolia the latitude/longitude under the __geoloc key.

This means we need to transform our GraphCMS responseData payload into an object that matches the below:

{
  "objectID":"cjyy2u004fftk0d53t777tyss",
  "title":"GraphQL Day Bodensee",
  "_geoloc":{
    "lat":47.66349785139189,
    "lng":9.174649342327939
  },
  "starts":"2019-09-06T07:30:00.000Z",
  "ends":"2019-09-06T17:30:00.000Z",
  "description":"GraphQL Day Bodensee is a hands-on one day developer conference and workshop for lovers of GraphQL, organised by GraphCMS and Honeypot. It's โ€Ša mini-conference with awesome speakers at the Kulturzentrum am Mรผnster in Konstanz on September 6."
}

Now we know what the object must look like when sending it to Algolia, let's next write the code to actually make it happen! ๐Ÿง™โ€โ™€๏ธ

Now this is where the real fun begins. We'll first want to import the algoliasearch library we installed previously and initialize the library with our credentials.

const algoliasearch = require('algoliasearch')

const ALGOLIA_APP_ID = '...'
const ALGOLIA_API_KEY = '...'

const INDEX_NAME = 'Event'

const createObserver = 'createEvent'
const updateObserver = 'updateEvent'
const deleteObserver = 'deleteEvent'

const algolia = algoliasearch(ALGOLIA_APP_ID, ALGOLIA_API_KEY)

Homework: You'll want to encrypt your ENV variables with now secrets. See Zeit docs on how-to.

Next inside our exported async function we want to use destructuring assignment to assign fieldName and responseData from the info object sent in the req.body payload:

const { info: { fieldName, responseData } } = req.body

Zeit has some handy helpers for parsing JSON, so if the request receives JSON, it can handle it accordingly and make it available via req.body.

The outcome of the serverless function is to:

  1. Add a new record to the Index if the fieldName is createEvent.
  2. Update an existing record in the Index if the fieldName is updateEvent.
  3. Delete a record if it exists and the fieldName is deleteEvent.

Because GraphCMS sends webhooks for EVERY type of mutation, we need to stop the serverless function from running if the event is NOT related to the creation, updating or deleting of an Event.

We can do that by returning a 422 response from the handler.

if (![createObserver, updateObserver, > deleteObserver].includes(fieldName)) {
  return res.status(422)
}

Next we'll go ahead and initialize our index 'Events' as index.

try {
  const index = algolia.initIndex(INDEX_NAME)

  let body

  res.status(200).send(body)
} catch (err) {
  res.status(500).json({ errors: [err.message] })
}

We also want to quickly execute and respond if the fieldName is to deleteEvent. We can do this by executing the deleteObject method on the Algolia index.

// ...

let body

if (fieldName === deleteObserver) {
  body = await index.deleteObject(responseData.id)
  return res.status(202).json(body)
}

// ...

Now all that's left to do is assign the fields we need from responseData to sync with Algolia, create the new object with the location lat/lng fields and call the applicable method on the index instance to add or update the indexes.

const {
  id: objectID,
  location: { latitude: lat, longitude: lng },
  ...rest
} = responseData

const indexable = { objectID, _geoloc: { lat, lng }, ...rest }

if (fieldName === createObserver) {
  body = await index.addObject(indexable)
} else if (fieldName === updateObserver) {
  body = await index.saveObject(indexable)
} else {
  throw new Error(`${fieldName} could not be processed.`)
}

The final code

const algoliasearch = require('algoliasearch')

const ALGOLIA_APP_ID = '...'
const ALGOLIA_API_KEY = '...'

const INDEX_NAME = 'Events'

const algolia = algoliasearch(ALGOLIA_APP_ID, ALGOLIA_API_KEY)

const createObserver = 'createEvent'
const updateObserver = 'updateEvent'
const deleteObserver = 'deleteEvent'

module.exports = async (req, res) => {
  const {
    info: { fieldName, responseData },
  } = req.body

  if (![createObserver, updateObserver, deleteObserver].includes(fieldName)) {
    return res.status(422)
  }

  try {
    const index = algolia.initIndex(INDEX_NAME)

    let body

    if (fieldName === deleteObserver) {
      body = await index.deleteObject(responseData.id)
      return res.status(202).json(body)
    }

    const {
      id: objectID,
      location: { latitude: lat, longitude: lng },
      ...rest
    } = responseData

    const indexable = { objectID, _geoloc: { lat, lng }, ...rest }

    if (fieldName === createObserver) {
      body = await index.addObject(indexable)
    } else if (fieldName === updateObserver) {
      body = await index.saveObject(indexable)
    } else {
      throw new Error(`${fieldName} could not be processed.`)
    }

    res.status(200).send(body)
  } catch (err) {
    res.status(500).json({ errors: [err.message] })
  }
}

That's it! ๐ŸŽ‰

In a further tutorial we will implement the Algolia InstantSearch widget library so we can search, filter our events and deploy it to Zeit Now. Stay tuned! โšก๏ธ