The import procedure is considered an advanced technical topic. Our existing query and mutation APIs have worked well for the vast majority of users to date, but there are edge cases where a proper import/export specification is still helpful.

Importing

The importing process requires your data to be normalized into the Normalized Data Format(NDF). Currently this is a manual process that needs to happen before any further step is accomplished. Future updates will add first-class citizen status to SQL, MongoDB and JSON. Read more about NDF format below.

NDF files need to be separated into three separate categories of files:

  • nodes: The individual database records
  • lists: Special list types for MySQL compatability
  • relations: Joiner tables

You can upload an unlimited number these files with a soft cap of 1mb per file. Anything more than 1mb could lead to timeout issues.

Important Notes

  • Node id's can be a max of 25 characters in length.
  • Imports are not idempotent. They will always add new data.
  • There is no data validation. Uploading bad data could break your project.
  • Because the import is not idempotent, and bad data can break your service, importing the same ID twice could break your app.

We suggest you inspect a data export from your project to see what kinds of data your project expects.

Import API

The import API can be found at the /import path of your project endpoint. For example:

  • https://api-euwest.graphcms.com/v1/cjsenhej6288g01bs36jkf0jw/master/import

Each request is capped at a 10mb payload and must be in NDF format. Each request needs to be authenticated with Bearer Auth using a systems token. Read more about System Tokens below.

Example: Uploading NDF nodes with curl :

curl 'YOUR_PROJECT_ENDPOINT/import' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_SYSTEM_TOKEN' \
-d '{"valueType":"nodes","values": YOUR_JSON_NODE_DATA}' \
-sSv

Exporting

You can export data of NDF shape in the JSON format by using the export path at the end of your project API endpoint.

Export API

The export API can be found at the /export path of your project endpoint. For example:

  • https://api-euwest.graphcms.com/v1/cjsenhej6288g01bs36jkf0jw/master/export

Each request is capped at a 10mb response and will be in the NDF shape. Each request needs to be authenticated with Bearer Auth using a systems token. Read more about System Tokens below.

The endpoint expects a POST request with a JSON encoded body of the following values:

{
  "fileType": "nodes",
  "cursor": {
    "table": 0,
    "row": 0,
    "field": 0,
    "array": 0
  }
}

The values in the cursor object refer to your data offsets, the response object includes these sames values with either:

  • Terminated (not full): If all the values for table, row, field and array are returned as -1 it means the export has completed.
  • Non-terminated (_full): If any of the values for table, row, field or array is different from -1, it means the maximum size of 10 MB for this response has been reached. If this happens, you can use the returned cursor values as the input for your next export request.

Here is an example curl command for uploading some JSON data (of NDF type nodes):

curl 'YOUR_PROJECT_ENDPOINT/export' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer YOUR_SYSTEM_TOKEN' \
-d '{"fileType":"nodes","cursor":{"table":0,"row":0,"field":0,"array":0}}' \
-sSv

Normalized Data Format

Normalized Data Format is a transport friendly data format that ensures expectations on the shape of your data. By adhering to a common format, we can import the widest amount of data possible. At the moment, transforming content into this format is a strictly manual process, but specialized importers are being worked on to support the most common data formats such as SQL, MySQL, MongoDB, etc.

NDF Value Types

NDF is composed of three different "Value Types":

  • Nodes: Contains data for the scalar fields of nodes
  • Lists: Contains data for list fields of nodes
  • Relations: Contains data to connect two nodes via a relation by their relation fields

Structure

A JSON document in the NDF shape has the following keys:

  • valueType: This can be one of "nodes", "lists" or "relations"
  • values: When importing, and adhering to an array of objects in the shape of their corresponding NDF type.

For the following examples, we will assume this data model:

type Hotel {
  id: String! @unique
  name: String!
  city: String!
  amenities: [String!]!
  overflow: Hotel
}

Nodes

When the valueType given is "nodes", the shape of values should resemble:

{
  "valueType": "nodes",
  "values": [
    { "_typeName": STRING, "id": STRING, "field": fieldType, "field": fieldType, ..., "field": fieldType },
    ...
  ]
}

Specifically, the _typeName and id must be String.

Importing two "Hotel" nodes would look like the following:

{
  "valueType": "nodes",
  "values": [
    {
      "_typeName": "Hotel",
      "id": "hilton1",
      "name": "Hilton",
      "city": "Orlando"
    },
    {
      "_typeName": "Hotel",
      "id": "westin1",
      "name": "Westin",
      "city": "Orlando"
    }
  ]
}

Lists

When the valueType is "lists", the array of value object should have the following shape:

{
  "valueType": "lists",
  "values": [
    { "_typeName": STRING, "id": STRING, "field": [fieldType] },
    ...
  ]
}

Specifically, the _typeName and id must be String.

Importing the hotels list values would look like the following:

{
  "valueType": "lists",
  "values": [
    {
      "_typeName": "Hotel",
      "id": "hilton1",
      "amenities": ["Wi-Fi", "Breakfast"]
    },
    {
      "_typeName": "Hotel",
      "id": "westin1",
      "amenities": ["Wi-Fi", "Breakfast"]
    }
  ]
}

Relations

When the valueType is "relations", the array of value object should have the following shape:

{
  "valueType": "relations",
  "values": [
    [
      { "_typeName": STRING, "id": STRING, "fieldName": STRING },
      { "_typeName": STRING, "id": STRING, "fieldName": STRING }
    ],
    ...
  ]
}

Specifically, the _typeName, id and fieldName must be String.

The _typeName is the name of an SDL type from your data model. The fieldName is the name of the of the relation field on your model. Since we are connecting two nodes, the array of values is comprised of arrays, with each of them containing exactly two objects, the objects we are linking together.

Linking our hotels together would look like the following:

{
  "valueType": "relations",
  "values": [
    [
      { "_typeName": "Hotel", "id": "hilton1", "fieldName": "overflow" },
      { "_typeName": "Hotel", "id": "westin1", "fieldName": "overflow" }
    ]
  ]
}

Getting the System Token

System Tokens can be generated from the project settings tab, near where the Permanent Auth Tokens also live. The Tokens can be scoped to one of either:

  • CONTENT_EXPORT: for authenticating export of data
  • CONTENT_IMPORT: for authenticating import of data

System Tokens