Skip to main content

Transformation using JQ

A system task that allows the processing of JSON data supplied to the task by using the popular JQ processing tool’s query expression language.

Definitions​

  {
     "name": "json_transform",
     "taskReferenceName": "json_transform_ref",
     "type": "JSON_JQ_TRANSFORM",
     "inputParameters": {
       "persons": [
         {
           "name": "some",
           "last": "name",
           "email": "mail@mail.com",
           "id": 1
         },
         {
           "name": "some2",
           "last": "name2",
           "email": "mail2@mail.com",
           "id": 2
         }
       ],
       "queryExpression": ".persons | map({user:{email,id}})"
     }
   }

Input Parameters​

AttributeDescription
inputParameters (Referred to as Script params in UI)JSON object defining configuration data for task execution.
queryExpressionA string that represents a JQ (JSON Query) expression. This expression will be used to transform JSON data.
optionalEnabling this option renders the task optional. The workflow continues unaffected by the task's outcome, whether it fails or remains incomplete.

Output Parameters​

AttributeDescription
resultThe result attribute stores the first element of the resultList.
resultListList of results returned by the JQ expression.
errorOptional error message if the JQ query fails.

Examples​



  1. Add task type JSON JQ TRANSFORM.
  2. Configure the JQ script and provide the input parameters.

Adding wait task

Sample Workflow

To demonstrate how this task operates, consider the following workflow definition:

    {
      "name": "jq_example_task",
      "taskReferenceName": "my_jq_example_task_ref",
      "type": "JSON_JQ_TRANSFORM",
      "inputParameters": {
        "key1": {
          "value1": [
            "a",
            "b"
          ]
        },
        "key2": {
          "value2": [
            "c",
            "d"
          ]
        },
        "queryExpression": "{ key3: (.key1.value1 + .key2.value2) }"
      }
    }

The inputParameters attribute contains:

  1. Key-value pairs key1/value1 and key2/value2, which are arbitrary names in this context.
  2. The queryExpression key holds a JQ expression that operates on these parameters. In this example:
  • key1 and key2 are objects containing value1 (["a", "b"]) and value2 (["c", "d"]) respectively.
  • The JQ expression { key3: (.key1.value1 + .key2.value2) } concatenates these arrays into a single array under key3.

Upon executing the above task, the output is structured as follows:

    {
      "result": {
        "key3": [
          "a",
          "b",
          "c",
          "d"
        ]
      },
      "resultList": [
        {
          "key3": [
            "a",
            "b",
            "c",
            "d"
          ]
        }
      ]
    }
  • result: Contains the result of queryExpression, where key3 holds the concatenated array ["a", "b", "c", "d"].
  • resultList: Stores all results generated by queryExpression, with a single entry similar to result.

Cleaning up a JSON response

An HTTP task initiates an API call to GitHub to retrieve a list of "stargazers" (users who have starred a repository). The API response snippet (for a single user) is as follows:

The snippet of ${hundred_stargazers_ref.output}:

    [
      {
        "starred_at": "2016-12-14T19:55:46Z",
        "user": {
          "login": "lzehrung",
          "id": 924226,
          "node_id": "MDQ6VXNlcjkyNDIyNg==",
          "avatar_url": "https://avatars.githubusercontent.com/u/924226?v=4",
          "gravatar_id": "",
          "url": "https://api.github.com/users/lzehrung",
          "html_url": "https://github.com/lzehrung",
          "followers_url": "https://api.github.com/users/lzehrung/followers",
          "following_url": "https://api.github.com/users/lzehrung/following{/other_user}",
          "gists_url": "https://api.github.com/users/lzehrung/gists{/gist_id}",
          "starred_url": "https://api.github.com/users/lzehrung/starred{/owner}{/repo}",
          "subscriptions_url": "https://api.github.com/users/lzehrung/subscriptions",
          "organizations_url": "https://api.github.com/users/lzehrung/orgs",
          "repos_url": "https://api.github.com/users/lzehrung/repos",
          "events_url": "https://api.github.com/users/lzehrung/events{/privacy}",
          "received_events_url": "https://api.github.com/users/lzehrung/received_events",
          "type": "User",
          "site_admin": false
        }
      }
    ]

We are interested only in the starred_at and login parameters for users who starred the repository after a specified date (${workflow.input.cutoff_date}). We will use JQ Transform to simplify the output:

    {
      "name": "jq_cleanup_stars",
      "taskReferenceName": "jq_cleanup_stars_ref",
      "inputParameters": {
        "starlist": "${hundred_stargazers_ref.output.response.body}",
        "queryExpression": "[.starlist[] | select (.starred_at > \"${workflow.input.cutoff_date}\") |{occurred_at:.starred_at, member: {github:  .user.login}}]"
      },
      "type": "JSON_JQ_TRANSFORM"
    }

Here's what's happening:

  • Input Parameters Explanation:
    • starlist: Contains the JSON data retrieved from the API call to GitHub.
    • queryExpression: Uses JQ syntax to filter and format the data:
      • select(.starred_at > "${workflow.input.cutoff_date}"): Filters entries where starred_at is after ${workflow.input.cutoff_date}.
      • { occurred_at: .starred_at, member: { github: .user.login } }: Constructs a JSON object with occurred_at set to the starred_at value and member containing GitHub login from user.

The entire queryExpression is enclosed in [] to denote that it's intended to produce an array of JSON objects. Each object corresponds to a user who meets the specified criteria (starred_at after ${workflow.input.cutoff_date}).

The queryExpression filters the JSON data, selecting entries where starred_at meets the specified date criteria, and formats the output JSON as follows:

    {
      "occurred_at": "date from JSON",
      "member": {
        "github": "github Login from JSON"
      }
    }