This article is intended for developers who plan to interact with Data360 DQ+ by utilizing its RESTful web services.

Security and User Authentication

Only valid Data360 application users can access the REST APIs. An application user will need to be set up at the Data360 side to allow these calls to be made.

An application user account can be created from the Admin->Users menu of the Data360 application.

The API will be validated using Basic Authentication (i.e. user name and password). Basic Authentication  is a Base64 encoded user name and password sent by an "Authorization" header in the following format:

<user name>:<password>

On successful authentication, the APIs will return an HTTP status of 200 OK. In the case of failure, it will return 401 Unauthorized.

If the application user does not have permission, the API responds with: HTTP Status 403 - Access is denied.

API endpoint URL

The REST endpoint URLs have a common root endpoint of http://<DATA360DQ+-URL>/

For example, to call the API /external/devenv/readbyname, you would invoke the following:

http://<DATA360DQ+-URL>/external/devenv/readbyname

Generic Error Response

In general, if an unexpected error occurs while making an API call, a response similar to the follwing will be returned:

{
   "isError": true,
   "success": false,
   "errorId": "1a7b43ffc80b4375983a9bce9899cf77",
   "errorMessage": "Dev env not found for owning member - 1000 and name FOO",
   "httpStatusCode": 400,
   "errorClassName": "com.infogix.component.devenv.export.error.DevEnvNotFoundException",
   "stackTraceText":<omitted> 
   "attributes": null
 }

Data Store APIs

Retrieve the environment ID by name

Data360 segregates its data by environments. This API allows the user to get the environment ID (long value) by its display name.

The API syntax is:

• URL
  • /external/devenv/readbyname
• Method
  • GET
• Request Parameters
  • devEnvName: The environment display name.
• Success Response
  • Sample Response: "1000"

Create or Update a Data Store Definition

A data store definition must be present on the Data360 side before its data can be uploaded to it. This API will create a new data store definition if doesn’t exist, or it will update the existing data store definition with the same itemAlternateId.

The API syntax is:

• URL
  • external/bomgroup/itemdef
• Method
  • PUT
• Content-Type: application/json;charset=UTF-8
• Request Data
  • itemType The item type, always set to DATASTORE
  • itemAlternateId The UUID identifying the data store. It’s recommended that it is prefixed with a constant unique to the client for easy identification. The same id must be used for this item across all environments: dev, test, production etc.
  • itemDisplayName The display name for the definition.
  • itemDeploymentId ID to identify client’s deployment environment. It must be unique for each environment of the application that is creating or updating the definition.
  • itemDeploymentDisplayName Display name of client’s deployment environment.
  • itemProductId Product ID of the application making the request. It must be unique for a given application.
  • itemProductDisplayName Product display name of the application making the request.
  • devEnvId Environment ID this request is targeting. It can be looked up by its display name using the API described at section 2.1.
  • dataFormatType The data format for the data store. Possible values are: DELIMITED, PARQUET, and AVRO. Default to DELIMITED if not specified.
  • recordDef The record definition for the data store.
    • name Name of the record, must be the same as the value of itemAlternateId.
    • displayName Display name of the record, should be the same as the value of itemDisplayName.
    • field The list of fields in the record.
      • name The name of a field.
      • displayName The display name of a field.
      • fieldType The type of a field. Possible values are: BIGINTEGER, BOOLEAN, CURRENCY, DATE, DATETIME, TIME, DECIMAL, FLOATINGPOINT, INTEGER, STRING, ARRAY, STRUCT, and MAP.
    • Sample Request:

      {
          "itemType": "DATASTORE",
          "itemAlternateId": "LS_7cb3d72a4a6711e8842f0ed5f89f718b",
          "itemDisplayName": "AvroTest",
          "itemDeploymentId": "LavaStorm_LS_DEV",
          "itemDeploymentDisplayName": "DEV",
          "itemProductId": "LavaStorm",
          "itemProductDisplayName": "Lava Storm",
          "dataFormatType":"AVRO",
          "devEnvId": "1140",
          "recordDef": {    
             "name":"LS_7cb3d72a4a6711e8842f0ed5f89f718b",
             "displayName":"AvroTest",
             "field": [{
                   "fieldType": "STRING",
                   "name": "username"          
                }, 
                {
                   "fieldType": "STRING",
                   "name": "tweet"             
                },
                {
                   "fieldType": "BIGINTEGER",
                   "name": "timestamp"             
                }
            ]     
         }
      }

• Success Response
  • Description: The data store definition as persisted in the system.
  • Sample Response:

      {
         "itemType": "DATASTORE",
         "itemAlternateId": "LS_7cb3d72a4a6711e8842f0ed5f89f718b",
         "itemDisplayName": "AvroTest",
         "itemParentAlternateId": null,
         "itemParentDisplayName": null,
         "itemDeploymentId": null,
         "itemDeploymentDisplayName": null,
         "itemProductId": null,
         "itemProductDisplayName": null,
         "devEnvId": "1140",
         "recordDef": {
             "name": "LS_7cb3d72a4a6711e8842f0ed5f89f718b",
             "displayName": "AvroTest",
             "field": [
                 {
                     "fieldType": "STRING",
                     "name": "username",
                     "displayName": null,
                     "system": false,
                     "processingInstructions": null,
                     "groupWithAccess": [],
                     "autoId": false,
                     "semanticType": "UNKNOWN",
                     "semanticTypeDetailsJson": null,
                     "location": null,
                     "encrypt": false,
                     "secure": false,
                     "hideWhenSecure": false,
                     "maskPatternType": "SHOW_NONE",
                     "maskPattern": null,
                     "displayMask": null,
                     "defaultValue": null,
                     "codedValue": [],
                     "maxLength": 255
                 },
                 {
                     "fieldType": "STRING",
                     "name": "tweet",
                     "displayName": null,
                     "system": false,
                     "processingInstructions": null,
                     "groupWithAccess": [],
                     "autoId": false,
                     "semanticType": "UNKNOWN",
                     "semanticTypeDetailsJson": null,
                     "location": null,
                     "encrypt": false,
                     "secure": false,
                     "hideWhenSecure": false,
                     "maskPatternType": "SHOW_NONE",
                     "maskPattern": null,
                     "displayMask": null,
                     "defaultValue": null,
                     "codedValue": [],
                     "maxLength": 255
                 },
                 {
                     "fieldType": "BIGINTEGER",
                     "name": "timestamp",
                     "displayName": null,
                     "system": false,
                     "processingInstructions": null,
                     "groupWithAccess": [],
                     "autoId": false,
                     "semanticType": "UNKNOWN",
                     "semanticTypeDetailsJson": null,
                     "location": null,
                     "encrypt": false,
                     "secure": false,
                     "hideWhenSecure": false,
                     "maskPatternType": "SHOW_NONE",
                     "maskPattern": null,
                     "displayMask": null,
                     "defaultValue": null,
                     "codedValue": [],
                     "formatPattern": null,
                     "displayFormatPattern": null
                 }
             ],
             "transformer": null,
             "screen": [],
             "identityField": [],
             "recordIndex": [],
             "foreignKey": [],
             "deletedField": [],
             "deletedForeignKey": [],
             "queryFilter": [],
             "editFilter": [],
             "identityFieldsUnique": false,
             "externallyDefined": false,
             "externallyDefinedIDFields": false,
             "locallyModifiedIDFields": false,
             "allowAccessOnQueryFilterFallthrough": false,
             "allowAccessOnEditFilterFallthrough": false
         },
         "storeRepositoryType": null,
         "dataFormatType": "AVRO"
     }

Read a Data Store Definition

Reads the definition of an existing data store. 

The API syntax is:   

• URL
  • external/bomgroup/itemdef/get
• Method
  • POST
• Request Parameters
  • devEnvId: The environment ID
• Content-Type: text/plain
• Request Data
  • The itemAlternateId of the data store
• Sample Request

  LS_7cb3d72a4a6711e8842f0ed5f89f718b

• Success Response
  • Description: The definition of the data store matching the given ID.
  • Sample Response:

    {
        "itemType": "DATASTORE",
        "itemAlternateId": "LS_7cb3d72a4a6711e8842f0ed5f89f718b",
        "itemDisplayName": "AvroTest",
        "itemParentAlternateId": null,
        "itemParentDisplayName": null,
        "itemDeploymentId": null,
        "itemDeploymentDisplayName": null,
        "itemProductId": null,
        "itemProductDisplayName": null,
        "devEnvId": "1140",
        "recordDef": {
           "name": "LS_7cb3d72a4a6711e8842f0ed5f89f718b",
           "displayName": "AvroTest",
           "field": [
              {
                 "fieldType": "STRING",
                 "name": "username",
                 "displayName": null,
                 "system": false,
                 "processingInstructions": null,
                 "groupWithAccess": [],
                 "autoId": false,
                 "semanticType": "UNKNOWN",
                 "semanticTypeDetailsJson": null,
                 "location": null,
                 "encrypt": false,
                 "secure": false,
                 "hideWhenSecure": false,
                 "maskPatternType": "SHOW_NONE",
                 "maskPattern": null,
                 "displayMask": null,
                 "defaultValue": null,
                 "codedValue": [],
                 "maxLength": 255
              },
              {
                 "fieldType": "STRING",
                 "name": "tweet",
                 "displayName": null,
                 "system": false,
                 "processingInstructions": null,
                 "groupWithAccess": [],
                 "autoId": false,
                 "semanticType": "UNKNOWN",
                 "semanticTypeDetailsJson": null,
                 "location": null,
                 "encrypt": false,
                 "secure": false,
                 "hideWhenSecure": false,
                 "maskPatternType": "SHOW_NONE",
                 "maskPattern": null,
                 "displayMask": null,
                 "defaultValue": null,
                 "codedValue": [],
                 "maxLength": 255
              },
              {
                 "fieldType": "BIGINTEGER",
                 "name": "timestamp",
                 "displayName": null,
                 "system": false,
                 "processingInstructions": null,
                 "groupWithAccess": [],
                 "autoId": false,
                 "semanticType": "UNKNOWN",
                 "semanticTypeDetailsJson": null,
                 "location": null,
                 "encrypt": false,
                 "secure": false,
                 "hideWhenSecure": false,
                 "maskPatternType": "SHOW_NONE",
                 "maskPattern": null,
                 "displayMask": null,
                 "defaultValue": null,
                 "codedValue": [],
                 "formatPattern": null,
                 "displayFormatPattern": null
              }
           ],
           "transformer": null,
           "screen": [],
           "identityField": [],
           "recordIndex": [],
           "foreignKey": [],
           "deletedField": [],
           "deletedForeignKey": [],
           "queryFilter": [],
           "editFilter": [],
           "identityFieldsUnique": false,
           "externallyDefined": false,
           "externallyDefinedIDFields": false,
           "locallyModifiedIDFields": false,
           "allowAccessOnQueryFilterFallthrough": false,
           "allowAccessOnEditFilterFallthrough": false
        },
        "storeRepositoryType": null,
        "dataFormatType": "AVRO"
     }

Check If Data Store Is Selected

Users in Data360 can select data stores to indicate that their data are of interest to the users. This API checks if a particular data store is selected for data to be pushed. Application should perform this check before pushing data to Data360.     

The API syntax is:   

• URL
  • external/bomgroup/itemdef/selected
• METHOD
  • POST
• Request Parameters
  • devEnvId: The environment ID
• Content-Type: text/plain
• Request Data
  • The itemAlternateId of the data store
• Sample Request

  LS_4551ebf447fb41808f28fa1008df2f53

• Success Response
  • Description: true if selected, false otherwise.
  • Sample Response:

    True 

Upload a Data Store File

Once a data store definition is created in Data360, data can be uploaded to the data store. Supported data file formats include:

• Tab delimited file
• Tab delimited file compressed in bz2 format with a .bz2 file extension.
• Avro file with .avro file extension.

The API syntax is:

• URL
  • external/datastore/upload
• Method
  • POST
• Request Parameters
  • devEnvId: The environment ID
  • storeAlternateId: The UUID of the data store definition
• Content-Type: multipart/form-data
• Request Data
  • dataFile: The uploaded data file
• Sample Request

  POST /cafe/external/datastore/upload?devEnvId=1140&storeAlternateId=LS_7cb3d72a4a6711e8842f0ed5f89f718b HTTP/1.1
   Host: cwoo-7040.infogix.com:8080
   Content-Type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW
   Cache-Control: no-cache
   
   ------WebKitFormBoundary7MA4YWxkTrZu0gW
   Content-Disposition: form-data; name="dataFile"; filename="twitter.avro"
   Content-Type: 
   
   
   ------WebKitFormBoundary7MA4YWxkTrZu0gW--

• Success Response
  • Description: The work ID for this upload.
  • Sample Response:

    5a8459069abc4b5aafdeea20247e92cd

Execution APIs

Execute a Stage

Executes a stage. A stage is an executable item in Data360 like a Data View, Analysis, Process Model, etc.

The API syntax is:

• URL
  • /external/bomgroup/item/execute
• Method
  • POST
• Request Parameters
  • itemId: The unique ID of the item to execute.
  • devEnvName: The Environment name. Required if more than one Environment is set up in the system.

    To find the itemId of a Data Stage, use the Infogix Data360 UI. Navigate to the Data Stage within its Pipeline. Then, right click the Data Stage and Select Edit Settings.

    

    The itemId will be the UUID listed underneath the Data Stage’s name.

    

• Data Parameters (optional)
  Additional execution parameters can be specified via the request body. For example:
  

  {param1: value1, param2:value2}

• Success Response
  • Code: 200
  • Content:

    {exception: null, executionId:123456789abcdef}

• Error Response
  • Code: 200
  • Content:

    {exception: <Exception details>,    executionId:123456789abcdef}

Get the Status of a Stage Execution

The API syntax is:

• URL
  • /external/bomgroup/item/status
• Method
  • GET
• Request Parameters
  • executionId: The unique ID of the execution. Normally obtained after the stage execution call.
• Success Response
  • Code: 200
  • Content: RUNNING, SUCCESS, COMPLETED_WITH_ERRORS, FAIL, or TERMINATE.
• Error Response
  • Code: 200
  • Content: Error message describing the error.
• Success Response
  • Code: 200
  • Content: RUNNING, SUCCESS, COMPLETED_WITH_ERRORS, FAIL, or TERMINATE.
• Error Response
  • Code: 200
  • Content: Error message describing the error.

Get the Completion Status of a Stage Execution

• URL
  • /external/bomgroup/item/completionstatus
• Method
  • GET
• Request Parameters
  • executionId: The unique ID of the execution.
• Success Response
  • Code: 200
  • Content: JSON string with name-value pairs for the properties set by a Data Stage.
• Error Response
  • Generic error response