Data3Sixty™ Analyze 3.2.5 is Now Available

I’m excited to announce the Data3Sixty™ Analyze 3.2.5 release is now available.

 

This Generally Available release provides the following enhancements to the Data3Sixty Analyze application:

The Calculate Fields node which simplifies the creation of fields that are based on other fields

A set of nodes that streamline integration with data stored in the cloud

 

Node enhancements

Calculate Fields Node

The Calculate Fields node enables you to create new fields whose values are derived from calculations on input fields. A grid editor allows you to easily create multiple output fields.

For each field you can specify its name, data type and the expression that determines the values in the output records. Data records are output on the node’s ‘calculated Fields’ pin. Records that result in an expression error when processing the calculations can be output on the ‘errors’ pin.

You can select the new field’s data type from the drop-down list. The selected data type must conform with data type of the evaluated expression.

The calculations use Python expressions. You can simply include references to input fields by name provided the name is not a reserved Python keyword or contains spaces. If it is a reserved keyword or contains spaces then use the fields['<my selected field name>'] notation.

Your expressions can include conditional statements of the form:

<Value if True> if <Logical Condition> else <Value if False>

For example a calculation to determine the cost of shipping an order could have an expression like the following:

10.80 if Total_Order_Value < 50.00 else 0.0

 

The node automatically imports Python libraries that help with the handling of date, time and datetime type fields, and calculations involving null values in your data (see the node help for details). 

Once a new field has been defined, it may be referenced in a subsequent field definition. This is useful when you want to re-use a calculated value in multiple calculations.

You can configure the default value to be used in a calculation when a field in an input record contains a null value. A default value can be specified for each data type and applies to all fields that have that data type, or null values can be retained. In addition to allowing you to enter values, you can specify 'contemporary' values that will change each time the node is run, e.g. when calculating a date value you can use 'Today' and for time and datetime values you can use 'Now'.

In addition to creating new fields, you can perform ‘in-place’ updates to an existing field by specifying the existing field’s name as the Name property in the Calculated Fields grid:

You can leverage the options to set default values for a data type and the in-place update capability to replace null values for specified fields. For example, to set missing due dates in a date field to today’s date you could use the following calculation and default for null date values:

When working with datetime fields, the node automatically performs data type conversions when the new output field’s data type differs from the data type of the evaluated expression. This simplifies the handling of Python datetime objects and, for example, can be used to:

• Extract the date and time elements from a datetime value
• Convert a date value to a datetime type (with the time element preset to 00:00:00).

 

Google Cloud Storage List Node

The Google Cloud Storage List node enables you to retrieve a list of objects stored in a specified bucket.

 

You can filter the list of objects to be retrieved by specifying a value for the ObjectPrefix property. When the ObjectPrefix is set, the results comprise objects whose names begin with this prefix. All objects are retrieved by default.

Only authenticated uses can access the contents of a Google Cloud Storage bucket. You can specify the file that contains the credentials for the Google Cloud Storage service account or use the Google Cloud Storage default strategy.

The values for the Bucket, ObjectPrefix and Credentials properties can be literals or obtained from an input field.

Google Cloud Storage does not store objects in a hierarchical file system. However you can name your objects to make it appear as if they were stored hierarchically by separating elements of the path with a delimiter e.g. europe/revenue_2018_Q1.csv

By default the node presents the objects as a simple list. When the DirectoryMode property is set to True,  you can configure the node to retrieve results as if the objects in the bucket were stored in a hierarchical folder structure, for example if the ObjectPrefix has a value of “test/” and the bucket contained the following objects:

• europe/revenue_2018_q1.csv
• europe/revenue_ 2018_q2.csv
• europe/revenue_ 2018_q3.csv
• test/revenue_ 2018_q4.csv
• test/revenue_ 2019_q1.csv
• test/revenue_ 2019_q2.csv

The results list would comprise the following objects:

• test/revenue_ 2018_q4.csv
• test/revenue_ 2019_q1.csv
• test/revenue_ 2019_q2.csv

 

Google Cloud Storage Get Node

The Google Cloud Storage Get node enables you to download objects stored in a specified bucket.

You must specify the name of the object to be downloaded from the bucket. Optionally, you can specify a name for the downloaded object. The object name is used by default if the FileName property is not set.

You can specify the file that contains the credentials for the Google Cloud Storage service account or use the Google Cloud Storage default strategy.

You must specify the root directory into which the object(s) is to be stored. If the object name includes the file path delimiter the file will be written to a sub-directory of the root directory.

The values for the Bucket, Object, Credentials and FileName properties can be literals or obtained from an input field. This is particularly useful when the node is used to download multiple files. Any failed download attempts are, by default, logged and the details of the failure are output on the node's errors pin.

 

Google Cloud Storage Put Node

The Google Cloud Storage Put node enables you to upload files from Data3Sixty Analyze to a specified bucket.

 

You must specify the filename of the file to be uploaded to the bucket. Optionally, you can specify an object name for the uploaded file. The filename is used by default if the Object property is not set.

You can specify the file that contains the credentials for the Google Cloud Storage service account or use the Google Cloud Storage default strategy.

The values for the Bucket, Object, Credentials and FileName properties can be literals or obtained from an input field. This enables you to upload multiple files in a single run of the node. Any upload failures are, by default, logged with the details output on the node's errors pin.

 

Google Cloud Storage Delete Node

The Google Cloud Storage Delete node enables you to delete objects in a specified bucket.

 

You must specify the name of the Object to be deleted.

You can specify the file that contains the credentials for the Google Cloud Storage service account or use the Google Cloud Storage default strategy.

The values for the Bucket, Object, Credentials and FileName properties can be literals or obtained from an input field. This enables you to delete multiple objects in a single run of the node.