Data360® Analyze 3.8.0 is Now Available - LTS Release

 

 

ATTENTION: In order to continue receiving product update notices, please Sign In and select “Follow” on  the Data360 Analyze announcements page here.

 

Data360 Analyze 3.8.0 release is now available.

Product downloads are available within our Data360 Analyze Download section. 

 

What’s New?

This Generally Available Long Term Support (LTS) release provides the following enhancements that are now available for the first time in any Data360 Analyze release:

The user interface now delivers improved design-time performance

Integration with data stored in Azure Data Lake Gen.2 

Improved support for generating a HTTP multi-party body

Increased flexibility when acquiring data from comma-separated value (csv) files

Improved support for metadata from in Azure blob and AWS S3 data sources

A number of nodes have been updated to not rely on Superseded nodes

Additional tools to allow administrative users to unlock locked documents

Command line inteface (CLI) tool to enable remote/scripted access to common tasks in Data360 Analyze

Changes to supported platforms.

 

Note: this overview includes functional changes that were delivered in the (unreleased) v.3.7.x releases.

 

 

Node Enhancements

Azure Data Lake Storage List Node
The new Azure Data Lake List node enables you to list files stored in Azure Data Lake Storage Gen2.

Only the contents of the specified directory are listed by default, but you can set the Recurse property to True to also list all sub-directories and their contents.

The AccountName property is mandatory. To authenticate the service user you can either:

• specify the AccountKey property
• specify the ClientID, ClientSecret and TenantID properties

The ‘from Field’ source option can be used with the FileSystem and RemotePath properties to allow the node to list the contents of multiple directories

When the node processes multiple records, the default action on encountering a transaction failure is to log the event and continue processing records. The FailureBehavior property can be set to Error, Log or Ignore transaction failures.

 

Azure Data Lake Storage Get Node

The new Azure Data Lake Get node enables you to download files/objects stored in Azure Data Lake Storage Gen2.

The user authentication options are the same as for the Azure Data Lake List node.

The ‘from Field’ source option can be used with the FileSystem and RemotePath properties to allow the node to download multiple files/objects.

The Directory property is used to specify the root directory in which downloaded files will be stored.

By default, the name of the object on the server is used for the name of retrieved object. 

You can specify the location for the object on the local server’s file system using the LocalPath property. The ‘from Field’ source option can be used to obtain the value for the property from an input field.

An existing file with the same name as the retrieved object will be overwritten by default, but you can set the Overwrite property to False to prevent this happening.

When downloading multiple objects, the default action on encountering a transaction failure is to log the event and continue processing input records. The FailureBehavior property can be set to Error, Log or Ignore transaction failures. Information on failed download transactions is output on the node’s ‘errors’ pin.

 

Azure Data Lake Storage Put Node

The new Azure Data Lake Put node enables you to upload files/objects to Azure Data Lake Storage Gen2.

The user authentication options are the same as for the Azure Data Lake List node.

The ‘from Field’ source option can be used with the FileSystem and RemotePath properties to allow the node to upload multiple files/objects to different storage locations.

The file to be uploaded is specified using the LocalPath property

• The file can be a literal entered manually or the file picker control can be used to select the file to be uploaded
• The ‘from File’ source option enables the path of the files to be uploaded to be obtained from an input field

An existing Azure storage object with the same name as the uploaded file will be overwritten by default, but you can set the Overwrite property to False to prevent this happening.

When the node processes multiple records, the default action on encountering a transaction failure is to log the event and continue processing records. The FailureBehavior property can be set to Error, Log or Ignore transaction failures.

 

Azure Data Lake Storage Delete Node

The new Azure Data Lake Delete node enables you to delete files/objects stored in Azure Data Lake Storage Gen2.

The file/object to be deleted is specified by the RemotePath property

• The ‘from File’ source option enables the files/objects to be deleted to be obtained from an input field
• The Recurse property can be set to True to delete all objects under the specified RemotePath.

The user authentication options are the same as for the Azure Data Lake List node.

The ‘from Field’ source option can be used with the FileSystem and RemotePath properties to allow the node to delete multiple files/objects in different storage locations.

When deleting multiple objects, the default action on encountering a transaction failure is to log the event and continue processing input records. The FailureBehavior property can be set to Error, Log or Ignore transaction failures.

Information on failed download transactions is output on the node’s ‘errors’ pin.

 

Multiparty Request Generator Node

The new Multiparty Request Generator node enables you to prepare a valid multi-part body, which can then be passed to the HTTP node for processing.

See the Help documentation for details.

 

Azure Blob List Node

The Azure Blob List node has been updated to return additional metadata including:

• Modified Date
• Create Date
• ETag
• ContentType
• ContentMD5
• Azure Metadata (e.g. the “hdi_isfolder” and “Data Classification” tags).

 

CSV/Delimited Node

You can now specify how the node handles duplicate input field names by specifying the ‘DuplicateFieldNameBehavior’ property. The node will generate an error by default.

The ‘Autoname’ option renames a duplicate field name to be <fieldname>_n where ‘n’ is the duplicate occurrence number.

You can set the duplicate field behavior on a per input record (i.e. file) basis using the ‘from field’ source option.

The ‘MissingFieldBehavior’ and ‘AdditionalFieldBehavior’ properties allow you to specify how the node handles missing fields or additional fields in the input file data. You can set the missing/additional field behavior on a per input record (i.e. file) basis using the ‘from field’ source option.

The node can be set to generate an error (default), log or ignore missing/additional fields.

The ‘MissingFieldNameBehavior’ property allows you to specify how the node handles a missing field name.

The node can be set to generate an error (default), ignore or Autoname

• If set to ignore, all data from that column are ignored
• If set to Autoname, the node generates a name for the field.

You can set the missing field name behavior on a per input record (i.e. file) basis using the ‘from field’ source option.

You can now specify multiple characters for the field delimiter.

Similarly, Record delimiters can now have up to two characters.

Certain characters such as colon (“:”) and newline (“\n”) characters are reserved characters in the metadata.

You can now specify whether the node should substitute invalid characters from the input file data that would be used in the record metadata. By default, the node generates an error if the resulting metadata would contain invalid characters.

The node now provides the ‘HeaderMode’ property.

The property enables you specify whether the input file has:

• No header
• Untyped header
• Typed header

The format of a typed header is:

<name>:<type>

In previous versions of Data360 Analyze the node provided the ‘HasHeaderLine’ and ‘TypedHeaders’ properties. These properties are now deprecated.

The node now provides the ‘DateFormat’, ‘TimeFormat’ and ‘DateTimeFormat’ properties. These properties enable you to specify the format of the data that is to be assumed when the node attempts to parse input fields with the specified type. Note: these properties are only used when the ‘HeaderMode’ property is set to ‘Typed’.

The node now provides the ‘NullMode’ property. The property enables you to specify what data values are to be interpreted as Null values. The following choices are available:

• Empty Value Is Null: Null will be used where two delimiters in a row with no quoting are found
• Quoted Empty Value Is Null: Null will be used where two identical quoting characters in a row are found
• Custom Value is Null: Null will be used where the custom value, as specified in the ‘NullString’ property is found
• Custom Quoted Value is Null: Null will be used where the custom value, as specified in the ‘NullString’ property is found within quotes
• None: There is no processing of Nulls, nothing will be treated as a Null value.

The node now provides the ‘MaxCharsPerColumn’ and ‘MaxColumns’ properties. These properties enable you to limit the amount of data that is imported from the source files and are useful when handling files that contain a very large amount of data in a single field value or contain many columns that are not relevant to your analysis. The node generates an error if a threshold is exceeded.

The new ‘CommentCharacter’ property allows you to specify the character that identifies comment lines in the input data. Comment lines are ignored during the import process. The default comment character is '#'. You can disable the identification of comment lines by setting the property to a value of \0

 

S3 List Node

The S3 List node has been updated to return any user-defined metadata as a JSON blob.

 

Database Metadata Node

The Database Metadata node has been changed to support a comma-separated list of patterns.

The Help descriptions of the CatalogFilter, SchemaFilter, TableViewFilter, FunctionFilter, ProcedureFilter, and TriggerFilter properties have been updated to reflect the new functionality.

 

Database Profiler Node

The Database Profiler node now allows you to choose whether to enable the default set of Logical Types, which are based on the locale.

 

JDBC Node Libraries

The JDBC Query, JDBC Execute and JDBC Store nodes now provide the 'DbDriverNativeLibraries' and 'DbDriverNativeLibraryDirectories' properties to enable you to specify the DB driver native libraries to load and where to load them from. See the relevant JDBC node documentation for further details.

 

Java Node Libraries

The Java node now provides the 'NativeLibraryPreloads' and 'NativeLibraryDirectories' properties to enable you to specify the native libraries (.so, .dlls) to load and where to load them from. See the Java node documentation for further details.

 

Publish to DQ+ Node (Experimental)

The Publish to DQ+ node now provides the ‘InvalidCharactersInFieldBehavior’ property.

This property defines the action to take when Data360 Analyze encounters a field format that is not supported by Data360 DQ+, for example, a field name containing spaces. By default, the node generates an error if an invalid character is in a field name but you can configure the node to replace the character and log a warning, or replace the character without generating a warning. See the Publish to DQ+ node documentation for further details.

 

Pivot Data to Names Node

A new Pivot Data to Names node is available in the node library. The new Pivot Data to Names node allows you to pivot on non-string values. 

The new node has been updated to not depend on Superseded nodes. The original Pivot Data to Names node is now superseded and has been renamed to Pivot Data to Names (Superseded). Data flows containing the original node continue to operate and the original node can be displayed in the node library when the ‘Show superseded Nodes’ option is checked.

 

Full Merge Node

The new Full Merge node is a replacement for the functionality provided by the Deep X-Ref node.

The Full Merge node is not dependent on Superseded nodes. The original Deep X-Ref node is now superseded and has been renamed to Deep X-Ref (Superseded). Data flows containing the original node continue to operate and the original node can be displayed in the node library when the ‘Show superseded Nodes’ option is checked.

 

X-Ref Metadata Node

The X-Ref Metadata node has been updated to not depend on Superseded nodes.

 

K-Means Advisor Node

A new K-Means Advisor node is available in the node library. The new node has been updated to not use the Bar Chart Engine library node. The original K-Means Advisor node is now superseded and has been renamed to K-Means Advisor (Superseded). Data flows containing the original node continue to operate and the original node can be displayed in the node library when the ‘Show superseded Nodes’ option is checked.

 

JSON Data Node and XML Data Node

The JSON Data node and XML Data node now provide the ‘ExcludeFieldPaths’ property. You can specify field paths within the input data which should not be written to the outputs. All fields under a specified path are excluded from the output data.

Previously, if the value of a field was always null or empty, it would never appear in the output of the XML or JSON nodes. An 'OutputAllNullFields' property has been introduced on the nodes, to determine whether or not such fields should be produced. The default is to output the fields. See the release notes for details of how to maintain system-wide compatibility with the previous operation of the nodes.

 

DeepSQL Node

The DeepSQL node now supports lineage generation for Google BigQuery data.

 

 

Application Enhancements

Design-time Performance Improvements
A range of changes have been made to improve the user experience when designing and building data flows:

• The performance of the application been significantly enhanced when navigating within, and making changes to, large data flows (i.e. data flows with over 3000 nodes)
• Previously, when two Composite nodes had run dependencies, and each Composite contained many nodes, this could lead to latency in the user interface when the user opened the data flow or navigated within it. This issue has now been resolved
• Previously, when connection bundles were connected to a series of other connection bundles this could lead to an out of memory event. This issue has now been resolved and it is now faster to navigate within this type of data flow
• The application is now more responsive when changing of graph and composite level properties
• When a data flow has resolution errors, navigation within the data flow is now faster
• The performance of the Apply Auto-Fixes functionality has been improved
• Save operations are now faster, together with the first edit operation on a saved data flow
• The resolution of permissions information is now faster
• The process of importing a legacy data flow (.brg file) is now faster.

 

Locked Documents

When a user starts to edit a document, it is locked for other users, so that two people cannot concurrently work on it. However, documents can remain locked if, for example, the user exits their browser for whatever reason, without first closing the document they are working on. Users with the administrative role can now view and unlock documents.

To view locked documents, click the ‘ALL FOLDERS’ tab and then select the ‘Locked Documents’ item in the navigation panel

To unlock a document, select the document to be unlocked in the list and click the ‘Unlock’ button

The Unlock Document dialog will be displayed to confirm your decision. The period in which the unsaved changes were made is shown. Note: Any unsaved changes that were made by the user will be lost. You have the option to save a copy of the document with unsaved changes.

When a document is locked for edit, the ‘Save As’ button is now disabled.

 

Property References

You can now insert a text substitution property reference to any run property defined on the parent run property set. 

 

Running Node Duration

When a node is running you can now see how long it has been running in the node’s properties panel.

 

Tomcat Version

The version of Tomcat that is used by the application has been upgraded to 9.0.44

 

Third-party Library Updates

A number of third-party libraries have been updated to benefit from the latest security updates

 

Python Version

The version of Python that is used by the application has been upgraded to 2.7.18

 

OpenSSL Version

The OpenSSL library has been updated to 1.1.1k

 

Audit Logs

Additional logging has been introduced to the audit log, which shows the progress of a system back up. The new audit codes are:

• backupWaiting
• backupStarted
• backupFinished

When a node is executed, the node type is now included in the event information in the audit log.

 

Third-Party License Information

Third-party license information is now available in the About Data360 Analyze dialog.

 

Postgres Database

The version of Postgres used by Analyze has been updated to version 13.3.

Note: Due to this change, upgrades to this release will require more disk space than normal. The additional space required is 1.5 times that of the current Postgres database size.

 

AnalyzeCli Tool

The Data360 Analyze command line interface (AnalyzeCli) tool is now available. 

The CLI is an independent Java application that can be used, for example, to retrieve information about your system, run commands, execute scripts and perform other operational and administrative tasks on a Data360 Analyze Analyze installation. See the ‘Data360 Analyze CLI’ topic in the Help documentation for further details.

 

Changes to Supported Platforms

Windows Server Core 2019 is now a supported platform

Windows Server 2012 R2 64-bit is no longer a supported platform

Red Hat Enterprise Linux 8 is now a supported platform

Red Hat Enterprise Linux 6 is no longer a supported platform

SUSE Linux Enterprise Server 12 is no longer a supported platform.

 

System Requirements

For Windows Server installations you must now also install Visual C++ Redistributable Packages for Visual Studio 2019.

 

Installer Check for Required .dll Files

The Windows version of the Data360 Analyze installer now checks for the required .dll files. If a required .dll file is not found, the Visual C++ Redistributable Packages for Visual Studio 2019 should be installed.

 

Superseded Nodes

Fixed Format File Node

The Fixed Format File node has been replaced by a new node with the same name.

The original Fixed Format File node is now named Fixed Format File (Superseded).

 

URL Query Builder Node
The URL Query Builder node has been replaced by a new node with the same name.
The original URL Query Builder node is now named URL Query Builder (Superseded).

 

Output Fixed Node
The Output Fixed node has been replaced by a new node with the same name.
The original Output Fixed node is now named Output Fixed (Superseded).

 

Data flows that contain the superseded nodes continue to operate as normal.  Superseded nodes are hidden by default in the node library but can be made visible by checking the ‘Show Superseded Nodes’ in the node library’s context menu. We recommend you use the new versions of the nodes in current projects and implement a plan to progressively replacing superseded nodes in existing data flows.

 

Deprecated Nodes

QVD File Node

The node is still available in the node library but is hidden by default. Deprecated nodes can be made visible by checking the ‘Show Deprecated Nodes’ option in the node library’s context menu.

 

Removed Functionality

Spark Nodes

The following Spark nodes are no longer supported and have been removed from the product:

Spark Agg, Spark Data Viewer, Spark Download, Spark End Session, Spark Filter, Spark Java Submit, Spark Join, Spark Sample, Spark Sort, Spark SQL Query, Spark Start Session, Spark Upload

It is possible to import a legacy data flow (BRG file) into Analyze that contains one or more of the above nodes. However, when the Data Flow is executed, an error message will be displayed

 

XMLpy Node

The XMLpy Node is no longer supported and has been removed from the product.

 

SharePoint 2010 Nodes

The following deprecated nodes are no longer supported and have been removed from the product:

Download for Sharepoint 2010, Folder List for Sharepoint 2010, Upload for Sharepoint 2010.

 

 

Resolved Issues

The following issues have been resolved in this release:

LAE-8880, LAE-9613, LAE-9851, LAE-9988, LAE-10423,
LAE-10846, LAE-22936, LAE-24352, LAE-23575, LAE-23865
LAE-23882, LAE-24521, LAE-24646, LAE-24692, LAE-24911
LAE-24956, LAE-24964, LAE-24989, LAE-25185, LAE-25277
LAE-25280, LAE-25468, LAE-25538, LAE-25550, LAE-25635
LAE-25678, LAE-25815, LAE-25816, LAE-25863, LAE-25902
LAE-25909, LAE-25911, LAE-25932, LAE-26026, LAE-26055
LAE-26240, LAE-26246, LAE-26287, LAE-26311, LAE-26382
LAE-26404, LAE-26430. 

 

See the release notes for details of the resolved issues.