Dataverse 3.2.0 is Now Available

I’m excited to announce the Dataverse 3.2.0 release is now available.

This Generally Available Long term Support (LTS) release provides a range of improvements that:

• Enhances the performance of interactive node execution
• Improves the process of importing data flows and handling of library node dependencies
• Makes it easier to share data flows that contain property references to password properties
• Updates Dataverse nodes that connect to MongoDB and SalesForce, and provides updated JDBC drivers for Amazon Redshift and Microsoft SQL Server.

 

Application Enhancements

In-Container Node Execution

Dataverse now enables multiple nodes to be executed in a ‘node container’. When In-Container node execution is used the start-up time for a node is reduced – significantly improving (up to 10x) the performance of the application when nodes are processing a moderate number of records.

In-Container node execution only applies to Java-based nodes. Legacy C++ nodes/ superseded nodes e.g. the Agg Ex node are not eligible for In-Container node execution.

When a node is to be executed, Dataverse automatically determines whether the node is eligible to be executed In-Container based on a number of factors including the type of node and the volume of data to be processed by the node.

The number of nodes that will be run In-Container is dependent on the number of threads that is available to the node container. By default, the node container is configured to use up to four threads.

Where the Dataverse Server services multiple users or multiple concurrent executions (e.g. due to scheduled jobs), the node container thread limit should be increased to match the expected load. The maximum number of threads can be changed by modifying the system configuration in a property file. If no thread is available for an eligible node when it is to be executed, the node is executed in a separate JVM.

The maximum heap size should be considered when modifying the In-Container execution settings. By default, no maximum is applied to the node container’s heap memory.
You can specify a maximum heap size for the node container by modifying the system configuration in a property file.

In-Container node execution is enabled by default. In-Container node execution can be disabled for the Dataverse instance by modifying the system configuration in a property file.

During installation, the port used for the node container is allocated to the next available port starting from the Dataverse server port + 1. If required, the port can be set by modifying the system configuration in a property file.

If required, In-Container node execution can be disabled for a particular node by modifying the node’s configuration to add a Boolean runtime property. This may be appropriate when the node will be performing operations that are memory-intensive.

 

See the System administration -> Performance tuning topic of the online documentation for details of the In-Container node execution configuration settings.

 

Improved import process

When a dataflow or library node is imported you are now prompted for the target folder in the Dataverse directory:

 

Improved Information for Import Conflicts
When an import conflict is detected, Dataverse now displays details of the conflicts and you are prompted to select the required action:

 

Improved handling of missing dependencies
When a dataflow is imported that depends on a missing library node, the error message now provides information on the last known name and location of the missing node. This may assist in determining whether the library node has been moved to a location that is not on the current search paths. When a dataflow or library node is opened if there are dependent library nodes that cannot be found the application displays a Library Node Conflicts dialog.

 

Password Property References
Under normal circumstances Dataverse encrypts the value entered into a Password property. However, when the value entered into the Password property is a textual substitution reference to another Password property, the value is now not encrypted. This increases the portability of data flows when they are transferred from one Dataverse system to another Dataverse system.

As the referenced property is itself encrypted, this change does not impact on the security of sensitive data stored in Dataverse. However, it is strongly recommended you only substitute password properties into other password properties. When the data flow is imported into another Dataverse system, only the referenced Password property needs to be configured.

 

JDBC Driver Updates

The Amazon Redshift and Microsoft SQL Server JDBC drivers shipped with Dataverse have been upgraded.

 

 

Node Enhancements

MongoDB Nodes

This release introduces replacements for the following MongoDB nodes that were included in previous versions of Dataverse:

• Get MongoDB Data
• Query for MongoDB
• Update for MongoDB.

The new nodes leverage the Java 3.6.3 driver for MongoDB which provides support for the following versions of MongoDB: 2.6, 3.0, 3.2, 3.4, 3.6.

Note: The replacement nodes are not backwards compatible. Dataflows containing previous MongoDB nodes must be modified to use the new MongoDB nodes. The format of the output data differs from that output by the previous MongoDB nodes.

 

Dataverse Update for Salesforce Node
The Dataverse Update for Salesforce node now allows you to specify whether active assignment rules are to be applied. See the Salesforce documentation for further details:
https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/headers_autoassign.htm

If the property is not specified, the Assignment Rule header is not provided in the Salesforce REST API request meaning the default is to use the active assignment rules.

 

Superseded Nodes

Superseded Node Palette Filter

The Dataverse node library has been changed to not display superseded nodes by default. This streamlines the selection of current Dataverse nodes. The superseded nodes are still available for use within Dataverse. However, Lavastorm recommends the Python-based nodes are used for current projects. The node library context menu enables you to show/hide the superseded nodes:

 

Resolved Issues

The following issues have been resolved in this release:

LAE-8372, LAE-9151, LAE-9408, LAE-9598, LAE-9671,

LAE-9722, LAE-9735, LAE-9737, LAE-9813, LAE-9824,

LAE-9869, LAE-9870, LAE-9871, LAE-9872, LAE-9877,
LAE-9890, LAE-9891, LAE-9906, LAE-9907, LAE-9909,
LAE-9919, LAE-9920, LAE-9923, LAE-9925, LAE-9935,
LAE-9937, LAE-9940, LAE-9953, LAE-9957, LAE-9962,
LAE-9984, LAE-9986, LAE-10003, LAE-10010, LAE-10012.

See the release notes for details of the resolved issues.

 

Deprecated Functionality

Dataverse Application Programming Interface version 2

The version 2 of the published Dataverse API has been deprecated in this release.

While the version 2 Dataverse API is still available for use, Lavastorm recommends upgrading to version 3 of the published Dataverse API.