Reimplement Pivot nodes in new syntax

Follow

M M

• February 15, 2021 09:14

is there a plan to reimplement the data-names pivot and  names-data pivot nodes in the new D360 Analyze? as of 3.6.2, i notice the underlying functionality is still in brainscript / java nodes, and is not particularly fast.

for example, i did a 3-way hash-split on 220 million records and performed a data->names pivot on each branch (70m records each), and d360 took 2.5 hours to complete on a 3GHz Intel 4-core device.

can the roadmap or releaseplan of added and enhanced functionality be made available?

0

• Official comment

  

  Tim Segall

  • February 16, 2021 21:49

  In terms of performance - the new node is dramatically faster.  In my brief testing typically 15 to 100 times faster.  For example, the new node processed 60 million rows in ~18 minutes.

  Comment actions Permalink
• 

  Adrian Williams

  • February 16, 2021 14:27

   A replacement node for the Pivot Data to Names node will be available in the 3.7.x release branch and, subsequently in the 3.8.x LTS branch. The replacement node does not utilize BRAINScript. The existing node will be superseded but will contine to be available and operate unchanged in these releases.

  The Pivot Names to Data node is already implemented using Java technologies and does not utilize BRAINScript. No changes are planned for this node at this time.

  While the Pivot Table node is implemented as a Composite library node, it does not expose BRAINScript to the user. It is likely that this node will be replaced by a new node in a future release, but the timeframe for this has not been confirmed at this time.

  The application currently provides the Transpose node, which is implemented using Java and can be used for a variety of simple data rotation use cases.

   

  0

  Comment actions Permalink
• 

  M M

  • February 17, 2021 09:36

  Thanks for the update. I look forward to the new nodes.

  0

  Comment actions Permalink

Please sign in to leave a comment.