Data Analyzer

Follow

Akihiko Katayama

• November 01, 2016 08:28

I've run the Data Analyzer node to analyze the output data from the Static Data node.  The data is just the sample one provided by Dataverse, and which is clearly set as an integer type from Static Data node but found by Data Analyzer as bint type.  I'm not sure how Dataverse created this data or why it finds it to be different at the end of the process.  How can I correct this?

0

• 

  Stony Smith

  • November 01, 2016 08:50

  Nothing to correct. bint is "binary integer".. meaning that the data is stored in an internal format which is quicker to process.

  Where are you seeing the bint? in the data viewer, or by looking in th BRD file itself?

  0

  Comment actions Permalink
• 

  Akihiko Katayama

  • November 01, 2016 12:53

  From the "analysis" output pin of data analyzer node.  I'm using this data as a filter node to change the metadata.  Unfortunately the metadata cannot use the bint type, and something I need to define for NewTypeField in Change Metadata Node.

  0

  Comment actions Permalink
• 

  Adrian Williams

  • November 01, 2016 14:23

  I assume your use case is similar to that shown below.

   

  

  The 'bint' data type from the 'analysis' pin of the Data Analyzer node can be converted by using the following script in the downstream Transform node:

  tmpType = if('discovered type' == "bint") then "int" else 'discovered type'

  
  emit 'field name' as "OldName",
     str(null) as "NewName",
    tmpType as "NewType"

   

   I would be interested to understand why your preferred solution is to use the Change Metadata node instead of using the output directly from the Data Analyzer node's 'converted data' pin. The data type on the converted data pin is valid and can be used by downstream nodes - i.e. no conversion is actually required in this case.

  Also, have you looked at the Data Converter node which provides automatic data type detection functionality and is provides better performance for larger data sets compared with the Data Analyzer node.

   

   

  0

  Comment actions Permalink
• 

  Akihiko Katayama

  • November 01, 2016 17:52

  Using the Filter + Change Metadata node because I was trying to filter out unused fields that are sitting in my dataset.

  I wasn't trying to convert any data from one type to another.  Would you still recommend Data Converter node?

  I'm interested to know why Dataverse nodes would even list out an unrecognised bint data type if other nodes can't recognise the type unless its called int... Maybe I have misunderstood something.

  I got my node to work so thank you very much.

  0

  Comment actions Permalink
• 

  Stony Smith

  • November 01, 2016 17:59

  The DataAnalyzer node is reporting the 'backoffice" data type for the field.. It makes sense not to confuse the issue.  In the background, there is a (slight) difference between integer and bint, so the node is technically correct, but it doesn't matter as far as you're concerned.  You will always refer to it as "int".

  If you want to drop some columns, then you can simply set up a Transform node this way:

  
  emit *
  exclude myField1
  exclude myField2
  exclude myField3

  You don't need to reference the data types to do this.

  0

  Comment actions Permalink

Please sign in to leave a comment.