Error on look up node - Maximum Buffer Size

Follow

Martin Hamilton

• September 04, 2017 19:25

Hi - am receiving an error below:

 

Node execution terminated while loading lookup hash by error: Maximum buffer size of 536870912 bytes exceeded. Aborting.

 

Input1 has 5 records

Input 2 has 1.74M records

Is there a setting that needs to be configured?

I am running the desktop version of dataverse.

Thanks

Martin

0

• 

  Stony Smith

  • September 04, 2017 20:47

  My suggestion is to do what I refer to as a "reverse lookup".

  Add another lookup in front of the one you have.. switch the inputs on it. put the 1.7m records on top and the 5 records on the bottom. 

  The code will be simply:
  
  emit 1:*
  where matchIsFound
  
  This way, only the records that could possibly match from pin2 will be fed to your existing lookup.
  
  There is a place where you can change the buffer setting, but you may fare better with trying the trick above.

  0

  Comment actions Permalink
• 

  Tim Segall

  • September 04, 2017 20:48

  Martin,

  As per the documentation https://preview-desktop.lavastorm.com/#e-node-help/Correlation/lookup.html?Highlight=lookup this node 'Enriches a large data set (left table) with a small data set (right table) by joining the two data sets based on common fields.'

  This node is meant for one input that is large and one that is small (e.g. a lookup table).  Given you have two large data sets you will need to use the Join or X-Ref node.

  Cheers, Tim.

  0

  Comment actions Permalink
• 

  Martin Hamilton

  • September 07, 2017 14:08

  Thanks Both

  i have run both options provided and each works for me.

  cheers

  Martin 

  0

  Comment actions Permalink

Please sign in to leave a comment.