Error on look up node - Maximum Buffer Size
Hi - am receiving an error below:
Node execution terminated while loading lookup hash by error: Maximum buffer size of 536870912 bytes exceeded. Aborting.
Input1 has 5 records
Input 2 has 1.74M records
Is there a setting that needs to be configured?
I am running the desktop version of dataverse.
Thanks
Martin
-
My suggestion is to do what I refer to as a "reverse lookup".
Add another lookup in front of the one you have.. switch the inputs on it. put the 1.7m records on top and the 5 records on the bottom.
The code will be simply:
emit 1:*
where matchIsFound
This way, only the records that could possibly match from pin2 will be fed to your existing lookup.
There is a place where you can change the buffer setting, but you may fare better with trying the trick above. -
Martin,
As per the documentation https://preview-desktop.lavastorm.com/#e-node-help/Correlation/lookup.html?Highlight=lookup this node 'Enriches a large data set (left table) with a small data set (right table) by joining the two data sets based on common fields.'
This node is meant for one input that is large and one that is small (e.g. a lookup table). Given you have two large data sets you will need to use the Join or X-Ref node.
Cheers, Tim.
Please sign in to leave a comment.
Comments
3 comments