We recommend switching to the latest versions of Edge, Firefox, Chrome or Safari. Using Internet Explorer will result in a loss of website functionality.
Our Support systems migrated on Saturday, May 21. We'll automatically forward you to the new location for this content.

Error on look up node - Maximum Buffer Size



  • Avatar
    Stony Smith

    My suggestion is to do what I refer to as a "reverse lookup".

    Add another lookup in front of the one you have.. switch the inputs on it. put the 1.7m records on top and the 5 records on the bottom. 

    The code will be simply:

    emit 1:*
    where matchIsFound

    This way, only the records that could possibly match from pin2 will be fed to your existing lookup.

    There is a place where you can change the buffer setting, but you may fare better with trying the trick above.

    Comment actions Permalink
  • Avatar
    Tim Segall


    As per the documentation https://preview-desktop.lavastorm.com/#e-node-help/Correlation/lookup.html?Highlight=lookup this node 'Enriches a large data set (left table) with a small data set (right table) by joining the two data sets based on common fields.'

    This node is meant for one input that is large and one that is small (e.g. a lookup table).  Given you have two large data sets you will need to use the Join or X-Ref node.

    Cheers, Tim.

    Comment actions Permalink
  • Avatar
    Martin Hamilton

    Thanks Both

    i have run both options provided and each works for me.



    Comment actions Permalink

Please sign in to leave a comment.