Hi Lavastorm Community,
I gave a condition:
Input CSV file contained 700k records (2 columns, SalesAgent ID & SalesOrder ID) had been performed HashSplit into 10 output which each output automatically contained around 70k records. Then, I do the linkage each of 10 output of HashSplit with 10 unit of DB Execute Node which perform sqlSelect script command to the same Oracle database table "SALES_ORDER" (contains 500 million records).
DB Execute Node
Query: select NN_Agent_No, NN_Order_Item_No from SALES_ORDER where NN_Agent_No = :1 and NN_Order_Item_No = :2
Each of 10 unit of DB Execute Node are running too long (almost 20 hours) just to retrieve that both ID in CSV list either exist or not in database.
I need to find a better way to do this. If try to use a while loop node, it seems the data of each 70k still getting longer to run since its looks bulk feed 70k records to DB Execute Node.
However, when I try to feed let say 1 record from CSV list to DB Execute Node, its getting fast and I can't find a way how to do this with While Loop Node.
Any advice or ideas to resolve this issue.
Please sign in to leave a comment.