We are having performance issues with the JDBC store node. We are writing to an Azure Synapse Analytics database (relational big data store) from a VM with 32GB Ram, 4 CPU vCores. Network wise, all date goes via the Microsoft network backbone (which usually is not a bottleneck).
- Reading in the table (220K records, 1 attribute) takes about 4 seconds
- Writing that same data to an existing table in Synapse takes +- 3 hours. We performed several trials with configurations of LoadMethod, BatchMode and CommitFrequency properties inside the node. The only thing that had a noticeble improvement was adding a "BulkInsert" property directly inside the connection string. That reduced the writing time to about +- 5 minutes (which still feels relatively long).
- Writing the same data to a BRD file is again in the order of seconds.
Could you give us some pointers on how to tackle the performance?
Thanks in advance!
Please sign in to leave a comment.