OutOfMemory Error in Data360 Analyze

There are two types of OutOfMemory (OOM) error that can be displayed to Data360 Analyze users - 

1. The Tomcat web application server can run out of memory. You will see an OOM error in the User Interface (UI) when you are trying to make a change to the data flow (Section A).

2. The node can run out of memory. You will see an OOM error when it is running and then fails (see section B below). 

SECTION A - STEPS TO RESOLVE TOMCAT OOM ERRORS:

If you hit the out of memory error in the UI, you can follow these instructions to increase the JavaMaxHeap size. Details on how to do that can be found in the help section: Getting Started > System Administration > Performance Tuning > Java Heap Space, and the steps for Windows and Linux are outlined here also.

NOTE: We request that you open a Support ticket if you encounter this issue - we'd like to know if the problem is due to a large number of users or if it's a bug that needs to be fixed. Also, there may be ways to tweak the dataflow so that it uses memory space more efficiently, which our Support staff can help you with. In some user cases, the JavaMaxHeap may already be high enough, and a different resolution is required.

STEPS FOR WINDOWS:

1. Open the command prompt as Administrator and cd to the Data360 Analyze installation directory by entering

cd "\program Files\Data3SixtyAnalyze\tomcat\bin"

2. For this step, you may need to look up the exact Tomcat Service. To do that, from the Start menu type the word "Services" and a list of the Windows services will come up. Scroll down to the Data360 Analyze services, there should be 3 of them, per the screenshot:

Hover over the third Data360 Analyze service and the exact details that should be entered into the command will be displayed to you. The command should look something like this if you're using version 3.4.1 on port 8081:

tomcat9w.exe //ES//Data3SixtyAnalyze-7731-TomcatServer3.4.1-8081

 So in the command prompt, step 2 should look something like this:

Press Enter and the Properties dialog will appear.

3. In the Properties dialog, select the Java tab and near the end adjust the Maximum memory pool parameter and click Apply. The new value depends on the amount of memory on your computer and the OS.

4. Restart the Data3Sixty Analyze Services

STEPS FOR LINUX:

1. Navigate to this directory <Data360 Analyze installation directory>/tomcat/bin and locate

setenv.sh

2. The Xmx value is the one that needs to be edited. 

For further reading, here is a link to some documentation that explains Xml well: Xml

3. Restart the Data360 Analyze Services

SECTION B - STEPS TO RESOLVE NODE OOM ERRORS:

ERROR MESSAGE:

Unexpected error occurred during processAll while running the node: java.lang.OutOfMemoryError: GC overhead limit exceeded.

This can occur in a Sort node if you're using a lot of unique key values. However, it can affect any nodes which have very narrow records (very few columns with smaller values in them) and the system is unable to allocate memory. 

It can also occur in the Excel nodes if the spreadsheet being read or written to is large enough. 

STEPS:

In the Define tab for the node, create a property called JvmMaxHeapSize, the type should be String, and set it to some big number e.g. `4g`.

To make this change globally, you could set it as a data flow level parameter. We recommend you test it first on a sample data flow. The value should remain at 4g at data flow level. Setting the parameter at dataflow level will work on all Java based nodes, the superseded nodes will not be affected. DB Query is C++ based, but JDBC Query is Java based.