JQ is a command-line tool for parsing JSON, and can be leveraged to be a diagnostic tool for Analyze performance issues. It can be used to retrieve and output specific information from your system, e.g. how many nodes or dataflows ran for more than 10 minutes, or who logged into the system.

Note: JQ is a third party tool. If you cannot install the binaries on your server, you can download the logs from the server and then run JQ on a different machine that has access to the logs. 

HOW TO SETUP JQ ON WINDOWS

1. Download jq from here: https://stedolan.github.io/jq/download/
2. Go into C:\Users\<username> and create a new folder called Bin
3. Place the downloaded file into the newly created Bin folder
4. Rename the file to “jq” (so the full name should be jq.exe)
5. Click on the Windows Start menu, and type Edit the System Environment Variables, and hit Enter
6. The System Properties window will be displayed. Click on Environment Variables

 

7. An Environment Variables window appears and two text boxes are displayed. In the top box, click on Path (third item in the list) and hit Edit

 

 

8. An Edit environment variable windows appears; click on the %USERPROFILE% entry and select Edit Text

 

 

9. An Edit User Variable windows appears. At the end of the %USERPROFILE% string, enter the path to the Bin folder that you created in step 2. The path should be separated from the %USERPROFILE% string by a semi-colon, and the Bin folder path should contain a semi-colon at the end of it

 

 

10. Click OK to exit the Edit User Variable window
     Click OK to exit the Environment Variables window
     Click OK to exit System Properties window

11. Open a command window (Windows Start menu > type CMD > select Command Prompt)

12. Enter this command: echo %PATH%. The new folder should be present at the end of the output text. This indicates that the environment variables have been changed and should now pick up the jq utility

 

 

13. Run ` jq ` in the command prompt and text should appear, indicating that jq is ready to be used

 

14. Cd into the folder where your lae-audit.log file is located <usually it’s under site-7731/logs, and start running jq commands

 

HOW TO RUN THE COMMANDS

• Create a notepad file in the folder you created in step 2, and call it “slownodes.jq” for example
• Paste the command into the file
• Remove the single quotes
• Remove the <lae-audit.log from the end. So if your command looks like this:

jq 'select(.auditCode=="nodeProcessed" and .arguments.elapsedTimeMS>500000)
 | {node: .arguments.node, timestamp: .timestamp, duration: .arguments.elapsedTimeMS, 
outputRecordCounts: .arguments.outputRecordCounts }' < lae-audit.log

It will become this in the notepad file:

• In the command prompt, enter this command and hit Enter:

jq –f slownodes.jq <lae-audit.log

• Repeat these steps for each command you use, modifying the command where necessary
• To make things easier, you can use this command to see the contents of the file:

type slownodes.jq

 

HOW TO INTERPRET THE OUTPUT

 

Here’s a quick overview of a sample output, with an explanation in orange below. 

Sample:

{

  "graph": " My Favourite Data FLow ",

  "node": " My Favourite Data Flow .Token My Node",

  "timestamp": "2020-11-16T01:03:23.811Z",

  "duration": 900269,

  "outputRecordCounts": "output:1"

}

Explanation:

 

{

  "graph": "My Favourite Data FLow",  the data flow name

  "node": " My Favourite Data FLow.Token My Node",

The data flow name and the name of the node

  "timestamp": "2020-11-16T01:03:23.811Z",

The time that it ran

  "duration": 900269,

How long it ran for

  "outputRecordCounts": "output:1"

The number of records on the output pin

}

 

COMMANDS 

 

There are a multitude of commands that you can run, below are a few to get started…

Please note, the commands containing "egrep" can't be used on Windows

This reads the audit log file:

jq<lae-audit.log

 

This will show you the graphs that ran last night and their status and how long they took: jq 'select(.auditCode=="dataflowProcessed") | { graph: .arguments.graph, status: .arguments.status, timestamp: .timestamp, duration: .arguments.duration }' < lae-audit.log This will show you how many graphs ran: jq 'select(.auditCode=="dataflowProcessed") | { graph: .arguments.graph, status: .arguments.status, timestamp: .timestamp, duration: .arguments.duration }' < lae-audit.log | egrep graph | wc -l This will show how many graphs ran for longer than one hour: jq 'select(.auditCode=="dataflowProcessed" and .arguments.duration > "01:00:00:000") | { graph: .arguments.graph, status: .arguments.status, timestamp: .timestamp, duration: .arguments.duration }' < lae-audit.log| egrep graph | wc -l This will show how many graphs ran for longer than two hours jq 'select(.auditCode=="dataflowProcessed" and .arguments.duration > "02:00:00:000") | { graph: .arguments.graph, status: .arguments.status, timestamp: .timestamp, duration: .arguments.duration }' < lae-audit.log| egrep graph | wc -l This will show the nodes that ran for more than 500s: jq 'select(.auditCode=="nodeProcessed" and .arguments.elapsedTimeMS>500000) | {node: .arguments.node, timestamp: .timestamp, duration: .arguments.elapsedTimeMS, outputRecordCounts: .arguments.outputRecordCounts }' < lae-audit.log This will show you how many nodes ran last night: jq 'select(.auditCode=="nodeProcessed") ' < lae-audit.log | egrep elapsedTimeMS | wc -l This counts the sleep nodes: jq 'select(.auditCode=="nodeProcessed")' < lae-audit.log | egrep 'Sleep"' | wc -l