Data360 DQ+ Log Locations

Data360 DQ+ logs are found within the following locations

Application logs ( named “cafe*” ), and app / webserver logs :

• The elastic beanstalk logs for the current hour are available from the AWS console as described here : https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html
• The elastic beanstalk logs for past hours where the logs have rotated out are available from the s3 location for rotated logs specified here: https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features.logging.html#health-logs-s3location. Please note, there is more than one instance and the logs for each instance are in separate sub folders. You can look up the instance ids of the currently executing instances from the AWS console in the "Health" section for the elastic beanstalk instance.

Job execution logs ( logs for a particular EMR job execution ) :

• For all versions of DQ.  The EMR logs for an application ( executed DQ+ analysis job ) can be found by looking up the "Log URI" for the shared cluster ( from the summary tab in the AWS console screen for the EMR cluster ). The logs for an application ( executed job ) are at /JobFlowId/containers/applicationId within the bucket as described here: https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-manage-view-web-log-files.html#emr-manage-view-web-log-files-s3. The JobFlowId is the first token, and the applicationId is the second token in the execution engine id shown in the DQ+ UI.
• For DQ+ environments within the shared cluster, the current EMR Log path is: emrLogs/<ID>/containers/<applicationId>.
  
• Starting in DQ+ v5.2, you can download EMR execution logs from the DQ+ UI directly.
  • The screen below can be located by going to: DQ+ Pipeline > {Analysis Job You Want Logs For} > Execute > Executions :