Leveraging "Storage Type" within Data Views to control where data is processed

Setting the Storage Type

The Storage Type can be chosen within the Data View's Detail tab:

DQ+ AWS editions have all three settings available. DQ+ Azure edition solely has High performance available. DQ+ Enterprise has High performance and Big Data available. The Big Data option for Enterprise deployments is dependent on the DATAVIEW_BIG_DATA_STORE_TYPE setting in the installation property file. 

Performance implications

Regular performance is recommended for dashboards that are viewed infrequently or contain small amounts of data. High performance is recommended for dashboards frequently viewed or with large data. 

Both regular and high performance will render data quicker than the Big Data setting.

In AWS deployments, the difference of regular and high performance is the use of traditional hard drives vs solid-state drives. 

DQ+ Enterprise edition

For the DQ+ Enterprise edition, the Storage Type settings correspond to:

• High performance: Vertica
• Big data: Hive/Impala

Leveraging Impala will result in slower executions than leveraging Vertica; however, leveraging Impala also reduces the data load in Vertica and can prevent overuse of the Vertica license.

Big data can use Hive or Impala based upon the DATAVIEW_BIG_DATA_STORE_TYPE setting in the installation property file. Example configuration of both Hive and Impala are also within the installation property file.

DQ+ Azure edition

DQ+ Azure has solely the High performance option available. 

DQ+ AWS edition

For the DQ+ AWS edition, the Storage Type settings correspond to:

• Regular performance: Redshift (defaults to dc2.large)
• High performance: Redshift (defaults to ds2.xlarge)
• Big data: Athena

For AWS deployments in a customer-cloud environments, the specific sizing of the Redshift clusters are controlled via the installation properties files. More information on updating the cluster sizes is within the "Changing the infrastructure" section of the DQ+ install guide.

More information on the Storage Type and other Data View features can be found within the Data View documentation.