Installation on Microsoft Azure HDInsight

Introduction

This section describes installing CDAP on Microsoft Azure HDInsight using Microsoft's website to:

Information on Microsoft Azure HDInsight is available online.

CDAP 5.0 is compatible with Microsoft Azure HDInsight 3.4, 3.5, and 3.6.

Creating a Cluster with CDAP Installed

  1. Log onto Microsoft Azure and navigate to HDInsight clusters (either through the New menu and under Intelligence + analytics, or directly through the HDInsight clusters icon in the left sidebar):

    ../_images/azure-hdinsight-1.png

    Microsoft Azure: Finding HDInsight

  2. Create an HDInsight cluster by clicking the "Create" button. Quick create will create a cluster, but without CDAP installed. We recommend instead using the Custom (size, settings, apps) tab to access additional options and add installing CDAP as part of the cluster creation.

    ../_images/azure-hdinsight-2.0.png

    Microsoft Azure HDInsight: Creating a cluster using Quick create

    Using the Custom (size, settings, apps) tab, enter a Cluster name (such as cdap-cluster). Under Cluster type, select "HBase". (To continue, click Select to close the Cluster configuration window):

    ../_images/azure-hdinsight-2.1.png

    Microsoft Azure HDInsight: Creating a cluster using Custom (size, settings, apps) settings

  3. Enter the required basic settings: Cluster login password, Resource group, and Location:

    ../_images/azure-hdinsight-3.0.png

    Microsoft Azure HDInsight: Configuring basic settings

  4. Under Storage, set a storage account, either an existing account or enter a unique name for a new account.

    ../_images/azure-hdinsight-3.1.png

    Microsoft Azure HDInsight: Setting a storage account

  5. Under Applications, search for CDAP and select the version you'd like to install and accept the legal terms:

    ../_images/azure-hdinsight-3.2.png

    Microsoft Azure HDInsight: Setting CDAP as an application on the cluster

  6. Under Cluster size, at a minimum, a CDAP cluster requires four D3 v2 region nodes. This is enough YARN capacity to run only simple examples. For any serious work, you will need a cluster with additional nodes (Microsoft recommends additional smaller nodes versus fewer larger nodes due to the nature of blob storage):

    ../_images/azure-hdinsight-3.3.png

    Microsoft Azure HDInsight: Setting cluster size

  7. Under Advanced settings, there are currently no settings required:

    ../_images/azure-hdinsight-3.4.png

    Microsoft Azure HDInsight: Advanced settings are currently not required

  8. Click Next to review the summary. Then click Create to start cluster initialization:

    ../_images/azure-hdinsight-4.png

    Microsoft Azure HDInsight: Summary and confirming configuration

  9. Once the cluster is ready—it may take a few minutes while the YARN containers come up—its icon will show on the portal dashboard:

    ../_images/azure-hdinsight-5.0.png

    Microsoft Azure HDInsight: Cluster is created and running

  10. Open the created cluster. Click the Applications button to see the installed applications:

    ../_images/azure-hdinsight-5.1.png

    Microsoft Azure HDInsight: The cluster dashboard, Applications button circled

  11. CDAP should show as installed successfully, with a Portal link to access it:

    ../_images/azure-hdinsight-5.2.png

    Microsoft Azure HDInsight: Installed applications, showing CDAP and its Portal link

Install CDAP as an Application on an Existing Cluster

You can add CDAP to an existing cluster. (If you added CDAP in the previous steps for creating a cluster, you can jump to verification.)

  1. Open the created HBase Cluster. Click the Applications button:

    ../_images/azure-hdinsight-6.png

    Microsoft Azure HDInsight: The cluster dashboard

  2. In the open window, click + Add to show the Available applications. Choose the CDAP version you'd like to install and accept the legal terms. Once confirmed in the following step, installation of CDAP starts:

    ../_images/azure-hdinsight-7.png

    Microsoft Azure HDInsight: Adding CDAP as an application

  3. The initial status starts at Accepted:

    ../_images/azure-hdinsight-8.png

    Microsoft Azure HDInsight: CDAP accepted as an application

  4. Once CDAP is installed successfully, its status shows as Installed, with a Portal link to access it:

    ../_images/azure-hdinsight-9.png

    Microsoft Azure HDInsight: CDAP installed as an application and its Portal link

Verification

  1. Click the Portal link to launch the login window. Login by entering the cluster login name (default admin) and password which are entered when you created the cluster:

    ../_images/azure-hdinsight-10.png

    Microsoft Azure HDInsight: Login window

  2. A new window will open with the CDAP "Welcome" page:

    ../_images/azure-hdinsight-11.png

    Microsoft Azure HDInsight: CDAP UI "Welcome" page

  3. Navigate to the Administration page from CDAP pulldown menu in the upper-right. (You can also directly enter the URL for the page, https://<cdap-host>:443/cdap/administration, substituting for <cdap-host> the host name of the CDAP server, as shown above in step 1 of Verification):

    ../_images/azure-hdinsight-12.png

    Microsoft Azure HDInsight: CDAP Administration page, showing CDAP up and running