Upgrading CDAP using Packages

Upgrading CDAP

When upgrading an existing CDAP installation from a previous version, you will need run an upgrade tool to make sure the CDAP table definitions in HBase are up-to-date.

To upgrade existing pipeline applications created using the 4.2.x versions of system artifacts, there are separate instructions on doing so. Run these after upgrading CDAP.

These steps will upgrade from CDAP 4.2.x to 5.1.1. If you are on an earlier version of CDAP, please follow the upgrade instructions for the earlier versions and upgrade first to 4.2.x before proceeding.

Upgrading CDAP Patch Release Versions

Upgrading between patch versions of CDAP refers to upgrading from one 5.1.x version to a higher 5.1.x version.

Upgrading CDAP Major/Minor Release Versions

Upgrading between major/minor versions of CDAP refers to upgrading from a 4.2.x version to 5.1.x. Upgrades between multiple major/minor versions must be done consecutively, and a version cannot be skipped unless otherwise noted.

Upgrade Steps

These steps will stop CDAP, update the installation, run an upgrade tool for the table definitions, and then restart CDAP:

  1. Stop all flows, services, and other programs in all your applications.

  2. Stop all CDAP processes:

    $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done
    
  3. Update the CDAP repository definition by running either of these methods:

    • On RPM using Yum:

      $ sudo curl -o /etc/yum.repos.d/cask.repo http://repository.cask.co/centos/6/x86_64/cdap/5.1/cask.repo
      

      This will create the file /etc/yum.repos.d/cask.repo with:

      [cask]
      name=Cask Packages
      baseurl=https://repository.cask.co/centos/6/x86_64/cdap/5.1
      enabled=1
      gpgcheck=1
      

      Add the Cask Public GPG Key to your repository:

      $ sudo rpm --import http://repository.cask.co/centos/6/x86_64/cdap/5.1/pubkey.gpg
      

      Update your Yum cache:

      $ sudo yum makecache
      
    • On Debian using APT:

      $ sudo curl -o /etc/apt/sources.list.d/cask.list http://repository.cask.co/ubuntu/precise/amd64/cdap/5.1/cask.list
      

      This will create the file /etc/apt/sources.list.d/cask.list with:

      deb [ arch=amd64 ] http://repository.cask.co/ubuntu/precise/amd64/cdap/5.1 precise cdap
      

      Add the Cask Public GPG Key to your repository:

      $ curl -s http://repository.cask.co/ubuntu/precise/amd64/cdap/5.1/pubkey.gpg | sudo apt-key add -
      

      Update your APT-cache:

      $ sudo apt-get update
      
  4. Update the CDAP packages by running either of these methods:

    • On RPM using Yum:

      $ sudo yum upgrade 'cdap*'
      
    • On Debian using APT:

      $ sudo apt-get install --only-upgrade '^cdap.*'
      
  5. Run the upgrade tool, as the user that runs CDAP Master (the CDAP user, indicated by <cdap-user>):

    $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run co.cask.cdap.data.tools.UpgradeTool upgrade
    

    Note that once you have upgraded an instance of CDAP, you cannot reverse the process; down-grades to a previous version are not possible. Also, note that authorization is disabled in the UpgradeTool so that the cdap user can upgrade all users' data.

    The UpgradeTool will produce output similar to the following, prompting you to continue with the upgrade:

    UpgradeTool - version 5.1.1-<build timestamp>.
    
    upgrade - Upgrades CDAP to 5.1.1
      The upgrade tool upgrades the following:
      1. User Datasets
          - Upgrades the coprocessor jars for tables
          - Migrates the metadata for PartitionedFileSets
      2. System Datasets
      3. UsageRegistry Dataset Type
      Note: Once you run the upgrade tool you cannot rollback to the previous version.
    Do you want to continue (y/n)
    y
    Starting upgrade ...
    

    You can run the tool in a non-interactive fashion by using the force flag, in which case it will run unattended and not prompt for continuing:

    $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run co.cask.cdap.data.tools.UpgradeTool upgrade force
    
  6. Restart the CDAP processes:

    $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done
    
  7. To upgrade existing pipeline applications created using the 4.2.x versions of system artifacts, there are separate instructions on doing so.

  8. Once CDAP has restarted, you can check the status of the upgrade using the Monitor HTTP RESTful API:

    $ curl -w"\n" -X GET "http://<cdap-host>:11015/v3/system/upgrade/status"
    

    Returning:

    {"metadata":true }
    

Upgrading Hadoop

These tables list different versions of CDAP and the Hadoop distributions for which they are supported. If your particular distribution is not listed here, you can determine its components and from that determine which version of CDAP may be compatible. Our blog lists the different components of the common Hadoop distributions.

Supported Cloudera Manager (CM) and Cloudera Distribution of Apache Hadoop (CDH) Distributions
CM Version CDH Version CDAP Parcel / CSD Version
5.10 5.9.x through 5.10.x 5.1.x
5.10 5.8.x 3.5.x through 5.1.x
5.10 5.7.x 3.4.x through 5.1.x
5.10 5.5.x through 5.6.x 3.3.x through 5.1.x
5.10 5.4.x 3.1.x through 5.1.x
5.10 no greater than 5.3.x 3.0.x through 5.1.x
 
5.9 5.9.x 5.1.x
5.9 5.8.x 3.5.x through 5.1.x
5.9 5.7.x 3.4.x through 5.1.x
5.9 5.5.x through 5.6.x 3.3.x through 5.1.x
5.9 5.4.x 3.1.x through 5.1.x
5.9 no greater than 5.3.x 3.0.x through 5.1.x
 
5.8 5.8.x 3.5.x through 5.1.x
5.8 5.7.x 3.4.x through 5.1.x
5.8 5.5.x through 5.6.x 3.3.x through 5.1.x
5.8 5.4.x 3.1.x through 5.1.x
5.8 no greater than 5.3.x 3.0.x through 5.1.x
 
5.7 5.7.x 3.4.x through 5.1.x
5.7 5.5.x through 5.6.x 3.3.x through 5.1.x
5.7 5.4.x 3.1.x through 5.1.x
5.7 no greater than 5.3.x 3.0.x through 5.1.x
 
5.6 5.5.x through 5.6.x 3.3.x through 3.6.x
5.6 5.4.x 3.1.x through 3.6.x
5.6 no greater than 5.3.x 3.0.x through 3.6.x
 
5.5 5.5.x 3.3.x through 3.6.x
5.5 5.4.x 3.1.x through 3.6.x
5.5 no greater than 5.3.x 3.0.x through 3.6.x
 
5.4 5.4.x 3.1.x through 3.6.x
5.4 no greater than 5.3.x 3.0.x through 3.6.x
 
5.3 no greater than 5.3.x 3.0.x through 3.1.x
5.2 no greater than 5.2.x 3.0.x through 3.1.x
5.1 no greater than 5.1.x Not supported
   
Supported Hortonworks Data Platform (HDP) Distributions
CDAP Series or Release Hadoop Distributions
CDAP 4.1.1, 4.2.x HDP 2.0 through HDP 2.6
CDAP 4.1.0 HDP 2.0 through HDP 2.5
CDAP 4.0.x HDP 2.0 through HDP 2.5
CDAP 3.6.x HDP 2.0 through HDP 2.4
CDAP 3.5.x HDP 2.0 through HDP 2.4
CDAP 3.4.x HDP 2.0 through HDP 2.4
CDAP 3.3.x HDP 2.0 through HDP 2.3
CDAP 3.2.x HDP 2.0 through HDP 2.3
CDAP 3.1.x HDP 2.0 through HDP 2.2
CDAP 3.0.x HDP 2.0 and HDP 2.1
   
Supported MapR Distributions for Apache Hadoop
CDAP Series MapR Distributions
CDAP 4.1.x MapR 4.1 through MapR 5.2
CDAP 4.0.x MapR 4.1 through MapR 5.2
CDAP 3.6.x MapR 4.1 through MapR 5.2
CDAP 3.5.x MapR 4.1 through MapR 5.2
CDAP 3.4.x MapR 4.1 through MapR 5.1
CDAP 3.3.x MapR 4.1 through MapR 5.1
CDAP 3.2.x MapR 4.1, MapR 5.0
CDAP 3.1.x MapR 4.1

Upgrade Steps

These steps cover what to do when upgrading the version of Hadoop of an existing CDAP installation. As the different versions of Hadoop can use different versions of HBase, upgrading from one version to the next can require that the HBase coprocessors be upgraded to the correct version. The steps below will, if required, update the coprocessors appropriately.

It is important to perform these steps as described, otherwise the coprocessors may not get upgraded correctly and HBase regionservers may crash.

  1. Upgrade CDAP to a version that will support the new Hadoop version, following the usual CDAP upgrade procedure for packages.

  2. After upgrading CDAP, start CDAP and check that it is working correctly.

  3. Stop all CDAP applications and services:

    $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i stop ; done
    
  4. Disable all CDAP tables; from an HBase shell, run the command:

    > disable_all 'cdap.*'
    
  5. Upgrade to the new version of Hadoop.

  6. Run the Post-Hadoop Upgrade Tasks—to upgrade CDAP for the new version of Hadoop—by running the CDAP Upgrade Tool, as the user that runs CDAP Master (the CDAP user, indicated by <cdap-user>):

    $ sudo -u <cdap-user> /opt/cdap/master/bin/cdap run co.cask.cdap.data.tools.UpgradeTool upgrade_hbase
    
  7. Enable all CDAP tables; from an HBase shell, run this command:

    > enable_all 'cdap.*'
    
  8. Restart CDAP:

    $ for i in `ls /etc/init.d/ | grep cdap` ; do sudo service $i start ; done