CDAP Documentation v3.1.2

The Cask™ Data Application Platform (CDAP) is an integrated, open source application development platform for the Hadoop ecosystem that provides developers with data and application abstractions to simplify and accelerate application development, address a broader range of real-time and batch use cases, and deploy applications into production while satisfying enterprise requirements.

CDAP is a layer of software running on top of Apache Hadoop® platforms such as the Cloudera Enterprise Data Hub, the Hortonworks® Data Platform, or the MapR Distribution. CDAP provides these essential capabilities:

  • Abstraction of data in the Hadoop environment through logical representations of underlying data;
  • Portability of applications through decoupling underlying infrastructures;
  • Services and tools that enable faster application creation in development;
  • Integration of the components of the Hadoop ecosystem into a single platform; and
  • Higher degrees of operational control in production through enterprise best practices.

CDAP exposes developer APIs (Application Programming Interfaces) for creating applications and accessing core CDAP services. CDAP defines and implements a diverse collection of services that land applications and data on existing Hadoop infrastructure such as HBase, HDFS, YARN, MapReduce, Hive, and Spark.

These documents are your complete reference to the Cask Data Application Platform: they help you get started and set up your development environment; explain how CDAP works; and teach how to develop and test CDAP applications.

It includes the CDAP programming APIs and client interfaces, with instructions on the installation, monitoring and diagnosing fully distributed CDAP in a Hadoop cluster.

  • Introduction to CDAP:
    • Simple Access to Powerful Technology: A comparison between using the current technologies available from the Hadoop ecosystem and using CDAP
  • Developers’ Manual:
    • Getting Started Developing: A quick, hands-on introduction to developing with CDAP
    • Overview: The overall architecture and technology behind CDAP
    • Building Blocks: The two core abstractions in CDAP: Data and Applications, and their components
    • Security: Perimeter security, configuration and client authentication
    • Testing and Debugging: Test framework plus tools and practices for debugging your applications
    • Ingesting Data: Different techniques for ingesting data into CDAP
    • Advanced Topics: Best practices for CDAP development, adding a custom logback
  • Application Templates:
    • Big Data without Big Development: How to use CDAP “out-of-the-box” to solve problems and use cases
    • Introduction to Application Templates: Applications that are reusable through configuration and extensible through plugins
    • ETL: Making performing ETL possible without writing code
    • Creating Custom ETL Plugins: For developers of custom ETL plugins
  • Administration Manual:
    • Installation: Putting CDAP into production, with installation, configuration, security setup, and monitoring
    • Operations: Logging, metrics, runtime arguments, scaling instances, resource guarantees, transaction service maintenance, troubleshooting and introduces the CDAP UI
  • Integrations:
    • Cloudera: Integrating CDAP into Cloudera, using Cloudera Manager, and running interactive queries with Impala
    • JDBC: The CDAP JDBC driver, included with CDAP
    • Pentaho: Pentaho Data Integration, a business intelligence tool that can be used with CDAP
    • Squirrel: SquirrelSQL, a simple JDBC client that can be integrated with CDAP
  • Examples, Guides, and Tutorials:
    • Examples: Included with the CDAP SDK, they range from a simple introductory to more elaborate examples
    • How-To Guides: Designed to be completed in 15-30 minutes, these guides provide quick, hands-on instructions
    • Tutorials: Designed to be completed in 2-3 hours, these tutorials provide deeper, in-context explorations
    • CDAP Apps and Packs Repository: data applications built using CDAP and useful building blocks for your data applications
  • Reference Manual:
    • Command Line Interface API: Methods for interacting with a CDAP instance from within a shell
    • HTTP RESTful API: HTTP interface for a multitude of purposes
    • Java Client API: Methods for interacting with CDAP from external Java applications
    • Javadocs: The Java APIs for writing CDAP applications
    • Trademarks, Licenses, and Dependencies: Trademark and License information for CDAP and lists of CDAP dependent packages
  • Glossary: Definitions of terms and phrases used in CDAP
  • FAQ: Answers to questions you might have about CDAP
  • Search: Search this documentation using Quick Search