Plugin Presentation

When a plugin is displayed in the CDAP UI, its properties are represented by widgets in the CDAP Studio. Each property of a plugin is represented, by default, as a simple textbox in the user interface. By including an appropriate plugin widget JSON file, you can customize that presentation to use a different interface, as desired.

The available widgets that can be used are described below. Note that some of the plugins available for CDAP may include use of additional widgets that are not listed here. We recommend only using the list of plugins shown here, as these other widgets are internal to CDAP and may not be supported in a future release.

This document describes version 1.5 of the plugin specification. Changes to the specification are described in the Specification Changes and should be checked if you are using a version of the specification earlier than the current.

Plugin Widget JSON

To customize the plugin display, a plugin can include a widget JSON file that specifies the particular widgets and the sets of widget attributes used to display the plugin properties in the CDAP UI.

The widget JSON is composed of:

Each configuration group consists of a list of the individual properties and the widgets to be used. For example:

{
  "metadata": {
    ...
  },
  "display-name": "Plugin Display Name",
  "icon": {
    "type": "builtin|link|inline",
    "arguments": {
       "url": "http://localhost/myicon.png",
       "data": "data:image/png;base64,..."
    }
  },
  "configuration-groups": [
    {"label": "Group 1",
      "properties": [
      ...
      ]
    },
    {"label": "Group 2",
      "properties": [
      ...
      ]
    },
    ...
  ],
  "inputs": {
    ...
  },
  "outputs": [
    {"output-property-1"},
    {"output-property-2"},
    ...
  ],
  "jump-config": {
    ...
  }
}

Metadata

Metadata refers to the top-level information about a plugin. The only information required is the a map consisting of the spec-version, the version of the specification which the JSON follows.

Current version: 1.5. For example:

{
  "metadata": {
    "spec-version" : "1.5"
  },
  ...
}

Display Name

The display-name field specifies the name of the plugin as it is displayed on the CDAP UI. A display name can be different from the plugin name, and is not guaranteed to be unique. A display name can contain whitespace. If not specified, the plugin name is used as the display name.

Icon

The icon field allows users to specify custom icons for their plugins. Users can specify icons for their plugins in three different ways, depending on the value of the type field contained inside the icon field:

  • builtin: The icon type builtin indicates that the icon for the plugin is defined in CDAP, and is not specified in the plugin's widget JSON file. This is the default icon type, if an icon is not specified. If such an icon is not found, the CDAP UI will use a default icon type.
  • link: The icon type link indicates that the icon for the plugin should be fetched from a URL specified in the arguments map. When the type of icon is link, the user is expected to provide a link to the icon image in a url attribute inside the arguments map.
  • inline: The icon type inline indicates that the icon for the plugin should be decoded from a base64 encoded image specified in the arguments map. When the type of icon is inline, the user is expected to provide a base64 encoded image in a data attribute inside the arguments map.

Icon Image Specification - The image used for an icon for a CDAP plugin should be in PNG format (http://www.libpng.org/pub/png/). - The size of the image should be 50 X 50 pixels. - The image background should be transparent.

Configuration Groups

Configuration groups are a simple grouping of properties of a plugin. A configuration group is represented as a JSON object with a label and an ordered list of plugin properties for that group.

For example, in a Batch Source plugin, properties such as name, basePath, duration, and delay could be grouped into a Batch Source Configuration.

In the case of a Batch Source plugin, it could look like this:

{
  "configuration-groups": [
    {
      "label": "Batch Source Configuration",
      "properties": [
        {
          "name": "name",
          ...
        },
        {
          "name": "basePath",
          ...
        },
        {
          "name": "duration",
          ...
        },
        {
          "name": "delay",
          ...
        }
      ]
    }
  ]
  ...
}

Once a group is established, you can configure how each of the individual properties inside the group is represented in the CDAP UI.

Property Configuration

Each individual property of the plugin is represented by a configuration, composed of:

  • name: Name of the field (as supplied by the CDAP server for the artifact).
  • label: Text string displayed in the CDAP UI beside the widget.
  • widget-type: The type of widget to be used to represent this property.
  • widget-attributes: A map of attributes that the widget type requires to be defined in order to render the property in the CDAP UI. The attributes required depend on the widget type used.
  • plugin-function: An optional map of plugin method and its widget attributes that can be applied to a particular plugin property.

Note that all properties and property values are case-sensitive.

To find the available field names, you can use the Artifact HTTP RESTful API to retrieve plugin details for an artifact, which will include all the available field names. (If the artifact is your own, you will already know the available field names from your source code.)

In this example of a Batch Source plugin and its configuration-groups, four different properties are defined; three use a textbox widget, while one uses a dataset-selector. The groupByFields property includes a plugin function, used to fetch an output schema for the widget:

{
  "configuration-groups": [
    {
      "label": "Batch Source",
      "properties": [
        {
          "name": "name",
          "label": "Dataset",
          "widget-type": "dataset-selector"
        },
        {
          "name": "basePath",
          "label": "Base Path",
          "widget-type": "textbox"
        },
        {
          "name": "groupByFields",
          "label": "Group By Fields",
          "widget-type": "textbox",
          "plugin-function": {
            "method": "POST",
            "widget": "outputSchema",
            "output-property": "schema",
            "plugin-method": "outputSchema",
            "required-fields": ["groupByFields", "aggregates"],
            "missing-required-fields-message":
              "'Group By Fields' & 'Aggregates' properties are required to fetch schema."
          }
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        ...
      ]
    }
  ]
}

Plugin Widgets

A widget in the CDAP UI represents a component that will be rendered and used to set the value of a property of a plugin. These are the different widgets—their type, their attributes (if any), their output data type, a description, sample JSON—that we support in CDAP pipelines as of version 5.1.2.

Widget Type Widget Attributes Output Data Type Description Example Widget JSON
csv No attributes Comma-separated string Comma-separated values; each value is entered in a separate box
{
  "name": "property-csv",
  "widget-type": "csv",
  "widget-attributes": {}
}
dataset-selector No attributes string A type-ahead textbox with a list of datasets from the CDAP instance
{
  "name": "property-dataset-selector",
  "widget-type": "dataset-selector",
  "widget-attributes": {}
}
ds-multiplevalues
  • delimiter: the delimiter between each set of values
  • numValues: number of values (number of delimiter-separated values)
  • placeholders: array of placeholders for each value's textbox
  • values-delimiter: the delimiter between each value
string A delimiter-separated values widget that allows specifying lists of values separated by delimiters
{
  "name": "property-ds-multiplevalues",
  "widget-type": "ds-multiplevalues",
  "widget-attributes": {
    "delimiter": ",",
    "values-delimiter": ":",
    "numValues": "3",
    "placeholders": [
      "Input Field",
      "Lookup",
      "Output Field"
    ]
  }
}
dsv delimiter: delimiter used to separate the values Delimiter-separated string Delimiter-separated values; each value is entered in a separate box
{
  "name": "property-dsv",
  "widget-type": "dsv",
  "widget-attributes": {
    "delimiter": ":"
  }
}
hidden default: default string value for the widget string This "hidden" widget allows values to be set for a property but hidden from users. A default can be supplied that will be used as the value for the property.
{
  "name": "property-hidden",
  "widget-type": "hidden",
  "widget-attributes": {
    "default": "defaultValue"
  }
}
input-field-selector No attributes string A dropdown widget with a list of columns taken from the input schema. Selecting sets the input column for that plugin property.
{
  "name": "Property1",
  "widget-type": "csv",
  "widget-attributes": {}
}
javascript-editor default: default string value for the widget string An editor to write JavaScript code as a value of a property
{
  "name": "property-javascript-editor",
  "widget-type": "javascript-editor",
  "widget-attributes": {
    "default":
      "function transform(input, emitter, context) {\
\\n  emitter.emit(input);\\n}"
  }
}
json-editor default: default serialized JSON value for the widget string A JSON editor that pretty-prints and auto-formats JSON while it is being entered
{
  "name": "property-json-editor",
  "widget-type": "json-editor",
  "widget-attributes": {
    "default": "{ \"p1\": \"value\" }"
  }
}
keyvalue
  • delimiter: delimiter for the key-value pairs
  • kv-delimiter: delimiter between key and value
string A key-value editor for constructing maps of key-value pairs
{
  "name": "property-keyvalue",
  "widget-type": "keyvalue",
  "widget-attributes": {
      "delimiter": ",",
      "kv-delimiter": ":"
  }
}
multi-select
  • delimiter: delimiter to concatenate multiple values to form a single string
  • options: array of options where each option is defined by - id: value to be set for the plugin property - label: a label to be rendered in UI for that specific option
  • defaultValue: an array of default value(s) to use when rendering the widget for the first time
string Multi-select dropdown to choose one or more from a list of available options.
{
  "name": "property-keyvalue",
  "widget-type": "multi-select",
  "widget-attributes": {
    "delimiter": ",",
    "defaultValue": [
      "value1",
      "value2"
    ],
    "options": [
      {
        "id": "value1",
        "label": "Value One"
      },
      {
        "id": "value2",
        "label": "Value two"
      }
    ]
  }
}
radio-group
  • layout: Layout for the radio buttons to render. Defaults to block to render one below the other. Another option is inline to render side-by-side
  • options: an array of options where each option is defined by - id: value to be set for the plugin property - label: a label to be rendered in UI for that specific option
  • default: default value to use when no value is specified
string A group of radio buttons to choose one among a list of available options.
{
  "name": "property-keyvalue",
  "widget-type": "radio-group",
  "widget-attributes": {
    "layout": "inline",
    "default": "value1",
    "options": [
      {
        "id": "value1",
        "label": "Value One"
      },
      {
        "id": "value2",
        "label": "Value two"
      }
    ]
  }
}
keyvalue-dropdown
  • delimiter: delimiter for the key-value pairs
  • dropdownOptions: list of drop-down options to display
  • kv-delimiter: delimiter between key and value
string Similar to keyvalue widget, but with a drop-down value list
{
  "name": "property-keyvalue-dropdown",
  "widget-type": "keyvalue-dropdown",
  "widget-attributes": {
      "delimiter": ",",
      "kv-delimiter": ":",
      "dropdownOptions": [ "Option1", "Option2"]
  }
}
non-editable-schema-editor schema: schema that will be used as the output schema for the plugin string A non-editable widget for displaying a schema
{
  "name": "property-non-editable-schema-editor",
  "widget-type": "non-editable-schema-editor",
  "widget-attributes": {}
}
number
  • default: default value for the widget
  • max: maximum value for the number box
  • min: minimum value for the number box
string Default HTML number textbox that only accepts valid numbers
{
  "name": "property-number",
  "widget-type": "number",
  "widget-attributes": {
    "default": "1",
    "min": "1",
    "max": "100"
  }
}
password No attributes string Default HTML password entry box
{
  "name": "property-password",
  "widget-type": "password",
  "widget-attributes": {}
}
python-editor default: default string value for the widget string An editor to write Python code as a value of a property
{
  "name": "property-python-editor",
  "widget-type": "python-editor",
  "widget-attributes": {
    "default":
      "def transform(input, emitter, context):\
\\n  emitter.emit(input)\\n"
  }
}
schema
  • schema-default-type: default type for each newly-added field in the schema
  • schema-types: list of schema types for each field from which the user can chose when setting the schema
string A four-column, editable table for representing the schema of a plugin
{
  "name": "property-schema",
  "widget-type": "schema",
  "widget-attributes": {
    "schema-default-type": "string",
    "schema-types": [
      "boolean",
      "int",
      "long",
      "float",
      "double",
      "bytes",
      "string",
      "map<string, string>"
    ]
  }
}
select
  • default: default value from the list
  • values: list of values for the drop-down
string An HTML drop-down with a list of values; allows one choice from the list
{
  "name": "property-select",
  "widget-type": "select",
  "widget-attributes": {
      "default": "Bananas",
      "values": ["Apples", "Oranges", "Bananas"]
  }
}
stream-selector No attributes string A type-ahead textbox with a list of streams from the CDAP instance
{
  "name": "property-stream-selector",
  "widget-type": "stream-selector",
  "widget-attributes": {}
}
textarea
  • default: default value for the widget
  • rows: height of the textarea
string An HTML textarea element which accepts a default value attribute and a height in rows
{
  "name": "property-textarea",
  "widget-type": "textarea",
  "widget-attributes": {
    "default": "Default text.",
    "rows": "1"
  }
}
textarea-validate
  • placeholder: placeholder text for the textarea
  • validate-endpoint: plugin function endpoint to hit to validate the contents of the textarea
  • validate-button-text: label of the validate button
  • validate-success-message: message to display when validation succeeds
string An HTML textarea element with a button to validate its contents using a plugin function endpoint
{
  "name": "property-to-validate",
  "widget-type": "textarea-validate",
  "widget-attributes": {
    "placeholder": "E.g. ((token['Data Quality']['error'] / token['File']['output']) * 100) > runtime['error_percentage']",
    "validate-endpoint": "validate",
    "validate-button-text": "Validate",
    "validate-success-message": "Expression is valid"
  }
}
textbox default: default value for the widget string An HTML textbox, used to enter any string, with a default value attribute
{
  "name": "property-textbox",
  "widget-type": "textbox",
  "widget-attributes": {
    "default": "Default text."
  }
}

Plugin Function

A plugin function is a method exposed by a particular plugin that can be used for a specific task, such as fetching an output schema for a plugin.

These fields need to be configured to use the plugin functions in the CDAP UI:

  • method: Type of request to make when calling the plugin function from the CDAP UI (for example: GET or POST)
  • widget: Type of widget to use to import output schema
  • output-property: Property to update once the CDAP UI receives the data from the plugin method
  • plugin-method: Name of the plugin method to call (as exposed by the plugin)
  • required-fields: Fields required to call the plugin method
  • missing-required-fields-message: A message for the user as to why the action is disabled in the CDAP UI, displayed when required fields are missing values

The last two properties (required-fields and missing-required-fields-message) are solely for use by the CDAP UI and are not required for all widgets. However, the first four fields are required fields to use a plugin method of the plugin in the CDAP UI.

With plugin functions, if the widget is not supported in the CDAP UI or the plugin function map is not supplied, the user will not see the widget in the CDAP UI.

Example Plugin

In the case of a Batch Source plugin example, the configuration-groups, with additional widgets to show the groupByFields and aggregates properties and using a plugin-function, could be represented by:

{
  "configuration-groups": [
    {
      "label": "Batch Source",
      "properties": [
        {
          "name": "name",
          "widget-type": "dataset-selector"
        },
        {
          "name": "basePath",
          "widget-type": "textbox"
        },
        {
          "name": "groupByFields",
          "widget-type": "textbox",
          "plugin-function": {
            "method": "POST",
            "widget": "outputSchema",
            "output-property": "schema",
            "plugin-method": "outputSchema",
            "required-fields": ["groupByFields", "aggregates"],
            "missing-required-fields-message":
              "Both 'Group By Fields' and 'Aggregates' properties are required to fetch the schema."
          }
        },
        {
          "name": "aggregates",
          "widget-type": "textbox"
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        {
          "name": "duration",
          "widget-type": "textbox"
        },
        ...
      ]
    }
  ]
}

Inputs

Beginning with version 1.2 of the specification, a plugin can accept multiple input schemas and from them generate a single output schema. Using the field multipleInputs and setting it to true tells the CDAP UI to show the multiple input schemas coming into a specific plugin, instead of assuming that all of the schemas coming in from different plugins are identical.

This is an optional object, and if it is not present, it is assumed that all of the schemas coming in from any connected plugins are identical. Currently, only one value (multipleInputs) is accepted.

For example:

{
  "metadata": {
    ...
  },
  "configuration-groups": [
    ...
  ],
  "inputs": {
    "multipleInputs": true
  },
  "outputs": [
    ...
  ]
}

Outputs

The outputs is a list of plugin properties that represent the output schema of a particular plugin.

The output schema for a plugin can be represented in two different ways, either:

  • via an explicit schema using a named property; or
  • via an implicit schema

Output properties are configured in a similar manner as individual properties in configuration groups. They are composed of a name and a widget-type, one of either schema (for an explicit schema) or non-editable-schema-editor (for an implicit schema).

With the schema widget type, a list of widget attributes can be included; with the non-editable-schema-editor, a schema to be displayed is added instead.

An explicit schema using a property can be defined as the output schema and then will be editable through the CDAP UI.

For example, a "Batch Source" plugin could have a configurable output schema named data-format, displayed for editing with the schema widget-type, with a default type of string, and a list of the types that are available:

{
  "outputs": [
    {
      "name": "data-format",
      "widget-type": "schema",
      "widget-attributes": {
        "schema-default-type": "string",
        "schema-types": [
          "boolean",
          "int",
          "long",
          "float",
          "double",
          "string",
          "map<string, string>"
        ]
      }
    }
  ]
}

An implicit schema is a pre-determined output schema for a plugin that the plugin developer enforces. The implicit schema is not associated with any properties of the plugin, but instead shows the output schema of the plugin, to be used for visual display only.

An example of this is from the KeyValueTable Batch Source plugin:

{
  "outputs": [
    {
      "widget-type": "non-editable-schema-editor",
      "schema": {
        "name": "etlSchemaBody",
        "type": "record",
        "fields": [
          {
            "name": "key",
            "type": "bytes"
          },
          {
            "name": "value",
            "type": "bytes"
          }
        ]
      }
    }
  ]
}

Widget types for output properties are limited to ensure that the schema that is propagated across different plugins in the CDAP UI is consistent.

Stream and Dataset Jumps

Beginning with version 1.3 of the specification, a plugin can be specified (using jump-config) with a map of stream and dataset "jumps". They specify which plugin property names are either a stream or dataset that can be used, in the CDAP UI, to directly jump to a detailed view of the stream or dataset.

This is an optional object, and if it is not present, no jump links will be created in the CDAP UI. Jump links are not active in the CDAP Studio.

For example:

{
  "metadata": {
    ...
  },
  "configuration-groups": [
    {
      "label": "KV Table Properties",
      "properties": [
        {
          "widget-type": "dataset-selector",
          "label": "Dataset Name",
          "name": "datasetName"
        }
      ]
    }
  ],
  "outputs": [
    {
      "widget-type": "non-editable-schema-editor",
      "schema": {
        "name": "etlSchemaBody",
        "type": "record",
        "fields": [
          {
            "name": "key",
            "type": "bytes"
          },
          {
            "name": "value",
            "type": "bytes"
          }
        ]
      }
    }
  ],
  "jump-config": {
    "datasets": [{
      "ref-property-name": "datasetName"
    }]
  }
}

In this example, the datasetName field of the dataset-selector will have a "jump" link added in the CDAP UI.

Example Widget JSON

Based on the above specification, we can write a widget JSON for a Batch Source plugin (with the properties of name, basePath, duration, delay, groupByFields, aggregates, and an editable output explicit schema) as:

{
  "metadata": {
    "spec-version": "<spec-version>"
  },
  "configuration-groups": [
    {
      "label": "Batch Source",
      "properties": [
        {
          "widget-type": "dataset-selector",
          "name": "name"
        },
        {
          "widget-type": "textbox",
          "name": "basePath"
        },
        {
          "widget-type": "textbox",
          "name": "duration"
        },
        {
          "widget-type": "textbox",
          "name": "delay"
        },
        {
          "widget-type": "textbox",
          "name": "groupByFields",
          "plugin-function": {
            "method": "POST",
            "widget": "outputSchema",
            "output-property": "schema",
            "plugin-method": "outputSchema",
            "required-fields": ["groupByFields", "aggregates"],
            "missing-required-fields-message":
              "Both 'Group By Fields' and 'Aggregates' properties are required to fetch the schema."
          }
        },
        {
          "widget-type": "keyvalue-dropdown",
          "name": "aggregates",
          "widget-attributes": {
            "showDelimiter": "false",
            "kv-delimiter" : ":",
            "delimiter" : ";",
            "dropdownOptions": [
              "Avg",
              "Count",
              "First",
              "Last",
              "Max",
              "Min",
              "Stddev",
              "Sum",
              "Variance"
            ]
          }
        }
      ]
    }
  ],
  "outputs": [
    {
      "name": "schema",
      "widget-type": "schema",
      "widget-attributes": {
        "schema-default-type": "string",
        "schema-types": [
          "boolean",
          "int",
          "long",
          "float",
          "double",
          "string",
          "map<string, string>"
        ]
      }
    }
  ]
}

Specification Changes

These changes describe changes added with each version of the specification.

  • 1.1: Initial version of the specification.
  • 1.2: Added multiple inputs for a plugin.
  • 1.3: Added jump-config to specify which property names are to be connected in the CDAP UI to a detailed view of a stream or dataset.
  • 1.4: Added widget-type: hidden.