Alexa.UIController Interface 3.1

Implement the Alexa.UIController interface in your Alexa skill so that users can voice-control onscreen navigation. You can voice-enable the user interface of entertainment devices, such as televisions and game consoles. For details about video skills, see Understand the Video Skill API.

When you implement the Alexa.UIController interface, you can send Alexa the full details of all user interface elements that appear on the screen, such as buttons, dialogues with options, menu items, and search results. For example, you can send a ChangeReport when a dialog appears on the screen with the options continue watching and exit. After that, the user can say "Alexa, continue watching." For more scenarios, see the Alexa.UIController Interface for Fire TV.

If you want to voice-enable onscreen navigation without sending the full user interface details, implement the Alexa.KeypadController interface instead.

For the list of languages that the UIController interface supports, see List of Alexa Interfaces and Supported Languages. For the definitions of the message properties, see Alexa Interface Message and Property Reference.

Utterances

The Alexa.UIController interface uses the pre-built voice interaction model. After the user says one of the following utterances, Alexa sends a corresponding directive to your skill.

The following examples show some user utterances:

Alexa, select The Aeronauts.
Alexa, select number one.

If you want users to be able to say "Alexa, go back" or "Alexa, scroll right", also implement the Alexa.KeypadController interface.

Properties and objects

Reportable properties

The Alexa.UIController interface supports the focusedUIElement and uiElements properties. You identify that you support the properties in your discovery response.

FocusedUIElement property

The Alexa.UIController interface uses the focusedUIElement property to represent the element on the user interface that has the focus. The focusedUIElement property value is an object.

The following table shows the definition of the FocusedUIElement object.

Property Description Type Required

scene

Identifying information about the user interface that currently appears on the user's screen.

Scene object

Yes

element

Identifies the element in the user interface that has the focus.

Element object

Yes

UiElements property

The Alexa.UIController interface uses the uiElements property to represent all the elements in the current user interface. The uiElements property value is an object.

The following table shows the definition of the UiElements object.

Property Description Type Required

scene

Identifying information about the user interface that currently appears on the user's screen.

Scene object

Yes

elements

Identifies the element in the user interface that has the focus.

Array of Element objects

Yes

Element object

The Alexa.UIController interface uses the Element object to represent an individual item that appears on the user's screen.

The following example shows a video element.

{
  "element": {
    "elementId": "elementId-003",
    "ordinal": 3,
    "uiSupportedActions": ["SELECT"],
    "entity": {
      "type": "AMAZON.VideoObject",
      "name": {
        "value": "The Dressmaker"
      },
      "externalIds": {
        "entityId": "video-ghi"
      }
    }
  }
}

The following table shows the definition of the Element object.

Property Description Type Required

elementId

Unique identifier for the user interface element in this scene.

String

Yes

ordinal

Number of the element when the user can refer to the element by number, such as "Alexa, select number one."

Integer

No

uiSupportedActions

List of actions that a user can perform on the element.
Valid values: SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD, SCROLL_BACKWARD.

Array of strings

Yes

entity

Information about the element.

Entity object

Yes

elements

Child elements of the element. Include the element.elements field for element objects in the uiElements property only.

Array of Element objects

No

Entity object

The Alexa.UIController interface uses the Entity object to represent the content of a UI element.

The following example shows a video entity.

{
  "entity": {
    "type": "AMAZON.VideoObject",
    "name": {
      "value": "The Marvelous Mrs. Maisel",
      "variants": ["Mrs. Maisel"]
    },
    "externalIds": {
      "entityId": "video-1"
    }
  }
}

The following table shows the definition of the Entity object.

Property Description Type Required

type

Describes the UI element. For more details, see Entity type values.
Valid values: AMAZON.ImageObject, AMAZON.ItemList, AMAZON.SoftwareApplication, AMAZON.Thing, AMAZON.VideoObject.

String

Yes

name

Name of the entity

Object

No

name.value

Preferred name of the entity, such as "The Marvelous Mrs. Maisel."

String

Yes

name.variants

List of alternate names for the entity, such as "Mrs. Maisel."

Array of strings

No

externalIds

External identifiers for the entity.

Map<String, String>

No

Entity type values

The following table shows the string values for the type property.

Value Description

AMAZON.ImageObject

An image or photo.

AMAZON.ItemList

A collection of entities of the same or different types, such as a collection of photos.

AMAZON.SoftwareApplication

An app, such as Netflix.

AMAZON.Thing

A type that you can use when the entity doesn't fit the other type categories.

AMAZON.VideoObject

A video item rendered on a UI. This entity is an abstraction for various types of videos, such as movies, trailers, TV shows.

Scene object

The Alexa.UIController interface uses the Scene object to identify the user interface that currently appears on the user's screen.

The following example shows the home screen.

{
  "scene": {
    "sceneId": "Home Screen 1234"
  }
}

The following table shows the definition of the Scene object.

Property Description Type Required

sceneId

Unique identifier for a set of UI elements that appear on the user's screen.

String

Yes

Discovery

You describe endpoints that support Alexa.UIController by using the standard discovery mechanism described in Alexa.Discovery.

Set retrievable to true for the properties that you report when Alexa sends your skill a state report request. Set proactivelyReported to true for the properties that you proactively report to Alexa in a change report.

For the full list of display categories, see display categories.

Discover response example

The following example shows a Discover.Response message for a device that supports the Alexa.UIController and Alexa.PowerController interfaces.

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa.Discovery",
      "name": "Discover.Response",
      "payloadVersion": "3",
      "messageId": "Unique identifier, preferably a version 4 UUID"
    },
    "payload": {
      "endpoints": [
        {
          "endpointId": "Unique ID of the endpoint",
          "manufacturerName": "Manufacturer of the endpoint",
          "description": "Description to be shown in the Alexa app",
          "friendlyName": "Device name, displayed in the Alexa app",
          "displayCategories": ["TV"],
          "cookie": {},
          "capabilities": [
            {
              "type": "AlexaInterface",
              "interface": "Alexa.UIController",
              "version": "3.1",
              "properties": {
                "supported": [
                  {
                    "name": "uiElements"
                  },
                  {
                    "name": "focusedUIElement"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": false
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa.PowerController",
              "version": "3",
              "properties": {
                "supported": [
                  {
                    "name": "powerState"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": true
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa",
              "version": "3"
            }
          ]
        }
      ]
    }
  }
}

Directives

Alexa sends the following Alexa.UIController interface directives to your skill.

ActionOnUIElement directive

Support the ActionOnUIElement directive so that users can voice-control your user interface.

ActionOnUIElement directive example

The following example shows a ActionOnUIElement directive that Alexa sends to your skill.

{
    "directive": {
        "header": {
            "namespace": "Alexa.UIController",
            "name": "ActionOnUIElement",
            "messageId": "Unique version 4 UUID",
            "correlationToken": "Opaque correlation token",
            "payloadVersion": "3.1"
        },
        "endpoint": {
            "endpointId": "Endpoint id",
            "cookie": {}
        },
        "payload": {
            "scene": {
                "sceneId": "Home Screen 1234"
            },
            "action": "SCROLL_FORWARD",
            "element": {
                "elementId": "list-001",
                "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
                "ordinal": 12,
                "entity": {
                    "type": "AMAZON.ItemList",
                    "name": {
                        "value": "Suggested for You",
                        "variants": ["Suggested"]
                    }
                }
            }
        }
    }
}

ActionOnUIElement directive payload

The following table shows the payload details for the ActionOnUIElement directive.

Property Description Type Required

scene

Identifies the user interface scene that the user is trying to act on.

Scene object

Yes

action

Identifies the action that the user wants to perform.
Valid values: SELECT, EXPAND, SCROLL_RIGHT, SCROLL_LEFT, SCROLL_UP, SCROLL_DOWN, SCROLL_FORWARD,SCROLL_BACKWARD.

String

Yes

element

Identifies the element that the user wants to perform the action on.

Element object

Yes

ActionOnUIElement response

If you handle a ActionOnUIElement directive successfully, respond with an Alexa.Response event. In the context object, include the values of all relevant properties.

The following example shows a ActionOnUIElement response.

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "Response",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {}
}

ActionOnUIElement directive error handling

If you can't handle a ActionOnUIElement directive successfully and the error is specific to video, respond with an Alexa.Video.ErrorResponse event. For general errors, respond with a generic Alexa.ErrorResponse event.

State reporting

Alexa sends a ReportState directive to request information about the state of an endpoint. When Alexa sends a ReportState directive, you send a StateReport event in response. The response contains the current state of all the retrievable properties in the context object. You identify your retrievable properties in your discovery response. For details about state reports, see Understand State and Change Reporting.

StateReport response example

Copied to clipboard.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "StateReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "OFF",
        "timeOfSample": "2020-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 0
      }
    ]
  }
}

Change reporting

You send a ChangeReport event to report changes proactively in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.

ChangeReport event example

Copied to clipboard.

{
    "event": {
        "header": {
            "namespace": "Alexa",
            "name": "ChangeReport",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2.0 bearer token"
            },
            "endpointId": "Endpoint id"
        },
        "payload": {
            "change": {
                "cause": {
                    "type": "VOICE_INTERACTION"
                },
                "properties": [{
                        "namespace": "Alexa.UIController",
                        "name": "uiElements",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "elements": [{
                                "elementId": "list-001",
                                "ordinal": 12,
                                "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
                                "entity": {
                                    "type": "AMAZON.ItemList",
                                    "name": {
                                        "value": "Suggested for You",
                                        "variants": ["Suggested"]
                                    }
                                },
                                "elements": [{
                                        "elementId": "elementId-001",
                                        "ordinal": 1,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "Captain Fantastic"
                                            },
                                            "externalIds": {
                                                "entityId": "video-abc"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-002",
                                        "ordinal": 2,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Aeronauts"
                                            },
                                            "externalIds": {
                                                "entityId": "video-def"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-003",
                                        "ordinal": 3,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Dressmaker"
                                            },
                                            "externalIds": {
                                                "entityId": "video-ghi"
                                            }
                                        }
                                    }
                                ]
                            }]
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    },
                    {
                        "namespace": "Alexa.UIController",
                        "name": "focusedUIElement",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "element": {
                                "elementId": "elementId-003",
                                "ordinal": 3,
                                "uiSupportedActions": ["SELECT"],
                                "entity": {
                                    "type": "AMAZON.VideoObject",
                                    "name": {
                                        "value": "Captain Fantastic"
                                    },
                                    "externalIds": {
                                        "entityId": "video-abc"
                                    }
                                }
                            }
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    }
                ]
            }
        }
    },
    "context": {
        "properties": [{
            "namespace": "Alexa.PowerController",
            "name": "powerState",
            "value": "ON",
            "timeOfSample": "2020-02-03T08:00:50.52Z",
            "uncertaintyInMilliseconds": 500
        }]
    }
}

ChangeReport event example to reset UI

To reset the UIController state, for example, when the user opens an app on the device that you don't control, send a change report with an empty uiElements property.

Copied to clipboard.

{  
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "ChangeReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "OAuth2.0 bearer token"
      },
      "endpointId": "Endpoint id"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "VOICE_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {},
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "ON",
        "timeOfSample": "2020-02-03T10:00:50.52Z",
        "uncertaintyInMilliseconds": 500
      }
    ]
  }
}
Was this page helpful?

Last updated: Aug 23, 2024