Alexa.UIController Interface 3.1

Implement the Alexa.UIController interface in your Alexa skill so that users can voice-control onscreen navigation. You can voice-enable the user interface of entertainment devices, such as televisions and game consoles. For details about video skills, see Understand the Video Skill API. When you implement the Alexa.UIController interface, you can send Alexa the full details of all user interface elements that appear on the screen, such as buttons, dialogues with options, menu items, and search results. For example, you can send a ChangeReport when a dialog appears on the screen with the options continue watching and exit. After that, the user can say For more scenarios, see the Alexa.UIController Interface for Fire TV. If you want to voice-enable onscreen navigation without sending the full user interface details, implement the Alexa.KeypadController interface instead. For the list of languages that the UIController interface supports, see List of Alexa Interfaces and Supported Languages. For the definitions of the message properties, see Alexa Interface Message and Property Reference.

Utterances

The Alexa.UIController interface uses the pre-built voice interaction model. After the user says one of the following utterances, Alexa sends a corresponding directive to your skill. The following examples show some user utterances:

English (All)

If you want users to be able to say or , also implement the Alexa.KeypadController interface.

Properties and objects

Reportable properties

The Alexa.UIController interface supports the focusedUIElement and uiElements properties. You identify that you support the properties in your discovery response.

FocusedUIElement property

The Alexa.UIController interface uses the focusedUIElement property to represent the element on the user interface that has the focus. The focusedUIElement property value is an object. The following table shows the definition of the FocusedUIElement object.

Property	Description	Type	Required
`scene`	Identifying information about the user interface that currently appears on the user’s screen.	`Scene` object	Yes
`element`	Identifies the element in the user interface that has the focus.	`Element` object	Yes

UiElements property

The Alexa.UIController interface uses the uiElements property to represent all the elements in the current user interface. The uiElements property value is an object. The following table shows the definition of the UiElements object.

Property	Description	Type	Required
`scene`	Identifying information about the user interface that currently appears on the user’s screen.	`Scene` object	Yes
`elements`	Identifies the element in the user interface that has the focus.	Array of `Element` objects	Yes

Element object

The Alexa.UIController interface uses the Element object to represent an individual item that appears on the user’s screen. The following example shows a video element.

{
  "element": {
    "elementId": "elementId-003",
    "ordinal": 3,
    "uiSupportedActions": ["SELECT"],
    "entity": {
      "type": "AMAZON.VideoObject",
      "name": {
        "value": "The Dressmaker"
      },
      "externalIds": {
        "entityId": "video-ghi"
      }
    }
  }
}

The following table shows the definition of the Element object.

Property	Description	Type	Required
`elementId`	Unique identifier for the user interface element in this scene.	String	Yes
`ordinal`	Number of the element when the user can refer to the element by number, such as	Integer	No
`uiSupportedActions`	List of actions that a user can perform on the element. Valid values: `SELECT`, `EXPAND`, `SCROLL_RIGHT`, `SCROLL_LEFT`, `SCROLL_UP`, `SCROLL_DOWN`, `SCROLL_FORWARD`, `SCROLL_BACKWARD`.	Array of strings	Yes
`entity`	Information about the element.	`Entity` object	Yes
`elements`	Child elements of the element. Include the `element.elements` field for element objects in the `uiElements` property only.	Array of `Element` objects	No

Entity object

The Alexa.UIController interface uses the Entity object to represent the content of a UI element. The following example shows a video entity.

{
  "entity": {
    "type": "AMAZON.VideoObject",
    "name": {
      "value": "The Marvelous Mrs. Maisel",
      "variants": ["Mrs. Maisel"]
    },
    "externalIds": {
      "entityId": "video-1"
    }
  }
}

The following table shows the definition of the Entity object.

Property	Description	Type	Required
`type`	Describes the UI element. For more details, see Entity type values. Valid values: `AMAZON.ImageObject`, `AMAZON.ItemList`, `AMAZON.SoftwareApplication`, `AMAZON.Thing`, `AMAZON.VideoObject`.	String	Yes
`name`	Name of the entity	Object	No
`name.value`	Preferred name of the entity, such as “The Marvelous Mrs. Maisel.”	String	Yes
`name.variants`	List of alternate names for the entity, such as “Mrs. Maisel.”	Array of strings	No
`externalIds`	External identifiers for the entity.	Map<String, String>	No

Entity type values

The following table shows the string values for the type property.

Value	Description
`AMAZON.ImageObject`	An image or photo.
`AMAZON.ItemList`	A collection of entities of the same or different types, such as a collection of photos.
`AMAZON.SoftwareApplication`	An app, such as Netflix.
`AMAZON.Thing`	A type that you can use when the entity doesn’t fit the other type categories.
`AMAZON.VideoObject`	A video item rendered on a UI. This entity is an abstraction for various types of videos, such as movies, trailers, TV shows.

Scene object

The Alexa.UIController interface uses the Scene object to identify the user interface that currently appears on the user’s screen. The following example shows the home screen.

{
  "scene": {
    "sceneId": "Home Screen 1234"
  }
}

The following table shows the definition of the Scene object.

Property	Description	Type	Required
`sceneId`	Unique identifier for a set of UI elements that appear on the user’s screen.	String	Yes

Discovery

You describe endpoints that support Alexa.UIController by using the standard discovery mechanism described in Alexa.Discovery. Set retrievable to true for the properties that you report when Alexa sends your skill a state report request. Set proactivelyReported to true for the properties that you proactively report to Alexa in a change report. For the full list of display categories, see display categories.

Discover response example

The following example shows a Discover.Response message for a device that supports the Alexa.UIController and Alexa.PowerController interfaces.

{
  "event": {
    "header": {
      "namespace": "Alexa.Discovery",
      "name": "Discover.Response",
      "payloadVersion": "3",
      "messageId": "Unique identifier, preferably a version 4 UUID"
    },
    "payload": {
      "endpoints": [
        {
          "endpointId": "Unique ID of the endpoint",
          "manufacturerName": "Manufacturer of the endpoint",
          "description": "Description to be shown in the Alexa app",
          "friendlyName": "Device name, displayed in the Alexa app",
          "displayCategories": ["TV"],
          "cookie": {},
          "capabilities": [
            {
              "type": "AlexaInterface",
              "interface": "Alexa.UIController",
              "version": "3.1",
              "properties": {
                "supported": [
                  {
                    "name": "uiElements"
                  },
                  {
                    "name": "focusedUIElement"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": false
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa.PowerController",
              "version": "3",
              "properties": {
                "supported": [
                  {
                    "name": "powerState"
                  }
                ],
                "proactivelyReported": true,
                "retrievable": true
              }
            },
            {
              "type": "AlexaInterface",
              "interface": "Alexa",
              "version": "3"
            }
          ]
        }
      ]
    }
  }
}

Directives

Alexa sends the following Alexa.UIController interface directives to your skill.

ActionOnUIElement directive

Support the ActionOnUIElement directive so that users can voice-control your user interface.

ActionOnUIElement directive example

The following example shows a ActionOnUIElement directive that Alexa sends to your skill.

{
    "directive": {
        "header": {
            "namespace": "Alexa.UIController",
            "name": "ActionOnUIElement",
            "messageId": "Unique version 4 UUID",
            "correlationToken": "Opaque correlation token",
            "payloadVersion": "3.1"
        },
        "endpoint": {
            "endpointId": "Endpoint id",
            "cookie": {}
        },
        "payload": {
            "scene": {
                "sceneId": "Home Screen 1234"
            },
            "action": "SCROLL_FORWARD",
            "element": {
                "elementId": "list-001",
                "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
                "ordinal": 12,
                "entity": {
                    "type": "AMAZON.ItemList",
                    "name": {
                        "value": "Suggested for You",
                        "variants": ["Suggested"]
                    }
                }
            }
        }
    }
}

ActionOnUIElement directive payload

The following table shows the payload details for the ActionOnUIElement directive.

Property	Description	Type	Required
`scene`	Identifies the user interface scene that the user is trying to act on.	`Scene` object	Yes
`action`	Identifies the action that the user wants to perform. Valid values: `SELECT`, `EXPAND`, `SCROLL_RIGHT`, `SCROLL_LEFT`, `SCROLL_UP`, `SCROLL_DOWN`, `SCROLL_FORWARD`,`SCROLL_BACKWARD`.	String	Yes
`element`	Identifies the element that the user wants to perform the action on.	`Element` object	Yes

ActionOnUIElement response

If you handle a ActionOnUIElement directive successfully, respond with an Alexa.Response event. In the context object, include the values of all relevant properties. The following example shows a ActionOnUIElement response.

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "Response",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {}
}

ActionOnUIElement directive error handling

If you can’t handle a ActionOnUIElement directive successfully and the error is specific to video, respond with an Alexa.Video.ErrorResponse event. For general errors, respond with a generic Alexa.ErrorResponse event.

State reporting

Alexa sends a ReportState directive to request information about the state of an endpoint. When Alexa sends a ReportState directive, you send a StateReport event in response. The response contains the current state of all the retrievable properties in the context object. You identify your retrievable properties in your discovery response. For details about state reports, see Understand State and Change Reporting.

StateReport response example

{
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "StateReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "correlationToken": "Opaque correlation token that matches the request",
      "payloadVersion": "3"
    },
    "endpoint": {
      "endpointId": "Endpoint id"
    },
    "payload": {}
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "OFF",
        "timeOfSample": "2020-02-03T16:20:50.52Z",
        "uncertaintyInMilliseconds": 0
      }
    ]
  }
}

Change reporting

You send a ChangeReport event to report changes proactively in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.

ChangeReport event example

{
    "event": {
        "header": {
            "namespace": "Alexa",
            "name": "ChangeReport",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2.0 bearer token"
            },
            "endpointId": "Endpoint id"
        },
        "payload": {
            "change": {
                "cause": {
                    "type": "VOICE_INTERACTION"
                },
                "properties": [{
                        "namespace": "Alexa.UIController",
                        "name": "uiElements",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "elements": [{
                                "elementId": "list-001",
                                "ordinal": 12,
                                "uiSupportedActions": ["SCROLL_FORWARD", "SCROLL_RIGHT"],
                                "entity": {
                                    "type": "AMAZON.ItemList",
                                    "name": {
                                        "value": "Suggested for You",
                                        "variants": ["Suggested"]
                                    }
                                },
                                "elements": [{
                                        "elementId": "elementId-001",
                                        "ordinal": 1,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "Captain Fantastic"
                                            },
                                            "externalIds": {
                                                "entityId": "video-abc"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-002",
                                        "ordinal": 2,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Aeronauts"
                                            },
                                            "externalIds": {
                                                "entityId": "video-def"
                                            }
                                        }
                                    },
                                    {
                                        "elementId": "elementId-003",
                                        "ordinal": 3,
                                        "uiSupportedActions": ["SELECT"],
                                        "entity": {
                                            "type": "AMAZON.VideoObject",
                                            "name": {
                                                "value": "The Dressmaker"
                                            },
                                            "externalIds": {
                                                "entityId": "video-ghi"
                                            }
                                        }
                                    }
                                ]
                            }]
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    },
                    {
                        "namespace": "Alexa.UIController",
                        "name": "focusedUIElement",
                        "value": {
                            "scene": {
                                "sceneId": "Home Screen 1234"
                            },
                            "element": {
                                "elementId": "elementId-003",
                                "ordinal": 3,
                                "uiSupportedActions": ["SELECT"],
                                "entity": {
                                    "type": "AMAZON.VideoObject",
                                    "name": {
                                        "value": "Captain Fantastic"
                                    },
                                    "externalIds": {
                                        "entityId": "video-abc"
                                    }
                                }
                            }
                        },
                        "timeOfSample": "2020-02-03T16:20:50.52Z",
                        "uncertaintyInMilliseconds": 0
                    }
                ]
            }
        }
    },
    "context": {
        "properties": [{
            "namespace": "Alexa.PowerController",
            "name": "powerState",
            "value": "ON",
            "timeOfSample": "2020-02-03T08:00:50.52Z",
            "uncertaintyInMilliseconds": 500
        }]
    }
}

ChangeReport event example to reset UI

To reset the UIController state, for example, when the user opens an app on the device that you don’t control, send a change report with an empty uiElements property.

{  
  "event": {
    "header": {
      "namespace": "Alexa",
      "name": "ChangeReport",
      "messageId": "Unique identifier, preferably a version 4 UUID",
      "payloadVersion": "3"
    },
    "endpoint": {
      "scope": {
        "type": "BearerToken",
        "token": "OAuth2.0 bearer token"
      },
      "endpointId": "Endpoint id"
    },
    "payload": {
      "change": {
        "cause": {
          "type": "VOICE_INTERACTION"
        },
        "properties": [
          {
            "namespace": "Alexa.UIController",
            "name": "uiElements",
            "value": {},
            "timeOfSample": "2020-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
          }
        ]
      }
    }
  },
  "context": {
    "properties": [
      {
        "namespace": "Alexa.PowerController",
        "name": "powerState",
        "value": "ON",
        "timeOfSample": "2020-02-03T10:00:50.52Z",
        "uncertaintyInMilliseconds": 500
      }
    ]
  }
}

Get Started

Connect Your Device to Alexa

Certify Your Device

Developer Reference

Utterances

Properties and objects

Reportable properties

FocusedUIElement property

UiElements property

Element object

Entity object

Entity type values

Scene object

Discovery

Discover response example

Directives

ActionOnUIElement directive

ActionOnUIElement directive example

ActionOnUIElement directive payload

ActionOnUIElement response

ActionOnUIElement directive error handling

State reporting

StateReport response example

Change reporting

ChangeReport event example

ChangeReport event example to reset UI

​Utterances

​Properties and objects

​Reportable properties

​FocusedUIElement property

​UiElements property

​Element object

​Entity object

​Entity type values

​Scene object

​Discovery

​Discover response example

​Directives

​ActionOnUIElement directive

​ActionOnUIElement directive example

​ActionOnUIElement directive payload

​ActionOnUIElement response

​ActionOnUIElement directive error handling

​State reporting

​StateReport response example

​Change reporting

​ChangeReport event example

​ChangeReport event example to reset UI

​Related topics

Utterances

Properties and objects

Reportable properties

FocusedUIElement property

UiElements property

Element object

Entity object

Entity type values

Scene object

Discovery

Discover response example

Directives

ActionOnUIElement directive

ActionOnUIElement directive example

ActionOnUIElement directive payload

ActionOnUIElement response

ActionOnUIElement directive error handling

State reporting

StateReport response example

Change reporting

ChangeReport event example

ChangeReport event example to reset UI

Related topics