Alexa.SmartVision.ObjectDetectionSensor Interface 1.0

Implement the Alexa.SmartVision.ObjectDetectionSensor interface in your Alexa add-on so that customers can receive notifications when their smart-vision device detects an object, such as a person or package. Customers can enable the class of objects for which they want to receive notifications. Your add-on reports smart-vision events to Alexa to notify the customer that the device detected an object of interest. Typically, you use the Alexa.SmartVision.ObjectDetectionSensor interface with the Alexa.RTCSessionController interface. Also, you can use the Alexa.DataController interface to enable customers to review and delete detection events. For the list of languages that the Alexa.SmartVision.ObjectDetectionSensor interface supports, see List of Alexa Interfaces and Supported Languages. For the definitions of the message properties, see Alexa Interface Message and Property Reference.

Object detection

Object detection is a computer vision technique to identify objects within an image or video stream. Some smart-vision devices can follow an object through a video stream, aggregate the frames of the same physical object, interpret the contents, and report the detected object information. Typically, customers can configure the types of objects (object classes) that they want their device to identify and report. Object detection occurs when the smart-vision device sees a configured object class in the video stream. After your add-on reports the object detection event to Alexa, Alexa notifies the customer. Customers can review the history of reported events in the Alexa app. If your smart-vision device can detect multiple objects in a single video stream, send an object detection event for each object, as soon as detection occurs. Don’t wait for the video processing session to end. As the session continues, your smart-vision processing software can add data to the event, group video frames showing the same physical object, and detect other objects of interest from the same or different object classes in the video stream. You send additional events for other objects of interest that appear in the stream. Event data includes the detection time, object class, unique ID for the detected object, and an image of the object. Alexa caches data associated with the detected event, such as the event ID. When the customer opens the Alexa app to review the detection event, Alexa uses the Alexa.DataController interface to retrieve the event data from your add-on.

Image guidelines

When you report the detected object to Alexa, you include a URI to the image. Alexa downloads and displays the image in the Alexa app and on Alexa-enabled devices with a screen. The Alexa.SmartVision.ObjectDetectionSensor interface supports two types of images:

Cropped image – A close-up view centered on the detected object.
Contextual frame – A wider view showing the detected object within the surrounding scene.

The image must meet the following guidelines for successful download and display in the Alexa app:

Image requirements:
- Supported media types: JPEG or PNG.
- Minimum image resolution: 360 pixels along its shortest side.
- Set a URI expiration time. Amazon recommends that the URI remain valid for 10 minutes.
- For optimal performance, keep image file sizes to 60 KB on average.
Authentication and security requirements:
- The URI must not be publicly accessible.
- Your service must require authentication to access the image.
- Alexa supports access token authentication and sends the access token as part of the Authorization header field when downloading the image from the URI.
- Your service must perform the following checks:
  - Verify that the access token exists.
  - Verify that the token represents a valid user in your resource server.
  - Verify that the given user has the correct access privileges for the given image.
HTTP specification details:
- Alexa doesn’t support URI redirects from the provided URI.
- Alexa downloads the image by sending an HTTP GET request to the URI with the following header fields:
  Accept: image/jpeg, image/png Authorization: Bearer {access token}
- In response, your service returns the image with the Content-type header field set to one of the following media types: image/jpeg or image/png.
  If not specified, the default is image/png.
- If the URI becomes invalid or expires, return HTTP Status 410.
To use a presigned URI instead of access tokens, contact Amazon developer support.

Utterances

The Alexa.SmartVision.ObjectDetectionSensor interface doesn’t define any user utterances. Instead, Alexa communicates with your add-on about the object detection classes that the customer configures in the Alexa app.

Reportable properties

The Alexa.SmartVision.ObjectDetectionSensor interface doesn’t define any reportable properties.

Discovery

You describe endpoints that support Alexa.SmartVision.ObjectDetectionSensor by using the standard discovery mechanism described in Alexa.Discovery. For smart-vision devices, use CAMERA for the display category. For video doorbells with a camera, use both CAMERA and DOORBELL for the display categories. For the full list of display categories, see display categories. Sensor devices must also implement Alexa.EndpointHealth.

Configuration object

In addition to the usual discovery response fields, for Alexa.SmartVision.ObjectDetectionSensor, include a configuration object that contains the following fields.

Property	Description	Type	Required
`objectDetectionConfiguration`	Object detection classes that the endpoint supports.	Array of objects	Yes
`objectDetectionConfiguration[*].imageNetClass`	Type of image that the endpoint can detect. Valid values: `person`, `package`.	String	Yes
`objectDetectionConfiguration[*].isAvailable`	Indicates whether you can enable the class on the endpoint. Default value: `true`.	Boolean	No
`objectDetectionConfiguration[*].unavailabilityReason`	Indicates why the class isn’t available on the endpoint. Valid values: `SUBSCRIPTION_REQUIRED`.	String	No

Discover response example

The following example shows a Discover.Response message for an Alexa add-on that supports the Alexa.SmartVision.ObjectDetectionSensor, Alexa.DataController, and Alexa.EndpointHealth interfaces.

{
    "event": {
        "header": {
            "namespace": "Alexa.Discovery",
            "name": "Discover.Response",
            "payloadVersion": "3",
            "messageId": "Unique identifier, preferably a version 4 UUID"
        },
        "payload": {
            "endpoints": [{
                "endpointId": "Unique ID of the endpoint",
                "manufacturerName": "Sample Manufacturer",
                "description": "Description that appears in the Alexa app",
                "friendlyName": "Your device name, displayed in the Alexa app",
                "displayCategories": ["CAMERA"],
                "additionalAttributes": {
                    "manufacturer": "Sample Manufacturer",
                    "model": "Sample Model",
                    "serialNumber": "Serial number of the device",
                    "firmwareVersion": "Firmware version of the device",
                    "softwareVersion": "Software version of the device",
                    "customIdentifier": "Optional custom identifier for the device"
                },
                "cookie": {},
                "capabilities": [{
                        "type": "AlexaInterface",
                        "interface": "Alexa.SmartVision.ObjectDetectionSensor",
                        "version": "1.0",
                        "properties": {},
                        "configuration": {
                            "objectDetectionConfiguration": [{
                                    "imageNetClass": "person"
                                },
                                {
                                    "imageNetClass": "package",
                                    "isAvailable": false,
                                    "unavailabilityReason": "SUBSCRIPTION_REQUIRED"
                                }
                            ]
                        }
                    },
                    {
                        "type": "AlexaInterface",
                        "interface": "Alexa.DataController",
                        "instance": "Camera.SmartVisionData",
                        "version": "1.0",
                        "properties": {},
                        "configuration": {
                            "targetCapability": {
                                "name": "Alexa.SmartVision.ObjectDetectionSensor",
                                "version": "1.0"
                            },
                            "dataRetrievalSchema": {
                                "type": "JSON",
                                "schema": "SmartVisionData"
                            },
                            "supportedAccess": ["BY_TIMESTAMP_RANGE"]
                        }
                    },
                    {
                        "type": "AlexaInterface",
                        "interface": "Alexa.EndpointHealth",
                        "version": "3.1",
                        "properties": {
                            "supported": [{
                                "name": "connectivity"
                            }],
                            "proactivelyReported": true,
                            "retrievable": true
                        }
                    },
                    {
                        "type": "AlexaInterface",
                        "interface": "Alexa",
                        "version": "3"
                    }
                ]
            }]
        }
    }
}

AddOrUpdateReport

You must proactively send an Alexa.Discovery.AddOrUpdateReport event if the feature support of your endpoint changes. For example, if the subscription status of a supported object class changes. For details, see AddOrUpdateReport event.

AddOrUpdateReport event example

The following example shows an AddOrUpdateReport message to report the package class no longer requires a subscription.

{
    "event": {
        "header": {
            "namespace": "Alexa.Discovery",
            "name": "AddOrUpdateReport",
            "payloadVersion": "3",
            "messageId": "Unique identifier, preferably a version 4 UUID"
        },
        "payload": {
            "endpoints": [{
                "endpointId": "Unique ID of the endpoint",
                "manufacturerName": "Sample Manufacturer",
                "description": "Description that appears in the Alexa app",
                "friendlyName": "Your device name, displayed in the Alexa app",
                "displayCategories": ["CAMERA"],
                "additionalAttributes": {
                    "manufacturer": "Sample Manufacturer",
                    "model": "Sample Model",
                    "serialNumber": "Serial number of the device",
                    "firmwareVersion": "Firmware version of the device",
                    "softwareVersion": "Software version of the device",
                    "customIdentifier": "Optional custom identifier for the device"
                },
                "cookie": {},
                "capabilities": [{
                        "type": "AlexaInterface",
                        "interface": "Alexa.SmartVision.ObjectDetectionSensor",
                        "version": "1.0",
                        "properties": {},
                        "configuration": {
                            "objectDetectionConfiguration" : [
                                {
                                   "imageNetClass" : "person"
                                },
                                {
                                   "imageNetClass" : "package"
                                }
                            ]
                        }
                    },
                    {
                        "type": "AlexaInterface",
                        "interface": "Alexa.EndpointHealth",
                        "version": "3.1",
                        "properties": {
                            "supported": [{
                                "name": "connectivity"
                            }],
                            "proactivelyReported": true,
                            "retrievable": true
                        }
                    },
                    {
                        "type": "AlexaInterface",
                        "interface": "Alexa",
                        "version": "3"
                    }
                ]
            }]
        }
    }
}

Directives and events

The Alexa.SmartVision.ObjectDetectionSensor interface defines the following directives and events.

SetObjectDetectionClasses directive

Support the SetObjectDetectionClasses directive so that the customer can configure the objects for which they want to receive notifications. The customer can configure the object classes in the Alexa app. For this endpoint, you must disable events for any object class that isn’t included in the request.

SetObjectDetectionClasses directive example

The following example shows a SetObjectDetectionClasses directive that Alexa sends to your add-on. This example enables detection of objects in the person and package classes.

{
    "directive": {
        "header": {
            "namespace": "Alexa.SmartVision.ObjectDetectionSensor",
            "name": "SetObjectDetectionClasses",
            "payloadVersion": "1.0",
            "messageId": "Unique version 4 UUID",
            "correlationToken": "Opaque correlation token"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2 bearer token"
            },
            "endpointId": "endpoint id",
            "cookie": {}
        },
        "payload": {
            "objectDetectionClasses": [{
                    "imageNetClass": "person"
                },
                {
                    "imageNetClass": "package"
                }
            ]
        }
    }
}

SetObjectDetectionClasses directive payload

The following table shows the payload details for the SetObjectDetectionClasses directive that Alexa sends to your add-on.

Property	Description	Type	Required
`objectDetectionClasses`	Types of physical objects that the customer wants the camera to detect. You must disable object detection for any other object classes that your smart-vision camera supports.	Array of objects	Yes
`objectDetectionClasses[].imageNetClass`	Image class. Valid values: `person`, `package`.	String

SetObjectDetectionClasses response

If you handle a SetObjectDetectionClasses directive successfully and you can configure events for the requested object classes, respond with an Alexa.Response. The following example shows a successful response to the SetObjectDetectionClasses directive.

{
    "event": {
        "header": {
            "namespace": "Alexa",
            "name": "Response",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "correlationToken": "Opaque correlation token that matches the request",
            "payloadVersion": "3"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2.0 bearer token"
            },
            "endpointId": "endpoint id",
            "cookie": {}
        },
        "payload": {}
    },
    "context": {
        "properties": [{
            "namespace": "Alexa.EndpointHealth",
            "name": "connectivity",
            "value": {
                "value": "OK"
            },
            "timeOfSample": "2017-02-03T16:20:50.52Z",
            "uncertaintyInMilliseconds": 0
        }]
    }
}

SetObjectDetectionClasses directive error handling

If you can’t handle a SetObjectDetectionClasses directive successfully, respond with an Alexa.SmartVision.ObjectDetectionSensor.ErrorResponse event. You can also respond with a generic Alexa.ErrorResponse event if your error isn’t specific to object detection.

ObjectDetection event

Send the ObjectDetection event to the Alexa Event Gateway when your device recognizes an object from one of the configured object classes. For details, see Send Events to the Event Gateway. On receipt of the event, Alexa notifies the customer about the detected object. Also, Alexa caches the event data by eventIdentifier and endpointId so that the customer can later view and delete the event data. Assign a unique eventIdentifier for each detected object in the video stream and send one event per detected object. Also, produce at most one detection event per detected object class and video stream. After your add-on reports an ObjectDetection event, you must wait at least 30 seconds before sending another ObjectDetection event for the same object.

ObjectDetection event example

The following example shows an ObjectDetection event that you send to Alexa. This example reports the detection of an object in the person class.

 {
    "event": {
        "header": {
            "namespace": "Alexa.SmartVision.ObjectDetectionSensor",
            "name": "ObjectDetection",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "payloadVersion": "1.0"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "OAuth2 bearer token"
            },
            "endpointId": "endpoint id"
        },
        "payload": {
            "events": [{
                "eventIdentifier": "2b3409df-d686-4a52-9bba-d361860bac61",
                "imageNetClass": "person",
                "timeOfSample": "2021-07-02T16:20:50.52Z",
                "uncertaintyInMilliseconds": 0,
                "objectIdentifier": "573409df-5486-7c52-b4ba-d361860bac73",
                "frameImageUri": "https://example.com/frames/frame1.jpg",
                "croppedImageUri": "https://example.com/images/image1.jpg"
            }]
        }
    }
}

ObjectDetection event payload

The following table shows the payload details for the ObjectDetection event.

Property	Description	Type	Required
`events`	Objects that the endpoint detected.	Array of objects	Yes
`events[*].eventIdentifier`	Uniquely identifies the event in the event history. You can use the ID to retrieve and delete the event from the camera stream. Generate an event ID for each detected object.	Version 4 UUID String	Yes
`events[*].imageNetClass`	Type of the detected object. Valid values: `person`, `package`.	String	Yes
`events[*].objectIdentifier`	Uniquely identifies the physical object detected in this session. Generate an ID for each detected object. If your device doesn’t distinguish between detected objects, don’t include this ID.	Version 4 UUID String	No
`events[*].frameImageUri`	Specifies the URI to the image of the detected object. For more details, see Image guidelines.	String	No
`events[*].croppedImageUri`	Specifies the URI to the cropped image of the detected object. For more details, see Image guidelines.	String	No
`events[*].timeOfSample`	Time the endpoint detected the object. Defined in ISO 8601 format, `YYYY-MM-DDThh:mm:ssZ`.	String	Yes
`events[*].uncertaintyInMilliseconds`	Uncertainty of the `timeOfSample` in milliseconds. This field represents the number of milliseconds before or after the endpoint detected the object. For example, uncertainty due to the transmission delay between the action in front of the camera and the corresponding object detection processing software.	Number	No

ObjectDetection response

If your add-on proactively sent an ObjectDetection event and Alexa handles the event successfully, your add-on receives HTTP status code 202 Success. On error, Alexa sends the appropriate HTTP status code. The following table shows the HTTP status codes sent in response to the event.

Status	Description
`202 Success`	Operation succeeded.
`400 Invalid Request`	Indicates that the request is invalid or badly formatted. Verify the event payload and check for any missing or invalid fields.
`401 Unauthorized`	Indicates that the request didn’t include the authorization token or the token is invalid or expired.
`403 Forbidden`	Indicates that the authorization token doesn’t have sufficient permissions or the add-on is disabled.
`404 Not Found`	Indicates that the add-on doesn’t exist in the corresponding stage.
`413 Request Entity Too Large`	Maximum number or size of a parameter exceeds the limit.
`429 Too Many Requests response`	Number of requests per minute is too high. Use exponential back-off and retry the request.
`500 Internal Server Error`	An error occurred on the server. The add-on can retry by using exponential back-off.
`503 Service Unavailable`	Server is busy or unavailable. The add-on can retry by using exponential back-off.

Event update and deletion

As soon as your device recognizes a configured object, you send the ObjectDetection event to Alexa. On receipt of the event, Alexa sends a notification to the customer. As the video stream continues, you can update the event data by using the Alexa.DataController interface to send a DataReport event to Alexa. For example, you might want to aggregate frames of the same physical object. Alexa doesn’t send a notification when you update the event data. You can also use the Alexa.DataController interface to delete data stored on Alexa. For example, you send a DataDeleted event to Alexa when the customer deletes the detection event directly from your camera or your camera app.

Smart-vision data schema example

You send smart-vision event data as an array in the data property of the DataReport event. The following example shows a DataReport event from a smart camera that includes data for two frames and the associated images of the detected object.

{
    "event": {
        "header": {
            "namespace": "Alexa.DataController",
            "name": "DataReport",
            "instance": "DataController-SmartVisionData",
            "messageId": "Unique identifier, preferably a version 4 UUID",
            "correlationToken": "Opaque correlation token that matches the request",
            "payloadVersion": "1.0"
        },
        "endpoint": {
            "scope": {
                "type": "BearerToken",
                "token": "Opaque correlation token"
            },
            "endpointId": "endpoint id"
        },
        "payload": {
            "paginationContext": {
                "nextToken": "token"
            },
            "dataSchema": {
                "type": "JSON",
                "schema": "SmartVisionData"
            },
            "data": [{
                    "eventIdentifier": "2b3409df-d686-4a52-9bba-d361860bac61",
                    "imageNetClass": "person",
                    "mediaId": "2c3409df-d686-4a52-9bba-d361860bac61",
                    "objectIdentifier": "4a3409df-d686-4a52-9bba-d361860bacbf",
                    "frameIndex": 2,
                    "frameWidthInPixels": 1980,
                    "frameHeightInPixels": 1080,
                    "frameImageUri": "https://example.com/frames/frame1.jpg",
                    "croppedImageUri": "https://example.com/images/image1.jpg"
                },
                {
                    "eventIdentifier": "2b3409df-d686-4a52-9bba-d361860bac61",
                    "imageNetClass": "person",
                    "mediaId": "2c3409df-d686-4a52-9bba-d361860bac62",
                    "objectIdentifier": "4a3409df-d686-4a52-9bba-d361860bacbf",
                    "frameIndex": 5,
                    "frameWidthInPixels": 1980,
                    "frameHeightInPixels": 1080,
                    "frameImageUri": "https://example.com/frames/frame2.jpg",
                    "croppedImageUri": "https://example.com/images/image2.jpg"
                }
            ]
        }
    }
}

Smart-vision data schema definition

The following table shows the JSON data schema defined by the Alexa.SmartVision.ObjectDetectionSensor interface.

Property	Description	Type	Required
`eventIdentifier`	Uniquely identifies the event in the event history. You can send updated data for the same camera session and customer.	Version 4 UUID String	Yes
`imageNetClass`	Type of detected object. Valid values: `person`, `package`.	String	Yes
`mediaId`	Uniquely identifies the media recording in which the event occurred.	Version 4 UUID String	No
`frameIndex`	Frame number.	Integer	No
`frameWidthInPixels`	Width of the frame in pixels.	Integer	No
`frameHeightInPixels`	Height of the frame in pixels.	Integer	No
`objectIdentifier`	Uniquely identifies the physical object detected in this session. Generate an ID for each detected object. If your device doesn’t distinguish between detected objects, don’t include this ID.	Version 4 UUID String	No
`frameImageUri`	Specifies the URI to the image of the detected object. For more details, see Image guidelines.	String	No
`croppedImageUri`	Specifies the URI to the cropped image of the detected object. For more details, see Image guidelines.	String	No

Change reporting

You send a ChangeReport event to report changes proactively in the state of an endpoint. You identify the properties that you proactively report in your discovery response. For details about change reports, see Understand State and Change Reporting.

Get Started

Connect Your Device to Alexa

Certify Your Device

Developer Reference

Alexa.SmartVision.ObjectDetectionSensor Interface 1.0

Object detection

Image guidelines

Utterances

Reportable properties

Discovery

Configuration object

Discover response example

AddOrUpdateReport

AddOrUpdateReport event example

Directives and events

SetObjectDetectionClasses directive

SetObjectDetectionClasses directive example

SetObjectDetectionClasses directive payload

SetObjectDetectionClasses response

SetObjectDetectionClasses directive error handling

ObjectDetection event

ObjectDetection event example

ObjectDetection event payload

ObjectDetection response

Event update and deletion

Smart-vision data schema example

Smart-vision data schema definition

Change reporting

​Object detection

​Image guidelines

​Utterances

​Reportable properties

​Discovery

​Configuration object

​Discover response example

​AddOrUpdateReport

​AddOrUpdateReport event example

​Directives and events

​SetObjectDetectionClasses directive

​SetObjectDetectionClasses directive example

​SetObjectDetectionClasses directive payload

​SetObjectDetectionClasses response

​SetObjectDetectionClasses directive error handling

​ObjectDetection event

​ObjectDetection event example

​ObjectDetection event payload

​ObjectDetection response

​Event update and deletion

​Smart-vision data schema example

​Smart-vision data schema definition

​Change reporting

​Related topics

Object detection

Image guidelines

Utterances

Reportable properties

Discovery

Configuration object

Discover response example

AddOrUpdateReport

AddOrUpdateReport event example

Directives and events

SetObjectDetectionClasses directive

SetObjectDetectionClasses directive example

SetObjectDetectionClasses directive payload

SetObjectDetectionClasses response

SetObjectDetectionClasses directive error handling

ObjectDetection event

ObjectDetection event example

ObjectDetection event payload

ObjectDetection response

Event update and deletion

Smart-vision data schema example

Smart-vision data schema definition

Change reporting

Related topics