How to create dynamic voice commands for RealWear®

Use cases

Dynamic voice commands are generated based on the data that is being displayed to users. This enables developers to follow the say-what-you-see design pattern, where users can look at the screen and speak the words they see in order to control the RealWear HMT device.

Typical use cases for barcode scanning include:

Displaying data in tables: This is useful for many use cases including:

  • Selecting items or parts during work order execution.
  • Selecting team members to call for remote assistance.
  • Selecting installed assets to perform maintenance or inspections on.

More intuitive navigation for integrated apps: Where data is being pulled in from existing systems, such as IBM® Maximo®, SAP®, Salesforce®, CMMS, or DMS systems, dynamic voice commands generate voice commands based on data being pulled in. This allows for more intuitive app navigation where commands can correspond to data being displayed and avoids the need for indicators or generic commands such as “Select item 2”.

What you’ll need

To use dynamic voice commands in your JourneyApps app for RealWear® you’ll need:

  • A RealWear® HMT device (for testing)
  • RealWear Explorer (for installing the JourneyApps APK if you haven’t already done so)
  • A JourneyApps OXIDE account – sign up here if you don’t already have one

Instructions:

Open OXIDE and navigate to the relevant app view

In OXIDE, click the Views tab and select the view where you’d like to implement the barcode scanning component from the list, or create a new view.

image
Views tab selected.

image
Example of views displayed in the left panel when the Views tab is selected.

Add dynamic voice command functionality

Dynamic voice commands make use of the data objects and structure defined in your app’s data model, or schema. Dynamic voice commands are implemented in your view’s logic, which is written in either JavaScript or TypeScript, depending on how you’ve set up your app.

Reset voice service

Once you’ve selected the view you want to implement dynamic voice commands on, the first step is to reset the voice service in order to ensure that any old commands are no longer registered. We’ll define this function under the init() function. The init() function is called whenever a view is navigated to by an app user.

For more best practices, refer to our documentation on RealWear® voice commands.

Here’s how we would do this using TypeScript in the view’s .ts file:

async function init() {
   // Reset voice service
   await journey.voice.reset();
}

Register dynamic voice commands

In our example, we would like to implement dynamic voice commands that allow the user to speak the name attribute of an asset data object in order to select a specific asset from multiple assets displayed in an object table. Once the user says the asset name, we would like to display a photo of that asset stored in the app’s database.

To do this we will define a function that will allow the voice service to listen for voice commands that match the name attributes of the assets being displayed.

We’ll set up this function to populate a constant that we’ll call “unit“ with the asset objects that exist in the app’s database. After that we’ll create another constant called commands that populates a command and callback instruction for the app based on the asset name that was spoken. The app will know to listen for asset names because a command instruction will be created for each asset name that exists in the app’s database. Once an asset’s name has been spoken, we’ll set the callback instruction to call a separate function that we’ll call selectUnit(), which we’ll define later.

Here’s how we would do this using TypeScript in the view’s .ts file:

async function registerUnitListeners() {
   const units = await DB.asset.all().orderBy(name).toArray();
   const commands = units.map((unit: DB.asset) => {
     return {
       command: `${unit.name}`,
       callback: (unit) => selectUnit(unit);,
     };
   });
   await journey.voice.registerCommands(commands);
}

Display asset photo based on dynamic voice command

The next step is to create a function that displays a photo of the asset whose name has been spoken. The photo will be displayed through a display-image UI component, defined in the view’s .xml file. The purpose of this function will therefore be to set the view variable that is referenced by the display-image component as equal to the photo of the asset that has been spoken. We’ve called the view variable asset_photo.

Here’s how we would do this using TypeScript in the view’s .ts file:

function selectUnit(unit: DB.asset) {
   view.asset_photo = unit.photo;
}

Add registerUnitListeners() function to init() function

The final step is to add the registerUnitListeners() function to the init() function to ensure that the dynamic voice commands are active whenever a user navigates to this view.

The full init() function will now look like this:

async function init() {
   // Reset voice service
   await journey.voice.reset();
   // Run dynamic voice commands function
   await registerUnitListerners();
}

Need support?

Documentation:

Questions?

Access OXIDE, our online dev environment.

Last updated: 2021/11/11


JourneyApps provides a rapid way to build custom apps for RealWear® HMT, mobile and desktop. Voice commands are simple to set up and manage, we provide offline support out of the box, and deploying apps happens with a single click. Comes with prebuilt ERP integrations. If you are interested, please contact us to schedule a demo. You can also visit our RealWear page to learn more and subscribe for notifications about new blog posts.


← Back to all posts

Subscribe to our blog

SUBSCRIBE

JourneyApps: The Rapid-Code Platform

To Build, Deploy and Run

Mission-Critical Industrial-Grade Apps

Try For Free