Human-Robot Interface: Efficient Control and Flight Data Display of a Quadcopter through Google Glass
Design Document v2
April 1, 2015
Prepared By: Prepared For:
Daniel Chen ECS193A/B
Laura Demesa Client:
Steven Rodriguez Dr. Ilias Tagkopoulos
Table of Contents
Technology Background 2
Google Glass 2
DJI Phantom Vision 2 3
Design Overview 5
Project Tasks and Flow 6
UAV and Glass 7
Glass and Database 8
Data Transfer 9
Data Storage 10
Gimbal Controls 11
Camera Controls 11
Glass UI 12
Database UI 12
Additional Notes 13
The Human-Robot Interface project address the desire to have a more hands-free experience when controlling an unmanned aerial vehicle (UAV). Modern UAV technology requires the use of physical devices to control not only the UAV but also attachments such as a camera. This project reduces the need for UAV operators to use their hands when controlling the camera and storing flight information to a database. The project accomplishes this by using Google Glass’s motion controls and wireless connectivity.
This document details the technology required and software design. The project’s design consist of four categories: communications, data, controls, and interface. Each segment explains the design choices, decisions, and implementation.
Technology Background Requirements
The human-robot interface project has two technology requirements. First, the device to interface with the UAV is Google Glass. Second, the UAV must be a programmable quadcopter capable of providing flight data and connecting with Google Glass.
Google Glass is a wearable electronic device that fits similar to a pair of glasses. Google Glass’s display screen rest slightly above eye level in front of one eye. Google Glass has these additional tech specs:
Wi-Fi - 802.11 b/g 2.4GHz
16 GB Flash Memory
Can be paired with an Android (4.0.3 or later) or iOS (7 or later) device.
Programming applications for Google Glass requires the Glass Development Kit, which is an additional package for Android Studio. Developing for Glass is similar to Android development, which is a combination of Java and XML. However, Glass’s user interface has a specific structure that differs from the typical Android device.
Glass introduces the concept of timeline, which manages processes called cards that can be live or static. Timeline allows the user to switch between different live cards, which update frequently and have access to low-level sensor data, and static cards, which rarely, if at all, update. Switching between processes in the timeline is analogous to tabbing through program windows on typical operating systems.
In order for a process to have full control, Glass uses the concept immersion, which runs a process outside the timeline and allows developers the ability to create their own UI and process the entire user input.
In addition to running native applications on Glass, users can install regular android software in a process known as sideloading. This requires access to the Glass debug bridge, where a user can manually load files to Glass, and the compiled .apk Android program. However, sideloading software does not guarantee compatibility with Glass. Therefore, undesired results may occur when using this method.
DJI Phantom 2 Vision
The Human-Robot Interface project uses the Phantom 2 Vision quadcopter made by DJI as its UAV. We selected the Phantom because it met the programmability and connectivity requirements of the project. We evaluated alternative quadcopters, but many did not meet the programmability requirement. Prior to selecting the Phantom, we used Parrot’s Bebop quadcopter; however, the quadcopter did not perform as described and lacked critical developer documentation. DJI offers a software development kit (SDK) to help developers.
The Phantom 2 Vision transmits its own Wi-Fi signal to a controller, which then extends the signal for other devices to connect. This Wi-Fi extender is how Google Glass and other devices can connect.
DJI restricts their SDK by application approval and offers several tiers of access. The first tier offers video uplink, gimbal control, camera controls, flight OSD, and battery. The second tier offers all the features of the first tier and flight controls. The second tier may require payment to use. However, this project only uses the first tier.
The Human-Robot Interface project has the following design goals:
Establish a connection between the Phantom quadcopter and Google Glass.
Transfer video from the UAV to the Google Glass display.
Transfer flight data from the UAV to Google Glass.
Store the flight data from Google Glass to a database.
Implement Google Glass head tracking to control the UAV’s gimbal.
Implement voice commands that Google Glass will transfer to the UAV.
Create an easy to use user-interface.
Design goals are listed in order of significance. Goals 6 and 7 are considered low priority and not critical to the core project requirements. However, the addition of these features creates a better application overall. In addition, goal four’s design is tentative and likely to change.
Figure 1 illustrates the overall communication and data transfer for the project. Note that the controller is an intermediate to the communication between the Phantom and Glass, which is done through the Wi-Fi extender. All communication is through wireless connections.
The project application is a native android application (i.e. .apk) that is sideloaded onto Glass, due to the limitations of the GDK.
Project Tasks and Flow
Figure 2 illustrates the project’s tasks and order of completion. The numbers represent the order of completion. The arrows show dependencies in the project — a task began once all prior tasks before it were completed.
This project requires establishing communication between the several devices involved. The connection between the Phantom and the controller is a part of the UAV product and does not need modifying. However, the connection between the Phantom drone and Glass is a project task. In addition, a connection between Glass and a database system is another task for this project.
Drone and Glass
Our project connects Google Glass to the Phantom drone through the controller transmitter. The connection type between the drone and controller is 5.8 GHz Wi-Fi, which is then transmitted to Glass through 2.8Ghz Wi-Fi. Figure 3 illustrates this architecture.
Glass must detect the controller to connect Glass to the drone; Google already implements this requirement. However, the login process requires knowing the ports, keys, and IPs to connect. This information is device specific and we obtained it through the devices. Once obtained by looking up the information, the program submits the login request with the appropriate information from Google Glass. If the information is correct then login is successful.
Once the program establishes a connection with the drone, Glass communicates to Phantom through application programming interface (API) calls. These are functions determined by DJI and returns data values, check values, and initiate commands. The program uses API calls to control the gimbal, transfer video and flight data, and issue camera commands.
There are no other practical ways to connect the Phantom and Glass.
Glass and Database
The setup of a database on a server and connecting Glass to the database is a task for this project. The database will store flight data transferred from Glass.
Note: The following design information on database communication is tentative.
Figure 4 illustrates the current design for database communication.
This architecture allows a user-friendly approach to storing data. Instead of directly communicating with the database, the user will simply just need to connect to the internet and the program automatically transfers the data transfer from the web page and Glass. This will require the creation of a program on Glass that when it detects a connection to the internet it begins to transfer the desired data. This program prepares the data for storage and transferring the data. The program eventually transfers data through HTTP commands such as GET and PUT. However, the program will have custom functions that lead to these commands.