REAL TIME AI to REVOLUTIONIZE BICYCLE SAFETY

The start to changing how we think about recreational safety

AUDIBLE WARNING ALERTS

Acknowledgements

Client: Tony Dobaj

Project Advisor: Robin Steele

Technical Advisor: Dr. Christopher Coulston

Luxonis Support: Brandon Gilles

Video

Elevator Pitch

Imagine biking on a road and a silent electric vehicle is approaching from behind. You can’t see it or hear it, but a device on your bicycle detects the vehicle and notifies you, without needing to take your eyes off the road in front of you. This capability is realized in the Gestr Hazard Notification System, a cutting-edge device in road safety. This product attaches securely to the back of a bicycle to detect and warn of any approaching hazard that is on a collision course with the rider. One day, we believe this system could be an integral and important part of riding safety, much like a helmet.

The Gestr HNS uses advanced algorithms utilizing machine learning and AI to enhance rider safety. The prototype detects vehicles, people, cyclists, and other potential hazards coming towards the rider from behind, all in real-time. The system uses artificial intelligence to classify a hazard and determine if it is currently (or likely to become) a threat to the rider, based on the object’s identity, velocity, and trajectory. This data can be sent to the user’s smartphone, which will alert the user to the vicinity of the hazard using 3D sound. This prototype has proven to be a functional proof of concept and is easily extended to a product that offers 360-degree protection. The ability to do this in real-time on a localized device that is not connected to the internet is revolutionary, and wouldn’t be possible without cutting-edge technologies like computer vision, machine learning, and smart antennas.

Design Approach

RESEARCH

To help realize the technological capabilities of the project, the team did a deep dive into researching several embedded hardware platforms that could support augmented awareness. These included the Google Coral (AI), Kendryte (AI), SiSpeed (AI), Luxonis (AI+CV), and Texas Instruments (RADAR) hardware options. The team went through several design paradigms such as trade studies, decision matrices,  and performance evaluations to determine which device’s capabilities would best fit our design requirements.

After testing all hardware options, the team settled on using the Luxonis (CV+AI) and Texas Instruments (RADAR) hardware. After the research phase was complete the team began brainstorming design solutions and creating a development plan for the rest of the project.

DEVELOPMENT PLAN

In order to successfully meet all of our clients’ requirements, it was essential to have a good plan in place that outlined development steps, project timeline, broke up the work between team members, and provided a clear and unambiguous testing plan. The team a document called the “gestr Hazard Detection Module Development Test Plan” to help support the development phase. This document includes project phases, GANTT charts, a project work breakdown structure, best design practices, a thorough testing plan, and documentation requirements.  The team created this plan and followed it in order to successfully complete the MVP by the end of the spring 2021 semester.

Design Solution

Radar

The Texas Instruments AWR1843 radar development board makes use of a smart antenna – a 2D microstrip antenna array that uses digital signal processing (DSP) to optimally steer the antenna beam and process the received data. This particular radar antenna operates in the frequency range of 76 to 81 GHz and has a detection range of about 80 meters. The HNS used the radar data in its range detection, object tracking, trajectory estimation, and speed algorithms.

Computer Vision

The computer vision capabilities are supported by the Luxonis development kit. The development kit includes two monochrome cameras and one RGB camera. These cameras allow for stereo vision and frame sampling capabilities.

The development kit allows for quick sampling of frames. With this design, the Gestr HNS can provide 30 FPS video without any delay.  With this frame data, the Gestr HNS software is able to run edge detection, depth estimation, and more to help with hazard prevention algorithms.

Depth AI

The embedded artificial intelligence processing is a key capability of our project. In order to achieve real-time detection, the code cannot run in the cloud or depend on an internet connection. Furthermore, to meet the response time requirements of the project, the AI subsystem needs to process data fast and provide neural inference in real time. The team decided to go with the Depth AI software stack to meet these requirements. 

Depth AI was created by Luxonis and is an embedded spatial AI platform built around Myriad X – a complete ecosystem of custom hardware, firmware, software, and AI training. It combines neural inference and depth vision all into a single platform. 

Currently, Depth AI is the only platform that can interface with the Myriad X. Luckily for the team, the embedded AI platform does an excellent job of this, while minimizing the data processing time. 

The team used Depth AI to train neural networks, apply object classification, process depth data, and minimize the false positives.

BLUETOOTH

The Gestr HNS incorporates a custom Bluetooth service to interact with smartphones. When a hazard is detected, a Bluetooth packet is sent from the Gestr HNS  device to a mobile device and alerts the user of the hazard. After the phone receives information about the hazard, it will play an audio message out saying “Hazard – Car” or “Hazard – Bike”. This warns the user of a potential collision before it happens and allows them to avoid the hazard. 

Software Architecture 

The software architecture is highly advanced, reliable, and minimizes response time between the first stage of the system (data sampling) to the final stage (hazard reporting). The software planning, design, development, and testing took up a majority of the team’s time over the course of the project. 

Getting less than 150 ms for an embedded system with massive amounts of incoming raw data, is a feat the team is extremely proud of. Here is a high-level overview of the software architecture that allowed us to complete this.

To see some of the software subsystems in action, check out the software demos section.

A big part of the Gestr HNS project was systems engineering and bringing all the subsystems together. The following diagram shows the connectivity/wiring diagram for the developed prototype.

The diagram shows the Luxonis raspberry pi demo board as the main hub of the system. On this board has a raspberry pi and Myriad-X integrated together. Connected to this hub is the radar data transfer line and Bluetooth dongle. The power bank distributes power to all devices that need it.

OVERCOMING CHALLENGES

1: RADAR DATA SAMPLING

Problem: The default TI software for receiving the RADAR serial data was written in MATLAB. MATLAB is not feasible for consumer products and is not compatible with Gestrs’ embedded environment.

Solution: The team had to study the MATLAB code to determine how it reads and processes data from the radar. Once this was understood, the team developed Python code to process the radar data

Iterations: The team began RADAR development through testing frameworks and code written in MATLAB. This code read serial data from the RADAR module and graphically displayed the RADAR data to the user, all within a MATLAB GUI interface.  After the team verified and evaluated the RADAR results from the MATLAB scripts, the team wrote a similar python script that would read in the serial data into the Gestr software. From there, the team added filtering capabilities into the sampling scripts to support a minimized system response time.

2: RADAR LOSING DATA CONNECTION

Problem: The TI RADAR board has had some intermittent trouble being recognized by a computer. There have been several instances where a computer cannot detect the presence of the TI board when connected via USB.

Solution: The team suspected that overheating may be a contributing factor. The radar draws a lot of current, but does not have a satisfactory heat sink. The current solution was to provide adequate ventilation to the TI board, but future iterations should incorporate a heat sink.

Iterations: The team first began realizing this issue midway through the spring semester. The first iteration involved de-powering the unit to allow it to cool down. As the team began to investigate the issue more, more ventilation was added to the casing to allow for more natural cooling. After adding this, the team no longer saw any issues with connectivity.

3: RASPBERRY PI VIDEO RECORDING

Problem: The Raspberry Pi’s processing power is limited. For testing purposes, the team needed to record massive amounts of data so that scenarios could be played back over and over again. The Pi has trouble handling the data buffering capabilities and the system would crash. 

Solution: A long-term solution will be to use a better processor with a real-time operating system. The short-term solution was to develop and test the  Python scripts on a laptop, and only run the scripts on the Raspberry Pi when the recording was not necessary.

Iterations: When the team realized there was a need to record data for playback, we first attempted to implement this on the raspberry pi. This quickly proved to be a challenge. In order to achieve this feature, the team implemented multiprocessing algorithms and developed the recording software on a laptop for testing/development.

4: INCORRECT STEREO DEPTH READINGS

Problem: Luxonis distance data was faulty. The Luxonis is advertised as being able to calculate distance to a detected object. When experimenting with this capability, the team discovered that the distance data ranged between 0.5 m and 2 m, regardless of the actual distance.

Solution: The team reached out to Luxonis for support. This was a tricky problem to troubleshoot, but fairly simple to fix. The team had to properly calibrate the stereo cameras and store the calibration files on the device’s EPROM, in order to get accurate distance data.

Iterations: The team first began to see this issue after creating the graphing/visualization capabilities. On the graph, it was obvious that the stereo depth distance was incorrect. After realizing the issue, the team began to perform isolated tests with people and cars at set distances. This gave the team a ground truth to evaluate results. The team reached out to Luxonis, who pointed the team in the right direction and gave guidance on calibration steps. After the device was calibrated, the team re-evaluated the results by comparing visualized results to ground truth measurements.

5: NEURAL NETWORK CLASSIFICATION RESOLUTION

Problem: There are few different types of neural networks that are supported by the Myriad-X chip. The initial model used by the team had terrible classification resolution for vehicles, people, and bikes. After an object was 10m away, the system could no longer identify it.

Solution: The team suspected that a neural network trained with images that had objects farther away may have a better classification range. The team found such a model that was pre-trained by Intel. This model classified objects at 20m-30m and did a much better job of detecting objects. 

Iterations: The team noticed these challenges right from the start. The initial demo model included with the Luxonis development kit was trained for 15 different objects including: Airplanes, Cats, Dogs, Goats, Bikes, Cars, etc. Furthermore, the model included was trained on zoom in or close up images of these objects. In our testing, it was clear that as objects got further away from the module, the system would begin to fail to recognize them. To fix this, the team began searching for alternative neural networks that were compatible with the hardware. The team found a model trained by Intel for vehicles, pedestrians, and cyclist that had better distance resolution and was a perfect use case for our project. The team implemented the model into our software, and tested the results. After implementing the model, the detection distance increased by a good margin.

6: SYSTEM RESPONSE TIME

Problem: There is a lot of data going through the Gestr HNS system. This requires a fair amount of compute power. When done incorrectly the system can become out of sync and very lose its real-time processing capabilities. 

Solution: The team implemented multiprocessing and parallel sampling in order to accomplish the system response requirement. 

Iterations: The system architecture went through many iterations before being able to have a less than 150 ms response time. In the first revision of the software, all data and algorithms were executed consecutively. This caused many issues and slowed down the response time by a ton. In the next phase of software architecture development, the team began to research multiprocessing and data pipelining. After redesigning the software architecture around these concepts, the team implemented multiprocessing into the Gestr HNS software stack. The results were obvious with an immediate speedup in response time. Finally, the team continued to refined various algorithms across the Gestr HNS software stack to continue minimizing response time.

7: GRAPHING AND VISUALIZATION

Problem: Although the final product will use audible alerts rather than visual alerts, a GUI showing data points from the Luxonis and TI sensors was helpful for accurate development and testing. Most graphing libraries tested slowed the system down by 300% or more.

Solution: The team developed a parallel GUI using multiprocessing using a supported python library to show a real-time polar plot of raw and processed data. Being able to visualize data in real-time helped optimize troubleshooting.

Iterations: The first attempt at including a graph to visualize data slowed the system down by a very large amount. The software was unusable. To fix this, the team began researching alternative python supported graphing methods that could handle large amounts of graphing data without any slow down to the system. The team found a solution, and implemented it into the project.

Next Steps

Several steps need to be taken to turn this into a marketable product:

• The hazard-detection algorithms will need to be made more robust to account for all possible edge cases.
• A real-time operating system (RTOS) and more powerful processor will replace the Raspberry Pi that we used, decreasing the response time and allowing for more advanced CV algorithms.
• A helmet-mounted 3D auralization system will need to be designed, to alert the user to hazards without the use of a distracting screen on a mobile device.
• The HNS and helmet system will need to be controlled by a mobile app. The development of such an app has been started, but needs further refinement. This will allow for an interface between the Gestr HNS and the user. 
• The different technologies should be integrated into a system with a single PCB, to allow for minimized module size.
• A universal housing unit should be developed that will attach to bicycles, motorcycles, and other outdoor recreational equipment. The team has started this design with an initial CAD drawing of an injection-molded housing, shown in the next section.
  

It is estimated that the Hazard Notification System will cost between $130-$150, for both the sensor system that attaches to the user’s equipment and the 3D auralization system that attaches to the user’s helmet. This price takes into consideration the estimated cost of production and the price range most stakeholders were willing to spend on such a product.

MOCKUP PRODUCT CASING DESIGN

SOFTWARE DEMOS

RADAR DEPTH

For long range distance (> 20m), Gestr HNS incorporates mmWave Radar to find and track objects behind the bike

LICENSE PLATE DETECTION

Accountability was extremely important to our stakeholders! Automated license plate detection allows for automated license plate recording when the rider is at risk. Video courtesy of Luxonis.

Meet the Team