Essays, Posts & Presentations

Data Science at Home: Leverage Wolfram’s Device Integration to Analyze Your Vehicle’s Performance

Andrew Steinacher

Author

Andrew Steinacher

Title

Data Science at Home: Leverage Wolfram’s Device Integration to Analyze Your Vehicle’s Performance

Description

Connect to a vehicle’s Controller Area Network (CAN) bus to gather driving data and create an interface using Wolfram Notebooks to report on the data. Incorporate GPS data for additional analysis.

Data Science at Home: Leverage Wolfram’s Device Integration to Analyze Your Vehicle’s Performance

December 31, 2020 — Andrew Steinacher, Lead Developer, Wolfram|Alpha Scientific Content

Cars are getting smarter and more connected, yet how much have you explored the technology that helps run our vehicles? I was curious to see how I could connect to my vehicle’s communication center and what kind of interface I could create in Wolfram Notebooks to report on the data gathered.

I started my data science side project by connecting a microcontroller to my car’s Controller Area Network (CAN) bus, and I gathered a fire hose of information by simply recording messages that were sent over the CAN bus. This massive amount of data has given me a large dataset (over 25 GB) from thousands of car trips—including time series for engine temperatures, steering angles, fuel flow and more. Using the Wolfram Language, it was easy to build a user interface for interactively analyzing and processing this CAN data, as well as an

EntityStore

to more programmatically analyze the data on longer timescales with

EntityValue

syntax. These tools have allowed me to analyze engine warmup times and my fuel usage trends, and after some alignment to GPS location data, revealed at which intersections I wait the longest.

Let’s explore how any of you could do this yourselves, along with the directions I’ve considered to take this even further.

Graduating from Web Forms I wrote a post in 2017 on how to create an app to track and analyze gas mileage in the Wolfram Cloud. This was an immediate option for gathering some basic data by punching numbers (e.g. fuel volume, fuel costs, mileage) into a

FormFunction

, storing it in a

Databin

and setting up an automated report (through a template notebook)—complete with 2D and 3D graphs plus

GeoHistogram

visualizations.

This method gave some interesting data correlation, linking things like worsening gas mileage to colder weather and uncovering fuel price drops, such as the one I observed at the beginning of 2019:

In[]:=

RowDateListPlot

Computed MPG

Actual MPG

,

Options

,Spacer[10],DateListPlotTimeSeries

Time: 20 Mar 2016 to 29 Sep 2019
Data points: 149

,

Options



Out[]=

Like all manual data collection, this process was tedious, and I really wanted to take it further. From watching the display in my car that shows instantaneous gas mileage and other averages, I knew that more data than I could collect by hand was being generated in order to populate the display. It was then that I began to wonder how I could gain access to this data, to the point I was inspired to research the on-board diagnostics standard (OBD-II) and CAN bus sniffing.

CAN Bus Sniffing Most modern cars have a small port under the dashboard that can be used to communicate with all of the car’s electronic systems, such as the engine, the body, the radio, etc. This port, defined by the OBD-II standard, is typically used for gathering emissions history and troubleshooting engine problems. But this is where all the car’s systems communicate with each other in an organized cacophony—on a set of two wires connecting lots of devices in the car, called the CAN bus. A CAN bus is a standard bus system frequently used in automotive vehicles, about as established as Mathematica itself. I had to learn quite a bit to not only connect to this communication center, but also to boil down the data that comes through it. All messages sent over the bus, such as engine temperatures, steering wheel angles and individual wheel speeds, have IDs that represent what they’re for.

Message Structure

One of the first things to understand is the structure of a single CAN message. There are roughly 12 bytes of really important information: about four bytes for the message’s ID, plus up to eight bytes of message data. Lower IDs are prioritized over higher IDs, and devices on the network acknowledge (ACK) when a message has been correctly received. Here’s a visualization of the parts of a CAN message (from CSS Electronics):

Physical Assembly

If you want to follow along with this too, you’ll need some hardware: a microcontroller of some kind, a way to interface with the CAN bus itself and a way to store collected data. Here’s what I used:

◼

Microcontroller: Arduino UNO Rev3

◼

CAN interface: Seeed CAN-BUS Shield V1.2

◼

RTC and storage: Adafruit data logger shield

Of course, you’ll also need a cable to connect the OBD-II port in your vehicle to your microcontroller, and another to provide power.

Before moving on to the software, there are a couple of other hardware considerations to keep in mind. For speed reasons, I connected a jumper wire from the square wave output of my RTC to an interrupt pin so I could keep track of time (more on that later!), and I’ve also removed some optional termination resistors. CAN typically has termination resistors at both ends of the two-wire bus (to avoid standing waves). The car’s CAN bus already has termination resistors, so if I connect my Arduino with another termination resistor, the overall resistance could change—and I don’t want to risk side effects from that.

Data Format

At first, I fought with the format of data and how to store it. My original, naive idea was to store the present hour, minute and second along with the bus information (CAN message ID plus the message’s data) in a plain text string with commas as delimiters. But this turned out to be really slow and large—generating around 40 bytes per CAN message (which you may recall has only about 12 “interesting” bytes):

In[]:=

StringLength["12:31:25,00D0,FF,FF,FF,FF,FF,FF,FF,FF\n"]

Out[]=

With a little work, I was able to optimize my code to store only the bare minimum to keep up with the large amount of traffic being sent over the bus. Using an interrupt connected to the RTC, I could quickly and efficiently count the number of seconds since the car was started, store the CAN ID and message content in a data structure, and write this data in binary. This conserves space and is overall more efficient, generating only 14 bytes per message:

In[]:=

//StoreCANdatainastructforfastbinarywritingvolatilestructCANMessage{unsignedintt;unsignedlongintid;unsignedchardata[8];}m;

Data Samples and Visualization

Once the raw data is stored in files, I can read it into the Wolfram Language for analysis. You can try this yourself—just download and extract the code and a portion of the data from the Wolfram Cloud:

In[]:=

zip=URLDownload["https://wolfr.am/CANGUI-zip"];ExtractArchive[zip];

Then browse to the extracted directory:

In[]:=

SetDirectory["CANGUI_STANDALONE"];

To read the files, I need to tell the Wolfram Language what kind of format to expect (based on the previously shown structure):

In[]:=

binaryFileFormat=Flatten[{"UnsignedInteger16",(*Secondssincestart*)"UnsignedInteger32",(*CANID(decimal)*)ConstantArray["UnsignedInteger8",{8}](*Data*)}];

Then I can use

BinaryReadList

with that spec to import data from individual files. Each file name includes a standardized start date and time. For instance, this imports the data for a specific car trip dated March 18, 2019:

In[]:=

data=BinaryReadList["DataFiles/19031818.58B",binaryFileFormat];data//Length

Out[]=

650376

As you can see, it consists of more than half a million points—pretty intense for a 10-minute car trip. Here’s a sample of what is stored, where the first column is the number of seconds that have elapsed, the second column is the CAN message ID in decimal and the remaining columns give the actual message data:

In[]:=

Griddata[[;;2000;;150]],



Out[]=

1	2	255	111	112	15	237	0	0	0
1	328	14	30	236	134	65	0	255	5
2	642	186	16	0	95	95	3	34	0
2	324	192	0	85	130	57	164	40	0
2	338	225	204	0	0	0	0	24	140
2	886	125	1	63	177	171	72	0	0
2	881	1	0	0	0	254	24	2	0
2	210	0	0	255	255	0	0	0	0
3	342	1	0	0	0	0	0	0	30
3	321	91	38	156	42	6	199	32	0
3	338	233	236	0	0	0	0	24	140
3	329	126	1	63	180	171	75	0	0
3	208	147	0	0	0	0	0	0	249
3	328	14	28	212	134	65	0	255	5

This is basically all I had when I got started, with no manual for reverse-engineering the data. After a lot of experimentation I started to piece together which IDs belonged to which system, and I began to extract useful information from this seemingly cryptic data. First, let’s make a function to translate some of this into a

TimeSeries

In[]:=

toMessageTimeSeries[data:{__List}]:=With[{times=data[[All,1]]},TimeSeries[#,{times}]&/@Transpose[data[[All,2;;]]]];

Now we will look at pieces of data from different IDs:

In[]:=

idToByteTimeSeries=KeySort@GroupBy[data,#[[2]]&,toMessageTimeSeries];

In[]:=

idToByteTimeSeries[[;;2,;;2]]

Out[]=

2TimeSeries

Time: 1 to 773
Data points: 773

,TimeSeries

Time: 1 to 773
Data points: 773

,208TimeSeries

Time: 1 to 773
Data points: 773

,TimeSeries

Time: 1 to 773
Data points: 773



And here’s a visualization of the engine coolant temperature data (ID: 0x360, Byte #4), using

DateListPlot

In[]:=

DateListPlotidToByteTimeSeries[864][[4]],

Options



Out[]=

CANGUI Manually working with files like this can be tedious, especially when exploring many trips. Creating a GUI will make it faster and easier to explore and plot data. I’ve already set up an interactive Wolfram Language GUI, called CANGUI, for processing and visualizing CAN bus messages recorded in my data format. This point-and-click interface and message space view was a quick and dirty attempt at using

DynamicModule

to browse data ranges, explore IDs of data within those ranges and plot those TimeSeries. It has worked pretty well for me so far!

You can evaluate the following function to bring up the full interface:

In[]:=

Get["CANGUI/CANGUI.wl"];CANGUI[]

On the left you can select a day, and then available trips based on their start and end times. In the middle gray-background area, you will see a number of message parts either fully identified or only suspected. Unfortunately, there is not freely available, complete documentation for which messages contain what data and in what units, so some are a guess. Since this is hard to keep track of, I’ve put the known IDs into a message space grid, which you can look at here:

Back to the GUI: on the right side, you will see a table of all IDs captured within the selected trips, along with a plot of the single ID selected from the gray-background area. From here you can use the Copy button to put that TimeSeries on your clipboard for further exploration:

In[]:=

DateListPlotTimeSeries

Time: 3.79× 9 10 to 3.79× 9 10
Data points: 8888



Out[]=

You can also use the Details drop-down menu at the bottom of the middle column to enhance the visualization in whatever way you’d like, or even add new entries:

Additionally, by selecting multiple trips on the far left, you can compare the time series data for all of the trips within a single plot, like this one for the odometer:

With this interface you can punch in IDs, give names and change the Wolfram Language code used to generate the TimeSeries, as well as add other options to aid in understanding and extracting useful data.

Individual Trip Analysis Although the GUI was helpful in initially exploring the data and understanding the message content, it quickly became too cumbersome for extracting information on timescales longer than a few trips. Additionally, I was interested in tying in other data, such as my existing form-based gas mileage information or even GPS data from my smartphone. After some thought on a good way to approach these issues, I set out to build an EntityStore.

Using this code, I can create and register the EntityStore and align it to the GPS data in my databin—again, only a small part of the total data. First, I import the entity store code and create an EntityStore:

In[]:=

Get["CANGUI/TripEntityStore.wl"];store=CreateTripEntityStore["DataFiles","PlotChoices",Databin["Fbuwpsdi"]]

Out[]=

EntityStore



Type: Trip

Entities: {2019-06-20:1,2019-06-20:2,2019-06-20:3,…,2019-03-13:4,2019-03-13:5,2019-03-13:6} (112)

Properties: {Label,File,Date,…,{WheelSpeedRLTimeSeries,{AggregationFunction_}},WheelSpeedRRTimeSeries,{WheelSpeedRRTimeSeries,{AggregationFunction_}}} (217)

Data not in notebook. Store now



Then I register the EntityStore:

In[]:=

EntityRegister[store]

Out[]=

{Trip}

This system is really nice for a few reasons. I can search and filter the trips with implicit

EntityClass

syntax. Plus, each trip entity only loads the core metadata based on file names and then keeps a reference to the file where the rest of the information can be extracted as needed (it’s really inefficient to load all the data at once). Here is a quick look at the kind of simple metadata that is gathered for each trip entity:

In[]:=

Trip #2 on 2019-03-22

TRIP



date

duration

start position

end position

start time

end time

file

label

trip number



Out[]=



Fri 22 Mar 2019

15.1333

min

,GeoPosition[{40.0993,-88.2493}],GeoPosition[{40.146,-88.2575}],

Fri 22 Mar 2019 18:11:00CDT

Fri 22 Mar 2019 18:26:00CDT

,File

/Data/CANGUI_STANDALONE/DataFiles/19032218.11B

,Trip #2 on 2019-03-22,2

Of course, extracting the information that I gathered from the GUI can be done by creating properties to programmatically produce a TimeSeries in much the same way. For example, now I can easily extract time series for the brake and accelerator pedals for a given trip:

In[]:=

accAndBrake=

Trip #1 on 2019-03-21

TRIP



,"PropertyAssociation"

Out[]=



Brake Pedal time series

TimeSeries

Time: 3.76× 9 10 to 3.76× 9 10
Data points: 2808

,

Accelerator Pedal time series

TimeSeries

Time: 3.76× 9 10 to 3.76× 9 10
Data points: 2808



Here is a plot of the time series for both pedals during the trip:

In[]:=

DateListPlotaccAndBrake,



Out[]=

Brake Pedal time series

Accelerator Pedal time series

Deep Diving into Fuel Economy

The ease of use with this system allows me to compare data in extremely useful ways when it comes to interpreting my car’s fuel economy. For example, I found the fuel flow rate with the help of the GUI:

In[]:=

fuelFlow=

Trip #1 on 2019-03-21

TRIP



Fuel Flow time series



Out[]=

TimeSeries

Time: 3.76× 9 10 to 3.76× 9 10
Data points: 2807



With a little work, one can integrate this TimeSeries to compute an estimate for the total fuel used for the entire trip:

In[]:=

timeSeriesTotalIntegral[td_TemporalData]:=Total[td]Quantity[MinimumTimeIncrement[td],"Seconds"]

In[]:=

timeSeriesTotalIntegral[fuelFlow]

Out[]=

0.180023

gal

To check the accuracy of this, I tried comparing the total for a week’s worth of trips from my full dataset to the numbers I had entered separately with my web form. The computed fuel usage from my CAN data is about 12 gallons:

In[]:=

EntityValueEntityClass"Trip","Date"Between

Day: Fri 14 Jun 2019

Day: Thu 20 Jun 2019

,

Fuel Flow time series

,Map[timeSeriesTotalIntegral]/*Total

Out[]=

12.0009

gal

This looks decent at a glance, since my car’s fuel tank holds about 14 gallons of fuel. And if we compare to the "FuelUsed" data that I recorded manually, the result is pretty close:

In[]:=

KeyTakeSelect

MPG Tracker Data

,DayRound[#Timestamp]==

Day: Thu 20 Jun 2019

&,{"FuelUsed","PreviousTime"}

Out[]=

FuelUsed	PreviousTime
12.311gal	Fri 14 Jun 2019 17:52:00

And that’s just one data point. If I take all of the data I’ve collected with the web form and compare it to what I’ve measured with my new CAN sniffing system, I can see how well the results match with a scatter plot:

In[]:=

fuelUsedData=Cases

MPG Tracker Data

[All,{#FuelUsed,#MeasuredFuelUsed}&],{_Quantity,_Quantity};

In[]:=

comparisonPlot=ListPlotfuelUsedData,

Options



Out[]=

Notice that most points lie near the dashed

y=x

line. To verify this result, I can compute a fit for the data:

In[]:=

fit=Fit[QuantityMagnitude[Normal@fuelUsedData,"Gallons"],x,x]

Out[]=

0.972275x

Here is the same plot shown previously, but with that (solid blue) trend line included:

In[]:=

Show[comparisonPlot,Plot[fit,{x,4,14}]]

Out[]=

Notice that the results are pretty well correlated, so the methods agree for the most part, though there are some outliers (e.g. I don’t use the CAN sniffing system for some trips).

Of course, this means that I can compute my fuel use on a per-trip basis accurately! Then I can do more interesting things, such as compare my fuel economy distribution by month. To do that, I first aggregate data from all trips (which is too large to share) and compute the median values:

In[]:=

fuelEconomyMedians=EntityValue["Trip",EntityProperty["Trip","FuelEconomyTimeSeries",{"AggregationFunction"Median}],"EntityAssociation"]//Normal;

Then I group the data by month and sort it appropriately:

In[]:=

monthlyMPGDistribution=KeySortBy[GroupBy[fuelEconomyMedians,First[#]["StartTime"]["MonthName"]&Last],Interpreter["Date"]];

From here I can plot my monthly fuel economy distribution with

BoxWhiskerChart

(note that your results using the reduced dataset will differ slightly from what’s shown here):

In[]:=

BoxWhiskerChartmonthlyMPGDistribution,

Options



Out[]=

Incorporating GPS Data After incorporating my manually tracked MPG data, I wanted to incorporate GPS information as well. So I began looking into using my smartphone to collect this data automatically.

Rather than build an Android app from scratch to do this myself, I found it much easier to use an automation app. I used MacroDroid, but just about any automation app should be up to the task. Here’s a quick screenshot of the configuration I used if you want to follow along:

With this in place, my smartphone’s GPS data is retrieved and uploaded to a Databin every 10 seconds—but only as long as I’m connected to my car’s Bluetooth (to conserve phone battery and data). The data available includes GPS position, speed, accuracy, time and whether I’m starting, ending or in transit.

As an example, consider this map of a short trip I made from Wolfram HQ up to a busier area of Champaign, Illinois (blue circles are simply for location positions, and red dots indicate accuracy):

In[]:=

GPSPlot

Trip #2 on 2019-03-22

TRIP

["GPSTimeSeries"],

Options



Out[]=

As you can see in this plot, the red accuracy indicators are much wider than the lanes of the roads, which means this data is probably not good enough for lane-level accuracy (e.g. detecting the right vs. the left lane). And to know which direction I was facing, I have to use context compared to the previously recorded point.

The plot includes tooltips that give all the details on the position, speed, time and accuracy:

Trip Date/Time Distributions

As of March 2020, I’d been collecting GPS data for about a year’s worth of trips, and I’ve accumulated over 70K points in my Databin. Here’s a quick

Manipulate

to see a

DateHistogram

of these trips at various timescales (ignoring several gaps in data recording due to technical difficulties):

In[]:=

ManipulateDateHistogram

Dates

Options

,{reduction,{None,"Year","Week","Day"}},

Options



Out[]=

reduction

None

Year

Week

Day

One noticeable trend is my tendency to drive more on weekends.

Single-City Speed Analysis

Let’s look at a GeoHistogram of all of the points recorded in Champaign (somewhat anonymized, of course!). The colors correlate to how many points there are in each bin:

In[]:=

champaignGeoPositionsTravel=Import["https://wolfr.am/R9FmQ8N1"]

Out[]=

GeoPosition

Number of points: 10698

Lat bounds: {40.1,40.2}

Lon bounds: {-88.3,-88.2}



In[]:=

GeoHistogramchampaignGeoPositionsTravel,{"Rectangle",Quantity[0.125,"Miles"]},

Options



Out[]=

As you can see, I spend a lot of time at specific places. We can look at only speeds under 15 miles per hour to see how much time I spend sitting still on these trips (intersections, construction, etc.). Notice the high concentrations in certain areas that I commonly visit, like my commute to work:

In[]:=

champaignGeoPositionsUnder15=Import["https://wolfr.am/R9GCouE7"]

Out[]=

GeoPosition

Number of points: 1163

Lat bounds: {40.1,40.2}

Lon bounds: {-88.3,-88.2}



In[]:=

GeoHistogramchampaignGeoPositionsUnder15,{"Rectangle",Quantity[0.125,"Miles"]},

Options



Out[]=

Parking Location Analysis

Zooming in a bit more, I thought it would be fun to find where I’ve parked in the Wolfram Research parking lot here in Champaign. Each dot represents a GPS reading from my parked car:

In[]:=

parkingSpotsWRI=GeoPosition

Number of points: 118

Lat bounds: {40.1,40.1}

Lon bounds: {-88.2,-88.2}

;

In[]:=

GeoGraphics{Green,PointSize[Medium],Point[GeoPosition[#]]&/@parkingSpotsWRI["Data"]},

Options



Out[]=

Now I’ve changed the grid size so we can see where I’ve parked and how often I choose to park there. Note that the geolocation data is probably skewed a bit because I’m fairly certain I’ve not parked in other lots, in the grass or across multiple spaces. That said, it’s clear that I’m pretty much a creature of habit when it comes to parking—there is a fairly narrow area in which I park:

In[]:=

GeoHistogramparkingSpotsWRI["Data"],{"Rectangle",{Quantity[10,"Feet"],Quantity[25,"Feet"]}},

Options



Out[]=

Future Directions Everything in this post was originally done before the pandemic; unsurprisingly, I’ve spent a lot less time in the car this year. So while I haven’t been collecting much driving data, I’ve thought a lot about how to take this further.

For instance, I’d like to be able to detect who is driving (having another driver download the automation app and adding a new Databin field for user identification). With a little more hardware, I could hook up a Wi-Fi chip for immediate data upload to the Wolfram Cloud, albeit with a smaller set of essential data. Another idea would be to livestream CAN data onto a screen to review and analyze in real time. Lastly, I think it would be interesting to investigate the different routes I take to get to and from places, possibly even computing the paths between GPS positions with odometry or dead reckoning. If you’ve explored anything like this or have more ideas on how to expand on the concept, let me know in the comments!

Visit Wolfram Community to embark on your own computational adventures!

Cite this as: Andrew Steinacher, "Data Science at Home: Leverage Wolfram’s Device Integration to Analyze Your Vehicle’s Performance" from the Notebook Archive (2020), https://notebookarchive.org/2020-12-eapfauw

1	2	255	111	112	15	237	0	0	0
1	328	14	30	236	134	65	0	255	5
2	642	186	16	0	95	95	3	34	0
2	324	192	0	85	130	57	164	40	0
2	338	225	204	0	0	0	0	24	140
2	886	125	1	63	177	171	72	0	0
2	881	1	0	0	0	254	24	2	0
2	210	0	0	255	255	0	0	0	0
3	342	1	0	0	0	0	0	0	30
3	321	91	38	156	42	6	199	32	0
3	338	233	236	0	0	0	0	24	140
3	329	126	1	63	180	171	75	0	0
3	208	147	0	0	0	0	0	0	249
3	328	14	28	212	134	65	0	255	5

1	2	255	111	112	15	237	0	0	0
1	328	14	30	236	134	65	0	255	5
2	642	186	16	0	95	95	3	34	0
2	324	192	0	85	130	57	164	40	0
2	338	225	204	0	0	0	0	24	140
2	886	125	1	63	177	171	72	0	0
2	881	1	0	0	0	254	24	2	0
2	210	0	0	255	255	0	0	0	0
3	342	1	0	0	0	0	0	0	30
3	321	91	38	156	42	6	199	32	0
3	338	233	236	0	0	0	0	24	140
3	329	126	1	63	180	171	75	0	0
3	208	147	0	0	0	0	0	0	249
3	328	14	28	212	134	65	0	255	5

1	2	255	111	112	15	237	0	0	0
1	328	14	30	236	134	65	0	255	5
2	642	186	16	0	95	95	3	34	0
2	324	192	0	85	130	57	164	40	0
2	338	225	204	0	0	0	0	24	140
2	886	125	1	63	177	171	72	0	0
2	881	1	0	0	0	254	24	2	0
2	210	0	0	255	255	0	0	0	0
3	342	1	0	0	0	0	0	0	30
3	321	91	38	156	42	6	199	32	0
3	338	233	236	0	0	0	0	24	140
3	329	126	1	63	180	171	75	0	0
3	208	147	0	0	0	0	0	0	249
3	328	14	28	212	134	65	0	255	5