MF4 Decoders - DBC Decode CAN Data to CSV/Parquet [+ Data Lakes] (2024)

The ASAM MDF (Measurement Data Format) is a popular, open and standardized format for storing bus data e.g.from CAN bus (incl. J1939, OBD2, CAN FD etc) andLIN bus.

The CANedge records raw CAN/LIN data in the latest standardized version, MDF4 (*.MF4). The log fileformatis ideally suited for pro specs CAN logging at high bus loads and enables both lossless recording and 100%powersafety. Further, the CANedge supports embedded encryption and compression of the log file data (both casesnatively supported by the MF4 decoders).

The raw MF4 data from the CANedge can be loaded natively in various software/API tools, including the asammdf GUI/API, our Python API, the MF4 converters - and the MF4decoders (detailed in this article).

To learn more about the MDF4 file format, see our MF4 intro.

The MF4 decoders can be deployed in numerous ways - below are two examples:

The decoders support DBC files for practically any CAN bus, CAN FD or LIN bus protocol. This includes e.g. OBD2,J1939, NMEA 2000, ISOBUS, CANopen and proprietary OEM-specific DBC files. For details, see the MF4 decoder docs. For eachDBCfile, you specify which CAN/LIN channel to apply it to (e.g. can1, can2, lin1 etc) and you canprovide multiple DBC files per channel.

The MF4 decoders enable you to output DBC decoded CAN/LIN data as either CSV files or Parquet files. Below webriefly outline the key differences between the formats:

• CSV files are simple, text-based, and universally compatible, ideal for small to medium-sized datasets andad hoc analyses. They are easy to use, but less efficient for large data
• Parquet files are binary and offer vastly faster performance and storage efficiency vs CSV - but requiremore specialized tools for analysis
• While CSVs are straightforward for basic tasks, Parquet is generally preferable for performance-intensiveanalysis and handling large-scale time series data

For most use cases, we recommend to use the Parquet file format if both options exist - and many of our plug &play integrations take outset in Parquet data lakes for the above reasons.

DownloaddecodersDownload MDF4data

Parquet data lakes are comprised of files, meaning they can be stored in file storage solutions like e.g. AWS S3.Storing data on S3 is incredibly low cost (0.023$/GB/month) compared to most databases (typically~1.5$/GB/month), which is relevant as many CAN/LIN data logging use cases can require terabytes of storage overtime.

The 'downside' to storing files on S3 vs. in a database is generally the fact that it is much slower to query thedata, e.g. for analytics or visualization. However, this is where the interface tools like Amazon Athena comeinto play as outlined below.

We refer to tools like Amazon Athena, Google BigQuery and Azure Synapse Analytics as 'interfaces' for simplicity.They can also be referred to as cloud based serverless data warehouse services. The serverless part isimportant: It means that it's simple to set up - and you pay only when you query data.

Many automotive OEM engineers often need to store terabytes of data for analysis - yet, the engineers may onlyneed to access small subsets of the data on an infrequent basis. Yet, when they access the data, the query speedhas to be fast - even if they query gigabytes of data. Tools like Amazon Athena are ideally suited for this.When you query the data, Athena spins up the necessarycompute and parallelization in real-time - meaning you can extract insights across gigabytes of your S3 datalake in seconds using standard SQL queries. At the same time, all the complexity is abstracted away - and thesolutions can be automatically deployed as per our step-by-step guides.

There are too many software/API integration examples for us to list - below is a more extensive recap oftools:

Direct integration examples

Below are examples of tools that can directly work with Parquet files:

• MATLAB:Natively supports local or S3 based Parquet data lakes, with powerful support for out-of-memory tall arrays
• Python:Natively supports local or S3 based Parquet data lakes and offers libraries for key interfaces(Athena, ClickHouse etc.)
• Power BI:Supports reading Parquet files from the local filesystem, Azure Blob Storage, and Azure Data Lake StorageGen2
• Tad: Free desktop tool forviewing and analyzing tabular data incl. Parquet files. Useful for ad hoc review of your data
• Apache Spark: A unified analytics engine forlarge-scale data processing that supports Parquet files
• Databricks: A platform for massive scale dataengineering and collaborative data science, supporting Parquet files
• Tableau: A data visualization tool that can connectto Parquet files through Spark SQL or other connectors
• Apache Hadoop: Supports Parquet file format forHDFS and other storage systems
• PostgreSQL: With the appropriate extensions, itcan query Parquet files
• Cloudera: Offers a platform that includes Parquetfile support
• Snowflake: A cloud data platform that can load andquery Parquet files
• Microsoft SQL Server: Can accessParquet files via PolyBase
• MongoDB: Can import data from Parquet files usingspecific tools and connectors
• Teradata: Supports querying Parquet files usingQueryGrid or other connectors
• Apache Drill: A schema-free SQLQuery Engine for Hadoop, NoSQL, and Cloud Storage, which supports Parquet files
• Vertica: An analytics database that can handleParquet file format
• IBM Db2: Can integrate with tools to loadand query Parquet files

Interface based integrations

Below are examples of tools that can integrate via interfaces like Athena, BigQuery, Synapse, ClickHouse etc.:

• Power BI(driver):By installing a JDBC/ODBC driver (for e.g. Athena), you can use SQL to query your data lake
• Excel (driver):By installing a JDBC/ODBC driver (for e.g. Athena), you can use SQL to query your data lake
• Grafana: Offers powerful and elegant dashboards for datavisualization, ideal for visualizing decoded CAN/LIN data
• Tableau: Known for its interactive datavisualization capabilities, especially popular for business intelligence applications
• Looker: Employs an analytics-oriented applicationframework, including business intelligence and data exploration features
• Google Data Studio: Customizablereports/dashboards, known for user-friendly design and integration with Google services
• AWS QuickSight: A fast, cloud-poweredbusiness intelligence service that integrates easily with e.g. AmazonAthena
• Apache Superset Apache Superset is anopen-source dataexploration and visualization platform written in Python
• Deepnote Deepnote is a collaborative data notebookbuiltfor teams to discover and share insights
• Zing Data Zing Data is a data exploration andvisualization platform supporting e.g. ClickHouse
• Explo Customer-facing analytics for any platform.Designedfor beautiful visualization. Engineered for simplicity
• Metabase Metabase is an easy-to-use, open source UItoolfor asking questions about your data
• Qlik: Offers end-to-end, real-time data integration andanalytics solutions, known for the associative exploration user interface
• Domo: Combines a powerful back-end with a user-friendlyfront-end, ideal for consolidating data systems into one platform
• Sisense: Known for its drag-and-drop user interface,enabling easy creation of complex data models and visualizations
• MicroStrategy: Offers a comprehensive suite ofBI tools, emphasizing mobile analytics and hyper-intelligence features
• Splunk: Specializes in processing and analyzingmachine-generated big data via a web-style interface
• Exasol: Offers a high-performance, in-memory, MPPdatabase designed for analytics and fast data processing
• Alteryx: Provides an end-to-end platform for datascience and analytics, facilitating easy data blending and advanced analytics
• SAP Analytics Cloud: Offersbusiness intelligence, augmented analytics, predictive analytics, andenterprise planning
• IBM Cognos Analytics:Integrates AI to help users visualize, analyze, and share actionable business insights
• GoodData: Provides cloud-based tools for big dataand analytics, with a focus on enterprise-level data management and analysis
• Dundas BI: Offers flexible dashboards, reporting, andanalytics features, allowing for tailored BI experiences
• Yellowfin BI: Delivers business intelligencetools and a suite of analytics products with collaborative features for sharing insights
• Reveal: Provides embeddedanalytics and a user-centric design, making data more accessible for decision makers and teams
• Chartio: A cloud-based data exploration tool, knownfor its ease of use and ability to blend data from multiple sources

Yes, you control 100% how you create and store your CSV/Parquet data lake.

In our examples, we frequently take outset in a setup where your CANedge2/CANedge3 uploads data to an AWS S3bucket - with uploaded data automatically processed via AWS Lambda functions. This is a common setup that weprovide plug & play solutions for - hence it will often be the simplest way to deploy your data processing anddata lake.

However, you can set this up in any way you want. For example, you might store your uploaded log files on aself-hosted MinIO S3 bucket instead. In such a scenario, you can periodically process new MF4 log files manually(e.g. via drag & drop or the CLI) to update your data lake - or you can set up e.g. a cron job or similarservice to handle this. The data lake can be stored in another MinIO S3 bucket - and you can then directly workwith the data lake from here (e.g. in MATLAB/Python) or integrate thedata using an open source system like ClickHouse or DuckDB.

The same principle applies if you upload data to Google Cloud S3 or Azure blob storage (via an S3 gateway) - hereyou can use their native data processing services to deploy the MF4 decoders if you wish to fully automate theDBC decoding of incoming data. We do not provide plug & play solutions for deploying this, however.

Of course, you can also simply use the MF4 decoders locally to create a locally stored Parquet data lake. Thiswill often suffice if you're e.g. using a CANedge1 to record your CAN/LIN data - and you simply wish to processthis data on your own PC. In such use cases, the Parquet data lake can still be a powerful tool, since it makesit much easier to perform large-scale data processing compared to tools like the asammdf GUI.

We provide two types of MF4 executables for use with the CANedge: The MF4 converters and MF4 decoders.

The MF4 converters let you convert the MDF log files to other formats like Vector ASC, PEAKTRCand CSV. Theseconverters do not perform any form of DBC decoding of the raw CAN/LIN data - they only change the fileformat.

The MF4 decoders are very similar in functionality as the MF4 converters. However, theseexcecutables DBC decodethe log files to physical values, outputting them as either CSV or Parquet files. When using the MF4decoders,you provide your DBC file(s) to enable the decoding. These executables are ideal if your goal is to analysethehuman-readable form of the data and/or e.g. create 'data lakes' for analysing/visualizing the data at scale.

No, you simply downloadtheconverter executables - no installation required.

As evident, there are almost limitless options for how you can deploy your MF4 decoders, how you can store theresulting data lake, how you provide interfaces for it - and what software/API tools you integrate it with.

You can use the CANedge and our MF4 decoders to facilitate any of these deployment setups. However, our teamoffers step-by-step guides and technical support only on limited sub sets of the deployments, such as ourGrafana-Athena integration.

Buy nowContact us