Modern Data Stack overview

CleverMaps and Modern Data Stack

As a data engineer or data analyst, you have probably heard about Modern Data Stack (MDS). The goal of this page is to describe the role and position of the CleverMaps components in the MDS ecosystem. There is a ton of information about MDS around the web, so we will briefly describe MDS from our perspective and focus on how CleverMaps fits in. 

Here are some resources if you are not familiar with MDS:

• preset.io blog

• getdbt.com blog

• moderndatastack.xyz

CleverMaps platform is purposely designed as a group of components, and thanks to this open architecture, it is compatible with MDS. You can freely combine CleverMaps components with MDS based on your current needs. Our vision is to blend Location Intelligence components into the MDS as seamlessly as possible.

We believe that the CleverMaps platform follows the principles of MDS mainly because:

• CleverMaps is a cloud-native, multitenant, RESTful platform

• CleverMaps components can be centred around the Data Warehouse

• CleverMaps analytic components are based on modern BI trends (semantic layer, logical data model, analytics as a code, self-service analytics, embeddable analytics, …)

• CleverMaps provides advanced options for automated analytics (REST API, SDKs, git workflows, ..)

From the broader perspective, we can divide CleverMaps components into two categories:

• Business Intelligence components

  • Semantic Layer

  • Studio

  • Stories Builder

• Data components

  • Data Marketplace

  • Data Toolkit

From the MDS architecture perspective, CleverMaps components can be placed as follows:

Here is a brief overview of the CleverMaps components from the MDS perspective:

• Data Marketplace

  • Source of the geospatial data for your analytics

  • Usually, it is combined with internal business data

  • More information about Data Marketplace can be found here Data Marketplace

• Ingestion from Data Marketplace

  • Datasets from the Marketplace are available over AWS S3 bucket in CSV format, and you can use any standard S3 connector to load them into your Data Warehouse (e.g., AirByte, Fivetran, Stitch).

• Transformation

  • You can use any of your favourite tools to transform your data (e.g. dbt), but we provide a few data components to make certain geospatial operations easier.

  • We also provide some components for the CleverMaps-specific jobs. More information can be found here Data Toolkit.

  • We are also planning to provide some dbt_packages with geospatial transformations in the near future.

• Data load

  • For loading data from the Data Warehouse into the CleverMaps Semantic Layer, see the next chapter.

• Business Intelligence Semantic and Visualization layer

  • This is the CleverMaps main layer, and it consists of the following components:

    • Semantic Layer

    • Studio

    • Stories Builder

  • More information about the Semantic layer can be found here Semantic layer

• Reverse ETL

  • We also provide a data component for extracting data and metrics from the CleverMaps Semantic Layer. More information can be found here Data Toolkit.

  • These tools can help you deploy a process which will send data and metrics from CleverMaps further to other organizational systems.

Data loading into the CleverMaps Semantic Layer

CleverMaps Semantic Layer is working on top of the Postgres database with the PostGIS extension. It can connect to externally hosted Postgres DWH directly. If you prefer Snowflake, BigQuery or Redshift, you will need to copy data from DWH to the CleverMaps Semantic layer. It means that you need to establish a loader from your Data Warehouse into the CleverMaps Semantic Layer. The loader can be implemented in a technology of your choice. The only requirement is that the data needs to be loaded in the CSV format. We provide a few ways to load the data:

• Using CleverMaps Writer (Docker)

• Using CleverMaps Shell (Docker or jar)

• Using pure CleverMaps Semantic Layer REST API

• Using CleverMaps Python SDK (beta)

• Using CleverMaps Java SDK (beta)

We also provide ready to use CleverMaps Connector for the Keboola Connection ETL, and we are also planning to implement a connector for other ETL/ELT technologies (e.g. AirByte, Fivetran, Stitch).

More information about data loading can be found here Data Toolkit

Common deployments

Let’s dive into some common scenarios which can be deployed in the MDS ecosystem.

Minimal headless deployment

• This scenario is helpful for use cases when you need a cloud analytical backend where you can access your metrics and data over the REST API from your own data visualization layer (custom data app, notebook, script, dashboard, …).

• This scenario will be deployed with CleverMaps Semantic Layer and optionally with some data from CleverMaps Data Marketplace.

• Data Ingestion, Data Warehouse and Data Visualization layer will be deployed with technologies of your choice.

Standard analytical deployment

• This scenario is the most common because it also delivers the Business Intelligence layer - CleverMaps Studio.

• This scenario will be deployed with CleverMaps Semantic Layer, CleverMaps Studio and optionally with some data from CleverMaps Data Marketplace.

• Data Ingestion and Data Warehouse will be deployed with technologies of your choice.

Full analytical storytelling deployment

• This scenario is common if you need to deliver data analytics for less technical users, typically for management or sales. Data analytics is delivered with a lightweight data story layer - CleverMaps Stories.

• This scenario will be deployed with CleverMaps Semantic Layer, CleverMaps Studio, CleverMaps Stories and optionally with some data from CleverMaps Data Marketplace.

• Data Ingestion and Data Warehouse will be deployed with technologies of your choice.