Writer (Docker)

Getting started

This component loads data into the CleverMaps platform from the CSV files. You need to have an existing CleverMaps project with defined data model.

The format of the input CSV file must match the target CleverMaps dataset. This includes the number of columns, their order and data types. It is possible to load one or more files in one configuration.

Authentication

To authenticate the Writer, you will need to Generate an AccessToken. This accessToken can be generated by CleverMaps UI:

Installation

To use Writer locally on your computer you just need to dowload the image from DockerHub with:

docker pull clevermaps/writer:latest

Running

There are few steps you need to do before running Writer:

Create an empty directory on your local drive (e.g. /home/user/clevermaps/writer).
Copy your CSV files into that folder.
Create a configuration file for Writer and save it as a JSON file into the folder. Please check the examples below in Configuration to get the idea how the configuration file should look.

Use following command to run Writer:

docker run -t -v /home/user/clevermaps/writer/:/tmp clevermaps/writer:latest --config /tmp/config.json

Remember that you need to change /home/user/clevermaps/writer/ to your working directory path.

Configuration

Writer needs a configuration to run, you need to pass a path of a JSON file into the --config parameter. Example config:

CODE

{
    "#access_token": "",
    "project_id": "",
    "server": "https://secure.clevermaps.io",
    "validation": false,
    "datasets": [
        {
            "dataset_enabled": true,
            "input_csv": "/tmp/branches.csv",
            "dataset": "branches_dwh",
            "mode": "full",
            "csv_null": "",
            "csv_force_null": ""
        },
        {
            "dataset_enabled": false,
            "input_csv": "/tmp/customers.csv",
            "dataset": "customers_dwh",
            "mode": "incremental",
            "csv_null": "",
            "csv_force_null": "phone_number"
        }
    ]
}

Configuration specification

#access_token: Your CleverMaps Access Token.
project_id: ID of the CleverMaps project.
server: URL of the CleverMaps environment.
validation: Boolean if the validation checks should be performed.
datasets: Array of the datasets to be loaded into the CleverMaps.
datasets.dataset_enabled: Boolean if the dataset should be loaded or not. Can be useful if you want to temporary disable the dataset.
datasets.input_csv: Absolute path to CSV file in mounted directory. Mounted directory is specified in docker run command.
datasets.dataset: Name of the dataset in CleverMaps in which the CSV file will be loaded.
datasets.mode: Load mode, full or incremental.
datasets.csv_null: Specifies which value should be replaced with null. For example to replace an empty string define single or double quotes.
datasets.csv_force_null: Specifies which columns should enforce the null replacement. Values need to be separated with commas in case of more columns.

How to prepare data for CleverMaps?

The format of the input table must match the target CleverMaps dataset. This includes the number of columns, their order and data types. See this part of our first tutorial to get a better idea.