This guide explains how to create a dataset for data push, i.e. you want to push your data to Luzmo and have it stored in our OLAP optimized database. If you are instead looking to create a dataset based on your connection to your database or plugin, refer to the Dataprovider API service or go to the Luzmo application and add it via the "Add datasets" modal.
In case you there's any data you'd like to programmatically store in Luzmo's OLAP data source instead of connecting your data source to Luzmo, you can use the Data API service to create a new dataset with data, and append or replace data inside previously created datasets. Note that there is a limit of pushing 10000 data points (i.e. rows) per API call, and a storage limit which can be found in our licensing information.
Out of the box, the performance of the underlying datastorage is extremely good for analytical queries. Luzmo can easily handle millions of datapoints. However, by linking several datasets together, the performance will go down which might be a problem for bigger datasets. How much the performance suffers depends on the cardinality of the columns that are linked. In general, it is preferable to store a table with many columns over many linked tables. As a rule of thumb: if the columns will be queried in the same dashboard, they probably belong together inside the same table.
The format of the data payload in the request is an array of arrays: each inner array represents one row of data, and each element inside of it is row value of a column.
This means that each inner array should consist of the same number of elements as there are columns (as defined in the header array), and the position of the data inside the array determines to which column they will be attributed.
You can programmatically create a new dataset by sending (a sample of) data to the Data endpoint, which will store the data in Luzmo's OLAP database.
To give names to your dataset and columns, the header array and name property need to be filled in.
The data sample will be analyzed to determine the types of the different columns. To make sure that the types are detected correctly, it is advised to send a sample for the initial creation where each column has at least a few values.
This example will create four columns in a new dataset and infer their types by detecting types from a data sample
import Luzmo from '@luzmo/nodejs-sdk';
const client = new Luzmo({
api_key: '<your Luzmo API key>',
api_token: '<your Luzmo API token>',
host: 'https://api.luzmo.com'
});
const response = await client.create('data',
{
type: "create",
data: [
[
"Chicken baconito ",
"3",
"500",
"menu"
],
[
"Quesarito",
"3.5",
"700",
"menu"
]
],
options: {
update_metadata: true,
header: [
"Burrito",
"price",
"weight",
"order type"
],
name: {
en: "Burritos"
}
}
}
); When replacing data in an existing dataset, you can provide the dataset's securable_id and set the type to replace . Note that this will replace all data inside a dataset with the new data.
For more information on how to find the id of a dataset, please refer to this section .
Note that you can only replace data in a dataset that was initially created by pushing data into a new dataset .
In case you want to automatically add/remove/update columns or their data type (use this with care), you can set the update_metadata boolean to true. As long as the update_metadata boolean is disabled, this will never change anything to your columns.
In the Append data with metadata update example, two columns are removed and a new column is added.
The columns will remain the same but the data will be completely replaced.
import Luzmo from '@luzmo/nodejs-sdk';
const client = new Luzmo({
api_key: '<your Luzmo API key>',
api_token: '<your Luzmo API token>',
host: 'https://api.luzmo.com'
});
const response = await client.create('data',
{
securable_id: "< dataset ID >",
type: "replace",
data: [
[
"New collection Guacarrito",
"3",
"500",
"menu"
],
[
"New collection Quasarito",
"3.5",
"700",
"menu"
]
]
}
); Similar to replacing data, you can provide an existing dataset's securable_id and set the type to append to append data to an existing dataset.
For more information on how to find the id of a dataset, please refer to this section .
Note that you can only append data to a dataset that was initially created by pushing data into a new dataset .
In case you want to automatically add/remove/update columns or their data type (use this with care), you can set the update_metadata boolean to true. As long as the update_metadata boolean is disabled, this will never change anything to your columns.
In the Append data with metadata update example, two columns are removed and a new column is added.
The columns will remain the same. Existing data inside the dataset will remain unchanged, the new data will be added into the dataset.
import Luzmo from '@luzmo/nodejs-sdk';
const client = new Luzmo({
api_key: '<your Luzmo API key>',
api_token: '<your Luzmo API token>',
host: 'https://api.luzmo.com'
});
const response = await client.create('data',
{
securable_id: "< dataset ID >",
type: "append",
data: [
[
"Fresh new burrito ",
"3",
"500",
"menu"
],
[
"Guacarrito",
"3.5",
"700",
"menu"
]
]
}
); Setting the property update_metadata to true allows you to:
specify a new name for your dataset
specify new header array for the columns in your dataset.
Additially, Luzmo will analyze the data sample to determine the types of the different columns. To make sure that the types are detected correctly, it is advised to send a sample for the initial creation where each column has at least a few values.
Be careful when setting update_metadata to true as this may lead to:
Removed columns. Any data stored in a removed column will be lost
Changed data types
Both removed columns and changed data types could cause query failed in dashboards that rely on these columns. Dashboards can rely on a column in a multitude of ways: it can be used directly in the chart slots, in the filters, to calculate a derived column or aggregated formula, or the column may be used to link two datasets which are in turn used in the dashboard, etc.
In below example, continuing with the dataset we created in the guide Create dataset for data push automatically , this call would introduce a new column 'newColumn' and remove the columns weight and order type.
import Luzmo from '@luzmo/nodejs-sdk';
const client = new Luzmo({
api_key: '<your Luzmo API key>',
api_token: '<your Luzmo API token>',
host: 'https://api.luzmo.com'
});
const response = await client.create('data',
{
securable_id: "< dataset ID >",
type: "append",
data: [
[
"Fresh new burrito ",
"3",
"500",
"menu",
"newColumndata"
],
[
"Guacarrito",
"3.5",
"700",
"menu",
"newColumndata"
]
],
options: {
update_metadata: true,
header: [
"Burrito",
"price",
"weight",
"order type",
"newColumn"
],
name: {
en: "Updated Burritos Name"
}
}
}
);