Wiki

What Are Datasets?

Datasets are where all your data is stored.

When data gets ingested through an Input Connector or submitted through
an Input Form it gets sent to a dataset.

Datasets allows you to:

• Look at your data
• Use AI tools to analyze your data
• Separate sets of data
• Organize your data

Example Uses of Datasets

Separate by Month

You may want to have your data separated by month, without requiring users
or API callers to specify what month they are in.

You can achieve this by changing your default dataset every month.

Archive Data

You may not want all your data to show up in your visualizations or output
connections, so you can archive old datasets and set up a new default for
new data to go into.

Test Dataset

Create a dataset for testing a new system or new additions/edits to an existing
system. By creating a new dataset and using it while testing you can isolate
any adverse effects to only that dataset before figuring out your new setup.

---

Dataset Explorer Overview and Tour

• Dataset Records
  • These are the records in the dataset you are currently looking at. This can show a portion or
    all of the data in a single record.
• Fields Shown
  • These are the fields you have selected to be shown form the dataset. You can drag and drop in
    new fields to see more data. The field tokens also let you know what type each field is.
• Available Fields
  • These are the fields that exist in this dataset that you can drag into the fields shown. This
    will update the data in the dataset records to show you the newly dragged in field.

Editing Data

To edit data, hover over the piece of data in the record you would like to edit and click.

Removing Records

Removing a record removes it permanently. In order to remove a record click the trash can button on the left hand side of the record's row.

AI Anomaly Detection

This AI helps quickly scan through your data to find potential inconsistencies and errors. The AI will even tell you the reason it believe that piece of data to be in error.

• The AI has 3 levels of scrutiny

  • Level 1 checks for general, obvious issues: like if a record has a missing piece of data where most records do not

  • Level 2 checks for more complex issues: like if a number falls outside of the model of expected values that it creates.

    • EX: most fields have a number between 35-200 but it finds a record with a value of 30.

  • Level 3 checks for ANY potential issues: level 3 issues can be very hard to catch, and sometimes these can be inherent in a dataset. Only use level 3 if you plan on heavy levels of scrutiny and want to make sure you're data is 100% clean.

Exporting Data

We export data into multiple different formats.

CSV and TSV

Arrays
Everything in the CSV and TSV format is standard except for Arrays:

Arrays store in the following format <value>|<value>|<value>

• Each value from the given field for that record and array is present
• Each value is separated by a bar "|"
• This field will be treated like a string upon use
  • In order to effectively transform it you must parse the string and seperate by "|"

Spreadsheets

NOTE: Spreadsheet export is still in it's early stages, expect issues, we suggest using the CSV and importing into your spreadsheet instead for now

The spreadsheet generated by exporting your data will include a sheet with the same name as the dataset. So if your dataset is named "March Dataset" the name of the file will be "March Dataset.xlsx" and the sheet in the file will be named "March Dataset"

Everything inside the spreadsheet format is standard except:

Arrays Each array creates a new sheet with the name of the field as the name of the sheet. This new sheet contains the record id and a list of values following it as columns in the same row.

JSON

THIS IS OUR RECOMMENDED FORMAT

The json export is structure as follows:

• The object is an array of json objects
• Each object contains a key called rid for the recordid, and a key for every field in your dataset
• Array are stored as arrays of values
• Text is in quotes ("")

---