Data-flo
Data-floSource CodeCGPS
  • INTRODUCTION
    • What is Data-flo
    • Getting Started - Sign In
    • Privacy and Terms Of Service
    • Contact - Help & reporting errors
    • Change log
  • USING DATA-FLO
    • Data-flo site navigation
      • Transformations Page
      • Run Page
      • Canvas
    • Data
      • Bringing data in to Data-flo
      • Getting data out of Data-flo
      • Data Types
        • Boolean
        • Datatable
        • File
        • Graph
        • List
        • Map
        • Number
        • Text
    • Regular Expressions (RegEx)
    • Adaptors overview
      • Components of an adaptor
      • Binding types
        • Bind to Data-flo input
        • Bind to value
        • Bind to another transformation
    • Specific adaptors
      • add-column
      • append-to-list
      • calculate-time-difference
      • change-column-case
      • columns-concatenation
      • concatenate-text
      • create-microreact-project
      • csv-file-to-datatable
      • csv-to-datatable
      • datatable-columns
      • datatable-to-csv-file
      • datatable-to-graph
      • datatable-to-list
      • datatable-to-map
      • datatable-to-sqlite-file
      • date-to-text
      • dbf-file
      • dot-to-graph
      • download-file
      • dropbox-file
      • epicollect-project
      • extend-datatable
      • figshare-file
      • file-to-text
      • filter-blank-values
      • filter-columns
      • filter-list
      • filter-rows
      • filter-rows-numerically
      • force-directed-layout
      • format-date-column
      • forward-geocoding
      • ftp-file
      • gather-rows
      • google-drive-file
      • google-spreadsheet
      • graph-to-dot
      • join-datatables
      • list-to-datatable
      • lookup-map-value
      • merge-datatables
      • merge-lists
      • microreact-project
      • mysql-database
      • newick-leaf-labels
      • oracle-database
      • postgress-database
      • prepend-to-list
      • remove-columns
      • remove-duplicate-rows
      • rename-columns
      • replace-blank-values
      • replace-column-values
      • replace-text
      • replace-text-in-list
      • replace-value
      • reverse-geocoding
      • row-column-value
      • s3-file
      • select-columns
      • send-email-message
      • slice-datatable
      • slice-list
      • smb-file
      • sort-datatable
      • sort-list
      • split-column
      • split-datatable-rows
      • split-list
      • split-text
      • spread-rows
      • spreadsheet-file
      • sql-server-database
      • sqlite-database
      • sum-rows
      • text-template
      • text-to-file
      • unique-list-items
      • update-epicollect-entries
      • update-microreact-project
      • update-smb-file
      • upload-file-to-google-drive
      • upload-files-to-google-drive
      • url-builder
      • yaml-to-json
    • Building a data-flo
      • Debugging mode
      • Show detailed errors on Run Page
      • Permissions - Access Control
    • Tips & Tricks
  • TUTORIALS
    • Prep outbreak data for Microreact
    • Common use cases, solved
      • Fixing datatable headers
      • Select, remove, rename, reorder columns
      • Data in separate files
      • There's no single-column unique row ID (primary key)
      • Ensure non-dates stay non-dates
      • Connect directly to a database
      • Access files on a drive
Powered by GitBook
On this page
  • VALID
  • NOT VALID
  • Problem: No column names in input data
  • Solution: Add a header row containing column names.
  • Problem: Extra header rows before column names
  • Solution: Limit the range of data imported

Was this helpful?

  1. TUTORIALS
  2. Common use cases, solved

Fixing datatable headers

All datatables require a single line of headers for column names.

PreviousCommon use cases, solvedNextSelect, remove, rename, reorder columns

Last updated 2 years ago

Was this helpful?

If data have no column names, or if there is extra header space before the row containing column names, the data require editing before use. The top row will be interpreted as column names, so inputs need to be formatted that way.

VALID

A datatable has a single row of headers as column names, without any other rows above it or extra rows after it before the data rows begin.

columnName1,columnName2,columnName3
data1a,data1b,data1c
data2a,data2b,data2c
data3a,data3b,data3c

NOT VALID

Problem: No column names in input data

This CSV has no headers, so the first row of data gets interpreted as column names.

data1a,data1b,data1c
data2a,data2b,data2c
data3a,data3b,data3c

Solution: Add a header row containing column names.

If the file is small:

  1. Whatever format your data are in, get them into CSV.

  2. Use Concatenate-text adaptor to prepend column names to the data. Left argument (Bind to value) is the comma-separated series of desired column names, ending with a newline. Right argument (Bind to another transformation) is the csv from the data input.

  3. Combination output argument then connects to the next step (probably CSV-to-datatable)

If the file is large: Avoid converting to text. Instead,

  1. Use Datatable-columns to turn the header row (which is really supposed to be your first row of data) into a list

  2. use List-to-datatable so the list becomes the data in a new datatable column

  3. use Add-column to create a dummy column (for use in Spread-rows)

  4. use Spread-rows to turn the column of column names into the values in a single row

  5. use Remove-column to get rid of the dummy column. Now your first row of data is both the header and the first row of data.

  6. Finally, use Merge-datatables with this as 'first data' and the original datatable (which has the first data-row as column names instead of as data) as 'second data'.

  7. Now you can Rename-columns and have all your data as data.

Problem: Extra header rows before column names

In this situation, the extra information (or empty space) in the header gets interpreted as column names, so you need to remove it.

information above the header
more information above the header
columnName1,columnName2,columnName3
data1a,data1b,data1c
data2a,data2b,data2c
data3a,data3b,data3c

Solution: Limit the range of data imported

  1. Use spreadsheet-file or google-spreadsheet adaptor to import the file.

  2. Use the **RANGE ** argument to remove rows at the top of the sheet. In the example data above, the real headers are in Row3. The desired import Range will be A3 until end of file.

    1. If you are unsure what the last row will be, choose something larger than expected (e.g. range=“A3:P20000”.

    2. The output of the adaptor will only include rows and columns that contain data within this specified range; you won’t get empty cells.

To see this example, make a copy of so you can see how it is built.

To see this example, make a copy of so you can see how it is built.

this Data-flo
this Data-flo
Screenshot of example data-flo
Screenshot of example data-flo