Data

JSON family

• JavaScript Object Notation (JSON)

• JSON Lines (JSONL), for streaming

• Newline delimited JSON (NDJSON), for streaming

• Prefixed JSONL

• Prefixed NDJSON

• AllThingsTalk

• Basetime, timezone = UTC

• Buildwise fiber

• CEMS

• Crodeon

• Daizy

• ElliTrack

• GeoJSON

  • Royal Meteorological Institute of Belgium

• HMS Argos

• IXON Edge

  • Siemens PLC

• Leica GeoMoS

• MicroSense

• MIMIC

• Munisense

  • • Ground water wells

    • Sound measurement points

• Obscape

• Profound, 2 s and 20 s

• Rail 1435

• Rijkswaterstaat

• ROAK

• Speedtest

• The Things Network

• Xeelas

The benefits of delimiter-separated values apply.

Delimiter-separated values

Most data formats will be imported in a generic fashion. This will automatically handle:

• All known byte order marks

• Detection of the character encoding

• Common value delimiters

• Cell trimming

• Date-time and timezone notation

• Partial or missing date-time information

• Multiple date and/or time columns

• Analysis of metadata schemata

• Metadata gathering, including serial numbers and firmware / hardware / software versions

• Analysis of data structures

• Concatenated data, for consistent delimiter and time column(s)

• (Live) coordinate transformation

Note this includes comma-separated values (CSV) and RFC 4180.

These column designations might come in handy (case and white-space insensitive):

• CHANNEL [ UNIT ]

• DEVICE | CHANNEL | UNIT

• EASTING, NORTHING, ELEVATION, and/or SURFACE

At least these delimiter-separated data formats are supported:

• AE Sensors LoRa

• ARGUS

• ARGUS (Iv)

• Azimut Monitoring Greenbee

• BARTEC SYSCOM, format versions 1, 3, and 4 at least

• Brem optical fiber, with and without extensive header

• Campbell Scientific (CSV)

• Campbell Scientific (TOACI1)

  • Measurand ShapeArray

• CAP (English)

• CoMeT

• Ekopower iBOX

• ElliTrack

• Empty files

• FBGS FBG-scan

• Fluvius

• GeoCSV, with a date-time column

• GeoMetiri

• Geosense Linx

• Geosense Linx Connect

• Immontec

• Inventec

• Keller ADT

• Keller Datamanager

• KISTERS WISKI, CSV format with metadata

• MEET HET, ambiguous profile data excluded

• Microsoft Excel (CSV)

• OSMOS

• Profound VIBRA(+) (CSV)

• Royal Meteorological Institute of Belgium

• RST Instruments data logger

• rsync.net

• Senceive (CSV), including in-place inclinometers

• Smet Group

• Soil Instruments VWLog8 GPRS

• SVANTEK (CSV)

• SVANTEK (SvanPC++)

• Sylex

• Villari (CSV and XLSX)

• Vision 42 (legacy)

• Vista Data Vision

Tip: Feel free to use our demo to try out (other) data formats.

Binary formats

• Apache Parquet, excluding the decimal128 floating-point format

• Hierarchical Data Format (HDF)

• LoRaWAN Payload Codec API

• Microsoft Excel (XLSX), a template can be downloaded from the Input section.

• Milesight

• miniSEED

• Network Common Data Form (netCDF)

  • Royal Netherlands Meteorological Institute

• Profound VIBRA(+)

• Protocol Buffers based on your proto definitions.

• Senceive.io

Other binary formats will be skipped.

Inclinometers

• DGSI Digitilt Uniform eXchange (DUX), including rotation

• Geosense

• GLÖTZL

  • Including German and legacy variants.

  • The legacy year 2.000 problem will be corrected automatically. 

• RST Instruments

• Soil Instruments Vertical System

The benefits of delimiter-separated values apply.

Other formats

• ARGUS (export)

• Campbell Scientific (TOA5)

  • Generic in-place profiles

  • Geokon in-place inclinometer

  • Osprey automatic settlement profiler (ASP), may be partially rolled-up

  • RST Instruments in-place inclinometer

• Diver

  • Including CTD-Divers.

  • Including LDM EAE- and VEI-edition software.

  • Including EnviroMon software.

  • Including Dutch variants.

• Endress+Hauser

• Geosense Wi-SOS

  • GEOSTRING in-place inclinometer

• Geosys

• Inducon

  • Exceptionally, the recognition of this data format is based solely on the file name, because no more metadata is available, unfortunately.

  • Both the instrument and sensor names will be taken from the file name.

• Measurand ShapeArray

• network Common Data form Language (netCDL)

• Sisgeo WR LOG

• Profound

• Solinst Levelogger software

Media files

• Exchangeable image file format (Exif)

• Joint Photographic Experts Group (JPEG)

• GeoJSON, file extension .geojson

• Graphics Language Transmission Format (glTF)

  • Self-contained binary .glb and JSON/ASCII .gltf are supported.

  • Large 3D models require a decent graphics card, at least.

  • Note the OpenGL axis convention (x,z,-y) versus typical GIS (x,y,z).

• Keyhole Markup Language (KML), file extension .kml

• Portable Document Format (PDF)

• Portable Network Graphics (PNG)

In absence of more specific information, these and all other formats will make use of:

• The date and time of the file itself

• The location of the upload device, when allowed

Identification of files

Vision 42 aims to function fully automatically, using a data chain that is as short as possible while avoiding Excel processing and intermediate servers.

The app will automatically link files to instruments based on data in the file, usually the header. In absence of this, the last alphanumeric word from the filename (for input by upload) and/or the full path (for continuous input) will be used instead. An underscore will usually be considered part of the identifier. This to match BH_14 and not BH in a filename, for example.

In case of a new file with previously unknown instruments / sensors, the application will populate the system with the correct instruments and their position.

Note that within an instrument, sensors will be matched by column number, if applicable.

Caution: As an administrator, you may change the identification of an instrument. This can break things, so only do this if you know exactly what you are doing.

Metadata

Barring exceptions, metadata such as units is only read the first time. This is consciously so, to make changes in the application possible. If not, the data would be overwritten every time, by metadata in files over which you may not have control.

Calculations and reimporting

Calculations are performed during import. A notable exception is the dynamic calculation of values relative to a given instrument datum. This method of calculating allows for parameters that vary over time. Here are some examples:

• A defective sensor has been replaced with a new one. The calibration formula needs to be changed and will be automatically applied to subsequent values.

• After a settlement, the instrument elevation has changed. Subsequent values will be calculated with the appropriately changed elevation parameter.

This method is simple and avoids having to define the full time course of each calculation parameter.

Reimporting requires little effort:

1. Plot menu > Download file(s)

2. Input > Upload > Overwrite previous values

3. Input > Upload > Choose files or just drag and drop

Coordinate transformations

Regional coordinates will be transformed and updated in bulk, using the project's given coordinate system. This means instrument locations will will be updated accordingly, even live.

• For coordinates relative to an instrument datum, the corresponding value in the file will be used. An example of this is difference in topography.

• In case of absolute coordinates, the most recent value from the database will be used. This allows for application of formulas on coordinates. For example, some tunnel boring machines only send the last digits. A simple addition can get you complete coordinates.

• Coordinate updates will adhere to the do-not-import flag.

About time

The timezone field, in the instruments and data loggers table, dictates how times should be interpreted on input. Time zones -12:00 to +14:00 denote a fixed offset from UTC (Coordinated Universal Time). These zones never adhere to daylight savings time. All other time zones are defined by politics and their definition tends to change throughout history. Past and future times will be interpreted accordingly. A zone in the time notation always gets priority.

Vision 42 can handle milliseconds. For the time format, fractional seconds of arbitrary precision are being read. For the database, this is rounded to milliseconds, because the hard limit of the plots is 1 millisecond.

Editing in Excel

For example, editing in Excel can be useful for correcting depth errors in inclinometer measurements. After missing a "click," the remaining column can be slid down and the missing value of the previous measurement can be inserted. Using an ordinary editor, this is all too laborious.

Inclinometers

On import, cumulative profiles will be calculated automatically. The deepest value is the origin, inherited from the previous, non-invalid profile. Usually, the origin will be zero, except for shortened inclinometers. The inclinometer length may vary over time.

A cumulative profile begins half a gauge deeper and ends half a gauge higher, while the intermediate values fall between the angles. These interlaced vertices will be numbered higher, following the original vertices.

Note that the top angle measurement has a relative depth of zero and that even depths may vary over time.

Internet of Things (IoT)

The Vision 42 app has extensive provisions for the Internet of Things. The following technologies have been fully integrated and tested:

• LoRa - Long Range

• LoRaWAN - Long Range Wide-Area Network

• The Things Network, including the Storage Integration.

Private LoRa networks

There is a good change that Vision 42 will be able to import data from your private LoRa network. Add a continuous input with a custom MQTT URL. Please see the documentation of your private LoRa server / gateway.

However, your private MQTT broker has to be reachable from our servers without the need for VPN or equivalent.