Input

Intro

This section describes how to get data into Vision 42. Under the input tab you can do this manually:

• Upload data, media files, or archived projects by drag-and-drop or file selection

• Data entry

• Fill in a spreadsheet template to upload

Automatic input can be configured under the IO tab, by administrators. The standard URL syntax is used to describe input locations. A password or other secret goes into the dedicated secret field. Transient errors will be indicated by a clickable exclamation mark. They will be cleared automatically, once solved.

A network protocol method can either push or pull data. Let's have a closer look at both. Live examples can be found in the demo app.

Note: all TLS connections (Transport Layer Security) use a certificate that is signed by a universal trusted CA (certificate authority), domain validated (DV) and valid for 90 days.

Push data

These kinds of protocol methods push data directly into Vision 42. They function real-time, efficiently, and are as such recommended. The polling interval in the input table can be left empty.

HTTP POST

Data or media files can easily be posted to this endpoint: https://vision42.net/{app}/post

Administrators can grant access in the input table, based on:

• User-password: URL http:{user} and secret password

• Token: URL http:token and secret token

• Fixed address: URL http:ip and secret IPv6 or IPv4

The first two require a standard Authorization header, respectively Basic and Bearer. The post endpoint is part of the REST API. Examples using curl:

curl \

--data '{}' \

--user demo:demo \

https://vision42.net/demo/post

curl \

--data @file \

--header 'Authorization: Bearer aGFybWxlc3M=' \

https://vision42.net/demo/post

Note: this method is being used by ElliTrack and Rail 1435.

MQTT subscribe

Vision 42 can subscribe to a data stream on a broker. URLs will typically look like this:

• mqtts://user@broker#topic

• wss://user@broker/path#topic

Available variants, including TLS and WebSocket: MQTT, MQTTS, WS, and WSS. Default, QoS 1 (Quality of Service) and standard version 5 will be used. If needed, exceptions can be made for versions 3.1.1 and 3.1. All topic wildcards are allowed.

To avoid message loss while offline, a persistent session with a 1-day-expiry is being used. However, it is up to the broker to respect this.

Examples and extensions:

• AllThingsTalk

mqtts://{token}@api.allthingstalk.io#device/{device}/#

• The KNMI Notification Service will automatically fetch dataset files, using our dedicated API key. A typical URL is:

wss://_@mqtt.dataplatform.knmi.nl#dataplatform/file/v1/Actuele10mindataKNMIstations/2/#

• Microsoft Azure Event Grid can authorize our certificate.

• The Things Network

mqtts://{application@tenant}@{region}.cloud.thethings.network#v3/+/devices/+/up

MQTT publish

Vision 42 comes with an integrated broker, on which a data stream can be published:

• Protocol: MQTTS

• Address: vision42.net

• Port 8883 (default): production

• Port 8884: quality assurance

• Port 8885: development

• Offline persistence: 1 week, including QoS 0 messages

• Credentials and topic: to be requested by ticket

Identical examples using Mosquitto and MQTT.js:

mosquitto_pub \

--url mqtts://demo:demo@vision42.net/demo/example \

--message '{}'

mqtt publish \

--protocol mqtts \

--username demo \

--password demo \

--hostname vision42.net \

--topic demo/example \

--message '{}'

In the input table, the default URL mqtt:{app}/# subscribes to all of your topics. This URL may be disabled, split up, or narrowed down.

AMQP consume

Vision 42 can consume a data stream from a broker queue. A typical URL will have this form: amqps://user@broker/vhost#queue

Available variants, including TLS: AMQP and AMQPS. Standard 0-9-1 is supported.

Extension:

• Senceive.io URLs can automatically be inserted into the input table, using their HTTP API described below. Binary data will be decoded according to the official definition.

SFTP PUT

Data and media files can be transferred to the integrated server:

• Address: vision42.net

• Port 22 (default)

• Credentials are provided to new customers

• Fingerprint records are publicly available (SSHFP in DNSSEC)

There are two authorization methods available:

• Add the sender's public key to .ssh/authorized_keys (recommended).

• Share the credentials with a trusted party.

Note: to avoid incomplete files, there is a 4 second input delay.

FTP PUT

Similar to SFTP, data and media files can be uploaded to the integrated server:

• Protocol: FTP only

• Address: vision42.net

• Port 21 (default)

• Credentials are provided to new customers and are identical for SFTP

FTP is not recommended because it is insecure and it requires you to share the credentials. However, we will keep supporting FTP PUT, as long as there are devices in use which have no better way to communicate.

Note: to avoid incomplete files, there is a 4 second input delay.

Pull data

These protocol methods pull data periodically into Vision 42. They are inefficient and not real-time, and are only to be used when no push mechanism is available. The polling interval in the input table has to be filled in.

HTTP GET or POST

URLs can have the following base forms, depending:

• https://user@server/endpoint for Basic authentication (secret password)

• https://server/endpoint for Bearer authentication (secret token)

• https://server/endpoint?key=SECRET for parameter authentication (secret value)

Available variants, including TLS: HTTP and HTTPS. To POST parameters as a JSON object, make use of a hash fragment: https://server/endpoint#{JSON}

Examples and extensions:

• Federation of Digital Seismograph Networks

https://{server}/fdsnws/dataselect/1/query?network={network}&station={station}&channel=L*

Wildcards are fully supported. The data will be reduced to a 10 s interval.

• Flanders Hydraulics

https://hicws.vlaanderen.be/KiWIS/KiWIS?timeseriesgroup_id={groups}&period=PT20M&format=csv&metadata=true&datasource=4&request=getTimeseriesValues

• HMS Argos

https://api.netbiter.net/operation/v1/rest/json/system/{system}/log/{parameter}?sortorder=desc&accesskey=SECRET

• KMI - Royal Meteorological Institute of Belgium

https://opendata.meteo.be/service/ows?typenames=aws:aws_10min&service=WFS&request=GetFeature&outputformat=json

https://opendata.meteo.be/service/ows?typenames=synop:synop_data&service=WFS&request=GetFeature&outputformat=json

Private static APIs are also supported, on behalf of paying KMI customers.

• Leica GeoMoS

http://{pc}:8000/v1/projects/{project}/pointsJSON?typesIds={0|100}&apiKey=SECRET

This URL will fetch data of all points in the given project. The typesIds parameter is optional; 0 for measurements and 100 for adjustments.

• Munisense

https://{user}@{portal}.water-munisense.net/webservices/v2/groundwaterwells/query?property=water_level

https://{user}@noise.munisense.net/webservices/v2/soundmeasurementpoints/query?property=daily_overview_lden_day

Any desired property can be included.

• Obscape

https://obscape.com/portal/api/v3/api?parameters=turbidity&username={user}&key=SECRET

This URL will fetch data from all stations. Any desired parameter can be included.

• Profound

https://api.profoundportal.online/v1/external/project/

This URL will fetch 20 s frequencies and velocities of all active projects.

• Rijkswaterstaat

https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/OphalenLaatsteWaarnemingen#{JSON-body}

• Senceive.io

https://{user}@senceive.io/api/consumers/current

This URL will automatically insert all available AMQP queues in the input table.

• Waterinfo

https://download.waterinfo.be/tsmdownload/KiWIS/KiWIS?timeseriesgroup_id={groups}&period=PT20M&format=csv&metadata=true&datasource=1&request=getTimeseriesValues

SFTP GET

URLs come in two flavors:

• sftp://user@server/directory/file to get a growing or rotating file

• sftp://user@server/directory/ to get new or modified files in a directory, non-recursively

While a password may be used, it is recommended to authorize our public key:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0skCWi+E3HMQgeuINQPYnO24K96JIGMVrca9xUzrtnbws31DeDkec0y+DH4ilUpZbEb2I+6krcQYEZnn92/IHBADJj7+iRBtmZJhPvguloaNygyBLF4E0yk6YB2PBdGXuoY/trGWj4h3h9NPKXpbkQwZtfmj8v4cQ7QGxbatmn62/piHfgY9tnL7ZpN9oxO4AQdSUisU4knPZViS7Go4Rj2bPDSVssqI+vCeNONjnvdSHXibwlIb/AiHMXJfSLEZytCnC1Ran9z9fPUWP0xCh+pHVKVv7hVwW6jc0R6081e4amBZOWsMiskoY/0CQGAfek9E6ON8yz7d+k7dKu2QZ Vision 42

Note: moving targets, like YYYY/MM/DD directories, are not supported. After all, any delay in writing would inevitably lead to data gaps.

FTP GET

URLs are similar to SFTP:

• ftps://user@server/directory/file to get a growing or rotating file

• ftps://user@server/directory/ to get new or modified files in a directory, non-recursively

Available variants, including TLS: FTP and FTPS. The former is insecure and thus not recommended. Only extended passive mode is supported (EPSV and EPRT). For large directories, the MLSD command is essential.

Note: moving targets, like YYYY/MM/DD directories, are not supported. After all, any delay in writing would inevitably lead to data gaps.

S3 GET

New or modified objects in a bucket can be fetched using a URL of this form: s3://{access_key_ID}@{endpoint}/{bucket}/{prefix}/#region

The secret access key goes into the secret field. The prefix is optional. For Amazon S3, the endpoint s3.amazonaws.com may be used. Other providers, like Cloudflare R2 or MinIO, will specify the endpoint.

Warning: commercial S3 providers will charge the request costs.

FILE read

The file:{app} URL is used internally to process received data and media files. It allows administrators to pause, slow down, or block all network input.

Our SFTP and FTP servers can be accessed through a client like FileZilla. In addition to uploading files, directories may be created. Transient files can briefly be observed. The meaning of the symbols is:

• ✓ File is complete

• ⯈ File is being processed

• ■ File contains an error

All received data will be stored in the Vision 42 directory, for downloading in bulk. This directory is read-only and divided into sub-directories based on the filename or URL. A sub-directory has generous limits:

• Maximum 200.000 files

• Maximum 2 GB storage

• Maximum 2 years inactive

These limits may change in the future. Stored media files reside in the Vision 42/Media directory.