Input
Intro
This section describes how to get data into Vision 42. Under the input tab you can do this manually:
Upload data, media files, or archived projects by drag-and-drop or file selection
Data entry
Fill in a spreadsheet template to upload
Automatic input can be configured under the IO tab, by administrators. The standard URL syntax is used to describe input locations. A readable password or other secret can be avoided by filling in the dedicated secret field. Errors will be indicated by a clickable exclamation mark. They will be cleared automatically, once solved.
A network protocol method can either push or pull data. Let's have a closer look at both. Live examples can be found in the demo app.
Note: all TLS connections (Transport Layer Security) use a certificate that is signed by a universal trusted CA (certificate authority), domain validated (DV) and valid for 90 days.
Push data
These kinds of protocol methods push data directly into Vision 42. They function real-time, efficiently, and are as such recommended. The polling interval in the input table can be left empty.
HTTP POST
Hypertext Transfer ProtocolData or media files can easily be posted to this endpoint: https://vision42.net/{app}/post
Administrators can grant access in the input table, based on:
User-password: URL http:{user} and secret password
Token: URL http:token and secret token
Fixed address: URL http:ip and secret IPv6 or IPv4
The first two require a standard Authorization header, respectively Basic and Bearer. The post endpoint is part of the REST API. Examples using curl:
curl \
--data '{}' \
--user demo:demo \
https://vision42.net/demo/post
curl \
--data @file \
--header 'Authorization: Bearer aGFybWxlc3M=' \
https://vision42.net/demo/post
Note: this method is being used by Rail 1435.
MQTT subscribe
Message Queuing Telemetry TransportVision 42 can subscribe to a data stream on a broker. URLs will typically look like this:
mqtts://user@broker#topic
wss://user@broker/path#topic
Available variants, including TLS and WebSocket: MQTT, MQTTS, WS, and WSS. Default, QoS 1 (Quality of Service) and standard version 5 will be used. If needed, exceptions can be made for versions 3.1.1 and 3.1. All topic wildcards are allowed.
To avoid message loss while offline, a persistent session with a 1-day-expiry is being used. However, it is up to the broker to respect this.
Examples and extensions:
mqtts://{token}@api.allthingstalk.io#device/{device}/#
The KNMI Notification Service will automatically fetch dataset files, using our dedicated API key. A typical URL is:
wss://_@mqtt.dataplatform.knmi.nl#dataplatform/file/v1/Actuele10mindataKNMIstations/2/#
Microsoft Azure Event Grid can authorize our certificate.
mqtts://{application@tenant}@{region}.cloud.thethings.network#v3/+/devices/+/up
MQTT publish
Message Queuing Telemetry TransportVision 42 comes with an integrated broker, on which a data stream can be published:
Protocol: MQTTS
Address: vision42.net
Port 8883 (default): production
Port 8884: quality assurance
Port 8885: development
Offline persistence: 1 week, including QoS 0 messages
Credentials and topic: to be requested by ticket
Identical examples using Mosquitto and MQTT.js:
mosquitto_pub \
--url mqtts://demo:demo@vision42.net/demo/example \
--message '{}'
mqtt publish \
--protocol mqtts \
--username demo \
--password demo \
--hostname vision42.net \
--topic demo/example \
--message '{}'
In the input table, the default URL mqtt:{app}/# subscribes to all of your topics. This URL may be disabled, split up, or narrowed down.
AMQP consume
Advanced Message Queuing ProtocolVision 42 can consume a data stream from a broker queue. A typical URL will have this form: amqps://user@broker/vhost#queue
Available variants, including TLS: AMQP and AMQPS. Standard 0-9-1 is supported.
Extension:
Senceive.io URLs can automatically be inserted into the input table, using their HTTP API described below. Binary data will be decoded according to the official definition.
SFTP PUT
SSH File Transfer ProtocolData and media files can be transferred to the integrated server:
Address: vision42.net
Port 22 (default)
Credentials are provided to new customers
Fingerprint records are publicly available (SSHFP in DNSSEC)
There are two authorization methods available:
Add the sender's public key to .ssh/authorized_keys (recommended).
Share the credentials with a trusted party.
FTP PUT
File Transfer ProtocolSimilar to SFTP, data and media files can be uploaded to the integrated server:
Protocol: FTP only
Address: vision42.net
Port 21 (default)
Credentials are provided to new customers and are identical for SFTP
FTP is not recommended because it is insecure and it requires you to share the credentials. However, we will keep supporting FTP PUT, as long as there are devices in use which have no better way to communicate.
Pull data
These protocol methods pull data periodically into Vision 42. They are inefficient and not real-time, and are only to be used when no push mechanism is available. The polling interval in the input table has to be filled in.
HTTP GET or POST
Hypertext Transfer ProtocolURLs can have the following base forms, depending:
https://user@server/endpoint for Basic authentication (secret password)
https://server/endpoint for Bearer authentication (secret token)
https://server/endpoint?key=SECRET for parameter authentication (secret value)
Available variants, including TLS: HTTP and HTTPS. To POST parameters as a JSON object, make use of a hash fragment: https://server/endpoint#{JSON}
Examples and extensions:
https://hicws.vlaanderen.be/KiWIS/KiWIS?timeseriesgroup_id={groups}&period=PT20M&format=csv&metadata=true&datasource=4&request=getTimeseriesValues
KMI - Royal Meteorological Institute of Belgium
https://opendata.meteo.be/service/ows?typenames=aws:aws_10min&service=WFS&request=GetFeature&outputformat=json
https://opendata.meteo.be/service/ows?typenames=synop:synop_data&service=WFS&request=GetFeature&outputformat=json
Private static APIs are also supported.
http://{server}:8000/v1/projects/{project}/pointsJSON?typesIds={0|100}&apiKey=SECRET
This URL will fetch data of all points in the given project. The typesIds parameter is optional; 0 for measurements and 100 for adjustments.
https://{user}@{portal}.water-munisense.net/webservices/v2/groundwaterwells/query?property=water_level
https://{user}@noise.munisense.net/webservices/v2/soundmeasurementpoints/query?property=daily_overview_lden_day
Indoor climate is also supported. Any desired property can be included.
https://obscape.com/portal/api/v3/api?parameters=turbidity&username={user}&key=SECRET
This URL will fetch data from all stations. Any desired parameter can be included.
https://api.profoundportal.online/v1/external/project/
This URL will fetch frequencies and velocities of all active projects.
https://waterwebservices.rijkswaterstaat.nl/ONLINEWAARNEMINGENSERVICES_DBO/OphalenLaatsteWaarnemingen#{JSON-body}
https://{user}@senceive.io/api/consumers/current
This URL will automatically insert all available AMQP queues in the input table.
https://download.waterinfo.be/tsmdownload/KiWIS/KiWIS?timeseriesgroup_id={groups}&period=PT20M&format=csv&metadata=true&datasource=1&request=getTimeseriesValues
SFTP GET
SSH File Transfer ProtocolURLs come in two flavors:
sftp://user@server/directory/file to get a growing or rotating file
sftp://user@server/directory/ to get new or modified files in a directory, non-recursively
While a password may be used, it is recommended to authorize our public key:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0skCWi+E3HMQgeuINQPYnO24K96JIGMVrca9xUzrtnbws31DeDkec0y+DH4ilUpZbEb2I+6krcQYEZnn92/IHBADJj7+iRBtmZJhPvguloaNygyBLF4E0yk6YB2PBdGXuoY/trGWj4h3h9NPKXpbkQwZtfmj8v4cQ7QGxbatmn62/piHfgY9tnL7ZpN9oxO4AQdSUisU4knPZViS7Go4Rj2bPDSVssqI+vCeNONjnvdSHXibwlIb/AiHMXJfSLEZytCnC1Ran9z9fPUWP0xCh+pHVKVv7hVwW6jc0R6081e4amBZOWsMiskoY/0CQGAfek9E6ON8yz7d+k7dKu2QZ Vision 42
Notes:
Only secure ciphers are supported. In case of connection issues, please update out-of-date SFTP software.
Moving targets, like YYYY/MM/DD directories, are not supported. After all, delayed writing would lead to data gaps.
For the moment, the natural order of filenames must be chronological.
FTP GET
File Transfer ProtocolURLs are similar to SFTP:
ftps://user@server/directory/file to get a growing or rotating file
ftps://user@server/directory/ to get new or modified files in a directory, non-recursively
Available variants, including TLS: FTP and FTPS. The former is insecure and thus not recommended. Only extended passive mode is supported (EPSV and EPRT).
Notes:
Moving targets, like YYYY/MM/DD directories, are not supported. After all, delayed writing would lead to data gaps.
For the moment, the natural order of filenames must be chronological.
FILE read
The file: URL is used internally to process received data. It allows administrators to pause or slow down all network input.
S3 GET
Simple Storage ServiceCurrently under evaluation