If you do not want to pollute the production environment with test data, you can use the QA environment at https://vision42.net/qa/customer/ (Quality Assurance). Note that data in QA will be overwritten daily by production data.
In QA, you can try things out with impunity. This is especially useful for new project files or for trying out new data formats. Large reports can be run in this QA-environment, without impact on production performance.
Vision 42 deliberately integrates well with password managers. We strongly advice to fully make use of a password manager. It can generate a strong random password, which it will remember and even fill in for you. Many password managers can even function across devices. A quick start is to use the Google Password Manager that is built into our recommended browser, Google Chrome.
Remark: In the rare case where you have to access Vision 42 instances of two different customers, make use of different domains in the URLs - for example vision42.net and vision42.eu. If not, a password manager can get confused, because it ony can handle a single cryptographically encoded password per domain (and per account).
Identity and access management
The authentication and authorization mechanisms of Vision 42 can be replaced by an OAuth2 integration with an external identity and access management (IAM) system. This may be your own IAM-platform or that of a third party like Google, LinkedIn, Facebook, and countless others. User properties and even roles can be synchronized.
The following limitations apply:
- Only a single OAuth2 provider can be integrated at once.
- The OAuth2 endpoints have to be reachable from our servers without the need for VPN or equivalent.
- Integration requires some professional services which are not included in the monthly subscription.
If necessary, any visible information in the app will be automatically refreshed after 30 seconds at most. This refresh will be delayed for elements the user is interacting with. For most elements, the focus will be used to determine this. For instance, a cursor in an editable input field will pause refreshing. For maps, this is an open plot window. It is not easy to keep all information up-to-date and to never disturb the user in the process.
Tip: The search function of the browser can be used throughout Vision 42. For most computers, its keyboard shortcut is Ctrl + F. For Apple computers, this is ⌘ F. In the Map section, this can be especially useful to quickly select a project by part of its name or code.
The edit function in the sliding footer can be used to position instruments visually on the map.
The markers on the map indicate instrument locations and names. Their color denotes their alert status: green, yellow, or red. A white marker stands for an outage alert: no measurements have been received for a while, ie. 10 minutes plus six intervals.
Businesses are hidden
In urbanized areas, like city centers, the map could become quite cluttered. Of course you could turn off the labels and end up with the aerial photos and nothing else. Granular hiding of businesses (bars, restaurants, retail, ...) has made the default hybrid map much less crowded, while retaining street names and other useful information such as metro stations, schools (traffic congestion) and the like.
Tunnel boring machines
Tunnel boring machines will be indicated by a specific icon. Their locations can be updated automatically and in real time.
In some cases, the automatic refreshing of the map markers may hinder your workflow. As a workaround, a plot window can be left open. This will effectively postpone refreshing. More information can be found here.
All plots and most tables have a download symbol. It lets you export the current table or the data of the current plot to a csv-file (comma-separated value) which is automatically or easily imported into any spreadsheet application, like Microsoft Excel.
By default, Vision 42 will plot a profile in the conventional way; with the x-axis pointing right and the y-axis pointing up. Vertical measurements will be plotted vertically, with an optional ground level indication. The orientations of the x-axis and the multiple y-axes will be set automatically, based on the metadata information received. Here are some examples, with the name of the vertical axis mentioned between brackets:
- Measurand SAA (Z)
- IPI via Campbell Scientific (z)
- RST Instruments inclinometer (Depth)
- Glötzl inclinometer (Step)
- SAA via Leica GeoMoS (Sensor_Z_)
Vision 42 was designed to function fully automatically and in real time. Usually, the loggers or instruments send their data directly to us, or the app will fetch them from somewhere. Next the cumulative profiles of the above inclinometers will be calculated, for example. Plotting of values relative to a dynamic instrument datum is embedded in the application, as well is recalculation to absolute levels in mASL.
Note that Vision 42 will calculate cumulative inclinometer profiles in between the given depths and that depths may even vary in time. Checksums will not be plotted by default, nor will their depths show up. This behavior can be controlled by the do-not-plot flag.
- can be marked using a graphical selection of the beginning and the end.
- will not be plotted.
- are excluded from tables with manual water level measurements.
- will not be sent through the REST API.
- will not be sent to the anomaly detector.
- will not be used to calculate formulas.
- will not be used when compensating with air pressure.
Marking measurements as invalid has the advantage over deleting, in that they do not return when one reimports data. Invalid measurements will only become deleted using Delete file or Delete series in the plot menu.
For profile data, marking a value of the first vertex (Z1, Depth1, ...) as invalid will exclude that time point from the slider.
- ARGUS, including automatic coordinate transformation and update of location.
- BARTEC SYSCOM
- Coordinate transformations and updates in bulk.
- Make use of this case-insensitive header:
- Instruments with a matching ID will have their elevation (if any) updated. If both easting and northing are present, they will be transformed using the project's given coordinate system before being updated in the database.
- Make use of this case-insensitive header:
- Campbell Scientific (TOA5)
- Campbell Scientific (comma separated)
- DGSI Digitilt Uniform eXchange
- The decimal separator can be either a point or a comma.
- Exceptionally, the recognition of this data format is based solely on the file name, because no more metadata is available, unfortunately.
- Both the instrument and sensor names will be taken from the file name.
- Iv Infra
- Always starts with
- Following column definitions must be unique and have the form of
- The field separator is a semicolon.
- The decimal separator is a point.
- The character encoding has to be Windows-1251.
- Always starts with
- JSON, for internal use.
- Leica GeoMoS, including automatic coordinate transformation and update of location.
- netCDF & CDL
- Profound VIBRA(+)
- RST Inclinometer
- Dates and times will be interpreted as UTC, because they have to be consistent across all Senceive projects.
- Many delimiter-separated values
Automatic detection of the date/time notation, the delimiter, and selected metadata.
- Brem optical fibers
- Campbell Scientific (TOACI1)
- Both English and French variants are supported.
- Including automatic coordinate transformation and update of location.
- CoMeT, including automatic coordinate transformation and update of location.
- Ekopower iBOX
- FBGS FBG-scan
- Geosense Linx
- Geosense Wi-SOS
- Soil Instruments VWLog8 GPRS
- Vista Data Vision
- Generic: a date and time, followed by measured values.
In these data formats, columns named easting, northing or elevation will be used as such. Only the most recent value in the database will be taken into account.
Please note that the files from ARGUS, Iv Infra, Leica GeoMoS, and Senceive are currently not stored within Vision 42. This is because they can not be linked with a single data logger or a single instrument. After all, these files contain a complete project or site. Only original data files are stored by Vision 42, for the moment.
For this type of data format, instrument identifications must be unique across projects. A compelling reason for this, is the lack of other (sufficiently) unique information such as serial numbers. A simple solution is to use a unique prefix per project. After the initial import, you can remove that prefix from the instrument names, as you see fit.
An error will be raised as long as there are duplicatie sensors.
Identification of files
Vision 42 aims to function fully automatically, using a data chain that is as short as possible while avoiding Excel processing and intermediate servers.
The app will automatically link files to instruments based on data in the file, usually the header. In absence of this, the last alphanumeric word from the filename (for input by upload) and/or the full path (for continuous input) will be used instead. An underscore will usually be considered part of the identifier. This to match BH_14 and not BH in a filename, for example.
In case of a new file with previously unknown instruments / sensors, the application will populate the system with the correct instruments and their position.
Note that within an instrument, sensors will be matched by column number, if applicable.
Caution: As an administrator, you may change the identification of an instrument. This can break things, so only do this if you know exactly what you are doing.
Vision 42 can import project data continuously from multiple FTP, FTPS, HTTP, HTTPS, SFTP, SCP, MQTT, MQTTS, (OPeN)DAP, and (OPeN)DAPS locations. The standard URL syntax must be used to configure import locations. We advice making use of the separate password field, to avoid a readable password in the URL.
A location may be a growing file, or a directory in which new files arrive. Possible errors can be found under the Setup section.
Continuous input will discard older values, except for:
Continuous input from directories, with the "Delete files" flag off, has a fairly strict requirement: the natural order of filenames must be chronological. Vision 42 keeps track of which file and which line were last imported, and will continue from there on the next time. This is necessary because directories can contain millions of files and obtaining modification times is on a per-file basis. In combination with often under-performing FTP servers, this would at least undermine the quasi-real-time nature of the Vision 42 app, and often become undesirably slow and cause heavy system load.
FTP and FTPS servers must support passive mode. If not, connections will fail.
Built-in SFTP and FTP servers
From within the application, an administrator can ask for access to the built-in SFTP and FTP servers. This is a one-time step. From then on, all received files will automatically be imported. You may pass the credentials to your trusted integrators.
Only files that are at least 5 seconds old will be imported via the built-in SFTP and FTP servers. This is so to prevent data loss. After all, while a file is being written to, an incomplete import and an optional deletion could otherwise occur.
In case of external SFTP and FTP connections, this check does not happen because there is no guarantee that the clocks of both servers will run synchronously.
Barring exceptions, metadata such as units is only read the first time. This is consciously so, to make changes in the application possible. If not, the data would be overwritten every time, by metadata in files over which you may not have control.
Accuracy in terms of time
Vision 42 can handle milliseconds. For the time format, up to one tenth of a microsecond is being read. For the database, this is rounded to milliseconds, because the hard limit of the plots is one millisecond.
Editing in Excel
For example, editing in Excel can be useful for correcting depth errors in inclinometer measurements. After missing a "click," the remaining column can be slid down and the missing value of the previous measurement can be inserted. Using an ordinary editor, this is all too laborious.
There is an automatic cleaning function, meant after editing in a European tinted Excel. Among other things, the added empty fields at the end of a line will be removed, the semicolons will be replaced with commas and numbers are given a point instead of a comma as a decimal point.
It goes without saying that standard CSV files are preferred and can avoid certain exotic problems. Not-too-old versions of Excel can write multiple types of CSV files using "save as."
A standardised exchange format that covers all needs is unfortunately unknown to us. Should there be or come one, we will not fail to implement it.
The most common form of file format is the well-known DSV (delimiter-separated values) with the date and time in the first column(s), followed by measurements in columns per quantity. Our software cleverly reads such files by estimating both the separator, the date-time notation, and the metadata.
At the time of writing, all message types are supported:
- DATA - UPLINK
- DATA - BIDIR: An acknowledgement will be sent back as downlink data.
- SERVICE - STATUS
- SERVICE - GEOLOC: The latitude and longitude of the instrument will be updated.
- SERVICE - ACKNOWLEDGE: The info message will be written to the error table.
- SERVICE - REPEATER
- SERVICE - DATA_ADVANCED
- ERROR: The severity and the error info will be written to the error table.
Two channels are supported: URL and BATCH_URL.
Use https://vision42.net/customer/iot.php as the URI pattern. Note the quality assurance environment is also supported.
All HTTP methods are supported (GET, POST, and PUT), as are both URL and JSON encoding.
Make use of the default variable names, as can be seen in this example:
When using the Sigfox custom payload configuration, prefix your data variables with "customData_".
Alternatively, a variable named "format" may be used to pass the payload data format. The syntax can be found here: http://php.net/manual/en/function.pack.php.
If no data format is specified, Vision 42 will try to guess the data format out of:
- 4-byte IEEE 754 floats, little-endian.
- 2-byte unsigned integers, little-endian.
- 1-byte unsigned integers.
Next to the variables in the payload, these standard variables will be imported, when not manually excluded:
- The signal to noise ratio (SNR) in dB.
- The received signal strength indication (RSSI) in dBm.
- The average signal to noise ratio in dB.
- The battery voltage in Volts.
- Temperature in degrees Celcius.
- The radius in meter.
- The number of white-listed devices.
- The number of repeated messages.
- The number of unwanted devices.
- The range of unwanted messages.
- The range of malformed frames.
- The range of messages over regulation.
Behavior of tables
The column header is like a search bar, with ...
- Dynamic search on text fields.
- Filtering on selected items.
- Sorting on all columns.
Some more features:
- It has input functionality, depending on given rights.
- You can delete entries, again depending on given rights.
- It’s smart if you want to see more lines, but fast because it does not show all data. It doubles the number of extra entries shown each time the icon in the caption area is clicked.
You can easily transfer a sensor to another instrument, using the drop-down list in the first column of the sensor table. This will not be a problem for new sensor data. Vision 42 will remember where the data was coming from.
Virtual and manual sensors
Instruments and sensors will be automatically created on import. In case of virtual or manual sensors, you will want to create these yourself, since there is no data available to import.
New sensors can be created under the SENSORS tab. Make use of the plus sign + at the bottom of the sensor table. An instrument (whether virtual or not) must be entered. Next, fill in the formula of the virtual sensor or set the manual flag to mark the sensor as such.
Virtual sensors will be exempted from outage indication and outage alerting. This because their interval is not guaranteed to be periodic.
Example: given two instrument sensors, BH1-level and BH2-level. Create a new sensor difference at the bottom of the sensor table, for example under instrument BH1. For the sensor difference, enter the formula  - , with the corresponding keys to the right of the sensor table.
In the tooltip of the formula you will find a summary example in terms of syntax:
[key1]>1,2 ? 3.4E-5**6 : log([key2];10)+elevation²-position³
Whenever a source sensor changes, the formula will be calculated. An exception to this behavior is a (calibration) formula which depends on its own sensor. The calculations can also be done for the past. Re-importing is the trigger for this.
You can clone a single sensor using the simple formula .
The measurement freshness in formula calculations is asymmetrical:
- 150% of the interval for earlier values, keeping late arrivals in mind.
- 110% of the interval for later values, to allow for small interval variations.
Calibrated sensor data can be achieved by adding a calibration formula to a sensor.
In case this formula depends on the own sensor, the original value will be replaced by the calibrated value. Such a formula will only be calculated for the points in time of the original sensor.
Alternatively, a virtual sensor can be used to store the calibrated data alongside the original data.
Air pressure compensation
The automatic air pressure compensation only uses linear interpolation between a time up to 110% of the interval before and a time up to 110% after. This provides the most correct value. The formulas do the same - using 150% / 110%, supplemented with substitution with the previous or the next measurement within 150% / 110% of the interval. Extrapolation is never considered because it produces incorrect values near inflection points. If no near measured values are found, previously calculated values are deleted by both mechanisms - except for upload without the overwrite check mark. When importing, all relationships between values and calculations are taken into account. This ensures, for example, that all air pressures are read first and then the dependent calculations are made.
The formula mechanism is therefore more recent, better and more suitable for continuous measurements and alarms. The intention is to replace the automatic air pressure compensation and all other implicit calculations with formulas, with or without a formula builder or the like.
In summary, the formulas will currently do much better than the automatic air pressure compensation. A difference between the measurement and the send interval can still get in the way here, but to a lesser extent. We do not wish to go much further than 125% of the interval, because we strive for a certain balance of correctness and usability.
Cleaning up files
Through a background process, where unreferenced files of at least one week old are removed; this way there will be multiple backups. References without a file are also continuously checked.
A coordinate system and a unit of elevation can be assigned per project. Among other things, importing among will make use of this.
Using this table, you can subscribe to email alerts on a per-project basis. There is a choice for receiving outage alerts or not. Email will be sent by blind carbon copy (bcc) for privacy.
The measurement interval, used inter alia for outage alerting, is defined within Vision 42 as the time between the last two measurements. This is the most logical choice and allows dynamic measurement frequencies:
- If the interval increases, for example from 1 minute to 1 hour, you get exactly one outage alert. With a sliding average or median, several fault alarms should occur, which would seem illogical.
- If the interval decreases, for example from 1 hour to 1 minute, you will not receive an outage alert in both cases. With a moving average or median, you may miss justified interference alarms shortly after the transition.
Currently, performance of the algorithm also plays a role. After all, the interval is recalculated with every value that enters the database, and that can be a huge number. In the future, we may approach this differently.
Deleting old values
Old values of a project, be it measured or calculated, can easily be deleted in bulk up to a given date. This operation may take a while, dependent on the size of your database. Closing or reloading the app window will break off the delete operation. Values marked invalid will be left behind, to aid future reimporting.
Limited by RAM memory
While generating large reports, a browser can become sluggish for a longer time. After a while, some browsers will offer the choice to close the app or to keep waiting on it. This is a indication that the RAM memory has been depleted and the report is better split into smaller parts.
In this table, you can define properties per instrument type. Leave the instrument type blank for generic properties, ie. all instrument types at once. The report flag toggles inclusion on reports. The information field may be used as a manual for the property. It will show up when users fill in the property values - the specifications of an instrument.
The input field is meant for advanced users. A little explanation about the syntax is in order:
- If you do not specify anything, Vision 42 will automatically choose between a text or a numeric field, depending on whether the unit field is filled in.
- A list of choices can be defined, by separating the options with a
|sign (vertical bar). Example:
full grouted | push in | sand pocket
- Some standard HTML input types and unquoted attributes may be specified: Example:
number min=0 max=10 step=2
Coordinates can be transformed in a general and fully automatic way. The desired coordinate systems can be defined here.
In terms of coordinate systems, the EPSG data set is the worldwide reference. On many sites, including epsg.io, you can find PROJ.4 or WKT definitions of coordinate systems. They can simply be pasted into the Vision 42 app. The EPSG field is optional and can be used as a reference to the source.
We chose not to include the full EPSG data set in the Vision 42 app, for the sake of clarity and to avoid confusion about similar and / or alternative definitions of the same coordinate system.
Tip: Sensor formulas can be combined with the automatic coordinate transformation and the update of instrument location.
This table contains errors, if any, encountered by your background jobs like continuous import or export.
Vision 42 can export project data continuously to multiple FTP, FTPS, SFTP and SCP locations. The standard URL syntax must be used to configure export locations. We advice making use of the separate password field, to avoid a readable password in the URL. The locations must be directories, in which files will be exported in an orderly fashion. Possible errors can be found in the Errors table.
The exported files contain times, instrument (or location) names, sensor (or quantity) names, units, and values. Note that this structure is identical to the Senceive CSV format. Additionally, the Vision 42 files adhere to several standards:
With usability in mind, exported files will not exceed 200.000 lines. This method of exporting is one-way only. It can therefore not be used to keep data in the target system fully synchronized.
Caution: Before exporting to another Vision 42 instance, conflicting instrument ids should be resolved.
FTP and FTPS servers must support passive mode. If not, connections will fail.
This section lets you manage the organisation structure and the users of the app. It is geared towards administrators, while regular users can view and browse the information.
It is divided into three parts:
- A representation of the organisation structure.
- A Users section.
- A Departments section.
In the structure graph you can select a department.
- The users linked to this department will be shown in the Users section.
- The administrators of a department are shown when hovering over the department in the graph.
- If you are an administrator in this department, you can link existing users to it.
Creating new users
New users are first created without a department:
- Go to the empty department, depicted by a empty circle.
- Add the user info at the bottom:
- Username, preferably the email address, so that a password reset is easier and more private.
- Administrator flag.
- Create the user by clicking the plus icon.
- Next select the department the user belongs to and link the user to it.
Password management is done by clicking the reset password link next to the name of the user.
If the identification of the user is an email address, the new login credentials will be emailed immediately. Next, the user can set a different pasword on his profile page.
This third section handles the organisation structure. New departments can be created by providing:
- A name.
- The optional parent organisational entity.
Add the new entity by clicking the plus icon.
Headers and footers
These are the standard headers and footers of Chrome, over which Vision 42 has no control. Simply turning these off once in the print dialog is the solution here.
Avoid using any print option in the browser's menus, since those will not resize maps and plots. Instead use the menu or buttons within the Vision 42 app. The Ctrl + P or ⌘P key combinations should also work as expected.
Create a ticket
This link allows you to create a ticket, directly in our software management system. Most fields will be filled in automatically. Please enter a one-line summary and a detailed description of the problem.
Tickets can cover:
- Code defects
- Feature requests
- Service requests
Once submitted, you will receive an automatic email whenever your ticket changes. In those emails, you will find links to edit or append the ticket and to add an attachment.
External applications can be linked with your Vision 42 database through the REST application programming interface (API). This powerful interface unlocks all relevant information over a secure channel. Open standards like GeoJSON, CSV, ISO 8601, and JSON are being used exclusively.
Data in Vision 42 can be requested programmatically via the REST interface. To this end, one or more system accounts can be created for specific customer demands, after which access to the interface is obtained with HTTP basic authentication, but only over TLS for security reasons. System accounts are meant to be used by trusted IT-systems exclusively. They cannot be restricted to selected projects.
The structure of a REST URI is https://vision42.net/customer/endpoint. Here endpoint can assume the values projects, instruments, alerts or sensors. The interface provides access to projects, instruments, data files, alerts, sensors and measurement values. GeoJSON is supplied unless otherwise indicated. For projects, a Polygon is used as bounding box, all others use Point.
The same REST interface is always available in the QA environment (https://vision42.net/qa/customer/) with the same credentials.
The following examples of partial URIs speak mostly for themselves:
- The response will also contain all available information about the projection used for coordinate transformation.
- The response will list all data files in CSV format.
- The response will list the last n data files in CSV format.
- The response will be the requested data file itself.
- The optional sensor offset(s) may be used to calculate the absolute sensor values.
Sensor values will be supplied as a JSON-array and a from and/or a to time can be specified, in ISO 8601 notation. Returned sensor values will be relative to the instrument datum, if any. Some examples:
To easily plot profile data with respect to a non-time axis (eg. a inclinometer), these URIs can be used:
- This call provides the discrete times for all measurements per instrument. The result is meant to be used to navigate through time.
- These calls provide the points (position, value) for the A and B series. Replace A or B by the desired series name. The values are again relative to the instrument datum, if any.