User guide: ViSiAnnoT¶

ViSiAnnoT has been designed to be as easy to use as possible, while being highly configurable so that it could meet a variety of needs. In this section, we introduce its features and we illustrate them with examples in the context of clinical research on preterm newborns.

The dataset used for these examples is stored in a folder named “data” and is organized as follows:

Folder “interval”: files with interval data to plot over signals * Pat01_2021-03-02T09-33-56_intervalA.txt * Pat01_2021-03-02T09-33-56_intervalB.txt
Folder “signal_sync”: signal files that are synchronized with first video * Pat01_2021-03-02T09-33-56_physio.h5 (contains ECG and respiration) * Pat01_2021-03-02T10-03-56_physio.h5 (contains ECG and respiration) * Pat01_2021-03-02T09-33-56_rr.txt * Pat01_2021-03-02T09-33-56_tqrs.txt
Folder “video”: video files for two cameras * Pat01_2021-03-02T09-33-56_BW1.mp4 * Pat01_2021-03-02T09-33-56_BW2.mp4 * Pat01_2021-03-02T10-03-56_BW1.mp4 * Pat01_2021-03-02T10-03-56_BW2.mp4 * Pat01_2021-03-02T10-33-56_BW1.mp4 * Pat01_2021-03-02T10-33-56_BW2.mp4
Pat01_audio.wav
physio_2021-03-02T09-34-09.h5 (contains ECG, respiration, TQRS and RR)

This dataset is not publicly available, but equivalent scripts are provided in this repository, along with an example dataset.

Video visualization with multiple cameras¶

ViSiAnnoT can be used as a simple video player. Supported formats are the ones that are supported by OpenCV (for instance .mp4 and .avi). It is possible to play as many videos as wanted simultaneously, provided that they are synchronized. This is useful when the user has video files of several synchronized cameras. The name of the video files must contain the beginning datetime of the video, so that the software can check if the videos are indeed synchronized (a shift of one second is authorized between the video files).

Here is an example:

from visiannot.visiannot import ViSiAnnoT

# video paths
path_video_1 = "data/video/Pat01_2021-03-02T09-33-56_BW1.mp4"
path_video_2 = "data/video/Pat01_2021-03-02T09-33-56_BW2.mp4"

# video configuration
video_dict = {}
video_dict["BW1"] = [path_video_1, '_', 1, "%Y-%m-%dT%H-%M-%S"]
video_dict["BW2"] = [path_video_2, '_', 1, "%Y-%m-%dT%H-%M-%S"]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoT(video_dict, {})

From the sub-package visiannot we import the class ViSiAnnoT. We have a set of 2 synchronized videos, located in the folder data. The configuration dictionary video_dict specifies to ViSiAnnoT where to find the videos and how to get the beginning datetime in the video file name. Each item corresponds to one camera, the key is the camera ID and the value is a list/tuple of 4 elements:

Path to the video file,
Delimiter to get the beginning datetime in the video file name,
Position of the beginning datetime in the video file name, according to the delimiter,
Format of the beginning datetime in the video file name (see keyword argument fmt of convert_datetime_to_string()).

The instance of ViSiAnnoT must be stored in a variable in order to have the window displayed.

Fig. 2 shows a screenshot of the resulting window. At the bottom, there is the progress bar which gives the current temporal position with the red dot and allows to navigate in the video. We call the red dot “navigation point”. The temporal axis is expressed in absolute datetime. It is possible to navigate in the video by clicking in the progress bar and dragging the navigation point. The user can start/stop the video playback with the space key.

Fig. 2 Screenshot of ViSiAnnoT used as a video player¶

If audio is contained in the video file, there is currently no audio playback.

Signal visualization¶

ViSiAnnoT can be used as a simple signal viewer. The user can define as many widgets as wanted (i.e. figures) and plot as many signals as wanted in a widget. As for video visualization, the signal files should be synchronized.

Signal format¶

Supported formats are .txt, .mat, .h5 and .wav. There are two ways to store the signal:

As a vector of length $n_{sample}$ , where $n_{sample} \in \mathbb{N}$ is the number of samples in the file. In this case, the frequency is constant and must be provided by the user.
As a matrix of shape $(n_{sample},2)$ , where the first column contains the timestamp of each sample and the second column contains the value of the samples. This is particularly useful for non regularly sampled signals. The timestamps are expressed in milliseconds relatively to the beginning datetime of the file.
As a matrix of shape $(n_{sample},m+1)$ , where the first column contains the timestamp of each sample and the remaining columns contain the value of the samples of $m$ signals. This is particularly useful for several non regularly sampled signals which share the same timestamps for samples.

An example of non regularly sampled signal is the RR series, which is extracted from the physiological signal ECG (electrocardiogram). The ECG measures the electrical activity of the heart beat. During a heart beat cycle, there is a peak that can be detected. The RR series is defined as the difference between two successive peaks in the ECG. Since these peaks are not regular, the RR series is non regularly sampled.

NB: it is strongly advised to use the .h5 format instead of .txt in order to have better speed performance.

Multiple signal plots in the same widget¶

ViSiAnnoT allows to plot as many signals as wanted in the same widget. Since plotting relies on Pyqtgraph, all the configurations available in this package can be used to customize plot style (see line style and point style keyword arguments of PlotDataItem constructor).

A default plot style can be used for up to 10 signals plotted in the same widget (no symbol for points, points connected by a line). Only the color of the connecting line changes from one signal to another. Above 10 signals, it is required to manually specify the plot style.

In case several signals are plotted in the same widget, the fact that their frequencies may be different is automatically managed.

Here is an example:

from visiannot.visiannot import ViSiAnnoT

# signal paths
path_physio = "data/signal_sync/Pat01_2021-03-02T09-33-56_physio.h5"
path_tqrs = "data/signal_sync/Pat01_2021-03-02T09-33-56_tqrs.txt"

# define plot style
plot_style_tqrs = {
    'pen': None,
    'symbol': '+',
    'symbolPen': 'r',
    'symbolSize': 10
}

plot_style_resp = {'pen': {'color': 'm', 'width': 1}}

# signal configuration
signal_dict = {}

signal_dict["ECG"] = [
    [path_physio, '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None],
    [path_tqrs, '_', 1, "%Y-%m-%dT%H-%M-%S", "tqrs", 0, plot_style_tqrs]
]

signal_dict["Respiration"] = [
    [path_physio, '_', 1, "%Y-%m-%dT%H-%M-%S", "resp", "resp/freq", plot_style_resp]
]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoT(
    {}, signal_dict, flag_pause_status=True, layout_mode=2
)

From the sub-package visiannot we import the class ViSiAnnoT. We have a set of 3 synchronized signals (ECG, respiration and QRS beat detection), located in the folder data. The configuration dictionary signal_dict specifies to ViSiAnnoT where to find the signal files, what is the frequency of the signals, how to get the beginning datetime of the signal file and how to plot. Each item corresponds to one signal widget. The key is the widget ID, which is used as Y axis label. The value is a nested configuration list where each element corresponds to one signal to plot and is a list of 7 elements:

Path to the signal file,
Delimiter to get the beginning datetime in the signal file name,
Position of the beginning datetime in the signal file name, according to the delimiter,
Format of the beginning datetime in the signal file name (see keyword argument fmt of convert_datetime_to_string()),
Key to access the data in the file (in case of .h5 or .mat, set it to '' otherwise), also used a legend - in case of 2D data with several value columns, then the column index must be specified, e.g. "key - 1" or "key - colName" if there is an attribute at key named columns with columns name being comma-separated (first column is always the timestamps),
Signal frequency (may also be a string with path to the frequency attribute in case of h5 file), set it to 0 in case of non-regularly sampled signal,
Dictionary with plot style, set to None for default plot style.

The keyword argument pause_status is set to True so that the video playback is disabled at launch. The instance of ViSiAnnoT must be stored in a variable in order to have the window displayed.

Fig. 3 shows a screenshot of the resulting window. On the first widget, there are two signals: ECG (sampled at 500 Hz) and QRS beat detection (non regularly sampled). On the second widget, there is one signal: respiration. The default plot style is used for the ECG (blue curve), whereas a custom plot style is defined for QRS beat detection (red dots) and respiration (purple curve). We call “temporal cursor” the red vertical line on the signal plots giving the current temporal position. It is linked to the red dot in the progress bar, which is above the signal widgets.

Fig. 3 Screenshot of ViSiAnnoT used as a signal viewer¶

Audio signal visualization¶

Regarding the visualization of an audio signal, the configuration is slightly different since the user must provide the channel to display (left or right). Here is an example:

from visiannot.visiannot import ViSiAnnoT

# audio path
path_audio = "data/Pat01_audio.wav"

# signal configuration
signal_dict = {}
signal_dict["Audio L"] = [[path_audio, '', None, '', "Left channel", 0, None]]
signal_dict["Audio R"] = [[path_audio, '', None, '', "Right channel", 0, None]]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoT(
    {}, signal_dict, flag_pause_status=True, layout_mode=2
)

We define two signal widgets: “Audio L” and “Audio R”. They both take the same audio file as input.

In order to specify the channel to display in each plot, we use the key to access data and set it to “Left channel” and “Right channel”. The key word for channel selection is “left” or “right”, regardless of the letter capitalization and the position in the string. If no channel is specified, then the left channel is displayed by default.

The signal frequency is automatically retrieved from the wav file, so in the configuration list it can be set to anything (in this example 0).

The beginning datetime is not contained in the audio file name, so one of the three related variables is set to None and a default beginning datetime is defined (2000/01/01 00:00:00).

Fig. 4 shows a screenshot of the resulting window.

Fig. 4 Screenshot of ViSiAnnoT used as an audio signal viewer¶

Zoom tools¶

The default zoom of Pyqtgraph is available for the Y axis of the signal plots and is overwritten for the X axis so that all the signal widgets are linked. Thus the zoom tools described here only affects the temporal axis.

Based on Fig. 3, Fig. 5 illustrates the temporal zoom. We call “temporal range” the period of the signals that is displayed and “temporal range duration” its duration. In the progress bar, the black lines delimit the temporal range. We can see that the temporal range duration in Fig. 3 is 30min00s and becomes 00min36s after zoom in Fig. 5. The black lines of the progress bar have also moved to show what part of the signals is displayed.

Fig. 5 Screenshot of ViSiAnnoT used as a signal viewer after zoom¶

The user can zoom in/out around the temporal cursor by using the two buttons looking like magnifying glass. It is also possible to directly zoom out in order to visualize the full signals by using the button looking like an eye. The buttons can be seen in the top left corner of the window.

YRange¶

The range of values on the Y axis of a specific signal widget may be fixed by the user.

This is done with the dictionary y_range_dict which is passed to ViSiAnnoT as a keyword argument. The key of the dictionary must correspond to a key of signal_dict, it specifies the signal widget where the Y range is fixed. The value of the dictionary is a tuple of length 2 with the minimum and maximum value on the Y axis.

Threshold values¶

Threshold values can be drawn as horizontal lines on a signal plot. It may be useful to identify temporal intervals where a signal is above or below a specific value.

This is done with the dictionary threshold_dict which is passed to ViSiAnnoT as a keyword argument. The key of the dictionary must correspond to a key of signal_dict, it specifies the signal widget where to draw the threshold. The value of the dictionary is a nested list of thresholds, each element is a list of length 2: threshold value and threshold color (RGB) or (RGBA).

Fig. 6 shows an example of a signal widget with thresholds.

Fig. 6 Detail of a screenshot of ViSiAnnoT used as a signal viewer with two thresholds¶

Temporal intervals¶

It is also possible to display temporal intervals on the signal widgets. This may be useful if the user has pre-annotations or results from a detection algorithm and wants to visually check their accuracy.

This is done with the dictionary interval_dict which is passed to ViSiAnnoT as a keyword argument. The key of the dictionary must correspond to a key of signal_dict, it specifies the signal widget where to display temporal intervals. The value of the dictionary is a nested list of configurations for each kind of interval to display on the same widget. The configuration is a list of length 7:

Path to the interval file,
Delimiter to get the beginning datetime in the interval file name,
Position of the beginning datetime in the interval file name, according to the delimiter,
Format of the beginning datetime in the interval file name (see keyword argument fmt of convert_datetime_to_string()),
Key to access the data in the file (in case of .h5 or .mat, set it to '' otherwise),
Interval frequency (may also be a string with path to the frequency attribute in case of h5 file),
RGBA color.

The intervals may be stored in two ways in the files:

As a vector of length $n_{sample}$ with 0 and 1, where $n_{sample} \in \mathbb{N}$ is the number of samples in the file,
As a matrix of shape $(n_{inter},2)$ , where $n_{inter} \in \mathbb{N}$ is the number of intervals in the file, each line is an interval with the starting sample and the ending sample.

Here is an example:

from visiannot.visiannot import ViSiAnnoT

# signal paths
path_physio = "data/signal_sync/Pat01_2021-03-02T09-33-56_physio.h5"
path_tqrs = "data/signal_sync/Pat01_2021-03-02T09-33-56_tqrs.txt"
path_interval_a = "data/interval/Pat01_2021-03-02T09-33-56_intervalA.txt"
path_interval_b = "data/interval/Pat01_2021-03-02T09-33-56_intervalB.txt"

# define plot style
plot_style_tqrs = {
    'pen': None,
    'symbol': '+',
    'symbolPen': 'r',
    'symbolSize': 10
}

# signal configuration
signal_dict = {}
signal_dict["ECG"] = [
    [path_physio, '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None],
    [path_tqrs, '_', 1, "%Y-%m-%dT%H-%M-%S", "tqrs", 0, plot_style_tqrs]
]

# interval configuration
interval_dict = {}
interval_dict["ECG"] = [
    [path_interval_a, '_', 1, "%Y-%m-%dT%H-%M-%S", '', 500, (0, 255, 0, 50)],
    [path_interval_b, '_', 1, "%Y-%m-%dT%H-%M-%S", '', 500, (255, 200, 0, 50)]
]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoT(
    {}, signal_dict, flag_pause_status=True, layout_mode=2,
    interval_dict=interval_dict
)

In this example, two kinds of intervals are defined on the "ECG" widget. A specific color is assigned to each kind of temporal intervals. Fig. 7 shows this particular plot.

Fig. 7 Detail of a screenshot of ViSiAnnoT used as a signal viewer with additional temporal intervals¶

Combined video and signal visualization¶

ViSiAnnoT allows to combine video and signal visualization. The videos and the signals must be synchronized. If they do not share the same frequency, it is automatically taken into account.

Here is an example:

from visiannot.visiannot import ViSiAnnoT

# video paths
path_video_1 = "data/video/Pat01_2021-03-02T09-33-56_BW1.mp4"
path_video_2 = "data/video/Pat01_2021-03-02T09-33-56_BW2.mp4"

# video configuration
video_dict = {}
video_dict["BW1"] = [path_video_1, '_', 1, "%Y-%m-%dT%H-%M-%S"]
video_dict["BW2"] = [path_video_2, '_', 1, "%Y-%m-%dT%H-%M-%S"]

# signal paths
path_physio = "data/signal_sync/Pat01_2021-03-02T09-33-56_physio.h5"
path_tqrs = "data/signal_sync/Pat01_2021-03-02T09-33-56_tqrs.txt"

# define plot style
plot_style_tqrs = {
    'pen': None,
    'symbol': '+',
    'symbolPen': 'r',
    'symbolSize': 10
}

# signal configuration
signal_dict = {}

signal_dict["ECG"] = [
    [path_physio, '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None],
    [path_tqrs, '_', 1, "%Y-%m-%dT%H-%M-%S", "tqrs", 0, plot_style_tqrs]
]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoT(video_dict, signal_dict)

Fig. 8 shows the resulting window. The temporal cursor is linked to the current video frame that is displayed. The user can navigate by clicking on a signal plot in order to change the position of the temporal cursor, then the video is displayed at the same position, as well as the navigation point in the progress bar. It is also possible to navigate by dragging the navigation point in the progress bar.

Fig. 8 Screenshot of ViSiAnnoT used as a combined video and signal visualizer¶

Tools for fast navigation¶

First, the user can set a truncation duration in order to split by default the display of the signals in several parts. For example, if the signal files last 30 minutes and the user chooses a 10 minutes truncation duration, then there is a combo box which allows to switch from a 10 minutes part to another (0 to 10 minutes , 10 to 20 minutes, 20 to 30 minutes). This feature is set by the keyword argument trunc_duration in ViSiAnnoT constructor. In the given example: trunc_duration=(10, 0) (tuple with minutes and seconds of the truncation duration).

Second, there is a combo box to select a temporal range duration in order to display a new temporal range that will begin at the current position of the temporal cursor. The list of available temporal range durations must be configured by the user with the keyword argument from_cursor_list in ViSiAnnoT constructor. For example, to have the choice between 30 seconds, 1 minute and 1 minute 30 seconds: from_cursor_list=[(0, 30), (1, 0), (1, 30)].

Third, there is a tool for defining a custom temporal range, as shown in Fig. 9. The user must define the start datetime of the temporal range. The push button “Current” can be used to define it as the current position of the temporal cursor. Then, the user must define the temporal range duration.

Fig. 9 Tool for defining a custom temporal range¶

Management of long recording¶

This section introduces the features for managing long recordings. All features introduced above are still available for long recordings. The class ViSiAnnoTLongRec inherits from ViSiAnnoT and adds specific features to manage long recordings.

A long recording is defined as a set of consecutive video and/or signal files. For example, a long recording lasting for two hours might be composed of four 30-minute length video files and eight 15-minute length signal files.

In this context, there are two additional buttons that allow to switch easily from one file to another and a combo box to directly select a specific file in the recording (with respect to the video files, or the first signal if no video). Fig. 10 shows these buttons and the combo box.

Fig. 10 Buttons and combo box for file selection in a long recording¶

We define the video configuration and the signal configuration almost the same way as for the class ViSiAnnoT, but instead of specifying the path to a file, we specify the directory containing the files and a pattern to find them.

Regarding video_dict, each item corresponds to one camera. The key is the camera ID and the value is a list of 5 elements:

Directory where to find the video files,
Pattern to find the video files,
Delimiter to get the beginning datetime in the video file name,
Position of the beginning datetime in the video file name, according to the delimiter,
Format of the beginning datetime in the video file name (see keyword argument fmt of convert_datetime_to_string()).

Regarding signal_dict, each item corresponds to one signal widget. The key is the widget ID. The value is a nested configuration list where each element corresponds to one signal to plot and is a list of 8 elements:

Directory where to find the signal files,
Pattern to find the signal files,
Delimiter to get the beginning datetime in the signal file name,
Position of the beginning datetime in the signal file name, according to the delimiter,
Format of the beginning datetime in the signal file name (see keyword argument fmt of convert_datetime_to_string()),
Key to access the data in the file (in case of .h5 or .mat, set it to '' otherwise), also used a legend - in case of 2D data with several value columns, then the column index must be specified, e.g. "key - 1" or "key - colName" if there is an attribute at key named columns with columns name being comma-separated (first column is always the timestamps),
Signal frequency (may also be a string with path to the frequency attribute in case of h5 file), set it to 0 in case of non-regularly sampled signal,
Dictionary with plot style.

Set of synchronized files¶

In this case, the different modalities are synchronized. In the constructor of ViSiAnnoTLongRec, the keyword argument flag_synchro is set to True (default value).

Here is an example:

from visiannot.visiannot import ViSiAnnoTLongRec

# data directory
dir_vid = "data/video"
dir_sig = "data/signal_sync"

# video configuration
video_dict = {}
video_dict["BW1"] = [dir_vid, "*BW1*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]
video_dict["BW2"] = [dir_vid, "*BW2*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]

# signal configuration
signal_dict = {}
signal_dict["ECG"] = [[dir_sig, "*_physio.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None]]
signal_dict["Respiration"] = [[dir_sig, "*_physio.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "resp", "resp/freq", None]]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoTLongRec(video_dict, signal_dict)

At launch, ViSiAnnoT loads and display the files “Pat01_2021-03-02T09-33-56_BW1.mp”, “Pat01_2021-03-02T09-33-56_BW2.mp4” and “Pat01_2021-03-02T09-33-56_physio.h5”. When clicking on the “next file” button, the files “Pat01_2021-03-02T10-03-56_BW1.mp4”, “Pat01_2021-03-02T10-03-56_BW2.mp4” and “Pat01_2021-03-02T10-03-56_physio.h5” are loaded and displayed.

In this mode, “holes” in files of any modality is supported. For example, when switching to the third file (timestamp 10:33:56), there is no signal file named “Pat01_2021-03-02T10-33-56_physio.h5” and consequently no signal is displayed.

Fig. 11 shows a screenshot of the resulting window, with the first file selected.

Fig. 11 Screenshot of ViSiAnnoT for a long recording¶

Set of asynchronous files¶

In the case where the different modalities are not synchronized with each other, ViSiAnnoTLongRec automatically synchronize them before display. The keyword argument flag_synchro of the constructor must be set to False.

We assume that the beginning datetime of each file is contained in its name.

The reference modality for synchronization is the video if there is any, otherwise it is the first signal to plot. Then, when loading a file of the reference modality, ViSiAnnoT loads the parts of the other signals that are covered by the reference file.

NB: if there are several cameras to display, they must be synchronized with each other, the synchronization process is applied only on signals.

Here is an example:

from visiannot.visiannot import ViSiAnnoTLongRec

# data directory
dir_vid = "data/video"
dir_sig = "data"

# video configuration
video_dict = {}
video_dict["BW1"] = [dir_vid, "*BW1*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]
video_dict["BW2"] = [dir_vid, "*BW2*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]

# signal configuration
signal_dict = {}
signal_dict["ECG"] = [[dir_sig, "physio_*.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None]]
signal_dict["Respiration"] = [[dir_sig, "physio_*.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "resp", "resp/freq", None]]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoTLongRec(
    video_dict, signal_dict, flag_pause_status=True, flag_synchro=False
)

Compared to previous example, there is only one signal file covering the 3 video files. So, this signal file is not synchronized with video.

Multi-label annotation tools¶

ViSiAnnoT provides two annotation tools:

Temporal events annotation,
Image extraction.

Events annotation tool¶

This tool allows to annotate temporal intervals. The user can provide as much labels as desired. This tool is useful for establishing the ground truth of a temporal segmentation or classification, as well as studying the occurrence and duration of specific events. It automatically creates a file for each label, where the annotations are written.

When creating an instance of ViSiAnnoT or ViSiAnnoTLongRec, the configuration dictionary of the annotation tool is given to the keyword argument annotevent_dict of the constructor. Here is an example:

annotevent_dict = {}
annotevent_dict["Label-1"] = [200, 105, 0, 50]
annotevent_dict["Label-2"] = [105, 205, 0, 50]

There are two labels (dictionary keys), to which is associated a color (dictionary values). It is worth to note that the label "DURATION" is not permitted because it is used internally by ViSiAnnoT.

Fig. 12 shows a screenshot of the events annotation tool.

Fig. 12 Events annotation tool¶

The radio buttons on the top allow to select the current label. The push buttons “Start” and “Stop” respectively set the beginning and ending datetime of the annotated temporal interval. In this example, the ending datetime is not defined yet. The push button “Add” validates the annotation and appends it in a file. The number of annotations is displayed next to it. The push button “Delete last” deletes the last added annotation. The push button “Display” enables or disables the display of the annotations on the signals plots.

The “Display mode” radio buttons allow to choose what to display:

“Current label”: only the annotations of the current label is displayed (current label is the one selected in the “Current label selection” box),
“All labels”: the annotations of all labels are displayed,
“Custom (below)”: the user can choose the labels to display thanks to the check boxes below.

Fig. 13 shows a screenshot of two signal plots with annotations displayed. They are displayed similarly to the additional temporal intervals. Each color corresponds to one label. As it can be seen on the progress bar, the temporal range is the first 5 minutes. The annotations outside of the temporal range are still displayed on the progress bar.

Fig. 13 Detail of a screenshot of ViSiAnnoT with annotations displayed, each color corresponding to one label¶

It is possible to display the duration of the annotated intervals by clicking with the left button of the mouse while pressing the alt key. The label of the annotated interval must be the current label in order to get the display. An example is given in Fig. 14.

_images/annotation_event_description.png

Fig. 14 Detail of a screenshot of ViSiAnnoT with annotations displayed, two of them with duration displayed¶

By default, it is not possible to overlap two annotations with the same label. In order to enable this feature, the keyword argument flag_annot_overlap of ViSiAnnoT constructor must be set to True.

Storage of events annotation¶

In the constructor of ViSiAnnoT, the keyword argument annot_dir specifies the directory where to store annotation files. By default it is the directory “Annotations”, located at the current working directory from where ViSiAnnoT is launched.

For each label, a text file is created with the intervals of the annotated events. The name of the annotation file is BASENAME_LABEL-datetime, where BASENAME is the basename of the annotation directory and LABEL is the label.

Each line in an annotation file corresponds to an annotated event: TS1 - TS2, where TS1 (resp. TS2) is the start (resp. stop) timestamp of the annotated event. The timestamp is formatted as follows: %Y-%m-%dT%H:%M:%S.%f, where %Y is the year in 4 digits, %m is the month in 2 digits, %d is the day in 2 digits, %H is the hour, %M is the minute, %S is the second and %f is the microsecond.

Image extraction tool¶

This tool allows to extract a still image from the video(s) and associate a label to it.

When creating an instance of ViSiAnnoT or ViSiAnnoTLongRec, the configuration of the annotation tool is given to the keyword argument annotimage_list. Here is an example:

annotimage_list = ["Label-A", "Label-B", "Label-C"]

Fig. 15 shows a screenshot of the image extraction tool. The user selects the label thanks to the radio buttons. Then the push button “Save” allows to extract the current frame for each camera and saves it in a directory named after the selected label.

Fig. 15 Image extraction tool¶

The extracted images are stored in the same directory than events annotation files. For each label, a sub-directory is created, named after the label, where are stored the extracted images. The image file name is "%s_%d.png", where %s is the video file name and %d is the frame index of the image.

Layout modes¶

In the context of combined video and signal visualization, the user may want to put the emphasis on either the video or the signal. For this purpose, we provide three default layout mode, to be selected with the keyword argument layout_mode (may be 1, 2 or 3) The user may also manually configure the layout of the window with the keyword argument poswid_dict.

Here is an example of combined video and signal visualization in the context of long recording with all features enabled (events annotation, image extraction, tools for fast navigation):

from visiannot.visiannot import ViSiAnnoTLongRec

# data directory
dir_vid = "data/video"
dir_sig = "data"

# video configuration
video_dict = {}
video_dict["BW1"] = [dir_vid, "*BW1*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]
video_dict["BW2"] = [dir_vid, "*BW2*.mp4", '_', 1, "%Y-%m-%dT%H-%M-%S"]

# define plot style
plot_style_tqrs = {
    'pen': None,
    'symbol': '+',
    'symbolPen': 'r',
    'symbolSize': 10
}

# signal configuration
signal_dict = {}
signal_dict["ECG"] = [
    [dir_sig, "physio_*.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "ecg", 500, None],
    [dir_sig, "physio_*.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "beat - TQRS", 0, plot_style_tqrs]
]
signal_dict["RR"] = [[dir_sig, "physio_*.h5", '_', 1, "%Y-%m-%dT%H-%M-%S", "beat - RR", 0, None]]

# event annotation dictionary
annotevent_dict = {}
annotevent_dict["Label-1"] = [200, 105, 0, 50]
annotevent_dict["Label-2"] = [105, 205, 0, 50]

# image annotation dictionary
annotimage_list = ["Label-A", "Label-B"]

# create ViSiAnnoT window
win_visiannot = ViSiAnnoTLongRec(
    video_dict, signal_dict,
    flag_pause_status=True,
    flag_synchro=False,
    annotevent_dict=annotevent_dict,
    annotimage_list=annotimage_list,
    trunc_duration=(5, 0),
    from_cursor_list=[(0, 30), (1, 0), (2, 0)],
    layout_mode=1
)

Mode 1 puts the emphasis on the video. If there is not enough space left for the signals, a scroll area is created.

Fig. 16 Layout mode 1¶

Mode 2 puts the emphasis on the signal.

Fig. 17 Layout mode 2¶

Mode 3 provides a more compact display since the following features are disabled: selection of truncated temporal range, selection of temporal range from cursor, and custom selection of temporal range.

Fig. 18 Layout mode 3¶

Keyboard/mouse interactions¶

Here is a synthesis of all the possible user interactions with the keyboard and the mouse.

Keyboard¶

Press

space: start/stop of the video playback
left: 1 second backward
- with control pressed: 1 minute backward
right: 1 second forward
- with control pressed: 1 minute forward
down: 10 seconds backward
- with control pressed: 10 minutes backward
up: 10 seconds backward
- with control pressed: 10 minutes backward
l: 1 sample backward
m: 1 sample forward
i: zoom in
o: zoom out
n: whole zoom out
a: start annotation
z: stop annotation
e: add annotation
s: display annotations
page down: switch to previous file (in long recordings only)
page up: switch to next file (in long recordings only)
home: set the position of the temporal cursor to the first sample of the current file
end: set the position of the temporal cursor to the last sample of the current file
d + control + shift: delete the display of annotation durations

Release

alt: show/hide the menu bar

Mouse click on the signal plots¶

left button: define the new position of the temporal cursor
- with both control and shift pressed: delete the annotation that is clicked on (the label must be the current label)
- with alt pressed: enable or disable to display the duration of the annotation that is clicked on (the label must be the current label)
right button: zoom in (3 clicks: the first two to define the new temporal range, the third click must be inside the new temporal range in order to validate and zoom in, or outside to cancel)
- with control pressed: add events annotation (3 clicks: the first two to define the start/stop of the annotation, the third click must be inside the temporal range in order to add the annotation, or outside to cancel)