JAABA Features

Per-frame features are computed from the input trajectories, and give information about the animal's state in the current frame. To classify behaviors, it also is necessary to provide contextual information about what the animal was doing in the frames before and after the current frame. For example, to detect whether an animal is walking, the speed in just the current frame is informative, but information about the speed in frames around the current frame, such as the mean or standard deviation of the speed in a window of frames around the current frame, can greatly improve performance. On top of the per-frame features, JAABA computes a large, fast, general-purpose set of window features to include this context.

Contents

• Window Feature Types
• Window Feature Transformations
• Window Feature Templates
• Window Feature Computation Files
• Select Features
• Basic Mode
• Advanced Mode
• Select Features Buttons
• Example Files

Window Feature Types

Each window feature is computed from a window of frames for a single per-frame feature. At each frame, for each per-frame feature, we compute around 100-200 window features (depending on the per-frame feature and the JAABA settings). A window feature is defined by which per-frame feature it operates on, which window of frames it operates on (relative to the current frame), and what function it computes from this window of frames. Each window feature is thus defined by the following parameters:

• Type: The type of computation that is performed on the window of data, for example, mean or standard deviation. The full list is given below.
• Transformation: It is often beneficial to perform some simple transformations on the data, either directly on the per-frame feature before computing the window feature, or after. For example, the dtheta per-frame feature is the difference in the animal's orientation between the next and previous frames. The sign of this per-frame feature is usually irrelevant, as it corresponds to whether the animal is turning left or right. Thus, we may want to use the absolute value transformation on this per-frame feature to symmetrize the feature. If we are computing e.g. the mean of the window, there will be less noise if we compute the absolute value after computing the mean. Depending on the type of window feature, the transformations may be computed before or after the window computation. A list of the types of window feature transformations is given below.
• Radius: The radius of the window in frames. The total size of the window is 2*radius+1, thus a radius of 0 corresponds to just a single frame.
• Offset: The where the window is in time, relative to the current frame. An offset of 0 means that the window is centered on the current frame. An offset of -1 means that the window ends on the current frame, that is, it is offset by -radius frames. An offset of 1 means the window starts at the current frame, that is, it is offset by radius frames. In general, the window for frame t is centered at t + offset*radius.
• Other parameters: Depending on the type of window feature, other parameters may be required. These are described in the table below.

The table below lists the type of window computations that are used in JAABA:

Window Feature Transformations

For some per-frame features, we transform the features in some subset of the following ways:

• Absolute value: Depending on the window feature type, we either take the absolute value of the per-frame feature before the window feature computation or the abolute value of the result of the window feature computation. Absolute-transformed features are used for per-frame features like change in orientation where the sign, indicating turning left or turning right, is often not discriminatory.
• Relative value: Normalize the per-frame feature by the distribution of that per-frame feature over the animal's entire trajectory. This is useful, for example, if walking speed is consistent for a given animal or subgroup of animals, but varies greatly between animals or groups of animals we want to apply the classifier to. Consider, for example, locomotion-impaired animals. In such cases, the speed at which the animal is perceived to walk or chase is much slower compared to "normal" animals. Converting feature values into relative values enables the classifier to detect behaviors for such cases, and generalize better to types of animals outside of the training set. We transform a feature value into a relative value by finding the percent of feature values over the whole trajectory that are smaller than the current value. For example, if the feature value of the current frame is the median then its relative transformed value is 50. If 10% of the features values over the animal's trajectory are smaller than the current value then the relative value is 10. In other words, the relative value of a feature value is its percentile value within its trajectory.
• Flipped value: Flip all the per-frame feature values by the sign of the per-frame feature in the current frame. Flip-transformed features are used in similar cases to absolute-transformed features, but allow the sign to be preserved within a window.

For the absolute value and "flip" transformations, we do this by either taking the transformation of the per-frame data first, then computing the window function on the transformed data, or we do this by computing the window function first, then computing the transformation on the output of the window data. Whether we do the transformation first or not depends on the window feature.

We do the transformation first for the min, max, hist, diff_neighbor_min, diff_neighbor_max window feature types. We do the transformation after for the mean, prctile, change, harmonic, diff_neighbor_mean, and zscore_neighbors window feature types. We do not do the transformation for the std feature types.

For the relative transformation, we always do the transformation first.

Window Feature Templates

For each per-frame feature, we need to specify a large number of parameters to define which window features should be computed from it. We found that we almost always set these features in one of a few ways, and thus created templates for the parameters. Thus, for each per-frame feature, we specify a template as to which types of window features to compute.

The table below specifies the templates that we use.

In the above table, wradius is a parameter specified by the user. It is roughly the radius of window size that the user thinks should be sufficient to capture the behavior.

Select Features

The Select Features dialog is a GUI the user can use to edit the window feature computation parameters for the current project. This interface can be accessed from the main JAABA window by selecting the menu item Classifier->Select Features.

Basic Mode

In basic mode, the user can select the classes of per-frame features and the window feature templates to be used for the class. The list of per-frame feature that belong to a particular class depends on the animal (Target-Type) that was selected for the project. Scores is a special class that has all the per-frame features that are the predictions of the previously trained classifiers. The list of per-frame features that belong to this particular class correspond to the list of the previously trained classifiers that were selected during project setup.

Screen capture of the Select Features dialog in Basic mode

Advanced Mode

In advanced mode, the user can set all the window feature parameters for each per-frame feature separately. The advanced mode also shows the histograms of the per-frame features over labeled data for all animals and unlabeled data for the currently selected animal. If the user changes any parameter for a per-frame feature then the window computation template for the corresponding class of per-frame features is set to custom.

Screen capture of the Select Features dialog in Advanced mode

Select Features Buttons

• Advanced > / Basic < : Toggles between advanced and basic modes
• Find hist bins: If user has selected to compute histogram (hist) features for any per-frame feature, then the Find hist bins button will set the bin edges at the 5, 15, 30, 50, 70, 85, and 90th prctiles of the per-frame features over all the flies and all the experiments that are currently loaded in.
• Done: Save the changes and return to JAABA. Also, if applicable, all window features are recomputed for all the labeled frames. When you quit JAABA, it'll ask you if you want to save the changes you made to window features to the project file so that the same window feature settings will be used when you select the same project. The window feature settings are also saved in the classifier file when you save a classifier and they are loaded in when you load the classifier.
• Cancel: Discard the changes made by the user and return back to main JAABA window