JAABA GUI Overview

On this page, we describe each component of the JAABA GUI.

Contents

• Main JAABA Interface
• Menu Items

Main JAABA Interface

Annotated Screen Capture of Main GUI

1. Menu Bar
2. Zoom Buttons: When toggled on, these buttons can be used to zoom in or out on the Movie Previewer view, the number of frames shown in all the timelines, and the y-axis limits for the Per-Frame Feature Timelines. Be sure to toggle these buttons off when you are done changing the axes limits to restore normal function.
3. Pan Button
When toggled on, this button can be used to pan the Movie Previewer view, or change the y-axis limits in the Per-Frame Feature Timelines.
4. Save Labels Button: Save the labels for the current project.
5. Movie Previewer: Window showing the current frame of the current video, annotated with the animals' tracked positions. This is the main interface for allowing the user to see the current animal's behavior.
6. Frame Slider: The current frame can be changed by moving the frame slider.
7. Manual Label Timeline: This timeline shows the labels the user has added for a window of frames centered on the current frame. The x-axis corresponds to time, and the color indicates the label. The size of this window can be set using the Zoom Buttons or through the View->Timeline View Options menu item.
8. Current Score: The raw score output by the current classifier for the current frame. This value will be positive if the behavior is predicted and negative if the None class is predicted. The magnitude of this number corresponds to the classifier's confidence. This number corresponds with the color shown in the Automatic Prediction Timeline's Raw Scores row.
9. Per-Frame Feature Selector: This drop-down menu can be used to select the per-frame feature shown in the timeline to the timeline to the right. If you select Remove from the list, this timeline will be removed. Help currently not implemented.
10. Current Per-Frame Feature Value: The value of the per-frame feature for the current frame. This number corresponds with the y-coordinate of the per-frame feature timeline for the current frame.
11. Add Per-Frame Feature Button: Push this button to add a new per-frame feature timeline to the display.
12. Status Bar: The Status Bar shows important information about the current state of JAABA. If the text is green, JAABA is not busy and you can click buttons, etc. If the text is magenta, then JAABA is busy computing something, and you should wait until it turns green again before using the GUI again. When it is not busy, the Status Bar shows information about the current video and the current animal.
13. Current Animal's Trajectory: The trajectory of the currently selected animal is plotted for a window of frames before and after the current frame. Centroid positions are shown as connected dots. The number of frames before and after the current frame plotted can be set in the View->Preview Options menu.
14. Current Animal: The currently selected animal's tracked position in the current frame is plotted. The currently selected animal's position is plotted with a thicker outline, the color of which is set by either the manual or automatic labels for the current frame, depending on the current configuration. For flies and mice, a triangle is plotted to indicate the size of the ellipse fit to the animal and its head direction. If the fly's wings are tracked, lines from the fly centroid to the tips of each wing are also plotted. For larvae, an 11-point line from the animal's head to its tail through its center is plotted. The head is indicated with a large circle. For flies and mice, the sex of the animal is indicated by the presence (male) or absence (female or unknown) of a dot at the animal's center.
15. Other Animals: All other tracked animals' positions are plotted. They can be distinguished from the selected animal by the thickness of the outline (their outlines are thinner). The color of the outline is unique per animal.
16. Add [Behavior] Label Button: This button is pushed to start or end labeling a bout of the behavior. It is equivalent to press the corresponding keyboard shortcut.
17. Add None Label Button: This button is pushed to start or end labeling a bout of the None behavior class. It is equivalent to press the corresponding keyboard shortcut.
18. Add Unknown Label Button: The labels for all frames is Unknown by default. This button can be used to set the labels for frames to Unknown, effectively erasing other labels given to the frame. Pressing this button starts or ends labeling a bout of Unknown label. It is equivalent to press the corresponding keyboard shortcut.
19. Toggle Select Mode Button: Pressing this button toggles whether Select Mode is enabled or not. If it is enabled, intervals of the timeline can be selected by clicking and dragging. These intervals can then be played or bookmarked.
20. Play Selection Button: If an interval of time is selected, pressing this button will play/stop playing the selected interval in a loop.
21. Clear Selection Button: Set no interval to be selected
22. Current Animal Information: Information about the currently selected animal. The information is:
    • Target: Animal identity number.
    • Sex: The sex of the animal in the current frame (for some trackers, sex is classified per-frame, in case identities are swapped by the tracker) and, in parentheses, the percent of frames over the entire trajectory that the animal is labeled as male and female.
    • Frames: Which frames the animal is tracked for.
    • Experiment Name: The name of the experiment directory for the current video.
23. Labeled Bout of [Behavior]: The red interval on the Manual Timeline shows a bout of frames that were labeled as the behavior.
24. Unlabeled Bout: Black indicates that the frames have not been labeled. Note that it is often difficult to tell exactly when a behavior begins or ends, so there will often be a short unlabeled interval of frames between a bout of the behavior and a bout of None.
25. Labeled Bout of None: The blue interval on the Manual Timeline shows a bout of frames that were labeled as None (the behavior is not occurring).
26. Current Frame Edit Box: Text edit box that shows the current frame number. This number can be modified to change the current frame.
27. Play Button: Pushing this button starts or stops playing the video. The playback speed can be set with the View->Preview Options menu item.
28. Automatic Predictions Timeline: This timeline shows the classifier's behavior label predictions for a window of frames centered on the current frame. The x-axis corresponds to time, and the color indicates the predicted behavior class. The size of this window can be set using the Zoom Buttons or through the View->Timeline View Options menu item. Within this timeline, there are three rows:
    a. Binary Predictions: The classifier's predictions of whether the behavior is or is not being performed in each frame. Usually, if the current classifier has predicted the current window of frames, then this timeline will be saturated red (behavior) or blue (not-behavior). If this timeline is black, then predictions may not have been generated for these frames. Pressing the Predict Button will apply the current classifier to a window of frames around the current frame and fill in this timeline. (Note that leaving a gap between the behavior and none thresholds can also result in frames that are predicted as unknown and are also colored black, but in general such thresholds are not used.) This timeline is generated by thresholding the Raw Scores timeline below, usually at 0.
    b. Raw Scores: The raw output from the current classifier. Red indicates that the behavior is predicted, and blue that None is predicted. The saturation of the color indicates the classifier's confidence. If this timeline is all black, then predictions may not have been generated for these frames. Pressing the Predict Button will apply the current classifier to a window of frames around the current frame and fill in this timeline.
    c. Configurable Scores and Predictions: This timeline is again broken into two sub-timelines. The first sub-timeline shows raw scores and the second sub-timeline shows the binary, thresholded version of these predictions. The following choices are available:
    • None: Nothing is shown.
    • Validated: The out-of-training-set scores and predictions produced using cross calidation.
    • Old: The scores and predictions from the previous classifier.
    • Loaded: The scores and predictions loaded from external files.
    • Postprocessed: The current scores and the predictions that result from Post Processing.
29. Train Button: Pressing this button trains a new classifier using the current labels.
30. Predict Button: Pressing this button computes the classifier's raw scores and predictions for a window of frames around the current frame.
31. Per-Frame Feature Timeline: This timeline shows the value of the selected per-frame feature over a window of frames around the current frame. The x-axis corresponds to time, the y-axis to the per-frame feature value. The y-axis limits can be set using the Zoom and Pan buttons.
32. Timeline Current Frame: This line indicates the current frame in the timelines.

Menu Items

• File
  • ->New: Create a new JAABA project. This is only enabled when no JAABA project is open. See Creating a New Project for more information.
  • ->Open: Open an existing JAABA project in training mode (in training mode, you can add labels and train new classifiers). This is only enabled when no JAABA project is open. See JAABA Projects for more information.
  • ->Open in Ground-Truthing Mode: Open an existing JAABA project in ground-truthing mode (in ground-truthing mode, you can quantitatively evaluate an existing classifier's performance by labeling ground-truth). This is only enabled when no JAABA project is open. See Ground-Truthing for more information.
  • ->Close: Close the current JAABA project.
  • ->Save: Save the current JAABA project. See JAABA Projects for more information.
  • ->Save As...: Save the current JAABA project to a new file. See JAABA Projects for more information.
  • ->Cache WindowData in the jab File: If this is checked, the window data for all labeled frames will be cached in the jab file. This can take a significant amount of disk space, but can save time when closing and opening the same project. See JAABA Projects for more information.
  • ->Change Behavior Name...: Change the name of the behavior being labeled. See Project Parameters for more information.
  • ->Change Score File Name...: Change the name of the score file associated with this project. See Project Parameters for more information.
  • ->Change Experiment List...: Modify the experiments (videos) within the current project. See Adding Experiments to a JAABA Project for details.
  • ->Remove Experiments With No Labels: Remove all experiments (videos) for which there are currently no labels from the project. This can be used for cleaning up a project to the bare minimum number of experiments necessary for training.
  • ->Import Classifier...: Import the classifier from another JAABA project into this one. This will allow you to try this classifier on experiments in the current project. If you train after importing, the imported classifier will no longer be used. See Project Parameters for more information.
  • ->Import Scores: Import the output of a classifier (termed scores) for experiments in the current project. Note that imported scores will show up in the third row of the automatic timeline if "Imported" is selected. See Loading Predictions for more information.
    • ->For current experiment from default location: Import scores stored in the scores file associated with the current JAABA project for the currently open experiment (video).
    • ->For current experiment from different location: Import scores stored in a selected MATLAB mat file for the currently open experiment (video).
    • ->For all experiments from default location: Import scores stored in the scores file associated with the current JAABA project for all experiments in the project.
  • ->Export Scores: Export the output of a classifier (termed scores) for experiments in the current project. See Saving Predictions for more information.
    • ->For current experiment to default location: Export the current classifier's output (scores) to the scores file associated with the current JAABA project for the currently open experiment (video). This scores file will be within the experiment directory for the current experiment.
    • ->For current experiment from different location: Export scores to a selected MATLAB mat file for the currently open experiment (video).
    • ->For all experiments from default location: Export scores to the scores file associated with the current JAABA project for all experiments in the project. These scores files will be within the respective experiment directories for each experiment in the project.
    • ->For all experiments to different location: Export scores for all experiments in the project to a MATLAB mat file within each experiment directory specified.
  • ->Quit: Quit JAABA.
• Edit
  • ->Undo: Not implemented yet.
  • ->Shortcuts for labeling...: Assign keyboard shortcuts to the label buttons. See Label Keyboard Shortcuts for more information.
  • ->Compression Preferences...: Change the compression preferences for bookmarking example video clips.
  • ->Normal mode vs ->Ground-truthing mode: Whether JAABA is opened in the normal mode, in which training data can be added and new classifiers can be trained, or in ground-truthing mode, in which the performance of an existing classifier can be quantitatively evaluated.
  • ->Basic mode vs ->Advanced mode: Whether labeling is being done in basic mode, in which there are three possible labels (the behavior, unknown, or not the behavior), or advanced mode, in which there are five possible labels (important example of the behavior, unimportant example of the behavior, unknown, unimportant example of not the behavior, important example of not the behavior). We use basic mode during training and advanced mode during ground-truthing.
  • ->Memory usage: Modify how much data JAABA uses for caching data. The default amount is 4GB.
• View
  • ->Keep target in view: The axes of the preview window will be moved as little as possible to keep the current animal within them. See Navigating in Space for more information.
  • ->Center on target: The currently selected animal will stay at the center of the video preview window. See Navigating in Space for more information.
  • ->Static view: The axes of the video preview window are not updated based on the current animal's position. See Navigating in Space for more information.
  • ->Show whole frame: Zoom out to show the entire video, and set view mode to Static view.
  • ->Plot tracks: Whether to overlay the animals' trajectories on the preview window.
  • ->Manual Labels vs Automatic Labels: Whether to color the current fly's plotted position and trajectory by the manually entered labels or the automatic predictions.
  • ->Timeline View Options...: Modify the number of frames shown in the timelines.
  • ->Preview Options...: Modify parameters related to the preview video window:
    • The playback speed of the video
    • The number of threads used to fill the frame cache. Matlab permits a maximum of 12 threads total, which JAABA must proportion between window feature computation and pre-loading of video frames. The number of window feature threads allocated is the smaller of the number of CPU cores and 12 less the specified number of frame cache threads.
    • The number of trajectory frames plotted into the past and into the future.
  • ->Show Bookmarked Clips: Show video clips for the current experiment that have been bookmarked.
  • ->Show Predictions/Hide Predictions: Show or hide the automatic predictions. We show the predictions in training mode and hide them during groundtruthin mode.
• Go: Navigation in time and between different animals and videos. See Navigating within and between Videos for more information.
  • ->Switch Target...: Switch to labeling a different animal in the same video or a different animal in a different video. See Switching to Different Animals and Videos for more information.
  • ->Navigation Preferences...: Modify how we can move around in the current video using the arrow keys. See Navigation Preferences for details.
  • ->Next frame (right arrow): Go forward one frame (can also be accomplished by pushing the right arrow key).
  • ->Previous frame (left arrow): Go backward one frame (can also be accomplished by pushing the left arrow key).
  • ->Forward X frames (down arrow): Go forward X frames (where X is set in Navigation Preferences (can also be accomplished by pushing the down arrow key).
  • ->Backward X frames (up arrow): Go backward X frames (where X is set in Navigation Preferences (can also be accomplished by pushing the up arrow key).
  • ->Next manual bout start (Ctrl+right arrow): Go forward to the start of the next labeled bout, if there is one (can also be accomplished by holding down control and pushing the right arrow key).
  • ->Previous manual bout end (Ctrl+left arrow): Go backward to the end of the previous labeled bout, if there is one (can also be accomplished by holding down control and pushing the left arrow key).
• Classifier
  • ->Select Features...: Select which per-frame and window features can be used by the classifier. See Select Features for more information.
  • ->Change Score Features...: JAABA can use the output of one classifier as the input to another. For example, we use the output of our "Jump" classifier as an input to our "Righting" classifier. This menu allows you to select which JAABA projects to use as inputs to this one.
  • ->Training Parameters...: Set the machine learning parameters. See Classifier Parameters (Advanced) for more information.
  • ->Predict: Run the current classifier either on the currently selected animal (Current fly), all animals in the currently selected experiment (Current experiment), or all animals in all experiments (All experiments). See Classifier's Predictions for more information.
• Help
  • ->About: Show JAABA version.
  • ->Documentation: Show this documentation.