Help Overview


RectLabel version 3.02.2

- Improved so that you can start editing items text on Attributes tab by single click, double click, and pressing enter key.

Key features

Draw bounding box, polygon, cubic bezier, and line

Draw keypoints with a skeleton

Label the whole image without drawing boxes

Label pixels with brush and superpixel tools

Automatically label images using Core ML model

1-click buttons make your labeling work faster

Customize the label dialog to combine with attributes

Settings for objects, attributes, hotkeys, and labeling fast

Read and write in PASCAL VOC XML format

Export to YOLO, Create ML, COCO JSON, and CSV format

Export index color mask image and separated mask images

Report issues

Post the problem to our Github issues page.

Have questions?

Send an email to support@rectlabel.com

Thank you.

Requested features

  • iPad app.

Troubleshooting

Find crash logs.

  1. You can find crash logs here. "~/Library/Logs/DiagnosticReports/".
  2. Please send the crash log to support@rectlabel.com

Clear cache files.

  1. Copy the settings file "~/Library/Containers/com.waysify.roi/Data/settings_labels.json".
  2. Remove cache files "~/Library/Containers/com.waysify.roi/".
  3. Import the copied settings file from app menu.

Solve the problem when you buy in-app purchase.

  1. Delete RectLabel via the Launchpad app
  2. Log out of the Mac App Store and iTunes.
  3. Reboot your Mac.
  4. Log back in to the Mac App Store and install RectLabel.

Papers cited RectLabel


Tutorials

We introduce some useful tutorials.


Settings

Projects

You can switch different objects and attributes settings for different labeling tasks.

To add another settings file to "Projects" table, import the settings file.

To use the newly added project, you have to check on the "Primary" check box.

To duplicate the project, right click on the row and "Duplicate" menu would open.

Settings for objects, attributes, hotkeys, and labeling fast

Objects

"Objects" table describes the object index of each object.

When you export, if the label name is not found on the objects table, it would be skipped.

You can assign 0-9 num keys and A-Z alphabet keys to objects.

To duplicate the object, right click on the row and "Duplicate" menu would open.

Right click on the objects table header, "Sort alphabetically", "Set default hotkeys", and "Clear objects table" menu would open.


When you label "sneakers" which uses 3 attributes "color", "brand", and "side".

Settings for objects, attributes, hotkeys, and labeling fast

Attributes

The label "sneakers-yellow-converse-right" is a combination of the object and attributes.

'-' is used as a separator so that '-' in the object name and attribute name would be replaced with '_'.

The prefix is used as '-' + prefix + attribute name.

if any objects are not using attributes, '-' in the label name is not replaced with '_'.

The attribute types are "Single select", "Multiple select", and "Text input".

To change the name on the items table, single click, double click, or press the enter key on the selected item.

For "Single select" type, you can use an empty string item so that the default label name becomes the object name.

To duplicate the attribute, right click on the row and "Duplicate" menu would open.

Settings for objects, attributes, hotkeys, and labeling fast

Hotkeys

Customize the hotkeys to make your labeling work faster.

Settings for objects, attributes, hotkeys, and labeling fast

Label fast

"Auto save" is to skip the confirm dialog to save the annotation.

"Skip label dialog when create" is to skip showing the label dialog when you create.

"Close label dialog when select" is to skip clicking the OK button on the label dialog.

"Use 0-9 and a-z hotkeys" is to prevent from changing label names accidentally.

"Use 1-click buttons" shows 1-click buttons of all objects on the label dialog.

"Maintain zoom and position" is to maintain zoom and position when you change the image.

"Label history max length" is the length of both the label history and the auto suggestions.

Settings for objects, attributes, hotkeys, and labeling fast

Others

"Fix the image position" is not to move the image position.

"Use circle edit points" shows the circle edit points instead of rectangle edit points.

"Use difficult tag" shows the difficult checkbox button on the label dialog.

"Show all edit points" is to show edit points of all objects. This is useful when you "Create point".

"Show edit points between corners on boxes" is to show edit points between corners on boxes and rotated boxes.

"Click 4 points when draw boxes" is used when you draw a box clicking xmin, xmax, ymin, and ymax of the object.

"Sort images" is to sort images by Alphabetic, Numeric, and Last modified.

"Annotator" is saved in the xml file.

"Brush size max" is to change the brush size max.

Settings for objects, attributes, hotkeys, and labeling fast

Export settings file

You can export the current settings file to import to another computer.

Here is an example of the JSON format.

{
    "projects": [
    {
        "fashion":
        {
            "objects": [
            {
                "hidden": false,
                "name": "sneakers",
                "attributes": ["color", "brand", "text"],
                "color":
                {
                    "red": 0,
                    "green": 1,
                    "blue": 1
                }
            },
            {
                "hidden": false,
                "name": "ignore",
                "attributes": [],
                "color":
                {
                    "red": 0,
                    "green": 1,
                    "blue": 1
                }
            }],
            "attributes": [
            {
                "prefix": "",
                "name": "color",
                "type": "Multiple select",
                "items": ["black", "red", "white", "yellow"]
            },
            {
                "prefix": "",
                "name": "brand",
                "type": "Single select",
                "items": ["", "adidas", "converse", "nike", "vans"]
            },
            {
                "prefix": "",
                "name": "text",
                "type": "Text input",
                "items": []
            }],
            "primary": true
        }
    }]
}

Import settings file

When you import the settings file, the content is added to "Projects" table.

To use the newly added project, you have to check on the "Primary" check box.

Export objects table

You can export the current objects table to import to another computer.

Here is an example of text format.

car
plate
person
dog

Import objects table

When you import the objects table, the content is added to "Objects" table.

To clear the objects table before importing, right clicking on the objects table header, "Clear objects table" menu would open.


File menu

Open images folder and annotations folder

Open images folder and annotations folder.

To mount an Amazon S3 bucket as a local filesystem, use goofys.

RectLabel reads and writes in PASCAL VOC XML format.


RectLabel and macOS Viewer show images with Exif orientation flags in the same way.

According to the Exif orientation flag, each image is rotated and shown in the front orientation.

So that the annotation file on RectLabel assumes that the image is rotated and shown in the front orientation.

<size>
    <width>760</width>
    <height>960</height>
</size>

Image file names which include "_pixels" are skipped because the suffix is used in the pixels image file.

Image file names which include "_depth" are skipped because the suffix is used in the depth image file.

To copy the current image file name, click on the image file name shown on the top-left corner.

Open images folder

For the annotations folder, we always use "images_folder/annotations" folder.

Next image and Prev image

To show the next image, press the right arrow key

To show the previous image, press the left arrow key.

Jump to image index

You can specify the image index to show.

Move image to another folder

You can move the image and the label file to the specified folder.

Save

The label file is saved as "{image_file_name}.xml".

Annotations are scaled to the current image size when you save.

Output format

The label file is saved in PASCAL VOC XML format.

The top-left pixel in the image has coordinates (1, 1).

The VOC2007 Development Kit

The rotated box is saved in the format as (center_x, center_y, width, height, rotation).

The rotation is counter-clockwise and ranged between 0 and 2Pi.

<annotation>
    <annotator>ryouchinsa</annotator>
    <folder>test_data_test</folder>
    <filename>aaron-burden-40491-unsplash.jpg</filename>
    <size>
        <width>4417</width>
        <height>3317</height>
        <depth>3</depth>
    </size>
    <object>
        <name>box</name>
        <bndbox>
            <xmin>501</xmin>
            <ymin>211</ymin>
            <xmax>893</xmax>
            <ymax>512</ymax>
        </bndbox>
    </object>
    <object>
        <name>rotated box</name>
        <rotated_box>
            <cx>1426</cx>
            <cy>311</cy>
            <width>360</width>
            <height>297</height>
            <rot>6.103084</rot>
        </rotated_box>
        <polygon>
            <x1>1223</x1>
            <y1>424</y1>
            <x2>1576</x2>
            <y2>489</y2>
            <x3>1629</x3>
            <y3>198</y3>
            <x4>1276</x4>
            <y4>133</y4>
        </polygon>
        <bndbox>
            <xmin>1223</xmin>
            <ymin>133</ymin>
            <xmax>1629</xmax>
            <ymax>489</ymax>
        </bndbox>
    </object>
    <object>
        <name>polygon</name>
        <polygon>
            <x1>1893</x1>
            <y1>505</y1>
            <x2>2300</x2>
            <y2>137</y2>
            <x3>2882</x3>
            <y3>739</y3>
        </polygon>
        <bndbox>
            <xmin>1893</xmin>
            <ymin>137</ymin>
            <xmax>2882</xmax>
            <ymax>739</ymax>
        </bndbox>
    </object>
    <object>
        <name>cubic_bezier</name>
        <cubic_bezier>
            <x1>3025</x1>
            <y1>501</y1>
            <x2>3381</x2>
            <y2>184</y2>
            <x3>3787</x3>
            <y3>414</y3>
        </cubic_bezier>
        <bndbox>
            <xmin>3025</xmin>
            <ymin>184</ymin>
            <xmax>3787</xmax>
            <ymax>501</ymax>
        </bndbox>
    </object>
    <object>
        <name>line</name>
        <line>
            <x1>403</x1>
            <y1>1159</y1>
            <x2>704</x2>
            <y2>920</y2>
            <x3>1146</x3>
            <y3>941</y3>
        </line>
        <bndbox>
            <xmin>403</xmin>
            <ymin>920</ymin>
            <xmax>1146</xmax>
            <ymax>1159</ymax>
        </bndbox>
    </object>
    <object>
        <name>keypoints</name>
        <keypoints>
            <x1>2094</x1>
            <y1>962</y1>
            <v1>2</v1>
            <x2>2344</x2>
            <y2>864</y2>
            <v2>2</v2>
            <x3>2902</x3>
            <y3>1277</y3>
            <v3>2</v3>
        </keypoints>
        <bndbox>
            <xmin>2094</xmin>
            <ymin>864</ymin>
            <xmax>2902</xmax>
            <ymax>1277</ymax>
        </bndbox>
    </object>
    <object>
        <name>pixels</name>
        <pixels>
            <id>0</id>
        </pixels>
        <bndbox>
            <xmin>1729</xmin>
            <ymin>1169</ymin>
            <xmax>1838</xmax>
            <ymax>1278</ymax>
        </bndbox>
    </object>
</annotation>

Export menu

Export train, val, and test.txt

Specify the split ratio "80/10/10" so that all images in the current folder are split into train, validation, and test set.

In the specified folder, train.txt, val.txt, and test.txt are saved.

raccoon-1.jpg
raccoon-2.jpg
...

Adding Prefix, you can make them full paths.

/Users/ryo/Desktop/test_annotations/raccoon_dataset-master/images/raccoon-1.jpg
/Users/ryo/Desktop/test_annotations/raccoon_dataset-master/images/raccoon-2.jpg
...

Export names for YOLO

The names file for YOLO is created from the objects table on the settings dialog.

raccoon

Export XML files to YOLO text files

Specify the folder to save the YOLO text files.

All objects are converted to boxes and a text file is saved for an image in the YOLO format.

If the label name is not found on the objects table, it would be skipped.

Refer to the YOLO document.

class_index center_x center_y width height
0 0.464615 0.594724 0.680000 0.769784

If your images folder is named "images", export YOLO text files into "labels" folder.

Otherwise export YOLO text files into your images folder.

darknet/src/data.c line 220
find_replace(path, "images", "labels", labelpath);
find_replace(labelpath, "JPEGImages", "labels", labelpath);
find_replace(labelpath, ".jpg", ".txt", labelpath);
find_replace(labelpath, ".JPG", ".txt", labelpath);
find_replace(labelpath, ".JPEG", ".txt", labelpath);

For cfg/raccoon.data, Export train, val, and test.txt as full paths.

classes= 1
train  = train.txt
valid  = val.txt
names = data/raccoon.names
backup = backup

Export label map for Tensorflow

The label map for TFRecord is created from the objects table on the settings dialog.

Bringing in your own dataset on Tensorflow Object Detection API

Here is an example of the label map.

item {
  id: 1
  name: 'raccoon'
}

Export mask images for the image

Specify the folder to save mask images.

The segmentation mask images for the current image are exported as the PNG format.

If the label name is not found on the objects table, it would be skipped.


You can specify which mask image to export.

  • "Export an image includes all objects": An index color image which includes all objects is saved as "{image_file_name}_all_objects.png".
  • "Export an image per object class": A gray image per object class is saved as "{image_file_name}_class_{class_name}.png".
  • "Export an image per object": A gray image per object is saved as "{image_file_name}_object{object_idx}.png".

For the index color image, all objects and their overlaps are based on the layer order on the label table.

Pixel values are set based on the "id" in the label map. For the background, 0 is set.

The index color table is created from object colors on the settings dialog.

For gray images, pixel values are set 255 for the foreground and 0 for the background.


Run an instance segmentation model on Tensorflow Object Detection API

Export index color mask image and separated mask images

Export mask images for all images

Specify the folder to save mask images.

The segmentation mask images for all images in the folder are exported.

If the label name is not found on the objects table, it would be skipped.

Export XML files to COCO JSON file

PASCAL VOC XML files are exported to COCO JSON file.

Refer to the COCO JSON document.

If the label name is not found on the objects table, it would be skipped.


For pixels and cubic bezier objects, "segmentation" is exported as RLE.

RLE is encoding the mask image using the COCO Mask API.

"segmentation":
{
    "size": [960, 896],
    "counts": "TP\\4<`m08J4L4M2M4M2N2N2N2N101N2N101O0O2O0O101O00000O10000000000001O0000000O2O000O2O0O2O1N1O2N2N2O1M3N3M2M4L5K5I[on4LiPQK4J6G9kROZO0Nkl0T1lNmNSUOW1kj0mNmTOZ1Qk0mNgTOU1Wk0SOcTOm0[k0m0O1N3M2O2M3M2O1N2N2N2O1N1O2O0O2N100002N1M3N2M201O1M3fNWUO\\Ojj0b0YUO\\Ohj0?cUOWOaj0f0dUOVO^j0i0dUOUO]j0h0gUOVOZj0i0_1O1N3M2M6M0N6K1L<DTck="
},

To decode the RLE in your python code, use the code below from rectlabel_create_coco_tf_record.py.

segm = object_annotations['segmentation']
m = mask.decode(segm)    
pil_image = PIL.Image.fromarray(m)

For box and polygon objects, "segmentation" is exported as polygon.

"segmentation" : [[x1, y1, x2, y2, ...]],

For keypoints objects, "keypoints" and "num_keypoints" are exported.

"segmentation": [],
"keypoints": [475, 124, 2, 482, 290, 2, 256, 405, 2, 752, 458, 2],
"num_keypoints": 4

For "categories", "keypoints" and "skeleton" are exported.

"categories": [
{
    "id": 1,
    "keypoints": ["0", "1", "2", "3", "4"],
    "name": "bare feet",
    "skeleton": [
        [1, 2],
        [2, 4],
        [2, 3]
    ]
},

To display an image with the annotation, you can use pycocoDemo.ipynb.

Import COCO JSON file to XML files

All objects in the COCO JSON file are imported to the XML files in the current folder.

Before importing, we would back up the current label files.

Export XML files to CSV file

When you export, check on "Convert all objects to rectangles" checkbox to convert all objects to rectangles.

(x, y) means the center of the box.

Refer to the Apple's Turi Create Object Detection document.

path,annotations
/Users/ryo/Desktop/test_annotations/test_data_csv/1.jpg,[{"label":"bare feet","type":"rectangle","coordinates":{"x":393,"y":838,"width":62,"height":118}},{"label":"bare feet","type":"rectangle","coordinates":{"x":392,"y":881,"width":52,"height":102}}]

To train a Turi Create model from the exported CSV file, use the code below.

train_turicreate.py

python train_turicreate.py "${EXPORTED_CSV_FILE}"

To export all objects as they are, check off "Convert all objects to rectangles" checkbox .

path,annotations
/Users/ryo/Desktop/test_annotations/keypoints_example_dataset/1_4.jpg,[{"label":"bare feet","type":"rectangle","coordinates":{"x":1167,"y":697,"width":223,"height":199}},{"label":"socks","type":"rotated_box","coordinates":{"cx":1484,"cy":576,"width":223,"height":199,"rot":5.980562}},{"label":"bare feet","type":"polygon","coordinates":{"x1":113,"y1":775,"x2":168,"y2":649,"x3":349,"y3":798}},{"label":"socks","type":"cubic_bezier","coordinates":{"x1":540,"y1":1078,"x2":630,"y2":950,"x3":767,"y3":1077}},{"label":"bare feet","type":"keypoints","coordinates":{"x1":476,"y1":125,"v1":2,"x2":483,"y2":291,"v2":2,"x3":257,"y3":406,"v3":0,"x4":753,"y4":459,"v4":2}},{"label":"socks","type":"pixels","id":0}]

Import CSV file to XML files

All objects in the CSV file are imported to the XML files in the current folder.

Before importing, we would back up the current label files.

Export XML files to Create ML JSON file

PASCAL VOC XML files are exported to Create ML JSON file.

Training Object Detection Models in Create ML

[{
    "image": "1.jpg",
    "annotations": [
    {
        "label": "bare feet",
        "coordinates":
        {
            "y": 838,
            "x": 393,
            "width": 62,
            "height": 118
        }
    },
    {
        "label": "bare feet",
        "coordinates":
        {
            "y": 881,
            "x": 392,
            "width": 51,
            "height": 102
        }
    }]
}]

Import Create ML JSON file to XML files

All objects in the Create ML JSON file are imported to the XML files in the current folder.

Before importing, we would back up the current label files.

Convert video to image frames

For "Video file", open a video file.

For "Output folder", open a folder to save the image frames.

For "Frames per a second", specify how many image frames to generate for a video second.

For "Image size max", both width and height would be less than or equal to the size.

Open the output folder to start labeling. To sort the image frames, set "Sort images" as Numeric.

Export objects and attributes stats

You can export text files which include the number of objects and attributes used in the current folder.

Resize all images and annotations

You can resize all images and label files in the current folder.

For "Image size max", both width and height would be less than or equal to the size.

If "Image size max" is empty, images are not resized and label files are resized to the same size as images.

Be careful that resizing would overwrite the current label files.

Before importing, we would back up the current label files.

Replace label name for all images

Replace the label name for all images in the folder.

Copy images to another folder

Combining "Search images", you can copy searched images and label files to the specified folder.

Move images to another folder

Combining "Search images", you can move unlabeled images to the specified folder.


Python tools

Export TFRecord for Tensorflow

Before starting these steps, be sure that you could set up Tensorflow Object Detection API in your environment.


To convert PASCAL VOC XML files to TFRecord file:


  1. Export train, val, and test.txt.
  2. Export label map for Tensorflow.
  3. To train the Mask-RCNN model, Export mask images for all images checking on "Export an image per object".
  4. Put the python code rectlabel_create_pascal_tf_record.py into "models/research/object_detection/dataset_tools".
  5. Change the current directory to "models/research/" and run.
python object_detection/dataset_tools/rectlabel_create_pascal_tf_record.py --images_dir="${IMAGES_DIR}" --train_txt_path="${TRAIN_TXT_PATH}" --val_txt_path="${VAL_TXT_PATH}" --annotations_dir="${ANNOTATIONS_DIR}" --label_map_path="${LABEL_MAP_PATH}" --output_dir="${OUTPUT_DIR}" --ignore_difficult_instances

flags.DEFINE_string('images_dir', '', 'Full path to images folder.')
flags.DEFINE_string('train_txt_path', '', 'Full path to train.txt.')
flags.DEFINE_string('val_txt_path', '', 'Full path to val.txt.')
flags.DEFINE_string('annotations_dir', '', 'Full path to annotations directory.')
flags.DEFINE_string('masks_dir', '', 'Full path to exported mask images folder.')
flags.DEFINE_string('label_map_path', 'label_map.pbtxt', 'Full path to label map file.')
flags.DEFINE_string('output_dir', '', 'Full path to output directory.')
flags.DEFINE_boolean('ignore_difficult_instances', False, 'Whether to ignore difficult instances')

To convert the COCO JSON file to the TFRecord file:


  1. Export train, val, and test.txt.
  2. Export XML files to COCO JSON file.
  3. Put the python code rectlabel_create_coco_tf_record.py into "models/research/object_detection/dataset_tools".
  4. Change the current directory to "models/research/" and run.
python object_detection/dataset_tools/rectlabel_create_coco_tf_record.py --images_dir="${IMAGES_DIR}" --train_txt_path="${TRAIN_TXT_PATH}" --val_txt_path="${VAL_TXT_PATH}" --annotations_path="${ANNOTATIONS_PATH}" --output_dir="${OUTPUT_DIR}"

flags.DEFINE_string('images_dir', '', 'Full path to images folder.')
flags.DEFINE_string('train_txt_path', '', 'Full path to train.txt.')
flags.DEFINE_string('val_txt_path', '', 'Full path to val.txt.')
flags.DEFINE_string('annotations_path', '', 'Full path to annotations JSON file.')
flags.DEFINE_string('output_dir', '', 'Full path to output directory.')
flags.DEFINE_boolean('include_masks', False, 'To train Mask-RCNN, add --include_masks.')

Train Mask R-CNN on Matterport

To train Matterport Mask R-CNN, follow these steps.

  1. Export XML files to COCO JSON file.
  2. Put the python code rectlabel_coco_matterport.py into "Mask_RCNN/samples/coco".
    Edit CocoConfig parameters if necessary.
  3. Change the current directory to "Mask_RCNN/samples/coco" and run.
# Train a new model starting from pre-trained COCO weights
python rectlabel_coco_matterport.py train --dataset=${DATASET_DIR} --annotations=annotations.json --weights=coco

# Resume training a model that you had trained earlier
python rectlabel_coco_matterport.py train --dataset=${DATASET_DIR} --annotations=annotations.json --weights=last

# Apply inference to an image
python rectlabel_coco_matterport.py inference --weights=last --image=${IMAGE_PATH}

Train Mask R-CNN / Keypoint Detection on Detectron2

To train Mask R-CNN / Keypoint Detection on Detectron2, follow these steps.

  1. Export XML files to COCO JSON file.
  2. Put the python code rectlabel_coco_detectron2.py into your preferred path.
    Edit cfg parameters if necessary.
  3. Run.
# Train a new model starting from pre-trained COCO weights
python rectlabel_coco_detectron2.py train --type=${TYPE} --images_dir=${IMAGES_DIR} --annotations_path=${ANNOTATIONS_PATH} --weights=coco

# Resume training a model that you had trained earlier
python rectlabel_coco_detectron2.py train --type=${TYPE} --images_dir=${IMAGES_DIR} --annotations_path=${ANNOTATIONS_PATH} --weights=last

# Apply inference to an image
python rectlabel_coco_detectron2.py inference --type=${TYPE} --weights=last --image=${IMAGE_PATH}

Convert COCO JSON file to SageMaker JSON files

To convert the COCO JSON file to the Amazon SageMaker Object Detection JSON files, you can use coco_json_to_sagemaker_json.py.

Convert DICOM to jpg images

To convert DICOM to jpg images, use dicom_to_images.py.


Edit menu

Create box

Change the mode to "Create box".

To draw a box, click 2 points.


Or click 4 points for xmin, xmax, ymin, and ymax of the object..

Refer to "Extreme clicking for efficient object annotation".

Press enter key to finish drawing when the number of points is less than 4.


When you finished drawing, the label dialog would open.

The label would be added to the label table on the right.

Drag the center of the box to move the box.

Drag one of the four corner points to transform the box.


To change the box color, use the color picker at the top-right corner of the image.

You can hide the cross hairs using "Hide cross hairs".

For the cross hairs, the default color is being used.

To change the default color, deselect all boxes and change the color.

To show the label name on each box, use "Show label on the box".

Create polygon, cubic bezier, and line

Change the mode to "Create polygon", "Create cubic bezier", or "Create line".

Click to add points.

Press enter key to finish drawing.

Press escape key to cancel drawing.

To show all edit points, use "Show all edit points".


When you right click on the point, edit menu would open.

"Add a point forward" and "Add a point backward" to add a point.

"Delete this point" to delete the point.

"Point size up" and "Point size down" to change the size of points.

"To polygon", "To cubic bezier", and "To line" change the polygon type.

Create keypoints

You can label keypoints for COCO Keypoint Detection Task.

Change the mode to "Create keypoints".

Click to add points.

Click holding option button, the point is added as invisible.

Press enter key to finish drawing.

Press escape key to cancel drawing.

To show all edit points, use "Show all edit points".

To hide keypoints names, use "Hide keypoints names" hotkey.


When you right click on the point, edit menu would open.

"Change keypoint name" to change the keypoint name.

"Make invisible" to make the point invisible.

"Delete edge with" to delete the edge with the point.


If you put empty string to the keypoint name, the keypoint name is hidden.

To add an edge, drag from one to another point holding option button.


When you right click on the label, edit menu would open.

"Flip" to flip the keypoints.

"Make visible" to make the point visible.


Keypoints names and edges are saved in the settings file.

For the first keypoints object, you have to press enter key to finish drawing, change keypoints names, and add edges.

From the second keypoints object, if currently selected object or lastly selected object has keypoints names/edges, the label dialog would appear without pressing the enter key and keypoints names/edges are automatically shown.

Label image

You can label the whole image without drawing boxes.

Label pixels

You can label pixels using brush and superpixel tools.


Brush size 1 means 1px in the image.

Using brush size history, you can change the brush size quickly.

For large images, adjust "Brush size max".

Erase is used to erase pixels.

Polygon is used to label pixels using the polygon tool.

Right clicking on the pixels, "Flood Fill" and "Clear pixels" menus would appear.

To hide pixels, use "Hide pixels" hotkey.

The pixels image file is saved as "{image_file_name}_pixels{pixels_idx}.png" in the annotations folder.

Label pixels with brush and superpixel tools

Superpixel size is used to adjust the segmentation size. For small objects, use small values.

Superpixel smoothness is used to adjust the segmentation boundary smoothness. For the detailed boundary, use small values.

On the pixels dialog, right arrow and left arrow keys change the superpixel size by 1px.

To hide superpixels, use "Hide superpixels" hotkey.

When the superpixel is calculated, the image is resized to 960px.

The superpixel calculation is done in async and it would take seconds.

If superpixels are not shown, reselect the label on the label table to start the calculation.

Label pixels with brush and superpixel tools

Import pixels

At first, "Label pixels" to create an empty pixels label.

Then, right click on the label, edit menu would open.

"Import pixels" to import your mask image.

Move

Change the mode to "Move".

You can switch between Create and Move mode holding space key when you are in Create mode.

Drag the box or the image to move its position.

For the image, mouse wheel can be used to move.

You can select multiple boxes and move them.

When you click on the box or the label, four corner points would appear.

Drag one of the four corner points to transform the box.

When you right click on the box or the label, edit menu would open.

"Focus" to quick zoom to the selected box, "Edit" to open the label dialog, "Duplicate" to duplicate the box, and "Delete" to delete the box.

When you double click on the box or the label, the label dialog would open.

To change the layer order of the box, drag the label in the label table upward or downward.

Rotate

Change the mode to "Rotate".

Drag up/down on the box to rotate the box.

You can select multiple boxes and rotate them.

Delete

You can select multiple boxes and delete them.

Layer up/down

Change the layer order of the box.

Change image brightness and contrast

Change the image brightness and contrast for dark images.

Change object color

Change the object color using color picker.

Clear object color

Clear the object color to the default color.

Undo/Redo

You can undo/redo operations.

Copy/Paste

You can select multiple boxes and copy/paste them across images.


Search menu

Search images

When you open the folder, we read all xml files in the folder and start indexing asynchronously.

When you edit some label files, we track which label file was edited.

To index those edited label files, press "Indexing" on the search dialog.

To clear the search result, use "Clear search images".

You can use Wildcard(*), AND(&), OR(|), NOT(!), and more in the search text.

To search unlabeled images, use empty search text.

Reload the images folder.


Core ML menu

Load Core ML model

Your macOS version has to be GTE 10.13.

Load the Core ML model and show the description in metadata.

This blog provides the trained model: MobileNetV2 + SSDLite with Core ML

NSString *desctiption = model.modelDescription.metadata[MLModelDescriptionKey];

Process the image using Core ML model

Your macOS version has to be GTE 10.13.

Process the image using the loaded Core ML model and add detected boxes or whole image label to the label table.


Before running the model, we check the user defined dictionary in the model.

NSDictionary *userDefinedDict = model.modelDescription.metadata[MLModelCreatorDefinedKey];

Check the class names.

NSArray *classesArray = [[NSArray alloc] init];
if([userDefinedDict objectForKey:(NSString*)kUserDefinedClasses] != nil){
    NSString *classesText = [userDefinedDict objectForKey:(NSString*)kUserDefinedClasses];
    classesArray = [classesText componentsSeparatedByString:(NSString*)kComma];
}

Check whether the model created by Turi Create so that box origin is the center of the box.

bool boxCentered = false;
if([userDefinedDict objectForKey:(NSString*)kTuriCreateVersion] != nil){
    boxCentered = true;
}

Check the confidence threshold.

float confidenceThreshold = kConfidenceThresholdDefault;
if([userDefinedDict objectForKey:(NSString*)kConfidenceThreshold] != nil){
    NSString *confidenceThresholdText = [userDefinedDict objectForKey:(NSString*)kConfidenceThreshold];
    confidenceThreshold = [confidenceThresholdText floatValue];
}

Check the Non-Maximum Suppression threshold.

float nmsThreshold = kNMSThresholdDefault;
if([userDefinedDict objectForKey:(NSString*)kNMSThreshold] != nil){
    NSString *nmsThresholdText = [userDefinedDict objectForKey:(NSString*)kNMSThreshold];
    nmsThreshold = [nmsThresholdText floatValue];
}

For the Non-Maximum Suppression, if your macOS is GTE 10.14, you can include the NMS in the Core ML model.

model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=True)

So that we can easily obtain the bounding box.

for(VNRecognizedObjectObservation *prediction in results){
    VNClassificationObservation *labels0 = [prediction.labels objectAtIndex:0];
    NSString *label = [NSString stringWithFormat:@"%@-%.2f", labels0.identifier, labels0.confidence];
    CGRect rect = prediction.boundingBox;
    ...

If your macOS is 10.13, you cannot include the NMS in the Core ML model.

model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=False)

So that we have to calculate the NMS.

VNCoreMLFeatureValueObservation *coordinatesObservation  = results[0];
VNCoreMLFeatureValueObservation *confidenceObservation = results[1];
MLMultiArray *coordinates  = coordinatesObservation.featureValue.multiArrayValue;
MLMultiArray *confidence = confidenceObservation.featureValue.multiArrayValue;
NSMutableArray *unorderedPredictions = [[NSMutableArray alloc] init];
int numBoundingBoxes = [confidence.shape[0] intValue];
int numClasses = [confidence.shape[1] intValue];
for (int b = 0; b < numBoundingBoxes; b++) {
    float maxConfidence = 0.0;
    int maxIndex = 0;
    for (int c = 0; c < numClasses; c++) {
        NSNumber *confNumber = confidence[b * numClasses + c];
        double conf = [confNumber doubleValue];
        if (conf > maxConfidence) {
            maxConfidence = conf;
            maxIndex = c;
        }
    }
    if (maxConfidence > confidenceThreshold) {
        float x = [coordinates[b * 4 + 0] floatValue];
        float y = [coordinates[b * 4 + 1] floatValue];
        float w = [coordinates[b * 4 + 2] floatValue];
        float h = [coordinates[b * 4 + 3] floatValue];
        if(boxCentered){
            x -= w / 2;
            y -= h / 2;
        }
        CGRect rect = CGRectMake(x, y, w, h);
        Prediction *prediction = [[Prediction alloc] init];
        prediction.labelIndex = maxIndex;
        prediction.confidence = maxConfidence;
        prediction.boundingBox = rect;
        [unorderedPredictions addObject:prediction];
    }
}
...

For the details, refer to the Apple's document.

Deploying to Core ML

For the whole image label, we obtain only class name without the bounding box.

VNClassificationObservation *prediction = results.firstObject;
NSString *label = [NSString stringWithFormat:@"%@-%.2f", prediction.identifier, prediction.confidence];
...

Process all images using Core ML model

Your macOS version has to be GTE 10.13.

Process all images in the folder using the loaded Core ML model and replace the label files with detected boxes or whole image label.

Before running, we would back up the current label files.


View menu

Zoom in, Zoom out

Click a position to zoom in or zoom out.

Or using command + mouse wheel down/up, you can zoom in/out.

Zoom fit

Clear zoom.

Trackpad gestures

Slide two fingers to move the image.

Double-tap with two fingers for smart zoom.

Pinch with two fingers to zoom in or out.

Swipe left or right with three fingers to change the image.


System Preferences > Trackpad > More Gestures tab.

For "Swipe between pages", select "Swipe with two or three fingers" or "Swipe with three fingers".

For "Swipe between full-screen apps", select "Swipe left or right with four fingers".

Magic Mouse gestures

Slide a finger to move the image.

Double-tap with a finger for smart zoom.

Swipe left or right with two fingers to change the image.


System Preferences > Mouse > Point & Click tab > Smart zoom.

System Preferences > Mouse > More Gestures tab.

For "Swipe between pages", select "Swipe with two fingers".

Focus box

You can quick zoom to the selected box.

Hide other boxes

Hide other boxes except the selected box. Toggle boxes alpha among 1.0, 0.3, and 0.0.

Hide cross hairs

Hide cross hairs when creating box.

Hide keypoints names

Hide keypoints names when creating keypoints.

Hide pixels

Toggle pixles alpha among 1.0, 0.5, and 0.0.

Hide superpixels

Toggle superpixels alpha among 1.0, 0.5, and 0.0.

Show depth channel

When the rgb image name is image_name.jpg, put the depth image as image_name_depth.png in the same folder.

Toggle depth image alpha among 0.0, 0.5, and 1.0.

Show label on the box

Show label name on each box.