Help Overview


RectLabel version 2.87

An image annotation tool to label images for bounding box object detection and segmentation.


- Improved so that you can draw or edit the box as square pressing Shift key.

- Fixed the problem that annotator name is cleared when you close RectLabel without opening the settings dialog.

Customize the label dialog to combine with attributes

Key features

Drawing bounding box, polygon, cubic bezier, line, and point

Label the whole image without drawing boxes

Label pixels with brush and superpixel tools

Export index color mask image and separated mask images

Export to the YOLO, KITTI, COCO JSON, and CSV format

Read and write in the PASCAL VOC XML format

Automatically label images using Core ML model

Auto-suggest speeds up selecting the object name

Customize the label dialog to combine with attributes

1-click buttons make your labeling work faster

Settings for objects, attributes, hotkeys, and labeling fast

Search images by label text

Convert video to image frames

Privacy policy

We Collect No Personal Information Using Our Applications.

Terms of Use

For a Full App, 1 user can use RectLabel on 2 devices.

Buy RectLabel

Start your 1 Month free trial.

Then $2.99 / month or $29.99 / Year.


Payment will be charged to your Apple ID account after the 1 month free trial.

Subscription automatically renews unless it is canceled at least 24 hours before the end of the 1 month free trial.

Your account will be charged for renewal within 24 hours prior to the end of the 1 month free trial.

You can manage and cancel your subscriptions by going to your account settings on the App Store after purchase.

Any unused portion of a free trial period, if offered, will be forfeited when the user purchases a subscription to that publication, where applicable.

For a Full App, 1 user can use RectLabel on 2 devices.

Report issues

Post the problem to our Github issues page.

Write a review

Give us your feedback to improve RectLabel.

Have questions?

Send an email to support@rectlabel.com

Thank you.

Requested features

  • iPad app.

Troubleshooting

Find crash logs.

  1. You can find crash logs here. "~/Library/Logs/DiagnosticReports/".
  2. Please send the crash log to support@rectlabel.com

Clearing cache files.

  1. Copy the settings file "~/Library/Containers/com.waysify.roi/Data/settings_labels.json".
  2. Remove cache files "~/Library/Containers/com.waysify.roi/".
  3. Import the copied settings file from app menu.

Cannot save bookmark to the folder.

Solving the problem when you buy in-app purchase.

  1. Delete RectLabel via the Launchpad app
  2. Log out of the Mac App Store and iTunes.
  3. Reboot your Mac.
  4. Log back in to the Mac App Store and install RectLabel.

Papers cited RectLabel

Training Sample Generation Software

SEMANTIC PHOTOGRAMMETRY: BOOSTING IMAGE-BASED 3D RECONSTRUCTION WITH SEMANTIC LABELING

Reducing the Pain: A Novel Tool for Efficient Ground-Truth Labelling in Images

Individual tree-crown detection in RGB imagery using self-supervised deep learning neural networks

Beef Cattle Instance Segmentation Using Mask R-Convolutional Neural Network

FingerReader2.0: Designing and Evaluating a Wearable Finger-Worn Camera to Assist People with Visual Impairments while Shopping

Fashion Object Detection and Pixel-Wise Semantic Segmentation

Mobile Object Detection using TensorFlow Lite and Transfer Learning

Deep learning in Dynamic Imager

A machine learning approach for jaundice detection using color corrected smartphone images

SCREEN-DR - Software Architecture for the Diabetic Retinopathy Screening

Faster R-CNN-Based Glomerular Detection in Multistained Human Whole Slide Images

Buy RectLabel

Start your 1 Month free trial.

Then $2.99 / month or $29.99 / Year.


Payment will be charged to your Apple ID account after the 1 month free trial.

Subscription automatically renews unless it is canceled at least 24 hours before the end of the 1 month free trial.

Your account will be charged for renewal within 24 hours prior to the end of the 1 month free trial.

You can manage and cancel your subscriptions by going to your account settings on the App Store after purchase.

Any unused portion of a free trial period, if offered, will be forfeited when the user purchases a subscription to that publication, where applicable.

For a Full App, 1 user can use RectLabel on 2 devices.

Settings

Projects

You can switch different objects and attributes settings for different labeling tasks.

To add another settings file to "Projects" table, import the settings file.

To use the newly added project, you have to check on the "Primary" check box.

To duplicate the project, right click on the row and "Duplicate" menu would open.

Settings for objects, attributes, hotkeys, and labeling fast

Objects

"Objects" table describes the object index of each object.

When you export, if the lable name is not found on the objects table, it would be skipped.

You can assign 0-9 num keys and A-Z alphabet keys to objects hotkeys.

To duplicate the object, right click on the row and "Duplicate" menu would open.

Right click on the objects table header, then "Sort alphabetically", "Set default hotkeys", and "Clear objects table" menu would open.


When you label "sneakers" which uses 3 attributes "color", "brand", and "side".

Settings for objects, attributes, hotkeys, and labeling fast

If you have many objects, auto-suggest speeds up selecting the object name.

Auto-suggest speeds up selecting the object name

Attributes

The label "sneakers-yellow-converse-right" is a combination of the object and attributes.

'-' is used as a separator so that '-' in the object name and attribute name would be replaced with '_'.

The prefix is used as '-' + prefix + attribute name.

if any objects are not using attributes, '-' in the label name is not replaced with '_'.

The type is among "Single select", "Multiple select", and "Text input". "Text input" is useful for text detection.

To change the name on the items table, click or press the enter key on the selected item.

To duplicate the attribute, right click on the row and "Duplicate" menu would open.

Settings for objects, attributes, hotkeys, and labeling fast

Hotkeys

Customize the hotkeys to make your labeling work faster.

Settings for objects, attributes, hotkeys, and labeling fast

Here is the default hotkeys.

Settings ⇧⌘T
Open images folder and annotations folder ⌘O
Open images folder ⇧⌘O
Open images folder recursively ⌃O
Next image
Prev image
Jump to image index ⌘J
Open image in the folder ⌃I
Move image to another folder ⌘⌫
Save ⌘S
Create box ⌘N
Create polygon ⌘P
Create cubic bezier ⇧⌘B
Create line ⌃L
Create point ⌃P
Label image ⌘L
Label pixels ⇧⌘P
Move ⌘M
Rotate ⌘R
Copy box ⌘C
Paste box ⌘V
Duplicate box ⌘D
Delete box
Layer up ⌘↑
Layer down ⌘↓
Clear current color ⇧⌘C
Load labels from prev image ⌃C
Search images ⌘F
Clear search images ⇧⌘F
Process the image using Core ML ⌃A
Zoom in ⇧⌘↑
Zoom out ⇧⌘↓
Zoom fit ⌘0
Focus box ⌘G
Hide other boxes ⌘B
Hide cross hairs ⌃G
Hide pixels ⇧⌘G
Hide superpixels ⇧⌘H
Show label on the box ⌃B

Label fast

"Auto Save" is to skip the confirm dialog to save the annotation.

"Skip Label Dialog When Create" is to skip showing the label dialog when you create.

"Use 0-9 and a-z hotkeys" is to prevent from changing label names accidentally.

"Use 1-click buttons" shows 1-click buttons of all objects on the label dialog.

"Close dialog when select a object" is to skip clicking the OK button on the label dialog.

"Maintain zoom and position" is to maintain zoom and position when you change the image.

"Use fast zoom for large images" is to make zooming fast for large images.

Settings for objects, attributes, hotkeys, and labeling fast

Mask

You can specify which mask image you want to export.

The mask image which includes all objects is saved as "{image_file_name}_all_objects.png".

The mask image per object class is saved as "{image_file_name}_class_{class_name}.png".

The mask image per object is saved as "{image_file_name}_object{object_idx}.png".

The class name is based on the object name on the "Objects" table.

The object index is based on the layer order on the label table.

Settings for objects, attributes, hotkeys, and labeling fast

Draw

"Use circle edit points" shows the circle edit points instead of rectangle edit points.

"Use aspect ratio when draw boxes" shows the aspect ratio on the "Objects" tab.

"Click 4 points when draw boxes" is used when you draw a box clicking xmin, xmax, ymin, and ymax of the object.

Refer to "Extreme clicking for efficient object annotation".

"Show all edit points" is to show edit points of all objects. This is useful when you "Create point".

"Show edit points between corners on boxes" is to show edit points between corners on boxes and rotated boxes.

"Superpixel size max" is to change the superpixel size max for large images. ( > 30)

"Superpixel image size" is to change the image size for superpixel calculation if the image size is larger than the size. (480px - 4096px)

Settings for objects, attributes, hotkeys, and labeling fast

Others

"Allow outside the image" is to move the box outside the image.

"Fix the image position" is not to move the image position.

"Use truncated, occluded and difficult" shows those parameters on the label dialog.

"Hide arrow buttons" is to hide arrow buttons on the image.

"Sort images" is to sort images by Alphabetic, Numeric, and Last modified.

"Annotator" is saved in the xml file.

Settings for objects, attributes, hotkeys, and labeling fast

Export settings file

You can export the current settings file to import to another computer.

Here is an example of the JSON format.

{
    "projects": [
    {
        "fashion":
        {
            "objects": [
            {
                "hidden": false,
                "aspect_ratio": "",
                "name": "sneakers",
                "attributes": ["color", "brand", "text"],
                "color":
                {
                    "red": 0,
                    "green": 1,
                    "blue": 1
                }
            },
            {
                "hidden": false,
                "aspect_ratio": "",
                "name": "ignore",
                "attributes": [],
                "color":
                {
                    "red": 0,
                    "green": 1,
                    "blue": 1
                }
            }],
            "attributes": [
            {
                "prefix": "",
                "name": "color",
                "type": "Multiple select",
                "items": ["black", "red", "white", "yellow"]
            },
            {
                "prefix": "",
                "name": "brand",
                "type": "Single select",
                "items": ["", "adidas", "converse", "nike", "vans"]
            },
            {
                "prefix": "",
                "name": "text",
                "type": "Text input",
                "items": []
            }],
            "primary": true
        }
    }]
}

Import settings file

When you import the settings file, the content is added to "Projects" table.

To use the newly added project, you have to check on the "Primary" check box.

Export objects table

You can export the current objects table to import to another computer.

Here is an example of text format.

car
plate
person
dog

Import objects table

When you import the objects table, the content is added to "Objects" table.

To clear the objects table before importing, right clicking on the objects table header, "Clear objects table" menu would open.

Import objects table from API

On the dialog, you can specify the API URL.

Here is an example of the JSON format from the API response.

[
  "car",
  "plate",
  "person",
  "dog"
]

File menu

Open images folder and annotations folder

Open the images folder and annotations folder.

Image file names which include "_pixels" or "_depth" are skipped.

RectLabel reads and writes in the PASCAL VOC XML format.

Open images folder

For the annotations folder, we always use "images_folder/annotations" folder.

Open images folder recursively

Open images from sub folders, too. For the annotations folder, we always use "images_folder/annotations" folder.

Next image and Prev image

To show the next image, press the right arrow hotkey or click the right arrow button on the image.

To show the previous image, press the left arrow hotkey or click the left arrow button on the image.

Jump to image index

You can specify the image index to show.

Open image in the folder

You can open an image in the folder using the file dialog.

This is useful when you want to browse images as thumbnails.

Move image to another folder

You can move the image and the label file to the specified folder.

Save annotations

The label file is saved as "{image_file_name}.xml".

Output format

The label file is saved in the PASCAL VOC XML format.

The top-left pixel in the image has coordinates (1, 1).

The VOC2007 Development Kit

The rotated box is saved in the format as (center_x, center_y, width, height, rotation).

The rotation is counter-clockwise and ranged between 0 and 2Pi.

Here is an example of the PASCAL VOC format.

<annotation>
    <folder>test_data</folder>
    <filename>26073093-275503062977108-7595223473371217920-n_2.jpg</filename>
    <size>
        <width>767</width>
        <height>959</height>
        <depth>3</depth>
    </size>
    <segmented>0</segmented>
    <object>
        <name>bag</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <bndbox>
            <xmin>108</xmin>
            <ymin>101</ymin>
            <xmax>251</xmax>
            <ymax>238</ymax>
        </bndbox>
    </object>
    <object>
        <name>boots</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <rotated_box>
            <cx>405</cx>
            <cy>170</cy>
            <width>144</width>
            <height>138</height>
            <rot>0.474627</rot>
        </rotated_box>
        <polygon>
            <x1>373</x1>
            <y1>264</y1>
            <x2>500</x2>
            <y2>198</y2>
            <x3>437</x3>
            <y3>76</y3>
            <x4>310</x4>
            <y4>142</y4>
        </polygon>
        <bndbox>
            <xmin>310</xmin>
            <ymin>76</ymin>
            <xmax>500</xmax>
            <ymax>264</ymax>
        </bndbox>
    </object>
    <object>
        <name>flats</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <polygon>
            <x1>136</x1>
            <y1>463</y1>
            <x2>205</x2>
            <y2>374</y2>
            <x3>343</x3>
            <y3>405</y3>
            <x4>276</x4>
            <y4>506</y4>
        </polygon>
        <bndbox>
            <xmin>136</xmin>
            <ymin>374</ymin>
            <xmax>343</xmax>
            <ymax>506</ymax>
        </bndbox>
    </object>
    <object>
        <name>glasses</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <cubic_bezier>
            <x1>397</x1>
            <y1>467</y1>
            <x2>466</x2>
            <y2>378</y2>
            <x3>604</x3>
            <y3>409</y3>
            <x4>537</x4>
            <y4>510</y4>
        </cubic_bezier>
        <bndbox>
            <xmin>397</xmin>
            <ymin>378</ymin>
            <xmax>604</xmax>
            <ymax>510</ymax>
        </bndbox>
    </object>
    <object>
        <name>line</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <line>
            <x1>133</x1>
            <y1>651</y1>
            <x2>229</x2>
            <y2>561</y2>
        </line>
        <bndbox>
            <xmin>133</xmin>
            <ymin>561</ymin>
            <xmax>229</xmax>
            <ymax>651</ymax>
        </bndbox>
    </object>
    <object>
        <name>point</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <point>
            <x1>456</x1>
            <y1>596</y1>
        </point>
    </object>
    <object>
        <name>image</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
    </object>
    <object>
        <name>person</name>
        <pose>Unspecified</pose>
        <truncated>0</truncated>
        <occluded>0</occluded>
        <difficult>0</difficult>
        <pixels>
            <id>0</id>
        </pixels>
        <bndbox>
            <xmin>207</xmin>
            <ymin>325</ymin>
            <xmax>291</xmax>
            <ymax>399</ymax>
        </bndbox>
    </object>
</annotation>

Export menu

Export train, val, and test.txt

Specify the split ratio "60/20/20" so that all images in the current folder are split into train, validation, and test set.

In the specified folder, train.txt, val.txt, and test.txt are saved.

If you check the "Full path" button, each line includes the full path.

/Users/ryo/Desktop/test_annotations/test_data_small/3.jpg
/Users/ryo/Desktop/test_annotations/test_data_small/2.jpg
/Users/ryo/Desktop/test_annotations/test_data_small/lukasz-lada-59786-unsplash.jpg

If you uncheck the "Full path" button, each line includes the image name.

2
3
lukasz-lada-59786-unsplash

Export names for YOLO

The names file for YOLO is created from the objects table on the settings dialog.

Here is an example of the names file.

car
plate
person
dog

Export XML files to YOLO txt files

Specify the folder to save the YOLO text files.

All objects are converted to boxes and a text file is saved for an image in the YOLO format.

If the lable name is not found on the objects table, it would be skipped.

Refer to the YOLO document.

class_index center_x center_y width height
0 0.511161 0.144792 0.103795 0.095833

Export XML files to KITTI txt files

Specify the folder to save the KITTI text files.

All objects are converted to boxes and a text file is saved for an image in the KITTI format.

If the lable name is not found on the objects table, it would be skipped.

If you use DetectNet, only type, truncated, and bbox are used.

Refer to the KITTI document.

type(1) truncated(1) occluded(1) alpha(1) bbox(4) dimensions(3) location(3) rotation_y(1)
person 0 0 0 174.0 367.0 423.0 602.0 0 0 0 0 0 0 0

Export label map for Tensorflow

The label map for TFRecord is created from the objects table on the settings dialog.

Bringing in your own dataset on Tensorflow Object Detection API

Here is an example of the label map.

item {
  id: 1
  name: 'car'
}

item {
  id: 2
  name: 'plate'
}

item {
  id: 3
  name: 'person'
}

item {
  id: 4
  name: 'dog'
}

Export TFRecord for Tensorflow

Before starting these steps, be sure that you could set up Tensorflow Object Detection API in your environment.

Check these tutorials could work in your environment.

  1. Object Detection Demo
  2. Training an object detector

Output files from RectLabel.

  1. Press "Settings" and add object names on the "Objects" table.
  2. Press "Open folder" to open Images Folder and Annotations Folder.
  3. Draw boxes and select an object name from the "Objects" pop-up button.
  4. Press "Save" button to save the XML file in the annotations folder.
  5. Press "Export train, val, and test.txt". Check on the "Full path" check box.
  6. Press "Export label map for Tensorflow".
  7. If you train Mask-RCNN, check on "Export an image per object" on the settings dialog. Then press "Export mask images for all images" and specify the current annotations folder to save.

Convert to the TFRecord file.

  1. Put the python code below to "models/research/object_detection/dataset_tools/" folder.
  2. Change the current directory to "models/research/".
  3. On the terminal, execute the python code.

To convert the PASCAL VOC XML files to the TFRecord file, use the code below.

For train.txt, val.txt, and test.txt, you have to execute the script 3 times to obtain train.tfrecord, val.tfrecord, and test.tfrecord.

rectlabel_create_pascal_tf_record.py

python object_detection/dataset_tools/rectlabel_create_pascal_tf_record.py \
    --images_dir="${IMAGES_DIR}" \
    --annotations_dir="${ANNOTATIONS_DIR}" \
    --image_list_path="${IMAGE_LIST_PATH}" \
    --label_map_path="${LABEL_MAP_PATH}" \
    --output_path="${OUTPUT_PATH}" \
    --include_masks

flags.DEFINE_string('images_dir', '', 'Full path to images directory.')
flags.DEFINE_string('annotations_dir', 'annotations', 'Relative path to annotations directory from images_dir.')
flags.DEFINE_string('image_list_path', 'train.txt', 'Full path to image list file which is one of the output files from "Export train, val, and test.txt".')
flags.DEFINE_string('label_map_path', 'label_map.pbtxt', 'Full path to label map file.')
flags.DEFINE_string('output_path', '', 'Full path to output TFRecord file.')
flags.DEFINE_boolean('include_masks', False, 'If you train Mask-RCNN, add --include_masks otherwise you can remove it. Mask images are expected to be png files and in the annotations folder.')

To convert the COCO JSON file to the TFRecord file, use the code below.

rectlabel_create_coco_tf_record.py

python object_detection/dataset_tools/rectlabel_create_coco_tf_record.py \
    --train_image_dir="${TRAIN_IMAGE_DIR}" \
    --val_image_dir="${VAL_IMAGE_DIR}" \
    --train_annotations_file="${TRAIN_ANNOTATIONS_FILE}" \
    --val_annotations_file="${VAL_ANNOTATIONS_FILE}" \
    --output_dir="${OUTPUT_DIR}" \
    --include_masks

flags.DEFINE_string('train_image_dir', '', 'Full path to training images directory.')
flags.DEFINE_string('val_image_dir', '', 'Full path to validation images directory.')
flags.DEFINE_string('train_annotations_file', '', 'Full path to training annotations JSON file.')
flags.DEFINE_string('val_annotations_file', '', 'Full path to validation annotations JSON file.')
flags.DEFINE_string('output_dir', '/tmp/', 'Full path to output data directory.')
flags.DEFINE_boolean('include_masks', False, 'If you train Mask-RCNN, add --include_masks otherwise you can remove it. "segmentation" is expected to be RLE format.')

With the code above and exporting mask images, you can train the Mask R-CNN model with your own data.

Export mask images for the image

Specify the folder to save mask images.

The segmentation mask images for the current image are exported as the PNG format.

If the lable name is not found on the objects table, it would be skipped.


You can specify which mask image you want to export.

For an index color mask image, all objects and their overlaps are based on the layer order on the label table.

Pixel values are set based on the "id" in the label map. For the background, 0 is set.

The index color table is created from object colors on the settings dialog.

checkIndexColorPalette.py

For separated mask images, pixel values are set 255 for the foreground and 0 for the background.


Run an instance segmentation model on Tensorflow Object Detection API

Export index color mask image and separated mask images

Export mask images for all images

Specify the folder to save mask images.

The segmentation mask images for all images in the folder are exported.

If the lable name is not found on the objects table, it would be skipped.

Export XML files to COCO JSON file

The PASCAL VOC XML files are exported to the COCO JSON file.

If the lable name is not found on the objects table, it would be skipped.

COCO JSON format

Check off the "Export segmentation as RLE" checkbox to export segmentation as polygon.

"segmentation" : [[x1, y1, x2, y2, ...]],

Check on the box to export segmentation as RLE.

The mask images are encoded as RLE using the COCO Mask API.

"segmentation" : {
  "size" : [
    960,
    768
  ],
  "counts" : "Q_h98bg06O1O1O1O1O1N2O2N1O1O1O1N2O1O1O1O1O1N2O1O1O1O1N2O1O1O1O1O1N2O2N1O1O1O1N2O1O1O1O1O1N2O1O1O1O1N2O1O1O1O1O1N2O2N1O1O1O1N2O1O1O1O1O1N2O1O1O1O1N2O1O1O1O1O1O101O000001O00000000000000001O0001O00001O001O01O01O00001O001O01O01O00001O001O00010O00001O00001O0010O0001O00001O001O01O0001O01O1O1O1O2O0O1O1O2N1O100O0000000001O01O00000001O00000000001O00000000010O00000000001O0001O001O010O001O001O01O01O001O0010O01O001O0001O00000000001O0000001O1O1N2O1O1O2N1O1O1O1O1O2N1O1O1O1O1O2N1O1O1O1O1O1O000000O100000O10O1000000O1000000O100000000O1000000O1000000O100000000O0100000O1000000O100000000O1000000O1000000O100000000O1000O10O1000000O100000000O1000000O100O1O100O1O1O1O100O1O1O10O01O1O1O100O1O100000000000000000000000001O000000000000000O10000O2O0O10000O10000O100O10001N100O1001O00001O00000010O0001O00000106I7I6K6I7I7F9VOPR]3"
},

To decode the RLE and use the mask images in your python code, use the code below from rectlabel_create_coco_tf_record.py.

binary_mask = mask.decode(object_annotations['segmentation'])
mask_shape = binary_mask.shape
binary_mask = np.ravel(binary_mask, order='F')
binary_mask = binary_mask.reshape(mask_shape, order='C')

Import COCO JSON file to XML files

All objects in the COCO JSON file are imported to the XML files in the current folder.

Be careful that importing would overwrite the current label files.

Before importing, we would back up the current label files.

Export XML files to CSV file

To start with, refer to the Apple's Turi Create Object Detection document.


When you export, check on "Convert all objects to rectangles" checkbox to convert all objects to rectangles.

(x, y) means the center of the box.

path,annotations
/Users/ryo/Desktop/test_annotations/test_data_csv/1.jpg,[{"label":"bare feet","type":"rectangle","coordinates":{"x":393,"y":838,"width":62,"height":118}},{"label":"bare feet","type":"rectangle","coordinates":{"x":392,"y":881,"width":52,"height":102}}]
...

To train from the exported CSV file, use the code below.

train_turicreate.py

python train_turicreate.py "${EXPORTED_CSV_FILE}"

If you opened images folder recursively, use the code below.

train_turicreate_recursive.py

python train_turicreate_recursive.py "${EXPORTED_CSV_FILE}" "${IMAGES_ROOT_FOLDER}"

When you export, check off "Convert all objects to rectangles" checkbox to export all objects as they are.

path,annotations
/Users/ryo/Desktop/test_annotations/test_data_create_types/gray.jpg,[{"label":"box","type":"rectangle","coordinates":{"x":445,"y":148,"width":253,"height":244}},{"label":"rotated box","type":"rotated_box","coordinates":{"cx":908,"cy":174,"width":247,"height":243,"rot":5.937303}},{"label":"polygon","type":"polygon","coordinates":{"x1":237,"y1":511,"x2":326,"y2":348,"x3":530,"y3":356,"x4":630,"y4":521,"x5":431,"y5":631}},{"label":"cubic_bezier","type":"cubic_bezier","coordinates":{"x1":701,"y1":545,"x2":790,"y2":383,"x3":993,"y3":390,"x4":1093,"y4":556,"x5":894,"y5":665}},{"label":"line","type":"line","coordinates":{"x1":247,"y1":855,"x2":336,"y2":692,"x3":540,"y3":699,"x4":640,"y4":865,"x5":441,"y5":975}},{"label":"point","type":"point","coordinates":{"x1":897,"y1":841}},{"label":"pixels","type":"pixels","id":0}]
...

Export XML files to JSON file for Create ML

You can export the annotations.json file for Training Object Detection Models in Create ML.

[{
    "image": "1.jpg",
    "annotations": [
    {
        "label": "bare feet",
        "coordinates":
        {
            "y": 838,
            "x": 393,
            "width": 62,
            "height": 118
        }
    },
    {
        "label": "bare feet",
        "coordinates":
        {
            "y": 881,
            "x": 392,
            "width": 51,
            "height": 102
        }
    }]
},
{
    "image": "2.jpg",
    "annotations": []
}]

Import CSV file to XML files

All objects in the CSV file are imported to the XML files in the current folder.

Be careful that importing would overwrite the current label files.

Before importing, we would back up the current label files.

Move images to another folder

Combining "Search images", you can move unlabeled images to the specified folder.

Copy images to another folder

Combining "Search images", you can copy searched images and label files to the specified folder.

Convert video to image frames

For "Video file", open a video file.

For "Output folder", open a folder to save the image frames.

For "Frames per a second", specify how many image frames to generate for a video second.

For "Image size max", both width and height would be less than or equal to the size.

Open the output folder to start labeling. To sort the image frames, set "Sort images" as Numeric.

Convert video to image frames

Replace label text for all images

Replace the label text for all images in the folder.

Resize all images and annotations

You can resize all images and label files in the current folder.

For "Image size max", both width and height would be less than or equal to the size.

If "Image size max" is empty, images are not resized and label files are resized to the same size as images.

Be careful that resizing would overwrite the current label files.

Before importing, we would back up the current label files.


Edit menu

Create box

Change the mode to "Create box".

Drag on the image to draw a box.

Or you can "Click 4 points when draw boxes".

When you finished dragging, the label dialog would open.


To change the box color, use the color picker at the top-right corner of the image.

The cross hairs would show up to label accurately.

You can hide the cross hairs using "Hide cross hairs".

For the cross hairs, the default color is being used.

To change the default color, deselect all boxes and change the color from main window.

To view the label name on each box, use "Show label on the box".


The label would be added to the label table on the right.

Drag the center of the box to move the box.

Drag one of the four corner points to transform the box.

To use aspect ratio, check on "Use aspect ratio when draw boxes".

Holding Shift key, you can draw or edit the box as square.

Create polygon, cubic bezier, line, and point

Change the mode to one of "Create polygon", "Create cubic bezier", "Create line", and "Create point".

Click to add points.

Press enter key to finish drawing.

Press escape key to cancel drawing.

For "Create point", when you add a point, the label dialog would open soon.

For "Create line", we do not close the polygon shape.

To show all edit points, use "Show all edit points".


When you right click on the point, edit menu would open.

"Add a point forward" and "Add a point backward" to add a point.

"Delete this point" to delete the point.

"Point size up" and "Point size down" to change the size of points.

"To polygon", "To cubic bezier", and "To line" change the polygon type.

Create box, rotated box, polygon, cubic bezier, line, and point

Label image

You can label the whole image without drawing boxes.

Label pixels

You can label pixels using brush and superpixel tools.

Superpixel size is used to adjust the segmentation size. When you label the small or narrow object, use small values.

Superpixel smoothness is used to adjust the segmentation boundary smoothness. When you label the detailed boundary, use small values.

Using right arrow and left arrow keys, you can change the superpixel size by 1px.

For large images, adjust both "Superpixel size max" and "Superpixel image size".


To hide superpixels, use "Hide superpixels" hotkey.

The superpixel calculation is being done in async and it would take seconds.

If superpixels are not shown, reselect the label on the label table to start the calculation.

If the superpixel calculation is slow, resize the image.

Label pixels with brush and superpixel tools

Brush size 1 means 1px in the image.

When you zoom in the image, the brush circle becomes larger.

Using right arrow and left arrow keys, you can change the brush size by 1px.

Erase is used to erase pixels.

To hide pixels, use "Hide pixels" hotkey.

The pixels image file is saved as "{image_file_name}_pixels{pixels_idx}.png" in the annotations folder.

Label pixels with brush and superpixel tools

Import pixels

For simplicity, we assume there is a mask image for the image.

The "pixels_idx" is 0.

So that the mask image file should be renamed as "{image_file_name}_pixels0.png".

Put this file into the current annotations folder.

For the xml file, just add the object tag which includes the label name and its pixels_idx.

This is an example of the XML file format.

<object>
    <name>person</name>
    <pixels>
        <id>0</id>
    </pixels>
</object>

Move

Change the mode to "Move".

Or you can switch between Create and Move mode holding space key when you are in Create mode.

Drag the box or the image to move its position.

For the image, mouse wheel can be used, too.

You can select multiple boxes and move them.

When you click on the box or the label, four corner points would appear.

Drag one of the four corner points to transform the box.

When you right click on the box or the label, edit menu would open.

"Focus" to quick zoom to the selected box, "Edit" to open the label dialog, "Duplicate" to duplicate the box, and "Delete" to delete the box.

When you double click on the box or the label, the label dialog would open.

To change the layer order of the box, drag the label in the label table upward or downward.

Rotate

Change the mode to "Rotate".

Drag up or down on the box to rotate the box.

You can select multiple boxes and rotate them.

Copy box

You can select multiple boxes and copy them.

Paste box

You can select multiple boxes and paste them across images.

Duplicate box

You can select multiple boxes and duplicate them.

Delete box

You can select multiple boxes and delete them.

Layer up and down

Change the layer order of the box.

Change object color

Change the object color using color picker.

Clear object color

Clear the object color to the default color.

Load labels from prev image

You can load labels from the previous image.


Search menu

Search images

When you open the folder, we read all xml files in the folder and start indexing asynchronously.

When you edit some label files, we track which label file was edited.

To index those edited label files, press "Indexing" on the search dialog.

To clear the search result, use "Clear search images".

You can use Wildcard(*), AND(&), OR(|), NOT(!), and more in the search text.

To search unlabeled images, use empty search text.

Settings for objects, attributes, hotkeys, and labeling fast

Reload the images folder.


Image menu

Change image brightness and contrast

Change the image brightness and contrast for dark images.

Change brightness and contrast

Rotate image

"Rotate image right" rotates the image by 90 degrees clockwise.

"Rotate image left" rotates the image by 90 degrees counter-clockwise.


Core ML menu

Load Core ML model

Your macOS version has to be GTE 10.13.

Load the Core ML model and show the description in metadata.

NSString *desctiption = model.modelDescription.metadata[MLModelDescriptionKey];

Process the image using Core ML model

Automatically label images using Core ML model

Your macOS version has to be GTE 10.13.

Process the image using the loaded Core ML model and add detected boxes or whole image label to the label table.


If you want to try with trained models, refer to these blogs.

Machine learning your first object detection

MobileNetV2 + SSDLite with Core ML


Before running the model, we check the user defined dictionary in the model.

NSDictionary *userDefinedDict = model.modelDescription.metadata[MLModelCreatorDefinedKey];

Check the class names.

NSArray *classesArray = [[NSArray alloc] init];
if([userDefinedDict objectForKey:(NSString*)kUserDefinedClasses] != nil){
    NSString *classesText = [userDefinedDict objectForKey:(NSString*)kUserDefinedClasses];
    classesArray = [classesText componentsSeparatedByString:(NSString*)kComma];
}

Check whether the model created by Turi Create so that box origin is the center of the box.

bool boxCentered = false;
if([userDefinedDict objectForKey:(NSString*)kTuriCreateVersion] != nil){
    boxCentered = true;
}

Check the confidence threshold.

float confidenceThreshold = kConfidenceThresholdDefault;
if([userDefinedDict objectForKey:(NSString*)kConfidenceThreshold] != nil){
    NSString *confidenceThresholdText = [userDefinedDict objectForKey:(NSString*)kConfidenceThreshold];
    confidenceThreshold = [confidenceThresholdText floatValue];
}

Check the Non-Maximum Suppression threshold.

float nmsThreshold = kNMSThresholdDefault;
if([userDefinedDict objectForKey:(NSString*)kNMSThreshold] != nil){
    NSString *nmsThresholdText = [userDefinedDict objectForKey:(NSString*)kNMSThreshold];
    nmsThreshold = [nmsThresholdText floatValue];
}

For the Non-Maximum Suppression, if your macOS is GTE 10.14, you can include the NMS in the Core ML model.

model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=True)

So that we can easily obtain the bounding box.

for(VNRecognizedObjectObservation *prediction in results){
    VNClassificationObservation *labels0 = [prediction.labels objectAtIndex:0];
    NSString *label = [NSString stringWithFormat:@"%@-%.2f", labels0.identifier, labels0.confidence];
    CGRect rect = prediction.boundingBox;
    ...

If your macOS is 10.13, you cannot include the NMS in the Core ML model.

model.export_coreml('MyDetector.mlmodel', include_non_maximum_suppression=False)

So that we have to calculate the NMS.

VNCoreMLFeatureValueObservation *coordinatesObservation  = results[0];
VNCoreMLFeatureValueObservation *confidenceObservation = results[1];
MLMultiArray *coordinates  = coordinatesObservation.featureValue.multiArrayValue;
MLMultiArray *confidence = confidenceObservation.featureValue.multiArrayValue;
NSMutableArray *unorderedPredictions = [[NSMutableArray alloc] init];
int numBoundingBoxes = [confidence.shape[0] intValue];
int numClasses = [confidence.shape[1] intValue];
for (int b = 0; b < numBoundingBoxes; b++) {
    float maxConfidence = 0.0;
    int maxIndex = 0;
    for (int c = 0; c < numClasses; c++) {
        NSNumber *confNumber = confidence[b * numClasses + c];
        double conf = [confNumber doubleValue];
        if (conf > maxConfidence) {
            maxConfidence = conf;
            maxIndex = c;
        }
    }
    if (maxConfidence > confidenceThreshold) {
        float x = [coordinates[b * 4 + 0] floatValue];
        float y = [coordinates[b * 4 + 1] floatValue];
        float w = [coordinates[b * 4 + 2] floatValue];
        float h = [coordinates[b * 4 + 3] floatValue];
        if(boxCentered){
            x -= w / 2;
            y -= h / 2;
        }
        CGRect rect = CGRectMake(x, y, w, h);
        Prediction *prediction = [[Prediction alloc] init];
        prediction.labelIndex = maxIndex;
        prediction.confidence = maxConfidence;
        prediction.boundingBox = rect;
        [unorderedPredictions addObject:prediction];
    }
}
...

For the details, refer to the Apple's document.

Deploying to Core ML

For the whole image label, we obtain only class name without the bounding box.

VNClassificationObservation *prediction = results.firstObject;
NSString *label = [NSString stringWithFormat:@"%@-%.2f", prediction.identifier, prediction.confidence];
...

Process all images using Core ML model

Your macOS version has to be GTE 10.13.

Process all images in the folder using the loaded Core ML model and replace the label files with detected boxes or whole image label.

Before running, we would back up the current label files.


View menu

Zoom in, Zoom out

Click a position to zoom in or zoom out.

Or using command + mouse wheel down/up, you can zoom in/out.

Zoom fit

Clear zoom.

Trackpad gestures

Slide two fingers to move the image.

Double-tap with two fingers for smart zoom.

Pinch with two fingers to zoom in or out.

Swipe left or right with three fingers to change the image.

System Preferences > Trackpad > More Gestures tab.

For "Swipe between pages", select "Swipe with two or three fingers" or "Swipe with three fingers".

For "Swipe between full-screen apps", select "Swipe left or right with four fingers".

Magic Mouse gestures

Slide a finger to move the image.

Double-tap with a finger for smart zoom.

System Preferences > Mouse > Point & Click tab > Smart zoom.

Swipe left or right with two fingers to change the image.

Focus box

You can quick zoom to the selected box.

Hide other boxes

Hide other boxes except the selected box. Toggle boxes alpha among 1.0, 0.3, and 0.0.

Hide cross hairs

Hide cross hairs when creating box.

Hide pixels

Toggle pixles alpha among 1.0, 0.5, and 0.0.

Hide superpixels

Toggle superpixels alpha among 1.0, 0.5, and 0.0.

Show depth channel

When the rgb image name is image_name.jpg, put the depth image as image_name_depth.png in the same folder.

Toggle depth image alpha among 0.0, 0.5, and 1.0.

Show label on the box

Show label name on each box.

Show memo from CSV file

This is useful when you edit specific label files based on some memo.

Here is the example of the CSV format.

image,text
26157415-386396491811054-4332315807620005888-n.jpg,stripes 2
26863805-1893959310636648-3775075699686113280-n.jpg,stiletto 3
...

Show loss list from CSV file

This is useful when you edit specific label files based on loss list from training.

Here is the example of the CSV format.

image,loss_color,loss_texture
20481859-332470370529981-2506007812560125952-n.jpg,0.254,0.050
20481859-332470370529981-2506007812560125952-n.jpg,0.205,0.011
...