I am going to implement Faster R-CNN for object detection in this tutorial, object detection is a computer vision and image processing technique that is used to locate instances of objects of a certain class (car, human or cat for example) in images and videos.
In object detection, we need to predict the class of objects and detect the bounding boxes surrounding objects, this means that a object detection model must do classification and regression. Faster R-CNN is a convolutional neural network (CNN or ConvNet) with a region proposal network (RPN). Faster R-CNN was invented by Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun, they improved on Fast R-CNN by creating an RPN with nearly cost-free region proposals.
Convolutional Neural Networks is designed to perform well with images, the arcitecture is inspired by the organization of the visual cortex in the human brain. A CNN reduces images into a form which is easier to process, without sacrificing predictive ability. An RPN is a convolutional network that predicts object boundaries and object scores at the same time for each individual position.
This code in this tutorial is written in Python and the code is adapted from Faster R-CNN for Open Images Dataset by Keras. I have downloaded a sample of images from Open Images V5 (you can use the latest version) and Keras Functional API is used to build models.
Download Images
You will need to download three .csv files to be able to download a sample of images from the open images project. Browse to Open Images Dataset and download class-descriptions-boxable.csv and train-annotations-bbox.csv, download train-images-boxable.csv from Figure Eight. I choosed to use only three categories (beer, laptop and goat) and 100 images for each class, 80 % is used for training and 20 % for testing/validation.
# Import libraries
import numpy as np
import random
import pandas as pd
from skimage import io # scikit-image
# The main entry point for this module
def main():
# Get classes from file
classes = pd.read_csv('C:\\DATA\\Python-data\\open-images-v5\\class-descriptions-boxable.csv', names=['label', 'name'])
# Get labels for Beer, Laptop and Goat
lbl_beer = classes.loc[classes['name']=='Beer', 'label'].iloc[0]
lbl_laptop = classes.loc[classes['name']=='Laptop', 'label'].iloc[0]
lbl_goat = classes.loc[classes['name']=='Goat', 'label'].iloc[0]
# Print labels
print('Beer: {0}, Laptop: {1}, Goat: {2}'.format(lbl_beer, lbl_laptop, lbl_goat))
# Load dataset of annotations
annotations = pd.read_csv('C:\\DATA\\Python-data\\open-images-v5\\train-annotations-bbox.csv')
# Get objects in annotations file
beer_bbox = annotations[annotations['LabelName']==lbl_beer]
laptop_bbox = annotations[annotations['LabelName']==lbl_laptop]
goat_bbox = annotations[annotations['LabelName']==lbl_goat]
# Print counts
print('There is {0} beer in the dataset.'.format(len(beer_bbox)))
print('There is {0} laptops in the dataset.'.format(len(laptop_bbox)))
print('There is {0} goats in the dataset.'.format(len(goat_bbox)))
# Get images
beer_images = np.unique(beer_bbox['ImageID'])
laptop_images = np.unique(laptop_bbox['ImageID'])
goat_images = np.unique(goat_bbox['ImageID'])
# Print count of images
print('There are {0} images which contain beer'.format(len(beer_images)))
print('There are {0} images which contain laptops'.format(len(laptop_images)))
print('There are {0} images which contain goats'.format(len(goat_images)))
# Variables
n = 100
train_size = int(n*0.8)
base_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\'
# Pick n of each object
beers_subset = beer_images[:n]
laptops_subset = laptop_images[:n]
goats_subset = goat_images[:n]
# Load dataset of images
images = pd.read_csv('C:\\DATA\\Python-data\\open-images-v5\\train-images-boxable.csv')
# Get 3 balanced subsets of images
beer_imgs_subset = [images[images['image_name']==name+'.jpg'] for name in beers_subset]
laptop_imgs_subset = [images[images['image_name']==name+'.jpg'] for name in laptops_subset]
goat_imgs_subset = [images[images['image_name']==name+'.jpg'] for name in goats_subset]
# Download images
for i in range(n):
beer_img = io.imread(beer_imgs_subset[i]['image_url'].values[0])
laptop_img = io.imread(laptop_imgs_subset[i]['image_url'].values[0])
goat_img = io.imread(goat_imgs_subset[i]['image_url'].values[0])
if(i < train_size):
io.imsave(base_path + 'train\\' + beer_imgs_subset[i]['image_name'].values[0], beer_img)
io.imsave(base_path + 'train\\' + laptop_imgs_subset[i]['image_name'].values[0], laptop_img)
io.imsave(base_path + 'train\\' + goat_imgs_subset[i]['image_name'].values[0], goat_img)
else:
io.imsave(base_path + 'test\\' + beer_imgs_subset[i]['image_name'].values[0], beer_img)
io.imsave(base_path + 'test\\' + laptop_imgs_subset[i]['image_name'].values[0], laptop_img)
io.imsave(base_path + 'test\\' + goat_imgs_subset[i]['image_name'].values[0], goat_img)
# Tell python to run main method
if __name__ == "__main__": main()Create Annotations
I have downloaded images and now it is time to create annotation files, one for training and one for testing. Annotation files tells us about the location of objects in images. These files show filename, bounding box and class name for each file and object. One image can include many objects.
# Import libraries
import os
import cv2
import pandas as pd
# Create an annotations file
def create_annotation_file(base_path, annotations, classes, labels, mode='train'):
# Create a dataframe
df = pd.DataFrame(columns=['filename', 'xmin', 'ymin', 'xmax', 'ymax', 'class'])
# Get all images
images = os.listdir(base_path + mode)
# Loop images
for i in range(len(images)):
img_name = images[i]
img_id = img_name.split('.')[0]
tmp_df = annotations[annotations['ImageID']==img_id]
img = cv2.imread((base_path + mode + '\\' + img_name))
height, width = img.shape[:2]
for index, row in tmp_df.iterrows():
label_name = row['LabelName']
for i in range(len(labels)):
if label_name == labels[i]:
df = df.append({
'filename': mode + '\\' + img_name,
'xmin': int(row['XMin'] * width),
'ymin': int(row['YMin'] * height),
'xmax': int(row['XMax'] * width),
'ymax': int(row['YMax'] * height),
'class': classes[i]},
ignore_index=True)
# Save annotations file
df.to_csv(base_path + mode + '_annotations.csv', index=None, header=False)
# The main entry point for this module
def main():
# Variables
base_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\'
classes = ['Beer', 'Laptop', 'Goat']
labels = ['/m/01599', '/m/01c648', '/m/03fwl']
# Load dataset of annotations
annotations = pd.read_csv('C:\\DATA\\Python-data\\open-images-v5\\train-annotations-bbox.csv')
# Create annotation files for train and test
create_annotation_file(base_path, annotations, classes, labels, mode='train')
create_annotation_file(base_path, annotations, classes, labels, mode='test')
# Tell python to run main method
if __name__ == "__main__": main()Common Methods
This module includes a lot of handy methods that is used in training and evaluation, it has a class for configuration and some image methods.
# Import libraries
import sys
import math
import cv2
import copy
import random
import keras
import numpy as np
import tensorflow as tf
# Minimum tensorflow version
MINIMUM_TF_VERSION = 1, 12, 0
# Restrict tensortlow to only use the one GPU
def setup_gpu(gpu_id):
if tf_version_ok((2, 0, 0)):
if gpu_id == 'cpu' or gpu_id == -1:
tf.config.experimental.set_visible_devices([], 'GPU')
return
# Get all gpus
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
# Restrict TensorFlow to only use the first GPU.
try:
# Currently, memory growth needs to be the same across GPUs.
for gpu in gpus:
#tf.config.experimental.set_virtual_device_configuration(gpu, [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=4096)])
tf.config.experimental.set_memory_growth(gpu, True)
# Use only the selcted gpu.
tf.config.experimental.set_visible_devices(gpus[gpu_id], 'GPU')
except RuntimeError as e:
# Visible devices must be set before GPUs have been initialized.
print(e)
logical_gpus = tf.config.experimental.list_logical_devices('GPU')
print(len(gpus), "Physical GPUs,", len(logical_gpus), "Logical GPUs")
else:
import os
if gpu_id == 'cpu' or gpu_id == -1:
os.environ['CUDA_VISIBLE_DEVICES'] = ""
return
os.environ['CUDA_VISIBLE_DEVICES'] = str(gpu_id)
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.5
config.gpu_options.allow_growth = True
tf.keras.backend.set_session(tf.Session(config=config))
# Get the Tensorflow version
def tf_version():
return tuple(map(int, tf.version.VERSION.split('-')[0].split('.')))
# Check if the current Tensorflow version is higher than the minimum version
def tf_version_ok(minimum_tf_version=MINIMUM_TF_VERSION):
return tf_version() >= minimum_tf_version
# This class is used for configuration
class Config:
# Initializes the class
def __init__(self):
# Print the process or not
self.verbose = True
# Settings for data augmentation
self.use_horizontal_flips = False
self.use_vertical_flips = False
self.rot_90 = False
# Anchor box scales
# Note that if im_size is smaller, anchor_box_scales should be scaled
# Original anchor_box_scales in the paper is [128, 256, 512]
self.anchor_box_scales = [64, 128, 256]
# Anchor box ratios
self.anchor_box_ratios = [[1, 1], [1./math.sqrt(2), 2./math.sqrt(2)], [2./math.sqrt(2), 1./math.sqrt(2)]]
# Size to resize the smallest side of the image
# Original setting in paper is 600. Set to 300 in here to save training time
self.im_size = 300
# image channel-wise mean to subtract
self.img_channel_mean = [103.939, 116.779, 123.68]
self.img_scaling_factor = 1.0
# number of ROIs at once
self.num_rois = 4
# stride at the RPN (this depends on the network configuration)
self.rpn_stride = 16
# scaling the stdev
self.std_scaling = 4.0
self.classifier_regr_std = [8.0, 8.0, 4.0, 4.0]
# overlaps for RPN
self.rpn_min_overlap = 0.3
self.rpn_max_overlap = 0.7
# overlaps for classifier ROIs
self.classifier_min_overlap = 0.1
self.classifier_max_overlap = 0.5
# Loss function settings
self.lambda_rpn_regr = 1.0
self.lambda_rpn_class = 1.0
self.lambda_cls_regr = 1.0
self.lambda_cls_class = 1.0
self.epsilon = 1e-4
# Paths
self.annotations_file_path = None
self.img_base_path = None
self.pretrained_model_path = None
self.model_path = None
self.records_path = None
# Parse the data from annotation file
def get_data(config):
# Make sure that there is configurations
if config == None:
config = Config()
# Variables
found_bg = False
all_imgs = {}
classes_count = {}
class_mapping = {}
visualise = True
i = 1
with open(config.annotations_file_path,'r') as f:
print('Parsing annotation file')
for line in f:
# Print process
sys.stdout.write('\r'+'idx=' + str(i))
i += 1
line_split = line.strip().split(',')
# Make sure the info saved in annotation file matching the format (path_filename, x1, y1, x2, y2, class_name)
(filename,x1,y1,x2,y2,class_name) = line_split
if class_name not in classes_count:
classes_count[class_name] = 1
else:
classes_count[class_name] += 1
if class_name not in class_mapping:
if class_name == 'bg' and found_bg == False:
print('Found class name with special name bg. Will be treated as a background region (this is usually for hard negative mining).')
found_bg = True
class_mapping[class_name] = len(class_mapping)
if filename not in all_imgs:
all_imgs[filename] = {}
img = cv2.imread(config.img_base_path + filename)
(rows,cols) = img.shape[:2]
all_imgs[filename]['filepath'] = filename
all_imgs[filename]['width'] = cols
all_imgs[filename]['height'] = rows
all_imgs[filename]['bboxes'] = []
all_imgs[filename]['bboxes'].append({'class': class_name, 'x1': int(x1), 'x2': int(x2), 'y1': int(y1), 'y2': int(y2)})
all_data = []
for key in all_imgs:
all_data.append(all_imgs[key])
# make sure the bg class is last in the list
if found_bg:
if class_mapping['bg'] != len(class_mapping) - 1:
key_to_switch = [key for key in class_mapping.keys() if class_mapping[key] == len(class_mapping)-1][0]
val_to_switch = class_mapping['bg']
class_mapping['bg'] = len(class_mapping) - 1
class_mapping[key_to_switch] = val_to_switch
return all_data, classes_count, class_mapping
# Loss function for rpn regression
def rpn_loss_regr(num_anchors, config=None):
# Make sure that there is configurations
if config == None:
config = Config()
def rpn_loss_regr_fixed_num(y_true, y_pred):
# x is the difference between true value and predicted vaue
x = y_true[:, :, :, 4 * num_anchors:] - y_pred
# absolute value of x
x_abs = keras.backend.abs(x)
# If x_abs <= 1.0, x_bool = 1
x_bool = keras.backend.cast(keras.backend.less_equal(x_abs, 1.0), tf.float32)
return config.lambda_rpn_regr * keras.backend.sum(
y_true[:, :, :, :4 * num_anchors] * (x_bool * (0.5 * x * x) + (1 - x_bool) * (x_abs - 0.5))) / keras.backend.sum(config.epsilon + y_true[:, :, :, :4 * num_anchors])
return rpn_loss_regr_fixed_num
# Loss function for rpn classification
def rpn_loss_cls(num_anchors, config=None):
# Make sure that there is configurations
if config == None:
config = Config()
def rpn_loss_cls_fixed_num(y_true, y_pred):
return config.lambda_rpn_class * keras.backend.sum(y_true[:, :, :, :num_anchors] * keras.backend.binary_crossentropy(y_pred[:, :, :, :], y_true[:, :, :, num_anchors:])) / keras.backend.sum(config.epsilon + y_true[:, :, :, :num_anchors])
# Return a function
return rpn_loss_cls_fixed_num
# Loss function for rpn regression
def class_loss_regr(num_classes, config=None):
# Make sure that there is configurations
if config == None:
config = Config()
def class_loss_regr_fixed_num(y_true, y_pred):
x = y_true[:, :, 4*num_classes:] - y_pred
x_abs = keras.backend.abs(x)
x_bool = keras.backend.cast(keras.backend.less_equal(x_abs, 1.0), 'float32')
return config.lambda_cls_regr * keras.backend.sum(y_true[:, :, :4*num_classes] * (x_bool * (0.5 * x * x) + (1 - x_bool) * (x_abs - 0.5))) / keras.backend.sum(config.epsilon + y_true[:, :, :4*num_classes])
# Return a function
return class_loss_regr_fixed_num
# Loss function for classification y_true, y_pred,
def class_loss_cls(config=None):
# Make sure that there is configurations
if config == None:
config = Config()
def class_loss_cls_fixed_num(y_true, y_pred):
return config.lambda_cls_class * keras.backend.mean(keras.losses.categorical_crossentropy(y_true[0, :, :], y_pred[0, :, :]))
# Return a function
return class_loss_cls_fixed_num
# Get image output length
def get_img_output_length(width, height):
def get_output_length(input_length):
return input_length//16
# Return output length for width and height
return get_output_length(width), get_output_length(height)
# Get a new image size
def get_new_img_size(width, height, img_min_side=300):
if width <= height:
f = float(img_min_side) / width
resized_height = int(f * height)
resized_width = img_min_side
else:
f = float(img_min_side) / height
resized_width = int(f * width)
resized_height = img_min_side
# Return resized width and height
return resized_width, resized_height
# Augument
def augment(img_data, config, augment=True):
assert 'filepath' in img_data
assert 'bboxes' in img_data
assert 'width' in img_data
assert 'height' in img_data
img_data_aug = copy.deepcopy(img_data)
img = cv2.imread(config.img_base_path + img_data_aug['filepath'])
if augment:
rows, cols = img.shape[:2]
if config.use_horizontal_flips and np.random.randint(0, 2) == 0:
img = cv2.flip(img, 1)
for bbox in img_data_aug['bboxes']:
x1 = bbox['x1']
x2 = bbox['x2']
bbox['x2'] = cols - x1
bbox['x1'] = cols - x2
if config.use_vertical_flips and np.random.randint(0, 2) == 0:
img = cv2.flip(img, 0)
for bbox in img_data_aug['bboxes']:
y1 = bbox['y1']
y2 = bbox['y2']
bbox['y2'] = rows - y1
bbox['y1'] = rows - y2
if config.rot_90:
angle = np.random.choice([0,90,180,270],1)[0]
if angle == 270:
img = np.transpose(img, (1,0,2))
img = cv2.flip(img, 0)
elif angle == 180:
img = cv2.flip(img, -1)
elif angle == 90:
img = np.transpose(img, (1,0,2))
img = cv2.flip(img, 1)
elif angle == 0:
pass
for bbox in img_data_aug['bboxes']:
x1 = bbox['x1']
x2 = bbox['x2']
y1 = bbox['y1']
y2 = bbox['y2']
if angle == 270:
bbox['x1'] = y1
bbox['x2'] = y2
bbox['y1'] = cols - x2
bbox['y2'] = cols - x1
elif angle == 180:
bbox['x2'] = cols - x1
bbox['x1'] = cols - x2
bbox['y2'] = rows - y1
bbox['y1'] = rows - y2
elif angle == 90:
bbox['x1'] = rows - y2
bbox['x2'] = rows - y1
bbox['y1'] = x1
bbox['y2'] = x2
elif angle == 0:
pass
img_data_aug['width'] = img.shape[1]
img_data_aug['height'] = img.shape[0]
return img_data_aug, img
# Yield the ground-truth anchors as Y (labels)
def get_anchor_gt(all_img_data, config, mode='train'):
while True:
for img_data in all_img_data:
try:
# read in image, and optionally add augmentation
if mode == 'train':
img_data_aug, x_img = augment(img_data, config, augment=True)
else:
img_data_aug, x_img = augment(img_data, config, augment=False)
(width, height) = (img_data_aug['width'], img_data_aug['height'])
(rows, cols, _) = x_img.shape
assert cols == width
assert rows == height
# get image dimensions for resizing
(resized_width, resized_height) = get_new_img_size(width, height, config.im_size)
# resize the image so that smalles side is length = 300px
x_img = cv2.resize(x_img, (resized_width, resized_height), interpolation=cv2.INTER_CUBIC)
debug_img = x_img.copy()
try:
y_rpn_cls, y_rpn_regr, num_pos = calc_rpn(config, img_data_aug, width, height, resized_width, resized_height)
except Exception as e:
print(e)
continue
# Zero-center by mean pixel, and preprocess image
x_img = x_img[:,:, (2, 1, 0)] # BGR -> RGB
x_img = x_img.astype(np.float32)
x_img[:, :, 0] -= config.img_channel_mean[0]
x_img[:, :, 1] -= config.img_channel_mean[1]
x_img[:, :, 2] -= config.img_channel_mean[2]
x_img /= config.img_scaling_factor
x_img = np.transpose(x_img, (2, 0, 1))
x_img = np.expand_dims(x_img, axis=0)
y_rpn_regr[:, y_rpn_regr.shape[1]//2:, :, :] *= config.std_scaling
x_img = np.transpose(x_img, (0, 2, 3, 1))
y_rpn_cls = np.transpose(y_rpn_cls, (0, 2, 3, 1))
y_rpn_regr = np.transpose(y_rpn_regr, (0, 2, 3, 1))
yield np.copy(x_img), [np.copy(y_rpn_cls), np.copy(y_rpn_regr)], img_data_aug, debug_img, num_pos
except Exception as e:
print(e)
continue
# Calculate union
def union(au, bu, area_intersection):
area_a = (au[2] - au[0]) * (au[3] - au[1])
area_b = (bu[2] - bu[0]) * (bu[3] - bu[1])
area_union = area_a + area_b - area_intersection
return area_union
# Calculate intersection
def intersection(ai, bi):
x = max(ai[0], bi[0])
y = max(ai[1], bi[1])
w = min(ai[2], bi[2]) - x
h = min(ai[3], bi[3]) - y
if w < 0 or h < 0:
return 0
return w*h
# Calculate IOU
def iou(a, b):
# a and b should be (x1,y1,x2,y2)
if a[0] >= a[2] or a[1] >= a[3] or b[0] >= b[2] or b[1] >= b[3]:
return 0.0
area_i = intersection(a, b)
area_u = union(a, b, area_i)
return float(area_i) / float(area_u + 1e-6)
# Calculate the rpn for all anchors
def calc_rpn(config, img_data, width, height, resized_width, resized_height):
downscale = float(config.rpn_stride)
anchor_sizes = config.anchor_box_scales
anchor_ratios = config.anchor_box_ratios
num_anchors = len(anchor_sizes) * len(anchor_ratios)
# calculate the output map size based on the network architecture
(output_width, output_height) = get_img_output_length(resized_width, resized_height)
n_anchratios = len(anchor_ratios)
# initialise empty output objectives
y_rpn_overlap = np.zeros((output_height, output_width, num_anchors))
y_is_box_valid = np.zeros((output_height, output_width, num_anchors))
y_rpn_regr = np.zeros((output_height, output_width, num_anchors * 4))
num_bboxes = len(img_data['bboxes'])
num_anchors_for_bbox = np.zeros(num_bboxes).astype(int)
best_anchor_for_bbox = -1*np.ones((num_bboxes, 4)).astype(int)
best_iou_for_bbox = np.zeros(num_bboxes).astype(np.float32)
best_x_for_bbox = np.zeros((num_bboxes, 4)).astype(int)
best_dx_for_bbox = np.zeros((num_bboxes, 4)).astype(np.float32)
# get the GT box coordinates, and resize to account for image resizing
gta = np.zeros((num_bboxes, 4))
for bbox_num, bbox in enumerate(img_data['bboxes']):
# get the GT box coordinates, and resize to account for image resizing
gta[bbox_num, 0] = bbox['x1'] * (resized_width / float(width))
gta[bbox_num, 1] = bbox['x2'] * (resized_width / float(width))
gta[bbox_num, 2] = bbox['y1'] * (resized_height / float(height))
gta[bbox_num, 3] = bbox['y2'] * (resized_height / float(height))
# rpn ground truth
for anchor_size_idx in range(len(anchor_sizes)):
for anchor_ratio_idx in range(n_anchratios):
anchor_x = anchor_sizes[anchor_size_idx] * anchor_ratios[anchor_ratio_idx][0]
anchor_y = anchor_sizes[anchor_size_idx] * anchor_ratios[anchor_ratio_idx][1]
for ix in range(output_width):
# x-coordinates of the current anchor box
x1_anc = downscale * (ix + 0.5) - anchor_x / 2
x2_anc = downscale * (ix + 0.5) + anchor_x / 2
# ignore boxes that go across image boundaries
if x1_anc < 0 or x2_anc > resized_width:
continue
for jy in range(output_height):
# y-coordinates of the current anchor box
y1_anc = downscale * (jy + 0.5) - anchor_y / 2
y2_anc = downscale * (jy + 0.5) + anchor_y / 2
# ignore boxes that go across image boundaries
if y1_anc < 0 or y2_anc > resized_height:
continue
# bbox_type indicates whether an anchor should be a target
# Initialize with 'negative'
bbox_type = 'neg'
# this is the best IOU for the (x,y) coord and the current anchor
# note that this is different from the best IOU for a GT bbox
best_iou_for_loc = 0.0
for bbox_num in range(num_bboxes):
# get IOU of the current GT box and the current anchor box
curr_iou = iou([gta[bbox_num, 0], gta[bbox_num, 2], gta[bbox_num, 1], gta[bbox_num, 3]], [x1_anc, y1_anc, x2_anc, y2_anc])
# calculate the regression targets if they will be needed
if curr_iou > best_iou_for_bbox[bbox_num] or curr_iou > config.rpn_max_overlap:
cx = (gta[bbox_num, 0] + gta[bbox_num, 1]) / 2.0
cy = (gta[bbox_num, 2] + gta[bbox_num, 3]) / 2.0
cxa = (x1_anc + x2_anc)/2.0
cya = (y1_anc + y2_anc)/2.0
# x,y are the center point of ground-truth bbox
# xa,ya are the center point of anchor bbox (xa=downscale * (ix + 0.5); ya=downscale * (iy+0.5))
# w,h are the width and height of ground-truth bbox
# wa,ha are the width and height of anchor bboxe
# tx = (x - xa) / wa
# ty = (y - ya) / ha
# tw = log(w / wa)
# th = log(h / ha)
tx = (cx - cxa) / (x2_anc - x1_anc)
ty = (cy - cya) / (y2_anc - y1_anc)
tw = np.log((gta[bbox_num, 1] - gta[bbox_num, 0]) / (x2_anc - x1_anc))
th = np.log((gta[bbox_num, 3] - gta[bbox_num, 2]) / (y2_anc - y1_anc))
if img_data['bboxes'][bbox_num]['class'] != 'bg':
# all GT boxes should be mapped to an anchor box, so we keep track of which anchor box was best
if curr_iou > best_iou_for_bbox[bbox_num]:
best_anchor_for_bbox[bbox_num] = [jy, ix, anchor_ratio_idx, anchor_size_idx]
best_iou_for_bbox[bbox_num] = curr_iou
best_x_for_bbox[bbox_num,:] = [x1_anc, x2_anc, y1_anc, y2_anc]
best_dx_for_bbox[bbox_num,:] = [tx, ty, tw, th]
# we set the anchor to positive if the IOU is >0.7 (it does not matter if there was another better box, it just indicates overlap)
if curr_iou > config.rpn_max_overlap:
bbox_type = 'pos'
num_anchors_for_bbox[bbox_num] += 1
# we update the regression layer target if this IOU is the best for the current (x,y) and anchor position
if curr_iou > best_iou_for_loc:
best_iou_for_loc = curr_iou
best_regr = (tx, ty, tw, th)
# if the IOU is >0.3 and <0.7, it is ambiguous and no included in the objective
if config.rpn_min_overlap < curr_iou < config.rpn_max_overlap:
# gray zone between neg and pos
if bbox_type != 'pos':
bbox_type = 'neutral'
# turn on or off outputs depending on IOUs
if bbox_type == 'neg':
y_is_box_valid[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 1
y_rpn_overlap[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 0
elif bbox_type == 'neutral':
y_is_box_valid[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 0
y_rpn_overlap[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 0
elif bbox_type == 'pos':
y_is_box_valid[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 1
y_rpn_overlap[jy, ix, anchor_ratio_idx + n_anchratios * anchor_size_idx] = 1
start = 4 * (anchor_ratio_idx + n_anchratios * anchor_size_idx)
y_rpn_regr[jy, ix, start:start+4] = best_regr
# we ensure that every bbox has at least one positive RPN region
for idx in range(num_anchors_for_bbox.shape[0]):
if num_anchors_for_bbox[idx] == 0:
# no box with an IOU greater than zero ...
if best_anchor_for_bbox[idx, 0] == -1:
continue
y_is_box_valid[
best_anchor_for_bbox[idx,0], best_anchor_for_bbox[idx,1], best_anchor_for_bbox[idx,2] + n_anchratios *
best_anchor_for_bbox[idx,3]] = 1
y_rpn_overlap[
best_anchor_for_bbox[idx,0], best_anchor_for_bbox[idx,1], best_anchor_for_bbox[idx,2] + n_anchratios *
best_anchor_for_bbox[idx,3]] = 1
start = 4 * (best_anchor_for_bbox[idx,2] + n_anchratios * best_anchor_for_bbox[idx,3])
y_rpn_regr[
best_anchor_for_bbox[idx,0], best_anchor_for_bbox[idx,1], start:start+4] = best_dx_for_bbox[idx, :]
y_rpn_overlap = np.transpose(y_rpn_overlap, (2, 0, 1))
y_rpn_overlap = np.expand_dims(y_rpn_overlap, axis=0)
y_is_box_valid = np.transpose(y_is_box_valid, (2, 0, 1))
y_is_box_valid = np.expand_dims(y_is_box_valid, axis=0)
y_rpn_regr = np.transpose(y_rpn_regr, (2, 0, 1))
y_rpn_regr = np.expand_dims(y_rpn_regr, axis=0)
pos_locs = np.where(np.logical_and(y_rpn_overlap[0, :, :, :] == 1, y_is_box_valid[0, :, :, :] == 1))
neg_locs = np.where(np.logical_and(y_rpn_overlap[0, :, :, :] == 0, y_is_box_valid[0, :, :, :] == 1))
num_pos = len(pos_locs[0])
# one issue is that the RPN has many more negative than positive regions, so we turn off some of the negative
# regions. We also limit it to 256 regions.
num_regions = 256
if len(pos_locs[0]) > num_regions/2:
val_locs = random.sample(range(len(pos_locs[0])), len(pos_locs[0]) - num_regions/2)
y_is_box_valid[0, pos_locs[0][val_locs], pos_locs[1][val_locs], pos_locs[2][val_locs]] = 0
num_pos = num_regions/2
if len(neg_locs[0]) + num_pos > num_regions:
val_locs = random.sample(range(len(neg_locs[0])), len(neg_locs[0]) - num_pos)
y_is_box_valid[0, neg_locs[0][val_locs], neg_locs[1][val_locs], neg_locs[2][val_locs]] = 0
y_rpn_cls = np.concatenate([y_is_box_valid, y_rpn_overlap], axis=1)
y_rpn_regr = np.concatenate([np.repeat(y_rpn_overlap, 4, axis=1), y_rpn_regr], axis=1)
return np.copy(y_rpn_cls), np.copy(y_rpn_regr), num_pos
# Non max suppression: http://www.pyimagesearch.com/2015/02/16/faster-non-maximum-suppression-python/
def non_max_suppression_fast(boxes, probs, overlap_thresh=0.9, max_boxes=300):
# if there are no boxes, return an empty list
if len(boxes) == 0:
return []
# grab the coordinates of the bounding boxes
x1 = boxes[:, 0]
y1 = boxes[:, 1]
x2 = boxes[:, 2]
y2 = boxes[:, 3]
np.testing.assert_array_less(x1, x2)
np.testing.assert_array_less(y1, y2)
# if the bounding boxes integers, convert them to floats --
# this is important since we'll be doing a bunch of divisions
if boxes.dtype.kind == "i":
boxes = boxes.astype("float")
# initialize the list of picked indexes
pick = []
# calculate the areas
area = (x2 - x1) * (y2 - y1)
# sort the bounding boxes
idxs = np.argsort(probs)
# keep looping while some indexes still remain in the indexes
# list
while len(idxs) > 0:
# grab the last index in the indexes list and add the
# index value to the list of picked indexes
last = len(idxs) - 1
i = idxs[last]
pick.append(i)
# find the intersection
xx1_int = np.maximum(x1[i], x1[idxs[:last]])
yy1_int = np.maximum(y1[i], y1[idxs[:last]])
xx2_int = np.minimum(x2[i], x2[idxs[:last]])
yy2_int = np.minimum(y2[i], y2[idxs[:last]])
ww_int = np.maximum(0, xx2_int - xx1_int)
hh_int = np.maximum(0, yy2_int - yy1_int)
area_int = ww_int * hh_int
# find the union
area_union = area[i] + area[idxs[:last]] - area_int
# compute the ratio of overlap
overlap = area_int/(area_union + 1e-6)
# delete all indexes from the index list that have
idxs = np.delete(idxs, np.concatenate(([last],
np.where(overlap > overlap_thresh)[0])))
if len(pick) >= max_boxes:
break
# return only the bounding boxes that were picked using the integer data type
boxes = boxes[pick].astype("int")
probs = probs[pick]
return boxes, probs
# Converts from (x1,y1,x2,y2) to (x,y,w,h) format
def calc_iou(R, img_data, config, class_mapping):
bboxes = img_data['bboxes']
(width, height) = (img_data['width'], img_data['height'])
# get image dimensions for resizing
(resized_width, resized_height) = get_new_img_size(width, height, config.im_size)
gta = np.zeros((len(bboxes), 4))
for bbox_num, bbox in enumerate(bboxes):
# get the GT box coordinates, and resize to account for image resizing
# gta[bbox_num, 0] = (40 * (600 / 800)) / 16 = int(round(1.875)) = 2 (x in feature map)
gta[bbox_num, 0] = int(round(bbox['x1'] * (resized_width / float(width))/config.rpn_stride))
gta[bbox_num, 1] = int(round(bbox['x2'] * (resized_width / float(width))/config.rpn_stride))
gta[bbox_num, 2] = int(round(bbox['y1'] * (resized_height / float(height))/config.rpn_stride))
gta[bbox_num, 3] = int(round(bbox['y2'] * (resized_height / float(height))/config.rpn_stride))
x_roi = []
y_class_num = []
y_class_regr_coords = []
y_class_regr_label = []
IoUs = [] # for debugging only
# R.shape[0]: number of bboxes (=300 from non_max_suppression)
for ix in range(R.shape[0]):
(x1, y1, x2, y2) = R[ix, :]
x1 = int(round(x1))
y1 = int(round(y1))
x2 = int(round(x2))
y2 = int(round(y2))
best_iou = 0.0
best_bbox = -1
# Iterate through all the ground-truth bboxes to calculate the iou
for bbox_num in range(len(bboxes)):
curr_iou = iou([gta[bbox_num, 0], gta[bbox_num, 2], gta[bbox_num, 1], gta[bbox_num, 3]], [x1, y1, x2, y2])
# Find out the corresponding ground-truth bbox_num with larget iou
if curr_iou > best_iou:
best_iou = curr_iou
best_bbox = bbox_num
if best_iou < config.classifier_min_overlap:
continue
else:
w = x2 - x1
h = y2 - y1
x_roi.append([x1, y1, w, h])
IoUs.append(best_iou)
if config.classifier_min_overlap <= best_iou < config.classifier_max_overlap:
# hard negative example
cls_name = 'bg'
elif config.classifier_max_overlap <= best_iou:
cls_name = bboxes[best_bbox]['class']
cxg = (gta[best_bbox, 0] + gta[best_bbox, 1]) / 2.0
cyg = (gta[best_bbox, 2] + gta[best_bbox, 3]) / 2.0
cx = x1 + w / 2.0
cy = y1 + h / 2.0
tx = (cxg - cx) / float(w)
ty = (cyg - cy) / float(h)
tw = np.log((gta[best_bbox, 1] - gta[best_bbox, 0]) / float(w))
th = np.log((gta[best_bbox, 3] - gta[best_bbox, 2]) / float(h))
else:
print('roi = {}'.format(best_iou))
raise RuntimeError
class_num = class_mapping[cls_name]
class_label = len(class_mapping) * [0]
class_label[class_num] = 1
y_class_num.append(copy.deepcopy(class_label))
coords = [0] * 4 * (len(class_mapping) - 1)
labels = [0] * 4 * (len(class_mapping) - 1)
if cls_name != 'bg':
label_pos = 4 * class_num
sx, sy, sw, sh = config.classifier_regr_std
coords[label_pos:4+label_pos] = [sx*tx, sy*ty, sw*tw, sh*th]
labels[label_pos:4+label_pos] = [1, 1, 1, 1]
y_class_regr_coords.append(copy.deepcopy(coords))
y_class_regr_label.append(copy.deepcopy(labels))
else:
y_class_regr_coords.append(copy.deepcopy(coords))
y_class_regr_label.append(copy.deepcopy(labels))
if len(x_roi) == 0:
return None, None, None, None
# bboxes that iou > config.classifier_min_overlap for all gt bboxes in 300 non_max_suppression bboxes
X = np.array(x_roi)
# one hot code for bboxes from above => x_roi (X)
Y1 = np.array(y_class_num)
# corresponding labels and corresponding gt bboxes
Y2 = np.concatenate([np.array(y_class_regr_label),np.array(y_class_regr_coords)],axis=1)
return np.expand_dims(X, axis=0), np.expand_dims(Y1, axis=0), np.expand_dims(Y2, axis=0), IoUs
# Formats the image size based on config
def format_img_size(img, config):
img_min_side = float(config.im_size)
(height,width,_) = img.shape
if width <= height:
ratio = img_min_side/width
new_height = int(ratio * height)
new_width = int(img_min_side)
else:
ratio = img_min_side/height
new_width = int(ratio * width)
new_height = int(img_min_side)
img = cv2.resize(img, (new_width, new_height), interpolation=cv2.INTER_CUBIC)
return img, ratio
# Formats the image channels based on config
def format_img_channels(img, config):
img = img[:, :, (2, 1, 0)]
img = img.astype(np.float32)
img[:, :, 0] -= config.img_channel_mean[0]
img[:, :, 1] -= config.img_channel_mean[1]
img[:, :, 2] -= config.img_channel_mean[2]
img /= config.img_scaling_factor
img = np.transpose(img, (2, 0, 1))
img = np.expand_dims(img, axis=0)
return img
# Formats an image for model prediction based on config
def format_img(img, config):
img, ratio = format_img_size(img, config)
img = format_img_channels(img, config)
return img, ratio
# Method to transform the coordinates of the bounding box to its original size
def get_real_coordinates(ratio, x1, y1, x2, y2):
real_x1 = int(round(x1 // ratio))
real_y1 = int(round(y1 // ratio))
real_x2 = int(round(x2 // ratio))
real_y2 = int(round(y2 // ratio))
return (real_x1, real_y1, real_x2 ,real_y2)
# Get map
def get_map(pred, gt, f):
T = {}
P = {}
fx, fy = f
for bbox in gt:
bbox['bbox_matched'] = False
pred_probs = np.array([s['prob'] for s in pred])
box_idx_sorted_by_prob = np.argsort(pred_probs)[::-1]
for box_idx in box_idx_sorted_by_prob:
pred_box = pred[box_idx]
pred_class = pred_box['class']
pred_x1 = pred_box['x1']
pred_x2 = pred_box['x2']
pred_y1 = pred_box['y1']
pred_y2 = pred_box['y2']
pred_prob = pred_box['prob']
if pred_class not in P:
P[pred_class] = []
T[pred_class] = []
P[pred_class].append(pred_prob)
found_match = False
for gt_box in gt:
gt_class = gt_box['class']
gt_x1 = gt_box['x1']/fx
gt_x2 = gt_box['x2']/fx
gt_y1 = gt_box['y1']/fy
gt_y2 = gt_box['y2']/fy
gt_seen = gt_box['bbox_matched']
if gt_class != pred_class:
continue
if gt_seen:
continue
iou_map = iou((pred_x1, pred_y1, pred_x2, pred_y2), (gt_x1, gt_y1, gt_x2, gt_y2))
if iou_map >= 0.5:
found_match = True
gt_box['bbox_matched'] = True
break
else:
continue
T[pred_class].append(int(found_match))
for gt_box in gt:
if not gt_box['bbox_matched']:# and not gt_box['difficult']:
if gt_box['class'] not in P:
P[gt_box['class']] = []
T[gt_box['class']] = []
T[gt_box['class']].append(1)
P[gt_box['class']].append(0)
#import pdb
#pdb.set_trace()
return T, P
# Format image for map. Resize original image to config.im_size (300 in here)
def format_img_map(img, config):
img_min_side = float(config.im_size)
(height,width,_) = img.shape
if width <= height:
f = img_min_side/width
new_height = int(f * height)
new_width = int(img_min_side)
else:
f = img_min_side/height
new_width = int(f * width)
new_height = int(img_min_side)
fx = width/float(new_width)
fy = height/float(new_height)
img = cv2.resize(img, (new_width, new_height), interpolation=cv2.INTER_CUBIC)
# Change image channel from BGR to RGB
img = img[:, :, (2, 1, 0)]
img = img.astype(np.float32)
img[:, :, 0] -= config.img_channel_mean[0]
img[:, :, 1] -= config.img_channel_mean[1]
img[:, :, 2] -= config.img_channel_mean[2]
img /= config.img_scaling_factor
# Change img shape from (height, width, channel) to (channel, height, width)
img = np.transpose(img, (2, 0, 1))
# Expand one dimension at axis 0
# img shape becames (1, channel, height, width)
img = np.expand_dims(img, axis=0)
return img, fx, fyLayers
This module includes custom layers that is used by the model builder to create a Faster R-CNN model.
# Import libraries
import cv2
import keras
import math
import numpy as np
import tensorflow as tf
import annytab.frcnn.common as common
# ROI pooling layer for 2D inputs
# K. He, X. Zhang, S. Ren, J. Sun
class RoiPoolingConv(keras.engine.Layer):
def __init__(self, pool_size, num_rois, **kwargs):
# keras.backend.image_dim_ordering()
self.dim_ordering = keras.backend.image_data_format()
self.pool_size = pool_size
self.num_rois = num_rois
super(RoiPoolingConv, self).__init__(**kwargs)
def build(self, input_shape):
self.nb_channels = input_shape[0][3]
def compute_output_shape(self, input_shape):
return None, self.num_rois, self.pool_size, self.pool_size, self.nb_channels
def call(self, x, mask=None):
assert(len(x) == 2)
# x[0] is image with shape (rows, cols, channels)
img = x[0]
# x[1] is roi with shape (num_rois,4) with ordering (x,y,w,h)
rois = x[1]
input_shape = keras.backend.shape(img)
outputs = []
for roi_idx in range(self.num_rois):
x = rois[0, roi_idx, 0]
y = rois[0, roi_idx, 1]
w = rois[0, roi_idx, 2]
h = rois[0, roi_idx, 3]
x = keras.backend.cast(x, 'int32')
y = keras.backend.cast(y, 'int32')
w = keras.backend.cast(w, 'int32')
h = keras.backend.cast(h, 'int32')
# Resized roi of the image to pooling size (7x7)
# tf.image.resize_images
rs = tf.image.resize(img[:, y:y+h, x:x+w, :], (self.pool_size, self.pool_size))
outputs.append(rs)
# Concentate tensors along an axis
final_output = keras.backend.concatenate(outputs, axis=0)
# Reshape to (1, num_rois, pool_size, pool_size, nb_channels)
# Might be (1, 4, 7, 7, 3)
final_output = keras.backend.reshape(final_output, (1, self.num_rois, self.pool_size, self.pool_size, self.nb_channels))
# permute_dimensions is similar to transpose
final_output = keras.backend.permute_dimensions(final_output, (0, 1, 2, 3, 4))
# Return final output
return final_output
def get_config(self):
config = {'pool_size': self.pool_size, 'num_rois': self.num_rois}
base_config = super(RoiPoolingConv, self).get_config()
return dict(list(base_config.items()) + list(config.items()))
# Create a rpn layer
def rpn_layer(base_layers, num_anchors):
x = keras.layers.Conv2D(512, (3, 3), padding='same', activation='relu', kernel_initializer='normal', name='rpn_conv1')(base_layers)
x_class = keras.layers.Conv2D(num_anchors, (1, 1), activation='sigmoid', kernel_initializer='uniform', name='rpn_out_class')(x)
x_regr = keras.layers.Conv2D(num_anchors * 4, (1, 1), activation='linear', kernel_initializer='zero', name='rpn_out_regress')(x)
return [x_class, x_regr, base_layers]
# Create a classifier layer
def classifier_layer(base_layers, input_rois, num_rois, nb_classes = 4):
input_shape = (num_rois,7,7,512)
pooling_regions = 7
# out_roi_pool.shape = (1, num_rois, channels, pool_size, pool_size)
# num_rois (4) 7x7 roi pooling
out_roi_pool = RoiPoolingConv(pooling_regions, num_rois)([base_layers, input_rois])
# Flatten the convlutional layer and connected to 2 FC and 2 dropout
out = keras.layers.TimeDistributed(keras.layers.Flatten(name='flatten'))(out_roi_pool)
out = keras.layers.TimeDistributed(keras.layers.Dense(4096, activation='relu', name='fc1'))(out)
out = keras.layers.TimeDistributed(keras.layers.Dropout(0.5))(out)
out = keras.layers.TimeDistributed(keras.layers.Dense(4096, activation='relu', name='fc2'))(out)
out = keras.layers.TimeDistributed(keras.layers.Dropout(0.5))(out)
# There are two output layer
# out_class: softmax acivation function for classify the class name of the object
# out_regr: linear activation function for bboxes coordinates regression
out_class = keras.layers.TimeDistributed(keras.layers.Dense(nb_classes, activation='softmax', kernel_initializer='zero'), name='dense_class_{}'.format(nb_classes))(out)
# note: no regression target for bg class
out_regr = keras.layers.TimeDistributed(keras.layers.Dense(4 * (nb_classes-1), activation='linear', kernel_initializer='zero'), name='dense_regress_{}'.format(nb_classes))(out)
return [out_class, out_regr]
# Apply regression layer to all anchors in one feature map
def apply_regr_np(X, T):
try:
x = X[0, :, :]
y = X[1, :, :]
w = X[2, :, :]
h = X[3, :, :]
tx = T[0, :, :]
ty = T[1, :, :]
tw = T[2, :, :]
th = T[3, :, :]
cx = x + w/2.
cy = y + h/2.
cx1 = tx * w + cx
cy1 = ty * h + cy
w1 = np.exp(tw.astype(np.float64)) * w
h1 = np.exp(th.astype(np.float64)) * h
x1 = cx1 - w1/2.
y1 = cy1 - h1/2.
x1 = np.round(x1)
y1 = np.round(y1)
w1 = np.round(w1)
h1 = np.round(h1)
return np.stack([x1, y1, w1, h1])
except Exception as e:
print(e)
return X
# Apply regression to x, y, w and h
def apply_regr(x, y, w, h, tx, ty, tw, th):
try:
cx = x + w/2.
cy = y + h/2.
cx1 = tx * w + cx
cy1 = ty * h + cy
w1 = math.exp(tw) * w
h1 = math.exp(th) * h
x1 = cx1 - w1/2.
y1 = cy1 - h1/2.
x1 = int(round(x1))
y1 = int(round(y1))
w1 = int(round(w1))
h1 = int(round(h1))
return x1, y1, w1, h1
except ValueError:
return x, y, w, h
except OverflowError:
return x, y, w, h
except Exception as e:
print(e)
return x, y, w, h
# Convert rpn layer to roi bboxes
def rpn_to_roi(rpn_layer, regr_layer, config, dim_ordering, use_regr=True, max_boxes=300,overlap_thresh=0.9):
# Make sure that there is configurations
if config == None:
config = common.Config()
# Create regression layer
regr_layer = regr_layer / config.std_scaling
anchor_sizes = config.anchor_box_scales # (3 in here)
anchor_ratios = config.anchor_box_ratios # (3 in here)
assert rpn_layer.shape[0] == 1
(rows, cols) = rpn_layer.shape[1:3]
curr_layer = 0
# A.shape = (4, feature_map.height, feature_map.width, num_anchors)
# Might be (4, 18, 25, 18) if resized image is 400 width and 300
# A is the coordinates for 9 anchors for every point in the feature map
# => all 18x25x9=4050 anchors cooridnates
A = np.zeros((4, rpn_layer.shape[1], rpn_layer.shape[2], rpn_layer.shape[3]))
for anchor_size in anchor_sizes:
for anchor_ratio in anchor_ratios:
# anchor_x = (128 * 1) / 16 = 8 => width of current anchor
# anchor_y = (128 * 2) / 16 = 16 => height of current anchor
anchor_x = (anchor_size * anchor_ratio[0])/config.rpn_stride
anchor_y = (anchor_size * anchor_ratio[1])/config.rpn_stride
# curr_layer: 0~8 (9 anchors)
# the Kth anchor of all position in the feature map (9th in total)
regr = regr_layer[0, :, :, 4 * curr_layer:4 * curr_layer + 4] # shape => (18, 25, 4)
regr = np.transpose(regr, (2, 0, 1)) # shape => (4, 18, 25)
# Create 18x25 mesh grid
# For every point in x, there are all the y points and vice versa
# X.shape = (18, 25)
# Y.shape = (18, 25)
X, Y = np.meshgrid(np.arange(cols),np. arange(rows))
# Calculate anchor position and size for each feature map point
A[0, :, :, curr_layer] = X - anchor_x/2 # Top left x coordinate
A[1, :, :, curr_layer] = Y - anchor_y/2 # Top left y coordinate
A[2, :, :, curr_layer] = anchor_x # width of current anchor
A[3, :, :, curr_layer] = anchor_y # height of current anchor
# Apply regression to x, y, w and h if there is rpn regression layer
if use_regr:
A[:, :, :, curr_layer] = apply_regr_np(A[:, :, :, curr_layer], regr)
# Avoid width and height exceeding 1
A[2, :, :, curr_layer] = np.maximum(1, A[2, :, :, curr_layer])
A[3, :, :, curr_layer] = np.maximum(1, A[3, :, :, curr_layer])
# Convert (x, y , w, h) to (x1, y1, x2, y2)
# x1, y1 is top left coordinate
# x2, y2 is bottom right coordinate
A[2, :, :, curr_layer] += A[0, :, :, curr_layer]
A[3, :, :, curr_layer] += A[1, :, :, curr_layer]
# Avoid bboxes drawn outside the feature map
A[0, :, :, curr_layer] = np.maximum(0, A[0, :, :, curr_layer])
A[1, :, :, curr_layer] = np.maximum(0, A[1, :, :, curr_layer])
A[2, :, :, curr_layer] = np.minimum(cols-1, A[2, :, :, curr_layer])
A[3, :, :, curr_layer] = np.minimum(rows-1, A[3, :, :, curr_layer])
curr_layer += 1
all_boxes = np.reshape(A.transpose((0, 3, 1, 2)), (4, -1)).transpose((1, 0)) # shape=(4050, 4)
all_probs = rpn_layer.transpose((0, 3, 1, 2)).reshape((-1)) # shape=(4050,)
x1 = all_boxes[:, 0]
y1 = all_boxes[:, 1]
x2 = all_boxes[:, 2]
y2 = all_boxes[:, 3]
# Find out the bboxes which is illegal and delete them from bboxes list
idxs = np.where((x1 - x2 >= 0) | (y1 - y2 >= 0))
all_boxes = np.delete(all_boxes, idxs, 0)
all_probs = np.delete(all_probs, idxs, 0)
# Apply non_max_suppression
# Only extract the bboxes. Don't need rpn probs in the later process
result = common.non_max_suppression_fast(all_boxes, all_probs, overlap_thresh=overlap_thresh, max_boxes=max_boxes)[0]
return resultModel Builder
This module is used to create training models and inference models. A VGG-16 model is used as a backbone model for image classification, it was choosed to speed up training time.
# Import libraries
import keras
import annytab.frcnn.common as common
import annytab.frcnn.layers as layers
# Get a VGG-16 model
def get_vgg_16_model(input=None):
# Make sure that the input is okay
input_shape = (None, None, 3)
if input is None:
input = keras.layers.Input(shape=input_shape)
else:
if not keras.backend.is_keras_tensor(input):
input = keras.layers.Input(tensor=input, shape=input_shape)
# Set backbone axis
bn_axis = 3
# Block 1
x = keras.layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv1')(input)
x = keras.layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2')(x)
x = keras.layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)
# Block 2
x = keras.layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv1')(x)
x = keras.layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2')(x)
x = keras.layers.MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool')(x)
# Block 3
x = keras.layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv1')(x)
x = keras.layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2')(x)
x = keras.layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv3')(x)
x = keras.layers.MaxPooling2D((2, 2), strides=(2, 2), name='block3_pool')(x)
# Block 4
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv1')(x)
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2')(x)
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv3')(x)
x = keras.layers.MaxPooling2D((2, 2), strides=(2, 2), name='block4_pool')(x)
# Block 5
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv1')(x)
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2')(x)
x = keras.layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv3')(x)
# x = keras.layers.MaxPooling2D((2, 2), strides=(2, 2), name='block5_pool')(x)
return x
# Get training models
def get_training_models(config=None, num_classes=3, weights_path=None):
# Make sure that there is configurations
if config == None:
config = common.Config()
# Calculate the number of anchors
num_anchors = len(config.anchor_box_scales) * len(config.anchor_box_ratios)
# Create input layers
img_input = keras.layers.Input(shape=(None, None, 3))
roi_input = keras.layers.Input(shape=(None, 4))
# Get a backbone model (VGG here, can be Resnet50, Inception, etc)
backbone = get_vgg_16_model(img_input)
# Create an rpn layer
rpn = layers.rpn_layer(backbone, num_anchors)
# Create a classifier
classifier = layers.classifier_layer(backbone, roi_input, config.num_rois, nb_classes=num_classes)
# Create models
rpn_model = keras.models.Model(img_input, rpn[:2])
classifier_model = keras.models.Model([img_input, roi_input], classifier)
total_model = keras.models.Model([img_input, roi_input], rpn[:2] + classifier)
# Load weights
if weights_path != None:
rpn_model.load_weights(weights_path, by_name=True)
classifier_model.load_weights(weights_path, by_name=True)
# Compile models
rpn_model.compile(optimizer=keras.optimizers.Adam(lr=1e-5), loss=[common.rpn_loss_cls(num_anchors, config=config), common.rpn_loss_regr(num_anchors, config=config)])
classifier_model.compile(optimizer=keras.optimizers.Adam(lr=1e-5), loss=[common.class_loss_cls(config=config), common.class_loss_regr(num_classes-1, config=config)], metrics={'dense_class_{}'.format(num_classes): 'accuracy'})
total_model.compile(optimizer='sgd', loss='mae')
# Return models
return rpn_model, classifier_model, total_model
# Get inference models
def get_inference_models(config=None, num_classes=3, weights_path=None):
# Make sure that there is configurations
if config == None:
config = common.Config()
# Calculate the number of anchors
num_anchors = len(config.anchor_box_scales) * len(config.anchor_box_ratios)
# Create input layers
img_input = keras.layers.Input(shape=(None, None, 3))
roi_input = keras.layers.Input(shape=(config.num_rois, 4))
feature_map_input = keras.layers.Input(shape=(None, None, 512))
# Get a backbone model (VGG here, can be Resnet50, Inception, etc)
backbone = get_vgg_16_model(img_input)
# Create an rpn layer
rpn = layers.rpn_layer(backbone, num_anchors)
# Create a classifier
classifier = layers.classifier_layer(feature_map_input, roi_input, config.num_rois, nb_classes=num_classes)
# Create models
rpn_model = keras.models.Model(img_input, rpn)
classifier_only_model = keras.models.Model([feature_map_input, roi_input], classifier)
classifier_model = keras.models.Model([feature_map_input, roi_input], classifier)
# Load weights
rpn_model.load_weights(weights_path, by_name=True)
classifier_model.load_weights(weights_path, by_name=True)
# Compile models
rpn_model.compile(optimizer='sgd', loss='mse')
classifier_model.compile(optimizer='sgd', loss='mse')
# Return models
return rpn_model, classifier_model, classifier_only_modelDebugging
The purpose of debugging is to make sure that annotations has been specified correctly, it is important to check this before training is initiated. The result of a run is a image with bounding boxes drawn for objects in the image.
# Import libraries
import time
import cv2
import random
import numpy as np
import matplotlib.pyplot as plt
import annytab.frcnn.common as common
import tensorflow as tf
# The main entry point for this module
def main():
# Create configuration
config = common.Config()
config.annotations_file_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\train_annotations.csv'
config.img_base_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\'
config.use_horizontal_flips = True
config.use_vertical_flips = True
config.rot_90 = True
config.num_rois = 4
# Set the gpu to use
common.setup_gpu('cpu')
#common.setup_gpu(0)
#print('Tf with cuda:{0}'.format(tf.test.is_built_with_cuda()))
#print('Tf with cuda:{0}'.format(tf.test.is_gpu_available(cuda_only=False, min_cuda_compute_capability=None)))
# Get data
st = time.time()
images, classes, mappings = common.get_data(config)
print()
print('Spend {0:0.2f} mins to load the data'.format((time.time()-st)/60))
# Make sure that we have a bg class. A background region, this is usually for hard negative mining
if 'bg' not in classes:
classes['bg'] = 0
mappings['bg'] = len(mappings)
# Print class distribution
print('Training images per class:')
print(classes)
print('Num classes (including bg) = {}'.format(len(classes)))
print(mappings)
# Randomize images to get different each time
random.shuffle(images)
# Get a train generator
train_generator = common.get_anchor_gt(images, config, mode='test')
# Get the next element in the training set
X, Y, image_data, debug_img, debug_num_pos = next(train_generator)
# Output debug information
print('Original image: height={0} width={1}'.format(image_data['height'], image_data['width']))
print('Resized image: height={0} width={1} im_size={2}'.format(X.shape[1], X.shape[2], config.im_size))
print('Feature map size: height={0} width={1} rpn_stride={2}'.format(Y[0].shape[1], Y[0].shape[2], config.rpn_stride))
print(X.shape)
print(str(len(Y)) + ' includes y_rpn_cls and y_rpn_regr')
print('Shape of y_rpn_cls {0}'.format(Y[0].shape))
print('Shape of y_rpn_regr {0}'.format(Y[1].shape))
print(image_data)
print('Number of positive anchors for this image: {0}'.format(debug_num_pos))
if debug_num_pos==0:
gt_x1, gt_x2 = image_data['bboxes'][0]['x1']*(X.shape[2]/image_data['height']), image_data['bboxes'][0]['x2']*(X.shape[2]/image_data['height'])
gt_y1, gt_y2 = image_data['bboxes'][0]['y1']*(X.shape[1]/image_data['width']), image_data['bboxes'][0]['y2']*(X.shape[1]/image_data['width'])
gt_x1, gt_y1, gt_x2, gt_y2 = int(gt_x1), int(gt_y1), int(gt_x2), int(gt_y2)
img = debug_img.copy()
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
color = (0, 255, 0)
cv2.putText(img, 'gt bbox', (gt_x1, gt_y1+5), cv2.FONT_HERSHEY_PLAIN, 0.7, color, 1)
cv2.rectangle(img, (gt_x1, gt_y1), (gt_x2, gt_y2), color, 2)
cv2.circle(img, (int((gt_x1+gt_x2)/2), int((gt_y1+gt_y2)/2)), 3, color, -1)
# Display the image
plt.grid()
plt.imshow(img)
plt.show()
else:
cls = Y[0][0]
pos_cls = np.where(cls==1)
print(pos_cls)
regr = Y[1][0]
pos_regr = np.where(regr==1)
print(pos_regr)
print('y_rpn_cls for possible pos anchor: {0}'.format(cls[pos_cls[0][0],pos_cls[1][0],:]))
print('y_rpn_regr for positive anchor: {0}'.format(regr[pos_regr[0][0],pos_regr[1][0],:]))
gt_x1, gt_x2 = image_data['bboxes'][0]['x1']*(X.shape[2]/image_data['width']), image_data['bboxes'][0]['x2']*(X.shape[2]/image_data['width'])
gt_y1, gt_y2 = image_data['bboxes'][0]['y1']*(X.shape[1]/image_data['height']), image_data['bboxes'][0]['y2']*(X.shape[1]/image_data['height'])
gt_x1, gt_y1, gt_x2, gt_y2 = int(gt_x1), int(gt_y1), int(gt_x2), int(gt_y2)
img = debug_img.copy()
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
color = (0, 255, 0)
# cv2.putText(img, 'gt bbox', (gt_x1, gt_y1-5), cv2.FONT_HERSHEY_DUPLEX, 0.7, color, 1)
cv2.rectangle(img, (gt_x1, gt_y1), (gt_x2, gt_y2), color, 2)
cv2.circle(img, (int((gt_x1+gt_x2)/2), int((gt_y1+gt_y2)/2)), 3, color, -1)
# Add text
textLabel = 'gt bbox'
(retval,baseLine) = cv2.getTextSize(textLabel,cv2.FONT_HERSHEY_PLAIN,0.5,1)
textOrg = (gt_x1, gt_y1+5)
cv2.putText(img, textLabel, textOrg, cv2.FONT_HERSHEY_PLAIN, 0.5, (255, 255, 255), 1)
# Draw positive anchors according to the y_rpn_regr
for i in range(debug_num_pos):
color = (100+i*(155/4), 0, 100+i*(155/4))
idx = pos_regr[2][i*4]/4
anchor_size = config.anchor_box_scales[int(idx/3)]
anchor_ratio = config.anchor_box_ratios[2-int((idx+1)%3)]
center = (pos_regr[1][i*4]*config.rpn_stride, pos_regr[0][i*4]*config.rpn_stride)
print('Center position of positive anchor: ', center)
cv2.circle(img, center, 3, color, -1)
anc_w, anc_h = anchor_size*anchor_ratio[0], anchor_size*anchor_ratio[1]
cv2.rectangle(img, (center[0]-int(anc_w/2), center[1]-int(anc_h/2)), (center[0]+int(anc_w/2), center[1]+int(anc_h/2)), color, 2)
# Display the image
print('Green bboxes is ground-truth bbox. Others are positive anchors')
plt.figure(figsize=(8,8))
plt.grid()
plt.imshow(img)
plt.savefig('C:\\DATA\\Python-data\\open-images-v5\\frcnn\\plots\\debug.png')
#plt.show()
# Tell python to run main method
if __name__ == "__main__": main()Training
Training models is created if no training has been done before, weights can be loaded from a pretrained model. Training models is loaded with saved weights if training has been conducted before, this enables the model to continue training (transfer learning). The result from a run is shown below the code.
# Import libraries
import os
import time
import cv2
import keras
import numpy as np
import pandas as pd
import tensorflow as tf
import annytab.frcnn.common as common
import annytab.frcnn.model_builder as mb
import annytab.frcnn.layers as layers
# The main entry point for this module
def main():
# Create configuration
config = common.Config()
config.annotations_file_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\train_annotations.csv'
config.img_base_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\'
config.pretrained_model_path = 'C:\\DATA\\Python-data\\open-images-v5\\frcnn\\vgg16_weights_tf_dim_ordering_tf_kernels.h5'
config.model_path = 'C:\\DATA\\Python-data\\open-images-v5\\frcnn\\training_model.h5'
config.records_path = 'C:\\DATA\\Python-data\\open-images-v5\\frcnn\\records.csv'
config.use_horizontal_flips = True
config.use_vertical_flips = True
config.rot_90 = True
config.num_rois = 4
# Set the gpu to use
common.setup_gpu('cpu')
#common.setup_gpu(0)
# Get data
st = time.time()
images, classes, mappings = common.get_data(config)
print()
print('Spend {0:0.2f} mins to load the data'.format((time.time()-st)/60))
# Make sure that we have a bg class. A background region, this is usually for hard negative mining
if 'bg' not in classes:
classes['bg'] = 0
mappings['bg'] = len(mappings)
# Print class distribution
print('Training images per class:')
print(classes)
print('Number of classes (including bg) = {}'.format(len(classes)))
print(mappings)
# Get a train generator
train_generator = common.get_anchor_gt(images, config, mode='train')
# Get training models
if os.path.isfile(config.model_path):
# Get training models
model_rpn, model_classifier, model_all = mb.get_training_models(config, len(classes), weights_path=config.model_path)
# Load records
record_df = pd.read_csv(config.records_path)
r_mean_overlapping_bboxes = record_df['mean_overlapping_bboxes']
r_class_acc = record_df['class_acc']
r_loss_rpn_cls = record_df['loss_rpn_cls']
r_loss_rpn_regr = record_df['loss_rpn_regr']
r_loss_class_cls = record_df['loss_class_cls']
r_loss_class_regr = record_df['loss_class_regr']
r_curr_loss = record_df['curr_loss']
r_elapsed_time = record_df['elapsed_time']
r_mAP = record_df['mAP']
print('Already trained {0} batches'.format(len(record_df)))
else:
# Check if we can load weights from pretrained model or not
if os.path.isfile(config.pretrained_model_path):
model_rpn, model_classifier, model_all = mb.get_training_models(config, len(classes), weights_path=config.pretrained_model_path)
else:
model_rpn, model_classifier, model_all = mb.get_training_models(config, len(classes))
# Create a records dataframe
record_df = pd.DataFrame(columns=['mean_overlapping_bboxes', 'class_acc', 'loss_rpn_cls', 'loss_rpn_regr', 'loss_class_cls', 'loss_class_regr', 'curr_loss', 'elapsed_time', 'mAP'])
# Settings for training
total_epochs = len(record_df)
r_epochs = len(record_df)
steps = 240 # 240 images (epoch_length)
num_epochs = 1
iter_num = 0
total_epochs += num_epochs
losses = np.zeros((steps, 5))
rpn_accuracy_rpn_monitor = []
rpn_accuracy_for_epoch = []
if len(record_df)==0:
best_loss = np.Inf
else:
best_loss = np.min(r_curr_loss)
# Start training (one image on each iteration)
start_time = time.time()
# Loop epochs
for epoch_num in range(num_epochs):
# Create a progress bar
progbar = keras.utils.Progbar(steps)
# Print the current epoch
print('Epoch {}/{}'.format(r_epochs + 1, total_epochs))
r_epochs += 1
while True:
try:
if len(rpn_accuracy_rpn_monitor) == steps and config.verbose:
mean_overlapping_bboxes = float(sum(rpn_accuracy_rpn_monitor))/len(rpn_accuracy_rpn_monitor)
rpn_accuracy_rpn_monitor = []
if mean_overlapping_bboxes == 0:
print('RPN is not producing bounding boxes that overlap the ground truth boxes. Check RPN settings or keep training.')
# Get the next element from an generator
# Generate X (x_img) and label Y ([y_rpn_cls, y_rpn_regr])
X, Y, img_data, debug_img, debug_num_pos = next(train_generator)
# Train rpn model and get loss value [_, loss_rpn_cls, loss_rpn_regr]
loss_rpn = model_rpn.train_on_batch(X, Y)
# Get predicted rpn from rpn model [rpn_cls, rpn_regr]
P_rpn = model_rpn.predict_on_batch(X)
# R: bboxes (shape=(300,4))
# Convert rpn layer to roi bboxes
R = layers.rpn_to_roi(P_rpn[0], P_rpn[1], config, keras.backend.image_data_format(), use_regr=True, overlap_thresh=0.7, max_boxes=300)
# note: calc_iou converts from (x1,y1,x2,y2) to (x,y,w,h) format
# X2: bboxes that iou > config.classifier_min_overlap for all gt bboxes in 300 non_max_suppression bboxes
# Y1: one hot code for bboxes from above => x_roi (X)
# Y2: corresponding labels and corresponding gt bboxes
X2, Y1, Y2, IouS = common.calc_iou(R, img_data, config, mappings)
# If X2 is None means there are no matching bboxes
if X2 is None:
rpn_accuracy_rpn_monitor.append(0)
rpn_accuracy_for_epoch.append(0)
continue
# Find out the positive anchors and negative anchors
neg_samples = np.where(Y1[0, :, -1] == 1)
pos_samples = np.where(Y1[0, :, -1] == 0)
if len(neg_samples) > 0:
neg_samples = neg_samples[0]
else:
neg_samples = []
if len(pos_samples) > 0:
pos_samples = pos_samples[0]
else:
pos_samples = []
rpn_accuracy_rpn_monitor.append(len(pos_samples))
rpn_accuracy_for_epoch.append((len(pos_samples)))
if config.num_rois > 1:
# If number of positive anchors is larger than 4//2 = 2, randomly choose 2 pos samples
if len(pos_samples) < config.num_rois//2:
selected_pos_samples = pos_samples.tolist()
else:
selected_pos_samples = np.random.choice(pos_samples, config.num_rois//2, replace=False).tolist()
# Randomly choose (num_rois - num_pos) neg samples
try:
selected_neg_samples = np.random.choice(neg_samples, config.num_rois - len(selected_pos_samples), replace=False).tolist()
except:
selected_neg_samples = np.random.choice(neg_samples, config.num_rois - len(selected_pos_samples), replace=True).tolist()
# Save all the pos and neg samples in sel_samples
sel_samples = selected_pos_samples + selected_neg_samples
else:
# in the extreme case where num_rois = 1, we pick a random pos or neg sample
selected_pos_samples = pos_samples.tolist()
selected_neg_samples = neg_samples.tolist()
if np.random.randint(0, 2):
sel_samples = random.choice(neg_samples)
else:
sel_samples = random.choice(pos_samples)
# Train classifier
loss_class = model_classifier.train_on_batch([X, X2[:, sel_samples, :]], [Y1[:, sel_samples, :], Y2[:, sel_samples, :]])
losses[iter_num, 0] = loss_rpn[1]
losses[iter_num, 1] = loss_rpn[2]
losses[iter_num, 2] = loss_class[1]
losses[iter_num, 3] = loss_class[2]
losses[iter_num, 4] = loss_class[3]
iter_num += 1
# Update the progress bar
progbar.update(iter_num, [('rpn_cls', np.mean(losses[:iter_num, 0])), ('rpn_regr', np.mean(losses[:iter_num, 1])),
('final_cls', np.mean(losses[:iter_num, 2])), ('final_regr', np.mean(losses[:iter_num, 3]))])
# Free memory before each step
#tf.keras.backend.clear_session()
# Start the next step
if iter_num == steps:
loss_rpn_cls = np.mean(losses[:, 0])
loss_rpn_regr = np.mean(losses[:, 1])
loss_class_cls = np.mean(losses[:, 2])
loss_class_regr = np.mean(losses[:, 3])
class_acc = np.mean(losses[:, 4])
mean_overlapping_bboxes = float(sum(rpn_accuracy_for_epoch)) / len(rpn_accuracy_for_epoch)
rpn_accuracy_for_epoch = []
if config.verbose:
print('Mean number of bounding boxes from RPN overlapping ground truth boxes: {0}'.format(mean_overlapping_bboxes))
print('Classifier accuracy for bounding boxes from RPN: {0}'.format(class_acc))
print('Loss RPN classifier: {0}'.format(loss_rpn_cls))
print('Loss RPN regression: {0}'.format(loss_rpn_regr))
print('Loss Detector classifier: {0}'.format(loss_class_cls))
print('Loss Detector regression: {0}'.format(loss_class_regr))
print('Total loss: {0}'.format(loss_rpn_cls + loss_rpn_regr + loss_class_cls + loss_class_regr))
print('Elapsed time: {0}'.format(time.time() - start_time))
elapsed_time = (time.time()-start_time)/60
curr_loss = loss_rpn_cls + loss_rpn_regr + loss_class_cls + loss_class_regr
iter_num = 0
start_time = time.time()
# Save the model
if curr_loss < best_loss:
if config.verbose:
print('Total loss decreased from {0} to {1}, saving model.'.format(best_loss,curr_loss))
best_loss = curr_loss
#model_all.save_weights(config.model_path)
model_all.save(config.model_path)
# Create a new records row
new_row = {'mean_overlapping_bboxes':round(mean_overlapping_bboxes, 3),
'class_acc':round(class_acc, 3),
'loss_rpn_cls':round(loss_rpn_cls, 3),
'loss_rpn_regr':round(loss_rpn_regr, 3),
'loss_class_cls':round(loss_class_cls, 3),
'loss_class_regr':round(loss_class_regr, 3),
'curr_loss':round(curr_loss, 3),
'elapsed_time':round(elapsed_time, 3),
'mAP': 0}
# Append a new record row and save the file
record_df = record_df.append(new_row, ignore_index=True)
record_df.to_csv(config.records_path, index=0)
break
except Exception as e:
print('Exception: {}'.format(e))
continue
# Free memory after each epoch
#tf.keras.backend.clear_session()
print('Training complete, exiting.')
# Tell python to run main method
if __name__ == "__main__": main()Parsing annotation file
idx=450
Spend 0.05 mins to load the data
Training images per class:
{'Laptop': 165, 'Beer': 140, 'Goat': 145, 'bg': 0}
Number of classes (including bg) = 4
{'Laptop': 0, 'Beer': 1, 'Goat': 2, 'bg': 3}
Already trained 20 batches
Epoch 21/21
1/240 [..............................] - ETA: 1:03:56 - rpn_cls: 4.7960 - rpn_regr: 0.0395 - final_cls: 0.6163 - final_regr: 0.0289
2/240 [..............................] - ETA: 41:03 - rpn_cls: 4.5647 - rpn_regr: 0.0483 - final_cls: 0.5607 - final_regr: 0.0983
3/240 [..............................] - ETA: 33:39 - rpn_cls: 4.0262 - rpn_regr: 0.0673 - final_cls: 0.5503 - final_regr: 0.2006
4/240 [..............................] - ETA: 29:19 - rpn_cls: 3.5737 - rpn_regr: 0.0771 - final_cls: 0.5335 - final_regr: 0.2581
5/240 [..............................] - ETA: 27:29 - rpn_cls: 3.4570 - rpn_regr: 0.0868 - final_cls: 0.5348 - final_regr: 0.3002
6/240 [..............................] - ETA: 25:52 - rpn_cls: 3.2962 - rpn_regr: 0.0957 - final_cls: 0.5232 - final_regr: 0.3184
7/240 [..............................] - ETA: 24:50 - rpn_cls: 3.2421 - rpn_regr: 0.1264 - final_cls: 0.5137 - final_regr: 0.3230
8/240 [>.............................] - ETA: 23:48 - rpn_cls: 3.1559 - rpn_regr: 0.1496 - final_cls: 0.5022 - final_regr: 0.3277
9/240 [>.............................] - ETA: 23:00 - rpn_cls: 3.0965 - rpn_regr: 0.1687 - final_cls: 0.4918 - final_regr: 0.3349
10/240 [>.............................] - ETA: 22:29 - rpn_cls: 3.0849 - rpn_regr: 0.1911 - final_cls: 0.4800 - final_regr: 0.3368
11/240 [>.............................] - ETA: 22:05 - rpn_cls: 3.0511 - rpn_regr: 0.2091 - final_cls: 0.4674 - final_regr: 0.3353
12/240 [>.............................] - ETA: 21:53 - rpn_cls: 3.0447 - rpn_regr: 0.2244 - final_cls: 0.4614 - final_regr: 0.3363
13/240 [>.............................] - ETA: 21:27 - rpn_cls: 3.0216 - rpn_regr: 0.2360 - final_cls: 0.4546 - final_regr: 0.3387
14/240 [>.............................] - ETA: 21:03 - rpn_cls: 2.9878 - rpn_regr: 0.2441 - final_cls: 0.4488 - final_regr: 0.3412
15/240 [>.............................] - ETA: 20:39 - rpn_cls: 2.9472 - rpn_regr: 0.2509 - final_cls: 0.4469 - final_regr: 0.3427
16/240 [=>............................] - ETA: 20:17 - rpn_cls: 2.9265 - rpn_regr: 0.2572 - final_cls: 0.4436 - final_regr: 0.3427
17/240 [=>............................] - ETA: 19:55 - rpn_cls: 2.8991 - rpn_regr: 0.2616 - final_cls: 0.4420 - final_regr: 0.3424
18/240 [=>............................] - ETA: 19:45 - rpn_cls: 2.8681 - rpn_regr: 0.2670 - final_cls: 0.4394 - final_regr: 0.3410
19/240 [=>............................] - ETA: 19:30 - rpn_cls: 2.8514 - rpn_regr: 0.2710 - final_cls: 0.4363 - final_regr: 0.3390
20/240 [=>............................] - ETA: 19:11 - rpn_cls: 2.8300 - rpn_regr: 0.2751 - final_cls: 0.4355 - final_regr: 0.3373
21/240 [=>............................] - ETA: 19:01 - rpn_cls: 2.8089 - rpn_regr: 0.2795 - final_cls: 0.4358 - final_regr: 0.3358
22/240 [=>............................] - ETA: 18:50 - rpn_cls: 2.7848 - rpn_regr: 0.2831 - final_cls: 0.4370 - final_regr: 0.3353
23/240 [=>............................] - ETA: 18:43 - rpn_cls: 2.7666 - rpn_regr: 0.2865 - final_cls: 0.4383 - final_regr: 0.3348
24/240 [==>...........................] - ETA: 18:36 - rpn_cls: 2.7463 - rpn_regr: 0.2892 - final_cls: 0.4400 - final_regr: 0.3347
25/240 [==>...........................] - ETA: 18:30 - rpn_cls: 2.7250 - rpn_regr: 0.2912 - final_cls: 0.4411 - final_regr: 0.3346
26/240 [==>...........................] - ETA: 18:23 - rpn_cls: 2.7032 - rpn_regr: 0.2927 - final_cls: 0.4422 - final_regr: 0.3344
27/240 [==>...........................] - ETA: 18:13 - rpn_cls: 2.6800 - rpn_regr: 0.2940 - final_cls: 0.4433 - final_regr: 0.3344
28/240 [==>...........................] - ETA: 18:06 - rpn_cls: 2.6558 - rpn_regr: 0.2952 - final_cls: 0.4438 - final_regr: 0.3340
29/240 [==>...........................] - ETA: 17:58 - rpn_cls: 2.6339 - rpn_regr: 0.2969 - final_cls: 0.4455 - final_regr: 0.3335
30/240 [==>...........................] - ETA: 17:58 - rpn_cls: 2.6112 - rpn_regr: 0.2982 - final_cls: 0.4468 - final_regr: 0.3329
31/240 [==>...........................] - ETA: 17:50 - rpn_cls: 2.5880 - rpn_regr: 0.2993 - final_cls: 0.4478 - final_regr: 0.3326
32/240 [===>..........................] - ETA: 17:42 - rpn_cls: 2.5649 - rpn_regr: 0.3000 - final_cls: 0.4485 - final_regr: 0.3323
33/240 [===>..........................] - ETA: 17:35 - rpn_cls: 2.5415 - rpn_regr: 0.3008 - final_cls: 0.4490 - final_regr: 0.3320
34/240 [===>..........................] - ETA: 17:27 - rpn_cls: 2.5196 - rpn_regr: 0.3014 - final_cls: 0.4495 - final_regr: 0.3318
35/240 [===>..........................] - ETA: 17:21 - rpn_cls: 2.4976 - rpn_regr: 0.3018 - final_cls: 0.4501 - final_regr: 0.3316
36/240 [===>..........................] - ETA: 17:12 - rpn_cls: 2.4754 - rpn_regr: 0.3021 - final_cls: 0.4506 - final_regr: 0.3314
37/240 [===>..........................] - ETA: 17:06 - rpn_cls: 2.4532 - rpn_regr: 0.3022 - final_cls: 0.4509 - final_regr: 0.3311
38/240 [===>..........................] - ETA: 16:54 - rpn_cls: 2.4329 - rpn_regr: 0.3022 - final_cls: 0.4510 - final_regr: 0.3306
39/240 [===>..........................] - ETA: 16:46 - rpn_cls: 2.4126 - rpn_regr: 0.3021 - final_cls: 0.4511 - final_regr: 0.3300
40/240 [====>.........................] - ETA: 16:35 - rpn_cls: 2.3962 - rpn_regr: 0.3022 - final_cls: 0.4513 - final_regr: 0.3296
41/240 [====>.........................] - ETA: 16:27 - rpn_cls: 2.3796 - rpn_regr: 0.3023 - final_cls: 0.4519 - final_regr: 0.3292
42/240 [====>.........................] - ETA: 16:17 - rpn_cls: 2.3628 - rpn_regr: 0.3023 - final_cls: 0.4523 - final_regr: 0.3288
43/240 [====>.........................] - ETA: 16:09 - rpn_cls: 2.3463 - rpn_regr: 0.3021 - final_cls: 0.4528 - final_regr: 0.3285
44/240 [====>.........................] - ETA: 16:04 - rpn_cls: 2.3296 - rpn_regr: 0.3019 - final_cls: 0.4536 - final_regr: 0.3283
45/240 [====>.........................] - ETA: 15:57 - rpn_cls: 2.3136 - rpn_regr: 0.3017 - final_cls: 0.4546 - final_regr: 0.3282
46/240 [====>.........................] - ETA: 15:51 - rpn_cls: 2.2989 - rpn_regr: 0.3016 - final_cls: 0.4555 - final_regr: 0.3283
47/240 [====>.........................] - ETA: 15:45 - rpn_cls: 2.2841 - rpn_regr: 0.3015 - final_cls: 0.4565 - final_regr: 0.3285
48/240 [=====>........................] - ETA: 15:39 - rpn_cls: 2.2692 - rpn_regr: 0.3014 - final_cls: 0.4573 - final_regr: 0.3285
49/240 [=====>........................] - ETA: 15:32 - rpn_cls: 2.2569 - rpn_regr: 0.3012 - final_cls: 0.4588 - final_regr: 0.3287
50/240 [=====>........................] - ETA: 15:26 - rpn_cls: 2.2450 - rpn_regr: 0.3010 - final_cls: 0.4605 - final_regr: 0.3288
51/240 [=====>........................] - ETA: 15:20 - rpn_cls: 2.2341 - rpn_regr: 0.3009 - final_cls: 0.4619 - final_regr: 0.3289
52/240 [=====>........................] - ETA: 15:15 - rpn_cls: 2.2231 - rpn_regr: 0.3007 - final_cls: 0.4630 - final_regr: 0.3288
53/240 [=====>........................] - ETA: 15:10 - rpn_cls: 2.2128 - rpn_regr: 0.3004 - final_cls: 0.4644 - final_regr: 0.3288
54/240 [=====>........................] - ETA: 15:04 - rpn_cls: 2.2024 - rpn_regr: 0.3001 - final_cls: 0.4656 - final_regr: 0.3287
55/240 [=====>........................] - ETA: 14:59 - rpn_cls: 2.1919 - rpn_regr: 0.2998 - final_cls: 0.4666 - final_regr: 0.3286
56/240 [======>.......................] - ETA: 14:53 - rpn_cls: 2.1813 - rpn_regr: 0.2994 - final_cls: 0.4674 - final_regr: 0.3286
57/240 [======>.......................] - ETA: 14:48 - rpn_cls: 2.1705 - rpn_regr: 0.2990 - final_cls: 0.4683 - final_regr: 0.3286
58/240 [======>.......................] - ETA: 14:43 - rpn_cls: 2.1618 - rpn_regr: 0.2990 - final_cls: 0.4690 - final_regr: 0.3286
59/240 [======>.......................] - ETA: 14:38 - rpn_cls: 2.1546 - rpn_regr: 0.2990 - final_cls: 0.4696 - final_regr: 0.3285
60/240 [======>.......................] - ETA: 14:33 - rpn_cls: 2.1477 - rpn_regr: 0.2991 - final_cls: 0.4707 - final_regr: 0.3285
61/240 [======>.......................] - ETA: 14:28 - rpn_cls: 2.1420 - rpn_regr: 0.2991 - final_cls: 0.4716 - final_regr: 0.3284
62/240 [======>.......................] - ETA: 14:23 - rpn_cls: 2.1362 - rpn_regr: 0.2995 - final_cls: 0.4725 - final_regr: 0.3283
63/240 [======>.......................] - ETA: 14:17 - rpn_cls: 2.1304 - rpn_regr: 0.2997 - final_cls: 0.4732 - final_regr: 0.3284
64/240 [=======>......................] - ETA: 14:13 - rpn_cls: 2.1258 - rpn_regr: 0.3000 - final_cls: 0.4740 - final_regr: 0.3284
65/240 [=======>......................] - ETA: 14:08 - rpn_cls: 2.1217 - rpn_regr: 0.3002 - final_cls: 0.4747 - final_regr: 0.3285
66/240 [=======>......................] - ETA: 14:04 - rpn_cls: 2.1175 - rpn_regr: 0.3005 - final_cls: 0.4754 - final_regr: 0.3286
67/240 [=======>......................] - ETA: 13:59 - rpn_cls: 2.1142 - rpn_regr: 0.3007 - final_cls: 0.4760 - final_regr: 0.3287
68/240 [=======>......................] - ETA: 13:54 - rpn_cls: 2.1108 - rpn_regr: 0.3009 - final_cls: 0.4767 - final_regr: 0.3289
69/240 [=======>......................] - ETA: 13:48 - rpn_cls: 2.1070 - rpn_regr: 0.3010 - final_cls: 0.4774 - final_regr: 0.3291
70/240 [=======>......................] - ETA: 13:42 - rpn_cls: 2.1030 - rpn_regr: 0.3013 - final_cls: 0.4783 - final_regr: 0.3293
71/240 [=======>......................] - ETA: 13:37 - rpn_cls: 2.0990 - rpn_regr: 0.3015 - final_cls: 0.4791 - final_regr: 0.3295
72/240 [========>.....................] - ETA: 13:31 - rpn_cls: 2.0950 - rpn_regr: 0.3016 - final_cls: 0.4800 - final_regr: 0.3297
73/240 [========>.....................] - ETA: 13:24 - rpn_cls: 2.0907 - rpn_regr: 0.3017 - final_cls: 0.4809 - final_regr: 0.3299
74/240 [========>.....................] - ETA: 13:19 - rpn_cls: 2.0871 - rpn_regr: 0.3019 - final_cls: 0.4817 - final_regr: 0.3302
75/240 [========>.....................] - ETA: 13:12 - rpn_cls: 2.0833 - rpn_regr: 0.3020 - final_cls: 0.4825 - final_regr: 0.3305
76/240 [========>.....................] - ETA: 13:07 - rpn_cls: 2.0798 - rpn_regr: 0.3021 - final_cls: 0.4833 - final_regr: 0.3308
77/240 [========>.....................] - ETA: 13:03 - rpn_cls: 2.0760 - rpn_regr: 0.3021 - final_cls: 0.4842 - final_regr: 0.3311
78/240 [========>.....................] - ETA: 12:59 - rpn_cls: 2.0731 - rpn_regr: 0.3023 - final_cls: 0.4851 - final_regr: 0.3314
79/240 [========>.....................] - ETA: 12:55 - rpn_cls: 2.0701 - rpn_regr: 0.3025 - final_cls: 0.4861 - final_regr: 0.3317
80/240 [=========>....................] - ETA: 12:49 - rpn_cls: 2.0669 - rpn_regr: 0.3027 - final_cls: 0.4869 - final_regr: 0.3319
81/240 [=========>....................] - ETA: 12:44 - rpn_cls: 2.0635 - rpn_regr: 0.3028 - final_cls: 0.4878 - final_regr: 0.3321
82/240 [=========>....................] - ETA: 12:39 - rpn_cls: 2.0599 - rpn_regr: 0.3029 - final_cls: 0.4886 - final_regr: 0.3323
83/240 [=========>....................] - ETA: 12:34 - rpn_cls: 2.0562 - rpn_regr: 0.3030 - final_cls: 0.4895 - final_regr: 0.3324
84/240 [=========>....................] - ETA: 12:29 - rpn_cls: 2.0531 - rpn_regr: 0.3030 - final_cls: 0.4904 - final_regr: 0.3326
85/240 [=========>....................] - ETA: 12:24 - rpn_cls: 2.0508 - rpn_regr: 0.3030 - final_cls: 0.4913 - final_regr: 0.3327
86/240 [=========>....................] - ETA: 12:18 - rpn_cls: 2.0483 - rpn_regr: 0.3031 - final_cls: 0.4922 - final_regr: 0.3328
87/240 [=========>....................] - ETA: 12:13 - rpn_cls: 2.0457 - rpn_regr: 0.3031 - final_cls: 0.4930 - final_regr: 0.3330
88/240 [==========>...................] - ETA: 12:09 - rpn_cls: 2.0433 - rpn_regr: 0.3032 - final_cls: 0.4938 - final_regr: 0.3331
89/240 [==========>...................] - ETA: 12:05 - rpn_cls: 2.0416 - rpn_regr: 0.3033 - final_cls: 0.4947 - final_regr: 0.3333
90/240 [==========>...................] - ETA: 12:01 - rpn_cls: 2.0398 - rpn_regr: 0.3034 - final_cls: 0.4955 - final_regr: 0.3335
91/240 [==========>...................] - ETA: 11:55 - rpn_cls: 2.0380 - rpn_regr: 0.3035 - final_cls: 0.4964 - final_regr: 0.3337
92/240 [==========>...................] - ETA: 11:51 - rpn_cls: 2.0361 - rpn_regr: 0.3035 - final_cls: 0.4972 - final_regr: 0.3338
93/240 [==========>...................] - ETA: 11:46 - rpn_cls: 2.0341 - rpn_regr: 0.3035 - final_cls: 0.4979 - final_regr: 0.3340
94/240 [==========>...................] - ETA: 11:40 - rpn_cls: 2.0328 - rpn_regr: 0.3036 - final_cls: 0.4986 - final_regr: 0.3341
95/240 [==========>...................] - ETA: 11:35 - rpn_cls: 2.0314 - rpn_regr: 0.3037 - final_cls: 0.4993 - final_regr: 0.3342
96/240 [===========>..................] - ETA: 11:29 - rpn_cls: 2.0299 - rpn_regr: 0.3038 - final_cls: 0.5000 - final_regr: 0.3342
97/240 [===========>..................] - ETA: 11:24 - rpn_cls: 2.0283 - rpn_regr: 0.3038 - final_cls: 0.5006 - final_regr: 0.3343
98/240 [===========>..................] - ETA: 11:19 - rpn_cls: 2.0267 - rpn_regr: 0.3039 - final_cls: 0.5013 - final_regr: 0.3344
99/240 [===========>..................] - ETA: 11:14 - rpn_cls: 2.0252 - rpn_regr: 0.3040 - final_cls: 0.5018 - final_regr: 0.3344
100/240 [===========>..................] - ETA: 11:09 - rpn_cls: 2.0235 - rpn_regr: 0.3040 - final_cls: 0.5025 - final_regr: 0.3345
101/240 [===========>..................] - ETA: 11:05 - rpn_cls: 2.0218 - rpn_regr: 0.3041 - final_cls: 0.5031 - final_regr: 0.3345
102/240 [===========>..................] - ETA: 11:00 - rpn_cls: 2.0200 - rpn_regr: 0.3041 - final_cls: 0.5037 - final_regr: 0.3346
103/240 [===========>..................] - ETA: 10:55 - rpn_cls: 2.0181 - rpn_regr: 0.3041 - final_cls: 0.5043 - final_regr: 0.3346
104/240 [============>.................] - ETA: 10:50 - rpn_cls: 2.0161 - rpn_regr: 0.3041 - final_cls: 0.5048 - final_regr: 0.3345
105/240 [============>.................] - ETA: 10:45 - rpn_cls: 2.0139 - rpn_regr: 0.3041 - final_cls: 0.5054 - final_regr: 0.3345
106/240 [============>.................] - ETA: 10:40 - rpn_cls: 2.0119 - rpn_regr: 0.3040 - final_cls: 0.5059 - final_regr: 0.3345
107/240 [============>.................] - ETA: 10:35 - rpn_cls: 2.0098 - rpn_regr: 0.3039 - final_cls: 0.5065 - final_regr: 0.3344
108/240 [============>.................] - ETA: 10:30 - rpn_cls: 2.0076 - rpn_regr: 0.3039 - final_cls: 0.5070 - final_regr: 0.3345
109/240 [============>.................] - ETA: 10:26 - rpn_cls: 2.0058 - rpn_regr: 0.3039 - final_cls: 0.5075 - final_regr: 0.3345
110/240 [============>.................] - ETA: 10:21 - rpn_cls: 2.0040 - rpn_regr: 0.3038 - final_cls: 0.5081 - final_regr: 0.3346
111/240 [============>.................] - ETA: 10:17 - rpn_cls: 2.0022 - rpn_regr: 0.3038 - final_cls: 0.5087 - final_regr: 0.3346
112/240 [=============>................] - ETA: 10:11 - rpn_cls: 2.0006 - rpn_regr: 0.3038 - final_cls: 0.5092 - final_regr: 0.3346
113/240 [=============>................] - ETA: 10:06 - rpn_cls: 1.9988 - rpn_regr: 0.3038 - final_cls: 0.5097 - final_regr: 0.3347
114/240 [=============>................] - ETA: 10:02 - rpn_cls: 1.9970 - rpn_regr: 0.3037 - final_cls: 0.5102 - final_regr: 0.3347
115/240 [=============>................] - ETA: 9:58 - rpn_cls: 1.9954 - rpn_regr: 0.3037 - final_cls: 0.5106 - final_regr: 0.3347
116/240 [=============>................] - ETA: 9:53 - rpn_cls: 1.9938 - rpn_regr: 0.3036 - final_cls: 0.5110 - final_regr: 0.3348
117/240 [=============>................] - ETA: 9:48 - rpn_cls: 1.9921 - rpn_regr: 0.3036 - final_cls: 0.5113 - final_regr: 0.3348
118/240 [=============>................] - ETA: 9:43 - rpn_cls: 1.9903 - rpn_regr: 0.3035 - final_cls: 0.5117 - final_regr: 0.3348
119/240 [=============>................] - ETA: 9:37 - rpn_cls: 1.9885 - rpn_regr: 0.3034 - final_cls: 0.5120 - final_regr: 0.3348
120/240 [==============>...............] - ETA: 9:32 - rpn_cls: 1.9869 - rpn_regr: 0.3033 - final_cls: 0.5124 - final_regr: 0.3348
121/240 [==============>...............] - ETA: 9:27 - rpn_cls: 1.9853 - rpn_regr: 0.3033 - final_cls: 0.5127 - final_regr: 0.3348
122/240 [==============>...............] - ETA: 9:22 - rpn_cls: 1.9837 - rpn_regr: 0.3032 - final_cls: 0.5131 - final_regr: 0.3348
123/240 [==============>...............] - ETA: 9:16 - rpn_cls: 1.9821 - rpn_regr: 0.3031 - final_cls: 0.5134 - final_regr: 0.3348
124/240 [==============>...............] - ETA: 9:11 - rpn_cls: 1.9804 - rpn_regr: 0.3030 - final_cls: 0.5137 - final_regr: 0.3348
125/240 [==============>...............] - ETA: 9:06 - rpn_cls: 1.9787 - rpn_regr: 0.3029 - final_cls: 0.5140 - final_regr: 0.3348
126/240 [==============>...............] - ETA: 9:01 - rpn_cls: 1.9771 - rpn_regr: 0.3028 - final_cls: 0.5143 - final_regr: 0.3347
127/240 [==============>...............] - ETA: 8:55 - rpn_cls: 1.9755 - rpn_regr: 0.3026 - final_cls: 0.5145 - final_regr: 0.3348
128/240 [===============>..............] - ETA: 8:50 - rpn_cls: 1.9738 - rpn_regr: 0.3025 - final_cls: 0.5148 - final_regr: 0.3348
129/240 [===============>..............] - ETA: 8:45 - rpn_cls: 1.9721 - rpn_regr: 0.3024 - final_cls: 0.5150 - final_regr: 0.3348
130/240 [===============>..............] - ETA: 8:40 - rpn_cls: 1.9703 - rpn_regr: 0.3022 - final_cls: 0.5153 - final_regr: 0.3348
131/240 [===============>..............] - ETA: 8:35 - rpn_cls: 1.9684 - rpn_regr: 0.3021 - final_cls: 0.5154 - final_regr: 0.3347
132/240 [===============>..............] - ETA: 8:30 - rpn_cls: 1.9664 - rpn_regr: 0.3019 - final_cls: 0.5157 - final_regr: 0.3347
133/240 [===============>..............] - ETA: 8:25 - rpn_cls: 1.9648 - rpn_regr: 0.3018 - final_cls: 0.5159 - final_regr: 0.3347
134/240 [===============>..............] - ETA: 8:20 - rpn_cls: 1.9630 - rpn_regr: 0.3017 - final_cls: 0.5161 - final_regr: 0.3347
135/240 [===============>..............] - ETA: 8:15 - rpn_cls: 1.9612 - rpn_regr: 0.3016 - final_cls: 0.5164 - final_regr: 0.3347
136/240 [================>.............] - ETA: 8:10 - rpn_cls: 1.9594 - rpn_regr: 0.3014 - final_cls: 0.5167 - final_regr: 0.3347
137/240 [================>.............] - ETA: 8:05 - rpn_cls: 1.9577 - rpn_regr: 0.3013 - final_cls: 0.5169 - final_regr: 0.3346
138/240 [================>.............] - ETA: 8:00 - rpn_cls: 1.9559 - rpn_regr: 0.3012 - final_cls: 0.5172 - final_regr: 0.3346
139/240 [================>.............] - ETA: 7:54 - rpn_cls: 1.9542 - rpn_regr: 0.3010 - final_cls: 0.5174 - final_regr: 0.3346
140/240 [================>.............] - ETA: 7:49 - rpn_cls: 1.9524 - rpn_regr: 0.3009 - final_cls: 0.5177 - final_regr: 0.3346
141/240 [================>.............] - ETA: 7:45 - rpn_cls: 1.9507 - rpn_regr: 0.3007 - final_cls: 0.5179 - final_regr: 0.3345
142/240 [================>.............] - ETA: 7:39 - rpn_cls: 1.9490 - rpn_regr: 0.3006 - final_cls: 0.5181 - final_regr: 0.3345
143/240 [================>.............] - ETA: 7:35 - rpn_cls: 1.9472 - rpn_regr: 0.3004 - final_cls: 0.5184 - final_regr: 0.3345
144/240 [=================>............] - ETA: 7:30 - rpn_cls: 1.9454 - rpn_regr: 0.3002 - final_cls: 0.5187 - final_regr: 0.3344
145/240 [=================>............] - ETA: 7:25 - rpn_cls: 1.9436 - rpn_regr: 0.3001 - final_cls: 0.5189 - final_regr: 0.3344
146/240 [=================>............] - ETA: 7:20 - rpn_cls: 1.9418 - rpn_regr: 0.2999 - final_cls: 0.5192 - final_regr: 0.3343
147/240 [=================>............] - ETA: 7:15 - rpn_cls: 1.9401 - rpn_regr: 0.2997 - final_cls: 0.5194 - final_regr: 0.3343
148/240 [=================>............] - ETA: 7:10 - rpn_cls: 1.9385 - rpn_regr: 0.2996 - final_cls: 0.5196 - final_regr: 0.3343
149/240 [=================>............] - ETA: 7:06 - rpn_cls: 1.9367 - rpn_regr: 0.2994 - final_cls: 0.5198 - final_regr: 0.3342
150/240 [=================>............] - ETA: 7:01 - rpn_cls: 1.9350 - rpn_regr: 0.2992 - final_cls: 0.5200 - final_regr: 0.3342
151/240 [=================>............] - ETA: 6:56 - rpn_cls: 1.9333 - rpn_regr: 0.2991 - final_cls: 0.5202 - final_regr: 0.3341
152/240 [==================>...........] - ETA: 6:51 - rpn_cls: 1.9315 - rpn_regr: 0.2989 - final_cls: 0.5203 - final_regr: 0.3341
153/240 [==================>...........] - ETA: 6:46 - rpn_cls: 1.9297 - rpn_regr: 0.2987 - final_cls: 0.5205 - final_regr: 0.3340
154/240 [==================>...........] - ETA: 6:41 - rpn_cls: 1.9279 - rpn_regr: 0.2985 - final_cls: 0.5206 - final_regr: 0.3339
155/240 [==================>...........] - ETA: 6:36 - rpn_cls: 1.9262 - rpn_regr: 0.2983 - final_cls: 0.5208 - final_regr: 0.3339
156/240 [==================>...........] - ETA: 6:31 - rpn_cls: 1.9246 - rpn_regr: 0.2981 - final_cls: 0.5209 - final_regr: 0.3338
157/240 [==================>...........] - ETA: 6:26 - rpn_cls: 1.9230 - rpn_regr: 0.2979 - final_cls: 0.5211 - final_regr: 0.3337
158/240 [==================>...........] - ETA: 6:22 - rpn_cls: 1.9213 - rpn_regr: 0.2977 - final_cls: 0.5212 - final_regr: 0.3336
159/240 [==================>...........] - ETA: 6:16 - rpn_cls: 1.9196 - rpn_regr: 0.2975 - final_cls: 0.5213 - final_regr: 0.3335
160/240 [===================>..........] - ETA: 6:11 - rpn_cls: 1.9179 - rpn_regr: 0.2973 - final_cls: 0.5214 - final_regr: 0.3334
161/240 [===================>..........] - ETA: 6:07 - rpn_cls: 1.9161 - rpn_regr: 0.2971 - final_cls: 0.5214 - final_regr: 0.3333
162/240 [===================>..........] - ETA: 6:01 - rpn_cls: 1.9143 - rpn_regr: 0.2969 - final_cls: 0.5215 - final_regr: 0.3332
163/240 [===================>..........] - ETA: 5:57 - rpn_cls: 1.9129 - rpn_regr: 0.2968 - final_cls: 0.5215 - final_regr: 0.3331
164/240 [===================>..........] - ETA: 5:52 - rpn_cls: 1.9115 - rpn_regr: 0.2966 - final_cls: 0.5216 - final_regr: 0.3330
165/240 [===================>..........] - ETA: 5:47 - rpn_cls: 1.9101 - rpn_regr: 0.2964 - final_cls: 0.5216 - final_regr: 0.3329
166/240 [===================>..........] - ETA: 5:42 - rpn_cls: 1.9088 - rpn_regr: 0.2962 - final_cls: 0.5216 - final_regr: 0.3328
167/240 [===================>..........] - ETA: 5:37 - rpn_cls: 1.9076 - rpn_regr: 0.2960 - final_cls: 0.5217 - final_regr: 0.3327
168/240 [====================>.........] - ETA: 5:32 - rpn_cls: 1.9064 - rpn_regr: 0.2958 - final_cls: 0.5217 - final_regr: 0.3326
169/240 [====================>.........] - ETA: 5:27 - rpn_cls: 1.9052 - rpn_regr: 0.2956 - final_cls: 0.5217 - final_regr: 0.3325
170/240 [====================>.........] - ETA: 5:22 - rpn_cls: 1.9039 - rpn_regr: 0.2954 - final_cls: 0.5217 - final_regr: 0.3324
171/240 [====================>.........] - ETA: 5:18 - rpn_cls: 1.9027 - rpn_regr: 0.2952 - final_cls: 0.5217 - final_regr: 0.3323
172/240 [====================>.........] - ETA: 5:13 - rpn_cls: 1.9015 - rpn_regr: 0.2951 - final_cls: 0.5217 - final_regr: 0.3322
173/240 [====================>.........] - ETA: 5:08 - rpn_cls: 1.9003 - rpn_regr: 0.2949 - final_cls: 0.5218 - final_regr: 0.3321
174/240 [====================>.........] - ETA: 5:04 - rpn_cls: 1.8991 - rpn_regr: 0.2947 - final_cls: 0.5218 - final_regr: 0.3320
175/240 [====================>.........] - ETA: 4:59 - rpn_cls: 1.8979 - rpn_regr: 0.2945 - final_cls: 0.5218 - final_regr: 0.3319
176/240 [=====================>........] - ETA: 4:54 - rpn_cls: 1.8967 - rpn_regr: 0.2943 - final_cls: 0.5218 - final_regr: 0.3318
177/240 [=====================>........] - ETA: 4:49 - rpn_cls: 1.8955 - rpn_regr: 0.2941 - final_cls: 0.5218 - final_regr: 0.3317
178/240 [=====================>........] - ETA: 4:45 - rpn_cls: 1.8943 - rpn_regr: 0.2939 - final_cls: 0.5219 - final_regr: 0.3316
179/240 [=====================>........] - ETA: 4:40 - rpn_cls: 1.8930 - rpn_regr: 0.2937 - final_cls: 0.5219 - final_regr: 0.3316
180/240 [=====================>........] - ETA: 4:35 - rpn_cls: 1.8917 - rpn_regr: 0.2935 - final_cls: 0.5219 - final_regr: 0.3315
181/240 [=====================>........] - ETA: 4:31 - rpn_cls: 1.8904 - rpn_regr: 0.2934 - final_cls: 0.5219 - final_regr: 0.3314
182/240 [=====================>........] - ETA: 4:26 - rpn_cls: 1.8890 - rpn_regr: 0.2932 - final_cls: 0.5220 - final_regr: 0.3313
183/240 [=====================>........] - ETA: 4:21 - rpn_cls: 1.8876 - rpn_regr: 0.2930 - final_cls: 0.5220 - final_regr: 0.3313
184/240 [======================>.......] - ETA: 4:16 - rpn_cls: 1.8863 - rpn_regr: 0.2929 - final_cls: 0.5220 - final_regr: 0.3312
185/240 [======================>.......] - ETA: 4:12 - rpn_cls: 1.8850 - rpn_regr: 0.2927 - final_cls: 0.5221 - final_regr: 0.3312
186/240 [======================>.......] - ETA: 4:07 - rpn_cls: 1.8837 - rpn_regr: 0.2925 - final_cls: 0.5221 - final_regr: 0.3311
187/240 [======================>.......] - ETA: 4:03 - rpn_cls: 1.8824 - rpn_regr: 0.2924 - final_cls: 0.5222 - final_regr: 0.3311
188/240 [======================>.......] - ETA: 3:58 - rpn_cls: 1.8811 - rpn_regr: 0.2922 - final_cls: 0.5222 - final_regr: 0.3310
189/240 [======================>.......] - ETA: 3:53 - rpn_cls: 1.8797 - rpn_regr: 0.2920 - final_cls: 0.5222 - final_regr: 0.3309
190/240 [======================>.......] - ETA: 3:49 - rpn_cls: 1.8783 - rpn_regr: 0.2918 - final_cls: 0.5223 - final_regr: 0.3309
191/240 [======================>.......] - ETA: 3:44 - rpn_cls: 1.8770 - rpn_regr: 0.2917 - final_cls: 0.5223 - final_regr: 0.3308
192/240 [=======================>......] - ETA: 3:39 - rpn_cls: 1.8756 - rpn_regr: 0.2915 - final_cls: 0.5223 - final_regr: 0.3307
193/240 [=======================>......] - ETA: 3:35 - rpn_cls: 1.8742 - rpn_regr: 0.2913 - final_cls: 0.5223 - final_regr: 0.3307
194/240 [=======================>......] - ETA: 3:30 - rpn_cls: 1.8728 - rpn_regr: 0.2911 - final_cls: 0.5224 - final_regr: 0.3306
195/240 [=======================>......] - ETA: 3:25 - rpn_cls: 1.8715 - rpn_regr: 0.2910 - final_cls: 0.5224 - final_regr: 0.3306
196/240 [=======================>......] - ETA: 3:21 - rpn_cls: 1.8702 - rpn_regr: 0.2908 - final_cls: 0.5224 - final_regr: 0.3305
197/240 [=======================>......] - ETA: 3:16 - rpn_cls: 1.8690 - rpn_regr: 0.2906 - final_cls: 0.5224 - final_regr: 0.3305
198/240 [=======================>......] - ETA: 3:11 - rpn_cls: 1.8678 - rpn_regr: 0.2904 - final_cls: 0.5224 - final_regr: 0.3305
199/240 [=======================>......] - ETA: 3:06 - rpn_cls: 1.8666 - rpn_regr: 0.2902 - final_cls: 0.5225 - final_regr: 0.3304
200/240 [========================>.....] - ETA: 3:02 - rpn_cls: 1.8654 - rpn_regr: 0.2901 - final_cls: 0.5225 - final_regr: 0.3304
201/240 [========================>.....] - ETA: 2:57 - rpn_cls: 1.8641 - rpn_regr: 0.2899 - final_cls: 0.5225 - final_regr: 0.3303
202/240 [========================>.....] - ETA: 2:52 - rpn_cls: 1.8629 - rpn_regr: 0.2897 - final_cls: 0.5225 - final_regr: 0.3303
203/240 [========================>.....] - ETA: 2:48 - rpn_cls: 1.8616 - rpn_regr: 0.2895 - final_cls: 0.5225 - final_regr: 0.3303
204/240 [========================>.....] - ETA: 2:43 - rpn_cls: 1.8603 - rpn_regr: 0.2894 - final_cls: 0.5225 - final_regr: 0.3302
205/240 [========================>.....] - ETA: 2:39 - rpn_cls: 1.8589 - rpn_regr: 0.2892 - final_cls: 0.5225 - final_regr: 0.3302
206/240 [========================>.....] - ETA: 2:34 - rpn_cls: 1.8577 - rpn_regr: 0.2890 - final_cls: 0.5225 - final_regr: 0.3301
207/240 [========================>.....] - ETA: 2:29 - rpn_cls: 1.8564 - rpn_regr: 0.2888 - final_cls: 0.5225 - final_regr: 0.3301
208/240 [=========================>....] - ETA: 2:25 - rpn_cls: 1.8552 - rpn_regr: 0.2886 - final_cls: 0.5225 - final_regr: 0.3301
209/240 [=========================>....] - ETA: 2:20 - rpn_cls: 1.8540 - rpn_regr: 0.2885 - final_cls: 0.5225 - final_regr: 0.3300
210/240 [=========================>....] - ETA: 2:16 - rpn_cls: 1.8527 - rpn_regr: 0.2883 - final_cls: 0.5225 - final_regr: 0.3300
211/240 [=========================>....] - ETA: 2:11 - rpn_cls: 1.8515 - rpn_regr: 0.2881 - final_cls: 0.5225 - final_regr: 0.3300
212/240 [=========================>....] - ETA: 2:06 - rpn_cls: 1.8502 - rpn_regr: 0.2879 - final_cls: 0.5225 - final_regr: 0.3300
213/240 [=========================>....] - ETA: 2:02 - rpn_cls: 1.8489 - rpn_regr: 0.2878 - final_cls: 0.5225 - final_regr: 0.3300
214/240 [=========================>....] - ETA: 1:57 - rpn_cls: 1.8476 - rpn_regr: 0.2876 - final_cls: 0.5225 - final_regr: 0.3300
215/240 [=========================>....] - ETA: 1:52 - rpn_cls: 1.8464 - rpn_regr: 0.2874 - final_cls: 0.5225 - final_regr: 0.3299
216/240 [==========================>...] - ETA: 1:48 - rpn_cls: 1.8451 - rpn_regr: 0.2872 - final_cls: 0.5225 - final_regr: 0.3300
217/240 [==========================>...] - ETA: 1:43 - rpn_cls: 1.8438 - rpn_regr: 0.2870 - final_cls: 0.5226 - final_regr: 0.3300
218/240 [==========================>...] - ETA: 1:39 - rpn_cls: 1.8424 - rpn_regr: 0.2868 - final_cls: 0.5226 - final_regr: 0.3300
219/240 [==========================>...] - ETA: 1:34 - rpn_cls: 1.8411 - rpn_regr: 0.2867 - final_cls: 0.5226 - final_regr: 0.3300
220/240 [==========================>...] - ETA: 1:30 - rpn_cls: 1.8397 - rpn_regr: 0.2865 - final_cls: 0.5226 - final_regr: 0.3300
221/240 [==========================>...] - ETA: 1:25 - rpn_cls: 1.8384 - rpn_regr: 0.2863 - final_cls: 0.5226 - final_regr: 0.3300
222/240 [==========================>...] - ETA: 1:20 - rpn_cls: 1.8371 - rpn_regr: 0.2861 - final_cls: 0.5226 - final_regr: 0.3300
223/240 [==========================>...] - ETA: 1:16 - rpn_cls: 1.8357 - rpn_regr: 0.2859 - final_cls: 0.5226 - final_regr: 0.3300
224/240 [===========================>..] - ETA: 1:11 - rpn_cls: 1.8343 - rpn_regr: 0.2857 - final_cls: 0.5226 - final_regr: 0.3299
225/240 [===========================>..] - ETA: 1:07 - rpn_cls: 1.8330 - rpn_regr: 0.2856 - final_cls: 0.5226 - final_regr: 0.3299
226/240 [===========================>..] - ETA: 1:02 - rpn_cls: 1.8317 - rpn_regr: 0.2854 - final_cls: 0.5226 - final_regr: 0.3299
227/240 [===========================>..] - ETA: 58s - rpn_cls: 1.8304 - rpn_regr: 0.2852 - final_cls: 0.5226 - final_regr: 0.3299
228/240 [===========================>..] - ETA: 53s - rpn_cls: 1.8291 - rpn_regr: 0.2850 - final_cls: 0.5226 - final_regr: 0.3299
229/240 [===========================>..] - ETA: 49s - rpn_cls: 1.8280 - rpn_regr: 0.2848 - final_cls: 0.5226 - final_regr: 0.3299
230/240 [===========================>..] - ETA: 44s - rpn_cls: 1.8269 - rpn_regr: 0.2847 - final_cls: 0.5226 - final_regr: 0.3299
231/240 [===========================>..] - ETA: 40s - rpn_cls: 1.8258 - rpn_regr: 0.2845 - final_cls: 0.5227 - final_regr: 0.3298
232/240 [============================>.] - ETA: 35s - rpn_cls: 1.8247 - rpn_regr: 0.2843 - final_cls: 0.5227 - final_regr: 0.3298
233/240 [============================>.] - ETA: 31s - rpn_cls: 1.8236 - rpn_regr: 0.2841 - final_cls: 0.5227 - final_regr: 0.3298
234/240 [============================>.] - ETA: 26s - rpn_cls: 1.8225 - rpn_regr: 0.2840 - final_cls: 0.5227 - final_regr: 0.3298
235/240 [============================>.] - ETA: 22s - rpn_cls: 1.8214 - rpn_regr: 0.2838 - final_cls: 0.5227 - final_regr: 0.3298
236/240 [============================>.] - ETA: 17s - rpn_cls: 1.8203 - rpn_regr: 0.2836 - final_cls: 0.5227 - final_regr: 0.3298
237/240 [============================>.] - ETA: 13s - rpn_cls: 1.8194 - rpn_regr: 0.2834 - final_cls: 0.5227 - final_regr: 0.3298
238/240 [============================>.] - ETA: 8s - rpn_cls: 1.8184 - rpn_regr: 0.2833 - final_cls: 0.5227 - final_regr: 0.3298
239/240 [============================>.] - ETA: 4s - rpn_cls: 1.8174 - rpn_regr: 0.2831 - final_cls: 0.5227 - final_regr: 0.3297
240/240 [==============================] - 1072s 4s/step - rpn_cls: 1.8164 - rpn_regr: 0.2829 - final_cls: 0.5227 - final_regr: 0.3297
Mean number of bounding boxes from RPN overlapping ground truth boxes: 9.629166666666666
Classifier accuracy for bounding boxes from RPN: 0.7625
Loss RPN classifier: 1.573888585749231
Loss RPN regression: 0.24293259833551323
Loss Detector classifier: 0.5250074003182817
Loss Detector regression: 0.3263928147032857
Total loss: 2.6682213991063115
Elapsed time: 1071.5359601974487
Total loss decreased from 2.883 to 2.6682213991063115, saving model.
Training complete, exiting.Evaluation
This module is used to perform visual evaluation of the model. I have only trained the model in 21 epochs, this type of models needs a lot of training to perform well. The result from an evaluation is shown below the code.
# Import libraries
import os
import cv2
import time
import keras
import random
import numpy as np
import matplotlib.pyplot as plt
import annytab.frcnn.common as common
import annytab.frcnn.model_builder as mb
import annytab.frcnn.layers as layers
# The main entry point for this module
def main():
# Create configuration
config = common.Config()
config.img_base_path = 'C:\\DATA\\Python-data\\open-images-v5\\imgs\\'
config.model_path = 'C:\\DATA\\Python-data\\open-images-v5\\frcnn\\training_model.h5'
config.use_horizontal_flips = False
config.use_vertical_flips = False
config.rot_90 = False
config.num_rois = 4
# Set the gpu to use
common.setup_gpu('cpu')
#common.setup_gpu(0)
# Create a dictionary with classes and switch key values
classes = {'Laptop': 0, 'Beer': 1, 'Goat': 2, 'bg': 3}
mappings = {v: k for k, v in classes.items()}
class_to_color = {mappings[v]: np.random.randint(0, 255, 3) for v in mappings}
print(mappings)
# Get inference models
model_rpn, model_classifier, model_classifier_only = mb.get_inference_models(config, len(classes), weights_path=config.model_path)
# Get a list of test images
images = os.listdir(config.img_base_path + 'test')
# Randomize images to get different images each time
random.shuffle(images)
# If the box classification value is less than this, we ignore this box
bbox_threshold = 0.7
# Start visual evaluation
for i, name in enumerate(images):
# Limit the evaluation to 4 images
if i > 4:
break;
# Print the name and start a timer
print(name)
st = time.time()
# Get the filepath
filepath = os.path.join(config.img_base_path + 'test', name)
# Get the image
img = cv2.imread(filepath)
# Format the image
X, ratio = common.format_img(img, config)
X = np.transpose(X, (0, 2, 3, 1))
# Get output layer Y1, Y2 from the RPN and the feature maps F
# Y1: y_rpn_cls
# Y2: y_rpn_regr
[Y1, Y2, F] = model_rpn.predict(X)
# Get bboxes by applying NMS
# R.shape = (300, 4)
R = layers.rpn_to_roi(Y1, Y2, config, keras.backend.image_data_format(), overlap_thresh=0.7)
# Convert from (x1,y1,x2,y2) to (x,y,w,h)
R[:, 2] -= R[:, 0]
R[:, 3] -= R[:, 1]
# Apply the spatial pyramid pooling to the proposed regions
bboxes = {}
probs = {}
for jk in range(R.shape[0]//config.num_rois + 1):
ROIs = np.expand_dims(R[config.num_rois*jk:config.num_rois*(jk+1), :], axis=0)
if ROIs.shape[1] == 0:
break
if jk == R.shape[0]//config.num_rois:
#pad R
curr_shape = ROIs.shape
target_shape = (curr_shape[0],config.num_rois,curr_shape[2])
ROIs_padded = np.zeros(target_shape).astype(ROIs.dtype)
ROIs_padded[:, :curr_shape[1], :] = ROIs
ROIs_padded[0, curr_shape[1]:, :] = ROIs[0, 0, :]
ROIs = ROIs_padded
[P_cls, P_regr] = model_classifier_only.predict([F, ROIs])
# Calculate bboxes coordinates on resized image
for ii in range(P_cls.shape[1]):
# Ignore 'bg' class
if np.max(P_cls[0, ii, :]) < bbox_threshold or np.argmax(P_cls[0, ii, :]) == (P_cls.shape[2] - 1):
continue
# Get the class name
cls_name = mappings[np.argmax(P_cls[0, ii, :])]
if cls_name not in bboxes:
bboxes[cls_name] = []
probs[cls_name] = []
(x, y, w, h) = ROIs[0, ii, :]
cls_num = np.argmax(P_cls[0, ii, :])
try:
(tx, ty, tw, th) = P_regr[0, ii, 4*cls_num:4*(cls_num+1)]
tx /= C.classifier_regr_std[0]
ty /= C.classifier_regr_std[1]
tw /= C.classifier_regr_std[2]
th /= C.classifier_regr_std[3]
x, y, w, h = layers.apply_regr(x, y, w, h, tx, ty, tw, th)
except:
pass
bboxes[cls_name].append([config.rpn_stride*x, config.rpn_stride*y, config.rpn_stride*(x+w), config.rpn_stride*(y+h)])
probs[cls_name].append(np.max(P_cls[0, ii, :]))
all_dets = []
for key in bboxes:
bbox = np.array(bboxes[key])
new_boxes, new_probs = common.non_max_suppression_fast(bbox, np.array(probs[key]), overlap_thresh=0.2)
for jk in range(new_boxes.shape[0]):
(x1, y1, x2, y2) = new_boxes[jk,:]
# Calculate real coordinates on original image
(real_x1, real_y1, real_x2, real_y2) = common.get_real_coordinates(ratio, x1, y1, x2, y2)
cv2.rectangle(img,(real_x1, real_y1), (real_x2, real_y2), (int(class_to_color[key][0]), int(class_to_color[key][1]), int(class_to_color[key][2])),4)
textLabel = '{0}: {1}'.format(key,int(100*new_probs[jk]))
all_dets.append((key,100*new_probs[jk]))
(retval,baseLine) = cv2.getTextSize(textLabel,cv2.FONT_HERSHEY_COMPLEX,1,1)
textOrg = (real_x1, real_y1-0)
cv2.rectangle(img, (textOrg[0] - 5, textOrg[1]+baseLine - 5), (textOrg[0]+retval[0] + 5, textOrg[1]-retval[1] - 5), (0, 0, 0), 1)
cv2.rectangle(img, (textOrg[0] - 5,textOrg[1]+baseLine - 5), (textOrg[0]+retval[0] + 5, textOrg[1]-retval[1] - 5), (255, 255, 255), -1)
cv2.putText(img, textLabel, textOrg, cv2.FONT_HERSHEY_DUPLEX, 1, (0, 0, 0), 1)
print('Elapsed time = {0}'.format(time.time() - st))
print(all_dets)
plt.figure(figsize=(10,10))
plt.grid()
plt.imshow(cv2.cvtColor(img,cv2.COLOR_BGR2RGB))
plt.savefig('C:\\DATA\\Python-data\\open-images-v5\\frcnn\\plots\\evaluate_' + str(i) + '.png')
#plt.show()
# Tell python to run main method
if __name__ == "__main__": main()
[URGENT]
In layers,
line 7 why did you import anytab.frcnn ??
Do you mean “import annytab.frcnn.common”? Because i make calls to functions in “common.py”, annytab.frcnn is a namespace (folder structure).
do you have any github link for this code ?