How to use Tensorflow Model Zoo frozen graphs in an Estimator pipeline?

Question

As Tensorflow* says on their website, the Estimators API should genereally make most ML tasks more friendly. In the past I've been using Tensorflow's Model Zoo for object detection as I didn't (and still don't) have the hardware to fully train models from scratch.

Lately, I've been dealing with more and more images, and the need to feed data on the fly during training (fine-tuning) has become important. My current setup is essentially the legacy train.py script for training and feed_dict method for inference.

I've been reading a lot on Estimators API and their usual pipeline, but I just cannot find tutorials or help on how to use pretrained models like I have with that pipeline; all examples work on image classification and do not mix dataset creation and model training, which is a real puzzle.

_{*Tensorflow r1.14}

---

Step 1: Dataset creation

So, given my data in the following format:

data = [ 
  [img_path_1, boxes_1],
  [img_path_2, boxes_2],
  ...,
  [img_path_N, boxes_N]
]

Where each boxes_i is in the format:

[
  [label_A, xa, ya, xb, xb],
  [label_B, xa, ya, xb, xb],
  [label_A, xa, ya, xb, xb],
  ...,
  [label_D, xa, ya, xb, xb],
]

According to some tutorial and tweaked for object detection instead of classification, it seems like I would need to process images like this (further preprocessing and batching functions left out for simplicity):

dataset = tf.data.Dataset.from_tensor_slices(data)

def load_image_and_annotations(path_and_boxes):
    path, boxes = path_and_boxes
    image_string = tf.read_file(path)

    image = tf.image.decode_jpeg(image_string, channels=3)
    image = tf.image.convert_image_dtype(image, tf.float32)
    return (image, boxes)

dataset = dataset.map(load_image_and_annotations, num_parallel_calls=...)

and then make use of the tf.data.Iterator object to traverse the dataset as needed.

Step 2: Training

Here is mostly where I'm unsure of how to proceed. The pre-made estimators and custom estimators tutorials seem inappropriate for my task since I already have a frozen model that I wish to convert to an estimator (instead of fully building one).

---

How should I proceed to transition from my old setup to this new pipeline ?