Allow specifying metric_definitions on ModelTrainer

[straygar]

Describe the feature you'd like
Similar to the Estimator abstraction, I would like to make use of SageMaker's CloudWatch metrics, based on job logs.

How would this feature be used? Please describe.
Not sure. Either an explicit argument in model_trainer or allowing the user to specify a create_job_args dictionary, to work around the abstraction if some API feature is not exposed:

trainer = ModelTrainer(
  ...,
  metric_definitions={
    {"Name": "training_iteration", "Regex": "Iteration (.+), Loss .+,"},
  }
)

(we could also just get rid of Name and Regex and just have a dict[MetricName, Pattern] there)

Describe alternatives you've considered
Using the Estimator class. I recently moved away from it, as the ModelTrainer abstraction makes more sense to me and other scientists/engineers on my team.

Additional context
n/a