AWS Transcoder already has some presets created for us to use. You can check if any of those suits your need, if not then you'll have to create your own preset using CreatePresetRequest.
Once you have the preset, you'll have to set it in job output object. Following is the pseudo code :
CreateJobOutput jobOutput = new CreateJobOutput();
jobOutput.setPresetId("1351620000001-000001");//can use already created presets jobReq.setOutput(jobOutput);
jobReq is the CreateJobRequest.