Imagine there is a need for a “map-reduce” kind of task. For example, one has to implement a Large Neighbourhood Search (LNS) with sub-models being solved on parallel workers. There are two possible ways to go. Either the LNS procedure itself is implemented on the client side (standalone application) and the sub-models are solved through Optimization Server jobs. Or the LNS procedure itself is implemented in a task, and ran as a job on a given worker. In this case, the root job that handles LNS, creates sub-jobs that will run on other worker(s). The root job is then parent of sub-jobs in a job flow.
In the first case described above, job flows are created by any kind of custom client. See the Master API Section for APIs documentation. To create a job in a job flow through a custom client, call the createJob API and set the JobCreation.flowId field. See also the FlowApi to create or retrieve a job flow.
The second possibility is both root job and sub-jobs are executed on Optimization Server workers. In this case, the root task and the sub-task(s) have to be exposed by different workers to prevent the root job to wait indefinitely. The ability for a custom worker to create sub-jobs is a built-in feature. Indeed, the ExecutionContext, that is one of the task inputs, exposes the following methods:
Optional<String> createJob(JobCreationDTO jobCreation);
Optional<JobStatus> startJob(JobSubmitRequestDTO jobSubmitRequest);
Optional<SolutionDTO> startJobAndWaitForCompletion(JobSubmitRequestDTO jobSubmitRequest);
The first method “createJob” allows asking the master for a job creation. This job is not started. Below is an example of a root job creating a sub-job:
JobCreationDTO jobCreation = JobCreationDTO.builder()
.taskId("MySubModelSolvingTask")
.flowParent(JobRelationDTO.builder()
.relatedJobId(executionContext.getJobId())
.build())
.description("MyAwesomeSubJob")
.build();
Optional<String> subJobId = executionContext.createJob(jobCreation);
The non-blocking method “startJob” allows asking the master to start the job asynchronously. The blocking method “startJobAndWaitForCompletion” allows asking the master to start the job. Once the sub-job has completed, the solution is returned.
There is a dedicated view in the web console to monitor such a job flow.