Separate job runner compute logic 
This is a proposal to separate management functions logic like run, backfill,skip,process, set_crashed, exceed_maximun_duration, ect.) from job runner. Job runner only compute, pre-compute and post-compute a response without other responsabilities (Like a operator).
This will make easier to switch to an orchestrator some day.

I introduce "job_manager" class which will be in charge of doing extra actions/ validations related to job pipelines an will invoke job runner whenever it is needed to compute.
A job runner do not need to know about other actions like store cache, validate if there already exist a parallel cache, validate if it has to skip the compute or other actions different than "compute" the response.
I also introduce parent classes dataset_job_runner, config_job_runner and split_job_runner (They are different in the attributes needed by each operator, it will close https://github.com/huggingface/datasets-server/issues/1074).
I think this way we could extend in the future more levels of granularity (Like partitions for example for https://github.com/huggingface/datasets-server/issues/1087

Sorry if there are many files but don't be afraid, most of them are related to changes in the imports because I moved some classes/functions to other files and changes on the tests now that process method does not exist in job runner. 
Separate job runner compute logic : This is a proposal to separate management functions logic like run, backfill,skip,process, set_crashed, exceed_maximun_duration, ect.) from job runner. Job runner only compute, pre-compute and post-compute a response without other responsabilities (Like a operator).
This will make easier to switch to an orchestrator some day.

I introduce "job_manager" class which will be in charge of doing extra actions/ validations related to job pipelines an will invoke job runner whenever it is needed to compute.
A job runner do not need to know about other actions like store cache, validate if there already exist a parallel cache, validate if it has to skip the compute or other actions different than "compute" the response.
I also introduce parent classes dataset_job_runner, config_job_runner and split_job_runner (They are different in the attributes needed by each operator, it will close https://github.com/huggingface/datasets-server/issues/1074).
I think this way we could extend in the future more levels of granularity (Like partitions for example for https://github.com/huggingface/datasets-server/issues/1087

Sorry if there are many files but don't be afraid, most of them are related to changes in the imports because I moved some classes/functions to other files and changes on the tests now that process method does not exist in job runner. 