Other job schedulers¶
eHive has a generic interface named Meadow that describes how to interact with an underlying grid scheduler (submit jobs, query job’s status, etc.). eHive is distributed with two meadow implementations:
- LOCAL
- A simple meadow that submits jobs locally via
system()(i.e.fork()). It is inherently limited by the specification of the machine Beekeeper is running on. The implementation is not able to control the memory consumption of the jobs vs the memory available. All jobs are supposed to be using one core each, and the total number of jobs is limited by (i) a machine-specific configuration found in thehive_config.jsonfile and (ii) the analysis_capacity and hive_capacity mechanisms. - LSF
- A meadow that supports IBM Platform LSF This meadow is extensively used by the Ensembl project and is regularly updated. It is fully implemented and supports workloads reaching thousands of parallel jobs.
Other meadows have been contributed to the project, though sometimes not all the features are implemented. Being developed outside of the main codebase, they may be at times out of sync with the latest version of eHive. Nevertheless, several are continuously tested on Travis CI using single-machine Docker installations. Refer to the documentation or README in each of those repositories to know more about their compatibility and support.
- SGE
- A meadow that supports Sun Grid Engine (now known as Oracle Grid Engine). Available for download on GitHub at Ensembl/ensembl-hive-sge.
- HTCondor
- A meadow that supports HTCondor. Available for download on GitHub at Ensembl/ensembl-hive-htcondor.
- PBSPro
- A meadow that supports PBS Pro. Available for download on GitHub at Ensembl/ensembl-hive-pbspro.
- SLURM
- A meadow that supports Slurm. Available for download on GitHub at tweep/ensembl-hive-slurm.
- DockerSwarm
- A meadow that can control and run on Docker Swarm. Available for download on GitHub at Ensembl/ensembl-hive-docker-swarm. See Docker Swarm for more information.
The table below lists the capabilities of each meadow, and whether they are available and implemented:
| Meadow | Submit jobs | Query job status | Kill job | Job limiter and resource management | Post-mortem inspection of resource usage |
|---|---|---|---|---|---|
| LOCAL | Yes | Yes | Yes | Partially implemented | Not available |
| LSF | Yes | Yes | Yes | Yes | Yes |
| SGE | Yes | Yes | Yes | Yes | Not implemented |
| HTCondor | Yes | Yes | Yes | Yes | Not implemented |
| PBSPro | Yes | Yes | Yes | Yes | Not implemented |
| SLURM | Yes | Yes | Yes | Yes | Yes |
| DockerSwarm | Yes | Yes | Not implemented | Yes | Not implemented |