Other job schedulers

eHive has a generic interface named Meadow that describes how to interact with an underlying grid scheduler (submit jobs, query job’s status, etc.). eHive is distributed with two meadow implementations:

A simple meadow that submits jobs locally via system() (i.e. fork()). It is inherently limited by the specification of the machine Beekeeper is running on. The implementation is not able to control the memory consumption of the jobs vs the memory available. All jobs are supposed to be using one core each, and the total number of jobs is limited by (i) a machine-specific configuration found in the hive_config.json file and (ii) the analysis_capacity and hive_capacity mechanisms.
A meadow that supports IBM Platform LSF This meadow is extensively used by the Ensembl project and is regularly updated. It is fully implemented and supports workloads reaching thousands of parallel jobs.

Other meadows have been contributed to the project, though sometimes not all the features are implemented. Being developed outside of the main codebase, they may be at times out of sync with the latest version of eHive. Nevertheless, several are continuously tested on Travis CI using single-machine Docker installations. Refer to the documentation or README in each of those repositories to know more about their compatibility and support.

A meadow that supports Sun Grid Engine (now known as Oracle Grid Engine). Available for download on GitHub at Ensembl/ensembl-hive-sge.
A meadow that supports HTCondor. Available for download on GitHub at Ensembl/ensembl-hive-htcondor.
A meadow that supports PBS Pro. Available for download on GitHub at Ensembl/ensembl-hive-pbspro.
A meadow that supports Slurm. Available for download on GitHub at tweep/ensembl-hive-slurm.
A meadow that can control and run on Docker Swarm. Available for download on GitHub at Ensembl/ensembl-hive-docker-swarm. See Docker Swarm for more information.

The table below lists the capabilities of each meadow, and whether they are available and implemented:

Meadow Submit jobs Query job status Kill job Job limiter and resource management Post-mortem inspection of resource usage
LOCAL Yes Yes Yes Partially implemented Not available
LSF Yes Yes Yes Yes Yes
SGE Yes Yes Yes Yes Not implemented
HTCondor Yes Yes Yes Yes Not implemented
PBSPro Yes Yes Yes Yes Not implemented
SLURM Yes Yes Yes Yes Yes
DockerSwarm Yes Yes Not implemented Yes Not implemented