9 Hive based processing is a concept based on a more controlled version
10 of an autonomous agent type system. Each worker is not told what to
do
11 (like a centralized control system - like the current pipeline system)
12 but rather queries a central database
for jobs (give me jobs).
14 Each worker is linked to an analysis_id, registers its
self on creation
15 into the Hive, creates a RunnableDB instance of the Analysis->module,
16 gets relevant configuration information from the database, does its
17 work, creates the next layer of job entries by interfacing to
18 the DataflowRuleAdaptor to determine the analyses it needs to pass its
19 output data to and creates jobs on the database of the next analysis.
20 It repeats
this cycle until it has lived its lifetime or until there are no
21 more jobs left to process.
22 The lifetime limit is a safety limit to prevent these from
'infecting'
23 a system and sitting on a compute node
for longer than is socially exceptable.
24 This is primarily needed on compute resources like an LSF system where jobs
25 are not preempted and
run until they are done.
27 The Queen
's primary job is to create Workers to get the work done.
28 As part of this, she is also responsible for summarizing the status of the
29 analyses by querying the jobs, summarizing, and updating the
30 analysis_stats table. From this she is also responsible for monitoring and
31 'unblocking
' analyses via the analysis_ctrl_rules.
32 The Queen is also responsible for freeing up jobs that were claimed by Workers
33 that died unexpectedly so that other workers can take over the work.
35 The Beekeeper is in charge of interfacing between the Queen and a compute resource
36 or 'compute farm
'. Its job is to query Queens if they need any workers and to
37 send the requested number of workers to open machines via the runWorker.pl script.
38 It is also responsible for interfacing with the Queen to identify workers which died
39 unexpectedly so that she can free the dead workers unfinished jobs.
43 Copyright [1999-2015] Wellcome Trust Sanger Institute and the EMBL-European Bioinformatics Institute
44 Copyright [2016-2024] EMBL-European Bioinformatics Institute
46 Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
47 You may obtain a copy of the License at
49 http://www.apache.org/licenses/LICENSE-2.0
51 Unless required by applicable law or agreed to in writing, software distributed under the License
52 is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
53 See the License for the specific language governing permissions and limitations under the License.
57 Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
61 package Bio::EnsEMBL::Hive;