ensembl-hive  2.7.0
LongMult_conf.pm
Go to the documentation of this file.
1 =pod
2 
3 =head1 NAME
4 
5  Bio::EnsEMBL::Hive::Examples::LongMult::PipeConfig::LongMult_conf;
6 
7 =head1 SYNOPSIS
8 
9  # initialize the database and build the graph in it (it will also print the value of EHIVE_URL) :
10  init_pipeline.pl Bio::EnsEMBL::Hive::Examples::LongMult::PipeConfig::LongMult_conf -password <mypass>
11 
12  # optionally also seed it with your specific values:
13  seed_pipeline.pl -url $EHIVE_URL -logic_name take_b_apart -input_id '{ "a_multiplier" => "12345678", "b_multiplier" => "3359559666" }'
14 
15  # run the pipeline:
16  beekeeper.pl -url $EHIVE_URL -loop
17 
18 =head1 DESCRIPTION
19 
20  This is the PipeConfig file for the long multiplication pipeline example.
21  The main point of this pipeline is to provide an example of how to write Hive Runnables and link them together into a pipeline.
22 
23  Please refer to Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf module to understand the interface implemented here.
24 
25  The setting. let's assume we are given two loooooong numbers to multiply. reeeeally long.
26  soooo long that they do not fit into registers of the cpu and should be multiplied digit-by-digit.
27  For the purposes of this example we also assume this task is very computationally intensive and has to be done in parallel.
28 
29  The long multiplication pipeline consists of three "analyses" (types of tasks):
30  'take_b_apart', 'part_multiply' and 'add_together' that we use to examplify various features of the Hive.
31 
32  * A 'take_b_apart' job takes in two string parameters, 'a_multiplier' and 'b_multiplier',
33  takes the second one apart into digits, finds what _different_ digits are there,
34  creates several jobs of the 'part_multiply' analysis and one job of 'add_together' analysis.
35 
36  * A 'part_multiply' job takes in 'a_multiplier' and 'digit', multiplies them and accumulates the result in 'partial_product' accumulator.
37 
38  * An 'add_together' job waits for the first two analyses to complete,
39  takes in 'a_multiplier', 'b_multiplier' and 'partial_product' hash and produces the final result in 'final_result' table.
40 
41  Please see the implementation details in Runnable modules themselves.
42 
43 =head1 LICENSE
44 
45  See the NOTICE file distributed with this work for additional information
46  regarding copyright ownership.
47 
48  Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License.
49  You may obtain a copy of the License at
50 
51  http://www.apache.org/licenses/LICENSE-2.0
52 
53  Unless required by applicable law or agreed to in writing, software distributed under the License
54  is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
55  See the License for the specific language governing permissions and limitations under the License.
56 
57 =head1 CONTACT
58 
59  Please subscribe to the Hive mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/ehive-users to discuss Hive-related questions or to be notified of our updates
60 
61 =cut
62 
63 
64 package Bio::EnsEMBL::Hive::Examples::LongMult::PipeConfig::LongMult_conf;
65 
66 use strict;
67 use warnings;
68 
69 use base ('Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf'); # All Hive databases configuration files should inherit from HiveGeneric, directly or indirectly
70 use Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf; # Allow this particular config to use conditional dataflow and INPUT_PLUS
71 
72 
73 =head2 pipeline_create_commands
74 
75  Description : Implements pipeline_create_commands() interface method of Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf that lists the commands that will create and set up the Hive database.
76  In addition to the standard creation of the database and populating it with Hive tables and procedures it also creates two pipeline-specific tables used by Runnables to communicate.
77 
78 =cut
79 
80 sub pipeline_create_commands {
81  my ($self) = @_;
82  return [
83  @{$self->SUPER::pipeline_create_commands}, # inheriting database and hive tables' creation
84 
85  # additional tables needed for long multiplication pipeline's operation:
86  $self->db_cmd('CREATE TABLE final_result (a_multiplier varchar(255) NOT NULL, b_multiplier varchar(255) NOT NULL, result varchar(255) NOT NULL, PRIMARY KEY (a_multiplier, b_multiplier))'),
87  ];
88 }
89 
90 
91 =head2 pipeline_wide_parameters
92 
93  Description : Interface method that should return a hash of pipeline_wide_parameter_name->pipeline_wide_parameter_value pairs.
94  The value doesn't have to be a scalar, can be any Perl structure now (will be stringified and de-stringified automagically).
95  Please see existing PipeConfig modules for examples.
96 
97 =cut
98 
99 sub pipeline_wide_parameters {
100  my ($self) = @_;
101  return {
102  %{$self->SUPER::pipeline_wide_parameters}, # here we inherit anything from the base class
103 
104  'take_time' => 1,
105  };
106 }
107 
108 
109 =head2 pipeline_analyses
110 
111  Description : Implements pipeline_analyses() interface method of Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf that defines the structure of the pipeline: analyses, jobs, rules, etc.
112  Here it defines three analyses:
113  * 'take_b_apart' that is auto-seeded with a pair of jobs (to check the commutativity of multiplication).
114  Each job will dataflow (create more jobs) via branch #2 into 'part_multiply' and via branch #1 into 'add_together'.
115 
116  * 'part_multiply' with jobs fed from take_b_apart#2.
117  It multiplies input parameters 'a_multiplier' and 'digit' and dataflows 'partial_product' parameter into branch #1.
118 
119  * 'add_together' with jobs fed from take_b_apart#1.
120  It adds together results of partial multiplication computed by 'part_multiply'.
121  These results are accumulated in 'partial_product' hash.
122  Until the hash is complete the corresponding 'add_together' job is blocked by a semaphore.
123 
124 =cut
125 
126 sub pipeline_analyses {
127  my ($self) = @_;
128  return [
129  { -logic_name => 'take_b_apart',
130  -comment => "A factory that takes in #b_multiplier# and dataflows all its unique non-trivial digits into #2",
131  -module => 'Bio::EnsEMBL::Hive::Examples::LongMult::RunnableDB::DigitFactory',
132  -meadow_type => 'LOCAL', # do not bother the farm with such a simple task (and get it done faster)
133  -analysis_capacity => 2, # use per-analysis limiter
134  -input_ids => [
135  { 'a_multiplier' => '9650156169', 'b_multiplier' => '327358788' },
136  { 'a_multiplier' => '327358788', 'b_multiplier' => '9650156169' },
137  ],
138  -flow_into => {
139  # creating a semaphored fan of jobs; filtering by WHEN; using INPUT_PLUS or templates to top-up the hashes.
140  #
141  # A WHEN block is not a hash, so multiple occurences of each condition (including ELSE) is permitted.
142  '2->A' => WHEN(
143  '#digit#>1' => { 'part_multiply' => INPUT_PLUS() }, # make parent job's parameters available to the kids
144 # ELSE { 'part_multiply' => { 'a_multiplier' => '#a_multiplier#', 'digit' => '#digit#' } },
145  ),
146  # creating a semaphored funnel job to wait for the fan to complete and add the results:
147  'A->1' => [ 'add_together' ],
148  },
149  },
150 
151  { -logic_name => 'part_multiply',
152  -comment => "Multiplies #a_multiplier# by #digit# and dataflows #product# into #1",
153  -module => 'Bio::EnsEMBL::Hive::Examples::LongMult::RunnableDB::PartMultiply',
154  -analysis_capacity => 4, # use per-analysis limiter
155  -flow_into => {
156  1 => [ '?accu_name=partial_product&accu_address={digit}&accu_input_variable=product' ],
157  },
158  },
159 
160  { -logic_name => 'add_together',
161  -comment => "Takes in #a_multiplier#, #b_multiplier# and the #partial_product# hash and dataflows the product of #a_multiplier# and #b_multiplier# into #1",
162  -module => 'Bio::EnsEMBL::Hive::Examples::LongMult::RunnableDB::AddTogether',
163  -flow_into => {
164  1 => [ '?table_name=final_result' ],
165  },
166  },
167  ];
168 }
169 
170 1;
171 
Bio::EnsEMBL::Hive::Examples::LongMult::RunnableDB::DigitFactory
Definition: DigitFactory.pm:25
Bio::EnsEMBL::Hive::Version
Definition: Version.pm:19
main
public main()
Bio::EnsEMBL::Hive::PipeConfig::HiveGeneric_conf
Definition: HiveGeneric_conf.pm:54
Bio::EnsEMBL::Hive
Definition: Hive.pm:38
Bio::EnsEMBL::Hive::Examples::LongMult::PipeConfig::LongMult_conf
Definition: LongMult_conf.pm:47