Inheritance diagram for Bio::EnsEMBL::Hive::RunnableDB::DatabaseDumper:

Public Member Functions
public	param_defaults ()

public	fetch_input ()

protected	_get_table_list ()

Public Member Functions inherited from Bio::EnsEMBL::Hive::RunnableDB::SystemCmd
public	param_defaults ()

public	run ()

public	write_output ()

public Void	complete_early_if_branch_connected ()

Public Member Functions inherited from Bio::EnsEMBL::Hive::Process
public	new ()

public	life_cycle ()

public	say_with_header ()

public	enter_status ()

public	warning ()

public	param_defaults ()

public	fetch_input ()

public	run ()

public	write_output ()

public Bio::EnsEMBL::Hive::Worker	worker ()

public	execute_writes ()

public Bio::EnsEMBL::Hive::DBSQL::DBAdaptor	db ()

public Bio::EnsEMBL::Hive::DBSQL::DBConnection	dbc ()

public Bio::EnsEMBL::Hive::DBSQL::DBConnection	data_dbc ()

public Returns	run_system_command ()

public Bio::EnsEMBL::Hive::AnalysisJob	input_job ()

public	input_id ()

public	param ()

public	param_required ()

public	param_exists ()

public	param_is_defined ()

public	param_substitute ()

public	dataflow_output_id ()

public	dataflow_output_ids_from_json ()

public	throw ()

public This	complete_early ()

public Int	debug ()

public	worker_temp_directory ()

public	cleanup_worker_temp_directory ()

Detailed Description

Synopsis

standaloneJob.pl RunnableDB/DatabaseDumper.pm -exclude_ehive 1 -exclude_list 1 \

-table_list "['peptide_align_%']" -src_db_conn mysql://ensro@127.0.0.1:4313/mm14_compara_homology_67 -output_file ~/dump1.sql

Description

This is a Runnable to dump the tables of a database (by default,
all of them).

The following parameters are accepted:

 - src_db_conn : the connection parameters to the database to be
    dumped (by default, the current eHive database if available)

 - exclude_ehive [boolean=0] : do we exclude the eHive-specific tables
    from the dump

 - table_list [string or array of strings]: the list of tables
    to include in the dump. The '%' wildcard is accepted.

 - exclude_list [boolean=0] : do we consider 'table_list' as a list
    of tables to be excluded from the dump (instead of included)

 - output_file [string] : the file to write the dump to. If the filename
    ends with ".gz", the file is compressed with "gzip" (default parameters)

 - output_db [string] : URL of a database to write the dump to. In this
    mode, the Runnable acts like MySQLTransfer

 - skip_dump [boolean=0] : set this to 1 to skip the dump


The decision process regarding which tables should be dumped is quite complex.
The following sections explain the various scenarios.

1. eHive database

1.a. Hybrid database

If "table_list" is undefined or maps to an empty list, the list
of tables to be dumped is decided accordingly to "exclude_list" (EL)
and "exclude_ehive" (EH). "exclude_list" controls the whole list of
non-eHive tables.

EL EH    List of tables to dump
0  0  => all the tables
0  1  => all the tables, except the eHive ones
1  0  => all the tables, except the non-eHive ones = only the eHive tables
1  1  => both eHive and non-eHive tables are excluded = nothing is dumped

If "table_list" is defined to non-empty list T, the table of decision is:

EL EH    List of tables to dump
0  0  => all the tables in T + the eHive tables
0  1  => all the tables in T
1  0  => all the tables, except the ones in T
1  1  => all the tables, except the ones in T and the eHive ones

1.b. eHive-only database

The decision table can be simplified if the database only contains eHive tables.
In particular, the "exclude_list" and "table_list" parameters have no effect.

EH    List of tables to dump
0  => All the eHive tables, i.e. the whole database
1  => No eHive tables, i.e. nothing

2. non-eHive database

The "exclude_ehive" parameter is ignored.

empty "table_list":
EL    List of tables to dump
0  => all the tables
1  => all the tables are excluded = nothing is dumped

non-empty "table_list" T:
EL    List of tables to dump
0  => all the tables in T
1  => all the tables, except the ones in T

Definition at line 92 of file DatabaseDumper.pm.

Member Function Documentation

◆ _get_table_list()

protected Bio::EnsEMBL::Hive::RunnableDB::DatabaseDumper::_get_table_list ( )

Undocumented method

Code:

click to view

◆ fetch_input()

public Bio::EnsEMBL::Hive::RunnableDB::DatabaseDumper::fetch_input ( )

Undocumented method

Code:

click to view

sub fetch_input {
    my $self = shift @_;
 
    # The final list of tables
    my @tables = ();
    my @ignores = ();
 
    # Connection parameters
    my $src_db_conn  = $self->param('src_db_conn');
    my $src_dbc = $src_db_conn ? go_figure_dbc($src_db_conn) : $self->data_dbc;
    $self->param('src_dbc', $src_dbc);
 
    $self->input_job->transient_error(0);
    die 'Only the "mysql" driver is supported.' if $src_dbc->driver ne 'mysql';
 
    my @ehive_tables = ();
    {
        ## Only query the list of eHive tables if there is a "hive_meta" table
        my $meta_sth = $src_dbc->table_info(undef, undef, 'hive_meta');
        if ($meta_sth->fetchrow_arrayref) {
            my $src_dba = Bio::EnsEMBL::Hive::DBSQL::DBAdaptor->new( -dbconn => $src_dbc, -disconnect_when_inactive => 1, -no_sql_schema_version_check => 1 );
            @ehive_tables = (@{$src_dba->hive_pipeline->list_all_hive_tables}, @{$src_dba->hive_pipeline->list_all_hive_views});
            unless (@ehive_tables) {
                my @ref_ehive_tables = qw(hive_meta pipeline_wide_parameters worker dataflow_rule analysis_base analysis_ctrl_rule job accu log_message job_file analysis_data resource_description analysis_stats role msg progress resource_class worker_resource_usage);
                # The hard-coded list is comprehensive, so some tables may not be
                # in this database (which may be on a different version)
                push @ehive_tables, @{$self->_get_table_list($src_dbc, $_)} for @ref_ehive_tables;
            }
        }
        $meta_sth->finish();
    }
    $self->param('nb_ehive_tables', scalar(@ehive_tables));
 
    # Get the table list in either "tables" or "ignores"
    my $table_list = $self->_get_table_list($src_dbc, $self->param('table_list') || '');
    $self->say_with_header(sprintf("table_list: %d %s", scalar(@$table_list), join('/', @$table_list)));
    my $nothing_to_dump = 0;
 
    if ($self->param('exclude_list')) {
        push @ignores, @$table_list;
        $nothing_to_dump = 1 if !$self->param('table_list');
    } else {
        push @tables, @$table_list;
        $nothing_to_dump = 1 if $self->param('table_list') and !@$table_list;
    }
 
    # eHive tables are ignored if exclude_ehive is set
    if ($self->param('exclude_ehive')) {
        push @ignores, @ehive_tables;
    } elsif (@ehive_tables) {
        if (@tables || $nothing_to_dump) {
            push @tables, @ehive_tables;
            $nothing_to_dump = 0;
        }
    }
 
    # Output file / output database
    $self->param('output_file') || $self->param('output_db') || die 'One of the parameters "output_file" and "output_db" is mandatory';
    unless ($self->param('output_file')) {
        $self->param('real_output_db', go_figure_dbc( $self->param('output_db') ) );
        die 'Only the "mysql" driver is supported.' if $self->param('real_output_db')->driver ne 'mysql';
    }
 
    $self->input_job->transient_error(1);
 
    $self->say_with_header(sprintf("tables: %d %s", scalar(@tables), join('/', @tables)));
    $self->say_with_header(sprintf("ignores: %d %s", scalar(@ignores), join('/', @ignores)));
 
    my @options = qw(--skip-lock-tables);
    # Without any table names, mysqldump thinks that it should dump
    # everything. We need to add special arguments to handle this
    if ($nothing_to_dump) {
        $self->say_with_header("everything is excluded, nothing to dump !");
        push @options, qw(--no-create-info --no-data);
        @ignores = ();  # to clean-up the command-line
    }
 
    # mysqldump command
    my $output = "";
    if ($self->param('output_file')) {
        if (lc $self->param('output_file') =~ /\.gz$/) {
            $output = sprintf(' | gzip > %s', $self->param('output_file'));
        } else {
            $output = sprintf('> %s', $self->param('output_file'));
        }
    } else {
        $output = join(' ', '|', @{ $self->param('real_output_db')->to_cmd(undef, undef, undef, undef, 1) } );
    };
 
    # Extra parameter to add to the command-line
    my $dump_options = $self->param('dump_options') // [];
 
    # Must be joined because of the redirection / the pipe
    my $cmd = join(' ', 
        @{ $src_dbc->to_cmd('mysqldump', undef, undef, undef, 1) },
        @options,
        @tables,
        ref($dump_options) ? @$dump_options : ($dump_options,),
        (map {sprintf('--ignore-table=%s.%s', $src_dbc->dbname, $_)} @ignores),
        $output
    );
 
    # Check whether the current database has been restored from a snapshot.
    # If it is the case, we shouldn't re-dump and overwrite the file.
    # We also check here the value of the "skip_dump" parameter
    my $completion_signature = sprintf('dump_%d_restored', defined $self->input_job->dbID ? $self->input_job->dbID : 0);
 
    if ($self->param('skip_dump') or $self->param($completion_signature)) {
        if ($self->param('skip_dump')) {
            $self->complete_early('Skipping the dump because "skip_dump" is defined');
        } else {
            $self->complete_early("Skipping the dump because this database has been restored from the target dump. We don't want to overwrite it");
        }
    } elsif ($self->param('nb_ehive_tables') and !$self->worker->adaptor) {
        # We add the signature to the dump, so that the
        # job won't rerun on a restored database
        # We're very lucky that gzipped streams can be concatenated and the
        # output is still valid !
        my $extra_sql = qq{echo "INSERT INTO pipeline_wide_parameters VALUES ('$completion_signature', 1);\n" $output};
        $extra_sql =~ s/>/>>/;
        $self->param('cmd', "$cmd; $extra_sql");
    } else {
        # No signature to add (non-eHive database or standalone job)
        $self->param('cmd', $cmd);
    }
}

◆ param_defaults()

public Bio::EnsEMBL::Hive::RunnableDB::DatabaseDumper::param_defaults ( )

Undocumented method

Code:

click to view

The documentation for this class was generated from the following file:

modules/Bio/EnsEMBL/Hive/RunnableDB/DatabaseDumper.pm

Public Member Functions

Detailed Description

Synopsis

Description

Member Function Documentation

◆ _get_table_list()

◆ fetch_input()

◆ param_defaults()