Multiple csv file rendering by Perl using single file open handle?

Question

You cannot read from a file opened in append >> mode. It seems you've just renamed the OUT filehandle to FP, but that isn't enough.

The original code included a chdir $seculert_dir – this makes sense because now we don't have to prefix our filenames with the directory all the time (it's the equivalent to the cd command in the shell).
The original code didn't exactly use best practices. I wrapped the scp and ssh commands with versions that do a minimal amount of error checking.
It is also unnecessary to invoke external tools for things we can do with Perl.

Here is my updated code, that can read the data from simplistic CSV files:

First, we start the preample. use warnings is more preferable than the -w switch.

#!/usr/bin/perl
use strict;
use warnings;

my $seculert_dir = "!FIXME!";
chdir $seculert_dir;  # go into that dir, so that we don't have to use prefixes

my @csv_files = ("seculert.csv"); # this is a csv which will contains IP addresses
                                  # of one specific category for e.g malware.
                                  # We only use the 2nd column

#ssh connection information 
my $qradar_console = '10.10.1.22';
my $qradar_ssh_key = "qr-id_dsa";
my $qradar_ssh_knownhosts = "known_hosts";

That was our configuration. Next we fetch the conf from the server using the wrapped SCP command:

# fetch the remotenet.conf from QRadar
print STDERR "Fetching configuration from QRadar...\n";
scp("root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf" => '.');

We now open a file where we put our changed configuration

# write the changed conf here before uploading:
open my $new_conf, ">", "remotenet.conf.changed" or die qq(Can't open "remotenet.conf.changed" for writing: $!);

Now we open the old configuration, copy it over, but skip lines starting with SECULERT.

# copy old conf over, delete lines starting with "SECULERT"
open my $old_conf, "<", "remotenet.conf" or die qq(Can't open "remotenet.conf" for reading: $!);
while (<$old_conf>) {
    print {$new_conf} $_ unless /^SECULERT/;
}
close $old_conf;

Notice how I've used “lexical filoehandles” (open my $fh, ...). This avoids some problems, and is more modern than using barewords.

Next, we loop through the CSV files. We open each one, then extract the 2nd column and print it with the other stuff into the changed configuration file.

# append the data from the CSVs
for my $csv_file (@csv_files) {
    my $source = 'BAD-IP-Addresses-LABEL';
    my $type_description = 'honeypots-for-examnple';

    open my $csv, "<", $csv_file or die qq(Can't open "$csv_file" for reading: $!);

    while (my $line = <$csv>) {
        my (undef, $ip) = split /,/, $line; # we're only interested in the 2nd column

        # Based upon the format described below I want to render the csv
        # as written in print OUT statement. This format is important, because the
        # endsystem process the file (remotenet.conf) based upon the provided layout.
        #
        # Columns in the output:
        # 1 - Name
        # 2 - Sub-Name
        # 3 - IP Address
        # 4 - is colour, deprecated
        # 5 - database length, deprecated
        # 6 - asset weight, deprecated
        # 7 - an ID for the 'record' each unique name pair (first 2 columns) gets an ID
        print {$new_conf} "$source $type_description $ip #FF0000 0 90  29\n";
    }
}

Now we have all the information we want in the new configuration file and can upload it to the server:

close $new_conf;

# copy the changed remotenet.conf back to QRadar
scp('remotenet.conf.changed' => "root\@$qradar_console:/store/configservices/staging/globalconfig/remotenet.conf");

# Remove our SECULERT list and the newly pushed out qradar conf
print STDERR "Cleaning up...\n";
unlink $_ or warn qq(Can't remove "$_": $!) for 'remotenet.conf', 'remotenet.conf.changed';

Next, we run the deploy script:

# QRadar magic -- run deploy script
print STDERR "Deploying in QRadar...(takes time to complete)\n";
ssh("root\@$qradar_console", '/opt/qradar/upgrade/util/setup/upgrades/do_deploy.pl');

print STDERR "Complete!\n\n";

Here are the wrappers for scp and ssh. They are subroutines (functions, procedures, or methods in other languages). The arguments are in the @_ array from which we unpack them into variables with better names. The system command takes a name of a command and a list of arguments. Because this circumvents the shell, we don't have to think about shell escapes. system returns zero on sucess, so we use that to check for errors.

# Wrappers for SSH and SCP commands
sub scp {
    my ($from, $to) = @_;
    my $success = 0 == system 'scp',
        '-i' => $qradar_ssh_key,
        '-o' => "UserKnownHostsFile=$qradar_ssh_knownhosts",
        '-o' => "StrictHostKeyChecking=no",
        $from => $to;
    return $success if defined wantarray;                # return failure when somebody checks for it
    die qq(Can't scp "$from" to "$to") if not $success;  # die when failure, and nobody checks.
}

sub ssh {
    my ($host, $command) = @_;
    my $success = 0 == system 'ssh',
        '-i' => $qradar_ssh_key,
        '-o' => "UserKnownHostsFile=$qradar_ssh_knownhosts",
        '-o' => "StrictHostKeyChecking=no",
        $host, $command;
    return $success if defined wantarray;                           # return failure when somebody checks for it
    die qq(Can't ssh into "$host" for '$command') if not $success;  # die when failure, and nobody checks.
}

This code could still be improved, e.g. by using a proper CSV parser like Text::CSV, using better SSH bindings than simply wrapping command line programs, automated error checking with autodie, and a better handling of tempfiles.

It seems you want different $source values for different files. For this, we should use a more complex data structure than @csv_files – I'd use a hash of arrays, e.g.

my %csv_files = (
  'malware.csv' => ['BAD-IP-Addresses-LABEL', 'honeypots-for-examnple'],
  'bot.csv'     => ['bot-net', 'top-10'],
);

This is a dictionary that maps keys (here filenames) to values (here the contents of two columns). Instead of looping over entries in an array, we would now loop over the keys in this hash:

for my $csv_file (keys %csv_files) {
  my ($source, $type_description) = @{ $csv_files{$csv_file} };

  ...
}

The expression $csv_files{$csv_file} accesses the entry called $csv_file in the hash $csv_files. This entry contains an array reference as value. The @{…} around that converts the array reference to an array, which we can unpack with a list assignment my ($foo, $bar) = @array.