Programmatically installing a module

Question

I need to install modules so that the database changes happen. I can't just let it be an install on logging to admin type of approach as I have a bash script that I am using to provision my test servers. It seems that as I clone modules from my git repo some of them are not installed so I need to forcefully install them in the php script I run that adds my test stores and what not. I have looked through all manner of class and the best I have found is

isModuleOutputEnabled()

But that is not going to cut it. Ideas on how to do a forceful install?

---

possible identification of the issue

It seems that simple clearing the cache and calling the Mage::app() and even trying to log in a admin user fails to trigger the modules to install. But if you clear the cache by hand, right before the first time you go to the /admin login screen, it then will install things as you would expect. As of yet getting the page (wget) before clearing the cache also fails to install the modules.

---

Update 1

I have tried to do some forcing here, but I don't get what is going on... the extensions load fine when I wait till after the fresh install and just place the files in the folders, but when I clone it in beforehand I don't see it in the core_resource table or the table for the modules. Example test for the post install script is

$resource = Mage::getSingleton('core/resource');
$modules = Mage::getConfig()->getNode('modules')->children();
$modulesArray = (array)$modules;
echo "Test modules";
if(isset($modulesArray['Aoe_AsyncCache'])) {
    echo "AsyncCache exists.";
    $tableName = $resource->getTableName('asynccache');

    /**
     * if prefix was 'mage_' then the below statement
     * would print out mage_catalog_product_entity
     */
    echo "known as ".$tableName;
} else {
    echo "AsyncCachedoesn't exist.";
}  

When ran I see AsyncCache exists. known as 'asynccache' yet it is not in the database. That table was not installed. Not again, the module does install fine if I wait... But oddly, if I do this git clone, it never installs. I'm really unsure what is going on.

---

Update 2

I seems that any module that has a database tied to it is not being installed. So background on what is going on. I'm using vagrant and it's on a url of http://local.mage.dev as the url. Now when I use the provision.sh script, I have it install Magento from the ssh. I does in stall. But a great example of the issue is that when I so a simple,

./mage config-set preferred_state alpha
./mage clear-cache
./mage sync
./mage download community BL_CustomGrid
./mage install community BL_CustomGrid

The table is not installed. I don't get why. Magento is installed. If I wait for the first log in I can use connect to do the install. I get notice on the command line that it has install the BL_CustomeGrid just fine, but it hasn't... the tables are not there.

---

Update3

I tried to combine all the suggestions in to one. I have the Mage app being invoked after each github clone, then the cache is cleared. I then clear the cache one last time before I go to log in. I can't see to get the table installed. I almost wonder if it has something to do with the vagrant but seems unlikely being that Magento does install. But proof of what I did, here is the sub question on calling the php script that is doing nothing but invoking mage and clearing the cache.

---

Update4

Important note:

I have not only done a test where I reconfirmed that if I do the install after the magento install. The normal log in, upload files, clear cache, log out, log back in and the module is installed. BUT when I add a programmatically login of the admin user, I echo that the user is loged in, and even thou I clear the cache, it still fails.

The goal is that one, I need to find the install('module_name') or something like that, or two, be able to 100% do what happens when the user signs in clears caches and logs back out.

Programmatic replication of how a module is installed we know (logout,clear cache,etc..) fails

---

---

The scripts in play

The script that invokes install-invoke-app.php And so there is full context, here is the function.sh that is called and where you can see that the invoking of Mage is done

provision.sh This is what is called from vagrant which will configure my test bed and install a full Magento install programmatically

start_seconds=`date +%s`
echo -en "\a" > /dev/tty5 | echo -e "\a"
cd /srv/www/
. scripts/install-functions.sh

if [[ has_network ]]
then
    apt-get install pv

    cd /srv/www/
    . scripts/install-prep.sh

    cd /srv/www/
    . scripts/main-install.sh

else
    echo -e "\nNo network connection available, skipping package installation"
fi

#double check the network
if [[ has_network ]]
then

    #check and install wp
    cd /srv/www/
    . scripts/db-install.sh   

    #check and install magento
    cd /srv/www/
    . scripts/mage-install.sh      

else
    echo -e "\nNo network available, skipping network installations"
fi
# Add any custom domains to the virtual machine's hosts file so that it
# is self-aware. Enter domains space delimited as shown with the default.
DOMAINS='local.mage.dev'
if ! grep -q "$DOMAINS" /etc/hosts
then echo "127.0.0.1 $DOMAINS" >> /etc/hosts
fi

end_seconds=`date +%s`
echo "-----------------------------"

install-functions.sh This is just some of the functions of this bash script. It's only what is used in this set of the server install. Note that this is where we are calling php "/srv/www/scripts/mage/install-invoke-app.php" which is what should be invoking the MAGE and installing any modules. Now the extensions should have programmatically been installed. This is not happening.

install_repo(){
    if [ $2 ]
    then
        echo "just 1"
        git clone $1 -q
    else
        echo "just 1 and 2"
        git clone $1 $2 -q
    fi
    success=$?
    if [[ $success -eq 0 ]];
    then
        echo "Repository successfully cloned."
        echo "cleaning"
        cd $r/
        rm -rf LICENSE.txt STATUS.txt README.md RELEASE_NOTES.txt modman
        cd ../
        cp -af $r/* .
        rm -rf $r/

        #eval $3
        php "/srv/www/scripts/mage/install-invoke-app.php"

        #if [ -z "$3" ]
        #then
        #    echo "no callback"
        #else
        #    eval $3
        #fi
    else
        echo "Something went wrong!"
    fi
    sleep 1 # slow it down to insure that we have the items put in place.
}

#declare -A list = ( [repo]=gitUser )
install_repolist(){
    gitRepos=$1
    for r in "${!gitRepos[@]}" #loop with key as the var
    do
        giturl="git://github.com/${gitRepos[$r]}/$r.git"
        echo "Adding $r From $giturl"
        if [ -z "$r" ];
        then
            echo
        else
            install_repo $giturl $2 $3
        fi
        echo
    done
    return 1
}

install-mage.sh This is called after the provision.sh script has installed all the basics of the server. It should already have nginx and php5/php-fpm and all the rest of the modules needed.

#!/bin/bash
cd /srv/www/mage/ #move to the root web folder


dbhost="localhost"
dbname="mage"
dbuser="devsqluser"
dbpass="devsqluser"
url="local.mage.dev"
adminuser="admin"
adminpass="admin2013"
adminfname="Mc"       
adminlname="Lovin"
adminemail="test.user@wsu.edu"

echo
echo "We will clear any past install"
echo
echo "--Clear old caches reports and sessions"
cd /srv/www/mage/ #move to the root web folder
rm -rf ./var/cache/* ./var/session/* ./var/report/* ./var/locks/*
rm -rf ./var/log/* ./app/code/core/Zend/Cache/* ./media/css/* ./media/js/*
echo
if [ -f /srv/www/scripts/mage/clean.sql ]
then
    mysql -u root -pblank $dbname < /srv/www/scripts/mage/clean.sql | echo -e "\n Initial custom mage cleaning MySQL scripting..."
else
    echo -e "\n COUNLDN'T FIND THE CLEANER SQL FILE"
fi



echo "Now installing Magento with sample data..."



#chack to see if there is already the files ready for instalation
if [ ! -f /srv/www/mage/app/Mage.php ]
then

    if [ ! -f /srv/www/mage/magento-1.7.0.2.tar.gz ]
    then

        echo
        echo "didn't find the packages, so now Downloading them..."
        echo

        wget http://www.magentocommerce.com/downloads/assets/1.7.0.2/magento-1.7.0.2.tar.gz
        wget http://www.magentocommerce.com/downloads/assets/1.6.1.0/magento-sample-data-1.6.1.0.tar.gz
    fi

    echo
    echo "Extracting data..."
    echo    
        pv -per magento-1.7.0.2.tar.gz | tar xzf - -C ./
        pv -per magento-sample-data-1.6.1.0.tar.gz | tar xzf - -C ./

    echo
    echo "Moving files..."
    echo        
        cp -af magento-sample-data-1.6.1.0/media/* media/
        cp -af magento-sample-data-1.6.1.0/magento_sample_data_for_1.6.1.0.sql data.sql
        cp -af magento/* magento/.htaccess .

        cd /srv/www/mage/ #move to the root web folder
    echo
    echo "Setting permissions..."
    echo

    chmod o+w var var/.htaccess app/etc
    chmod -R o+w media

fi

echo
echo "Installing Adminer..."
if [ ! -f /srv/www/mage/adminer.php ]
then
    wget http://downloads.sourceforge.net/adminer/adminer-3.7.1-mysql-en.php > adminer.php
fi

echo
echo "Importing sample products..."
mysql -h $dbhost -u $dbuser -p$dbpass $dbname < data.sql




#pear mage-setup .
#pear install magento-core/Mage_All_Latest-stable

#./mage mage-setup .
#./mage config-set preferred_state stable

echo
echo "Initializing PEAR registry..."


pear mage-setup .

echo
echo "Downloading packages..."


pear install magento-core/Mage_All_Latest


echo
echo "Cleaning up files..."


rm -rf downloader/pearlib/cache/* downloader/pearlib/download/*
rm -rf magento/ magento-sample-data-1.6.1.0/
#rm -rf magento-1.7.0.2.tar.gz magento-sample-data-1.6.1.0.tar.gz data.sql
rm -rf index.php.sample .htaccess.sample php.ini.sample LICENSE.txt
rm -rf STATUS.txt LICENSE.html LICENSE_AFL.txt  RELEASE_NOTES.txt

echo
echo "Installing Magento..."

    php -f install.php --\
    --license_agreement_accepted yes \
    --locale en_US \
    --timezone America/Los_Angeles \
    --default_currency USD \
    --db_host $dbhost \
    --db_name $dbname \
    --db_user $dbuser \
    --db_pass $dbpass \
    --url $url \
    --use_rewrites yes \
    --skip_url_validation yes \
    --use_secure no \
    --secure_base_url "" \
    --use_secure_admin no \
    --admin_firstname "$adminfname" \
    --admin_lastname "$adminlname" \
    --admin_email "$adminemail" \
    --admin_username "$adminuser" \
    --admin_password "$adminpass"

if [ ! -f /srv/www/mage/app/etc/local.xml ]
then
    echo "failed install try it again"
else


    if [ -f /srv/database/init-mage.sql ]
    then
        mysql -u root -pblank < /srv/database/init-mage.sql | echo -e "\nInitial custom mage MySQL scripting..."
    else
        echo -e "\nNo custom MySQL scripting found in database/init-mage.sql, skipping..."
    fi

    cd /srv/www/mage/
    echo "Starting to import base WSU modules from connect"
    #./mage install http://connect20.magentocommerce.com/community Flagbit_ChangeAttributeSet
    ./mage config-set preferred_state alpha
    ./mage clear-cache
    ./mage sync
    ./mage download community Flagbit_ChangeAttributeSet
    ./mage download community BL_CustomGrid
    ./mage install community Flagbit_ChangeAttributeSet
    #./mage install community BL_CustomGrid


    echo "Starting to import base WSU modules fro github"
    declare -A gitRepos
    #[repo]=gitUser
    gitRepos=(
        [wsu_admin_base]=jeremyBass
        [wsu_base_theme]=jeremyBass
        [Storeutilities]=jeremyBass
        [StructuredData]=jeremyBass
        #[Storeuser]=jeremyBass
        [sitemaps]=jeremyBass
        [webmastertools]=jeremyBass 
        [ldap]=jeremyBass 
        [Aoe_FilePicker]=jeremyBass         #https://github.com/fbrnc/Aoe_FilePicker.git
        [Aoe_ClassPathCache]=jeremyBass     #https://github.com/fbrnc/Aoe_ClassPathCache.git
        [Aoe_Profiler]=jeremyBass           #https://github.com/fbrnc/Aoe_Profiler.git
        [Aoe_ManageStores]=jeremyBass       #https://github.com/fbrnc/Aoe_ManageStores.git
        [Aoe_LayoutConditions]=jeremyBass   #https://github.com/fbrnc/Aoe_LayoutConditions.git
        [Aoe_AsyncCache]=jeremyBass         #https://github.com/fbrnc/Aoe_AsyncCache.git
        [Aoe_ApiLog]=jeremyBass             #https://github.com/fbrnc/Aoe_ApiLog.git
        #[mage-enhanced-admin-grids]=mage-eag
    )
    cd /srv/www/mage/
    install_repolist $gitRepos 0 "wget http://$url"
    unset gitRepos         #unset and re-declare to clear associative arrays
    declare -A gitRepos

    cd /srv/www/mage/

    echo "importing WSU favicon"
    wget -q http://images.dev.edu/favicon.ico -O favicon.ico

    #run actions that are hard to do direct sql quries on 

    php /srv/www/scripts/mage/install-post.php


    echo
    echo "doing the first index"
    echo
    cd shell && php -f indexer.php reindexall


    mysql -u root -pblank $dbname -e "DELETE FROM $dbname.adminnotification_inbox;" | echo -e "\n >> Removed admin notifications ..."

    # Enable developer mode
    #if [ $MAG_DEVELOPER_MODE == 1 ]; then
    #    sed -i -e '/Mage::run/i\
    #Mage::setIsDeveloperMode(true);
    #' -e '1,$s//Mage::run/' $WWW_PATH/index.php
    #fi


    echo
    echo "Finished installing Magento"
    echo

 fi

install-post.php This file happens after the install-mage.sh file has completed the Magento install. All modules have been downloaded and what we need is to have had all the module's tables installed. IE: the sql installs for the modules should have been ran by now

<?php
//just as a guide, no real purpose
echo getcwd() . " (working from)\n";

//set up the store instance
require_once "app/Mage.php";
umask(0);
Mage::app();
Mage::app()->getTranslator()->init('frontend');
Mage::getSingleton('core/session', array('name' => 'frontend'));
Mage::registry('isSecureArea'); // acting is if we are in the admin
Mage::app('admin')->setUseSessionInUrl(false);
Mage::getConfig()->init();
/**
 * Get the resource model
 */
$resource = Mage::getSingleton('core/resource');

/**
 * Retrieve the read connection
 */
$readConnection = $resource->getConnection('core_read');

/**
 * Retrieve the write connection
 */
$writeConnection = $resource->getConnection('core_write');

// switch off error reporting
error_reporting ( E_ALL & ~ E_NOTICE );

$changeData = new Mage_Core_Model_Config();

echo "applying default store settings\n";
//pattern
//\((.*?),  '(.*?)',    (.*?),  '(.*?)',    '?(.*?)'?\),
//$changeData->saveConfig('$4', "$5", 'default', 0);
echo " - applying design settings\n";
    $changeData->saveConfig('design/package/name', "wsu_base", 'default', 0);
    $changeData->saveConfig('design/theme/template', "default", 'default', 0);
    $changeData->saveConfig('design/theme/skin', "default", 'default', 0);
    $changeData->saveConfig('design/theme/layout', "default", 'default', 0);
    $changeData->saveConfig('design/theme/default', "default", 'default', 0);
    $changeData->saveConfig('design/theme/locale', "NULL", 'default', 0);

function make_store($categoryName,$site,$store,$view){
    //#adding a root cat for the new store we will create
    // Create category object
    $category = Mage::getModel('catalog/category');
    $category->setStoreId(0); // No store is assigned to this category

    $rootCategory['name'] = $categoryName;
    $rootCategory['path'] = "1"; // this is the catgeory path - 1 for root category
    $rootCategory['display_mode'] = "PRODUCTS";
    $rootCategory['is_active'] = 1;

    $category->addData($rootCategory);
    $rootCategoryId=0;
    try {
        $category->save();
        $rootCategoryId = $category->getId();
    }
        catch (Exception $e){
        echo $e->getMessage();
    }
    if($rootCategoryId>0){
    //#addWebsite
        /** @var $website Mage_Core_Model_Website */
        $website = Mage::getModel('core/website');
        $website->setCode($site['code'])
            ->setName($site['name'])
            ->save();

    //#addStoreGroup
        /** @var $storeGroup Mage_Core_Model_Store_Group */
        $storeGroup = Mage::getModel('core/store_group');
        $storeGroup->setWebsiteId($website->getId())
            ->setName($store['name'])
            ->setRootCategoryId($rootCategoryId)
            ->save();

    //#addStore
        /** @var $store Mage_Core_Model_Store */
        $store = Mage::getModel('core/store');
        $store->setCode($view['code'])
            ->setWebsiteId($storeGroup->getWebsiteId())
            ->setGroupId($storeGroup->getId())
            ->setName($view['name'])
            ->setIsActive(1)
            ->save();
    }
    return $rootCategoryId;
}
echo "Applying the default multi-store setup\n";
$installed_stores = array();
$installed_stores['studentstore'] = make_store("Student store root",
                array('code'=>'studentstore','name'=>'Student store'),
                array('name'=>'Student Store'),
                array('code'=>'studentstore','name'=>'base default veiw')
              );
$installed_stores['teststore'] = make_store("Test store root",
                array('code'=>'teststore','name'=>'Test store'),
                array('name'=>'Test Store'),
                array('code'=>'teststore','name'=>'base default veiw')
              );

// let us refresh the cache
try {
    $allTypes = Mage::app()->useCache();
    foreach($allTypes as $type => $blah) {
      Mage::app()->getCacheInstance()->cleanType($type);
    }
} catch (Exception $e) {
    // do something
    error_log($e->getMessage());
}

$types = Mage::app()->getCacheInstance()->getTypes();
try {
    echo "Cleaning data cache... \n";
    flush();
    foreach ($types as $type => $data) {
        echo "Removing $type ... ";
        echo Mage::app()->getCacheInstance()->clean($data["tags"]) ? "[OK]" : "[ERROR]";
        echo "\n";
    }
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}

echo "\n";

try {
    echo "Cleaning stored cache... ";
    flush();
    echo Mage::app()->getCacheInstance()->clean() ? "[OK]" : "[ERROR]";
    echo "\n\n";
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}

$modules = Mage::getConfig()->getNode('modules')->children();
$modulesArray = (array)$modules;
echo "Test modules";
if(isset($modulesArray['Aoe_AsyncCache'])) {
    echo "AsyncCache exists.";
    $tableName = $resource->getTableName('asynccache');

    /**
     * if prefix was 'mage_' then the below statement
     * would print out mage_catalog_product_entity
     */
    echo "known as ".$tableName;
} else {
    echo "AsyncCachedoesn't exist.";
}   

install-invoke-app.php

<?php
//just as a guide, no real purpose
echo getcwd() . " (working from)\n";

//set up the store instance
require_once "app/Mage.php";
umask(0);
Mage::app();
Mage::app()->getTranslator()->init('frontend');
Mage::getSingleton('core/session', array('name' => 'frontend'));
Mage::registry('isSecureArea'); // acting is if we are in the admin
Mage::app('admin')->setUseSessionInUrl(false);
Mage::getConfig()->init();

// switch off error reporting
error_reporting ( E_ALL & ~ E_NOTICE );

// let us refresh the cache
try {
    $allTypes = Mage::app()->useCache();
    foreach($allTypes as $type => $blah) {
      Mage::app()->getCacheInstance()->cleanType($type);
    }
} catch (Exception $e) {
    // do something
    error_log($e->getMessage());
}


$types = Mage::app()->getCacheInstance()->getTypes();
try {
    echo "Cleaning data cache... \n";
    flush();
    foreach ($types as $type => $data) {
        echo "Removing $type ... ";
        echo Mage::app()->getCacheInstance()->clean($data["tags"]) ? "[OK]" : "[ERROR]";
        echo "\n";
    }
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}

echo "\n";

try {
    echo "Cleaning stored cache... ";
    flush();
    echo Mage::app()->getCacheInstance()->clean() ? "[OK]" : "[ERROR]";
    echo "\n\n";
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}
echo "finish invoking the mage app.";

//time to login as admin
Mage::app('admin')->setUseSessionInUrl(false);
Mage::getSingleton('core/session', array('name' => 'adminhtml'));

// supply username
$user = Mage::getModel('admin/user')->loadByUsername('admin'); // user your admin username

if (Mage::getSingleton('adminhtml/url')->useSecretKey()) {
   Mage::getSingleton('adminhtml/url')->renewSecretUrls();
}

$session = Mage::getSingleton('admin/session');
$session->setIsFirstVisit(true);
$session->setUser($user);
$session->setAcl(Mage::getResourceModel('admin/acl')->loadAcl());
Mage::dispatchEvent('admin_session_user_login_success',array('user'=>$user));
if ($session->isLoggedIn()) { echo "Logged in \n"; } else{ echo "Not Logged \n"; }

Answer 1

Answer

It turns out that just clearing the cache and doing a wget, or login a user in programmatically, or even both was not enough to trigger the code to install a module.

This is what works.

rm -rf var/cache/*
php "/srv/www/mage/index.php"

Where /mage is the store root folder, I just needed to execute the index.php file and then it worked. I can't say why it wouldn't on wget, or why calling MAGE:app() and all combinations didn't work, but clearing the cache folder and calling the file is the ticket for stably rebuilding the cache from the command line.

Hope this saves someone some time. Cheers

Answer 2

Not sure if I got your question right but if you want to trigger setup script w/o opening the website in your browser why not just create the shell script which will just wget the home page of your Magento installation?

Answer 3

You can use the following php script to clear the cache from the commandline after you applied all the updates thru git. After this you can use wget to get the homepage of your site to rebuild the cache and run all the database changes.

(code from http://www.yameveo.com/development/php/flush-every-magento-cache-from-the-command-line):

<?php
date_default_timezone_set("Europe/Madrid");
echo "Start Cleaning all caches at ... " . date("Y-m-d H:i:s") . "\n\n";
ini_set("display_errors", 1);

require '../app/Mage.php';
Mage::app('admin')->setUseSessionInUrl(false);
Mage::getConfig()->init();

$types = Mage::app()->getCacheInstance()->getTypes();

try {
    echo "Cleaning data cache... \n";
    flush();
    foreach ($types as $type => $data) {
        echo "Removing $type ... ";
        echo Mage::app()->getCacheInstance()->clean($data["tags"]) ? "[OK]" : "[ERROR]";
        echo "\n";
    }
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}

echo "\n";

try {
    echo "Cleaning stored cache... ";
    flush();
    echo Mage::app()->getCacheInstance()->clean() ? "[OK]" : "[ERROR]";
    echo "\n\n";
} catch (exception $e) {
    die("[ERROR:" . $e->getMessage() . "]");
}