In an extbase extension, how to access the persistence layer from a scheduler task?

Question

What sounds a bit academic in the title is actually quite straightforward: I have set up a TYPO3 6.1 extbase extension that I've equipped with a scheduler task. The task is supposed to import a CSV file and save it into the extension's database fields.

But how do I tell the scheduler task to use the extension's model etc. and save the received data into the persistence layer?

I've seen this answer to a similar question: Execute repository functions in scheduler task and I think it points the right way, but I think need a full example to start understanding how the dependency injection works.

Answer 1

First you have to consider the aspect of performance:

If you want to insert a big amount of data, you should not use the Extbase persistence for such a task. Because if you do so, it will generate an object for each row you want to insert and persist it immediately. This is quite slow and has a big memory footprint.

If you don't have much data or you split the jobs (e.g. perform 100 import jobs per scheduler run), then use the Extbase persistence.

You can have both in CommandController context, and since CommandControllers are straight-forward to set up, you should go for them instead of an own Scheduler task.

Using Extbase persistence

In the CommandController, inject your repository:

/**
 * myRepository
 *
 * @var \Venor\Myext\Domain\Repository\MyRepository
 * @inject
 */
protected $myRepository

Then iterate through the rows you want to import (foreach) and create a new object for every row and add it to your repository:

 $myObject = $this->objectManager->get('Vendor\Myext\Domain\Model\MyModel');
 $myObject->setProperty('foo');
 $myObject->setOtherProperty('bar');
 $this->myRepository->add($myObject);

To actually save the objects to the database, you need to persist them. So you also inject the persistenceManager:

/**
 * @var \TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager
 * @inject
 */
protected $persistenceManager;

And use it:

 $this->persistenceManager->persistAll();

You should not do that for every single object (for performance reasons); but for memory usage reasons you should not wait until after thousands of objects to persist, either. So you just insert an iterator to your foreach loop and persist every 20th, 40th, or whatever loop.

Please don't forget that the Scheduler works in Backend context, so the TypoScript must be available by module.tx_yourext. If you want to share the same settings/storagePid with the frontend part of your app, use

module.tx_yourext.persistence < plugin.tx_yourext.persistence
[...]

The TypoScript needs to be present in the root page of your website for backend modules/CommandControllers to use them. I suggest you add the stuff to myext/Configuration/TypoScript/setup.txt and add the static template of your extension to the root page.

Using DataHandler

The TYPO3 DataHandler (formerly TCEmain) is the engine the TYPO3 backend uses for inserting and modifying database records. It is very powerful.

Instead of an object, inside your loop you create an array containing all the data. The first array index is the table, the next level is the affected record, where NEW means that a new record is created. Then you can just set every field of a table with the desired value

    $data = array();
    $data['be_users']['NEW'] = array(
        'pid' => 0,
        'username' => $staffMember['IDPerson'],
        'password' => md5(GeneralUtility::generateRandomBytes(40)), // random password
        'usergroup' => '1,2',
        'email' => $staffMember['Email'],
        'realName' => $staffMember['Nachname'] . ' ' . $staffMember['Vorname'],
        'lang' => 'de',
    );

Now you can make an Instance of DataHandler and persist the changes:

    /** @var $tce t3lib_TCEmain */
    $tce = GeneralUtility::makeInstance('TYPO3\CMS\Core\DataHandling\DataHandler');
    $tce->bypassAccessCheckForRecords = TRUE;
    $tce->start($data, array());
    $tce->admin = TRUE;
    $tce->process_datamap();
    $newRecordsUidArray = $tce->substNEWwithIDs['NEW'];

Please note the line $tce->admin = TRUE. This suggests to DataHandler that an admin is performing the action. This is convenient because you don't have to set allowed exclude fields for the Scheduler user and can also insert records to PID 0. But it is a possible security flaw, so carefully consider its usage.

Records inserted/updated by DataHandler logged correctly, can be reverted etc.. You can find some examples (such as adding pictures, resolving MM relations) here. In this case all DataHandler related functions were moved to an external repository class that is injected to the CommandController as described above (it's just named in Extbase convention).

A good overview of DataHandler functions can be found here.

Answer 2

In Addition to lorenz's answer: Beginner's Guide to set up a Command Controller Scheduler task:

My example is an import task. Change the Name part "Import" to your needs.

Create a new file EXT:Classes/Controller/ImportCommandController.php

    <?php
    namespace NAMESPACE\Myextension\Controller;

    /***************************************************************
     *  Copyright notice
     *
     *  (c) 2014 
     *  All rights reserved
     *
     *  This script is part of the TYPO3 project. The TYPO3 project is
     *  free software; you can redistribute it and/or modify
     *  it under the terms of the GNU General Public License as published by
     *  the Free Software Foundation; either version 3 of the License, or
     *  (at your option) any later version.
     *
     *  The GNU General Public License can be found at
     *  http://www.gnu.org/copyleft/gpl.html.
     *
     *  This script is distributed in the hope that it will be useful,
     *  but WITHOUT ANY WARRANTY; without even the implied warranty of
     *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
     *  GNU General Public License for more details.
     *
     *  This copyright notice MUST APPEAR in all copies of the script!
     ***************************************************************/

    /**
     *
     *
     * @package Myextension
     * @license http://www.gnu.org/licenses/gpl.html GNU General Public License, version 3 or later
     *
     */
    class ImportCommandController extends \TYPO3\CMS\Extbase\Mvc\Controller\CommandController {

        /**
         * itemRepository
         *
         * @var \NAMESPACE\Myextension\Domain\Repository\ItemRepository
         * @inject
         */
        protected $itemRepository;

        /**
         * @var \TYPO3\CMS\Extbase\Persistence\Generic\PersistenceManager
         * @inject
         */
        protected $persistenceManager;


        /**
         *
         * @param \integer $storagePid
         * @param \string $url
         *
         * @return bool
         */
        // very nice: parameters will be fields in the scheduler!
        public function importCommand($storagePid = 0,$url = NULL) {

            $source = utf8_encode(utf8_encode(file_get_contents($url)));

            // set storage page ourselves
            // not sure if really necessary

            $querySettings = $this->itemRepository->createQuery()->getQuerySettings();
            $querySettings->setRespectStoragePage(FALSE);
            $this->itemRepository->setDefaultQuerySettings($querySettings);

            // do your stuff here
            $source = 'foo';
            $rendered = 'bar';


            // then store it

            // this seems to be only necessary if we don't have an existing item yet
            // but as we don't know that here, we have to do it
            $item = $this->objectManager->get('STUBR\Therapiestellen\Domain\Model\Item');

            // find all existing items
            $all = $this->itemRepository->findAll();

            // if we have an item already, take the first (and only one)
            if(count($all) > 0){
                $item = $all->getFirst();
            }

            // set / update properties
            $item->setSource($source);
            $item->setRendered($r); 
            $item->setPid($storagePid);

            // either update or add it
            if(count($all) > 0){
                $this->itemRepository->update($item);
            }
            else {
                $this->itemRepository->add($item);
            }

            // persist it
            $this->persistenceManager->persistAll();
        }
}
?>

In EXT:ext_localconf.php, add the command controller:

$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['extbase']['commandControllers'][] = 'NAMESPACE\\Myextension\\Controller\\ImportCommandController';

Configure in Scheduler:

That's basically it!