The controller for URI is not callable, using annotations

Question

I have a symfony2 controller that is returning 500, with this log entry:

[2014-03-26 01:25:48] request.INFO: Matched route "searchtempestsite_direct_sponsored" (parameters: "_controller": "SearchTempest\Bundle\SiteBundle\Controller\SearchController::DirectResultsSponsoredAction", "_route": "searchtempestsite_direct_sponsored") [] []
[2014-03-26 01:25:48] request.CRITICAL: Uncaught PHP Exception InvalidArgumentException: "The controller for URI "/search/direct/sponsored" is not callable." at [...]/releases/20140326082503/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Controller/ControllerResolver.php line 82 {"exception":"[object] (InvalidArgumentException: The controller for URI \"/search/direct/sponsored\" is not callable. at [...]/releases/20140326082503/vendor/symfony/symfony/src/Symfony/Component/HttpKernel/Controller/ControllerResolver.php:82)"} []

The controller action is defined as

/**
 * Display the Sponsored Results frame in Direct Results
 *
 * @return Symfony\Component\HttpFoundation\Response
 *
 * @Route("/search/direct/sponsored", name="searchtempestsite_direct_sponsored")
 */
public function DirectResultsSponsoredAction()
{
    $response = $this->get('legacy_bridge')
        ->request('direct_sponsored.php');

    $parameters = $response->getTemplateParameterBag();
    $content = $this->renderView(
        'SearchTempestSiteBundle:Search:directSponsored.html.twig',
        $parameters->all()
    );
    $response->setContent($content);

    return $response;
}

What's strange is everything is working properly, both in dev and prod modes, on my local test server. It is only when I try to deploy this to the production server using capifony that I get this error. However, all the other routes defined under the same controller are working properly. Only this new one fails. Here is a working route:

/**
 * Display the Direct Results index
 *
 * @return Symfony\Component\HttpFoundation\Response
 *
 * @Route("/search/direct", name="searchtempestsite_direct")
 */
public function DirectResultsAction()
{
    $response = $this->get('legacy_bridge')
        ->request('adv_control.php');

    $parameters = $response->getTemplateParameterBag();

    $content = $this->renderView(
        'SearchTempestSiteBundle:Search:directResults.html.twig',
        $parameters->all()
    );
    $response->setContent($content);

    return $response;
}

I've tried changing the route (from /search/direct/sponsored to /search/direct_sponsored), but I still received the same 500 error at the new path. (With the new path in the error, of course.)

Beyond that I'm sort of stuck as far as what to try, short of digging into the guts of the Symfony and Sensio code to trace how its reconciling these annotations. Presumably this has something to do with installing to the server, since it works locally, but as far as I can tell, everything should be the same. Both use composer to install dependencies, so all the vendor code should be identical. We don't share a cache between releases to the server, so it's working with a fresh cache. (Capifony does call app/console cache:warmup --env=prod.)

It sounds like usually this error is due to the controller specification in the routing yml not matching the name of the Action function, or alternately the function not being public. Neither applies here.

I'm happy to provide any additional information requested.

Edit: It's working now, although I haven't changed anything. I think what's going on has to do with Capifony's multistage extension. We are running three stages (development, beta, prod). I had been testing this on the dev stage, but on a whim I tried deploying it to the other stages to see if that would make any difference. It didn't at first, but after I had deployed to all three, when I then re-deployed to any one of them, the route started working.

Edit 2: Multistage was most of the problem, but there was more to it. See my answer below.

Answer 1

It turns out the problem was due to the ApcClassLoader. These two lines in app.php instruct sf2 to use APC to cache the locations (filesystem paths) of various classes used by the application:

$loader = new ApcClassLoader('sf2', $loader);
$loader->register(true);

You're supposed to change the sf2 to a unique prefix to avoid cache key conflicts. We did that, but used the same 'unique' prefix for each of our stages, which meant that when we ran a deployment and cleared the apc cache, whichever stage was accessed first would set the file locations in the cache, and those same files would then be used for all stages. Since it's a busy site, this meant the production stage files ended up being used for the other stages.

The solution should be to simply deploy a separate version of app.php for each stage, with a separate prefix. (Or pass in a variable for the prefix; however you want to do it.) However, I also found that very occasionally the previous release of the same stage was polluting the cache, even though we clear the APC cache ([as described here][1]) immediately after deploying. I can't currently explain why that happens, since it seems to happen even if we add a delay before clearing, in case any requests are in progress.

So for now, we're simply going to use composer instead of apc to save the class map, as described in the "Use Composer's Class Map Functionality" section here: http://symfony.com/doc/current/book/performance.html

Edit: Ah, and this explains the last bit, how the earlier releases were polluting the cache even after the new ones were released: https://stackoverflow.com/a/23419954/160565. We use symlinks for deployment, and PHP was caching the old symlink routes, causing the previous deployment to refill the cache before the symlink was updated. The answer linked there explains how to work around that.