Pluses and minuses of using Factories in a Rails test suite?

Question

I'm currently looking at a hefty Rails test suite. It's nothing I can get into specifics about, but the run time for the entire suite (unit/functional/some integration) can run upward of 5 minutes.

We're completely reliant on fixtures and are we're not mocking and stubbing as much as we should be.

Our next few sprints are going to be completely focused on the test suite, both improving coverage, writing better tests and most importantly writing more efficient tests.

So aside from more mocking and stubbing within our tests, we're considering replacing our fixtures with most likely Factory Girl. I see a lot of happy folks doing similar situations but haven't been able to find a good resource on any minuses of moving to a factory. I have seen some slower benchmarks when using benchmarks from various resources but cannot find a definitive this why factories are good and this is why you might not want to use them.

Can anyone educate me on why or why I shouldn't be using factories?

Thanks!

Answer 1

There could be some issues with setting up all dependencies between entities for good test suite. Anyway, it's still much easier than maintaing a lot of fixtures.

Fixtures:

• hard to maintain relationships (especially many-to-many);
• test suite runtime is usually slower due more DB hits;
• tests are very sensitive to changes in schema.

Factories:

• you stub everything you don't test at current unit test;
• you prepare entities that you are testing with factories. This is where factories show their real advantage — it's easy to set up new test cases, as you don't need to maintain a ton of YAML-files for that;
• you concentrate on testing. If tests require changing scenario, you don't shift your mindset. As long as stubs are reasonable and factories are easily customized, you should be fine.

So, factories seem a good way to go. The only possible drawbacks I see, are:

• time you spent migrating from fixtures;
• keeping a sane set of scenarios can require some effort.

Answer 2

Oleg's answer is great, but let me offer the perspective of someone who is using both.

Fixtures have sort of been the whipping boy of the Rails community for a while. Everyone understands the drawbacks of fixtures, but no one is really championing their strengths. In my experience, factories by themselves can easily become just as difficult to maintain as fixtures (it really depends on the schema, but I digress). The real strength of factories is in selective replacement of fixture-based pain. Let's talk about a couple specifics.

The first issue is performance. If you can test most of your app without hitting the database then you will see a significant speed up, but for most applications I don't think it's wise to test without hitting the database entirely. At some point you want to test the whole stack. Every time you mock or stub you are making an assumption about an interface that may contain subtle bugs. So, assuming that you need to hit the database on some significant percentage of tests, transactional fixtures (you are using transactional fixtures right?) could well be much much faster than instantiating a whole environment for every test.

I'd say, with the size of your test suite that you really need to look towards Continuous Integration to scale your development to the next level. No matter how much you speed them up, it's still a long time for developers to wait. Maybe look at autotest as well to help at the individual level. But ultimately CI is going to allow you to maintain testing discipline without sacrificing developer agility.

The place where fixtures really shine is in functional/integration testing. The way I look at it is that the fixtures should set up a healthy base state for the app to be tested. Most unit tests don't really need this. You can get very good unit coverage using factories. However when it comes to functional testing, any given page may be hitting dozens of models. I don't want to set up all that stuff in each test. As I construct ever more complex scenarios, I'm getting closer and closer to recreating a global data state which is exactly what fixtures were designed to do in the first place.

One controversial belief I hold is that all else being equal, I prefer one functional test to 20 unit tests (using Rails parlance). Why? Because the functional test proves that the end result that is sent to the user is correct. The unit tests are great for getting at nuances of functionality, but at the end of the day, you could still have a bug along an interface that breaks your entire site. Functional tests are what give me the confidence hitting deploy without actually loading up the page in my browser. I know that I could stub everything out and test both interfaces and get the same coverage, but if I can test the whole stack in one simple test at the expense of a little CPU, I'd much rather do that.

So what are my best practices for fixtures?

• Set up a handful for every model to cover the broadest categories of data
• When adding a major new feature that cuts across many models and controllers, add some new fixtures to represent the major states
• Avoid editing old fixtures except for adding/removing fields
• Use factories for more smaller/more localized variations
• Use factories for testing pagination or other mass creation that is only needed for a few tests

Also, let me recommend Jay Fields' blog for really good pragmatic testing advice. The thing I like most about Jay's blog is that he always acknowledges that testing is very project-specific, and what works for one project does not necessarily work for another. He's short on dogma and long on pragmatism.