IDE manual productivity test

Question

I am looking for a "test" to determinate which IDE is more productive.

I imagine this is a project in the desired language (my case would be javascript react, or Ruby) and I need to follow instructions like:

• Refactor class A and rename it to B
• Write a short function that does "this little job".
• Search for a method called C and change it so that it returns an integer
• Add a logging line to method F

Then you could determinate productivity by timing the time it takes to complete the changes.

But maybe there are other ways of achieving the same result, which I'm not aware of.

My goal is to determinate how productive I am using different IDEs, and even comparing how well two people can use their IDE of choice, possibly stopping a few discussions in the office, which would also increase productivity :)

Answer 1

This doesn't seem to be a particularly complicated problem.

• Collect metrics for performing various tasks.
• Normalise the results per person.
• Aggregate the results per IDE.

Are you asking what metrics to collect, or what task to perform?

The latter is largely irrelevant as long as it reflects 'day to day' tasks for people. The former is probably a mix of something like time taken, defects produced, LOC generated.

The Personal Software Process is an old, but easy, way to collect this sort of metric and there are plenty of digital tools that support it.

The biggest problem you'll face is gathering enough data for it to be statistically significant, and avoiding the 'I don't like this IDE so I am deliberately misreporting or misperforming when using it'.

...but if you want to show A is better/worse than B, the steps to take are the same:

• Collect baseline data.
• Collect test data.
• Compare baseline to test data.
• Generalize taking the error margin into account.