Python Profiling In Windows, How do you ignore Builtin Functions

Question

I have not been capable of finding this anywhere online. I was looking to find out using a profiler how to better optimize my code, and when sorting by which functions use up the most time cumulatively, things like str(), print, and other similar widely used functions eat up much of the profile. What is the best way to profile a python program to get the user-defined functions only to see what areas of their code they can optimize?

I hope that makes sense, any light you can shed on this subject would be very appreciated.

Answer 1

OK, I assume your real goal is to make your code as fast as reasonably possible, right?

It is natural to assume you do that by finding out how long your functions take, but there is another way to look at it.

Consider as your program runs it traces out a call tree, which is kind of like a real tree outside your window. The trunk is like the main function, and where any branch splits off from it is like calling another function.

Suppose each "leaf" takes a certain amount of time, and what you want to do is prune the tree so as to remove as many leaves as possible.

One way is to find branches with lots of leaves and cut off the leaves. Another way is to cut off whole branches if you don't need them. The problem is to find heavy branches that you don't need.

One bone-simple way to do this is to pick several leaves at random, like 10, and on each one, trace a line down its branch all the way back to the trunk. Any branch point will have some number of these lines running through it, from leaf to trunk. The more lines run through that branch point, the more leaves are on that branch, and the more you could save by pruning it.

Here's how you can apply this to your program. To sample a leaf, you pause the program at random and look at the call stack. That is the line back to the trunk. Each call site on it (not function, call site) is a branch point. If that call site is on some fraction of samples, like 40%, then that is roughly how much you could save by pruning it.

So, don't think of it as measuring how long functions take. Think of it as asking which call sites are "heavy". That's all there is to it.