Pause Python program on the fly (and resume)

Question

I am building a machine learning algorithm(like neural network) where class variables(i.e numpy matrices) represent various parameters of the system

Training the system is done by iteratively update all class variables. The more iterations the better. I want to get up every morning and check the class variables. After that I want to resume the program

I am calling the program in an interactive terminal. Here is what I can think of:

1. Print to terminal -> matrices too large, won't be helpful
2. save to disk and load in another terminal
3. set_trace(), but requires knowing when to pause beforehand

Is it possible to pause the program on the fly and play with the class variables and then resume ?

If anyone needs more details, the program is here: github link

Answer 1

I'm not familiar with numpy, but here is a simple class that can stop and resume:

class Program():

    def run(self):
        while 1:
            try:
                self.do_something()
            except KeyboardInterrupt:
                break

    def do_something(self):
        print("Doing something")


# usage:

a = Program()
a.run()

# will print a lot of statements

# if you hit CTRL+C it will stop
# then you can run it again with a.run()

Answer 2

What if you modify the model.do_EM() method to save the current state at each step and check a configuration file?

 def do_EM(self, n_iteration = 10):
    self.visualizer.visualize(self.param_alpha, self.param_mu, self.param_sigma)
    for i in range(n_iteration):
      print "iteration:", i 
      self.step_E()
      print "done step_E. ",
      self.step_M()
      print "done step_M. "
      self.visualizer.visualize(self.param_alpha, self.param_mu, self.param_sigma)

      # Save current state
      self.log.write( ... )
      # Check for config changes
      self.config.update( ... )