How to figure out why my streaming data stops?

Question

This is a question about how I can find the failure point in a complicated process. (If you can figure out what is actually going wrong... wow.)

I am using QuickFix with Python 2.7 to connect to futures markets, and I am using pandas to deal with data and put it in dataframes etc. The process goes like this:

1. Connect to TT FIX Adapter, which provides exchange access.

2. Submit MarketDataIncrementalRefreshRequest, which results in streaming data (every time a trade is made, this is reported). As a result, the time between incoming messages during busy periods can be on the order of 10 milliseconds.

3. Each message is parsed, converted into a pandas dataframe, and concatenated with the preexisting dataframe for that market. The code for this:

   #df is dataframe of trades with 10 columns
   df.index = pd.to_datetime(df.TIME)
   #concatenate with prior data 
   #TS_DIC is a dictionary holding trade data for various markets
   try:
       df_prev = TS_DIC[market_key]
       TS_DIC[market_key] = pd.concat([df_prev,df])
   
   except:
       #in the case this is the first message received:
       TS_DIC[market_key] = df
   
   #now write to disk
   try: 
   #if file exists just write
       to_file = open('path/for/data', 'a+')
       df.to_csv(mode='a+', path_or_buf= to_file, header=False, index=False)
       to_file.close()
   except: 
   #create the file with headers and write
       to_file = open(path+name, 'wb')
       df.to_csv( path_or_buf= to_file, index=False)
       to_file.close()
   

This process works fine, sometimes for hours, sometimes for minutes, then it stops working. There are never any errors, it just stops. The result is data with gaps. I can make the process start again by doing step 2 again.

I would appreciate the help of anyone used to high-throughput data and maybe even these software packages.

What is likely to be the problem here? How do I figure out what is going wrong?

Answer 1

Figured I would let anyone who came across this question know how I sorted it out:

Printing the errors as suggested by the comments above did not actually help solve the problem. The data would still stop without printing errors. The reason was that sometimes QuickFix would get booted offline, for whatever reason, and would automatically log back on. Evidently this is something that FIX applications have to deal with.

What was happening: because I was initiating the data download manually, everytime I got booted offline, the data stopped. So by putting the request for data into the OnLogon function of QuickFix, I was able to make download requests repeat automatically whenever the program logged on.

This solved my problem. Thanks to CasualDemon and cpburnz.