How to continuously read data from xively in a (python) heroku app?

Question

I am trying to write a Heroku app in python which will read and store data from a xively feed in real time. I want the app to run independently as a sort of 'backend process' to simply store the data in a database. (It does not need to 'serve up' anything for users (for site visitors).)

Right now I am working on the 'continuous reading' part. I have included my code below. It simply reads the datastream once, each time I hit my app's Heroku URL. How do I get it to operate continuously so that it keeps on reading the data from xively?

import os
from flask import Flask
import xively

app = Flask(__name__)

@app.route('/')
def run_xively_script():

   key = 'FEED_KEY'  
   feedid = 'FEED_ID'  


   client = xively.XivelyAPIClient(key) 
   feed = client.feeds.get(feedid)
   datastream = feed.datastreams.get("level")

   level = datastream.current_value


   return "level is %s" %(level)

I am new to web development, heroku, and python... I would really appreciate any help(pointers)

{ PS: I have read about Heroku Scheduler and from what I understand, it can be used to schedule a task at specific time intervals and when it does so, it starts a one-off dyno for the task. But as I mentioned, my app is really meant to perform just one function->continuously reading and storing data from xively. Is it necessary to schedule a separate task for that? And the one-off dyno that the scheduler will start will also consume dyno hours, which I think will exceed the free 750 dyno-hours limit (as my app's web dyno is already consuming 720 dyno-hours per month)... }

Answer 1

Using the scheduler, as you and @Calumb have suggested, is one method to go about this.

Another method would be for you to setup a trigger on Xively. https://xively.com/dev/docs/api/metadata/triggers/

Have the trigger occur when your feed is updated. The trigger should POST to your Flask app, and the Flask app can then take the new data, manipulate it and store it as you wish. This would be the most near realtime, I'd think, because Xively is pushing the update to your system.

Answer 2

This question is more about high level architecture decisions and what you are trying to accomplish than a specific thing you should do.

Ultimately, Flask is probably not the best choice for an app to do what you are trying to do. You would be better off with just pure python or pure ruby. With that being said, using Heroku scheduler (which you alluded to) makes it possible to do something like what you are trying to do.

The simplest way to accomplish your goal (assuming that you want to change minimal amount of code and that constantly reading data is really what you want to do. Both of which you should consider) is to write a loop that runs when you call that task and grabs data for a few seconds. Just use a for loop and increment a counter for however many times you want to get the data.

Something like:

for i in range(0,5):
    key = 'FEED_KEY'  
    feedid = 'FEED_ID'  
    client = xively.XivelyAPIClient(key) 
    feed = client.feeds.get(feedid)
    datastream = feed.datastreams.get("level")

    level = datastream.current_value

    time.sleep(1)

However, Heroku has limits on how long something can run before it returns a value. Otherwise the router will return a 503 or 500. But you could use the scheduler to then schedule this to run every certain amount of time.

Again, I think that Flask and Heroku is not the best solution for what it sounds like you are trying to do. I would review your use case and go back to the drawing board on what the best method to accomplish it our.