Looks like I managed to access the raw data from the inline objects using the following code in the template:
<script>
var data = google.visualization.arrayToDataTable([
["Date", "Value"],
{% for form in inline_admin_formset %}
["{{form.original.date}}", {{form.original.value}}],
{% endfor %}
]);
</script>
So it looks like now I only need to parameterize what fields the graph is going to plot from the inline, or from the object (date and value in this case). Perhaps add a graph_fields() method to both.
In any case, thanks to whoever gave this a thought.