Approach to data-analysis [closed]

Question

Closed. This question needs to be more focused. It is not currently accepting answers.

---

Want to improve this question? Update the question so it focuses on one problem only by editing this post.

Closed 8 years ago.

Improve this question

I'm looking to write a reporting tool. The data resides in a ~6GB postgresql database. The application is an online store/catalog application that has items and orders. The stakeholders are requesting a feature that will allow them to search for an item and give a count of all those orders in the last 2 years.

Some rows contain quantities, and units of measure, which would require multiplication of quantity and UoM for each row.

It's also possible that other reporting functions will be necessary in the future.

I have not delved much into the data analysis aspect of programming. I enjoy Clojure, so I would be thrilled to find a solution that uses Clojure, but only if Clojure offers competitive tools for my needs.

Here are some options I'm considering:

• merely SQL
• Clojure
  • core.reducers
  • a clojure hadoop library
• Hadoop

Can anyone shed some insight into these kinds of problems for me? Are there articles that you would recommend?

Answer 1

Hadoop is likely overkill for this project. It seems most likely that simply using Clojure-jdbc or Korma to read the data form the database and filter/reduce it in Clojure is likely to be fine. At work we routinely work with sequences of that size, though this depends on the expected response time. You may need to do some preprocessing and caching if instantaneous responses are expected.