Twitter style trends with php/mysql

Question

I am coding a social network and I need a way to list the most used trends, All statuses are stored in a content field, so what it is exactly that I need to do is match hashtag mentions such as: #trend1 #trend2 #anothertrend

And sort by them, Is there a way I can do this with MySQL? Or would I have to do this only with PHP?

Thanks in advance

Answer 1

I think it is better to store tags in dedicated table and then perform queries on it. So if you have a following table layout

trend | date

You'll be able to get trends using following query:

SELECT COUNT(*), trend FROM `trends` WHERE `date` = '2012-05-10' GROUP BY trend

18  test2
7   test3

Answer 2

The maths behind trends are somewhat complex; machine learning may be a bit over the top, but you probably need to work through some examples.

If you go with @deadtrunk's sample code, you would miss trends that have fired up in the last half hour; if you go with @eggyal's example, you miss trends that have been going strong all day, but calmed down in the last half hour.

The classic solution to this problem is to use a derivative function (http://en.wikipedia.org/wiki/Derivative); it's worth building a sample database and experimenting with this, and making your solution flexible enough to change this over time.

Whilst you want to build something simple, your users will be used to trends, and will assume it's broken if it doesn't work the way they expect.

Answer 3

You should probably extract the hash tags using PHP code, and then store them in your database separately from the content of the post. This way you'll be able to query them directly, rather then parsing the content every time you sort.

Answer 4

1. Create a table that associates hashtags with statuses.

2. Select all status updates from some recent period - say, the last half hour - joined with the hashtag association table and group by hashtag.

3. The count in each group is an indication of "trend".