Email - Process Ads website emails with custom solution or using Celery/RabbitMQ?

Question

I need to process e-mails for a medium project that will be expectecting to send 20 or 30 emails per hour.

I've designed a solution in other project that uses a database table and a cronjob running every 5 or 10 minutes to handle this. The Database table is very simple. Looks like this:

CREATE TABLE "atem_emails_envios" (
    "id_email_envio" int4 NOT NULL,
    "id_email_msg" varchar(20) NOT NULL,
    "dat_inserted" timestamp NOT NULL,
    "dat_sended" timestamp,
    "try_number" int4,
    "max_tries" int4 NOT NULL,
    "email_from" varchar(500) NOT NULL,
    "email_to" varchar(500) NOT NULL,
    "email_cc" varchar(500),
    "email_bcc" varchar(500),
    "email_subject" varchar(500) NOT NULL,
    "email_msg" text NOT NULL,
    "error_msg" text,
    "i_started" timestamp,
    "pid" int4,
    "coment" varchar(2000),
    "id_utiliz_ins" varchar(45),
    "id_utiliz_upd" varchar(45),
    "data_ult_actual" timestamp,
  PRIMARY KEY("id_email_envio"),
  CONSTRAINT "Ref_atem_emails_envios_to_atem_mensagens_email" FOREIGN KEY ("id_email_msg")
    REFERENCES "atem_mensagens_email"("id_email_msg")
    MATCH SIMPLE
    ON DELETE NO ACTION
    ON UPDATE NO ACTION
    NOT DEFERRABLE
);

When an e-mail is being processed I just store the PID to the table to avoid collisions.

My question goes in this direction. I've been using this table to process e-mails in a website with low traffic, and works well. What advantages I have using a Queue manager like Celery and a broker like RabbitMQ? It seems to me that I will add another layer of complexity. What benefits I will gain using a solution like Celery/RabbitMQ?

Please give me some clues.

Best Regards,

Answer 1

Well like the old adage "it it ain't broke, don't fix it". Doesn't sound like you really need to complicate the architecture if you aren't planning on scaling or worrying about the current CRON/PG setup.

That said, there are benefits to using an async framework like Celery. To name a few:

• You may very well have future use cases that can take advantage of an async system (SMS sending, user processing, integration, cache warming, webhooks, etc.)
• easier to maintain if/when you scale as it's well documented and has a large community
• greater visibility and control

As for complexity, we built an integration to Celery so you can replace the broker with our cloud message queue service IronMQ. Check out http://Iron.io/celery. This will help reduce your complexity and broker point of failure by replacing RabbitMQ.

Hope this helps!