What types of Google Cloud application should I consider for something running as a continually available service that subscribes to a pub/sub?

Question

I want to create an application that will run as a service that subscribes to a Google Cloud pub/sub topic, and delivers emails based on 'alerts' that it receives through the pub/sub subscription. It will read from Cloud DataStore to determine the email addresses to mail to, and it will probably use some type of 3rd party service and API to do the actual mailing. In future, there will be a requirement to send other types of notification, for example SMS alerts.

What types of Google Cloud application should I consider for this? I'm not sure what type of application would be appropriate for something that effectively acts as a constantly available 'service'. Could I just create a normal Java application that subscribes to the topic in its 'main' method, and deploy this as an AppEngine app? Or are there more complex considerations I need to bear in mind? I've previously used DataFlow, which handles parallel processing of data very well, when subscribed to a pub/sub topic, however it's really designed for processing Big Data and I don't think would be appropriate.

Apologies for the noob question - I'm still learning to think in terms of Cloud architectures as opposed to traditionally running services. I'd really appreciate some basic guidance on what type of application I should be looking at creating.

Answer 1

In general GAE apps are intended to run as web services, in the sense that they're waiting for HTTP(S) requests and perform actions in response to them. While it might be possible to have them designed to register for pub/sub topics and immediately respond to them, achieving that may be a bit awkward and probably costly.

IMHO a better approach would be to implement the pub/sub event input portion as a cloud function, which has native support for Google Cloud Pub/Sub Triggers. Especially effective if the stream of input events is highly irregular - you would only pay per event (i.e. per cloud function invocation), not for an "always on" service capable to handle the highest peaks of events - Google takes care of that.

Depending on the number and complexity of the actions to be executed in response to the pub/sub events, they can be implemented either in the cloud function itself (in simple cases) or in a GAE app which would be triggered by the cloud function via HTTP(S) requests. See also When to choose App Engine over Cloud Functions?.

Such GAE app would still need to be able to handle the peaks of activity, but it wouldn't need to be an always-on service, it could scale down to zero in periods of no activity.

A side effect of such architecture is decoupling of the input events from actions, you can even re-distribute the actions as needed. For example you could delay sending some of the emails a bit to meet an email sending rate limit restriction, or implement features like group/summary emails or duplicate emails prevention.